This publication was produced for review by the United States Agency for International Development. It was prepared by Tetra Tech. FINAL EVALUATION REPORT: IMPACT EVALUATION OF THE KENYA RESILIENT ARID LANDS PARTNERSHIP FOR INTEGRATED DEVELOPMENT ACTIVITY JULY 2021 ACKNOWLEDGEMENTS The evaluation team thanks the families, water managers, and local leaders in the counties who shared their time to participate in this study. The National Drought Management Authority and the Kenyan Ministry of Water and Sanitation both deserve praise for their assistance, cooperation, and patience with the evaluation process and team members. Special thanks go to the data collection teams from Ipsos Public Affairs Kenya for conducting the water borehole surveys and downloading data from each borehole. The team is grateful to the Kenya Resilient Arid Lands Partnership for Integrated Development Activity staff for their continued collaboration, cooperation, and communication. The team could not have implemented the evaluation without the ongoing and patient cooperation of SweetSense staff, particularly Evan Thomas and Styvers Kathuni, as well as Pranav Chintalapati from the University of Colorado at Boulder. The team also acknowledges Mary Gichuhi for her quality oversight during data collection, translation support, and general guidance as the local coordinator for this evaluation. Finally, the evaluation team appreciates the technical review and support for this report from Tetra Tech’s Siobhan Girling and Morris Israel. The team also thanks Elizabeth Jordan (USAID/RFS/CW), Vicky Liyai (USAID/KEA), Martin Mulongo (USAID/KEA), Walter Ong’eng’a (USAID/KEA), Amanda Robertson (USAID/KEA), Albert Waudo (USAID/KEA), and Nicholas Owuor (USAID/KEA) for their continued support of this study. Prepared by: Dr. Katherine Dickinson (Principal Investigator, University of Colorado) Jacob Patterson-Stein (Evaluation Coordinator, MSI, A Tetra Tech Company) Cover Photo: Camels drink from strategic borehole in Kenya, Jacob Patterson-Stein, MSI. Prepared for the United States Agency for International Development by the Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) project under Task Order number AID-OAA-TO-16-00016 of the Water and Development Indefinite Delivery Indefinite Quantity Contract (WADI IDIQ), contract number AID-OAA-I-14-00068. Tetra Tech Contacts: Morris Israel, Project Director morris.israel@washpals.org Jeff Albert, Deputy Project Director jeff.albert@washpals.org Lucia Henry, Project Manager lucia.henry@tetratech.com Tetra Tech 1320 North Courthouse Road, Suite 600, Arlington, VA 22201 Tel: 703 387-2100, Fax: 703-414-5593 www.tetratech.com/intdev FINAL EVALUATION REPORT IMPACT EVALUATION OF THE KENYA RESILIENT ARID LANDS PARTNERSHIP FOR INTEGRATED DEVELOPMENT ACTIVITY JULY 2021 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY i TABLE OF CONTENTS TABLE OF CONTENTS................................................................................................................................................... I LIST OF FIGURES............................................................................................................................................................. III LIST OF TABLES.............................................................................................................................................................. IV ACRONYMS....................................................................................................................................................................... V EXECUTIVE SUMMARY................................................................................................................................................ VI METHODS .......................................................................................................................................................... VI RESULTS ............................................................................................................................................................. VII CONCLUSIONS............................................................................................................................................... VII RECOMMENDATIONS................................................................................................................................. VIII 1. INTRODUCTION............................................................................................................................................... 1 2. KENYA RAPID ACTIVITY BACKGROUND ............................................................................................... 2 2.1. NATIONAL CONTEXT......................................................................................................................... 2 2.2. KENYA RAPID ACTIVITY DESCRIPTION........................................................................................ 2 2.3. DEVELOPMENT HYPOTHESIS AND THEORY OF CHANGE................................................... 3 3. EVALUATION PURPOSE, AUDIENCES, AND USES ................................................................................ 4 3.1. PURPOSE .................................................................................................................................................... 4 3.2. AUDIENCE ................................................................................................................................................. 5 3.3. INTENDED USE........................................................................................................................................ 5 4. EVALUATION DESIGN..................................................................................................................................... 6 5. DATA COLLECTION ........................................................................................................................................ 9 5.1. BOREHOLE ASSET SURVEY ................................................................................................................. 9 5.2. BOREHOLE SUBSTITUTES.................................................................................................................. 10 5.3. SENSOR DATA....................................................................................................................................... 10 5.4. QUALITATIVE DATA ........................................................................................................................... 11 5.4.1. Key Informant Interviews........................................................................................................ 14 5.4.2. Focus Group Discussions........................................................................................................ 14 5.4.3. Qualitative Data Analysis Methods....................................................................................... 15 6. FINDINGS ............................................................................................................................................................ 16 6.1. EVALUATION QUESTION 1: DOES THE INTERVENTION USING REAL-TIME REMOTE SENSING DATA OF WATER POINTS FOR STRATEGIC BOREHOLE MANAGEMENT IN KENYA RAPID COUNTIES LEAD TO INCREASED ON-TIME OF STRATEGIC BOREHOLES DURING THE DROUGHT SEASON?............................................................... 16 6.1.1. Borehole Sensor and Context Descriptive Analysis ........................................................ 16 6.1.2. Impact Analysis.......................................................................................................................... 22 6.1.3. EQ1 conclusions........................................................................................................................ 30 FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY ii 6.2. EVALUATION QUESTION 2: HOW DO WATER MANAGERS PERCEIVE THE IMPACT OF SENSOR-BASED SYSTEMS ON THEIR ABILITY TO ADDRESS BOREHOLE FUNCTIONALITY AND HOW DOES THIS COMPARE TO PERCEPTIONS OF BOREHOLE FUNCTIONALITY IN NON-KENYA RAPID COUNTIES?............................ 31 6.2.1. Key Informant Interview Sample Characteristics.............................................................. 31 6.2.2. Borehole Failures and Perceptions of the Sensor-Based System.................................. 32 6.2.3. Water Managers’ Perceptions of Water System Functionality...................................... 34 6.2.4. Perceptions of Water System Management Structure and Roles................................. 36 6.2.5. Resources for Borehole Repairs........................................................................................... 37 6.2.6. EQ2 conclusions........................................................................................................................ 39 6.3. EVALUATION QUESTION 3: DO KENYA RAPID’S SENSOR-BASED SYSTEMS AFFECT USER PERCEPTIONS OF BOREHOLE FUNCTIONALITY AND ACCESS?...................... 40 6.3.1. Focus Group Discussion Sample Characteristics.............................................................. 40 6.3.2. Users’ perceptions of Water System Functionality.......................................................... 41 6.3.3. Borehole Access and Use ....................................................................................................... 43 6.3.4. Perceptions of Management Structure ................................................................................ 44 6.3.5. EQ3 conclusions........................................................................................................................ 47 6.4. COVID-19 IMPACTS ON WATER USE AND MANAGEMENT............................................... 47 7. CONCLUSIONS ................................................................................................................................................ 49 7.1. EQ1 CONCLUSIONS............................................................................................................................ 49 7.2. EQ2 CONCLUSIONS............................................................................................................................ 49 7.3. EQ3 CONCLUSIONS............................................................................................................................ 49 7.4. OVERALL CONCLUSIONS................................................................................................................. 50 8. RECOMMENDATIONS ................................................................................................................................... 51 ANNEXES ......................................................................................................................................................................... 52 ANNEX A: EVALUATION STATEMENT OF WORK.......................................................................................... 53 ANNEX B: COMPARISON COUNTY FOLLOW-UP BOREHOLE ASSET SURVEY INSTRUMENT...... 60 ANNEX C: REFERENCES ............................................................................................................................................. 72 ANNEX D: STATISTICAL POWER........................................................................................................................... 74 ANNEX E: COVID-19 CONSIDERATIONS ........................................................................................................... 76 ANNEX F: INTERVIEW CONSENT SCRIPTS ........................................................................................................ 77 ANNEX G: INTERVIEW GUIDES .............................................................................................................................. 80 ANNEX H: ANONYMIZED RESPONDENT LISTS............................................................................................... 93 ANNEX I: ADDITIONAL ANALYTICAL DETAILS............................................................................................... 95 ANNEX J: MATCHING PROCESS AND BALANCE ......................................................................................... 105 ANNEX K: IMPLEMENTATION FIDELITY........................................................................................................... 109 ANNEX L: ROUND 1 AND ROUND 3 COMPARISON................................................................................. 110 FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY iii LIST OF FIGURES FIGURE 1: THEORY OF CHANGE FOR THE KENYA RAPID ACTIVITY’S REMOTE SENSOR INTERVENTION 4 FIGURE 2: TREATMENT AND COMPARISON COUNTIES FIGURE 3: SAMPLE NDMA BULLETIN FIGURE 4: ROUND II BOREHOLE ON-TIME BY COUNTY 7 10 12 FIGURE 5: DAILY ROUND II ON-TIME FOR SELECTED QUALITATIVE COUNTIES AND BOREHOLES 14 FIGURE 6: AVERAGE BOREHOLE ON-TIME BY ASSIGNMENT 18 FIGURE 7: DISTRIBUTION OF DAILY ON-TIME BY ASSIGNMENT AND ROUND 19 FIGURE 8: DISTRIBUTION OF WEEKLY BOREHOLE ON-TIME BY ASSIGNMENT AND ROUND 19 FIGURE 9: OVERALL AVERAGE WEEKLY ON-TIME PERCENTAGE BY ASSIGNMENT 19 FIGURE 10: MAP OF BOREHOLES AS PERCENT OF WATER SOURCES 20 FIGURE 11: AVERAGE RAINFALL BY MONTH FOR TREATMENT AND COMPARISON COUNTIES 22 FIGURE 12: AVERAGE MONTHLY RAINFALL AND BOREHOLE WATER SOURCES 23 FIGURE 13: MODEL 1 ESTIMATES FOR ALL AND DRY MONTHS 27 FIGURE 14: MODEL 1 RESULTS FOR 2019 DATA 27 FIGURE 15: MODEL 2 ESTIMATES 28 FIGURE 16: MODEL 2 RESULTS FOR 2019 29 FIGURE 17: PROBABILITY OF BOREHOLE PUMPS REMAINING ACTIVITY FOR AT LEAST AN HOUR 30 32 33 38 42 42 46 56 BY WEEK BY TREATMENT STATUS FIGURE 18: FREQUENCY OF THEMATIC CODES FOR KIIS FIGURE 19: WATER SYSTEM THEMES BY ASSIGNMENT FIGURE 20: KII REPAIR THEMES AND MENTIONS FIGURE 21: FUNCTIONALITY THEMES BY ASSIGNMENT FIGURE 22: WATER SYSTEM FUNCTIONALITY SENTIMENT POLARITY BY COUNTY FIGURE 23: TEXT CORRELATION WITH MENTIONS OF WOMEN FIGURE A-1: KENYA RAPID THEORY OF CHANGE FIGURE D-1: STATISTICAL POWER BY EFFECT SIZE 77 FIGURE I-1: BOREHOLE PUMP ON-TIME BY COUNTY AND MONTH FOR ROUNDS II AND III 97 FIGURE I-2: NUMBER OF BOREHOLES THAT LEFT THE SAMPLE BY ASSIGNMENT AND COUNTY99 FIGURE I-3: AVERAGE BOREHOLE WATER SOURCE PERCENTAGE AND RAINFALL BY COUNTY 100 FIGURE I-4: MODEL SENSITIVITY TEST RESULTS 101 FIGURE I-5: MODEL 1 RESULTS WITH AND WITHOUT CLUSTERING FOR THE OPTIMALLY MATCHED SAMPLE 102 FIGURE I-6: ROUND II MODEL 1 ANALYSIS 103 FIGURE I-7: ROUND II MODEL 1 REANALYSIS 104 FIGURE I-8: ROUND II MODEL 2 WEEKLY ON-OFF ANALYSIS 104 FIGURE I-9: MODEL 3 HAZARD RATIO RESULTS 105 FIGURE I-10: SENTIMENT ANALYSIS FOR FGDS 106 FIGURE J-1: UNMATCHED AND MATCHED SAMPLES WITH NEW MODEL INPUTS 107 FIGURE J-2: BALANCE ACROSS KEY VARIABLES WITH REVISED MATCHING PARAMETERS 106 FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY iv FIGURE J-3: NEAREST NEIGHBOR PROPENSITY SCORE DISTRIBUTIONS BEFORE AND AFTER MATCHING 107 FIGURE J-4: NEAREST NEIGHBOR COVARIATE BALANCE ACROSS MATCHED AND UNMATCHED SAMPLES 108 FIGURE J-5: TWO-TO-ONE PROPENSITY SCORE DISTRIBUTIONS BEFORE AND AFTER MATCHING 110 FIGURE J-6: TWO-TO-ONE COVARIATE BALANCE ACROSS MATCHED AND UNMATCHED SAMPLES LIST OF TABLES TABLE 1: SAMPLED COUNTIES 6 TABLE 2: CHARACTERISTICS OF BOREHOLES SELECTED FOR QUALITATIVE DATA COLLECTION 14 TABLE 3: REPORTED FEES BY BOREHOLE 38 TABLE A-1. EVALUATION DELIVERABLES 59 TABLE D-1: EX ANTE MINIMUM DETECTABLE EFFECT SIZE (MDES) ESTIMATES 76 TABLE D-2: MDES ESTIMATES BASED ON BOREHOLE SAMPLE SIZES 76 TABLE H-1: KII LIST 95 TABLE H-2: FGD LIST 95 TABLE I-1: ROUND III COMPARISON BOREHOLE DATA COLLECTION SUMMARY 98 TABLE K-1: ICT INTERVENTION IMPLEMENTATION SUMMARY 112 TABLE L-1: SUMMARY OF QUALITATIVE THEMES, FINDINGS, AND CONCLUSIONS 113 108 FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY v ACRONYMS ASAL Arid and Semi-Arid Lands CAPI Computer-Assisted Personalized Interview CI Confidence interval COVID-19 Coronavirus Disease 2019 CHIRPS Climate Hazards Group InfraRed Precipitation with Station E3 Bureau for Economic Growth, Education, and Environment (USAID) EDE Ending Drought Emergency EQ Evaluation Question FGD Focus Group Discussion GARWASCO Garissa Rural Water and Sanitation Company FGD Focus Group Discussion GPS Global Positioning System ICT Information and Communication Technology IE Impact Evaluation KEA Kenya and East Africa KEMRI Kenya Medical Research Institute KII Key Informant Interview MWA Millennium Water Alliance NACOSTI National Commission for Science, Technology, and Innovation NDMA National Drought Management Authority NGO Nongovernmental Organization NRC National Research Council Canada Kenya-RAPID Kenya Resilient Arid Lands Partnership for Integrated Development RE-AIM Reach, Effectiveness, Adoption, Implementation, and Maintenance RFS Bureau for Resilience and Food Security SDC Swiss Development Corporation SO Strategic Objective SOW Statement of Work TAWASCO Tana Water and Sanitation Company WASCO Water and Sanitation Company WASH Water, Sanitation, and Hygiene WASHPaLS Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability USAID United States Agency for International Development FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY vi EXECUTIVE SUMMARY This document presents the results of an impact evaluation (IE) of the Kenya Resilient Arid Lands Partnership for Integrated Development Activity’s (Kenya RAPID) borehole sensor intervention. A five￾year (2015–2020), US$35.5 million activity implemented under a Global Development Alliance (USAID agreement number AID-615-A-15-00008), Kenya RAPID was co-funded by USAID, the Swiss Development Corporation, private sector partners, and the Millennium Water Alliance (MWA) and its sub-recipients. 1, 2 The intervention component of Kenya RAPID that the team evaluated involved installation of sensors on a set of “strategic” boreholes—defined as those deemed critical during periods of drought—in order to track pump functionality and share the information with local water managers and officials. The IE investigated whether this intervention increased borehole functionality in counties served by Kenya RAPID (Garissa, Isiolo, Marsabit, Turkana, and Wajir) compared to a set of non-RAPID counties in northern and central Kenya. In parallel, the team conducted a qualitative analysis of water managers’ and water users’ perceptions of the intervention and water management issues more generally. Kenya RAPID’s theory of change for this activity envisioned that improved functionality of strategic boreholes would result from 1) installation of sensors, with data sharing through mobile applications and online dashboards as well as accompanying training on sensor data use; 2) establishment of county operations and maintenance teams; and 3) a dedicated budget for strategic borehole repairs. The evaluation team designed the IE to answer three specific evaluation questions (EQs): • EQ 1: Does the intervention using real-time remote sensing data of water points for strategic borehole management in Kenya RAPID counties lead to increased on-time of strategic boreholes during the drought season?3 • EQ 2: How do water managers perceive the impact of sensor-based systems on their ability to address borehole functionality, and how does this compare to perceptions of borehole functionality in non-Kenya RAPID counties? • EQ 3: Do Kenya RAPID’s sensor-based systems affect user perceptions of borehole functionality and access? METHODS To answer EQ 1, the team employed a quasi-experimental matching study design. To provide a counterfactual for analyzing the effects of the sensor-based systems installed on strategic boreholes in Kenya RAPID counties, evaluators installed “comparison” sensors on strategic boreholes in a set of counties not covered by Kenya RAPID (Baringo, Kitui, Laikipia, Mandera, Meru, Samburu, Tana River, and West Pokot). The non-RAPID boreholes were chosen to create a comparison sample similar to the intervention boreholes in key observable characteristics, such as power type, tariff scheme, and livestock use. To estimate the sensor-based intervention’s effects, the team ran statistical models comparing the on-time of Kenya RAPID boreholes to comparison boreholes. 1 A total of $35.5 million was invested through Kenya RAPID: $12.5 million from USAID, $12.5 million in leveraged funds from private sector partners, $7.5 million from SDC, and $3 million in cost share from MWA and its sub-recipients. 2 The MWA members for Kenya RAPID include CARE, Catholic Relief Services, Food for the Hungry, and World Vision. SweetSense and IBM Research are private sector sub-recipients under this award. Other private sector partners include the Coca Cola Foundation, Acacia Water, and KCB Foundation. 3 “On-time” is a measure of the borehole pump running. It is defined as the time within a 24-hour period that the borehole pump sensor recorded the borehole pump as running: for example, a value of 50 percent would indicate the borehole pump ran for 12 out of 24 hours. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY vii To answer EQs 2 and 3, the evaluators conducted i) 16 key informant interviews (KIIs) with water officers, operations and maintenance managers, water engineers and technicians, and borehole operators; and ii) eight focus group discussions (FGDs) among water users. Both the KIIs and FGDs took place in two Kenya RAPID counties (Garissa and Turkana) and two comparison counties (Tana River and West Pokot). RESULTS For EQ 1 (sensor effects on borehole on-time): Across multiple analyses, results suggest that, on average, Kenya RAPID did not have a significant impact on borehole pump on-time during the drier months of the intervention (2018–2020) relative to comparison county strategic boreholes. After controlling for borehole characteristics and rainfall, the difference between Kenya RAPID and non￾Kenya RAPID dry month strategic borehole pump on-time was much smaller than anticipated during the design phase (e.g., 35% effect for a sensor-based intervention in Rwanda [Nagel et al., 2015]). At most, the IE team’s model estimates suggest that the sensor-based intervention resulted in less than an hour of additional borehole pump-on time per day in Kenya RAPID counties compared to non-RAPID counties. For EQ 2 (water manager perceptions of sensor impacts): Water managers reported similar timelines for repairs in RAPID and comparison countries. While county- and sub-county-level water managers in Kenya RAPID counties viewed the sensor-based system favorably and said it provided useful data to support water management activities, they also pointed out that lack of access to resources for repairs continues to limit borehole functionality in Kenya RAPID counties. Officials in Garissa (Kenya RAPID county) reported that they did not yet have full access to the data dashboard, and others reported that a lack of office internet and issues with network connectivity for using mobile devices limited access to the data dashboard. Perhaps most importantly, a lack of dedicated resources for borehole repairs remains a key barrier to improved functionality in both treatment and comparison counties. Water user fees are collected at most of the selected boreholes, but large repairs require funds from the county, national government, or nongovernmental organizations, and the process for obtaining these funds is complicated. Local managers in both RAPID and comparison counties also lack spare parts and technical capacity to maintain and repair water systems. For EQ 3 (user perceptions of borehole functionality): Given the lack of a clear impact on borehole on-time and the broader challenges confronting water managers, the sensor-based intervention did not improve users’ perceptions of borehole functionality and water access in Kenya RAPID counties as compared to comparison counties. All strategic borehole users continued to identify a range of water access and supply issues not directly addressed by the intervention, including breakages in distribution pipes and taps bringing water from boreholes to people’s homes and villages. CONCLUSIONS Improving water service delivery is a challenge that includes technical, social, economic, and political components. The Kenya RAPID’s information and communication technology (ICT) intervention component introduced an innovative technological solution to one part of the problem: lack of timely information about strategic borehole breakages. Taken together, the results of this IE show that information provision alone—without effective solutions to a broader range of social, economic, and political management challenges—had at best a small impact on strategic borehole functionality. Increased attention to the social, economic, and political context in which technical solutions operate is imperative to realize the full potential of these tools and uncover more effective water management solutions. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY viii RECOMMENDATIONS The Kenya RAPID borehole sensor intervention was unable to compensate for inadequate investment in the areas that pose the biggest constraints facing water managers for achieving major improvements in functionality and improving water users’ experience. Based on the findings and conclusions for each EQ, the evaluation team offers the following recommendations to USAID: 1. Continue to focus on water system governance, clarifying roles and responsibilities for water management and establishing dedicated and sustainable funding sources for water system maintenance and repairs. 2. Address community concerns carefully in planning for delivery of water services. The team’s results revealed a number of problems cited by users that were not directly addressed by the sensor-based intervention. 3. Consider rural water ICT intervention costs and context. Use of ICT to collect and share information in some contexts may be worthwhile. USAID should consider implementation costs and systemic challenges in thinking about the theory of change for ICT interventions given large structural constraints, such as limited budgets and climate change. 4. For future evaluation efforts, make sure that implementation monitoring is included as a key, funded component, following established guidelines such as the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework (Glasgow et al., 1999). Sparse implementation data on specific activities outside of the ICT intervention and detailed budget information limited the team’s ability to track progress over time. These data are key to understanding how and why impacts (or lack of impacts) are observed. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 1 1.0 INTRODUCTION This report presents findings from the impact evaluation (IE) of the Kenya Resilient Arid Lands Partnership for Integrated Development Activity’s (Kenya RAPID) borehole sensor intervention, commissioned by the United States Agency for International Development (USAID)/Kenya and East Africa (USAID/KEA) and the Center for Water Security, Sanitation, and Hygiene in USAID’s Bureau for Resilience and Food Security (RFS).4 USAID’s Bureau for Economic Growth, Education, and Environment (E3) Analytics and Evaluation Project designed the evaluation, and the USAID Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability Project (WASHPaLS) implemented it. The intervention component of Kenya RAPID that the team evaluated involved installation of sensors on a set of “strategic” boreholes—defined by water managers and county officials as those deemed critical during periods of drought—in order to track pump functionality and share the information with local water managers and officials. The IE investigated whether this intervention increased borehole functionality in counties served by Kenya RAPID (Garissa, Isiolo, Marsabit, Turkana, and Wajir) compared to a set of non-RAPID counties in northern and central Kenya. The evaluation incorporates a quasi-experimental matching design to rigorously test how remote sensing technology and information sharing affect how long water borehole pumps are on (i.e., pump on-time) and borehole pump management. Additionally, the IE team conducted a qualitative analysis of water managers’ and water users’ perceptions of the intervention and water management issues more broadly. The statement of work (SOW) for the evaluation is provided Annex A. This document provides detailed results of the IE, which was implemented in three annual rounds of data collection and analysis from 2018 through 2020. The report begins with background on Kenya RAPID, followed by details of the evaluation purpose and design, including a description of data collection methods for both quantitative and qualitative analyses. Next the report provides in-depth results corresponding to the IE’s three key evaluation questions (EQs). Specifically, the report presents findings from the quasi-experimental design and analysis comparing functionality of boreholes in Kenya RAPID counties with boreholes in the non-RAPID counties. Findings from key informant interviews (KIIs) with water managers at the local, county, and sub-county level are summarized to assess perceptions of the sensor-based intervention and perspectives on water management challenges. The report also summarizes results from focus group discussions (FGDs) with local water users in Kenya RAPID and comparison counties to assess perceptions of borehole functionality. While not part of the initial design, the onset of the Coronavirus Disease 2019 (COVID-19) pandemic led the evaluation team to collect qualitative data on the perceived impacts of the pandemic on water access, use, and management issues. The report concludes with a summary of key findings of the IE and implications for the use of information and communication technology (ICT) to improve water service delivery. 4 USAID restructured since the initiation of this evaluation, and the relevant office is now referred to as the Center for Water Security, Sanitation, and Hygiene in the Bureau of Resilience and Food Security (RFS). FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 2 2.0 KENYA RAPID ACTIVITY BACKGROUND 2.1 NATIONAL CONTEXT Over the past decade, Kenya has gone through a period of major institutional reform, including the devolution of authority and resources from the national government to newly elected county governments. County governments now have the political mandate and more autonomy over use of funds to provide water to their communities; however, they are relatively new institutions with limited operational capacity. As part of these developments, the Government of Kenya launched its “Common Programme Framework to End Drought Emergencies,” which arose from a series of meetings with development partners between 2013 and 2014.5 The institutional framework for water management in Kenya consists of multiple stakeholders, with counties operating at the regional and local levels. The Common Programme includes the Ending Drought Emergencies (EDE) Initiative to better align stakeholders involved in drought mitigation and water management across all levels of government. The EDE initiative is a framework to improve targeting and coordination with the goal of reducing the impact of droughts, early warning and response, and institutional capacity for climate resilience (Kenya, 2014). Challenges to the provision of sustainable and reliable water service, coupled with a changing institutional environment and scarce water resources in Kenya, reinforce the need for stronger and more accountable institutions, enhanced coordination and integration of development programs across sectors, private sector participation, and empowered communities with the knowledge and ability to exercise rights and responsibilities regarding water resources. Kenya RAPID aimed to tackle these challenges. 2.2 KENYA RAPID ACTIVITY DESCRIPTION Kenya RAPID was a five-year activity implemented under a Global Development Alliance (USAID agreement number AID-615-A-15-00008) co-funded by USAID, the Swiss Development Corporation (SDC), private sector partners, and the Millennium Water Alliance (MWA) and sub-recipients.6 USAID awarded the activity in 2015 to MWA, a consortium of non-profit water-related organizations, to build on the successes and lessons learned from USAID’s Kenya Arid Lands Disaster Risk Reduction – Water, Sanitation, and Hygiene (WASH) Program and SDC’s Water for Livestock Program. Both programs were implemented in northern Kenya from 2012 to 2014.7 Kenya RAPID sought to contribute to sustainable and resilient livelihoods for communities in Kenya’s arid and semi-arid lands (ASALs) by improving water availability and water service delivery to people and livestock and by improving rangelands in those regions within Marsabit, Garissa, Isiolo, Wajir, and Turkana counties. Three strategic objectives (SOs) guided the activity toward the overall goal of sustainable and resilient livelihoods for communities in the ASALs: • SO 1: A responsive and accountable governance framework is in place and operational at the county government level that ensures sustainable provision of water and pasture; • SO 2: Replicable and scalable business models for sustainable WASH and livestock service delivery have been developed and operationalized; and 5 See: http://www.ndma.go.ke/index.php/resource-center/ede-reports/send/43-ending-drought-emergencies/4251-common-programme￾framework 6 A total of $35.5 million was invested through Kenya RAPID: $12.5 million from USAID, $12.5 million in leveraged funds from private sector partners, $7.5 million from SDC, and $3 million in cost share from MWA and its sub-recipients. 7 The MWA members for Kenya RAPID include CARE, Catholic Relief Services, Food for the Hungry, and World Vision. SweetSense and IBM Research are private sector sub-recipients under this award. Other private sector partners include the Coca Cola Foundation, Acacia Water, and KCB Foundation. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 3 • SO 3: Communities have increased access to sustainable WASH services and improved rangeland management. Through one sub-activity under these SOs, Kenya RAPID committed to making data and ICT tools available and accessible to improve decision-making for better water service delivery. The sub-activity, referred to in this report as the ICT intervention, installed approximately 400 sensors to measure borehole pump functionality and flag system failures on water boreholes. Of these 400 sensors, 69 are in areas identified as “strategic” by local authorities due to the risk of drought in those areas and the subsequent importance of the water boreholes. Through the ICT intervention, Kenya RAPID developed customized data dashboards for each county to display water borehole status in near real time. The sensors transmit data to the dashboards to track whether or not a borehole pump was turned on.8 SweetSense, one of Kenya RAPID’s consortium members, processed the sensor data, complemented it with near real-time survey information obtained via mobile phone surveys of local borehole operators or staff deployed in the local region when notable changes to operations were identified, and used the information to make inferences about causes for usage disruptions and changes. Kenya RAPID made these data accessible to relevant authorities such as county governments and the appropriate service providers. Throughout this report, reference to “Kenya RAPID” specifically refers to the ICT intervention unless otherwise noted. 2.3 DEVELOPMENT HYPOTHESIS AND THEORY OF CHANGE The theory of change envisions that if the Kenya RAPID activity: 1) installs sensors on strategic boreholes, shares the data through mobile applications and online dashboards, and provides training on sensor data use; 2) supports the development of county operations and maintenance teams; and 3) facilitates a dedicated budget for strategic borehole repairs, then this will lead to increased strategic borehole functionality, including more borehole pump on-time during critical drought periods and reduced drought impacts on ASAL communities. Kenya RAPID components are intended to work together to promote improved strategic borehole management by addressing key information and resource constraints. Figure 1 illustrates the causal linkages relevant to this evaluation that USAID envisions for translating results under each of the sub-activities into the Kenya RAPID sensor intervention’s intended outcomes. The Round I report provides more detail on the assumptions and reasoning behind this theory of change (USAID, 2019). 8 A pump turning on is a proxy for general water production and accessibility. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 4 FIGURE 1: THEORY OF CHANGE FOR THE KENYA RAPID REMOTE SENSOR INTERVENTION A key part of the theory of change is that officials in charge of water management used this information to deploy resources and address pump failure. The sensor data fed into and informed other core pieces of Kenya RAPID’s support for management processes, specifically the development of operations and management teams in each county with clear roles and responsibilities and budget support for strategic boreholes (i.e., boreholes in areas local authorities deem to be at risk of drought), to promote the goal of improving water service delivery. County and sub-county officials, in theory, should have been able to use the sensor data to improve their management and deployment of staff and resources—areas that received support through other Kenya RAPID interventions. 3.0 EVALUATION PURPOSE, AUDIENCES, AND USES This evaluation comes at an opportune time. As investments shift toward sustainable water provision, drought risk management, and service quality, innovative tools with the potential to improve service delivery, managerial decision-making, and efficient use and allocation of resources need to be evaluated to determine which are appropriate and how, or if, to bring them to scale. 3.1 PURPOSE This evaluation will help USAID understand the effectiveness of real-time remote sensing on the functionality of water services during the drought season to improve decision-making for better water service delivery and drought risk management. The results of this evaluation will be made widely available to encourage replication or scaling-up of interventions and analytical activities within and beyond Kenya, as applicable. As such, this evaluation applies USAID’s Evaluation Policy guidance with respect to using the most rigorous methods possible to demonstrate accountability for achieving results. The evaluation is also designed to capture practical lessons from USAID/KEA experience to increase sustainability in WASH programs and investment in water resource management systems, specifically in strategic, drought-prone areas. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 5 3.2 AUDIENCE The evaluation is aimed at several audiences. First, the evaluation’s findings are expected to be valuable to USAID/KEA and the USAID/RFS Center for Water Security, Sanitation, and Hygiene so they can better understand whether decision-making utilizing data from real-time remote sensing can lead to improved borehole functionality. Second, findings and lessons learned from this evaluation will be valuable to the MWA, its partners, and other practitioners in the water sector, including the Government of Kenya, which is seeking to improve water resource management, drought risk management, water coverage, and quality of services. Finally, for donors, implementers, and scholars, the evaluation will make an important contribution to the evidence base on water service delivery and information interventions in drought-prone and at-risk areas. 3.3 INTENDED USE Results from this evaluation will be used to determine whether additional investments should be made in ICT tools for improved borehole functionality in Kenya or beyond. The evaluation’s findings will also inform the design of future USAID programming targeting the sustainability of water service delivery to increase resilience and livelihoods for communities. Lastly, the evaluation will add to a growing body of evidence about drought risk management, to which the evaluations and studies conducted by USAID and other institutions also contribute. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 6 4.0 EVALUATION DESIGN The IE addresses three questions derived from the theory of change. The evaluation team developed and finalized these EQs in collaboration with USAID. • EQ 1: Does the intervention using real-time remote sensing data of water points for strategic borehole management in Kenya RAPID counties lead to increased on-time of strategic boreholes during the drought season? • EQ 2: How do water managers perceive the impact of sensor-based systems on their ability to address borehole functionality, and how does this compare to perceptions of borehole functionality in non-Kenya RAPID counties? • EQ 3: Do Kenya RAPID’s sensor-based systems affect user perceptions of borehole functionality and access? To answer the EQs, the team designed a quasi-experimental, mixed-methods evaluation focused on boreholes of strategic importance for mitigating drought risks. The evaluation team implemented a quantitative quasi-experimental evaluation approach to answer the first question (increased on-time) and a qualitative data collection and analysis approach to answer the second and third questions (water managers’ and users’ perceptions of functionality). The evaluation design involves units of analysis at two levels. Strategic boreholes (and the communities they serve) are the primary units and are nested within counties. The selection of Kenya RAPID counties was not random but based on their aridity, and the selection of boreholes for the sensor intervention based on their “strategic” designation. While the team considered random assignment within Kenya RAPID counties, there were not enough strategic boreholes to make this feasible. Instead, the evaluation team worked with USAID to identify eight ASAL counties that are nominally comparable to the Kenya RAPID counties based on general information regarding other USAID activities, aridity, and security (Table 1). This was a purposive process based on USAID staff experience and knowledge of county-level characteristics and use of verifiable county information. Table 1 highlights the purposive nature of county selection for implementation—all the Kenya RAPID counties (treatment counties in Table 1) are arid, with a relatively high level of borehole use. TABLE 1: SAMPLED COUNTIES Assignment County Arid/Semi-Arid Boreholes as a percentage of total water sources in January 2018 Comparison Baringo Arid 9% Comparison Kitui Semi-Arid 28% Comparison Laikipia Semi-Arid 29.1% Comparison Mandera Arid 15% Comparison Meru Semi-Arid 43.8% Comparison Samburu Arid 25.6% Comparison Tana River Arid 14.3% Comparison West Pokot Semi-Arid 19.6% Treatment Garissa Arid 31.7% Treatment Isiolo Arid 28.6% Treatment Marsabit Arid 48% Treatment Turkana Arid 30% FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 7 Assignment County Arid/Semi-Arid Boreholes as a percentage of total water sources in January 2018 Treatment Wajir Arid 33.3% Source: (National Drought Management Authority [NDMA], n.d.) After identifying the eight comparison counties, the evaluation team worked with NDMA and each county’s water officer to generate lists of the strategic boreholes within each comparison county.9 As noted above, strategic boreholes are meant to alleviate drought risk, for example, through selective use during drought. However, in several counties, there were no clear criteria for designating boreholes as “strategic,” and local authorities appeared unsure whether they should follow specific guidelines. Kenya RAPID staff communicated similar challenges in obtaining a consistent roster of strategic boreholes in intervention counties.10 Due to the nature of the Kenya RAPID activity, assigning strategic boreholes to the treatment group was not random. The Kenya RAPID counties were chosen because of their specific characteristics: namely, their arid to semi-arid climates and associated challenges in reliable water access, particularly during the drought season. Given this reality, we designed the IE to select a comparable set of boreholes with similar observable characteristics and controlled for these characteristics through a matching algorithm. The identifying assumption for this design is that, conditional on these observable characteristics, we would expect similar functionality outcomes across boreholes in Kenya RAPID and non-RAPID counties in the absence of the intervention. Annex J provides additional details on the matching and covariate balance between treatment and comparison groups. County representatives selected 132 boreholes across the eight comparison counties and 69 boreholes in the five treatment counties, which were selected as part of Kenya RAPID’s initial roll-out. We 9 For more detail on county selection, see the Round I report (USAID, 2019). 10 Although outside the scope of the evaluation, a full comparison between borehole characteristics across strategic and non-strategic boreholes in the RAPID counties may provide more insight into determinants of the “strategic” label. FIGURE 2: TREATMENT AND COMPARISON COUNTIES FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 8 assessed similarity based on the following observable borehole characteristics that we expected to affect functionality during droughts: • Rainfall; • Remoteness/distance from Nairobi and the county seat; • Type of pump; • Populations served (human and livestock); and • History of breakages/repair times. By comparing sensor data in treatment versus comparison borehole pumps, we can assess how the ICT intervention’s suite of information sharing, clarification of roles and responsibilities, and budget facilitation affects the main outcome variable for this evaluation: the proportion of a 24-hour period that a borehole pump is actively pumping water, known as on-time. For example, an on-time value of 50 percent would indicate the borehole pump ran for 12 out of 24 hours. Throughout the report, we use the terms “percent-on” and “on-time” interchangeably as our measures of overall borehole functionality. As a secondary measure, the evaluation team created a seven-day rolling average of borehole pump on-time, and then developed a binary, on-off measure for any periods of this seven-day average where on-time was zero. This binary on-off measure provides a measure of borehole pump functionality that attempts to account for periods of deliberate off-time; when on-time is above zero based on the seven-day rolling average, the binary measure is equal to 1. This measure helps account for any use and smooths out average on-time over a longer period, such as after a tank is filled, to better capture when a pump may be functional and unused. The matching process attempts to create as rigorous a comparison group as possible to provide a sense of what would have happened to borehole pump on-time in the absence of Kenya RAPID’s ICT intervention. However, there are several key limitations to this design: • County Sample: The counties in which Kenya RAPID operated were the most arid in Kenya. The intervention provided services that covered water management across the entire county, which limited the ability to compare intervention and non-intervention boreholes within the same county context. While the evaluation team sought to select counties that have similar levels of rainfall, the remaining counties do differ geographically and climatologically. The model-estimation approach attempts to control for county differences, but there are almost certainly unobserved variables that influence water management for which we cannot account. • Measurement Challenges: The main measure of pump on-time does not equate to direct use of the water supply. It is conceivable that a borehole pump could be turned on to fill a tank and turned off once the tank is full, while users draw down the tank water level regardless of whether the pump is on or off. While the IE design cannot capture the fact that users would still have temporary access to water from a full tank with a non-active pump, in large part water access is reliant on borehole pumps, especially over extended periods. This limitation is considered in the analysis and discussion below. Qualitative findings were also used to enhance the quantitative analysis for EQ1 by providing additional context to the distinction between water access and pump on-time, in addition to directly answering EQ2 and EQ3, exploring perspectives among managers and users across the two assignment groups. • Borehole Sample: The number of strategic boreholes designated to address drought-prone areas is limited in each county. As a result, the total population of available boreholes for consideration in this study was limited ex-ante. This limited sample size presents a challenge to reliably estimating an impact across treatment and comparison counties, particularly where impacts are small. • Lack of Drought: The evaluation was designed to test a theory of change based around boreholes that serve a strategic drought mitigation role. However, Kenya experienced record rainfall during the 2018–2020 evaluation period, with 2020 seeing an increase in flooding and overall high levels of FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 9 rainfall compared to the previous September 2018 to September 2019 period. While the evaluation analysis and design attempts to account for rainfall and borehole substitutes, it is difficult to completely control for unanticipated changes in use, access, and management. 5.0 DATA COLLECTION The evaluation design initially envisaged three rounds of data collection: baseline, midline, and endline. However, because the ICT intervention was already underway when the evaluation team initiated the first round of data collection, it is not a true baseline. Rounds of data collection throughout this report are therefore referred to as Round I, Round II, and Round III. Round I was initiated in August 2018 with a visit to each of the 132 comparison boreholes to install a sensor on the borehole pump and conduct the initial borehole asset survey. Round I also included qualitative research in two comparison and two treatment counties to conduct KIIs with sub-county water managers and borehole operators and to conduct FGDs with water users. Each comparison borehole was visited again in September 2019 for Round II, providing an opportunity to download data stored on each sensor. At the same time, Kenya RAPID shared sensor data from the treatment boreholes for the period covering April 2017 through December 2019. For Round III in August 2020, the evaluation team visited comparison boreholes that were still part of the sample at Round II to download sensor data covering the period since Round II (i.e., starting in September 2019), and Kenya RAPID shared updated sensor data for the period through October 2020.11 In Rounds II and III, the evaluation team conducted a limited version of the borehole asset survey to capture any relevant changes to the borehole context. The evaluation team also conducted additional KIIs and FGDs at different boreholes in September 2020. 5.1 BOREHOLE ASSET SURVEY Each round of data collection included an asset survey of borehole characteristics.12 At Round I, the evaluation team employed a survey that was almost identical to the survey that Kenya RAPID used for their monitoring data. As part of the intervention, SweetSense visited each strategic borehole in Kenya RAPID counties and conducted a borehole asset survey at the time of sensor installation. This survey collected data on borehole characteristics as well as information on the borehole context through interviews with borehole managers and sub-county and county officials. Strategic borehole data were collected in the Kenya RAPID (treatment) counties between November 2017 and September 2018; the activity subsequently provided these data to the evaluation team. The evaluation team adapted the borehole asset survey and used it to collect comparable data on borehole characteristics in the comparison counties between August and September 2020. Round II employed a limited borehole asset survey to determine what changes may have occurred across key metrics, such as power source.13 The Round II survey included a subset of questions from the survey utilized in Round I. The follow-up survey in the comparison counties was employed to track key changes in the borehole context that might influence borehole pump functionality. The evaluation team repeated the same limited borehole asset survey for Round III. 11 As noted in the Round II report, out of the original 132 comparison boreholes, 115 had sensors still installed (or re-installed) at the end of Round II. In some cases, boreholes closed, the team was unable to reinstall new sensors when sensors were lost or stolen, or the pump type changed, and the sensor was unable to be installed on the new pump. 12 See Annex B for survey instrument. 13 The full list of metrics are what was collected in the survey instrument. See Annex B for survey instrument. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 10 At each round, the asset survey was programmed into iField, a computer-assisted personalized interview (CAPI) application that can be run on a mobile device. The CAPI survey allowed enumerators to take photos, record global positioning system (GPS) stamps, and upload data in near-real time. Given the remote location of many of the boreholes, Ipsos Public Affairs, the data collection partner for this evaluation, provided each survey team with a separate GPS logger to ensure the most accurate coordinate stamp possible.14 5.2 BOREHOLE SUBSTITUTES A key issue raised in the Round II report (USAID, 2020) was how to account for borehole substitutes— that is, a water source used in lieu of a borehole. The Round I qualitative interviews and other research suggest that rainfall provides a borehole substitute in many cases, but this substitution does not account for all of the variation in borehole pump on-time, even within Kenya RAPID counties.15 To account for rainfall and other borehole substitutes, the evaluation team compiled data from NDMA on the percentage of each county’s water sources comprised of boreholes. NDMA issues monthly drought early warning bulletins for every county in Kenya. Figure 3 shows an example of the front page of a typical bulletin. In most months, the bulletin includes either a table, figure, or text reporting where people in the county get water. The inclusion of this section varies from month-to-month and county-to-county. We reviewed 287 of the monthly bulletins to pull the reported borehole percentage numbers into an analyzable format and providing an update to the initial figures shown in Table 1. The team used a Bayesian linear regression to impute missing values for months where data were not available or reported.16 The resulting variable provides a measure of the percentage of water sources in the county that are boreholes, on a monthly basis.17 Put differently, this new variable provides an inverse measure of the percentage of substitutes for borehole water. For example, this variable has a value of 10 percent for Baringo in November 2019, which implies that 90 percent of water sources accessed by the population were not boreholes, with many of these not providing safe drinking water, such as rivers and temporary ponds. 5.3 SENSOR DATA Borehole sensor data are the primary basis for the quantitative analyses to answer EQ 1. The borehole pump sensors were installed on 132 comparison boreholes during Round I, following sample selection 14 The Round II report provides a detailed summary of the data collection and downloading process. 15 For example, see Thomas et. al., 2020. 16 For more, see Buuren & Groothuis-Oudshoorn, 2010. 17 After reviewing a draft of this report, Kenya RAPID noted that there may be data quality issues with NDMA’s reports, including that some reports may suggest water sources are available when they are not, or bulletins may suggest the opposite, no availability when there is indeed water access. The evaluation team does not have a way of verifying the data quality on the ground but is aware of the issue. FIGURE 3: SAMPLE NDMA BULLETIN FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 11 described earlier, and boreholes were revisited on an annual basis to download the sensor data. Our evaluation team, through local subcontractor Ipsos Public Affairs, successfully downloaded data from 114 sensors at Round II and reconnected or reinstalled sensors on 115 borehole pumps. Ipsos then revisited each of the 115 comparison boreholes to download sensor data and conduct a limited survey of borehole characteristics between August 26 and September 3, 2020 for Round III. As noted above, Round III employed the limited borehole survey to determine what, if any, changes occurred across key metrics. SweetSense provided sensor data for intervention boreholes for the Round II and Round III measurement periods through October 16, 2020. Though the intervention and comparison sensors are different models with distinct specifications, they produce similar data, including a start-time stamp, end-time stamp, and the pump on-time percentage. Whereas the intervention sensors have satellite connectivity, comparison sensors procured for this evaluation forgo that feature in consideration of cost control. As such, the comparison sensors must be physically accessed to download data as a special file format that allows for reviewing the data at various intervals from one to 60 minutes, as well as at a daily interval. In contrast, the SweetSense data were provided at a daily level and are reviewed on an ongoing basis. SweetSense also provided additional variables, such as the seven- and 14-day rolling averages, as well as their classification for the borehole’s functionality, e.g., no use, offline, and seasonal disuse. Such classifications are the result of SweetSense’s ability to check on the borehole status in person and then train a machine learning algorithm on the field-confirmed data. 5.4 QUALITATIVE DATA In addition to the quantitative data, Rounds I and III included qualitative data collection to provide contextual information related to EQs 2 and 3. Interview and focus group questions focused on managers’ and users’ experiences during the most recent drought period.18 The Colorado Multiple Institutional Review Board (COMIRB, Protocol #18-1438) and the Kenya Medical Research Institute (KEMRI, Protocol #637) reviewed and approved the qualitative study protocol. The KEMRI Institutional Review Board approval was the first step in a process to meet Kenya’s National Commission for Science, Technology, and Innovation’s (NACOSTI) national ethical approval, which is obtained through an affiliation with an approved local organization. For this evaluation, the team partnered with Ipsos, which has a NACOSTI accreditation, to assist with qualitative data collection planning, translation during field work, and quality oversight. Ipsos ensured compliance with NACOSTI standards. In addition, Ipsos trained all its staff to comply with local COVID-19 requirements, such as maintaining a two-meter distance between staff and any respondents, as well as equipping all field staff with personal protective equipment.19 During Rounds I and III, qualitative data were collected through KIIs with water managers and FGDs with water users. The evaluation team purposefully sampled a total of four counties for inclusion in the qualitative subsample. The evaluation team used a sampling approach in Round III that was similar to that used in Round I, which attempted to capture high- and low-traffic boreholes with a mix of livestock and human uses; however, the Round III sampling approach also took into consideration borehole pump on￾time, which was not available during the sampling process at Round I when it was based entirely on the borehole asset survey. Three of the four counties selected for data collection in Rounds I (November–December 2018) were also included in the Round III sample (September 17–20, 2020): Garissa and Turkana in the treatment group, and Tana River in the comparison group. Baringo, the second comparison county in Round I, was 18 See Annex G for the interview and discussion guides. 19 See Annex E for additional detail on COVID-19 considerations. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 12 replaced by West Pokot for Round III due to accessibility challenges in the Round I qualitative interviews due to intense rainfall. Within counties, the specific boreholes chosen at Round III were different from those included in Round I, given that data on on-time from the evaluation period drove the selection of these boreholes in Round III. In addition, security concerns that arose during Round I qualitative data collection pushed the team to revisit the sampling process. The evaluation team used Round II data to inform selection of a revised qualitative sample. As in Round I, the evaluation team sought to capture experiences from users and borehole operators in both low on-time boreholes and high on-time boreholes across low and high borehole pump use counties. The goal of this approach is to cover a range of potential experiences and contexts while finding thematic overlap in the qualitative data and investigating where perspective vary based on location. A key consideration for sample selection at Round III was overall on-time. As shown in Figure 4, Isiolo in the treatment group and Laikipia in the comparison group had the lowest overall average on-time from 2018 to 2019. West Pokot and Turkana had the highest. Turkana was sampled for Round I qualitative data collection but was also a high on-time borehole, making it a good choice within the selection criteria. FIGURE 4: ROUND II BOREHOLE ON-TIME BY COUNTY Although Isiolo boreholes exhibited low percent on-time within the Kenya RAPID sample, there are only three boreholes in that county, rendering it unsuitable for qualitative sampling (because with such a small number of boreholes, there is no room for re-sampling should an issue arise at one of the selected boreholes). In light of transportation and security challenges in Round I, the evaluation team worked to ensure that replacement boreholes could be easily sampled within the selected counties. In addition, the spread of COVID-19 presented additional considerations (see Annex E), which meant that the evaluation team had to build in as much flexibility into the qualitative sample as possible. Two of Isiolo’s three boreholes also reported very little on-time variation in 2018–2019. While one of the boreholes had on-time of around 34 percent, the other two reported less than two percent on-time during the Round II period. Due to the limited number of boreholes and lower level of variation in Isiolo FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 13 compared to other counties, Garissa was chosen as the “low” on-time county in the treatment group for qualitative sampling. Among comparison counties, Laikipia had the lowest average on-time at Round II, but Tana River, which had the third lowest on-time, was selected at Round I and had similar average on-time to Garissa in the treatment group. West Pokot was the clear leader at Round II in average on-time, while Turkana was a leader in average on-time at Round II among RAPID counties. Within the chosen treatment counties (Turkana and Garissa) and comparison counties (Tana River and West Pokot) for Round III, the evaluation team selected the top two and bottom two on-time boreholes with the aim of capturing a range of experiences and perceptions.20 Figure 5 shows the daily on-time for the selected counties. Darker symbols represent the four boreholes selected for accompanying qualitative data collection (two high average on-time, two low average on-time) within each county, while the solid line is the smoothed average for the county as a whole.21 Tana River provides a clear illustration of how the qualitative sampling includes both the highest on-time boreholes (dots at the top of the figure) and lowest on-time boreholes, which are slightly closer to the overall county average. FIGURE 5: DAILY ROUND II ON-TIME FOR SELECTED QUALITATIVE COUNTIES AND BOREHOLES In addition to on-time, the evaluation team also examined criteria from the asset survey, as shown in Table 2. Except for the Turkana boreholes, all the sites selected for qualitative data collection are used for livestock, and there is a mix of generator and other power sources. TABLE 2: CHARACTERISTICS OF BOREHOLES SELECTED FOR QUALITATIVE DATA COLLECTION 20 This sampling was done after filtering out boreholes with fewer than 200 timestamps and with average on-time above 0.01%. Dropping boreholes with very few observations and very little use was done to avoid selecting boreholes that are no longer functional or used so rarely that user and manager perceptions might be unique to very specific conditions. 21 More specifically, this is the locally smoothed seven-day mean of on-time, which helps minimize the influence of outliers. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 14 # County Power Type Households Served Fixed Tariff Average On￾Time (Round II) Used for Livestock 1 Tana River (Comparison) Generator 560 Yes 5% Yes 2 Tana River (Comparison) Utility Power 6,250 Yes 52% Yes 3 West Pokot (Comparison) Solar 250 No 36% Yes 4 West Pokot (Comparison) Solar 105 No 44% Yes 5 Garissa (Treatment) Generator 500 Yes 26% Yes 6 Garissa (Treatment) Hybrid 300 Yes 32% Yes 7 Turkana (Treatment) Solar 100 Yes 14% No 8 Turkana (Treatment) Solar 74 Yes 44% No 5.4.1 KEY INFORMANT INTERVIEWS We purposively selected a total of sixteen key informants—four per county—based on their expected knowledge of borehole management and repair. We identified respondents through the borehole asset survey data in coordination with Kenya RAPID staff. In the treatment counties, respondents included county-level individuals most closely involved in managing the data dashboard, the system that conveys information from the SweetSense sensors,22 as well as leaders in the administration of water and irrigation services, water and sanitation companies (WASCOs), technical managers, and engineers. In comparison counties, we identified individuals holding similar positions with respect to county-level water management. In all counties, we also interviewed sub-county water officers and local pump attendants or borehole operators for each borehole selected for qualitative data collection (Table 2). Annex H provides a full, anonymized list of the KII respondents. Interviews followed a semi-structured format, allowing for follow-up questions and flexibility in the evolution of the discussion. The KII protocol, which is provided in Annex G, was structured to gather background information about the local context, including the local economy, any prior water management or drought mitigation schemes, and implementation of Kenya RAPID interventions from the perspectives and experiences of treatment area respondents. Given that the study poses minimal risks to participants, the team obtained oral consent for KII subjects. The consent script for qualitative data collection is provided in Annex F. We voice recorded the interviews and took detailed notes during each to inform analysis. 5.4.2 FOCUS GROUP DISCUSSIONS To investigate perceptions of borehole functionality further and gain a better understanding of user experiences, the evaluation team conducted focus group discussions (FGDs) with between eight and 10 water users served by each of the eight selected boreholes. In the study context, the evaluation team determined that having both men and women in mixed discussions would not be acceptable. The team thus developed a procedure for determining whether men or women were the primary users of each 22 IBM is implementing the dashboard portion of Kenya RAPID. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 15 borehole and selected single-sex FGD participants accordingly. Specifically, women generally collect water for domestic (household) use, while men are often in charge of managing livestock and their water consumption. The team therefore determined whether the primary use of the borehole was for domestic use or for livestock (through the KIIs) and conducted FGDs with women in the former case and men in the latter. Additionally, the evaluation team worked with the chief or the local administration to draw a map of the villages that used the borehole and to help recruit at least one water user from each village for the FGD. While the evaluation team held each FGD with either men or women only, discussion questions elicited information about water access patterns and challenges among users of the opposite gender. As part of the planning and facilitation process, the team informally met with whichever gender group was not part of the planned FGD to explain the evaluation and selection process and collect a few basic details to corroborate with the FGD data. This informal questionnaire is provided in Annex G. This approach allowed the evaluation team to hold in-depth data collection with the most relevant borehole users while ensuring some level of gender parity. It also served to note divergence in perceptions between gender groups on key factors, such as borehole functionality and management staff availability. A moderator from the evaluation team facilitated the FGDs in Swahili, or the most prevalent local language, utilizing semi-structured instruments with specific questions to guide the discussion. The evaluation team designed the FGD protocol, provided in Annex G, to obtain participant experiences on the following key issues that are relevant for this evaluation, irrespective of the county: ● General livelihoods strategies and borehole access and use; ● Overall perception of borehole management, functionality, and changes; ● Usage patterns, perceived challenges to use, and threat of drought; ● Perception of borehole management-community engagement and accessibility for different community populations (e.g., women and youth); ● Prior experiences with water management and preservation programs, perception of program influence, shocks, and mitigation strategies; and ● Impacts of the COVID-19 pandemic on water use and access issues. 5.4.3 QUALITATIVE DATA ANALYSIS METHODS For both Round I and Round III, the evaluation team analyzed data from the FGDs and KIIs using content analysis techniques, in which text was coded according to key themes of interest across the interviewees and discussion participants. Using MaxQDA qualitative analysis software, the team coded all interview notes. The team applied a mixed coding approach, with thematic codes determined ex-ante, followed by open coding to capture themes and broader trends as they emerged during an initial review of the data. The evaluation team reviewed and summarized coded segments by theme, exported the codes into Excel, and further analyzed them using the R software package to capture counts, frequencies, and relationships between themes and words. The evaluation team employed text mining methods to the coded segments to further analyze the reported roles of women, repairs, and water system functionality (Feinerer, Hornik, & Meyer, 2008). This analysis included basic topic mapping as well as analysis of what words came up most often within the context of women’s roles in accessing and using water resources to understand better the relevant gender themes that may not come out of traditional qualitative coding approaches. The team also FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 16 employed sentiment analysis techniques to see how the qualitative responses align with broad sentiment categories that have been verified in the text mining literature.23 6.0 FINDINGS This section presents findings for each of the three EQs. Discussion of EQ1 begins with a descriptive analysis of borehole sensor and contextual data, followed by a more in-depth discussion of the impact analysis. Additional information on key EQ1 variables can be found in Annex I. EQ2 presents detailed characteristics for KII sample and highlights major themes uncovered including water system functionality, management structure and roles, information about borehole failures and perceptions of the sensor-based system, and resources for repairs. EQ3 presents characteristics of the FGD sample and presents major themes uncovered, including water system functionality, borehole access and use, and perceptions of management structure. This section concludes with additional perspectives on COVID-19 impacts on water use and management. Annex L provides a summary comparison of qualitative Round I and Round III findings and themes. 6.1 EVALUATION QUESTION 1: DOES THE INTERVENTION USING REAL-TIME REMOTE SENSING DATA OF WATER POINTS FOR STRATEGIC BOREHOLE MANAGEMENT IN KENYA RAPID COUNTIES LEAD TO INCREASED ON-TIME OF STRATEGIC BOREHOLES DURING THE DROUGHT SEASON? 6.1.1 BOREHOLE SENSOR AND CONTEXT DESCRIPTIVE ANALYSIS We developed and applied a set of matching models to develop a sample for estimating the effects of the Kenya RAPID intervention on borehole functionality. These models use data on borehole characteristics to match Kenya RAPID boreholes with one or more comparison boreholes that are as similar as possible. The Round I report presented an initial set of matching models; based on feedback, we refined these models in Round II using higher resolution data on rainfall and an alternative definition of remoteness to measure distance to the county seat rather than Nairobi, which was used in Round I. These refinements resulted in better overlap across assignment groups, improving the ability of the comparison group to serve as a convincing counterfactual for the Kenya RAPID counties. 23 Taboada, M., Brooke, J., Tofiloski, M., Voll, K., & Stede, M. "Lexicon-based methods for sentiment analysis." Computational linguistics 37, no. 2 (2011): 267-307. Key Findings for EQ1 • Drought season impact: Kenya RAPID did not have a meaningful impact on strategic borehole pump on-time during the drought season (2019 and 2020). • Overall on-time: Strategic borehole pump on-time was 21.4 percent on average in Kenya RAPID counties and 15 percent on average in comparison counties across for the period August 2018 through the end of August 2020. − Weekly functionality: Borehole pumps in Kenya RAPID and comparison counties had almost equal odds of being on over the course of a week (OR 0.91). − 2019 impact: Kenya RAPID appears to have increased strategic borehole pump on-time by around 3 percent after controlling for key characteristics, rainfall, and substitutes for the 2019 drought periods. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 17 The Round I and Round II reports provide detailed analysis of the borehole pump context, including power sources, people served, and location. Given that the main impact analysis approach controls for borehole context at Round I, i.e., before Kenya RAPID’s intervention was fully operational and scaled, this report will not repeat the contextual analysis summarizing the borehole characteristics.24 However, there are several variables that are critical for estimating impact that are important to understand descriptively, including overall borehole pump on-time, rainfall, and borehole substitutes. 6.1.1.1 BOREHOLE PUMP ON-TIME There are several ways to calculate borehole pump on-time using the sensor data output. The main metrics of interest for this evaluation are daily on-time and a binary measure tracking whether or not a borehole was on at all within a week-long period. Strategic borehole pump on-time percentage is the main outcome measure for EQ1. This metric is the percentage of a 24-hour period for which the borehole pump was on; for example, 0.5, or 50 percent, is equivalent to 12 hours in a 24-hour day. The overall borehole pump on-time was 21.4 percent on average in Kenya RAPID counties and 15 percent on average in comparison counties for the period August 2018 through the end of August 2020. FIGURE 6: AVERAGE BOREHOLE ON-TIME BY ASSIGNMENT As illustrated in Figure 6, the average borehole pump on-time fell for both Kenya RAPID and comparison counties between Round II and Round III (i.e., October 2019 to August 2020), with the average on-time declining from 25 percent to 17 percent in Kenya RAPID counties and 18 percent to 11 percent in comparison counties. This is likely due in part to the higher-than-usual rainfall amounts during the two years of the evaluation since borehole use is inversely related to rainfall (as shown below in section 6.1.1.3). As shown in Figure 7, borehole pump data are heavily right skewed, implying that borehole pumps were more often turned off than on. 24 Annex I provides summary statistics on Round I borehole characteristics, which are used as controls in the inferential analysis. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 18 While the sensors do not capture direct use, the measure of borehole pump on-time does provide a helpful proxy for borehole pump use since extended periods of down-time are a sign of pump breakages, particularly after controlling for rainfall. A better estimate of performance is the rolling seven-day average of on-time, which better distinguishes between temporary and extended down-times. Discussions with SweetSense suggested that looking at a rolling weekly average of on-time provides a measure that is less sensitive to temporary off-periods due to water tank use; for example, a borehole pump might be used to fill a tank, which then lasts for several days resulting in pump off-time, but with full water access for users. As shown in Figure 8, moving to a weekly average adjusts the skew of the data slightly as compared to Figure 7, but also reduces the number of observations. The number of observations with zero percent on-time on both rounds and assignment groups falls from 47,289 on a daily basis to 32,688 on a weekly basis—a 31 percent decline. FIGURE 7: DISTRIBUTION OF DAILY ON-TIME BY ASSIGNMENT AND ROUND This weekly average is also important for generating a basic “on-off” binary measure to understand general functionality (discussed further in Section 6.1.2). If a borehole had a rolling weekly average of zero (i.e., no on-time on average for a week, implying no pump use) then this is recorded as zero; otherwise, the on-off binary is coded as one. This allows the evaluation team to capture general functionality, with RAPID counties reporting about 71 percent weekly average functionality, while comparison counties report 68 percent weekly functionality, based on this binary measure. However, weekly functionality during the dry months only was 70.3 percent for the comparison counties and 71.7 percent for Kenya RAPID counties. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 19 FIGURE 8: DISTRIBUTION OF WEEKLY BOREHOLE ON-TIME BY ASSIGNMENT AND ROUND FIGURE 9: AVERAGE WEEKLY FUNCTIONALITY PERCENTAGE BY ASSIGNMENT As shown in Figure 9, using the binary on-off measure based on a seven-day rolling average shows that borehole pumps were largely functional. The contrast between these higher averages and the daily rates comes from the fact that many boreholes report very low on-time percentages, as shown by the right￾skewed figures above, but are in fact operational. By looking at whether there was any pumping, this metric highlights the fact that four comparison counties had general functionality above Kenya RAPID’s average. The benefit of this approach is that it removes much of the noise from the data (see Annex I FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 20 for a figure depicting the daily on-times for each borehole by county). Many of the boreholes within each county report periods of very low, but not zero, on-time, as well as periods of higher pump on-time percentage. The binary on-off measure based on a seven-day rolling average helps to reduce this variation by removing any distinction between lower and higher on-time. While this reduces precision, it does provide a measure of general functionality. 6.1.1.2 BOREHOLE PUMP SUBSTITUTES Boreholes comprise an average of 37 percent of all water sources in Kenya RAPID counties and 26 percent of water sources in comparison counties. These water sources include all available sources, whether improved, such as boreholes, or not, such as rivers. As indicated in Figure 10, boreholes comprise 20 to 40 percent of a county’s water sources for most of the counties in the sample, with some variation throughout the year. Isiolo and Meru, in the treatment and comparison groups, respectively, stand out in that over the year 42 percent of the water sources in each county are boreholes. Baringo, a comparison county, also stands out with boreholes representing 14 percent of county water sources over the year. The months with the highest and FIGURE 10: MAP OF BOREHOLES AS PERCENT OF lowest percentage of borehole water WATER SOURCES use varied across assignment groups. Across the entire sample of Kenya RAPID counties, boreholes comprised 47 percent of water sources on average in July, but only 27 percent in November. This variation over time aligns with one of the assumptions underlying the Kenya RAPID theory of change: that boreholes are used to meet water demand during the driest months. In the comparison counties, September saw the highest percentage of water borehole sources at 28 percent of all water sources, while November was the lowest month at 23 percent. While the values vary and data quality is an issue given that NDMA’s reporting varies month-to-month and county￾to-county, a key takeaway from the addition of this metric is that there is seasonal variation in the type of available water sources for the sampled counties. This is important for understanding factors outside of the intervention that might impact borehole pump on-time. Because this measure is at the county level, some of the within-county variation in water sources is lost. For example, it may be the case that a county overall has a lower percentage of water borehole sources, but the most arid portion of the county has a high percentage of boreholes among its water sources. 6.1.1.3 RAINFALL The evaluation uses remotely sensed rainfall data, known as Climate Hazards Group InfraRed Precipitation with Station (CHIRPS). CHIRPS is a satellite database of global rainfall that combines FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 21 satellite and in-situ precipitation data collection from the University of Santa Barbara, with funding from the US Geological Survey, USAID, NASA, and NOAA. CHRIPS data used for this evaluation are at 0.05° resolution (Funk et al., 2015). The data consist of a continuous measure of rainfall within a 20-meter radius around each borehole. Rainfall appears to be increasing during the periods that are typically considered the driest. This is important to keep in mind given the theory of change, which assumes that Kenya RAPID’s information sharing will help mitigate borehole pump breakages during dry periods. The overall average daily rainfall increased from 1.7 mm to 3.8 mm on average in comparison counties and from 0.85 mm to 2.04 mm on average in Kenya RAPID counties over the period between October 2019 and August 2020. This continues the trend noted in the Round II report. Indeed, a historical analysis comparing recent rainfall to average rainfall dating to 1981 shows that rainfall was around 150 percent above median levels for the period between February and September 2020 across almost all of Kenya.25 As shown in Figure 11, rainfall in October 2018 was 1.6 mm and 1.2 mm on average in the comparison and treatment counties, respectively. By October 2019, average rainfall for that month was nearly 6 mm and 4 mm in the comparison and treatment counties, respectively. The gap between comparison and treatment county average rainfall is particularly notable for November and December 2019. FIGURE 11: AVERAGE MONTHLY RAINFALL FOR TREATMENT AND COMPARISON COUNTIES The evaluation team noted that September, January, and February were the driest months for both the treatment and comparison counties in the 2018–2019 measurement period. In the 2019–2020 measurement period, there is more divergence in rainfall between the assignment groups, with February, June, and July reporting the driest months on average for the Kenya RAPID counties, while the driest months for comparison counties were January, February, and June. Figure 11 highlights what has been stated previously: 2020 rainfall was much higher than previous years within the evaluation period.26 This is important context not only in that it affects how people use borehole water sources generally, but 25 See FEWSNET, https://earlywarning.usgs.gov/fews/product/596. 26 According to FEWSNET data, for 2020 almost all of Kenya experienced at least a 100 percent increase in rainfall above the historical average (1981–2010). FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 22 also given that Kenya RAPID reported at least 18 of its boreholes were temporarily closed because they were meant to serve as strategic drought water sources. With the heavy rainfall, these boreholes were not needed to mitigate drought risk. Information on similar borehole closure due to heavy rainfall was not available in the comparison counties.27 As noted in the discussion on borehole substitutes, there appears to be a relationship between rainfall and the percentage of a county’s water sources comprised by boreholes. Figure 12 shows that there is indeed a negative relationship between rainfall and borehole prevalence (Pearson coefficient -0.11). The relationship appears to be stronger in Kenya RAPID counties but is still present in comparison counties. There is also variation within the assignment groups, with Marsabit reporting a particularly strong negative relationship (-0.15, p< 0.001), while there appears to be almost no relationship in Tana River (0.003, p = 0.77). FIGURE 12: AVERAGE MONTHLY RAINFALL AND BOREHOLE WATER SOURCES 6.1.2 IMPACT ANALYSIS The main estimation model to determine the intervention’s impact uses several variables from the borehole asset survey, as well as third-party data, such as rainfall, to control for differences in the main outcome of interest: borehole pump on-time. The model follows a similar specification as the matching model outlined in the Round I and Round II reports, with the sole addition of the borehole substitutes variable. The key asset survey covariates are as follows: ● Number of households using the borehole scheme: integer, the borehole manager reports this figure. 27 The information about treatment county boreholes was shared with the evaluation team after field data collection, so it was not possible to verify whether borehole operators shut down comparison county boreholes during heavy rainy periods. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 23 ● Livestock use (Y/N): binary 1-0 indicator of whether the borehole is used by livestock, the borehole manager reports this figure. ● Service area for the scheme (in km2): numeric measure in kilometers based on length of pipes from borehole to access point, the borehole manager reports this figure. ● Out of service one or more days in the past month (Y/N): binary 1-0 indicator, the borehole manager reports this figure. ● Borehole pump power type: recorded as either solar, utility, generator, or hybrid, observed by enumerators. ● Average daily on-time (in hours): numeric measure, the borehole manager reports this figure ● Borehole construction year: numeric date, the borehole manager reports this figure; in some cases, enumerators were able to verify based on commemorative documentation or plaques at the borehole site. This was not part of the matching process. ● Fixed tariff scheme (Y/N): binary 1-0 indicator denoting whether a tariff is applied for use of the borehole, the borehole manager reports this figure. ● Water quality test (Y/N): binary 1-0 indicator of whether there has been any water quality testing on the borehole; the borehole manager reports this figure. In addition to CHIRPS rainfall data, mileage data are also included in the estimation model and calculated based on the travel distance in miles from the borehole location to the county seat, where county officials who support water management are located. Travel miles were calculated using the Google Maps API in R.28 The number of households, service area, rain, and miles are rescaled by centering the variables and dividing by two standard deviations per Gelman and Hill (2006). This makes the variables easier to interpret and compare, especially in the case of the miles measure for which there is a large amount of variation. Three matching algorithms were run as a sensitivity test to develop a range of matched samples from which to estimate the impact of Kenya RAPID: 1) optimal; 2) nearest neighbor; and 3) many-to-one. While researchers often run multiple models or tests and then only report on one, we have included each of our matching tests to show how researcher decisions, such as selection of matching algorithm, affect estimation. Discussion of estimates in the subsequent sections references these matched samples as appropriate. Annex J provides additional detail on the matching approach, which is described in further detail in the Round II report. The number of boreholes varies across each sample, as shown in Table 3, with each sample used across the main model specifications summarized below. TABLE 3: BOREHOLES PER SAMPLE BY MATCHING ALGORITHM Matching Algorithm Comparison boreholes Treatment boreholes Unmatched 119 69 Optimal 55 60 Many-to-one 108 60 Nearest Neighbor 13 28 28 See https://developers.google.com/maps/documentation/distance-matrix/intro for more detail. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 24 6.1.2.1 IMPACT MODEL AND ESTIMATION APPROACH The evaluation team employed three models to estimate the impact of the intervention on borehole functionality. Model 1 estimates daily on-time percentage; Model 2 estimates weekly rolling average on￾time percentage; and Model 3 estimates time to repair. Each are discussed in more detail below. For the full panel of data, the following general model specification (Model 1) is used to estimate the percentage of the day that the borehole pump was on during dry month periods. For example, if the borehole pump was cumulatively on for 12 hours, this variable would be equal to 50 percent. This model is fit to the full population of boreholes over the period of the evaluation. 𝑌𝑌𝑖𝑖𝑖𝑖 = 𝛾𝛾0 + 𝛽𝛽1𝑋𝑋𝑖𝑖 + 𝛽𝛽2𝑇𝑇𝑖𝑖 + 𝛿𝛿𝑖𝑖𝑖𝑖 + 𝛽𝛽3𝑇𝑇𝑖𝑖 ∗ 𝛿𝛿𝑖𝑖𝑖𝑖 + 𝛽𝛽4𝐶𝐶𝑖𝑖𝑖𝑖 + 𝛽𝛽5𝑊𝑊𝑖𝑖 + 𝛾𝛾𝑖𝑖 + 𝜀𝜀𝑖𝑖𝑖𝑖 Model 1 Where: ● Yit is the outcome of interest, daily percentage that borehole pump was on, for borehole I at time t; ● Xit is the vector of survey and mileage covariates discussed above; ● T is an indicator variable equal to 1 for members of the treatment group; ● δ is an indicator variable equal to 1 for the dry months (January, February, June, and September); ● Cit is the CHIRPS rainfall data; ● Wit is a set of indicators denoting the percentage of alternative water sources based on monthly NDMA county bulletins for each county; ● γi is a vector of county fixed effects; and ● εit is a random error term. The main outcome of interest in Model 1, daily on-time percentage, does not account for why a borehole might be on or off. A key question of the evaluation is whether Kenya RAPID’s interventions result in improved borehole performance. As noted above, a challenge to this analysis is the absence of an accurate and cost-effective way to verify why a borehole recorded off-time. One way to address this is to move from the reported percent of borehole on-time to a weekly rolling average, Model 2. 𝑌𝑌𝑖𝑖𝑖𝑖 = 𝛾𝛾0 + 𝛽𝛽1𝑋𝑋𝑖𝑖 + 𝛽𝛽2𝑇𝑇𝑖𝑖 + 𝛽𝛽2𝑇𝑇𝑖𝑖 ∗ 𝛿𝛿𝑖𝑖 + 𝛽𝛽3𝐶𝐶𝑖𝑖𝑖𝑖 + 𝛽𝛽4𝑊𝑊𝑖𝑖 + 𝛾𝛾𝑖𝑖 + 𝜀𝜀𝑖𝑖𝑖𝑖 Model 2 Where: ● Yiw is the outcome of interest, is a binary indicator for whether or not a borehole was on for borehole I for week w; ● Xi is the vector of survey and mileage covariates discussed above; ● T is an indicator variable equal to 1 for members of the treatment group; ● δ is an indicator variable equal to 1 for the dry months (January, February, June, and September); ● Ciw is CHIRPS rainfall data collapsed to a rolling weekly average; ● Ww is a set of indicators denoting the percentage of alternative water sources based on monthly NDMA county bulletins for each county; ● γi is a vector of county fixed effects; and ● εiw is a random error term. For Model 2, the dependent variable is equal to one if the borehole pump was turned on at all in a given week, and is otherwise zero, which may suggest a broken pump. Like Model 1, this model accounts for fixed county and random borehole effects. The data for Models 1 and 2 are limited to the period FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 25 between August 8, 2018 and August 31, 2020, the period for which both treatment and comparison data are available.29 A potential limitation is that Models 1 and 2 do not appropriately capture borehole use in a way that accounts for the duration of breakages as distinct from periods of non-use. Given the distribution of on￾time, with most observations around zero (i.e., no borehole pumping), the normality assumption of linear regression does not hold. While this assumption is often dismissed, in the context of borehole pumps, there is a long tail, with some pumps reporting relatively high on-time despite most others reporting less than 50 percent on average. In addition to this challenge, there are time-invariant independent variables in our model, such as rainfall and the percentage of alternative water sources, which complicate the use of linear regression given the basic assumptions of ordinary least squares. To address these potential model weaknesses, we fit a duration model that estimates the time of borehole pump survival (i.e., how long a pump lasts before breaking). If boreholes report less than an hour of pump on-time within a month (i.e., four weeks), they are recorded as no longer functional. (We note, however, that borehole sensors in the comparison group were reportedly removed during repairs, so longer off-time may actually be a measure of repairs rather than breakages.) The evaluation team employed a Cox model (Cox 1972) that takes the following form: ℎ(𝑡𝑡) = ℎ0(𝑡𝑡) + 𝑒𝑒𝑒𝑒𝑒𝑒 (𝑏𝑏1 1 + 𝑏𝑏2 2 + 𝑏𝑏3 3 + ⋯ 𝑏𝑏𝑘𝑘 𝑘𝑘) Model 3 Where: ● ℎ(𝑡𝑡) represents the risk of breakage at time, t; ● x represents the covariates of interest, which are the same as in Models 1 and 2; ● b, are the coefficients to be estimated for the covariates, x; ● ℎ0 provides an estimate of time varying off-time if all b for covariates x are zero; and ● 𝑒𝑒𝑒𝑒𝑒𝑒 implies the exponentiation of the covariate estimates to obtain a ratio that indicates the odds of repair. This model will yield an estimate of the lifespan of a borehole pump as a function of key borehole characteristics and other variables, such as rainfall, across treatment rounds. A key assumption of this approach is that longer periods of borehole pump off-time, after accounting for rainfall, are the result of breakages rather than filling of local water tanks. Given the lack of drought and higher-than-expected rainfall in Kenya during the period of the evaluation, it is sensible to examine subsets of the borehole data in an attempt to control for climate conditions. To this end, we fit Models 1 and 2 on data only for 2019, where precipitation was closer to, though still above, levels expected during the intervention’s design. The tradeoff in limiting the data to 2019 is that we know the activity was implemented beyond this point, so it does not necessarily provide an estimate for the overall impact of the intervention as planned. 6.1.2.2 MODEL ESTIMATES Model 1 estimates the impact of Kenya RAPID’s ICT intervention during the driest periods of the year (January, February, June,30 and September). However, it is also worth seeing how the intervention impacts on-time overall, given the variation in rainfall across assignment counties and the fact that drought may have been less of a challenge than anticipated over the intervention period. 29 Treatment data were provided through October 16, 2020, whereas comparison data is available through August 31, 2020. 30 These months had the lowest average rainfall across both treatment and comparison groups, as well as low differences between assignment groups. Additional sensitivity testing is included in the annex. The sensitivity testing changes the drought variable to include the months with the highest rainfall. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 26 As shown in Figure 13, Kenya RAPID’s sensor intervention did not appear to have an impact on borehole pump on-time during the dry months.31 For the main matching approaches, the model estimates that Kenya RAPID either slightly reduced on-time or increased it by one-to-five percent depending on the matched sample.32 The fact that the confidence intervals for these estimates cross zero (i.e., are both positive and negative) suggests no real impact during the dry months. The optimally matched sample yields an overall positive impact on borehole pump on-time for all months, of around 10 percent additional on-time for treatment boreholes relative to comparison boreholes. In contrast, the nearest neighbor matched sample yields a negative impact of the RAPID ICT intervention. However, this sample includes a notably smaller number of boreholes.33 FIGURE 13: MODEL 1 ESTIMATES FOR ALL AND DRY MONTHS 31 The Model 1 estimates are robust to multiple specification tests (i.e., running a reduced specification of just treatment and drought status on borehole on-time). They are sensitive, however, to how standard errors are treated. As noted above, all models are run with standard errors clustered at the county level. While this may seem like an overly technical issue, it is important to note that there is some variation in outcomes depending on whether or not the model groups or “clusters” estimates based on the county. This approach is justifiable given that the sample itself was selected starting with counties, which is what the treatment assignment is based on. For more see Abadie, A., Athey, S., Imbens, G. W., and Wooldridge, J. “When should you adjust standard errors for clustering?” No. w24003. National Bureau of Economic Research, 2017. 32 Nearest neighbor matching was implemented as a sensitivity test but results in a very small overall sample (n=41). 33 As noted in the Annex J, while this sample has the best overlap in propensity scores among the matched samples and very strong covariate balance, it only includes 13 comparison county boreholes and 28 treatment county boreholes across 11 counties (six comparison, five treatment). FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 27 6.1.2.3 MODEL 1 RESULTS FOR 2019 DATA Limiting the analysis to the 2019 dry months results in point estimates that are similar in magnitude to the full sample (3 percent increase in strategic borehole pump on-time). In contrast to the full data estimates reported above, the unmatched and many-to-one estimates are statistically significant at the 5% level (Figure 14). However, the lower bounds of these effects at the 95 percent confidence intervals are very close to 0 (upper bounds around 6%), and the overall takeaway is similar to the full sample: any observed increases in strategic borehole pump on-time are small. FIGURE 14: MODEL 1 RESULTS FOR 2019 DATA 6.1.2.4 MODEL 2 RESULTS Applying Model 2, which uses a binary on-off measure of functionality at the weekly level, there is no estimated impact from Kenya RAPID’s ICT intervention. Because this model uses a binary outcome— on/off for the week—a logistic regression is run, which results in an odds ratio.34 As shown in Figure 15, for the optimally matched sample, Kenya RAPID and comparison counties had almost equal odds of being on (OR 0.91). Put another way, the probability of Kenya RAPID boreholes being on at some point in the week relative to comparison boreholes was about 50 percent. The small number of boreholes within a limited number of clusters (i.e., counties) leads to large dispersion of the confidence intervals, which speaks to the uncertainty of the treatment estimates (all months). 34 An odds ratio presents the odds that an outcome will occur—in this case that a borehole will be on at some point during the week— relative to whether it was treated or not. For example, an odds ratio of 1 would mean that both comparison and treatment counties have the same odds of having their borehole pump on for the week, while an odds ratio above 1 suggests that treatment counties have higher odds of being on relative to comparison boreholes. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 28 FIGURE 15: MODEL 2 ESTIMATES We find similar Model 2 results for Kenya RAPID’s impact during the dry months using data only from 2019. Overall, none of the estimates are statistically significant, with the odds of a borehole being on over a rolling seven-day period no different between Kenya RAPID and non-Kenya RAPID counties for our main matched samples (Figure 16). FIGURE 16: MODEL 2 RESULTS FOR 2019 FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 29 6.1.2.5 MODEL 3 RESULTS A final specification (Model 3) estimated probability of continued borehole functionality to capture longer periods of non-use that could indicate breakages. Boreholes were considered “dead” or broken if they had average on-time that was less than one hour in total for a four-week period.35 The average on￾time for this four-week period was around four hours. A key measure here is whether or not the borehole pump sensors were registering any activity at or above an hour per month. If borehole pumps were functional, we would expect the probability of a recording of an hour or more within a given month to be high assuming at least some demand over this period. Kenya RAPID strategic borehole pumps more frequently reported on-time of more than an hour over a four-week period relative to comparison counties, 68 percent to 52 percent. This model essentially provides an estimate of how long we would expect boreholes pumps on average to continue showing activity at or above an hour each month. We first model the probability of borehole functionality as a function of the number of boreholes at a given time that are still recording pump activity relative to the total number of boreholes in each assignment group.36 As shown in Figure 17, both treatment and comparison county boreholes had relatively similar continued functionality rates, particularly for the first 50 weeks in the measurement period out of a total of around 114 weeks. After 50 weeks, we see the probability of a borehole pump showing a sensor reading decrease for comparison counties at a faster rate than for Kenya RAPID boreholes. By week 73, we would only expect to see around 80 comparison boreholes still producing sensor readings of an hour or more a month, while all 69 treatment boreholes are expected to still be in our sample. From here, the probability of boreholes showing functionality readings in the comparison groups continues to drop.37 This drop is driven by the fact that many sensors, particularly in the comparison group, disappeared as the evaluation period went on; either due to theft or removal. 35 A period of four weeks was chosen to account for a longer duration of consistent off-time. Based on a review of qualitative data and discussions with the implementing partner, a single week of off-time may not imply the borehole pump is actually broken. For several borehole sensors, there was a very small, but not zero, amount of on-time detected at seemingly random intervals. Seeing the threshold below 1, and inclusive of 0, captures these cases to avoid potentially overstating borehole pump functionality. 36 More formally, Sti= Sti-1(1-dini) where Stiis the probability of a borehole pump i being on during week t , n is the number of boreholes, and d is the number of stoppage events, i.e., a borehole being on for an hour or less for a whole month. 37 Additional results can be found in Annex I. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 30 FIGURE 17: PROBABILITY OF BOREHOLE PUMPS REMAINING ACTIVE FOR AT LEAST AN HOUR BY WEEK BY ASSIGNMENT STATUS AND 95% CONFIDENCE INTERVAL 6.1.3 EQ1 CONCLUSIONS Kenya RAPID did not have a significant impact on borehole pump on-time during the drier months of the intervention from 2018 to 2020, nor did it improve overall borehole pump functionality (based on a binary, on/off measure). EQ 1 sought to understand whether or not use of real-time sensing increased strategic borehole pump on-time during the drought season. Our results suggest that information constraints alleviated through sensor information sharing may not have been the primary challenge facing strategic borehole functionality. We note that Kenya experienced record rainfall during the intervention period, which influenced borehole functionality as well as the availability of “substitutes” – water supply alternatives to boreholes. With respect to our binary measure, Kenya RAPID county boreholes experienced functionality slightly above 70 percent on average and comparison counties experienced functionality slightly below 70 percent on average. We caution that the absence of major benefits is not necessarily a finding that Kenya RAPID interventions did not “work,” but the effect sizes are small enough that we cannot attribute the difference between the average functionality rates in Kenya RAPID counties vs. non-Kenya RAPID counties to the sensor intervention. Indeed, several non-Kenya RAPID counties had average functionality above Kenya RAPID county levels. Regardless of the analytical approach, we consistently find that Kenya RAPID boreholes had an average of around two percent more on-time compared to non-RAPID county boreholes for 2018–2020, controlling for covariates and applying the optimally matched sample during the drought season.38 Similarly, across the optimally matched, unmatched, and many-to-one matched samples, we estimate a 38 See Annex I for more detail. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 31 consistent three percent increase in borehole pump on-time during the dry months for Kenya RAPID counties relative to comparison counties for the 2019 sub-set. The smaller-than-expected effects of the sensor intervention are nonetheless valuable to understanding the costs and benefits of information sharing. Additional context from Evaluation Questions 2 and 3 help fill in the reasons why sensors alone may not substantially improve functionality. 6.2 EVALUATION QUESTION 2: HOW DO WATER MANAGERS PERCEIVE THE IMPACT OF SENSOR-BASED SYSTEMS ON THEIR ABILITY TO ADDRESS BOREHOLE FUNCTIONALITY AND HOW DOES THIS COMPARE TO PERCEPTIONS OF BOREHOLE FUNCTIONALITY IN NON-KENYA RAPID COUNTIES? Key Findings for EQ2 • Limited impact of sensor-based systems on ability to address functionality: taken together, the findings for EQ2 highlight that, although well received where available, the sensor￾based system did not address larger constraints to improved strategic borehole functionality. • Largely positive perceptions of Kenya RAPID system: County- and sub-county-level water managers in Kenya RAPID counties viewed the sensor-based system favorably and said it provided highly relevant and useful data to support water management activities. However, officials in Garissa reported that they did not yet have full access to the data dashboard. • Roles and responsibilities for water management are unclear: Water management systems vary across counties and boreholes. The roles and responsibilities of local water committees, water and sanitation companies, county government, national government, and nongovernmental organizations (NGOs) in this sector are often unclear to water managers, in both treatment and comparison counties. Strategic boreholes are currently not managed differently from other boreholes, though this may be changing in Garissa with the formation of the Garissa Rural Water and Sanitation Company (GARWASCO). • Dedicated budgets for strategic borehole repairs have not been established: A lack of dedicated resources for borehole repairs remains a key barrier to improved functionality, in both treatment and comparison counties. Water user fees are collected routinely or as needed in most of the selected boreholes, but large repairs require funds from the county government, national government, or NGOs and the process for obtaining these funds is complicated. In addition to monetary resources, local areas also lack spare parts and technical capacity to maintain and repair water systems. 6.2.1 KEY INFORMANT INTERVIEW SAMPLE CHARACTERISTICS According to the Kenya RAPID theory of change, the ICT intervention was designed to facilitate faster response to fix broken boreholes and increase borehole functionality by 1) increasing the speed with which information about borehole breakages reaches service providers at the sub-county and county level, 2) clarifying roles and responsibilities for operations and maintenance, and 3) setting aside dedicated budgets and resources to address strategic borehole breakages. The KIIs shed light on these processes and changes that occurred, which may have affected borehole functionality over time in both treatment and control counties. In Round III, we conducted KIIs with 16 individuals across the four selected counties (Garissa and Turkana in the treatment group and Tana River and West Pokot in the comparison group). Key informants included water officers, operations and maintenance managers, water engineers and technicians, and borehole operators. Most of these participants (14/16) were male; Turkana and West Pokot each had one female interviewee. On average, these individuals had been employed in their FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 32 current role for 8.2 years (range: 1–20 years). Annex H contains more detailed information on respondent characteristics. Figure 18 shows the frequency of 12 different codes that were included in qualitative analysis of KII interview transcripts, in treatment and control counties. The most frequently referenced code was Management Structure and Roles followed by Water System Functionality, and Resources for Borehole Repairs. Results for each of these key themes, and related sub-themes, are summarized below. FIGURE 18: FREQUENCY OF THEMATIC CODES FOR KIIS 6.2.2 BOREHOLE FAILURES AND PERCEPTIONS OF THE SENSOR-BASED SYSTEM Among respondents in the two Kenya RAPID counties, local borehole operators had limited knowledge of the sensor-based system, while sub-county and county officials in both counties were highly knowledgeable about this system. We asked all key informants how information about borehole breakages is communicated from the local level up to those responsible for repairs. All respondents were also asked whether they had any knowledge of the sensors that had been installed on boreholes, including what they were for. In Kenya RAPID counties, respondents at the local and sub-county/county level were also asked about their knowledge of the Kenya RAPID intervention and their role in accessing and using sensor-based data. In Garissa, neither of the borehole operators who were interviewed had heard of Kenya RAPID or knew what the sensors were for. Kenya RAPID has noted, however, that it faced implementation “Let me give you one challenge, a layman reports that we have a break down at Kartumba, now for me to visualize what kind of breakage it is, it can be very difficult so I don’t receive the size of that data, I just give attention to the data based on how it was given to me because for me to move from this point go to site and assess the extent of the breakdown it’s almost impossible. We have no logistics, no vehicles, no what so you cannot even get there. Sometimes you are lucky when they report a breakdown in Kartumba and then you have a job closer to the borehole site with another organization then you combine.” - KII, Tana River FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 33 challenges in Garissa due to security concerns. Meanwhile, both the Director of Water Service and the sub-county water officer explained the system in detail but also noted challenges with implementation and lack of full access to the data dashboard. Both individuals had a favorable assessment of the system’s potential, while also noting that they did not yet have full access to the data to be able to see problems in real time. Still, the Director was very familiar with the system and noted many benefits related to having this information. He explained that the sensors provide information through a system called the Water Management and Service platform (WAMASP), which uses color coding to show which boreholes are functional. When the system shows an issue with a borehole, the Director said they call the Kenya RAPID team to see if the sensor is still working. However, the Director noted that the internet infrastructure to connect to the real-time WAMASP system had not yet been connected to the office, so they are only able to examine and analyze past data. He noted that these data are still quite useful for analyzing use patterns and making annual development plans. Meanwhile, the sub-county officer also reported that he was not able to view the data dashboard, explaining that he had an app installed on his phone at one point but lost his phone and had not re-installed the app. He sees the system as highly relevant and useful and recommends that the program be given more time to complete its work. In Turkana, implementation and use of the sensor-based system appears to be more advanced. Of the two local operators interviewed, one had not heard of Kenya RAPID, but the other noted that the system actually alerted them to a minor issue with the borehole before they were aware of it, which was subsequently addressed. Meanwhile, one Turkana sub-county officer said that he had the app and used it regularly to get information on borehole issues. She noted that time to repair is shorter for boreholes with sensors and said she would like to see sensors on more boreholes in her sub-county. One challenge noted by this individual involved network connectivity, explaining that “some places need a stronger network for you to use the app” (KII, Turkana). The other Turkana sub-county officer was also aware of the system and able to explain how it operates. This person said she received a report every Monday telling her which boreholes are working, which ones are not working, and where the sensors are not transmitting a signal. For these latter cases, she then gets in touch with SweetSense so they replace the sensor or see what the problem is. This person noted that with the conclusion of the Kenya RAPID project, an outstanding issue involved handing over management of the sensor-based system from SweetSense to the county. Finally, this officer suggested that the system would be even more useful if it was able to predict breakages before they happened, allowing water managers to take action to prevent these failures rather than responding to them when they happened. “Yes, the operator will inform you the problem, number two you can also analyze the duration during the breakdown and the fixing of this borehole. Once there is a breakdown, the time taken to produce a ticket and fix the borehole is noted in the system. You can see how long you took to attend to this breakdown because it will even show you the when the borehole stopped and when it resumed. So, you will see the graph, in between you can see I took these hours or I took a day. Every year we make plans, departmental, annual development plans so when we were making annual development plans, we usually use this data we are talking of. I advise as a technical person, I advise both the chief officer and the CEC you need to have to factual things like ABC based on the data extracted. We are able to approach the boreholes management based on the data seen and we able to know if there were charges on water user. Also we get to know the number of people who have been extracting money from water users. Sometimes we put a smart meter for the accountability of how much money was collected.” - KII, Garissa FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 34 It’s still under the SweetSense guys but I think they are planning before the project ends, they were planning to set an office at the county level then we will have one of our staff I think the IT guy, he will be the one to manage.” - KII, Turkana One additional challenge observed by the RAPID implementation team involved intentional and unintentional disconnections of Kenya RAPID sensors. KII respondents were asked whether they were aware of such issues in their jurisdictions. In Garissa, the Director said they had not encountered this challenge, while the sub-county officer said some sensors had been disconnected there. Meanwhile, in Turkana, both sub-county officers said disconnections had been a challenge in that county. One respondent said that some operators disconnected sensors because they did not understand what the sensor was doing and thought it was interfering with the borehole’s operation. Both Turkana officers also reported instances where rats had chewed through sensor cables. “It [Reporting time] varies because we have some boreholes that are not installed with sensors, where the Kenya RAPID program and CRS did not install all boreholes with sensors so we have some boreholes that we depend on the operator to call us and report any breakdown. Those boreholes normally takes time either three days or one week to report and this also depends if he has the credit to call us, some areas do not have network and it take them time to come to Lakori and report.” - KII Turkana 6.2.3 WATER MANAGERS’ PERCEPTIONS OF WATER SYSTEM FUNCTIONALITY Many of the same themes were mentioned with roughly the same frequency across treatment and comparison respondents, though there are key areas of divergence. In discussing issues around borehole functionality, many KII respondents mentioned seasonal variation of borehole use and functionality issues. In Garissa, West Pokot, and Tana River, water managers said that borehole use was substantially higher during the dry season, when other sources of surface water (rivers, impoundments, ponds) were not available. By contrast, operators in Turkana said that the boreholes represented the main water source year round. In this county, some KII subjects noted that water availability is actually lower during the rainy season since the boreholes rely on solar power and cloud cover reduces available energy. “It differs seriously because during rainy season people and animals get water from dams and other water catchment area but during dry season the congestion and overcrowding is very high.” - KII, Garissa Regarding breakages and other issues affecting borehole functionality, seasonality was also a key consideration. Many respondents noted that breakdowns were more frequent during the dry season because of more use. KII respondents mentioned several different components of the water system that are prone to failure or functionality issues. The most commonly mentioned issues involved broken pipes, particularly in Turkana and West Pokot. A borehole operator in Turkana suggested that pipe breakages are due to high volume of water being pumped during the dry season, while another Turkana respondent said pipes also broke due to the heat. Other system components prone to breakages or functionality issues included generators and solar panels. Some respondents complained that their water tanks were too small, leading to challenges “When it’s wet season we don’t have much breakdowns because there is limited operations of those machines but when it’s dry this machines are exposed to running for long hours hence they break down every time. During the wet season there isn’t a lot of problems because the water is available, they even abandon pumping, they use those water pans, even others use the stagnant water on the road so during that time we don’t have any problem.” - KII, Garissa FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 35 maintaining sufficient water supply. Water quality issues were mentioned by borehole operators in Garissa and Tana River. In Garissa, one borehole operator said that the water from the borehole was too salty for human consumption and was mainly used by livestock, which is consistent with the fact that the FGDs for Garissa both targeted male livestock owners as the primary borehole users in these areas. KII respondents in treatment and comparison counties were asked about time to repair strategic boreholes when they broke. All water managers said that it depends on the circumstances and type of breakage. Minor issues are typically handled locally and can sometimes be fixed within hours or days. For example, managers in Garissa, West Pokot, and Turkana also reported that time to repair locally is shorter in the dry season, when there are no alternative water sources, compared to the rainy season. For more substantial issues, local water managers inform the sub-county or county government, and repairs can take months. For strategic boreholes managed by local water committees, and even for those managed by WASCOs, KII respondents noted that major repairs require the assistance of the county government. At the local level, operators noted that information about breakages reaches them quite quickly, within hours, either through their own observations or through communication from local users. Most KII respondents also said that information about breakages travels quickly—within a day or two—from the local level up to the county or sub-county, typically through phone calls from the local operator or water committee. At that point, the time to repair depends on a few factors. First, respondents often noted that diagnosing the reasons for the breakage and needed repairs can take time: someone from the county or sub-county must travel to the borehole, take pictures, and ascertain what resources are needed to fix the issue. One respondent from West Pokot reported that this can take as long as two weeks. Second, funding for repairs was often mentioned as a key barrier. The reported time to repair for strategic boreholes in treatment and comparison counties, based on recall of the last breakage, was between a few days to several months. The longest time to repair was for one of the selected boreholes in Garissa (treatment), which the local borehole operator said had been broken for seven months at the time of the interview, because users had access to an alternative source. The operator of the other borehole in Garissa said that it had not had any issues recently but told a story about a previous time (before the Kenya RAPID project) when the borehole was broken for five years and attributed this to political factors and mismanagement. In the other treatment county, Turkana, operators of the two strategic boreholes visited said that repairs typically take two to three months, and one month, respectively. In Tana River (comparison), one borehole manager reported that a borehole had been broken for a year without being fixed, while “The engine of generator had started to malfunctioned and new engine was brought in 2017 by the current Member of Parliament who was aspiring then. Since 2013-2017 there was no water at all. It was total water crisis in this town. There is billboard erected before you enter the town that indicating this borehole expansion and services were done by county government and national government which is a total lie. The billboard further indicates the funding on this borehole was done through specific funds set aside for drought management. We never saw it; it was drained in someone’s pocket.” - KII, Garissa “Whenever there is a problem, we call the water officers immediately to come and fix the problem because we cannot stay without water. The delay is usually from the side of the officers coming to fix the problem that is when it may take even a month.” - KII, Turkana “The pipes got broken, they were leaking after the cows trampled over them. So, what I did is, I called the sub-county water officer immediately, by good luck his number was going through, I told him that the pipe had broken down and they need a replacement. After two days he came with a new pipe and replaced the worn-out ones. This is because he has been replacing the pipes, so he knew the sizes very well. In two days, the pipe had been replaced successfully.” - KII, West Pokot FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 36 the other said their borehole took six months for the county to repair the last time it had broken. In West Pokot (comparison), both borehole operators reported recent instances when pipes broke and were fixed by the county or sub-county within days. 6.2.4 PERCEPTIONS OF WATER SYSTEM MANAGEMENT STRUCTURE AND ROLES Part of the ICT intervention involved clarifying roles and responsibilities for water system management. Under Kenya’s devolved system of water governance, most boreholes are managed by local water committees. Local communities are primarily responsible for repairing borehole breakages, with larger issues are elevated to the sub-county and county. Of the eight boreholes selected for qualitative data collection, seven were managed by these local committees. The size and composition of these committees varied from four to 17 individuals. In most cases, committees were elected by the local community or water user groups, while in one borehole in West Pokot, the local church drilled and manages the borehole and appoints members of the local committee. Just one selected borehole, in Tana River, is managed by a local Water and Sanitation Company (TAWASCO). These companies typically operate in more urban areas, while the selected strategic boreholes were all located in rural areas. However, Garissa’s Director of Water Services also reported that a new GARWASCO had recently been formed and is intended to take over management of strategic boreholes in the county. In boreholes managed by local water committees, these organizations handle day to day operations and minor repairs and seek the assistance of the county or sub-county government for larger repairs. Many of the interviewed local borehole operators expressed frustration with perceived slow response times from county governments. In Tana River, a county water engineer noted that there is some lack of clarity around roles and responsibilities when it comes to strategic boreholes. He stated an expectation that the county government should monitor operational activities of strategic boreholes and indicated this responsibility has fallen to the community. NGOs are also key players in the water management structure. Several respondents noted that NGOs play a large role in building boreholes, and many also step in to help when boreholes break. However, their roles are also often informal and not clearly defined. A borehole manager in Turkana expressed frustration that boreholes were built by NGOs, who then left without teaching the community how to operate and manage them. Meanwhile, an operator in Garissa said that NGOs serve as a backstop resource that they go to when they cannot get help from the government. Similarly, Tana River’s county water engineer said that they often look to NGOs for materials and support when they cannot address issues on their own. KII respondents were also asked about the management structure for strategic boreholes specifically, including whether they knew how and why certain boreholes were designated as strategic, and whether management of these boreholes differed from other boreholes. Awareness of the strategic borehole system was higher among sub-county and county water officials, all of whom were aware of this designation. These individuals explained that the factors used to designate certain boreholes as strategic included serving a large population of people and/or livestock, having a high and consistent yield, and being located in an area with few alternative water sources during the dry season. The sub-county deputy water officer in Turkana (treatment) specifically mentioned NDMA and their role in designating strategic boreholes. “We have requested support from the county government so many times and in vain, forget about the national government. We hope NGOs could help.” - KII, Garissa FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 37 Knowledge of the strategic borehole designation was more variable among local borehole operators; two of the eight respondents in this group (one in Garissa and one in Tana River) had not heard of this system, while six said they were aware of this designation and provided varying explanations for what constituted a “strategic borehole.” One operator in Turkana said he thought that strategic boreholes were powered by solar, while others used generators. Other operators generally pointed to the reliance on these boreholes as a main water source for people and livestock during drought. Across the board, all KIIs reported that the management structure for strategic boreholes did not differ from that of other boreholes. However, as noted above, in Garissa (treatment county) plans appear to be in motion to put the recently formed GARWASCO in charge of strategic boreholes in the county. 6.2.5 RESOURCES FOR BOREHOLE REPAIRS The Kenya RAPID ICT intervention also aimed to ensure dedicated budgets for strategic borehole repairs. Key informants provided information about the resources required to repair broken boreholes, and challenges accessing these resources. Figure 20 shows the frequency with which different resources were mentioned in treatment and control counties. Under the devolved governance system, local communities are primarily responsible for repairing borehole breakages, with sub-county officials and then county officials serving as key points of contact for elevating any issues or making resource requests. For five of the eight selected boreholes, user fees are routinely collected, and these funds are used to repair minor issues and support borehole maintenance. In two of the remaining boreholes (one in Turkana and one in West Pokot), fees are collected on an ad-hoc basis, when there is an issue that needs to be addressed; and the West Pokot borehole managed by a local church does not collect any user fees; operators reported that users expect the church to maintain the borehole, since the church built it. Multiple operators reported that the ability to collect fees varies throughout the year (see Table 4). Interestingly, in Garissa, a local borehole operator stated that fees are harder to collect during the dry season because livestock owners have less available money during this time, while in Tana River an operator reported that fee collections are higher during the dry season when borehole use increases for both human and livestock consumption. More generally, “I think if we can have a system that does not use clamps because currently what we have there is a transmitter and the sensor itself so sometimes people within a given community, some of them are malicious, they usually go and disconnect that clamp and make that sensor not to function, so if we can have a sensor that is not directly connected to where people can access that connecting wire, it could be okay.” - KII, Garissa “The community living here are herders and the use their money to pay the children school fees and other livelihoods. During dry/drought season they mostly undergo crisis to feed their family forgets about paying water fees. But during rainy season the situation is better because water is available in dams and animals’ market is good to pay back debt of the drought season and some refused to pay it. So, it differs.” - KII, Garissa “Water users should be charged a fee this is to help get money for maintenance. When we have external people either government or other organization come and repair, the community members just take it easy and they mishandle available the equipment’s within the borehole especially the taps they break most often.” - KII, West Pokot “The Red Cross taught us with committee on how to get fee from the community, but the community refused to pay and said that water is free. The community need to be educated on water management. The church always repairs where they can.” - KII, West Pokot FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 38 multiple operators noted that it can be difficult to enforce or implement fee collection. In Tana River, an operator reported that people will often say they do not have enough money to pay fees, creating a difficult decision for operators about whether to provide access. In West Pokot, where neither sampled borehole requires routine user fees, both operators noted this as a key barrier to better maintenance. FIGURE 20: KII REPAIR THEMES AND MENTIONS TABLE 4: REPORTED FEES BY BOREHOLE Borehole User Fees Collected? Details Garissa 1 Yes, routinely Per animal; 2 Ksh per 20 liters Garissa 2 Yes, routinely Per animal; 2 Ksh per 20 liters Turkana 1 Yes, ad hoc No routine fees; “The money is just given whenever there is a problem.” Turkana 2 Yes, routinely 300 Ksh per household per month Tana River 1 Yes, routinely 5 Ksh per jerrycan Tana River 2 Yes, routinely Fees collected by TAWASCO; used for water treatment, maintenance, paying operators West Pokot 1 Yes, ad hoc Ad hoc collection when maintenance is needed. West Pokot 2 No Church manages borehole and users expect water to be free. When user fees are not sufficient to address a given issue, funding must be requested from the county, national government, and/or NGOs. All KII respondents noted significant challenges with this process. One issue involves the timing of funding allocations throughout the year, sometimes related to the financial year. As one respondent in West Pokot put it, “before money comes from Nairobi, we won’t be doing anything, that is from July to August but when we get money then it becomes almost constant.” Others noted that these financial years are not consistent between the federal government and NGOs/donors, which can create additional confusion. Another borehole manager in Garissa echoed this sentiment, noting that having access to a pool of funds that was easier to access would help to address breakdowns quickly, rather than having to go to NGOs for money. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 39 Budget constraints are only one challenge for borehole management. One Turkana respondent noted that access to parts and tools also presents a challenge. The county engineer in Tana River also expressed frustration at the lack of consistent funding for borehole maintenance, noting that this funding lags behind resources for drilling new water sources. In Garissa, one of the treatment counties, a sub￾county officer perceived that Kenya RAPID had made it easier to access funds for repairs. Water managers also emphasized a need for staffing and capacity to address borehole maintenance issues, as well as spare parts and other equipment, and vehicles. Staffing and capacity needs include skills in borehole repairs and maintenance, as well as capacity to write proposals for funding to the county government or NGO. Several water managers in West Pokot and Garissa discussed challenges accessing spare parts for repairs, noting that these parts had to come from Nairobi or other distant areas, causing delays. Some said that having consistent access to spare parts locally could speed the repair process. Another borehole operator in Turkana noted that they also lacked the right tools to conduct repairs, making them reliant on outside technicians to address issues. Respondents in all four sampled counties mentioned transportation issues and lack of vehicles as constraints on their ability to maintain and repair boreholes. “Yes, So, you see they normally want to sink more boreholes than the one existing, the county normally budgets for new boreholes rather than maintaining the existing ones. They normally expect the communities to maintain the existing boreholes, but the communities are very poor, they cannot manage with the small funds they collect.” - KII, Tana River “What am saying the request process is the same but now the release of funds has changed because with Kenya RAPID there are have funds so when we make requests they avail them immediately.” - KII, Garissa 6.2.6 EQ2 CONCLUSIONS While water managers see the ICT intervention as a positive and welcome addition to their management toolkits, their ability to take advantage of the system’s potential benefits is limited by other significant barriers to faster repair times and increased water service reliability across treatment and comparison counties. County and sub-county water managers in Garissa and Turkana had positive perceptions of the Kenya RAPID intervention and described the data provided by sensors as highly relevant and useful for water management decision￾making. In Turkana, in particular, a local borehole operator reported that the sensor-based system had alerted sub-county officers to an issue with their borehole before it was noted at the local level and were able to repair the problem promptly. In Garissa, sub-county and county water managers described the benefits of the sensor-based system and its data in detail, but also reported that they did not yet have full access to the data dashboard and expressed a wish that the program could continue longer to achieve its full benefits. While implementation data shared with the evaluation team revealed that roles and responsibilities for water management had been clarified as part of the Kenya RAPID activities, qualitative data indicate that confusion remains on the part of water managers about these roles. Kenya’s devolved system of water governance is complex; in most (but not all) cases, local water committees manage rural boreholes and are responsible for minor repairs, while larger breakages must be addressed by some combination of sub-county, county, or national governments, and/or NGOs. Water managers are not always sure who to turn to when a borehole breaks. Managers in both treatment and comparison counties indicated that strategic boreholes are not managed differently from other boreholes. However, in Garissa, officials indicated that the newly formed FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 40 GARWASCO will be taking over management of strategic boreholes; RAPID implementation data indicates that GARWASCO staff received training on the data dashboard in March 2020. This institutional change may help streamline roles and responsibilities for these boreholes. However, ensuring that local communities continue to be involved and engaged in water management processes will be important. Along with lack of clarity around responsibility for water management, KIIs confirm a continuing shortage of dedicated resources for borehole repairs. Water fees vary widely and are not routinely collected in three of the eight sampled boreholes; even where they are collected, they are not sufficient to cover the costs of large repairs. In these cases, information about breakages must be communicated up to the sub-county or county level (which the Kenya RAPID intervention facilitated), and then budgets must be allocated for repairs. This is a complex process that often requires navigating federal financial calendars, budget allocation processes, and grant proposal writing. When government funding is not readily available, NGOs may step in to fill gaps, but the precise roles and accountability for these organizations is also unclear in many cases. Beyond financial resources, local water committees lack spare parts, equipment, and technical capacity to address functionality issues. While the Kenya RAPID intervention acknowledged these constraints and may have attempted to target these barriers, systems change is a slow process and appears to be ongoing as of the end of the intervention period. 6.3 EVALUATION QUESTION 3: DO KENYA RAPID’S SENSOR-BASED SYSTEMS AFFECT USER PERCEPTIONS OF BOREHOLE FUNCTIONALITY AND ACCESS? Key Findings for EQ3 ● Given broader challenges in water service delivery and the small, measured impact on borehole functionality (EQ1), the sensor-based intervention had little impact on users’ perceptions of borehole functionality and water access. The Kenya RAPID sensor-based intervention sought to speed access to information about borehole breakages to sub-county and county-level water managers so these managers could identify and address issues more quickly. This intervention was not designed to address many of the challenges highlighted by users, such as congestion and long waiting times, or breakages in distribution pipes and taps bringing water from boreholes to people’s homes and villages. Users in treatment and comparison counties all cited these issues as continuing challenges limiting reliable water access. ● Water management systems, and users’ views of these systems, vary widely: Boreholes in rural areas must provide reliable water supply for both livestock and humans, and the systems developed to manage these boreholes vary across and within counties. Most boreholes are managed by local water committees, which are viewed by local users to be accountable and accessible but also lacking resources for major repairs. ● Water users encounter a wide range of issues in functionality and access in treatment and comparison counties: Many users view breakages as a key challenge that limits water availability, particularly during the dry season when other water sources are not available. Crowding and congestion at boreholes lead to long waiting times. Access issues vary seasonally and have a particularly large effect on vulnerable groups, including women and the elderly. 6.3.1 FOCUS GROUP DISCUSSION SAMPLE CHARACTERISTICS In Round III, the evaluation team held eight FGDs with local users of the selected strategic boreholes (two FGDs per county) during September 17–20, 2020. As shown in Table 2 above, the selected boreholes include a mix of power types, range of households served, and some variety in the average on-time at Round II. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 41 Both FGDs in Garissa were held with males only, given that these boreholes were used primarily for watering livestock, while the remaining six discussions were held with women at boreholes primarily for domestic use. In total, 69 people participated in the FGDs, with an average of 8.6 people per discussion. FGD participants were 42 years old on average, with an age range between 22 and 78. Annex H provides further details on age and village of FGD participants. 6.3.2 USERS’ PERCEPTIONS OF WATER SYSTEM FUNCTIONALITY Borehole and water system functionality is the key focus of this IE, and ultimately, the objective of the ICT intervention was to meet water users’ diverse needs more reliably through the use of information sharing on pump functionality. FGD participants highlighted several continuing challenges in this area, cutting across both treatment and comparison boreholes. Figure 21 shows a breakdown of sub-codes under borehole functionality in treatment and comparison counties. FIGURE 21: FUNCTIONALITY THEMES BY ASSIGNMENT Participants in all counties mentioned borehole breakages as an ongoing issue. In Garissa, male livestock owners complained that long repair times caused challenges and hardship: “Most of the times the borehole will take a week to be repaired and that causes a lot of inconvenience and sufferings. It will be better if few times is taken to repair when it breaks down.” (FGD, Garissa). Similarly, a female respondent in Tana River described repeated experiences with breakdowns and slow response times. However, other Tana River respondents disagreed, saying that breakdowns were more of an issue in the past but had not occurred since the current newer borehole was built. Figure 22 shows the sentiment polarity for each county for all segments of the FGDs related to water system functionality. Sentiment analysis takes all of the words in the text and assigns them to either a positive or negative sentiment group using a database called the National Research Council Canada (NRC) lexicon. Sentiment polarity is simply the difference between positive and negative sentiments, which helps provide an overall measure of sentiment.39 Results show that water users in Garissa and 39 For more, see http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 42 West Pokot had overall negative sentiments about borehole functionality, while Tana River and Turkana users viewed functionality more positively. FIGURE 22: WATER SYSTEM FUNCTIONALITY SENTIMENT POLARITY BY COUNTY Several respondents noted that functionality varies by season, though there were differences across counties in opinions about when breakages are more likely. In Garissa, respondents said that breakdowns were more likely in the dry season because the generators overheat and use is high, while in Turkana water users said that breakages are more common during the rainy season, particularly broken pipes. Pipe breakages were some of the most commonly reported functionality issues in Turkana and were mentioned as a key challenge in West Pokot. Users also mentioned issues with solar powered borehole pumps. One user in West Pokot explained that because the solar pumps only operate during the day, water is not available early in the morning or in the evening. In Turkana, echoing comments heard during the KIIs, water users said that reduced solar energy during the rainy season reduces the amount of water that is pumped. To address this issue, one Turkana water user suggested that generators could be used as a backup energy sources during the rainy season: “We request that the generators to be returned and to be used during cloudy and rainy season so that we do not have to go without water supply.” (FGD, Turkana). In West Pokot, one water user expressed similar dissatisfaction with solar pumps, saying that the quantity of water pumped had decreased since it was installed. However, another user in Garissa stated that a solar pump installation had led to a decrease in water user fees. Water users in all four counties raised concerns about the size of water storage tanks, expressing a “…During short rainy season we have a problem, because when it rains there is erosion of soil or removal of soil which leads pipe exposure to the surface. When pipes are exposed, they are damaged either by big animals like camels, cattle and even vehicles causing breakage and leakages. We don’t have the tools or the equipment to help in repairing of the pipes and so we have to wait for the experts nearby and that may take even two weeks without water.” - GD, Turkana FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 43 desire for more and larger tanks to increase storage capacity and distribution. A female respondent in Tana River suggested that the government install a tank in every village: “As I told you I have a farm and I told you I tried to plant for three seasons and I have failed, the government should give every village a tank, even if they are going to sell the water to us we are the ones who are going to benefit.” (FGD, Tana River). These comments relate to a broader, widespread concern about water shortages. Respondents agreed that the current system does not supply enough water for all users and use types. Male livestock owners in Garissa and female water users in West Pokot highlighted challenges supplying sufficient water for both people and livestock. Users in Turkana and Tana River said that population growth over time was contributing to water shortages. A few respondents noted that the lack of sufficient water supply limited opportunities for other productive activities such as farming and businesses. Water quality concerns were mentioned less frequently than water quantity. Quality issues were most prominent in Garissa, where most respondents agreed that the water was too salty for human consumption and was mainly used for watering livestock and other household chores. A few users in other counties also raised concerns about water quality. One participant in Turkana said, “Sometimes the water from the tank is dirty due to the rust in the tank.” (FGD, Turkana). Another participant in Tana River expressed her concerns: “You buy water at the borehole but the water is not clean, the water is not treated. I know there is a way I can treat water in my house. At times you can just feel in your heart that this water is not clean.” (FGD, Tana River). In contrast, a West Pokot participant said that the quality of borehole water was substantially better than alternatives and that health issues in the area improved after the borehole was built: “Since we started drinking this water, typhoid went down in this area. So, this water is very good and clean for drinking compared to river water.” (FGD, West Pokot). “During rainy season when there are floods, the flood carry away the pipes and even breaks. In addition, clouds coverage also prevent solar energy from reaching the solar panels and thus water is not pumped well from the borehole leading to less water supply.” - GD, Turkana “The livestock that depend on it are so many. It is not enough for both house chores and for livestock, we fetch in turns even the livestock come and drink in turns. If you come today you are not supposed come for the next two days.” - GD, West Pokot 6.3.3 BOREHOLE ACCESS AND USE Consistent with findings from KIIs, borehole users reported that borehole use varies by season. In Garissa, Tana River, and West Pokot, FGD respondents reported that they primarily use surface water (rivers, streams, dams) during the rainy season and rely on the borehole only during the dry season. In Turkana, both groups said that the borehole is their main water source year-round. As noted above, Turkana users also—somewhat paradoxically—noted that water shortages are a bigger issue during the rainy season due to reduced solar power for the borehole pumps. Water users noted a number of challenges affecting water access and use. Chief among these were issues related to excessive demand leading to water shortages during the dry season. Several water users mentioned traffic and congestion around boreholes. One user in Garissa noted: “During the traffic it is very elusive getting the water or access the borehole as there a lot of scrambling for the access of water from the borehole.” (FGD, Garissa). Other livestock owners from Garissa attributed congestion to people coming from surrounding areas to access the borehole during the dry season and said that this causes long queues and waiting times that can exceed a week. Respondents in Tana River and West Pokot also reported long queues to access water. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 44 Water users also explained that a shortage of water sources causes them to travel long distances during the dry season. The water users who participated in FGDs were primarily selected from communities near the selected boreholes, but users reported that others travel from far away to these boreholes during the dry season. One water user in Turkana reported that people journey up to three hours to reach the water point. In West Pokot, one participant said that she herself walks from far away to collect water at the selected borehole. Other West Pokot water users reported that they sometimes travel to Uganda during the dry season because the nearby borehole is so congested. Water users also reported that water shortages can lead to conflict and violence, with one male Garissa FGD participant explaining, “Sometimes when there is water shortage, violence erupts due to the water shortage and sometimes brings tension among people.” (FGD, Garissa). Like many other aspects of water use, the impacts of conflict are closely tied to gender. A female water user in West Pokot stated, “During the dry season almost everyone and the livestock depend on the borehole entirely. This is the season where the borehole breaks down most of the time, people fight because of water scarcity. […] Men dominates during this time of the year because of their livestock thus for women we cannot easily access the water except at late hours or very early in the morning. (FGD, West Pokot). Even more troubling, another male FGD participant in Garissa noted, “During water crisis, women fall vulnerable for those with those with bad intention of rape and that put their lives in danger.” (FGD, Garissa). Other respondents in West Pokot also noted how the impacts of water shortages and queuing were particularly difficult for certain vulnerable groups, such as pregnant women, children, and the elderly. Analyzing these FGD data using a machine-driven approach reinforces findings from the content-driven approach above. As in Round I, the evaluation team broke each sentence into two-word pairs to see which words have the highest correlation with mention of women. As shown in Figure 23, the issue of pregnancy and disability are most strongly correlated with a respondent mentioning women. As noted above, water access is particularly challenging for vulnerable groups. Figure 23 highlights just how prevalent this issue is in the text and how women’s access to water is directly related to many of the issues of vulnerability. For most of the selected sites, users collect water by traveling to the borehole. The exception was one of the selected boreholes in Tana River where respondents said that many people have tap water that is piped to their homes from the borehole. This allows users to avoid many of the challenges involved in water access (congestion, conflict), but breakages still occur and affect water supply. “The disabled, pregnant women and those that delivered the other day. These are the people who face challenge of fetching water because of long queues and distance covered to access the borehole from their homes.” - GD, West Pokot “No, the elderly have to queue long same in equal pace as others since everyone one is longing to get little water in the evening. The younger women run and outsmart the elders when going to the borehole, same to the disable. The 20 liters is heavy for them to carry because of distance back home.” – GD, West Pokot 6.3.4 PERCEPTIONS OF MANAGEMENT STRUCTURE Water users provided information about how they view water management systems. The evaluation team conducted sentiment analysis using the NRC lexicon of FGD participants’ feelings expressed in connection with different water management actors (borehole operators, local water committees, county government, NGOs, national government). The sentiment results show that most of the responses to questions related to borehole management structure and the associated borehole FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 45 managers can be broadly categorized as “trusting,” while a much smaller portion of the responses are categorized as angry or disgusted.40 FIGURE 23: TEXT CORRELATION WITH MENTIONS OF WOMEN Fees and affordability were a major theme in users’ discussions of water management systems. As previously noted, the fee system varies across selected boreholes, with no fees collected routinely in West Pokot and one of the Turkana boreholes, while some fee structure is in place at the other boreholes. Some users acknowledged that fees are necessary to maintain the water system. A Turkana respondent stated, “Yes, they need to pay. Because whenever there is need for maintenance of the pipes, that money is used to repair the pipes. The committee does not have any other sources of money apart from the money collected from the members. They use the money.” (FGD, Turkana). Meanwhile, others thought that the current fees were too high, and many agreed that inability to pay fees was a significant problem limiting water access for some users, particularly the elderly. One elderly respondent in Tana River said that she was given water for free when she was not able to pay. In response to water shortages, several water users reported that water managers enacted rationing schemes at certain times of year. In Garissa, livestock owners explained that in times of shortage, priority is given to home consumption first, then to certain types of livestock (goats and cattle), while camels are supposed to go to the river. In Turkana, users described a system for allocating water from one borehole to multiple villages in turns: “There are plans and strategies in water use. It is a rule that only one village gets water in a day. Villages alternate in water channeling. If one village gets water today, the other waits.” (FGD, Turkana). Tana River respondents also mentioned rationing schemes during the short rainy season. In discussing perceptions of water management schemes, FGD participants were also asked whether they knew who to contact when there was an issue with their borehole, and whether they felt that their 40 Mohammad, S., Kiritchenko, S. and Zhu, X. "NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets." Proceedings of the seventh international workshop on Semantic Evaluation Exercises (SemEval-2013), June 2013, Atlanta, USA. arXiv preprint arXiv:1308.6242 (2013). FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 46 voices and concerns were heard when it came to water needs. Water users’ satisfaction with the current system varied across individuals and across counties. In Garissa, users of both boreholes knew who the local borehole manager was and had contacted him often, even daily. These individuals were able to name the local technician in charge of repairing the borehole, and users of one of the boreholes noted that a member of parliament had stepped in the last time their borehole had a serious problem. In Turkana, most users seemed generally satisfied with their access to water managers and the process for resolving issues. At one borehole, a water user talked about community meetings held to discuss water, while noting challenges collecting water fees: “We usually have meetings, and we talk about how we can properly manage water usage in this area. Most of the time, what we say is just mere talk because sometimes it is not practical considering our living situation in this area. Sometimes even the 300 monthly contribution is hard to contribute and we end up with only few people contributing in a month. Out of 20 users, only 8 might be able to pay.” (FGD, Turkana). Users of the second Turkana borehole explained that they contact the secretary of the water committee or the water manager in person or by phone, and that they communicate frequently and resolve issues effectively: “We speak with them often. I even spoke with them last week on Wednesday when my tap was not working properly, and they came and fixed it.” (FGD, Turkana). In contrast, water users at the Tana River borehole managed by the water company, TAWASCO, expressed frustration at a lack of input into decisions about water management, particularly when it comes to fees. One user stated, “We are never involved, they just plan and do what they want, you just see it on the bill, and you cannot change anything.” (FGD, Tana River). Others felt similarly, expressing a desire to be more involved in management decisions. Another Tana River water user thought that public health officials should also play more of a role in water management issues: “There are people from the public health, when there was shortage of water they come and ask what is going on or if the water is dirty then they would intervene, nowadays they don’t come, there is a borehole near the hospital so I don’t know if they have stopped coming here but they know that we need them because if there is any problem with the water, they need to know.” (FGD, Tana River). In West Pokot, users of one borehole managed by a local water committee, expressed some confusion around the water management structure as well as some distrust of the water committee. Multiple FGD participants in this group said they did not know who was responsible for fixing the borehole when it breaks. Others said that anyone can contact the water manager when there is an issue. Some users complained about the state of the boreholes and said the local water committee had not done enough to fix the repairs. One user stated, “The committee has not called us for repairs, yet they see the pipes are leaking day and night. The committee eat money meant for repairs.” (FGD, West Pokot). The other West Pokot borehole is also managed by a local water “There are those people who live a comfortable life and there are those people who live a miserable life. You will find an old person who struggles so much to get twenty shillings. You know, at that age you do not have strength to go fetch water everyday unless you have a tank. […] I do not know how they were paying before but at my home we paid a flat rate of KES250. If you spent more than that then you had to add money, but you were supposed to pay 250 every month. The flat rate was later increased so even those who have piped water do not have a meter, the flat rate was increased to 550, right now it’s 600. This is high and there is someone who can afford to pay that but there are others struggling to get that 600. You must pay 600 whether you have used the water or not, so even if I do not get water for the whole month, I will still pay the flat rate. So, our main question did this water supply came to help us or it came to oppress us because most people have decided to dig their own boreholes, so we are suffering a lot.” - GD, Tana River “No money is set aside for future use for maintenance. The manager depends on the community for minor repairs and maintenance. For major repairs and maintenance, it is the county government and other well-wishers like Red Cross.“ - GD, West Pokot FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 47 committee, but members are appointed by the bishop of the local church that constructed the borehole. Water users said they were able to contact the water manager easily, meeting him at his home near the borehole or about town or calling him. While water managers interviewed in KIIs reported that the church pays for all minor repairs, water users claimed that the community does contribute for minor issues on an ad hoc basis, and more major issues are handled by the county. 6.3.5 EQ3 CONCLUSIONS The sensor-based intervention does not appear to have had a large impact on water users’ perceptions of water access and reliability. This likely due in large part to limited impacts on borehole on-time (see EQ1), as well as to a suite of other continuing water access challenges highlighted by users across treatment and comparison counties. FGDs with water users in treatment and comparison counties paint a picture of a highly heterogeneous landscape of water uses, management systems, and functionality challenges. In Garissa, use of selected boreholes is highly seasonal and primarily for livestock, while the boreholes selected in Turkana are used year round for both people and animals. In most cases, water users must travel to the borehole— often from far away—to collect water, encountering congestion, queuing, and even conflict along the way. However, for one of the selected boreholes in Tana River, water is pumped from the borehole to people’s homes, reducing many access challenges but introducing different problems around pipe breakages, for example. We also observed variation in the level at which water is managed, and in users’ perceptions of different management systems. The Kenya RAPID sensor-based intervention was designed to facilitate water system management at the county level, aiming to provide water managers with better and faster information about borehole breakages to facilitate the allocation of resources for repairs. Meanwhile, nearly all of the selected boreholes are managed by local water committees whose members are locally elected and widely seen as accountable to the people they serve but lack sufficient resources to maintain and repair boreholes adequately, necessitating intervention from the higher-level actors the Kenya RAPID intervention targeted. However, somewhat lost in this discussion are the wide range of local challenges that are not within the scope of this ICT intervention—issues such as overcrowding and congestion, as well as “last-mile” infrastructure getting water from boreholes to people and livestock. Integrated service delivery interventions examining and addressing water access challenges from source to end user may be needed going forward. 6.4 COVID-19 IMPACTS ON WATER USE AND MANAGEMENT In light of the ongoing COVID-19 pandemic, the evaluation team included questions in both KIIs and FGDs to assess any impacts of the evolving health situation on issues related to water use and management. Overall, respondents indicated that the pandemic had not had major health impacts in selected counties. Indeed, external data indicate that as of October 1, 2020, shortly after data collection concluded, there had been one confirmed COVID-19 case in West Pokot and none in the other three counties.41 Accordingly, some FGD participants said that borehole and water use had not changed substantially since the pandemic started. One user in Garissa explained, “The goodness of this time is that the corona time coincided with a rainy season whereby people had plenty of water and the dams were full -so it has not changed anything with the coming of the new epidemic.” (FGD, Garissa). However, many other FGD and KII respondents noted impacts of the COVID-19 pandemic on water use and water management, from increased need for water for handwashing to impacts on budgets and finances. FGD respondents in Turkana and Tana River noted that people had changed their practices to 41 See https://www.africanews.com/2020/10/02/coronavirus-kenya-distribution-of-cases-by-counties-01-october-2020// FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 48 require visitors to wash hands before entering their homes. Some respondents also noted that the need to purchase more water was causing economic hardships, or that these costs may be a barrier to increased hygiene practices for some individuals. A Garissa water user reported: “The pandemic had also come with economic impact where you will have to incur lot of expenses that comes with the government directives. Water is used 4 times before corona virus.” (FGD, Garissa). For another FGD participant in Tana River, other economic impacts of the COVID-19 pandemic reportedly left less money available to purchase water. This respondent noted, “I used to buy four or five jerricans of water but because of corona I only buy two or three jerricans based on cash I have.” (FGD, Tana River). At the level of local governments, inability of water users to pay fees, or an expectation that water should be provided for free during the pandemic, has also resulted in decreased budgets for water management. A Tana River borehole operator explained, “Most people owe us money, we usually tell them to pay but they continue to tell us that the president had said that water is free but when corona ends we are going to be in a quarrel since they will need to pay for the water.” (KII, Tana River). Some respondents also mentioned that movement restrictions and fears around spreading the virus had effects on water use and water collection patterns. A borehole operator in West Pokot noted that the timing of the pandemic’s onset, during the rainy season, meant that people could access water from other sources and avoided gathering at boreholes early on. A county water official in Tana River noted that movement restrictions affected livestock owners in particular: “Yes [the situation] has changed, there is restricted movement of livestock because most of the people they don’t want other people who are not residing within their community to come to their area because of corona situation more especially the people who are coming from town and going to the inter-land or remote areas.” (KII, Tana River). Other water managers noted that movement restrictions and work-from-home policies affected water management practices, including access to spare parts and availability of personnel to repair boreholes. Several water managers discussed impacts of the COVID-19 pandemic and response on budget allocations and funding for water management. Officials in Garissa noted that funding had been diverted from the Ministry of Water to the Ministry of Health, presenting challenges for local efforts to supply water and handwashing materials. The timing of the pandemic near the end of the country’s fiscal year also constrained available funding, and NGOs’ and other partners’ rollback of activities left local governments with few sources of money to address increased water needs. Water managers in Turkana and West Pokot both mentioned water trucking to provide water to distant communities during the pandemic. However, limited funding reportedly caused water trucking to stop in Turkana. Like many other areas around the world, Kenya experienced a surge in COVID-19 cases in October and November. The pandemic is likely to continue to create challenges for water supply and water management, and these issues should be considered in national response strategies. “If a visitor comes to my house, she must wash her hands before she enters my house, that is water been used so you will get that the water usage has gone up. I buy the twenty liters jerrican and I have put it on the doorstep so anyone coming in must wash his hands. You can imagine there is an old woman or an orphan who buy a jerrican of water at ten shillings then that water is going to be used to wash hands, no way. Let God help us, water to us is so important but the water usage has increased because of corona, I can also say that the water is not enough and we recently started getting water and the water usage is a lot.” - GD, Tana River FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 49 7.0 CONCLUSIONS 7.1 EQ1 CONCLUSIONS The ICT intervention did not lead to increased on-time for strategic boreholes relative to a comparison group of boreholes during the driest months between 2018 and 2020. The theory of change for the Kenya RAPID ICT intervention assumed, informed by historic data, that drought impacts would be severe and somewhat predictable. This assumption also motivated the original design, which anticipated larger effects as a result of the increased borehole water demand during drought periods. If we limit our data just to the relatively dry year of 2019, we do find that during the dry months of this year, Kenya RAPID increased borehole pump on-time by 3 percent on average (95% CI ± 0, 0.06). In sum, our qualitative results indicate that Kenya RAPID did not increased borehole pump on-time to a large degree. Qualitative data from water managers and water users add context to these findings, illustrating how the ICT intervention’s attempt to share information is just one piece of a much larger water system that is facing multiple constraints unrelated to information access. Breakages elsewhere in the water system (e.g., delivery pumps, pipes) also present access challenges that fall outside of sensor measurement. 7.2 EQ2 CONCLUSIONS Qualitative interviews suggest that while the sensor intervention was well-liked, where implemented, it did little to address the ongoing challenges water managers face. Water managers across both treatment and comparison counties continue to have questions about roles and responsibilities of different actors in the water management landscape, and perhaps most importantly, no county reported having clearly delineated adequate budgets for borehole repairs. Implementation of key aspects of the intervention remain incomplete at the conclusion of the project. Officials in Garissa reported that they did not have full access to the data dashboard. While officials in Kenya RAPID counties saw value in the data sensor-based systems provided, it appears that this information was not sufficient to spur large changes in functionality over the intervention period. Barriers to acting on this information include lack of clarity around who is in charge of maintenance and lack of budgets and other resources to support repairs. However, it is important to note that systems change takes time, and there are signs—such as the formation of GARWASCO in Garissa—that the full impact of the intervention may only be seen as these changes continue to unfold. 7.3 EQ3 CONCLUSIONS Discussions with diverse water users in two treatment and two comparison counties indicate that water access, use, and management systems are highly variable in the Kenyan context. Users face a wide range of challenges to meeting their water needs, and our qualitative analysis did not reveal large differences between treatment and comparison groups in perceived functionality or before and after the intervention. This is unsurprising for two reasons. First, our quantitative analysis shows that the Kenya RAPID intervention did not result in large changes in functionality over the evaluation period. Second, many of the challenges highlighted by local users—congestion, “last mile” distribution—are largely beyond the scope of Kenya RAPID’s sensor-based intervention. Understanding and focusing on users’ perspectives may help to guide future policy directions. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 50 7.4 OVERALL CONCLUSIONS Improving water service delivery is a challenge that includes technical, social, economic, and political components. The ICT intervention component of Kenya RAPID introduced an innovative technological solution to one component of the problem: lack of timely information about strategic borehole breakages. Taken together, the results of this impact evaluation show that information provision alone, without effective solutions to a broader range of social, economic, and political management challenges, had at best a small impact on strategic borehole functionality. After controlling for borehole characteristics, county fixed effects, and rainfall, we find a very small average effect of the sensor intervention. At most, our model estimates suggest that the ICT intervention resulted in a little more than an hour of additional borehole pump-on time per day in Kenya RAPID counties compared to comparison counties. Meanwhile, evaluation questions 2 and 3 illuminate the broader context surrounding this intervention, a context where users often substitute away from water boreholes when it rains, where some strategic boreholes do not operate during certain periods of heavier rainfall, where responsibility for water management is distributed across multiple actors in inconsistent and unclear ways, and where funds to repair broken boreholes are illusive. As part of an integrated strategy that addresses each of these challenges together, ICT interventions like the one studied here may be an effective tool for improving water service delivery and increasing resilience in the face of drought. Increased attention to the social, economic, and political context in which technical solutions operate is imperative to realize the full potential of these tools and uncover more effective water management solutions. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 51 8.0 RECOMMENDATIONS Based on our experience and findings, the evaluation team offers the following recommendations for future USAID programmatic and evaluation decisions: 1. Continue to focus on water system governance by clarifying roles and responsibilities for water management and establishing dedicated and sustainable funding sources for water system maintenance and repairs. Evaluate and rethink the role of NGOs in this ecosystem. While donors provide crucial sources of funds in low-income countries, their lack of accountability and clearly delineated role contributes to disjointed and unsustainable systems. How can we ensure that NGO priorities match local and national priorities? How can these actors be engaged to support not just initial infrastructure development, but sustainable maintenance and, crucially, good governance systems? 2. Ensure community concerns are addressed in planning for delivery of water services. Our results show several issues and concerns expressed by users that were not directly addressed by the sensor-based intervention. While water management necessarily involves multiple actors at different scales, the perspectives and expertise of local water users should be central to any effort to improve water service delivery. Walking the line between giving communities agency and voice, while also providing the necessary resources and support from higher levels of government to support user needs, may be challenging but necessary. 3. Consider cost-effectiveness for these interventions in the future. This IE was not designed to consider cost implications, but these should be part of any interpretation of the findings. The small effect sizes estimated here may still be worthwhile if they are cost effective; evaluating this question requires additional data on program costs (in comparison to other approaches to water service delivery improvement). 4. For evaluation efforts, ensure that implementation monitoring is included as a key, funded component, following established guidelines such as the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework.42 A lack of good implementation data was a challenge in this evaluation study, and a lack of budget for this task limited the team’s ability to track progress over time. These data are key to understanding how and why impacts (or lack of impacts) are observed. 42 Glasgow, R.E., Vogt, T.M., Boles, S.M. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health. 1999; 89(9): 1322–1327. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 52 ANNEXES FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 53 ANNEX A: EVALUATION STATEMENT OF WORK Impact Evaluation of the Kenya Resilient Arid Lands Partnership for Integrated Development Project This Statement of Work is for an impact evaluation commissioned by the Office of Water in the United States Agency for International Development’s Bureau for Economic Growth, Education, and Environment (USAID/E3/Water) that will examine the Kenya Resilient Arid Lands Partnership for Integrated Development (Kenya RAPID) activity. 1. Activity Information Kenya RAPID is a five-year activity that began in October 2015 and is funded by USAID, the Swiss Development Corporation (SDC), and the private sector. Kenya RAPID aims to contribute to sustainable and resilient livelihoods for communities in Kenya’s arid and semi-arid lands (ASALs) by improving water availability and water services delivery to people and livestock and improve rangelands in the ASALs. Kenya RAPID mobilizes financial and technical resources from development partners, the national government, county governments, and the private sector to address the complex problems created by inadequate water access and poor governance of natural resources in the ASALs. Kenya RAPID targets five northern ASAL counties—Marsabit, Garissa, Isiolo, Wajir, and Turkana. Each has high poverty rates, chronic water shortages and food insecurity, and low access to basic services. Kenya RAPID uses a public-private partnership model to combine the assets and experience of development actors, private and public institutions—leveraging their capital and investments, innovation, and access to markets—to address the complex problems created by inadequate water access and poor governance of natural resources in the ASALs. Kenya RAPID will directly contribute to USAID/Kenya’s Country Development Cooperation Strategy (CDCS) 2015–2018, whose goal is Kenya’s governance and economy sustainably transformed, and the SDC’s Cooperation Strategy for the Horn of Africa goal to contribute to reduction of poverty, improve human security and instability, and address migration challenges. 2. Development Hypothesis USAID/Kenya envisions that building the capacity of relevant private and public stakeholders for improved WASH service provision and improved rangeland management practices will lead to better health and more resilient livelihoods in targeted areas. Kenya RAPID activity components work in concert to promote water access and delivery and enhanced rangeland environments. Access to water for both domestic and livestock use is a critical component to the livelihoods of ASAL communities. Frequently, ASAL communities have limited availability of water resources, which can adversely affect WASH practices; instead of engaging in hygienic practices like handwashing, individuals may choose to use the water for other purposes. Poor water access can also limit livestock growth and inhibit economic growth for individuals. This adversely affects the health and economic wellbeing of communities and individuals. Kenya RAPID will endeavor to add to this growing body of knowledge during the life of the activity by testing appropriate hypotheses that will be specified at a later date. Figure A-1 illustrates the causal linkages that USAID/E3/Water and USAID/Kenya envision for translating results under the activities into Kenya RAPID’s intended intermediate and final outcomes that this evaluation will be expected to examine. In this theory of change diagram, the improvement of governance frameworks and WASH coverage leads to improvements in water and sanitation access for individuals, water access for livestock, and rangeland-management practices. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 54 FIGURE A-1: KENYA RAPID THEORY OF CHANGE 3. Existing Performance Information Sources USAID/E3/Water, in coordination with USAID/Kenya, provided the evaluation team with the following documents related to existing performance information: ● Kenya RAPID activity documents: ● Kenya RAPID Year 1 Work Plan ● Kenya RAPID fully executed Task Order ● CARE Implementation Activities Progress Presentation The above list, which is non-exhaustive, highlights relevant data sources shared with the evaluation team. The evaluation team did not have access to the following documents, but they will be shared as the evaluation progresses: ● All future quarterly project management and progress reports provided by each of the four implementing partners (CARE, Catholic Relief Services, Food for the Hungry, and World Vision) ● Documents pertaining to selection and implementation of WASH, water coverage, and rangeland management projects ● Annual USAID/Kenya WASH Survey materials In addition to information provided by USAID and each of the implementing partners, the evaluation team will need to access other types of secondary data, including administrative information on the municipalities from a variety of sources. This will likely involve accessing published government sources or obtaining the information from Kenya RAPID staff who are knowledgeable about existing data for FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 55 specific municipalities. The evaluation should also collect and analyze information related to WASH, water coverage, and rangeland management in Kenya, other activities to improve WASH services, issues that may affect social cohesion and gender inclusion in Kenya, and other factors exogenous to Kenya RAPID that could influence activity impacts or survey responses. 4. Evaluation Purpose, Audience, and Intended Use Purpose This impact evaluation will allow the Agency to learn more about how WASH and rangeland management interventions can lead to improved health and economic outcomes. The results of this evaluation will be made widely available to encourage replication and/or scaling up of pilot activities within and beyond Kenya, as applicable. As such, this evaluation will apply USAID’s Evaluation Policy guidance with respect to using the most rigorous methods possible to demonstrate accountability for achieving results. The evaluation is also designed to capture practical lessons from USAID/Kenya’s experience regarding increasing sustainable WASH programs and investment in water and rangeland resource management systems. Audience The evaluation is aimed at several audiences. First, the findings are expected to be of value from an accountability and learning standpoint to USAID/E3, particularly in the Office of Water, and USAID/Kenya. Second, findings and lessons learned from this evaluation will also be of interest to MWA, its partners, and other practitioners in the WASH and rangeland management sectors, including the Government of Kenya, which is seeking ways to improve water resource management, WASH coverage and quality of services. Finally, the evaluation will be of interest to donors, implementers, and scholars more generally by making an important contribution to the evidence base on WASH service delivery interventions. Intended Use This evaluation will be used to inform the design of future USAID programming that aims to improve the sustainability of WASH services to increase resilience and sustainable livelihoods for communities. Depending on the intervention/hypotheses USAID elects to examine through an impact evaluation, it may also contribute to a growing body of evidence about WASH effectiveness, to which other USAID evaluations are also contributing as are studies conducted by other institutions. 5. Evaluation Questions The evaluation questions for Kenya RAPID are still in development. Ultimately, they will reflect USAID’s learning priorities for WASH and rangeland management investments and Agency programming for WASH and rangeland management. The evaluation is expected to focus on how ICT solutions can affect water management in drought prone areas. 6. Gender Considerations In line with USAID’s Gender Policy (ADS 203.3.1.5), the evaluation will consider gender-specific and differential effects of Kenya RAPID activities. The evaluation team will disaggregate access and participation data by gender at multiple points along the theory of change diagram to analyze the potential influence it has on pilot activities and outcomes. Data collected through surveys will be gender￾disaggregated to identify gender differences with respect to benefits and outcomes. The evaluation team will conduct further inquiry on gender themes as they emerge during data analysis. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 56 7. Evaluation Methods Impact Evaluation Design Impact evaluations identify activity impact by comparing outcomes between activity beneficiaries to those of a control or comparison group of non-beneficiaries. The control or comparison group is intended to represent the counterfactual, or what would have happened in the absence of the Kenya RAPID intervention. As per the USAID Evaluation Policy, impact evaluations using experimental designs—whereby units are randomly assigned to treatment and control groups—provide the most rigorous evidence of activity impact, and this will be the preferred approach for the Kenya RAPID impact evaluation. Where randomized assignment is not feasible, quasi-experimental impact evaluation designs can be employed as an alternative. The evaluation team responding to this SOW will work with USAID/E3/Water, USAID/Kenya, and the implementing partner to develop a design that suits the objectives, timing, and constraints of Kenya RAPID. The evaluation team will produce an evaluation design proposal to be approved by USAID/E3/Water prior to any site selection or randomization. It is expected that the evaluation questions will be answered using an experimental or, if necessary, quasi-experimental design, and that a mixed-method approach may be suitable to answer the evaluation questions. Data Collection Methods USAID anticipates that data collection for this evaluation will involve the use of household-level surveys that cover all communities targeted for Kenya RAPID. This is likely to include a baseline survey that would be conducted before major interventions commence. The survey would collect information on basic the outcomes of interest that the evaluation will measure. The evaluation team responding to this SOW shall provide further details on data collection methods and the specific survey methodology in the evaluation design proposal, including proposing specific data collection methods on a question-by￾question basis. 8. Data Analysis Methods In its evaluation design proposal, the evaluation team responding to this SOW should propose specific data analysis methods on a question-by-question basis, including the appropriate mix of methods necessary to estimate the impact Kenya RAPID has on the primary outcomes of interest. Potential data analysis methods include difference-in-difference and multivariate regressions. The evaluation design proposal should also explain what statistical tests will be conducted on data collected to address all evaluation questions, how qualitative data will be analyzed, and whether that analysis will allow the evaluation team to transform some data obtained from qualitative into quantitative form. The evaluation design proposal should also indicate and justify the evaluation team’s proposed sequencing of quantitative and qualitative data collection. For example, if key informant qualitative interviews are conducted during the endline data collection process, these lines of data may be collected and analyzed in parallel and only synthesized once data from all other sources are available. 9. Strengths and Limitations The strengths and limitations of the Kenya RAPID impact evaluation will depend on the final design proposed by the evaluation team in consultation with USAID and the implementing partner. The final design should reflect a rigorous approach to answering the evaluation questions and contribute to the global knowledge on water delivery and rangeland management practices. One key contribution of this evaluation is that it is expected to specifically test the impact of private sector engagement on improving access and quality of WASH services. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 57 Sample size, activity reach, and implementation fidelity could all create internal validity limitations for this evaluation. Ensuring that the sample size achieves sufficient statistical power will be critical for identifying impact and answering the evaluation questions. In addition, ensuring that randomization is done properly and random assignment, if applied, is systematic will improve the internal validity of the evaluation, but must be done in a transparent manner. Indirect contamination across treatment arms and comparison groups is always a possibility, which is why it is important for the evaluation team and the implementation team to coordinate from the outset. 10. Evaluation Deliverables The evaluation team expects to be responsible for the deliverables listed in Table 5. A final list of proposed deliverables and due dates will be included in the evaluation design proposal for USAID’s approval. TABLE A-1. EVALUATION DELIVERABLES Deliverable Estimated Due Date 1. Concept Paper, describing design and methodological options to answer the evaluation questions TBD in consultation with USAID/E3/Water 2. Draft Evaluation Design Proposal TBD in consultation with USAID/E3/Water 3. Final Evaluation Design Proposal, including data collection and analysis methods, evaluation instruments, team composition, proposed timeline, and estimated budget TBD in consultation with USAID/E3/Water 4. Baseline Report o/a 60 days following completion of baseline data collection 5. Draft Evaluation Report o/a 60 days following completion of endline data collection 6. Final Evaluation Report o/a 21 days following receipt of USAID comments on Draft Evaluation Report All documents and reports will be provided electronically to USAID no later than the dates indicated in the approved evaluation design proposal. The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template. 11. Team Composition The evaluation design proposal should describe the specific composition and qualifications of the team members who will be carrying out this evaluation, including CVs for core team members. General qualifications and roles anticipated for core evaluation team are listed below. Local survey research firm(s) with experience in the conduct of household surveys at the village level and/or qualitative data collection may also support the evaluation team, as necessary. Principal Investigator. The Principal Investigator for this impact evaluation should hold a Ph.D. in a relevant economic development field. S/he will have previous experience with WASH programs and will have previously served as a team leader for one or more impact evaluation(s) that include a counterfactual. Familiarity with a range of impact evaluation designs and with USAID evaluation guidance will be sought for this position. Experience in publishing evaluation research in peer-reviewed journals is desirable, as is experience working in East Africa. A demonstrated ability to gather and integrate both quantitative and qualitative findings to answer evaluation questions is expected. Demonstrated experience managing multinational teams and producing highly readable reports for USAID and its FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 58 developing country partner audiences on a timely basis is expected. This individual will be primarily responsible for the quality of the evaluation design and its execution, particularly with respect to the evidence obtained on questions involving causality and the attribution of outcomes to USAID’s intervention. This is not a full-time position. Evaluation Specialist. The Evaluation Specialist should have a graduate degree in a relevant social science field and may be a Kenyan national. The individual will have sufficient previous experience with evaluations and other types of studies involving sample surveys to be actively engaged in efforts to oversee and ensure the quality of the evaluation's multiple rounds of household surveys and ensure that data codebooks are clearly written and all study data prepared by local firms are properly transferred to USAID. Gender analysis experience is also desirable. This is not anticipated to be a full-time position. 12. USAID Participation The desirability of USAID participation in evaluation activities such as data collection will be considered in consultation with USAID and the evaluation team, and any specific roles and responsibilities of USAID staff will be described in the evaluation design proposal. 13. Scheduling and Logistics The following table provides the originally anticipated timeframe for evaluation activities and deliverables. The evaluation team will be responsible for procuring all logistical needs such as workspace, transportation, printing, translation, and any other forms of communication. USAID will offer some assistance in providing introductions to partners and key stakeholders as needed and will ensure the provision of data and supporting documents as possible. 14. Reporting Requirements The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note-preparing-evaluation￾reports). The final version of the evaluation report will be submitted to USAID and should not exceed 30 pages, excluding references and annexes. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 59 All members of the evaluation team will be provided with USAID’s mandatory statement of the evaluation standards they are expected to meet, shown in the following text box below, along with USAID’s conflict of interest statement that they sign and return before field work starts. USAID EVALUATION POLICY, APPENDIX 1 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why. Evaluation reports shall address all evaluation questions included in the scope of work. The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. Evaluation findings will assess outcomes and impact on males and females. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. Sources of information need to be properly identified and listed in an annex. Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. 15. Budget The evaluation team responding to this SOW will propose a notional budget in its concept paper for this evaluation, including cost implications of the methodological options proposed. A full detailed budget will then be prepared and included in the evaluation design proposal for USAID’s approval. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 60 ANNEX B: COMPARISON COUNTY FOLLOW-UP BOREHOLE ASSET SURVEY INSTRUMENT SweetSense designed this instrument to collect information on boreholes. The evaluation team used a slightly revised version as part of baseline data collection, which is provided below. Interviewer details: Name of the interviewer ID no. of the interviewer Name of the supervisor ID no. of the supervisor Introduction Good morning/afternoon! I am ____ from Ipsos, a survey and market research company. We are currently conducting a study to better understand water use in this community. Your opinion and knowledge would be incredibly helpful for supporting national efforts to address water management and drought resiliency. Many of the questions I will ask are related to the local water borehole and based on the current context here. There are no right or wrong answers, and please be assured that the information collected from you will be treated completely confidentially. ./ Subax wanaagsan / galab! Waxaan ahay ____ ka socda Ipsos, shirkad cilmi baaris ah iyo suuqayada. Hadda waxaan sameyneynaa daraasad si aan si fiican u fahanno isticmaalka biyaha ee bulshada ama deegankan. Fikraddaada iyo aqoontaadu waxay noqonaysaa mid aad u caawin karta si ay u taageerata dadaalada qaranka ee si wax loga qabta maareynta biyaha iyo wax ka qabadka adkeeysiga abaarta. Su'aalo badan oo aan weydiin doono waxay ku saab san yihin ceelesha biyaha ee dagankan ah oo ku salaysan xaaladda hadda jirta. Ma jiraan jawaabo sax ah ama khaldan, fadlan agaanteed laho in macluumaadka laga soo ururiyey adiga laguula dhaqmi doono si qarsoodi ah. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 61 Water supply facilities General Information Water system (distribution scheme linked to this update) A1. Name/description of the water system/Magaca/sharraxaadda nidaamka biyaha A2. Unique water system ID (map A1 &A2)/Nidaamka Biyaha ee Gaarka ah (Khariidad A1 A3. GPS of the borehole location (This is to be done at the borehole exact location.) A4. Take picture of the water system: (INTERVIEWER NOTE: THE PICTURE SHOULD CAPTURE THE WHOLE BOREHOLE SYSTEM PLUS THE SURROUNDINGS [I.E., TANKS, KIOSK] IN ONE CAPTION.) A5. County/Ismaamulka  Baringo  Kitui  Laikipia  Mandera  Meru  Samburu  Tana River  West Pokot A6. Sub-County/ismaamulka hoose (insert list) A7. Village (insert list)/Xaafada (geli liiska) FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 62 A8. Rural/Urban (map)/Baadiyaha/Magaalada (khariidada)  Rural  Urban A9. Local Officer Name (INTERVIEWER NOTE: RECORD THE NAME OF THE PERSON RESPONSIBLE FOR BOREHOLE MAINTENANCE/PERSON IN CHARGE.) A10. Local Officer Position  Operator  Water committee chairperson  Water Committee Officials/Saraakiisha Guddiga Biyaha  Other (specify) A11. Local officer telephone number Users/ Hali ya matumizi B1. Total number of households currently served from the scheme?/tirade guud ee iminka isticmasha nidaamkas B6. Is the water scheme used for livestock?/nidaamka biyaha loo isticmaala xoolaha  Yes/haa  No/Maya If Yes/hadii haa tahay B11. Is the water scheme used for water trucking?/Mashruuca biyaha ma loo isticmaalaa biyo dhaamin?  Yes/haa  No/maya (SKIP TO C3) FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 63 B13. How many days per week does the water trucking occur?/Meega jeer toddobaadikii ayaa biyo dhaaminaya? Borehole Information C3. Physical state of the well/borehole/xalada guud ee ceelka (INTERVIEWER NOTE: THIS INFORMATION REFERS TO THE PERCEPTION OF THE COMMUNITY ON THE BOREHOLE.)  Function well/Si ficaan u saqeeya  Poor/hoseesa  Doesn't function/ma shaqeeyo  Unsure / (DO NOT READ OUT)/ma hubo(HAA AKHRIN) C4. Type of Power/Nooca tamarta (MULTIPLE RESPONSE)  Generator/Matoor  Solar/oorahada  Utility Power (Grid power)/isticmaalka tamarta(tamarta roobka) C7. Is there a water meter at the water source?/Ma yeela cabirka biyaha goobta biyaha?(WAREESTAHA HA EEGA) (INTERVIEWER TO OBSERVE.)  Yes/haa  No (SKIP TO C11)/Maya(u kac c13) Scheme Functionality/Shaqeeynta nidaamka C11. Is the water scheme currently functional?/nidaamka biyaha iminka ma shaqeeynaya? (INTERVIEWER NOTE: THIS REFERS TO WHETHER THE BOREHOLE IS PRODUCING WATER. IF SO, IS THE AMOUNT OF WATER PRODUCED AS PER DESIGN OR IS IT REDUCED YIELD?)  Functional (producing as designed) (SKIP TO C15)/Shaqeeynta(wax soo sarka sida logu qasdi)  Partially functional (reduced yield)/xogaaha shaqeena(wax soo sarka oo is dhimi) FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 64  Non-functional/aan shaqeenin  Abandoned SKIP TO SECTION E)/laga tagay (u kac E) If the water scheme is partially functional or non- functional/:Hadii nidaamka biyaha ee xoogaha ama aan shaqeenini: C12. Please take a picture illustrating the non-functionality or partial functionality /(INTERVIEWER NOTE: TAKE MORE THAN ONE PICTURE IF MORE THAN ONE PART OF THE SYSTEM IS NOT FUNCTIONING.)/fadlan ka quad sawir tusinaaya shageeyn laanta ama xoogaha ka shageeyni If the water scheme is partially functional or non-functional/Hadii nidaamka biyaha ayan shaqeenin ama xogaha shaqeeyna: C13. Main cause of non-functionality or partial functionality/Sababta ugu weyn keentay shaqeeynta laanta ama uu xogaha ugu shaqeeyna: :JAWAABA BADAN (MULTIPLE RESPONSE)  Insufficient water at source/Biya yari ka jirta isha ceelka  Distribution pipeline or tap failure/dummoyinka qeeybiya oo fashalmi  No gas for generator/motoorki oo ka dhammaday gaaski  Generator failure/matoorki oo fashalmi  Grid power failure/tamarta griidka oo fashalmay  Solar power failure/tamarta qoraxda oo fashalmay  Submersible pump failure/doloolkadummoyinka oo fashilmi  Switchboard (electrical) failure/failure/qalabka korontada oo ka fashalmi  Other (please specify)/wax kale(fadlan fahfaahi) If the water scheme is partially functional or non-functional: C14. Number of months since non-functional/Idadi ya miezi tangu kituo hiki kufanya kazi (SKIP TO QUESTION D1 IF C11 IS CODED NON-FUNCTIONAL) FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 65 If the water scheme is currently functional or partially functional/hadii nidaamka biyuhu uu shaqeenin ama xogaha ka shaqeyniya: C15. Was the water scheme out of service one or more days in the last month?/ Nidaamka biyaha musan shaqeeynin hal malmood iyo wixi ka badan bisha aan soo dhafni ? WARESTAHA:HUBI INUU JAWAAB BIXIYAHA UU FAHMA WAQTIGA INTUSAN KA JAWAABIN (INTERVIEWER: ENSURE THAT RESPONDENT UNDERSTANDS THE TIME FRAME BEFORE RESPONDING [I.E., THE PAST 4 WEEKS FROM THE DAY OF INTERVIEW].)/ Nidaamka biyaha musan shaqeeynin hal malmood iyo wixi ka badan bisha aan soo dhafni ? WARESTAHA:HUBI INUU JAWAAB BIXIYAHA UU FAHMA WAQTIGA INTUSAN KA JAWAABIN  Yes/Haa  No (SKIP TO C17)/Maya(U Kac C17 C16. If Yes, number of days the scheme was out of service in the last month/Hadii haa tahay, inta maalmood uu nidaamka aha shaqaa laan bisha laso dhaafay C17. Was the water scheme out of service one or more days in the past 12 months? (INTERVIEWER: ENSURE THAT RESPONDENT UNDERSTANDS THE TIME FRAME BEFORE RESPONDING.)/Nidaamka biyaha musan shaqeeynin hal malmood iyo wixi ka badan sanadki aan soo dhaafni ? WARESTAHA:HUBI INUU JAWAAB BIXIYAHA UU FAHMA WAQTIGA INTUSAN KA JAWAABIN  Yes/Haa  No/La (SKIP TO C20)/Maya (ukac C18. How many times was the water scheme broken in the past 12 months?/imisaa mar ayuu nidaamka biyaha uu jabay sanadki lasoo dhaafay?  Has never broken in the past 12 months/wali ma jabin sanadkii lasoo dhaafay (SKIP TO C20) C18a. What was the nature of the failure? (SELECT MULTIPLE)/maxaa uu ahay dhibaatada la xiriira fashilitaanka?(xula jawaaba badan)  Pump failure/fasilitaanka dhuumaha  Generator/power failure/fashilka matoorka/fashilka tamarta  Pipe failure/fashilka dhuumaha FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 66  Tank failure/fashilka taangiga  Other (please specify)/Wax kale C19. Describe the functionality problem/s over the past 12 months./fahfaahi dhibaatooyinka la xiriira shaqeynta sandki laso dhaafay? C20. Does the scheme have any emerging problems that might lead to non-functionality in the near future?/nidaamka byaha maleeyhay dhibaatooyin hadaa soo if baxayo oo laga yaabo iney shaqeynta dhib ugeystaan mustaqbalka dhow?  Yes/haa  No (SKIP TO D1)/Maya(SKIP TO D1) C21. If yes, please describe the problems that might lead to non-functionality in the near future. Hadii haa tahay/?/fadlan fahfaahi dhibaatooyinka laga yaabo iney u hor seedaan shaqeyn laan mustaqbalka dhow Scheme Usage Patterns/Mpango wa Matumizi ya Mfumo D1. How many days per week does the pump run on average?/imisaa maalin isbuucii D2. How many hours per day does the pump run on average?/imisaa saacadood maalinti ayu bamka biyaha soo jiido socdaa isku celcelis ahaan? D3. Is this a seasonal water scheme that commonly fails in the dry season?/nidaamkani mayahay mid sanadla ah oo inta badan go’a xiliyada roobabka jirin?  Yes/haa  No/Maya  Don’t know (DO NOT READ OUT)/GARAN MAYO(HAA KHRIN) D4. Does frequency of use of the pump depend if it is wet/rainy season?/xawaaraha loo isticmaalo bamkan biyaha miyuu ku xiranyahay xiliyada roobabka da’aan misna abaaraha ah?  Yes/Haaa  No?/Maya FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 67  Don’t know (DO NOT READ OUT)/GARAN MAYO(HAA KHRIN) D5. During wet/rainy seasons, how many days a week does the pump run on average?/xiliyada roobabka da’ayaan meeqa maalin ayuu bamka shaqeeyaa isbuuci? D6. During wet/rainy seasons, how many hours per day does the pump run on average? /xiliyada roobabka da’ayaan meeqa sac ayuu bamka shaqeeyaa isbuuci? Pump G1. Pump Controller Manufacturer/ Soo saaraha xakameeyah bambada/dhumooyinka (INTERVIEWER NOTE: SEE TECHNICAL DETAILS FROM THE PLACARDS. IF NOT AVAILABLE, CHECK WITH COUNTY/WATER AUTHORITIES FROM THE COUNTY ENGINEERS, WATER RESOURCE AUTHORITY.)/WARESTAHA XASUUSNOW:EEG CALAAMADAHA MACLUMAADKA FARSAMADA /HADII UU YEELIN WAA INAD KA EEGTID MACLUMAADKA MAAMULKA BIYAHA KAUNTIGA.INJINEERKA KAUNTIGA,GUDIGA BIYAHA IYO QEYRAADKA)  Grundfos  Lorentz  Davis & Shirtliff  ABB  Dayliff  Tormak  Other (please specify) )/Wax kale(fadlan fahfaahi)  Don’t Know/)/GARAN MAYO(HAA KHRIN) Sensor Information/Macluumaadka Xasaasiga ah I1. Is there a sensor installed at the site?/ma jira qalabka ogaanshaha lagu xirahay goobta(WARESTAHA FADLAN XASUUSNOW TANI TIXRAACA UMA AHAOGSHANSHA DHUUMAHA) (INT: VERIFY IF THE SENSOR IS ACTUALLY CLAMPED TO THE LIVE WIRE )  Yes/Haa FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 68  No/La >>>>> skip to I7/MAYA(U KAC 17) I5. Take a picture of the sensor (INT: TAKE A PICTURE OF INSTALLED SENSOR BEFORE UNCLAMPING IT, BROKEN OR VANDALIZED SENSOR AND OPEN PUMP CONTROLLER ON SITE IF THE SENSOR IS NOT ON SITE )/Sawirka xiritaanka qalabka ogaanshaha I7. If No, what are the reasons to why the sensor is not installed/Hadday Maya tahay, waa maxay sababaha keenay in qalabka uusan looga rakibin.  Unclamped and set aside :(please ask reason for uninstalling)  Broken  Stolen  Other specify 17.1 For how long has the sensor not been installed? Record in months. If less than 1 month record zero)/Muddo intee la'eg ayaa qalabka aan la rakibin? Ku diiwan geli bilo. Haddii wax kayar 1 bilood diiwaan geli eber) I8. Was data downloaded from the sensor?/Xogta miyaa laga soo qaaday qalabka? (DOWNLOAD DATA FROM SENSOR WHETHER INSTALLED OR NOT)  Yes (SKIP TO I10)/Haa (u kac 110)  No/Maya La  Sensor not at site ( logic check . only applicable if 17 is any other code other than code 1 or 2 ) >>>>>> 113/qalabka meesha usan yaallin (hubinta caqligal ayaa lagu dabaqi karaa oo keliya haddii 17 uu jiro koodh kale oo aan ka ahayn koodhka 1 ama 2) >>>>>> 113 I9. If No, what are the reasons for not downloading the data./hadii maya tahay waa maxay sababta aad u helin xogta  Sensor is broken/qalabka la jabiyay  Sensor malfunctioning. Cannot download data to the computer/qalabka muu shaqeenayo. Xogta kombuutarka ushan gashani karin  Other (please specify)/wax kale (fadlan fahfaahi) FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 69 I10. Was the same sensor installed back?  Yes/Haa  No/La (SKIP TO I12)/Maya 111. If Yes, take a Picture of sensor installation. (INT: TAKE A PICTURE OF THE SENSOR IMMEDIATE AFTER REINSTALLING)/Hadday Haa tahay, qaado Sawir rakibaadda qalabka.(DIG;KA QAADO SAWIR QALABKA KADIIB MARKA LA SAMEEYA IF YES SKIP TO QUESTION J1 112. If No, what is the reason of not installing the sensor/hadii maya ,waa maxay sababaha ee loo rakibin qalabka I13. was the sensor replaced?/qalabka ma la badalay  Yes/Haa  No (SKIP TO I15)/Maya 114. If Yes, take a Picture of the sensor after replacing. (INT: TAKE A PICTURE OF THE SENSOR IMMEDIATE AFTER INSTALLING)/haddii haa tahay ka qaado sawir qalabka kadib marka la rakibay(DIG:KA QAADO SAWIR KADIB QALABKA LA SAMEEYA) 115. If No, what is the reason of not replacing the sensor./haddii maya ,waa maxay sababta qalabka loo badalin Management Usimamizi J1. Management Body  Utility  WASHCO  No management organization  Others (specify)_______________ FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 70 If management body is utility or WASHCO: J2. Current Management Status  WASHCO or utility is active  WASHCO or utility is not active Tariffs Malipo/Canshuuraha K1. Type of tariff system (most common)nooca canshuurta (Sida badan)  Fixed tariff per visit/canshuur joogto ah mar walboo lasoo booqdo  Fixed tariff per week/canshuur joogto ah isbuuciba  Fixed tariff per month/canshuur joogto ah bishiba  Fixed tariff per half year/canshuur joogto ah sanadki barkiisa  Fixed tariff per year/canshuur joogto ah sandkiba  Tariff per jerrycan (20 litre) )/cashuurta jergan kasta (20 liitar)  Tariff per cubic meter (m3) )/canshuurta miitar sadex lab kasta (m3)  Ad hoc contributions/Malipo maalum inayotozwa kwa dharura (SKIP TO K3)  No payment (SKIP TO K3)/lacaq laan(U KAC k3)  If type of tariff system (most common) is one of fixed tariff per visit, fixed tariff per week, fixed tariff per month, fixed tariff per half year, fixed tariff per year, tariff per jerrycan (20 litre), or tariff per cubic meter (m3): )/hadii nooca canshuurta (sida badan) uu yahay mid joogto ah mar walboo lasoo booqdo, mid joogto ah isbuuc walbo, mid joogto ah bil walbo, mid joogto ah kala barka sand walbo, mid jooto ah sanad walbo, canshuurta jerganka (20 liitar) canshuurta miitarka sadex laabka (m3) K2. What is the tariff amount (in KES)?/waa imisaa qiimaha canshuurta (KES)? K3. Is there a special tariff for livestock?/ma jirtaa canshuur gaar ah oo laga qaado xoolaha (calaa neef)  Yes/yaa  No (SKIP TO K8) )/Maya (U KAC K8) FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 71 Other notes/Observations FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 72 ANNEX C: REFERENCES Abadie, A., & Imbens, G. W. (2006). Large sample properties of matching estimators for average treatment effects. econometrica, 74(1), 235-267. doi:doi:10.1111/j.1468-0262.2006.00655.x Altman, D. G. J. T. S. (1985). Comparability of randomised groups. The Statistician: Journal of the Royal Statistical Society. Series D, 34(1), 125-136. Buuren, S. van, & Groothuis-Oudshoorn, K. (2010). Mice: Multivariate imputation by chained equations. R. Journal of Statistical Software, 1-68. Cox, D. (1972). Regression Models and Life-Tables. Journal of the Royal Statistical Society. Series B (Methodological), 34(2), 187-220. Cronk, R., & Bartram, J. (2017). Factors influencing water system functionality in Nigeria and Tanzania: A regression and Bayesian Network Analysis. Environmental Science & Technology, 51(19), 11336- 11345. doi:10.1021/acs.est.7b03287 Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053-1062. doi:10.1080/01621459.1999.10473858 Feinerer, I., Hornik, K., & Meyer, D. (2008). Text mining infrastructure in R. 2008, Journal of Statistical Software, 25(5), 1-54. doi:10.18637/jss.v025.i05 Fisher, M. B., Shields, K. F., Chan, T. U., Christenson, E., Cronk, R. D., Leker, H., ... & Bartram, J. (2015). Understanding handpump sustainability: Determinants of rural water source functionality in the Greater Afram Plains region of Ghana. Water Resources Research, 51(10), 8431-8449. doi:10.1002/2014WR016770 Funk, C. C., Peterson, P. J., Landsfeld, M. F., Pedreros, D. H., Verdin, J. P., Rowland, J. D., . . . Verdin, A. P. (2014). A quasi-global precipitation time series for drought monitoring. US Geological Survey data series, 832(4), 1-12. Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge university press. Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. (2016). Impact evaluation in practice. The World Bank. Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9): 1322– 1327. Harvey, P. A., & Reed, R. A. (2007). Community-managed water supplies in Africa: sustainable or dispensable? Community Development Journal, 42(3), 365-378. Ho, D., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. 2011, Journal of Statistical Software, 42(8), 28. doi:10.18637/jss.v042.i08 Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge: Cambridge University Press. Kenya, R. O. (2014). Ending drought emergencies: Common programme framework for drought risk management. Retrieved from Nairobi. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 73 King, G., & Nielsen, R. (2018). Why propensity scores should not be used for matching. Political Analysis. Nagel, C., Beach, J., Iribagiza, C., & Thomas, E. A. (2015). Evaluating cellular instrumentation on rural handpumps to improve service delivery—A longitudinal study in rural Rwanda. Environmental Science & Technology, 49(24), 14292-14300. doi:10.1021/acs.est.5b04077 NDMA. (n.d.). Early warning bulletins. Retrieved from http://www.ndma.go.ke/index.php/resource￾center/early-warning-reports Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55. doi:10.1093/biomet/70.1.41 Thomas, E. A., S. Kathuni, D. Wilson, C. Muragijimana, T. Sharpe, D. Kaberia, D. Macharia, A. Kebede, & P. Birhane. (2020). The Drought Resilience Impact Platform (DRIP): Improving Water Security Through Actionable Water Management Insights. Front. Clim. 2: 6. doi: 10.3389/fclim. Thomas, E. A., Needoba, J., Kaberia, D., Butterworth, J., Adams, E.C., Odur, P., Macharia, D., Mitheu, F., Mugo, R., Nagel, C. (2019). Quantifying increased groundwater demand from prolonged drought in the East African Rift Valley. Submitted to Lancet Planetary Health. USAID. (2019). Round 1 evaluation report: Impact evaluation of the Kenya Resilient Arid Lands Partnership for Integrated Development Activity. March 22, 2019. USAID. (2020). Round II Evaluation Report: Impact Evaluation of The Kenya Resilient Arid Lands Partnership for Integrated Development Activity. May 12, 2020. Walters, J. P., & Javernick-Will, A. N. (2015). Long-Term functionality of rural water services in developing countries: A system dynamics approach to understanding the dynamic interaction of factors. Environmental Science & Technology, 49(8), 5035-5043. doi:10.1021/es505975h World Health Organization. (2015). Progress on sanitation and drinking water: 2015 update and MDG assessment. World Health Organization. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 74 ANNEX D: STATISTICAL POWER As outlined in the evaluation design proposal, there is a trade-off between sample size and the reliability of estimates, with larger sample sizes more likely to detect the causal effect of the activity. Table D-1 shows the original sample size calculations from the evaluation design proposal. A key feature to note in the table below is that the minimum detectable effect size (MDES)was much larger than what the evaluation team observed in the actual pump on-time date, which was generally under 0.10 depending on the sample and specification. TABLE D-1: EX ANTE MINIMUM DETECTABLE EFFECT SIZE (MDES) ESTIMATES Sample Size (# of Boreholes) MDES 95% Confidence Interval 150 0.350 0.118, 0.581 175 0.338 0.114, 0.561 190 0.327 0.110, 0.543 The initial estimates were based on a review of the relevant literature and assumptions regarding potential sample sizes prior to the commencement of the evaluation. Table D-2 updates sample sizes based on the matching options explored in previous rounds. The MDES by sample size is presented based on standard parameters, such as significance level (alpha) of 0.05 and power of 0.8. For the purposes of this exercise, the evaluation team has also reused the 25 percent covariate variance figure from the evaluation design, which is close to the actual covariate variance of the different models, ranging from 20-25 percent. The table below shows the full Round I samples. In reality, the unmatched sample is not the initially anticipated 208 as sensor data in Rounds II and III were unavailable for all boreholes. Instead, the final sample at Round III was 163 boreholes (66 treatment, 97 comparison) for the full unmatched sample. TABLE D-2: MDES ESTIMATES BASED ON BOREHOLE SAMPLE SIZES43 Sample Size (# of Boreholes) MDES 95% Confidence Interval 45 0.561 0.188, 0.934 114 0.391 0.132, 0.649 163 0.35 0.12, 0.59 208 0.312 0.105, 0.518 The actual effect sizes observed and reported in the body of this report are substantially lower than the MDES. Using the model that was fit for the analysis in this report and simulating a range of potential effects produces Figure D-1. We find that based on the 14 covariates, as well as number of total boreholes (n=163) across 13 counties, it would take a minimum effect size of 0.36 to achieve statistical power of 0.8. 43 Assuming blocked design with the following parameters: alpha = 0.05, power = 0.8, 14 included covariates that explain 25 percent of the variation in the outcome. The percentage assigned to treatment is based on borehole survey data and matching: 37 percent for the unmatched sample; 50 percent for the matched sample; and 38 percent for the trimmed matched sample. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 75 FIGURE D-1: STATISTICAL POWER BY EFFECT SIZE FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 76 ANNEX E: COVID-19 CONSIDERATIONS In late March 2020, the Kenyan government closed airports, limited public gatherings, and limited movement within the country severely. These restrictions were lifted just as the data collection process was being planned for this impact evaluation. However, to maintain the safety of the field teams and respondents (both qualitative respondents and local borehole staff who were needed to facilitate sensor downloads), the evaluation team and its local partner, Ipsos, implemented a series of safety precautions. Enumerator training was held in a large hall with open space that allowed for proper air circulation and social distancing of two meters between all participants. Field team members were provided with hand sanitizer and face masks, which were worn throughout the training process. The evaluation team also updated the language in its consent scripts to acknowledge that not all potential respondents may feel comfortable participating in an interview and reiterated that participation is voluntary, and respondents can leave at any time. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 77 ANNEX F: INTERVIEW CONSENT SCRIPTS Qualitative Evaluation of Kenya Resilient Arid Lands Partnership for Integrated Development INFORMED CONSENT: GROUP DISCUSSIONS INTRODUCTION Hello, my name is ______. I am working with Ipsos, and with the Colorado School of Public Health and WASHPALS in the United States. We are doing a study about water services, drought, and other issues in this area. We are not talking to all residents in this area, only some of them. You have been selected as someone who lives in this area and uses the local borehole. STUDY PROCEDURE We are interested in learning about how boreholes are managed for water service delivery in this area, especially during times of drought. If you agree to participate in this study, we conduct a group discussion with you that will take about one hour. If you are willing, we will come back in about one year to conduct a group discussion with you and other users in this area again. RECORDING With your permission, I will audiotape, and my colleague will take notes during the discussion. The recording is to accurately capture the information you provide and will be used for transcription purposes only. Excerpts from the recordings/transcripts may be used to illustrate the research findings. This will always be done in a way to protect your identity (that is, your name will not be used). VOLUNTARINESS Taking part of this study is completely voluntary. You have every right to refuse to participate. If you should refuse, you will not suffer any consequences. COVID-19 Our team has taken every precaution to conduct this discussion safely. If at any point you are uncomfortable, it is entirely permissible to decide to leave the group discussion. WITHDRAWAL If you chose to participate in this study, you have the right to withdraw from it at any point in time without any consequences to you. You are free to skip any questions you do not wish to answer or to stop at any time. You may ask the researchers any questions you have at any time. COMPENSATION You will not be paid for your participation in this study. CONFIDENTIALITY We will need to connect your name to the information you give us, but only for a short time—while we are gathering information from many households. After that, we will save the information and report what we learn using numbers, not names. Only we, the researchers, will ever see the information with people’s names. QUESTIONS If you have any questions concerning the study, you can contact Mary Gichihi [PHONE NUMBER]. [Add info here on IRB approvals once completed – which boards reviewed and approved, and who to contact] FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 78 ORAL CONSENT OF PARTICIPANT Do you agree to participate in this study? ◻ Yes ◻ No Qualitative Evaluation of Kenya Resilient Arid Lands Partnership for Integrated Development INFORMED CONSENT: KEY INFORMANT INTERVIEWS INTRODUCTION Hello, my name is ______. I am working with Ipsos, and with the Colorado School of Public Health and WASHPALS in the United Stated. We are doing a study about water services, drought, and other issues in this area. We understand that you are involved in the management of boreholes in this area, and would like to talk to you about your experiences. STUDY PROCEDURE We are interested in learning about how boreholes are managed for water service delivery in this area, especially during times of drought. If you agree to participate in this study, we conduct an interview with you that will take about one hour. If you are willing, we will come back in about one year to interview you again. RECORDING With your permission, I will audiotape, and my colleague will take notes during the interview. The recording is to accurately capture the information you provide and will be used for transcription purposes only. You have the right to review, edit, or erase any information from the interview that you do not want on record. Excerpts from the recordings/transcripts may be used to illustrate the research findings. This will always be done in a way to protect your identity (e.g., your name will not be used). VOLUNTARINESS Taking part of this study is completely voluntary. You have every right to refuse to participate. If you should refuse, you will not suffer any consequences. COVID-19 Our team has taken every precaution to conduct this interview safely. If at any point you are uncomfortable or would prefer to reschedule this discussion to be held online or over the phone, please let us know. WITHDRAWAL If you chose to participate in this study, you have the right to withdraw from it at any point in time without any consequences to you. You are free to skip any questions you do not wish to answer or to stop at any time. You may ask the researchers any questions you have at any time. COMPENSATION You will not be paid for your participation in this study. CONFIDENTIALITY FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 79 We will need to connect your name to the information you give us, but only for a short time—while we are gathering information from many water managers. After that, we will save the information and report what we learn using numbers, not names. Only we, the researchers, will ever see the information with people’s names. QUESTIONS If you have any questions concerning the study, you can contact Mary Gichihi [PHONE NUMBER]. [Add info here on IRB approvals once completed – which boards reviewed and approved, and who to contact] ORAL CONSENT OF PARTICIPANT Do you agree to participate in this study? ◻ Yes ◻ No FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 80 ANNEX G: INTERVIEW GUIDES G-1 KEY INFORMANT INTERVIEW SCRIPT: COUNTY/SUB-COUNTY OFFICIALS Respondent Information Respondent name:/Jina la mhojiwa: Location of interview: (county, town/village)/Eneo la mahojiano: (kaunti, mji/kijiji) Respondent organization:/Shirika la mhojiwa: Respondent job title:/Taji la kazi ya mhojiwa: Date of Interview:/tarehe ya mahojiano: Interview Questions (semi-structured interview – use script as a guide) 1. Please describe your involvement in water management./Tafadhali taja uhusiano wako na usimamizi wa maji. (Have respondent describe their role, and then probe as needed). a. How long have you held this job?/Je umefanya kazi hii kwa muda gani? b. How long have you worked in water management?/ Je umefanya kazi katika usimamizi wa maji kwa muda gani? c. What are the main water sources in the area?/Je kuna njia gani kuu za kupata maji katika eneo ambalo unasimamia? d. Where do people in this area mainly get their water? What about during the rainy season?/Je ni wapi watu katika eneo hili hupata maji yao kwa ukuu? Je na wakati was msimu wa mvua mingi? i. Do you know how many of the boreholes you manage are designated as “ending drought emergency (EDE)” or strategic boreholes?/Je unafahamu visima ngapi ambavyo unasimamia vimeteuliwa kama “ending drought emergency (EDE)” (visima vya dharura wakati wa ukame) ama visima vya strategic boreholes? Visima ambavyo zingatiwa kuwa za muhimu wakati wa kiangazi? (Note: not all water managers will know which boreholes are designated as EDE. • Probe: If they don’t know: which boreholes are critical during the dry season? Throughout the rest of the script, use either “EDE borehole” or “county strategic” borehole to refer to these critical boreholes, depending on how the respondent identifies them. ii. Do you know how the EDE/strategic boreholes are decided or defined? What makes these boreholes different from other boreholes?/Je unafahamu jinsi visima vya EDE/strategic boreholes vinachaguliwa au kutengwa vile? Ni nini hufanya visima hivi tofauti na visima vingine? 2. What is the management structure for the water in this county/sub-county?/Je muundo wa usimamizi wa maeneo haya ya maji ambayo huwa unasaidia kusimamia ni gani? a. How do the different actors involved in managing this borehole interact with one another?/Je washiriki tofauti wanaohusika katika kusimamia kisima hiki huwa wanashirikia ki vipi? (Where relevant, probe for: i. Local operator relationship with county government/Mahusiano ya mwendeshaji wa kisima na serikali ya kaunti ii. WASCO relationship with county government/Mahusiano ya WASCO na serikali ya kaunti FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 81 iii. County government relationship with operations and maintenance organizations like Catholic Diocese in Turkana/Mahusiano ya serikali ya kaunti na mashirika ya kuendesha na kutunza kama vile Kanisa la Katoliki Diocese ya Turkana iv. WASCO relationship with rate payers)/Mahusiano ya WASCO na wanaolipa ada 3. What are the main challenges you face in providing reliable water service to users in this area?/Je nichangamoto gani kuu huwa unapitia katika kutoa huduma za maji inayotegemewa kwa wakaazi katika eneo hili? a. Do you receive any complaints about the borehole from users during the drought season?/Je huwa unapokea malalamishi yoyote kuhusu kisima kutoka watumizi katika msimu ya ukame? • Probe: What kind of complaints?/Je ni malalamishi ya aina gani? • Probe: How do you typically receive these complaints?/Je huwa unapokea malalamishi haya vipi kwa kawaida? b. Are you able to address these complaints? If not, why?/Je huwa unaweza kusuluhisha malalamishi haya? Kama huwa huwezi, ni kwa nini? 4. What are the main challenges that you experience in providing reliable water service during the dry season?/Je ni changamoto gani kuu huwa unapata katika kutoa huduma ya maji ya kutegemewa katika msimu wa ukame? a. Do the challenges differ during the dry season from other times of year? If so, how?/Je changamoto huwa zinatofautiana katika msimu wa ukame ukilinganisha na nyakati zingine za mwaka? Kama ni hivyo, ni vipi? b. How do you or your team manage or mitigate these challenges?/Ni vipi wewe ama jopo lako linasimamia huwa linashughulikia au kusuluhisha changamoto hizi? 5. Has the water situation changed as a result of COVID-19?/Je hali ya maji imebadilika kufuatia COVID-19? a. Has there been any change in funding or support for borehole pumped water? What about water from other sources?/Je kumekuwa na mabadiliko yoyote katika usaidizi wa fedha au wowote kwa maji ya bomba? Je na kwa maji kutoka njia zingine? b. Have there been changes related to water use patterns as a result of COVID-19?/Je kushawahi kuwa na mabadiliko yanayohusiana na mitindo ya matumizi ya maji kufuatia COVID-19? 6. When EDE/county strategic boreholes in your area breakdown, how long does it take…/Je EDE/visima vikiharibika katika eneo lako, je huwa inachukua muda gani… a. For your office to find out the borehole is broken?/Afisi ya maji kujua kisima kimeharibika? b. To get information about the breakages (i.e., specifically what is broken and what repairs are needed)?/Kupata habari kuhusu kuharibika (hiyo ni nini haswa imeharibika na ni nini inafaa kutengenezwa)? c. To repair the broken boreholes?/Kutengeneza kisima ambacho kimeharibika? • Probe: Do the answers to these questions differ between the dry season and wet season?/Je majibu ya maswali haya yanatofautiana na msimu wa ukame na msimu wa mvua mingi? • Probe: how does this vary from borehole to borehole? What are the reasons for this variation? 7. What are the main reasons for delays in fixing broken EDE/county strategic boreholes?/Je sababu gani kuu ziliangazia kuchelewa kutengeneza EDE/Strategic boreholes za kaunti? a. During the long dry season?/Nyakati za ukame wa muda mrefu? b. At other times of year?/Nyakati zingine za mwaka? FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 82 c. Has COVID-19 affected borehole repairs? If so, how? If not, why not?/Je COVID-19 imeadhiri utengenezaji wa visima? Kama ni hivyo, ni vipi? Kama sivyo, ni kwa nini? 8. Does the management of EDE/county strategic boreholes differ from management of other boreholes in this area? If so, how?/je usimamizi wa EDE/strategic boreholes za kaunti ni tofauti na usimamizi wa visima vingine katika eneo hili? Kama ni hivyo, je ni vipi? a. Are these boreholes operated year round, or only during the dry season?/Je visima hivyo hufanya kazi mwaka mzima, ama ni misimu ya ukame pekee? b. Does the management structure for these boreholes differ from what was discussed in Question 2? If so, how?/Je muundo wa usimamizi wa visima hivi ni tofauti na chenye kilizungumziwa katika Swali la 2? Kama ni hivyo? Ni vipi? 9. I would like to learn more about the resources here to address borehole issues. How often do you need to request funds or resources, such as equipment or skilled technicians, to fix an EDE/county strategic borehole?/Ningependa kujua zaidi kuhusu rasilimali ambazo ziko hapa ili kushughulikia shida za kisima. Je, ni mara ngapi, unahitaji kuomba usaidizi wa kifedha ama rasilimali, kama vile vifaa au mafundi walio na ujuzi kutengeneza EDE/strategic borehole ya kaunti? a. Who do you talk to about the budget for these boreholes?/Ni nani huwa unazungumza naye kuhusu bajeti ya visima hivi? • Probe: What is that process like? Describe it, please./Je mkakati huo uko vipi? Tafadhali ieleze? b. Are there constraints to obtaining the funds needed to address the management and maintenance of these boreholes? Explain./Je kuna shida/ changamoto ambazo za kupata usaidizi wa fedha unaohitajika kushughulikia usimamizi na kutunzwa wa visima? Eleza. c. If you can obtain the budget needed, do you feel that you have the right staff capacity for managing these boreholes? Which staff are most important?/Iwapo unaweza kupata bajeti inayohitajika, je unahisi kama una idadi ya wafanyakazi wanaohitajika kusimamia visima hivi? Je ni wafanyakazi wagani ndio wa muhimu zaidi? d. Does the availability of funds vary throughout the year, i.e. are there times when there might be more money, times when there might be less?/Je, uwezo fedha zinazopata hubadilika kwa nyakati tofauti za mwaka, kama vile, kuna nyakati pesa ni mingi na nyakati zingine pesa ni kidogo? 10. What suggestions do you have for ways to improve management of boreholes in this area?/Je ni maoni gani unayo ya njia za kuboresha usimamizi wa visima katika eneo hili? • Probe: Who would need to be involved in implementing these suggestions?/Je, ni nani anafaa kuhusishwa katika kutekeleza maoni haya? • Probe: What opportunities do you see for taking these steps?/Je ni nafasi gani unaona za kuchukua hatua hizi? • Probe: What barriers do you see for better drought management?/Je ni vizuizi gani unaona katika usimamizi bora wa ukame? For respondents in Kenya RAPID counties only: 11. How familiar are you with the sensor-based systems that have been implemented to manage EDE/strategic boreholes in this county?/ Je ni kwa kiwango gani unafahamu mifumo ya sensa ambayo imetekelezwa kusimamia EDE/visima katika kaunti hii? a. Do you use the Kenya RAPID system?/Je huwa unatumia mfumo wa Kenya RAPID system? • Probe: If so, describe how you use it./Kama ni hivyo, eleza jinsi huwa unaitumia. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 83 12. Who is responsible for reviewing and managing the sensor data in your county?/Je ni nani ana wajibu wa kurejelea na kusimamia data ya sensa katika kaunti yako? 13. Where would you get resources to repair borehole after a breakdown detected by the sensors?/je ni wapi unaweza pata rasilimali ya kutengeneza kisima kichoharibika unapogundua kupitia sensa? a. Can you describe this process?/unaweza nieleza utaratibu unaotumiwa? b. Has the process for requesting resources for repairs changed since Kenya RAPID began implementing here?/Je utaratibu wa kuagiza rasilimali za kutengeneza umebadilika tangu mfumo Kenya RAPID ulianza kutekeleza hapa? c. What are the procedures for repair after obtaining resources?/Je, utaratibu mgani wa kutengeneza hufuatwa baada ya kupata rasilimali? 14. Have you personally used the data dashboard to view sensor information on EDE/strategic boreholes?/Je ushawahi tumia dashboard ya data kutazama habari ya sensa kutoka EDE/visima maalum? 15. Do you manage or interact with other people who have used the dashboard?/Je huwa unasimamia ama kuzungumza na watu wengine ambao wametumia dashboard? 16. Have you encountered any challenges using the sensor/ dashboard/ phone application?/Je umepata changamoto zozote ukitumia sensa/ dashboard/ programu ya simu? What happened?/Je ni nini ilifanyika? [interviewer: ensure you clarify whether the respondent is describing the sensor, dashboard, or phone app]. 17. What do you think about the system with respect to your water borehole management activities?/Je unahisi vipi kuhusu mfumo huo ukihusha shughuli zako za kusimamia visima? a. Do you feel the information/ data transmitted by the sensor is relevant? Do you feel that it is accurate?/Je unahisi habari/data ambayo hutumwa na sensa inafaa/ni ya muhimu? Je unahisi ni sahihi? • Probe: What about it is or is not relevant?/Je ni nini kuihusu inaifaa au haifai? b. Do you feel that the training you received on the dashboard/phone application is sufficient to help you in managing the boreholes? Why/why not?/Je unahisi mafunzo ambayo ulipata kwenye dashboard/programu ya simu inatosha kukusaidia kusimamia visima? Ni kwa nini? [interviewer: ensure you clarify whether the respondent is describing the dashboard or phone app] c. Are there any management challenges that you feel the Kenya RAPID system does not help address with respect to strategic borehole management?/Je kuna changamoto zozote ambazo unahisi mfumo wa Kenya RAPID haushughulikii kuhusiana na usimamizi wa kisima? d. Do you feel that the county has enough capacity to manage the data dashboards? Why or why not?/Je unahisi kaunti inauwezo wa kutosha kusimamia/kuenendeleza data ya dashboard? Ni kwa nini au kwa nini sio? e. How could this system be improved?/Je mfumo huu unaweza kuboreshwa vipi? 18. Have you had any problems with sensors being disconnected or vandalized in your area?/Je umekuwa na shida zozote za sensa kuzimwa ama kuharibiwa katika eneo lako? For non-Kenya RAPID respondents only: 19. Suppose that you were able to access information very quickly about which boreholes were working/not working. How would you use this information? Do you think this would help with management of these boreholes? If so, how? Explain./Je kama ungeweza kupata habari kwa haraka sana kuhusu ni visima gani vinafanya kazi/havifanyi kazi kwa njia ya haraka. Je unaweza kutumia vipi habari hii? Je unafiri hii inaweza kusaidia na usimamizi wa visima hivi? Kama ni hivyo, ni vipi? Eleza. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 84 G-2 KEY INFORMANT INTERVIEW SCRIPT: BOREHOLE MANAGERS/OPERATORS Respondent Information Respondent name:/Jina la mhojiwa: Location of interview: (county, town/village)/Eneo la mahojiano: (kaunti, mji/kijiji) Respondent organization:/Shirika la mhojiwa: Respondent job title:/Taji la kazi ya mhojiwa: Date of Interview:/tarehe ya mahojiano: Interview Questions (semi-structured interview – use script as a guide) 20. Please describe your involvement in water management./Tafadhali taja uhusiano wako na usimamizi wa maji. (Have respondent describe their role, and then probe as needed). a. How long have you held this job?/Je umefanya kazi hii kwa muda gani? b. How long have you worked in water management?/ Je umefanya kazi katika usimamizi wa maji kwa muda gani? c. What are the main water sources in the area that you manage?/Je kuna njia gani kuu za kupata maji katika eneo ambalo unasimamia? d. Where do people in this area mainly get their water? What about during the rainy season? During the dry season How many boreholes do you help manage?/Je ni wapi watu katika eneo hili hupata maji yao kwa ukuu? Je na wakati was msimu wa mvua mingi? Wakati wa msimu wa ukame ni visima ngapi huwa unasaidia kusimamia? i. Do you know how many of the boreholes you manage are designated as “ending drought emergency (EDE)” or strategic boreholes?/Je unafahamu visima ngapi ambavyo unasimamia vimeteuliwa kama “ending drought emergency (EDE)” (visima vya dharura wakati wa ukame) ama visima vya strategic boreholes? Visima ambavyo zingatiwa kuwa za muhimu wakati wa kiangazi? (Note: not all water managers will know which boreholes are designated as EDE. • Probe: If they don’t know: which boreholes are critical during the dry season? Throughout the rest of the script, use either “EDE borehole” or “county strategic” borehole to refer to these critical boreholes, depending on how the respondent identifies them. ii. Do you know how the EDE/strategic boreholes are decided or defined? What makes these boreholes different from other boreholes?/Je unafahamu jinsi visima vya EDE/strategic boreholes vinachaguliwa au kutengwa vile? Ni nini hufanya visima hivi tofauti na visima vingine? e. What is your role in managing these boreholes?/Je kazi yako ni gani katika kusimamia visima hivi? f. Are the water points you manage mainly rural, urban, or a mix?/Je maeneo haya ya maji ambayo unasimamia yako kwa ukuu yako kijijini, mjini, ama ni mchanganyiko? 21. What is the management structure for the water points that you help manage?/Je muundo wa usimamizi wa maeneo haya ya maji ambayo huwa unasaidia kusimamia ni gani? a. Are these water points managed by local water committees? WASCOs (utilities)? Other organizations?/Je maeneo haya ya maji yanasimamiwa na kamati za maji katika eneo hilo, huduma za WASCO au mashirika mengine? FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 85 i. For water committees: how many people are in these committees, and how does someone become a member?/Ya kamati za maji: ni watu wangapi wako katika kamati hizi, na mtu anaweza kuwa mwanachama vipi? b. Who is responsible for operating the site (e.g., turning the borehole on and off)?/Ni nani ana wajibu wa kuendesha eneo hilo (kwa mfano kuwasha na kuzima kisima) c. Who is responsible for maintaining and repairing boreholes?/Ni nani ana wajibu wa kuhudumia na kuregeza visima? d. Are fees collected for water use?/Je kuna ada ambazo huwa zinachukuliwa kwa matumizi ya maji? e. Are fees collected when the water point needs maintenance?/Je kuna ada ambazo huwa zinachukuliwa wakati eneo la maji linahitaji kuundwa? f. Who is responsible for collecting and managing fees?/Ni nani ana wajibu wa kuchukua na kusimamia ada? g. How do the different actors involved in managing this borehole interact with one another?/Je washiriki tofauti wanaohusika katika kusimamia kisima hiki huwa wanashirikia ki vipi? (Where relevant, probe for: i. Local operator relationship with county government/Mahusiano ya mwendeshaji wa kisima na serikali ya kaunti ii. WASCO relationship with county government/Mahusiano ya WASCO na serikali ya kaunti iii. County government relationship with operations and maintenance organizations like Catholic Diocese in Turkana/Mahusiano ya serikali ya kaunti na mashirika ya kuendesha na kutunza kama vile Kanisa la Katoliki Diocese ya Turkana iv. WASCO relationship with rate payers)/Mahusiano ya WASCO na wanaolipa ada 22. What are the main challenges you face in providing reliable water service to users in this area?/Je nichangamoto gani kuu huwa unapitia katika kutoa huduma za maji inayotegemewa kwa wakaazi katika eneo hili? a. Do you receive any complaints about the borehole from users during the drought season?/Je huwa unapokea malalamishi yoyote kuhusu kisima kutoka watumizi katika msimu ya ukame? • Probe: What kind of complaints?/Je ni malalamishi ya aina gani? • Probe: How do you typically receive these complaints?/Je huwa unapokea malalamishi haya vipi kwa kawaida? b. Are you able to address these complaints? If not, why?/Je huwa unaweza kusuluhisha malalamishi haya? Kama huwa huwezi, ni kwa nini? 23. What are the main challenges that you experience in providing reliable water service during the dry season?/Je ni changamoto gani kuu huwa unapata katika kutoa huduma ya maji ya kutegemewa katika msimu wa ukame? a. Do the challenges differ during the dry season from other times of year? If so, how?/Je changamoto huwa zinatofautiana katika msimu wa ukame ukilinganisha na nyakati zingine za mwaka? Kama ni hivyo, ni vipi? b. How do you or your team manage or mitigate these challenges?/Ni vipi wewe ama jopo lako linasimamia huwa linashughulikia au kusuluhisha changamoto hizi? 24. Has the water situation changed as a result of COVID-19?/Je hali ya maji imebadilika kufuatia COVID-19? a. Has there been any change in funding or support for borehole pumped water? What about water from other sources?/Je kumekuwa na mabadiliko yoyote katika usaidizi wa fedha au wowote kwa maji ya bomba? Je na kwa maji kutoka njia zingine? FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 86 b. Have there been changes related to water use patterns as a result of COVID-19?/Je kushawahi kuwa na mabadiliko yanayohusiana na mitindo ya matumizi ya maji kufuatia COVID-19? 25. How many of the strategic boreholes that you manage are currently functional?/Je ni ngapi kati ya visima ambayo unasimamia vinafanya kazi kwa sasa? a. About how many times did an EDE/country strategic borehole break during the past long dry season? Was this more/less/about the same as usual during this time of year? If they manage more than one borehole: Is this the same for all of the boreholes that you manage?/Ni takriban mara ngapi EDE/ kisima ya kaunti iliharibika msimu wa ukame wa muda mrefu uliopita? Je hii ilikuwa zaidi/kidogo/takriban idadi sawa na kawaida wakati huu wa mwaka? b. What do you think is the main cause of breakages?/Je unahisi ni jambo gani kuu ilisababisha uharibifu? • Probe: Which parts typically break – pump, generator, power source, etc.?/Je ni maeneo gani huwa yanaharibika kwa kawaida - bomba, jenereta, umeme, na kadhalika? 26. When EDE/county strategic boreholes in your area breakdown, how long does it take…/Je EDE/visima vikiharibika katika eneo lako, je huwa inachukua muda gani… a. For the water office to find out the borehole is broken?/Afisi ya maji kujua kisima kimeharibika? b. To get information about the breakages (i.e., specifically what is broken and what repairs are needed)?/Kupata habari kuhusu kuharibika (hiyo ni nini haswa imeharibika na ni nini inafaa kutengenezwa)? c. To repair the broken boreholes?/Kutengeneza kisima ambacho kimeharibika? • Probe: Do the answers to these questions differ between the dry season and wet season?/Je majibu ya maswali haya yanatofautiana na msimu wa ukame na msimu wa mvua mingi? 27. Please think about the most recent time when you experienced a broken EDE/county strategic borehole during the drought season. Can you describe what happened?/Tafadhali fikiria kuhusu mara ya mwisho ulipata EDE/kisima kiliharibika katika msimu wa ukame? Je unaweza eleza ni nini ilifanyika? a. How did the borehole break?/Je kisima hicho kiliharibika vipi? b. Where was this borehole?/Je kisima hicho kilikuwa wapi? c. Who was responsible for repairing the borehole?/Je ilikuwa wajibu wa nani kutengeneza kisima hicho? d. How did the person/organization responsible for fixing the borehole learn that it was broken?/Je mtu/shirika ambalo linawajibu wa kutengeneza kisima hicho, a/lilifahamu vipi kuwa kilikuwa kimeharibika? • Probe: Who reported the information?/Ni nani aliripoti kuharibika? • Probe: How did they report it?/Je waliripoti vipi? • Probe: How long was the borehole broken before it was reported?/Je kisima kilikuwa kimeharibika kwa muda gani kabla kuripotiwa? e. After the information was reported, what happened?/Baada ya habari hiyo kuripotiwa, ni nini ilifanyika? • Probe: Who repaired it?/Ni nani alitengeneza? • Probe: How long did it take for the borehole to be repaired?/Je ilichukua muda gani kabla kisima hicho kutengenezwa? FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 87 • Probe: Where did the funds for the repair come from?/Je pesa za kutengeneza kisima zilitoka wapi? f. Looking back on this episode, what worked well?/Ukikumbuka kipindi hicho, ni nini ilifanya kazi vyema? g. Looking back on this episode, what could have worked better?/Ukikumbuka kipindi hicho, ni nini ingefanya kazi vyema? • Probe: What problems were encountered?/Je ni shida gani ziliwakumba? 28. What are the main reasons for delays in fixing broken EDE/county strategic boreholes?/Je sababu gani kuu ziliangazia kuchelewa kutengeneza EDE/Strategic boreholes za kaunti? a. During the long dry season?/Nyakati za ukame wa muda mrefu? b. At other times of year?/Nyakati zingine za mwaka? c. Has COVID-19 affected borehole repairs? If so, how? If not, why not?/Je COVID-19 imeadhiri utengenezaji wa visima? Kama ni hivyo, ni vipi? Kama sivyo, ni kwa nini? 29. Does the management of EDE/county strategic boreholes differ from management of other boreholes in this area? If so, how?/je usimamizi wa EDE/strategic boreholes za kaunti ni tofauti na usimamizi wa visima vingine katika eneo hili? Kama ni hivyo, je ni vipi? a. Are these boreholes operated yearround, or only during the dry season?/Je visima hivyo hufanya kazi mwaka mzima, ama ni misimu ya ukame pekee? b. Does the management structure for these boreholes differ from what was discussed in Question 2? If so, how?/Je muundo wa usimamizi wa visima hivi ni tofauti na chenye kilizungumziwa katika Swali la 2? Kama ni hivyo? Ni vipi? 30. I would like to learn more about the resources here to address borehole issues. How often do you need to request funds or resources, such as equipment or skilled technicians, to fix an EDE/county strategic borehole?/Ningependa kujua zaidi kuhusu rasilimali ambazo ziko hapa ili kushughulikia shida za kisima. Je, ni mara ngapi, unahitaji kuomba usaidizi wa kifedha ama rasilimali, kama vile vifaa au mafundi walio na ujuzi kutengeneza EDE/strategic borehole ya kaunti? a. Who do you talk to about the budget for these boreholes?/Ni nani huwa unazungumza naye kuhusu bajeti ya visima hivi? • Probe: What is that process like? Describe it, please./Je mkakati huo uko vipi? Tafadhali ieleze? b. Are there constraints to obtaining the funds needed to address the management and maintenance of these boreholes? Explain./Je kuna shida/ changamoto ambazo za kupata usaidizi wa fedha unaohitajika kushughulikia usimamizi na kutunzwa wa visima? Eleza. c. If you can obtain the budget needed, do you feel that you have the right staff capacity for managing these boreholes? Which staff are most important?/Iwapo unaweza kupata bajeti inayohitajika, je unahisi kama una idadi ya wafanyakazi wanaohitajika kusimamia visima hivi? Je ni wafanyakazi wagani ndio wa muhimu zaidi? d. Does the availability of funds vary throughout the year, i.e. are there times when there might be more money, times when there might be less?/Je, uwezo fedha zinazopata hubadilika kwa nyakati tofauti za mwaka, kama vile, kuna nyakati pesa ni mingi na nyakati zingine pesa ni kidogo? 31. What suggestions do you have for ways to improve management of boreholes in this area?/Je ni maoni gani unayo ya njia za kuboresha usimamizi wa visima katika eneo hili? • Probe: Who would need to be involved in implementing these suggestions?/Je, ni nani anafaa kuhusishwa katika kutekeleza maoni haya? FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 88 • Probe: What opportunities do you see for taking these steps?/Je ni nafasi gani unaona za kuchukua hatua hizi? • Probe: What barriers do you see for better drought management?/Je ni vizuizi gani unaona katika usimamizi bora wa ukame? For respondents in Kenya RAPID counties only: 32. How familiar are you with the sensor-based systems that have been implemented to manage EDE/strategic boreholes in this county? Note: if they are not familiar with the sensors, probe to determine if they are aware of the sensor installed on the pump, if they are not familiar with it, conclude interview, if they are familiar with it continue./Je ni kwa kiwango gani unafahamu mifumo ya sensa ambayo imetekelezwa kusimamia EDE/visima katika kaunti hii? a. Do you use the Kenya RAPID system?/Je huwa unatumia mfumo wa Kenya RAPID system? • Probe: If so, describe how you use it./Kama ni hivyo, eleza jinsi huwa unaitumia. 33. Who is responsible for reviewing and managing the sensor data in your county?/Je ni nani ana wajibu wa kurejelea na kusimamia data ya sensa katika kaunti yako? 34. Where would you get resources to repair borehole after a breakdown detected by the sensors?/je ni wapi unaweza pata rasilimali ya kutengeneza kisima kichoharibika unapogundua kupitia sensa? a. Can you describe this process?/unaweza nieleza utaratibu unaotumiwa? b. Has the process for requesting resources for repairs changed since Kenya RAPID began implementing here?/Je utaratibu wa kuagiza rasilimali za kutengeneza umebadilika tangu mfumo Kenya RAPID ulianza kutekeleza hapa? c. What are the procedures for repair after obtaining resources?/Je, utaratibu mgani wa kutengeneza hufuatwa baada ya kupata rasilimali? 35. Do you manage or interact with other people who have used the dashboard?/Je huwa unasimamia ama kuzungumza na watu wengine ambao wametumia dashboard? 36. Have you encountered any challenges using the sensor/ dashboard/ phone application?/Je umepata changamoto zozote ukitumia sensa/ dashboard/ programu ya simu? What happened?/Je ni nini ilifanyika? [interviewer: ensure you clarify whether the respondent is describing the sensor, dashboard, or phone app]. 37. What do you think about the system with respect to your water borehole management activities?/Je unahisi vipi kuhusu mfumo huo ukihusha shughuli zako za kusimamia visima? a. Do you feel the information/ data transmitted by the sensor is relevant? Do you feel that it is accurate?/Je unahisi habari/data ambayo hutumwa na sensa inafaa/ni ya muhimu? Je unahisi ni sahihi? • Probe: What about it is or is not relevant?/Je ni nini kuihusu inaifaa au haifai? b. Do you feel that the training you received on the dashboard/phone application is sufficient to help you in managing the boreholes? Why/why not?/Je unahisi mafunzo ambayo ulipata kwenye dashboard/programu ya simu inatosha kukusaidia kusimamia visima? Ni kwa nini? [interviewer: ensure you clarify whether the respondent is describing the dashboard or phone app] c. Are there any management challenges that you feel the Kenya RAPID system does not help address with respect to strategic borehole management?/Je kuna changamoto zozote ambazo unahisi mfumo wa Kenya RAPID haushughulikii kuhusiana na usimamizi wa kisima? d. Do you feel that the county has enough capacity to manage the data dashboards? Why or why not?/Je unahisi kaunti inauwezo wa kutosha kusimamia/kuenendeleza data ya dashboard? Ni kwa nini au kwa nini sio? e. How could this system be improved?/Je mfumo huu unaweza kuboreshwa vipi? FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 89 38. Have you had any problems with sensors being disconnected or vandalized in your area?/Je umekuwa na shida zozote za sensa kuzimwa ama kuharibiwa katika eneo lako? For non-Kenya RAPID respondents only: 39. Suppose that you were able to access information very quickly about which boreholes were working/not working. How would you use this information? Do you think this would help with management of these boreholes? If so, how? Explain./Je kama ungeweza kupata habari kwa haraka sana kuhusu ni visima gani vinafanya kazi/havifanyi kazi kwa njia ya haraka. Je unaweza kutumia vipi habari hii? Je unafiri hii inaweza kusaidia na usimamizi wa visima hivi? Kama ni hivyo, ni vipi? Eleza. G-3 GROUP DISCUSSION GUIDE This guide should be used for all FGD respondent categories./Mwongozo huu unafaa kutumika kwa vitengo vyote vya wahojiwa. Total number of people in FGD:/Idadi ya jumla ya watu wa kundi la mjadala: ____________ Number of women:/Idadi ya wanawake: ____________ A. Overview of Water Use History [15 minutes] 1. Please tell us about this area./Tafadhali tueleze kuhusu eneo hili. Facilitator probes: a. How do most people earn a living here?/Je watu wengi katika eneo hili hupata kipato vipi? b. What are the main sources of water that people in this area use?/Je njia kuu za maji ambazo watu wengi katika eneo hili hutumia ni gani? i. During the short dry season?/Msimu wa ukame wa muda mfupi? ii. During the short rainy season?/Msimu wa mvua mingi wa muda mfupi? iii. During the long dry season?/Msimu wa ukame wa muda mrefu? iv. During the long rainy season?/Msimu wa mvua mingi wa muda mrefu? 2. How many people here use the local borehole?/Ni watu wangapi hapa ambao hutumia kisima ya eneo hili? Refer to the focus borehole. Facilitator probes: a. For what types of activities do people use the borehole?/Ni aina gani za shughuli watu hutumia kisima? (Probe for: personal use, livestock, irrigation) b. For those that do not use local boreholes (if any): Why don’t you use them?/Ni kwa nini huwa hamzitumii? B. Borehole Access and Use [20 minutes] 1. Can you access the water from this borehole without any challenges?/Je unaweza kupata maji kutoka kisima hii bila changamoto zozote? a. How many people here have personally experienced a challenge accessing water from this borehole?/Je ni watu wangapi huku washawahi kupata changamoto ya kupata maji kutoka kisima hii? b. What challenges have you faced?/Ni changamoto gani ushawahi pata? Facilitator probes: Physical access? Any rules imposed by managers?/Uwezo wa kupata maji? Kuna sheria zozote ambazo zimewekwa na mameneja/ msimamizi wa kisima hiki? c. Are there any groups or types of people who face more of these types of challenges, or have greater challenges accessing water?/Je kuna makundi yoyote ama aina ya watu ambao wanapata changamoto kubwa au changamoto zaidi za kupata maji? i. Women? Elderly?/Wanawake? Wazee? FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 90 ii. (If yes): Why?/Ni kwa nini? d. Have people here changed their use of the borehole since the COVID-19 pandemic started? /Je watu hapa washawahi badilisha matumizi yao ya kisima tangu janga la COVID-19 kuanza? 2. Do people need to pay to get water from this borehole? How much?/Je watu huwa wanahitajika kulipa kupate maji kutoka kisima hii? Ni pesa ngapi? 3. For (each main challenge listed regarding access), what is the solution?/Kwa (kila changamoto kuu iliyoorodheshwa kuhusiana na uwezo wa kupata) je suluhisho lake ni gani? a. Are there any institutions available here to help resolve access challenges? How well do you trust these institutions to assist you? Can you give me an example?/Je kuna taasisi zozote ambazo ziko eneo hili ambazo zinasaidia kusuluhisha shida za upatikanaji wa maji? Je ni kwa kiwango gani unaamini taasisi hizi kukusaidia? Tafadhali nipe mfano? 4. Does borehole accessibility vary over the month, or over the year? For example, based on the season or who is in charge?/Je uwezo wa kupata huduma za kisima huwa unabadilika kulingana na miezi, ama mwaka? Kwa mfano, kulingana na msimu ama mwenye anasimamia? 5. Have people here in this group, or others you are familiar with, contacted the local borehole manager regarding accessibility and service?/Je watu katika kundi hili, ama wengine ambao unafahamu, wameweza kuwasiliana na msimamizi wa kisima kuhusu upatikanaji wa huduma katika kisima? a. What were the reasons for this?/Je sababu za hii zilikuwa nini? b. Is this issue common here, or is it uncommon?/Je shida hii ni ya kawaida hapa, ama siyo ya kawaida? c. How often have you contacted the water manager or heard about people contacting him or her?/Je ni mara ngapi wewe umewasiliana na meneja wa maji ama kusikia kuhusu watu wengine waliwasiliana naye? d. When was the last time you or someone you know contacted the local borehole manager?/Je ni lini mara ya mwisho wewe ama mtu unayejua aliwasiliana na meneja wa kisima ya eneo lako? C. Borehole Maintenance and Management [35 minutes] 1. Do you know who is in charge of fixing this borehole when it breaks? Who is it?/Je unafahamu mwenye anasimamia kutengeneza kisima hii inapoharibika? Je ni nani? a. Is there someone who is in charge of contacting the water manager when the borehole breaks? How do they contact the water manager?/Je kuna mtu ambaye anasimamia kuwasiliana na meneja wa maji kisima inapoharibika? Je huwa wanawasiliana na meneja wa maji vipi? 2. When was the last time this borehole broke?/Je ni lini mara ya mwisho kisima hii iliharibika? 3. Are there financial resources for the local water borehole managers to maintain the borehole?/Je kuna rasilimali za fedha kwa meneja wa kisima ya maji ya eneo lako kutunza kisima? a. Where do they come from?/Je huwa zinatoka wapi? b. Are they sufficient? Why or why not? Give examples./Je zinatosha? Ni kwa nini? Toa mifano. c. Have the managers or other representatives from the government or aid program talked to borehole users about maintenance?/ Je mameneja ama wawakilishi wengine kutoka serikali ama mradi wa usaidizi wamezungumza na watumizi wa kisima kuhusu utunzaji? 4. Are there currently any maintenance challenges at this borehole? For example, equipment failure/breakdowns?/Je kuna changamoto zozote za kutunza hivi sasa katika kisima hii? Kwa mfano, vifaa kuharibika/ kuacha kufanya kazi? a. Are there some times of year when maintenance challenges are more likely to occur?/Je FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 91 kuna nyakati zingine za mwaka ambazo changamoto za kutunza zina uwezekano mkubwa wa kufanyika? b. Do you think the borehole managers plan for seasonal or other known issues that arise each year?/Je unahisi mameneja wa kutunza huwa wanafanya mipango ya msimu ama shida zingine zinazojulikana mambazo hutokea kila mwaka? 5. Was the management of this borehole always like this? Has it changed over time?/Je usimamizi wa kisima hii ulikuwa hivi kila wakati? Je imebadilika na muda? a. What has changed?/Je ni nini imebadilika? • Probe: Ask for examples, such as staff, community outreach, resources. b. Do you think these changes are positive or negative for accessing water?/J unahisi mabadiliko haya ni nzuri ama siyo nzuri katika kupata maji? D. Drought Risk [35 minutes] 1. When was the last drought season? 2. When was the last dry season? [NOTE: if the drought season was not in the past year, proceed asking about the dry season] 3. Please think about the most recent time dry/drought season./Tafadhali fikiria kuhusu msimu wa ukame wa mwisho. a. Were you able to access water for you and your household during the dry season/drought?/ Je uliweza kupata maji yako na nyumba yako msimu wa ukame? b. What water sources did you use during the drought?/Ni njia gani ya maji ulitumia wakati wa ukame? c. Did you have any trouble accessing water from this borehole?/Je ulipata shida zozote za kupata maji kutoka kisima hii? d. Did this borehole break?/Je kisima hii iliharibika? • Probe: Do you know how it broke?/Je unajua jinsi iliharibika? • Probe: Who was responsible for repairing the borehole?/Je ilikuwa wajibu ya nani kuunda kisima hiyo? • Probe: How long did it take for the borehole to get repaired?/Je ilichukua muda gani kwa kisima kutengenezwa? 4. During the dry/drought season, do you think that some people here face larger water challenges than others?/Katika msimu wa ukame, je unahisi kuwa watu wengine hapa hupata changamoto kubwa kuliko wengine? a. Do women or the elderly (or other groups) have the same access to water as everyone else?/Je wanawake ama wazee (ama kundi zingine) wana uwezo wa kupata maji sawia na watu wengine? 5. What suggestions do you have for ways to improve management of boreholes in this area?/Je ni maoni yapi unayo ya njia za kuboresha usimamizi wa visima katika eneo hili? • Probe: Who would need to be involved in implementing these suggestions?/Ni nani anahitaji kujumuishwa katika kutekeleza ushauri huu? • Probe: What opportunities do you see for taking these steps?/Je ni nafasi gani unaona kuchukua hatua hizi? • Probe: What barriers do you see for better drought/dry season management?/Ni vizuizi gani unaona katika usimamizi wa ukame? 6. Has anyone noticed sensors or gadgets that were installed on this borehole?/Je kuna yeyote ambaye amegundua sensa ama vidude ambavyo viliwekwa katika kisima hii? FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 92 a. Do you know what these sensors are for?/Je unafahamu sensa hizi ni za nini? b. Has anyone in this area tried to take the sensors off or stop them from working?/Je kuna yeyote katika eneo hili amejaribu kutoa sensa hizi ama kuziharibu? (Facilitator can then explain that the sensors help the USAID project measure whether or not boreholes are working so we can understand problems with the water systems and how those systems might be fixed in the future.) E. Recorded responses [10 minutes] No. Question # of hands YES # of hands for NO 1 How many people here feel they can rely on the local borehole for their water needs during the wet season?/Ni watu wangapi hapa wanahisi wanaweza kutegemea kisima ya eneo hili kwa matumizi yao ya maji msimu wa mvua mingi? 2 How many people here feel they can rely on the local borehole for their water needs during the dry season?/Ni watu wangapi hapa wanahisi wanaweza kutegemea kisima ya eneo hili kwa matumizi yao ya maji msimu wa ukame? 3 How many people here are comfortable contacting the local water pump operator or water committee member if they see there is an issue with the borehole?/Ni watu wangapi hapa wanaweza kufurahia kuwasiliana na opareta wa bomba la maji la eneo lako ama mwanachama katika kamati ya maji wanapoona shida katika kisima? 4 How many people here are confident that the water pump operator or water committee member will address an issue in a reasonable amount of time if it is reported by a member of the community?/Ni watu wangapi hapa wana imani opareta wa bomba la maji la eneo lako ama mwanachama katika kamati ya maji wanaweza kushughulikia shida kwa wakati unaofaa inaporipotiwa na mtu katika jamii? 5 How many people here believe the borehole is at risk during the next drought?/Ni watu wangapi hapa wanaamini kisima iko katika hatari katika msimu wa ukame ujao? F. Conclusion [5 minutes] Thank you for discussing these issues with us today. We asked a lot of questions. Is there anything that you want to add, or would like to ask us?/Asante kwa kujadili maswala haya nasi leo. Tumeuliza maswali mengi. Je kuna chochote ungependa kuongeza , ama ungependa kutuuliza? FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 93 ANNEX H: ANONYMIZED RESPONDENT LISTS TABLE H-1: KII LIST County Position of Key Informants Date of Interview Tana River County Chief Engineer: Technical and operations 17.09.2020 Sub-County Water Officer 18.09.2020 Borehole Manager/Operator 20.09.2020 Borehole Manager/Operator 19.09.2020 Turkana Sub-county Water Officer/Technical Operator 19.09.2020 Sub-county Water Officer/Technical Operator 18.09.2020 Borehole Manager/ Operator 20.09.2020 Borehole Manager/Operator 19.09.2020 Garissa Director of Water Services 20.09.2020 Sub-County Water Officer 18.09.2020 Borehole Manager/Operator 20.09.2020 Borehole Manager/Operator 18.09.2020 West Pokot Deputy Water Engineer 17.09.2020 Sub-County Water Officer 19.09.2020 Borehole Manager/Operator 18.09.2021 Borehole Manager/Operator 19.09.2020 TABLE H-2: FGD LIST FGD # County Date Age Gender 1 Tana River 19.09.2020 50 Female 1 Tana River 19.09.2020 57 Female 1 Tana River 19.09.2020 22 Female 1 Tana River 19.09.2020 23 Female 1 Tana River 19.09.2020 25 Female 1 Tana River 19.09.2020 22 Female 1 Tana River 19.09.2020 32 Female 1 Tana River 19.09.2020 67 Female 2 Tana River 20.09.2020 65 Female 2 Tana River 20.09.2020 52 Female 2 Tana River 20.09.2020 51 Female 2 Tana River 20.09.2020 42 Female 2 Tana River 20.09.2020 39 Female 2 Tana River 20.09.2020 24 Female 2 Tana River 20.09.2020 36 Female 2 Tana River 20.09.2020 52 Female 3 West Pokot 18.09.2020 32 Female 3 West Pokot 18.09.2020 41 Female 3 West Pokot 18.09.2020 29 Female 3 West Pokot 18.09.2020 42 Female 3 West Pokot 18.09.2020 61 Female 3 West Pokot 18.09.2020 31 Female 3 West Pokot 18.09.2020 22 Female 3 West Pokot 18.09.2020 40 Female FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 94 FGD # County Date Age Gender 3 West Pokot 18.09.2020 36 Female 4 West Pokot 19.09.2020 45 Female 4 West Pokot 19.09.2020 43 Female 4 West Pokot 19.09.2020 40 Female 4 West Pokot 19.09.2020 71 Female 4 West Pokot 19.09.2020 23 Female 4 West Pokot 19.09.2020 25 Female 4 West Pokot 19.09.2020 41 Female 4 West Pokot 19.09.2020 23 Female 5 Turkana 19.09.2020 40 Female 5 Turkana 19.09.2020 41 Female 5 Turkana 19.09.2020 32 Female 5 Turkana 19.09.2020 33 Female 5 Turkana 19.09.2020 60 Female 5 Turkana 19.09.2020 37 Female 5 Turkana 19.09.2020 48 Female 5 Turkana 19.09.2020 62 Female 5 Turkana 19.09.2020 38 Female 5 Turkana 19.09.2020 29 Female 6 Turkana 20.09.2020 45 Female 6 Turkana 20.09.2020 40 Female 6 Turkana 20.09.2020 42 Female 6 Turkana 20.09.2020 36 Female 6 Turkana 20.09.2020 53 Female 6 Turkana 20.09.2020 28 Female 6 Turkana 20.09.2020 27 Female 6 Turkana 20.09.2020 35 Female 6 Turkana 20.09.2020 56 Female 6 Turkana 20.09.2020 39 Female 7 Garissa 17.09.2020 78 Male 7 Garissa 17.09.2020 69 Male 7 Garissa 17.09.2020 66 Male 7 Garissa 17.09.2020 25 Male 7 Garissa 17.09.2020 29 Male 7 Garissa 17.09.2020 30 Male 7 Garissa 17.09.2020 68 Male 7 Garissa 17.09.2020 26 Male 8 Garissa 19.09.2020 53 Male 8 Garissa 19.09.2020 60 Male 8 Garissa 19.09.2020 45 Male 8 Garissa 19.09.2020 46 Male 8 Garissa 19.09.2020 73 Male 8 Garissa 19.09.2020 32 Male 8 Garissa 19.09.2020 38 Male 8 Garissa 19.09.2020 32 Male FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 95 ANNEX I: ADDITIONAL ANALYTICAL DETAILS Figure I-1 below shows the overall borehole pump on-time average as a thick line, with Kenya RAPID counties in red, and individual borehole pump on-times as thinner lines. As with Round II, borehole pump performance varied by county across and within assignment groups and throughout the year. For example, between August 2018 and August 2020, the treatment counties of Turkana and Marsabit had the highest borehole pump on-time in the sample (at 29% and 24%, respectively), but comparison county West Pokot had the third highest on-time over this period with 24% average on-time. The variation in on-time is notable in some counties, such as Baringo (reporting an average borehole pump on-time of 21%, but with a standard deviation of 30%) and Tana River (average on-time was 18% (i.e., around 4.5 hours), with a standard deviation of 29%). FIGURE I-1: BOREHOLE PUMP ON-TIME BY COUNTY AND MONTH FOR ROUNDS II AND III I-1 BOREHOLE ATTRITION The comparison sensor data at Round III cover a period from October 11, 2019 through August 31, 2020. The data collection team was able to visit each of the 115 boreholes that remained in the sample after Round II, but only 110 (96%) of the boreholes had the asset survey readministered, while data downloading only took place at 97 (84%) of the boreholes. In three of the five cases where it was not possible to attempt a download, the borehole was abandoned, and the field team was unable to access the sensor. In the two other cases, local conflict inhibited access. Of the 110 boreholes accessed, 107 still had sensor on site, of which 97 had data that was successfully downloaded at Round III. This represents a 15% decline between rounds, and a 26% attrition rate from the original 132 boreholes included in the sample. As shown in Table I-1 below, there was a difference of 1 to 3 sensors between the target and actual sensor downloads. In several cases, the field team went through the download procedures, but subsequent review of the downloaded material found that the file was either corrupted or there were no readings. For example, one of the 17 borehole data files from Baringo were found not to contain any information. In Laikipia, one of the sensors had been damaged, while in West Pokot one of the sensors was missing. In each of the other counties where data FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 96 were not available for download, the team found the sensor on site, but it was unclamped and did not contain any data. TABLE I-1: ROUND III COMPARISON BOREHOLE DATA COLLECTION SUMMARY County Target Boreholes Visited Boreholes with survey administered Boreholes where sensor data was downloaded successfully Boreholes where Sensor was reinstalled Baringo 17 17 17 15 16 Kitui 20 20 20 19 19 Laikipia 23 23 23 22 21 Mandera 12 12 11 11 11 Meru 7 7 7 7 7 Samburu 14 14 11 8 9 Tana River 9 9 8 5 7 West Pokot 13 13 13 10 9 Grand Total 115 115 110 97 99 In Kenya RAPID counties, the sensors broadcast data and the RAPID partner team can identify quickly when sensors are disconnected or stop recording data, as well as make follow-up calls or send field staff to verify the situation. The evaluation team does not have a full record explaining why sensors were uninstalled or missing. In May and June of 2020, the evaluation team’s local coordinator made calls to the borehole operators of 25 randomly chosen boreholes to check on their status. In five cases, the local coordinator was unable to reach the operator, while the sensor was reported to still be installed and operational in the other 20 cases where contact was made. In some cases, the enumeration team determined it was likely that new borehole managers who were unfamiliar with the sensors removed them during repairs. For example, at a borehole in Tana River, a sensor was found to have been unclamped and set aside when the borehole system was repaired. Given that there is not continuous monitoring in the comparison counties, it is not possible to verify why sensors were removed in all cases. Figure I-2 shows the number of boreholes that dropped out of the sample between Round II and Round III, which includes three treatment boreholes. Conversations with SweetSense and review of quarterly reports show that they also dealt with boreholes moving, which may explain why the boreholes dropped out of the data. An attrition rate of nearly 30% presents challenges to the overall fidelity of the design and ability of the evaluation to detect an effect. This does not mean that there is no longer value in comparing Kenya RAPID borehole pump performance to non-Kenya RAPID borehole performance. Such comparisons, both through model estimation and descriptive analysis, provide useful information for responding to the evaluation questions. The decline in the sample size does limit how general the findings are and means that it is more difficult to rely on estimates where effect sizes are small. That is, with a smaller sample size and small effect sizes, it is likely that the directionality of the impact, whether Kenya RAPID increased or decreased pump-on time, will be more difficult to estimate. A summary of the approach to statistical power is provided in Annex D. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 97 FIGURE I-2: NUMBER OF BOREHOLES THAT LEFT THE SAMPLE BY ASSIGNMENT AND COUNTY I-2 RAINFALL BY COUNTY As noted in the main report, the relationship between the proportion of water sources that are boreholes and rainfall varies by county. As shown in Figure I-3, the inverse relationship is strongest for Marsabit and Wajir, while there is largely no relationship between these measures for Turkana. FIGURE I-3: AVERAGE BOREHOLE WATER SOURCE PERCENTAGE AND RAINFALL BY COUNTY FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 98 I-3 MODEL SENSITIVITY TESTING The key outcome of interest for this impact evaluation (IE) is borehole on-time during the drought season. As noted in the text, rainfall was higher than expected during the implementation period. Between 2018 and 2020, the traditional dry months (roughly May through September and December through February) were notably rainy. The drought indicator used for this IE includes September, January, February, and June, based on rainfall around the boreholes in the sample. To test whether our drought indicator is potentially mis-specified, we ran Model 1, using a sensitivity dummy that is equal to 1 for the months of November and April, two of the highest rainfall months. As shown in Figure I-4, the overall average on-time for treatment counties relative to comparison counties remains around 11% higher for the optimally matched sample, controlling for other baseline characteristics, rainfall, and the percentage of borehole water sources. This is similar to the finding for the non-sensitivity test outcome. However, our impact estimate for the rainy months dummy shows that there was essentially no difference between Kenya RAPID and non-Kenya RAPID counties, CI - 0.019 0.027. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 99 FIGURE I-4: MODEL SENSITIVITY TEST RESULTS As a separate sensitivity test, we look at how clustering the standard errors and confidence intervals affects estimates. The main report text explains why clustering makes sense, but Figure I-5 below highlights how this affects the confidence intervals around our estimates. A key take-away here is that the point estimates do not change. The impact of Kenya RAPID during the drought season is still about 0.02 more on-time than comparison boreholes on average, controlling for key covariates. The magnitude of the estimate is constant regardless of how we treat the standard errors, or as shown below whether we use a mixed effects model.44 44 This approach has the same specification as Model 1 but allows the intercept to vary by county. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 100 FIGURE I-5: MODEL 1 RESULTS WITH AND WITHOUT CLUSTERING FOR THE OPTIMALLY MATCHED SAMPLE I-4 COMPARISON WITH ROUND 2 RESULTS Round II analysis found that during drought periods, on-time was about 0.04 percentage points (-0.02 0.09 95% CI) higher in Kenya RAPID county strategic boreholes compared to non-Kenya RAPID strategic boreholes during the drought season45 for the optimally matched sample. At Round II, the evaluation team also ran an analysis that only included the dry months of January, February, and September, which found a treatment effect between 0.18 to 0.39 for the optimally matched sample. As the evaluation team noted in the Round II report, however, this approach throws out data that may be 45 At Round II, we defined the drought season as only include January, February, and September based on the rainfall data for the implementation period covered. During Round III, we also extended this to include June. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 101 moderating some of the effect on on-time and explaining the variation in borehole pump on-time, which limits the estimates’ reliability. The evaluation team re-ran its updated analytical approach only on the Round II series to see the impact of including borehole substitutes on the estimates. As shown in Figure I-6 below, controlling for borehole substitutes and clustering at the county-level, Model 1 estimates an impact that ranges from slightly less than 0 to a 9% increase (95% CI) for the optimally matched sample, which is similar to what was found at Round II. This range suggests that there is no statistically significant effect, though this is somewhat sensitive to the modelling approach (e.g., whether clustering is applied). As noted in the descriptive analysis, there is a relationship between the variation in on-time and rainfall and borehole substitution, which helps explain the change in analytical outcomes when revisiting the data. The evaluation team reflected on this point in the Round II report: absent randomization and a true baseline, omitted variables likely explain some of the variation in borehole pump on-time, but are not properly accounted for in the available data. The evaluation team sought to address this limitation with the inclusion of the borehole substitutes variable, but this issue remains an important consideration for drawing conclusions for EQ1, as well as the other EQs. Using the weekly model with a binary on-off outcome presents a similarly inconclusive picture. As shown in Figure I-8, the odds of a borehole in a treatment county during the dry season being on at some point during the week relative to a comparison county borehole are statistically even. FIGURE I-6: ROUND II MODEL 1 ANALYSIS FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 102 FIGURE I-7: ROUND II MODEL 1 REANALYSIS FIGURE I-8: ROUND II MODEL 2 WEEKLY ON-OFF ANALYSIS FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 103 I-5 ADDITIONAL MODEL 3 RESULTS Figure I-9 below shows the full estimates for Model 3. The full output provides what are known as hazard ratios. A ratio below 1 suggests decreased odds of a borehole going a month or more without recording less than an hour of on-time, while values above one suggest that these variables are correlated with increased odds of a borehole going a month with less than an hour of on-time. As shown in the figure, receiving Kenya RAPID’s intervention lowers the likelihood of boreholes recording less than an hour of on-time relative to comparison counties, but this estimate is not statistically significant. FIGURE I-9: MODEL 3 HAZARD RATIO RESULTS FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 104 I-6 SENTIMENT ANALYSIS The sentiment analysis referenced in EQ3 can be visualized and is shown in Figure I-10 below. FIGURE I-10: SENTIMENT ANALYSIS FOR FGDS FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 105 ANNEX J: MATCHING PROCESS AND BALANCE The main model used for matching in this evaluation employs an “optimal” matching algorithm, which attempts to minimize the overall distance between propensity scores across the data. This is a way to improve overlap across the entirety of our treatment sample. As shown in Figure J-1, without applying matching, only six comparison boreholes have a propensity score above 0.50 and 47 comparison boreholes have a propensity score of zero. The Round II matching model with the new rainfall and mileage variables results in only one comparison borehole receiving an estimated propensity score of zero. As shown in Figure J-1, the matching process smooths out the distribution of estimated propensity scores in the comparison group, improving the area of overlap between the treatment and comparison scores overall. As noted in the Round I report, matching requires a full set of data with no missing values that can diminish the overall sample size. To address the need for a full dataset without missing observations of key variables, such as travel miles, the evaluation team input missing values by estimating miles using a basic linear regression with all of the other matching variables as covariates. FIGURE J-1: UNMATCHED AND MATCHED SAMPLES WITH NEW MODEL INPUTS As shown in Figure J-2 below, using an optimal matching approach achieved overall balance. Balance here is defined by an absolute standardized mean difference below 0.25, which suggests that the average across treatment and comparison groups is statistically similar for the variables of interest. This balance measure provides a scale-free measure of the average difference between assignment groups, which allows for balance across the set of key variables to be easily compared and interpreted. It is important to note that before matching, the average dry season rainfall across treatment and comparison counties was, on average, very different: average daily rainfall at comparison county boreholes was 0.62 mm, while it was 0.23 mm at the treatment county boreholes, with a standardized mean difference around 1, suggesting little balance across assignment groups for this metric. After the Round II matching approach is applied, the average comparison county borehole dry season rainfall drops to 0.29 mm, with an absolute standardized mean difference of 0.07 for matched boreholes. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 106 FIGURE J- 2: BALANCE ACROSS KEY VARIABLES WITH REVISED MATCHING PARAMETERS As noted below, the nearest neighbor matching approach yielded improved overlap, but somewhat limited gains in overall balance, while the two-to-one matching approach did not meaningfully improve balance across the treatment and comparison boreholes. As shown in Figure J-3, nearest neighbor matching improved overlap, suggesting a similar probability of receiving treatment based on observed confounding factors. However, the sample size under this method is quite small at 13 comparison boreholes and 28 treatment boreholes. The covariate balance for nearest neighbor matching is improved on 10 of 14 variables; however, it is notably worse on the “new miles to county seat” variable. Two-to-one matching yields little improvement in overlap or balance. This is due to the small number of available matches from the overall population of 127 comparison and only 60 treatment boreholes. Two-to-one matching would likely be more effective if the number of boreholes overall were much higher. Instead, this approach serves to simply filter out the boreholes in the comparison group that have the lowest propensity scores. As shown in Figure J-5, however, there is very little difference in overlap, with 120 comparison boreholes and 60 treatment boreholes in this sample. This model improves balance across four variables within the 0.25 threshold, but provides limited additional values compared to the unmatched sample FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 107 FIGURE J-3: NEAREST NEIGHBOR PROPENSITY SCORE DISTRIBUTIONS BEFORE AND AFTER MATCHING FIGURE J-4: NEAREST NEIGHBOR COVARIATE BALANCE ACROSS MATCHED AND UNMATCHED SAMPLES FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 108 FIGURE J-5: TWO-TO-ONE PROPENSITY SCORE DISTRIBUTIONS BEFORE AND AFTER MATCHING FIGURE J- 6: TWO-TO-ONE COVARIATE BALANCE ACROSS MATCHED AND UNMATCHED SAMPLES FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 109 ANNEX K: IMPLEMENTATION FIDELITY To track implementation progress and fidelity, the evaluation team distributed a brief questionnaire to the implementation team, which was completed in April 2020. According to these reports, implementation was similar across all five counties, apart from Isiolo, where only three sensors were installed on strategic boreholes. For the remaining counties, the number of sensors installed ranged from 13 (Turkana) to 22 (Marsabit), and the number of functioning sensors was between 9 and 20. The implementation team reported that roles and responsibilities for borehole repairs had been clarified and defined in all counties and that the dashboard had been developed and was being used by relevant officials. However, implementers reported that dedicated budgets for strategic borehole repairs had not been set aside in any of the Kenya RAPID counties. TABLE K-1: ICT INTERVENTION IMPLEMENTATION SUMMARY County Sensors Installed Sensors Functional Roles Defined? Dashboard developed? Dashboard used? Budget for repairs? Garissa 14 12 Yes Yes Yes No Wajir 20 20 Yes Yes Yes No Turkana 13 9 Yes Yes Yes No Marsabit 22 16 Yes Yes Yes No Isiolo 3 3 Yes Yes Yes No In addition to this questionnaire, the evaluation team received a log of training events and financial activities from SweetSense, as well as available quarterly reports. SweetSense reported that they held a training event with county staff in each of the five Kenya RAPID counties in September 2019. In addition, in Garissa they held a training with the newly formed rural water and sanitation company in March 2020. SweetSense provided the evaluation team with information on funding contributions from United Nations International Children’s Fund (UNICEF) and National Drought Management Authority (NDMA) in support of staff training and borehole repairs in Garissa, Isiolo, and Wajir between May and November of 2019, as well as a donation of materials that NDMA made in Wajir to support borehole repairs in October 2019. The Kenya RAPID team shared quarterly activity reports with the evaluation team, which provided additional insight into some of the challenges encountered throughout the implementation process. The reports highlight the differential rollout and training that took place across the treatment counties, and the fact that the number of sensors operating varies over time due to breakages, theft, or disconnection, as well as new installations and replacements. The reports also provide some insight into implementation fidelity. By the end of September 2019, Kenya RAPID reported the installation of the data dashboard on 579 devices.46 However, the digital system was not the only way relevant authorities accessed information; quarterly reports highlight how “SweetSense shared weekly reports via email with 75 selected county and NDMA staff” (ibid) to provide borehole on-time information. 46 See “Kenya RAPID Program Progress Report for the period between July 1 and September 30, 2019 (Qtr.4-FY2019)” FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 110 ANNEX L: ROUND 1 AND ROUND 3 COMPARISON TABLE L-1: SUMMARY OF QUALITATIVE THEMES, FINDINGS, AND CONCLUSIONS Round 1 Theme Round 1 Conclusions Round III Conclusions Borehole Access and Use ● Users face significant challenges to borehole use across all areas. ● Kenya RAPID intervention may affect some of these (e.g., reduced breakages) but is unlikely to affect others (e.g., excessive demand). ● Providing sufficient water supply to meet demand remains challenging. Staff Roles and Management ● Variations in water committee quality and effectiveness are likely to affect outcomes (functionality, runtime). ● Intervention will be most effective if it succeeds in building capacity and clarifying roles across levels (i.e., from county level down to local operators). ● Heterogeneity in management structures and unclear roles persist in many cases. ● Garissa’s move toward more centralized management of strategic boreholes through GARWASCO could help clarify roles, but continued engagement of local communities will be needed. Resources for Borehole Repairs ● There is a strong need for dedicated budgets and clearer processes for accessing resources for borehole repairs. ● Similarities in concerns across RAPID and comparison counties indicate that intervention has not yet fixed this problem. ● Information and communications technology (ICT) intervention has not resulted in dedicated budgets for borehole repairs with clear mechanisms for accessing funds. ● Local user fees are not sufficient to maintain boreholes; roles of county and national governments and NGOs in sustaining funds need to be clarified. Water System Functionality ● Concerns about functionality are widespread; demand for systems that address these problems is high. ● Functionality concerns persist at all levels, showing continued need for investment in system improvements. Information Sharing Systems ● Intervention will not achieve its intended impacts unless users are willing and able to access data through the dashboard. Evaluation team needs to expand its data collection on implementation progress across Kenya RAPID counties. ● Need exists for better outreach and integration of local operators in sensor-based intervention. ● ICT intervention viewed favorably by water managers, who see data as relevant and useful. Implementation should continue to ensure full impacts of system are realized over time. ● Some local borehole operators are aware of ICT systems while others are not. Continued outreach to local operators and users encouraged to support buy-in and engagement. FINAL REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 111 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524