This publication was produced for review by the United States Agency for International Development. It was prepared by Tetra Tech. PHOTO CREDIT: JACOB PATTERSON-STEIN ROUND I1 EVALUATION REPORT: IMPACT EVALUATION OF THE KENYA RESILIENT ARID LANDS PARTNERSHIP FOR INTEGRATED DEVELOPMENT ACTIVITY 12 MAY 2020 ACKNOWLEDGEMENTS The evaluation team thanks the families, water managers, and local leaders in the counties who shared their time to participate in this study. The National Drought Management Authority and the Kenyan Ministry of Water and Sanitation both deserve praise for their assistance, cooperation, and patience with the evaluation process and team members. Special thanks go to the data collection teams from Ipsos Public Affairs Kenya for conducting the water borehole surveys and downloading data from each borehole. The team is grateful to the Kenya Resilient Arid Lands Partnership for Integrated Development Activity staff for their continued collaboration, cooperation, and communication. The team could not have implemented the evaluation without the ongoing and patient cooperation of SweetSense staff, particularly Evan Thomas, Pranav Chintalapati, and Styvers Kathuni. The team also acknowledges Mary Gichuhi for her quality oversight during data collection, translation support, and general guidance as the local coordinator for this evaluation. Finally, the evaluation team appreciates the technical review and support for this report from Tetra Tech’s Siobhan Girling and Morris Israel. The team also thanks Elizabeth Jordan (USAID/E3/Water), Vicky Liyai (USAID/Kenya), Martin Mulongo (USAID/Kenya), Walter Ong’eng’a (USAID/Kenya), Amanda Robinson (USAID/Kenya), Albert Waudo (USAID/Kenya), and Nicholas Owuor (USAID/Kenya) for their continued support of this study. Prepared by: Dr. Katherine Dickinson (Principal Investigator, University of Colorado) Jacob Patterson-Stein (Evaluation Coordinator, MSI, A Tetra Tech Company) Cover Photo: Camels drink from strategic borehole in Kenya, Jacob Patterson-Stein, MSI. Prepared for the United States Agency for International Development by the Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) project under Task Order number AID-OAA-TO-16-00016 of the Water and Development Indefinite Delivery Indefinite Quantity Contract (WADI IDIQ), contract number AID-OAA-I-14-00068. Tetra Tech Contacts: Morris Israel, Project Director morris.israel@washpals.org Jeff Albert, Deputy Project Director jeff.albert@washpals.org Lucia Henry, Project Manager lucia.henry@tetratech.com Tetra Tech 1320 North Courthouse Road, Suite 600, Arlington, VA 22201 Tel: 703 387-2100, Fax: 703-414-5593 www.tetratech.com/intdev ROUND 1I EVALUATION REPORT IMPACT EVALUATION OF THE KENYA RESILIENT ARID LANDS PARTNERSHIP FOR INTEGRATED DEVELOPMENT ACTIVITY May 12, 2020 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ROUND I1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY i CONTENTS ACKNOWLEDGEMENTS ......................................................................................................................................... III ACRONYMS.................................................................................................................................................................. III EXECUTIVE SUMMARY............................................................................................................................................. IV 1 INTRODUCTION........................................................................................................................................... 1 2 KENYA RAPID ACTIVITY BACKGROUND ........................................................................................... 2 2.1 NATIONAL CONTEXT..................................................................................................................... 2 2.2 KENYA RAPID ACTIVITY DESCRIPTION .................................................................................... 3 2.3 DEVELOPMENT HYPOTHESIS AND THEORY OF CHANGE............................................... 4 2.4 IMPLEMENTATION FIDELITY .......................................................................................................... 5 3 EVALUATION BACKGROUND PURPOSE, AUDIENCES, AND USES........................................... 7 3.1 PURPOSE ................................................................................................................................................ 7 3.2 AUDIENCE ............................................................................................................................................. 7 3.3 INTENDED USE.................................................................................................................................... 7 4 EVALUATION DESIGN................................................................................................................................. 8 4.1 EVALUATION QUESTIONS ............................................................................................................. 8 5 DATA COLLECTION .................................................................................................................................. 12 5.1 ROUND I RECAP ............................................................................................................................... 12 5.2 INTERIM PERIOD............................................................................................................................... 12 5.3 ROUND II DATA COLLECTION.................................................................................................. 13 5.3.1 COMPARISON BOREHOLE ATTRITION..................................................................... 13 5.3.2 COMPARISON BOREHOLE SENSOR DATA............................................................... 16 5.3.3 TREATMENT BOREHOLE SENSOR DATA .................................................................. 16 6 FINDINGS ........................................................................................................................................................ 18 6.1 ROUND I RECAP............................................................................................................................... 18 6.2 ROUND II OVERVIEW..................................................................................................................... 19 6.3 LIMITED BOREHOLE ASSET SURVEY ......................................................................................... 19 6.3.1 BOREHOLE USE.................................................................................................................... 19 6.3.2 BOREHOLE MANAGEMENT............................................................................................. 22 6.3.3 CONCLUSIONS.................................................................................................................... 27 6.4 BOREHOLE SENSOR DATA DESCRIPTIVE ANALYSIS .......................................................... 28 7 PROPENSITY SCORE MATCHING.......................................................................................................... 34 7.1 OVERVIEW........................................................................................................................................... 34 7.2 ROUND I RECAP............................................................................................................................... 34 7.2.1 ROUND I METHOD ............................................................................................................ 34 7.2.2 ROUND I BALANCE ........................................................................................................... 35 7.3 ROUND II MATCHING ................................................................................................................... 36 7.3.1 ADJUSTMENTS FROM ROUND I.................................................................................... 36 7.3.2 ROUND II METHOD........................................................................................................... 36 7.3.3 ROUND II BALANCE .......................................................................................................... 37 8 INITIAL BOREHOLE FUNCTIONALITY COMPARISONS ............................................................... 38 ROUND I1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY ii 8.1 MODEL DEFINITIONS ..................................................................................................................... 38 8.2 PRIMARY OUTCOME MEASURE .................................................................................................. 39 8.3 INFERENTIAL ANALYSIS................................................................................................................. 43 9 CONCLUSION .............................................................................................................................................. 45 9.1 SUMMARY OF CONCLUSIONS.................................................................................................... 45 9.2 NEXT STEPS ........................................................................................................................................ 45 10 ANNEXES ........................................................................................................................................................ 47 ANNEX A: EVALUATION STATEMENT OF WORK...................................................................................... 48 ANNEX B: COMPARISON COUNTY FOLLOW-UP BOREHOLE ASSET SURVEY INSTRUMENT.. 56 ANNEX C: REFERENCES ......................................................................................................................................... 68 ANNEX D: RAINFALL............................................................................................................................................... 70 ANNEX E: SUMMARY STATISTICS ...................................................................................................................... 71 ANNEX F: MATCHING REVISITED ...................................................................................................................... 73 ANNEX G: REGRESSION RESULTS...................................................................................................................... 76 ANNEX H: IMPLEMENTATION FIDELITY MONITORING QUESTIONNAIRE...................................... 79 ROUND I1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY iii ACRONYMS ADS Automated Directives System (USAID) ASAL Arid and Semi-Arid Land CAPI Computer-Assisted Personalized Interview CHIRPS Climate Hazards Group InfraRed Precipitation with Station E3 Bureau for Economic Growth, Education, and Environment (USAID) EDE Ending Drought Emergency EQ Evaluation Question GD Group Discussion GPS Global Positioning Software HH Household ICT Information and Communication Technology IE Impact Evaluation KEMRI Kenya Medical Research Institute KII Key Informant Interview LOWASCO Lodwar Water and Sanitation Company MCA Member of County Assembly MWA Millennium Water Alliance NACOSTI National Commission for Science, Technology, and Innovation NDMA National Drought Management Authority NDVI Normalized Difference Vegetation Index NGO Nongovernmental Organization PSM Propensity Score Matching RAPID Resilient Arid Lands Partnership for Integrated Development SDC Swiss Development Corporation SDG Sustainable Development Goal SO Strategic Objective SOW Statement of Work WAMSP Water Management as a Service Platform WASCO Water and Sanitation Company WASH Water, Sanitation, and Hygiene WASHPaLS Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability USAID United States Agency for International Development ROUND I1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY iv EXECUTIVE SUMMARY This document reports on the findings of Round I1 data collection for Kenya Resilient Arid Lands Partnership for Integrated Development’s (Kenya RAPID’s) impact evaluation. Kenya RAPID is a five￾year activity implemented under a Global Development Alliance (USAID agreement number AID-615-A￾15-00008) and is co-funded by the United States Agency for International Development (USAID), the Swiss Development Corporation, private sector partners, and the Millennium Water Alliance and its sub-recipients. Kenya RAPID aims to contribute to sustainable and resilient livelihoods for communities in Kenya’s arid and semi-arid lands by improving water availability and water service delivery to people and livestock and by improving rangelands in those regions. This report provides preliminary findings after 12 months of data collection on borehole pump functionality to gain initial insight into whether the use of real-time remote sensing increases pump on-time1 1 On-time is defined as the time during which the borehole pump is actively pumping. during the drought season. One component of this activity involves the use of information and communication technology (ICT) tools to improve water service delivery. Specifically, the evaluation team installed sensors on approximately 400 boreholes in the five Kenya RAPID counties to measure flow and water extraction rates and to detect system failures. Of these sensors, 69 were in boreholes characterized as “strategic” by local authorities due to the risk of drought. For these boreholes, the ICT intervention also involved developing a response operations and maintenance team in each county and reaching an agreement with the county governments to provide a dedicated budget for strategic boreholes. These strategic boreholes are the focus of the impact evaluation. Specifically, this Round II report presents quantitative results comparing borehole on-time between treatment (Kenya RAPID counties) and comparison (non￾Kenya RAPID counties) groups during the 2019 drought season. The evaluation team designed a quasi-experimental matching study to rigorously test the effects of the ICT intervention on borehole on-time and management decisions during the drought season. For this evaluation, the team identified a set of eight comparison counties that are not part of the Kenya RAPID activity, and identified 132 strategic boreholes in these counties. In Round I, the team installed sensors on these comparison county boreholes and conducted a borehole asset survey across all intervention and comparison boreholes to provide data on borehole characteristics (e.g., pump type, power source, number of households and livestock served, and presence of tariff schemes). The evaluation team developed and applied a set of matching models to estimate the effects of the Kenya RAPID intervention on borehole functionality. These models use data on borehole characteristics to match Kenya RAPID boreholes with one or more comparison boreholes that are as similar as possible, and then estimates differences in borehole on-time across Kenya RAPID boreholes and matched controls. The Round I report presented an initial set of matching models; based on feedback, we refined these models in Round II using higher resolution data on rainfall (which is hypothesized to be strongly related to borehole use) and an alternative definition of remoteness. These refinements result in better overlap across assignment groups, improving the ability of the comparison group to serve as a convincing counterfactual for the Kenya RAPID counties. Across multiple matching models, regression results consistently indicate that on-time was higher for Kenya RAPID strategic boreholes than for matched comparison boreholes during the 2019 drought season, controlling for a set of borehole characteristics and county fixed effects. These differences amount to roughly one to six hours of additional on-time each day. While a portion of this difference may be attributable to the Kenya RAPID intervention, the evaluation team is cautious about overinterpreting these results at this stage for at least two reasons. First, while many important observable differences between intervention and comparison boreholes are accounted for using ROUND I1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY v available data, we remain concerned about potential unobserved factors that may differ across counties and be correlated with borehole on-time. For instance, if there are fewer alternative water sources in arid Kenya RAPID counties, users may be more reliant on boreholes in these areas. This factor is not well captured in our current dataset; the team is exploring possible ways to account for such factors in Round III data analysis. Second, at this stage the evaluation team has incomplete data on Kenya RAPID’s ICT intervention status, but the data that are available indicate that implementation is incomplete in several counties. An intervention that has not yet occurred cannot plausibly influence outcomes. Thus, the evaluation team seeks to gather better data on implementation progress and conduct additional analyses (e.g., looking at a subset of counties where implementation is farther along) to gain a more nuanced understanding of observed effects and to aid interpretation. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 1 1 INTRODUCTION This Round II data collection report is part of the impact evaluation of the Kenya Resilient Arid Lands Partnership for Integrated Development (Kenya RAPID) Activity commissioned by the United States Agency for International Development’s (USAID’s) Mission in Kenya and the Office of Water in USAID’s Bureau for Economic Growth, Education, and Environment (E3). USAID’s E3 Analytics and Evaluation Project designed the evaluation, and the Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability Project (WASHPaLS) is implementing it. The evaluation uses a quasi-experimental matching design to rigorously test how remote sensing technology and information sharing affect water borehole pump on-time and management. Annex A provides USAID’s statement of work (SOW) for the evaluation. This report provides a summary of the Kenya RAPID context and background, followed by an overview of the evaluation design, including details about the Round II data collection process. To provide additional context, this report includes a summary of the Round I results before presenting findings from Round I1 data collection. Round II included a limited version of the borehole asset survey from Round I, a review of data from intervention borehole sensors, and downloading data from comparison borehole pump sensors installed during Round I. The report revisits the matching methods used in Round I based on feedback provided since report completion. Finally, the report presents initial inferential borehole functionality estimates. Unlike Round I, Round II collected no qualitative data. The findings from this report are one step in the overall evaluation process. They should not be taken as definitive evidence about the “impact” of Kenya RAPID. Rather, the goal of this report is to highlight initial trends in borehole functionality, as well as to describe preliminary differences between treatment and comparison borehole pump functionality during the first year of measurement. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 2 2 KENYA RAPID ACTIVITY BACKGROUND 2.1 NATIONAL CONTEXT Water supply coverage in Kenya increased from 33 percent in 1990 to 57 percent in 2015 (Organization, Supply, & Programme, 2015).2 2 Defined as “piped” or “other improved” water sources. However, reliable and sustained water service delivery in rural areas remains a challenge, particularly in drought-prone areas. Problems maintaining the water pumps and boreholes result from social, logistical, and technical issues like the breakdown of community management structures, insufficient human resources to provide services and repairs, and lack of spare parts (Harvey & Reed, 2007). In addition, there is a lack of reliable and regular information for monitoring and increasing maintenance provider responsiveness. Decisions about service provision may depend on having accurate and timely information as well as on political, social, and economic pressures that may influence decision-making within any given local environment. Politically, Kenya is going through a period of major institutional reform including the devolution of authority and resources from the national government to newly elected county governments. County governments now have the political mandate and financial resources to provide water to their communities; however, they are new institutions with limited operational capacity. As part of these developments, the Government of Kenya launched its “Common Programme Framework to End Drought Emergencies,” which arose from a series of meetings with development partners between 2013 and 2014.3 3 See: http://www.ndma.go.ke/index.php/resource-center/ede-reports/send/43-ending-drought-emergencies/4251- common-programme-framework As shown in Figure 1, the institutional framework for water management in Kenya consists of multiple stakeholders, with local counties operating at the regional and local levels. The Common Programme includes the Ending Drought Emergencies (EDE) initiative to better align stakeholders involved in drought mitigation and water management across all levels of government. The EDE initiative is a framework to improve targeting and coordination with the goal of promoting drought reduction, early warning and response, and institutional capacity for climate resilience (Kenya, 2014). ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 3 FIGURE 1. INSTITUTIONAL ROLES IN THE WATER AND SANITATION SECTORS (World Bank, 2011) Challenges to the provision of sustainable and reliable water service, coupled with a changing institutional environment and scarce water resources in Kenya, reinforce the need for stronger and more accountable institutions, enhanced coordination and integration of development programs across sectors, private sector participation, and empowered communities with the knowledge and ability to exercise rights and responsibilities regarding water resources. Kenya RAPID aims to tackle these challenges. 2.2 KENYA RAPID ACTIVITY DESCRIPTION Kenya RAPID is a five-year activity implemented under a Global Development Alliance (USAID agreement number AID-615-A-15-00008) that is co-funded by USAID, the Swiss Development Corporation (SDC), private-sector partners, and the Millennium Water Alliance (MWA) and sub￾recipients.4 4 A total of $35.5 million will be invested through Kenya RAPID: $12.5 million from USAID; $12.5 million in leveraged funds from private sector partners; $7.5 million from SDC; and $3 million in cost share from MWA and its sub-recipients. USAID awarded the activity in 2015 to MWA, a consortium of non-profit water-related organizations, to build on the successes and lessons learned from USAID’s Kenya Arid Lands Disaster ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 4 Risk Reduction – Water, Sanitation and Hygiene (WASH) Program and SDC’s Water for Livestock Program. Both programs were implemented in northern Kenya from 2012 to 2014.5 5 The MWA members for Kenya RAPID include CARE, Catholic Relief Services, Food for the Hungry, and World Vision. SweetSense and IBM Research are private sector sub-recipients under this award. Other private sector partners include the Coca Cola Foundation, Acacia Water, and KCB Foundation. Kenya RAPID aims to contribute to sustainable and resilient livelihoods for communities in Kenya’s arid and semi-arid lands (ASALs) by improving water availability and water service delivery to people and livestock and by improving rangelands in those regions. Three strategic objectives (SOs) guide the activity toward the overall goal of sustainable and resilient livelihoods for communities in the ASALs: • SO 1: A responsive and accountable governance framework is in place and operational at the county government level that ensures sustainable provision of water and pasture; • SO 2: Replicable and scalable business models for sustainable water, sanitation, and hygiene (WASH) and livestock service delivery have been developed and operationalized; and • SO 3: Communities have increased access to sustainable WASH services and improved rangeland management. Under these SOs, Kenya RAPID is committed to making data and information and communication technology (ICT) tools available and accessible to improve decision-making for better water service delivery. The activity will install approximately 400 sensors to measure flow rates and water extraction rates and detect system failures on water boreholes. Of these 400 sensors, 69 are in areas identified as strategic by local authorities due to the risk of drought in those areas and the subsequent importance of the water boreholes. The activity developed customized data dashboards for each county to display water borehole status in near-real time. This is possible by facilitating county coordination units, wherein local county officials lead activity implementation with support from RAPID partners, such as SweetSense, which developed remote sensor technology to improve service delivery in multiple countries, and IBM, which developed the data dashboards. SweetSense processes the sensor data, complements it with near-real-time survey information obtained via mobile phone surveys when notable changes to operations are identified, and uses this information to make inferences about causes for usage disruptions and changes. Kenya RAPID makes these data accessible to relevant authorities such as county governments and the appropriate service providers. The sensor data is intended to feed into and inform other core pieces of Kenya RAPID’s support for management processes, specifically the development of operations and management teams in each county with clear roles and responsibilities, and budget support for strategic boreholes, to promote the goal of improving water service delivery. County and sub-county officials will, in theory, be able to use the sensor data to improve their management and deployment of staff and resources—areas that are receiving support through other Kenya RAPID interventions. 2.3 DEVELOPMENT HYPOTHESIS AND THEORY OF CHANGE The theory of change envisions that if the Kenya RAPID activity, 1) installs sensors on strategic boreholes, shares the data through mobile applications and online dashboards, and provides training on sensor data use; 2) supports the development of county operations and maintenance teams; and 3) facilitates a dedicated budget for strategic borehole repairs, this will lead to increased strategic borehole functionality, including more borehole pump on-time during critical drought periods and reduced drought impacts on ASAL communities. Kenya RAPID components are intended to work together to promote improved strategic borehole management by addressing key information and resource constraints. Figure 2 illustrates the causal linkages relevant to this evaluation that USAID envisions for ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 5 translating results under each of the activities into the Kenya RAPID sensor intervention’s intended outcomes. The Round I report provides more detail on the assumptions and reasoning behind this theory of change. FIGURE 2. THEORY OF CHANGE FORTHE KENYA RAPID ACTIVITY’S REMOTE SENSOR INTERVENTION 2.4 IMPLEMENTATION FIDELITY A key take-away from the Round I report and discussion with USAID was the need for implementation fidelity monitoring in the context of the impact evaluation (IE). Given Kenya RAPID’s implementation approach, wherein a consortium of partners provide non-uniform services across the intervention counties, it is critical to the IE’s analysis to account for variation in activities. The absence of this data undermines a key assumption of the evaluation design: that the matched boreholes are statistically similar on average after accounting for observable metrics and thus any change in borehole pump on￾time is a result of the remote sensor intervention. Tracking the progress of the sensor intervention, associated dashboard, and other Kenya RAPID activities are critical for understanding confounding factors that may affect the IE team’s analysis, relevant for both the quantitative analysis of sensor data and sample selection for qualitative analysis. First, the evaluation assumes that the intervention is rolled out as per the design and in a timely manner. Lack of clear implementation data also greatly limits the IE team’s ability to interpret quantitative results. At the extreme, if no activities are implemented in a particular county but quantitative results still show a large “treatment effect,” this would be an important signal that matching models have not fully accounted for unobservable differences across counties. In addition, we know that other observable activities that are not accounted for in our analysis are likely to influence the evaluation outcomes. Specifically, activities by Kenya RAPID partners outside of the sensor intervention have the potential to impact borehole management and pump functionality (EQ1), and perceptions around water access and management (EQ 2 & 3). For example, if Kenya RAPID partners support a complaint system in Wajir or the adoption of water related policies in Turkana, this would be relevant to the IE’s analysis. Without data on the delivery of supplementary support activities within the broader Kenya RAPID approach, impact estimates will likely overstate the effect of the ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 6 sensor and information sharing activities. Without access to Kenya RAPID’s formal reporting, it is difficult to know what services the activity offered, in which counties, and when, to control for this during analysis and isolate the effect of the sensor intervention. To obtain the necessary data, the evaluation team requested access to the Kenya RAPID quarterly and annual reports, which are not available publicly through the USAID Development Experience Clearinghouse (DEC). Quarterly reports have not yet been shared with the evaluation team and it remains unclear whether the reports contain the needed context in a format that is easily integrated into the evaluation at this late stage. The evaluation team also sought to hold quarterly check-in calls with Kenya RAPID. These calls provided important context and indicated that some delays or adjustments to implementation occurred, which we can expect to influence results. However, these verbal check-ins were limited in their ability to provide detailed and systematic data that can be incorporated into quantitative analysis. The evaluation team developed an implementation questionnaire (provided in Annex H) that identifies the specific information needed, including the status of intervention sensors and the data dashboard, clarification of roles and budget for borehole maintenance, and supplemental activities. The evaluation team shared the questionnaire with MWA in October 2019 and requested that a questionnaire be completed for each county on a quarterly basis. The evaluation team received questionnaires for each Kenya RAPID county on April 15, which provides an update on the implementation status. The evaluation team will continue working with Kenya RAPID’s COP to obtain quarterly and annual reports in order to understand how implementation changes over time. Once analyzed, if the above sources do not provide insight into aspects of sustainability plans, the evaluation team will also seek this information. Given that the Kenya RAPID program will be coming to an end, possibly before the evaluation is completed at the end of the next drought season, information on whether and how components of the intervention will continue after the funding period ends, and how this may vary across counties will be particularly important for interpreting quantitative results on borehole functionality. The evaluation team will continue to coordinate with USAID and Kenya RAPID to incorporate detailed implementation information into the Round III analysis. The evaluation team will move forward with selecting sites for Round III qualitative data collection based on borehole functionality and precipitation data, as well as the initial implementation fidelity summaries received on April 15. Given that key details related to the local borehole context will still be critical for analyzing qualitative data, the evaluation team will select boreholes, then share these selections with Kenya RAPID to learn what, if any, additional activities the activity implemented in the selected counties, sub-counties, and at the borehole sites. This will require input from Kenya RAPID to ensure timely selection of borehole sites for Round III qualitative data collection. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 7 3 EVALUATION BACKGROUND PURPOSE, AUDIENCES, AND USES This evaluation comes at an opportune time, when the Sustainable Development Goals (SDGs) include not only the provision of safe and affordable drinking water but also support for and strengthening of local community participation in improving water management (SDG 6). As investments shift toward sustainable water provision, drought risk management, and service quality, innovative tools with the potential to improve service delivery, managerial decision making, and efficient use and allocation of resources need to be evaluated to determine which are appropriate and how to bring them to scale. 3.1 PURPOSE This evaluation will help USAID understand the effectiveness of real-time remote sensing of the functionality of water points during the drought season to improve decision-making for better water service delivery and drought risk management. The results of this evaluation will be made widely available to encourage replication or scaling-up of interventions and analytical activities within and beyond Kenya, as applicable. As such, this evaluation applies USAID’s Evaluation Policy guidance with respect to using the most rigorous methods possible to demonstrate accountability for achieving results. The evaluation is also designed to capture practical lessons from USAID/Kenya’s experience to increase sustainability in WASH programs and investment in water resource management systems, specifically in strategic drought areas. 3.2 AUDIENCE The evaluation is aimed at several audiences. First, the evaluation’s findings are expected to be valuable to USAID/Kenya and the USAID/E3 Office of Water, so they can better understand whether decision￾making utilizing data from real-time remote sensing can lead to improved borehole functionality. Second, findings and lessons learned from this evaluation will be interesting to MWA, its partners, and other practitioners in the water sector, including the Government of Kenya, which is seeking to improve water resource management, drought risk management, water coverage, and quality of services. Finally, for donors, implementers, and scholars, the evaluation will make an important contribution to the empirical evidence base on water service delivery and information interventions in drought-prone and at-risk areas. 3.3 INTENDED USE Results from this evaluation will be used to determine whether additional investments should be made on ICT tools for improved borehole functionality in Kenya or beyond. The evaluation’s findings will also inform the design of future USAID programming targeting the sustainability of water service delivery to increase resilience and livelihoods for communities. In addition, the evaluation will add to a growing body of evidence about drought risk management, to which the evaluations and studies conducted by USAID and other institutions also contribute. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 8 4 EVALUATION DESIGN 4.1 EVALUATION QUESTIONS The Kenya RAPID evaluation addresses three questions derived from the theory of change. The evaluation team developed and finalized these evaluation questions (EQs) in collaboration with USAID. 1. Does the intervention using real-time remote sensing data of water points for strategic borehole management in Kenya RAPID counties lead to increased on-time of strategic boreholes during the drought season? 2. How do water managers perceive the impact of sensor-based systems on their ability to address borehole functionality and how does this compare to perceptions of borehole functionality in non￾Kenya RAPID counties? 3. Do Kenya RAPID’s sensor-based systems affect user perceptions of borehole functionality and access? This report provides a preliminary assessment to address EQ1 only, using sensor data to determine borehole pump on-time. The final Round III report will include a more comprehensive analysis addressing EQ1, as well as qualitative data collection to respond to EQs 2 and 3. To answer the evaluation questions, the team designed a quasi-experimental, mixed-methods evaluation focused on boreholes that are strategic for mitigating drought risks. The team is implementing a quantitative quasi-experimental evaluation approach to answer the first question and qualitative data collection and analysis to address the second and third questions. The evaluation design involves two nested units of analysis. Strategic boreholes (and the communities they serve) are the primary units and are nested within counties. To select comparison counties, the evaluation team worked with USAID to identify eight ASAL counties (Table 1) that are nominally comparable to the Kenya RAPID counties based on general information regarding other USAID activities, aridity, and security. This was a purposive process based on USAID staff experience and knowledge of county-level characteristics and use of verifiable county information. After identifying the eight counties, the evaluation team worked with the National Drought Management Authority (NDMA) and each county’s water officer to generate lists of the strategic boreholes within each county. In several counties, there were no clear criteria for designating boreholes as “strategic” and local authorities FIGURE 3. TREATMENT AND COMPARISON COUNTIES ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 9 appeared unsure whether they should follow specific guidelines. Kenya RAPID staff communicated their challenges in obtaining a consistent roster of strategic boreholes in intervention counties, and the evaluation team had a similar experience.6 6 Although outside the scope of the evaluation, a full comparison between borehole characteristics across strategic and non-strategic boreholes in the RAPID counties may provide more insight into determinants of the “strategic” label. TABLE 1. COMPARISON COUNTIES County Arid/Semi-Arid Boreholes as a percentage of total water sources in January 2018 Baringo Arid 9% Kitui Semi-Arid 28% Laikipia Semi-Arid 29.1% Mandera Arid 15% Meru Semi-Arid 43.8% Samburu Arid 25.6% Tana River Arid 14.3% West Pokot Semi-Arid 19.6% (NDMA, n.d.) In contrast, Table 2 shows the characteristics of treatment boreholes, those included in the Kenya RAPID intervention. This table highlights the purposive nature of county selection for implementation— all the Kenya RAPID counties are arid, with a relatively high level of borehole use. TABLE 1.TREATMENT COUNTIES County Arid/Semi-Arid Boreholes as a percentage of total water sources in January 2018 Garissa Arid 31.7% Isiolo Arid 28.6% Marsabit Arid 48% Turkana Arid 30% Wajir Arid 33.3% (NDMA, n.d.) ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 10 County representatives selected a total of 132 boreholes across the eight comparison counties for the entirety of the evaluation. The evaluation’s quasi-experimental design then attempts to identify a set of boreholes that are as similar as possible to the treatment group such that we would expect outcomes (in this case, borehole functionality) to be similar in the absence of treatment. If this is achieved, the comparison group’s outcomes can serve as a proxy for the treatment group’s counterfactual, allowing the team to estimate the specific effect of the ICT intervention. To assess similarity, the evaluation team collected data on the following observable borehole characteristics that are expected to affect functionality during droughts: • Rainfall • Remoteness/distance from Nairobi and the county seat • Type of pump • Populations served (human and livestock) • History of breakages/repair times The research team measured each of these variables using the borehole asset survey (described below) and other sources (e.g., meteorological databases). As noted in the Round I report, this survey was based on a similar survey that Kenya RAPID partners conducted on the intervention boreholes. Round I used the borehole asset survey data to “match” comparison boreholes to treatment county boreholes. The nature of the Kenya RAPID activity meant that assigning strategic boreholes to the “treatment” group that received the sensor-based intervention was not random. This intervention included the five Kenya RAPID counties because of their specific characteristics: namely, their arid to semi-arid climates and associated challenges in reliable water access, particularly during the drought season. Given this reality, the team designed the impact evaluation to select a comparable set of boreholes with similar observable characteristics and controlled for these characteristics through a matching algorithm. The identifying assumption for this design is that, conditional on these observable characteristics, we would expect similar functionality outcomes across boreholes in Kenya RAPID and non-RAPID counties in the absence of the intervention. The goal of Round II and Round III of the evaluation is to compare sensor data from across treatment and comparison borehole pumps to assess how Kenya RAPID’s suite of information sharing, roles and responsibilities clarification, and budget facilitation affects borehole pump on-time. The percentage of the day that a borehole pump is turned on, a measure we call “percent on,” is the main outcome variable for this evaluation. This provides the percentage of on-time, defined as time within a 24-hour period that the meters recorded the borehole pump as running; for example, a value of 50 percent would indicate the borehole pump ran for 12 hours (i.e. 0.5 x 24 hours). Throughout the report, we use the terms “percent-on” and “on-time” interchangeably. The main measure of on-time does not capture direct use. It is conceivable that a borehole pump could be turned on to fill a tank, turned off once the tank is full, and users simply draw water from the tank while the pump is dormant. The IE data captured the time that water was pumped, but not accessed from the tank. As noted below, the borehole sensors provide data for treatment counties on a daily basis that is aggregated up to this level for the comparison counties. This means that we have a measure that captures the overall running of the pump to meet immediate demand or fill a tank. If a tank is filled and then the pump breaks, the IE design cannot capture the fact that users in the area would still have access to water, but can capture the lack of future pumping. It is worth keeping in mind that pump functionality is different from water access, but for the sampled boreholes, water access is reliant on borehole pumps, even if there is a lag in pumping and use. A key role of qualitative data in this evaluation is to better understand water user perspectives, as well as borehole management practices. This will ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 11 enhance the quantitative analysis at Round III by providing additional context to the distinction between use and on-time. In addition, Round III will assess perspectives among managers and users across the two assignment groups through qualitative data analysis. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 12 5 DATA COLLECTION 5.1 ROUND I RECAP As noted in the Round I report, this impact evaluation relies on data from three main primary sources: 1) sensor data on borehole functionality; 2) a borehole asset survey on borehole characteristics; and 3) qualitative data on water managers’ and water users’ experiences related to borehole access and functionality.7 7 Note: only motor-powered boreholes are included in the evaluation since the sensors use the electrical current to track functionality. The evaluation also uses secondary data sources, such as meteorological variables and travel distance. During August and September 2018, for Round I, the evaluation team performed a borehole asset survey on comparison boreholes to collect observable data on borehole characteristics and conducted interviews with borehole managers and sub-county and county officials in both comparison and treatment counties to collect information on the borehole context. The evaluation team also held focus group discussions with water users at eight boreholes across four counties (two intervention and two comparison counties) to understand perceptions around water access and management. In addition, while conducting the borehole asset survey on comparison boreholes, the evaluation team installed sensors to collect near-continuous data on borehole pump on-time for the duration of the evaluation. Per the evaluation design, the Kenya RAPID implementation team is responsible for providing similar data for the treatment boreholes. Kenya RAPID partner SweetSense collected observable data on borehole characteristics as part of its implementation strategy from roughly April 2017 through September 2018, collecting verifiable information, such as the borehole power source, as well as broader contextual information, including the number of households served and the presence of tariffs. The evaluation team adapted the intervention survey for use in comparison counties, to ensure consistency across treatment and comparison boreholes. The evaluation team selected and installed sensors on comparison boreholes to parallel intervention borehole sensors installed during the course of implementation. The comparison sensors are produced by Dent Instruments, can store up to 32,576 on/off transitions, data is easily accessed through a USB connection, and has a battery life of five years.8 8 More formally, the comparison sensor is known as the Dent Instruments TOUCT-4G CT Data Logger. While the intervention borehole sensors transmit borehole pump on-time remotely, the sensors used in comparison counties require a laptop connection for in-person downloading. 5.2 INTERIM PERIOD Conversations with the Kenya RAPID implementation team (SweetSense and MWA) since Round I suggest that conditions at the intervention borehole sites are often dynamic, with boreholes moving (that is, being closed and re-drilled in a different location using some of the same equipment), power sources changing, and sensors being removed. Because the intervention sensors transmit data, this team is able to identify outages in borehole pump functionality immediately and quickly followup with local contacts to determine whether this is the result of local users or a changing borehole context. In July 2019, nearly one year after installation, the evaluation team attempted to contact 11 randomly selected comparison borehole managers to determine if the borehole sensors were still installed and whether ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 13 any major changes to the borehole had occurred. The team successfully contacted ten borehole managers and reported that the comparison sensors remained installed and functioning. 5.3 ROUND II DATA COLLECTION Borehole sensors provide the main source of data for Round II. The evaluation team, through local subcontractor, Ipsos Public Affairs, revisited each of the 132 comparison boreholes to download sensor data and conduct a limited survey of borehole characteristics between September 27th and October 11th, 2019. Round II employed the limited borehole survey to determine what, if any, changes occurred across key metrics. It includes a subset of questions from the survey utilized in Round I and can be found in Annex B: Comparison County Follow-up Borehole Asset Survey Instrument. The goal of the follow￾up survey in the comparison counties was to track key changes in the borehole context that might influence borehole pump functionality. In contrast to Kenya RAPID, the evaluation team is not actively monitoring the boreholes, so this survey serves as a way to check on the status of each borehole in the comparison group beyond simply downloading the data. Kenya RAPID partner SweetSense provided data from intervention borehole sensors from April 2017 through December 2019. The Round II recording period for comparison boreholes sensors is August 2018 through October 2019. 5.3.1 COMPARISON BOREHOLE ATTRITION The comparison sensor data cover a period from August 8, 2018 through October 10, 2019, with 114 out of 132 successful downloads at Round II and 115 sensors left intact to record for the Round III period. Data collection aligned with the installation of sensors and implementation of the borehole asset survey at Round I in 2018. As shown in Table 3 below, when revisited at Round II, 116 (88%) comparison boreholes still had sensors properly installed, while 10 (7%) had a sensor on site, but not installed. In six cases (5%), the team could not find the sensors at the borehole site and they could not be traced. Of the 116 sensors that remained properly installed, two sensors experienced issues during data download, yielding 114 successful downloads.9 9 One sensor in Samburu that was still connected to the borehole pump malfunctioned and its data could not be downloaded, while another sensor in Samburu yielded a dataset that did not register any on-time for the borehole pump due to a community member unclamping it prior to the arrival of the data collection team for “safe keeping.” In Kenya RAPID counties, the sensors broadcast data and the RAPID partner team can quickly identify when sensors are disconnected or stop recording data, as well as make follow-up calls or send field staff to verify the situation. The evaluation team does not have a full record explaining why sensors were uninstalled or missing. In some cases, the enumeration team determined it was likely that new borehole managers who were unfamiliar with the sensors removed them during repairs. Given that there is not continuous monitoring in the comparison counties, it is not possible to verify why sensors were removed in all cases. After briefly disconnecting each sensor in order to complete the download, the data collection team reinstalled the same Round I sensor on 104 boreholes (excluding non-functional boreholes) and installed previously unused sensors on 11 boreholes. The evaluation team removed a total of seventeen boreholes from the sample for various reasons, including change of power source and apparent borehole abandonment, which was the case for four boreholes that were no longer functional.10 10 In five of the cases, the borehole converted to a solar power system that was not compatible with the sensors, while seven boreholes were out of service when the data collection team arrived to download the data. In three cases, the power source was changed to have a direct line to the borehole pump such that the sensors ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 14 It is important to note that not all of the boreholes for which the team downloaded data at Round II will remain in the sample for Round III for the reasons described above (e.g. abandoned boreholes). Given that EQ1 estimated an overall treatment effect, rather than heterogenous effects within or between counties, this does not undermine the overall fidelity of the evaluation. In addition, this level of attrition is considered acceptable, although similar attrition before the final round of data collection and absent a large treatment effect (i.e. effect size 30% or more functionality compared to non-RAPID counties) will limit the ability of the evaluation team to estimate an impact where one exists. Prior to Round III, the evaluation team will make random spot checks on comparison boreholes to flag potential attrition threats. could not be installed on a live wire. One borehole manager refused to let the team re-install or replace the sensor, and security was a concern at a borehole in Mandera where the sensor installed at Round I was stolen. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 15 TABLE 2. COMPARISON COUNTY BOREHOLE DOWNLOAD DETAILS A B C D E F G H I J County Sensors Installed at Round I Initial Sensor Status at Round II Successful Data Downloads at Round II End Sensor Status at Round II Total Sensors Available for Round III Remain Installed On-site but disconnected Lost Original Sensor Installed New Sensor Installed No Sensor Baringo 18 16 2 - 14* 12 5 1 17 Kitui 22 21 1 21 20 - 2 20 Laikipia 30 26 3 1 25* 23 - 7 23 Mandera 13 11 - 2 11 11 1 1 12 Meru 7 7 - - 7 7 - - 7 Samburu 14 11 2 1 12* 12 2 - 14 Tana River 13 11 1 1 10* 7 2 4 9 West Pokot 15 13 1 1 14* 12 1 2 13 Total 132 116 10 6 114 104 11 17 115 *Note: In cases where successful data downloads for a county (Column F) is not equal to the number of sensors found installed (Column C), this indicates that either a partial download was completed from one of the disconnected sensors (column D) or that there was a malfunction with one of the original sensors. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 16 5.3.2 COMPARISON BOREHOLE SENSOR DATA Ipsos led a team of 16 data collectors, 13 of whom participated in Round I data collection. Two data collectors—a supervisor and an enumerator—worked in each comparison county. At least one member of each team worked in their assigned county during Round I. After one day of training in Nairobi, one day piloting the data download process on two boreholes in Laikipia, and a debrief meeting back in Nairobi to review the pilot experience and training, the eight data collection teams travelled to their assigned counties. Data collectors uploaded borehole sensor data to a cloud drive on a daily basis, and the evaluation team conducted daily data quality checks to assess whether there were any issues with the sensor data. In addition, Ipsos conducted regular data quality checks on the supplemental borehole asset survey to ensure that data collectors addressed all survey items and to flag any potentially questionable responses, such as extremely high or low values. The data collection team encountered several challenges, including: • Poor road quality and weather conditions. In Samburu, Laikipia, Tana River, and West Pokot, the Ipsos team faced notable travel challenges due to the onset of the rainy season and poor road quality. • Removal of sensors. In cases where the sensor was found uninstalled, the data collection team found that this was often due to repair technicians’ failure to reinstall the sensor (in cases where wiring was part of the repair process) or removing it out of ignorance. The evaluation team will attempt to make more regular check-ins with borehole managers to ensure awareness and that local stakeholders know that the sensors do not interfere with water access or borehole pump functionality. • Technology failure. The comparison sensors require a unique USB connection to convert the compressed data into a readable format. The cable on one sensor was faulty and a replacement needed to be ordered from the United States. Customs held the replacement cable, which delayed the deployment of one data collection team. As with the Round I data collection, outreach to county and sub-county officials prior to data collection and on the day that teams deployed to the boreholes was critical for ensuring easy access to the borehole sites. This helped ensure communities were aware that the data collection team was arriving. Given the sensitive nature of water access in many of the comparison county borehole sites, mitigating potential community hostility is critical for the ongoing fidelity of the evaluation and for the data collection team’s safety. A key take-away from this round of data collection is that commencing data collection earlier in August or September will improve accessibility.11 11 Final timing for Round III data collection will be determined in coordination with USAID. 5.3.3 TREATMENT BOREHOLE SENSOR DATA Kenya RAPID partner SweetSense provided two datasets that captured output from the intervention borehole sensors to the evaluation team via a cloud-based storage link. The first dataset contained 23 variables for all of the intervention sensor data (strategic and non-strategic boreholes), including dates, borehole pump on-time as a percentage of the day (i.e. on-time percentage over 24 hours), various summary measures for the borehole (e.g. median on-time), and repair status. After cleaning the initial dataset, SweetSense also provided a revised version of the aforementioned dataset limited to 10 variables, including a date stamp, geo-coordinates, and borehole repair status. While, the repair status is ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 17 interesting for understanding how the intervention may prompt county officials to make repairs, there is no equivalent value in the comparison sensor data, which only contains the date, time stamp, and percent-on. After reviewing both datasets, the evaluation team decided to use the original file’s measures of percent of day with the pump on (percent-on) and date stamp, since the missing values in this file were similar to what is found in the comparison data. In contrast, the cleaned file largely had missing values replaced by zeros, which affects analysis. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 18 6 FINDINGS 6.1 ROUND I RECAP This section summarizes the main quantitative and qualitative findings from the Round I report, which provide relevant context for interpretation of Round II findings. The Round I report focused on descriptive summaries of the borehole asset survey and qualitative responses from interviews with county officials, borehole managers, and water users. At Round 1, the evaluation team also conducted balance tests to assess the comparability of the treatment and comparison groups and found that 1) about half of the borehole survey metrics suggested strong balance without matching and 2) matching improved equivalence across assignment groups for key metrics. Matching and balance are discussed further in Section 6. The Round I borehole asset survey found the following:12 12 Note: these figures were updated from the Round I report to include Isiolo county, which was not included in the previous report due to a coding error in the intervention borehole data, which has since been corrected. • Water source access and the number of households served was similar in treatment and comparison boreholes. There were 450 households at the median served by the treatment group strategic boreholes and 400 at the median in the comparison group, with most of the boreholes in low-density service areas. • Livestock use is common across assigned groups. Round 1 data suggest that 83 percent of the boreholes in the comparison counties and 71 percent of the boreholes in the treatment counties are used for livestock. • Boreholes were largely constructed in the past ten years. The median construction date for boreholes was 2010 in the Kenya RAPID counties and 2012 in the comparison counties. Only 2 of the 17 comparison boreholes with a physical state rated “poor” were constructed before 2000. • Generator power is widespread in the comparison counties, while the treatment counties have a broader mix of power sources. The mix of power sources across treatment and comparison counties varied, with none of the comparison counties reporting the use of hybrid borehole pump power sources (e.g., solar power with a generator back-up), compared to 33 hybrid boreholes in the treatment counties. In six of the eight comparison counties, the majority of sampled boreholes had generator-powered borehole pumps, with Tana River reporting 46 percent of its sampled boreholes using generator power. West Pokot was something of an outlier in this group, with only seven percent of the sampled boreholes powered by generator. • Boreholes across both assignment groups are largely functional and run multiple days a week. Across all of the sampled boreholes included in Round 1 data collection, only four treatment boreholes were non-functional. Partial functionality is defined by reduced yield per the borehole’s design. Qualitative data collection found that generator-powered borehole users reported facing frequent issues with functionality due to broken borehole power sources, pipes, or pumps. Despite these service issues, more than 50 percent of the boreholes across both assignment groups operated eight or more hours a day, and 149 boreholes in the sample operated seven days a week. Although median operating hours were the same in both the treatment and comparison counties, average operating hours were slightly higher in the treatment group at around ten hours a day compared to eight hours a day in the comparison group. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 19 • Use of tariffs varies across assignment groups. The presence of fixed and formalized tariffs may reflect the management capacity of the borehole operators or their ability to address and budget for repairs. Notably, 19 percent (n = 38) of the comparison boreholes had no tariff system in place at Round I, with four comparison county boreholes reporting an “ad hoc” system, which may not be applied to all users or with the same rates. In contrast, all but two of the treatment boreholes did not have a tariff. Qualitative analysis at Round I provides some context to these figures in that users reported long wait times and travel constraints. This was particularly true during dry periods when alternative modes of water access, such as tarps and temporary streams, were unavailable. Round I qualitative data is also relevant to the above findings in that many borehole managers reported reliance on user fees for funding borehole repairs and maintenance. During Round I qualitative data collection, county and sub￾county officials noted limited internet access as a key challenge facing potential use of an information sharing dashboard. These findings will be investigated further at Round III. 6.2 ROUND II OVERVIEW As noted above, the main sources of data for this round are borehole sensor on-time data for both treatment and comparison boreholes and a limited borehole asset survey for the comparison boreholes. This section presents summary statistics for each of these data sources. The evaluation team analyzed differences across the Round I and Round II borehole asset surveys to investigate the extent of change for certain comparison borehole characteristics. In many cases, there was variation in characteristics reported by borehole managers, in addition to observed, verifiable changes (e.g. borehole power source). 6.3 LIMITED BOREHOLE ASSET SURVEY The Round II limited borehole asset survey tracked 76 measures, including photographic verification of the borehole conditions. The Round I asset survey included 170 measures in contrast. Annex E provides an extended table of summary statistics for the survey outcomes. This section presents the descriptive statistics for key metrics, as well as comparisons to the Round I findings to highlight where results changed over time. Given that the Kenya RAPID team reported changing conditions at treatment boreholes, it was important for the evaluation team to assess whether and to what extent conditions changed for the comparison boreholes. Variation between rounds is not necessarily a cause for concern; where differences exist, the following section outlines implications and considerations. No comparisons are made across assignment groups, since the Round II team conducted the limited borehole asset survey only in the comparison group. 6.3.1 BOREHOLE USE At Round II, the median number of households served across the 132 comparison boreholes was 372, with an average of 821. However, as shown in Figure 4 there is quite a bit of variation in the number of reported households served by each borehole across counties and within each Round. The number of households reportedly using a borehole is an important measure given that it speaks to the level of demand at each borehole, as well as the character of constituencies for resources. However, it is important to note that this is an unofficial figure that local borehole managers reported and are likely prone to some amount of error. While aggregated census data for the county or sub-county are accessible, official population figures are difficult to obtain for the remote areas around the boreholes. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 20 Use of census data that is aggregated at a high administrative level is also complicated given that boreholes serve multiple villages. Instead, it is likely best to consider these figures general estimates of the magnitude of use and whether it increased or decreased between rounds. In Round II, Mandera and Kitui served the highest number of households on average, at 1,877 and 1,558 respectively. Across survey rounds, Kitui saw the largest increase in the average number of households served per borehole, going from 1,003 households at Round I to 1,558 at Round II. In contrast, Baringo saw the largest decrease in the average number of households served, going from 732 households on average per borehole to 340 at Round II. Kitui had four of its 22 boreholes report service of 2,000 or more additional households between Round I and Round II. In contrast, of the 18 sampled boreholes in Baringo, only four did not see a decrease, one of which reported no change in the number of households served. FIGURE 4.AVERAGE NUMBER OF COMPARISON HOUSEHOLDS SERVED BY ROUND Round I qualitative interviews suggest that livestock use is also a critical factor influencing demand for boreholes. It is important to note that this measure captures whether a borehole is used for watering livestock to any degree; it does not focus on exclusive use. Changes in the percentage of boreholes used for watering livestock could be the result of many factors but are worth tracking because a third of all comparison county boreholes apply tariffs to watering livestock. The overall average number of boreholes used for watering livestock fell slightly, from 83 percent across all comparison boreholes on average at Round I to 80 percent at Round II. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 21 Both Baringo and Tana River saw a 16 percent decrease in boreholes used for livestock. West Pokot and Laikipia were the only counties that reported an increase in the percent of boreholes used for livestock, with all 15 borehole managers in West Pokot reporting watering livestock at Round II, and 24 of the 30 borehole managers in Laikipia reporting the same. FIGURE 5. PERCENTAGE OF COMPARISON BOREHOLES USED FOR LIVESTOCK BY COUNTY AND ROUND Water trucking is often a major source of demand on boreholes in drought prone areas. In the comparison counties, 29 percent (n=38) of boreholes in the sample reported providing water for trucking at Round I. The Round II data suggest that only about 21 percent (n=27) of boreholes are used for trucking. As shown in the figure below, the number of boreholes used for trucking in Laikipia decreased from 30 boreholes to 4 between Rounds I and II. The one borehole in Meru that reported water trucking use at Round I reported no trucking at Round II. Figure 6 shows the number of boreholes per county used for trucking. It is notable that Mandera and West Pokot are the only counties to report an increase in the number of boreholes used for trucking. Related to this overall decline in the number of boreholes used for trucking, borehole managers reported a small decline in the average number of days the borehole was used for trucking, from 4.1 days in Round I to 3.9 days at Round II. However, while five comparison counties reported a decline in the average number of days boreholes were used for trucking, Tana River, Kitui, and Laikipia actually saw increases in the number of days their sampled boreholes were used for trucking; in Tana River use increased from an average of 3.5 to 5 days, in Kitui it increased from 5 to 6 days, and in Laikipia trucking increased from 4.75 days on average to 5 days. As discussed below, the borehole sites in Tana River and Kitui were two of the most arid counties in the comparison group in the Round II data collection period. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 22 Rainfall in 2019 was heavy, including a 400 percent increase in rainfall above the average since 1981 for the last quarter of the year.13 13 See FEWSNet Kenya Food Security Update December 2019 https://reliefweb.int/sites/reliefweb.int/files/resources/KENYA_Food_Security_Outlook_Update_December_20 19_final.pdf Indeed, in its September National Drought Bulletin, NDMA reported improved vegetation conditions for all of the sampled counties.14 14 See https://www.ndma.go.ke/index.php/resource-center/national-drought-bulletin/send/39-drought￾updates/5334-national-monthly-drought-updates-september-2019 The borehole survey is not designed to ask about trucking demand, which is often driven by access constraints from existing water sources. Additional qualitative inquiry at Round III will probe changes in borehole use and how perceptions and experience of increased rainfall may affect demand for trucking. FIGURE 6. NUMBER OF COMPARISON BOREHOLES USED FOR TRUCKING BY ROUND 6.3.2 BOREHOLE MANAGEMENT The borehole asset survey asks borehole managers about borehole operating hours per day. This is a somewhat nuanced survey item in that it is self-reported whereas the sensor data show the observed functionality of the borehole pump, which is slightly different. The operating hours response provides a measure of how many hours a day a borehole may be “open” or operational and available for users to access. This is distinct from pump on-time as captured by the borehole sensors: in the case of a generator powered borehole, for example, the pump may be on for long enough to fill a water tank that is then drawn down on while the borehole is open for use. These two measures together help present an understanding of use and functionality. As shown in the Figure 7, median operating hours did not significantly change across rounds. The largest change occurred for generator powered boreholes, which went from a median of eight hours of ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 23 operational time to six hours; the overall number of generator powered boreholes did not change across survey rounds, with 81 total generator powered boreholes. In contrast, the number of solar powered boreholes increased from 29 at Round I to 33 at Round II, but median operating hours did not change. The four new solar powered boreholes in Round II were utility powered at Round I, with the total number of utility powered boreholes decreasing from 22 at Round I to 18 at Round II; median utility operating hours declined by an hour to 7.5 at Round II. FIGURE 7. COMPARISON BOREHOLE OPERATING HOURS PER DAY BY SURVEY ROUND As discussed in the review of the borehole sensor data, borehole use often varies seasonally. At Round I, about 77 percent (n=101) of borehole managers reported that use depended on whether or not it was the rainy season (reported as yes/no). The motivation behind this question is that borehole managers may choose to shut down or operate at reduced hours when rainwater is prevalent. At Round II, however, only 67 percent (n=89) of borehole managers reported that use was contingent on rainfall. Samburu and Laikipia saw the largest change in reported average seasonal operation: at Round I, of Samburu’s 14 boreholes, 64 percent (n = 9), reported operating based on rainfall, but this dropped to 36 (n=5) percent at Round II; the reported percentage of Laikipia’s 30 seasonally operated boreholes decreased from 93 percent (n=28) to 70 percent (n = 21). Apart from Baringo and Tana River counties, all of the comparison county boreholes also reported that the average number of days boreholes were open during the rainy season increased. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 24 FIGURE 8. AVERAGE NUMBER OF DAYS COMPARISON BOREHOLES ARE OPERATIONAL DURING THE RAINY SEASON BY SURVEY ROUND Across all comparison counties at Round II, borehole managers reported operating around three days a week during the rainy season, which should correspond to April, October, and December based on the rainfall data described below. Comparison sensor data for these rainiest months suggests that boreholes were operational (i.e. recording more than zero on-time) around four days a week on average for the period of 2018 and 2019 for which there is data. The data for all months, not just the rainiest months, shows that pump on-time was around three days a week in 2018, while it is a bit higher for 2019, at about 4 days a week. These self-reported figures are helpful for understanding borehole functionality perceptions, as well as potentially filling in contextual gaps where the sensor may not have registered on-time. However, it does appear that borehole managers are somewhat precise in their estimates of rainy season functionality. As noted in the Round I report, the presence of fixed and formalized tariffs may reflect the management capacity of the borehole operators or their ability to address and budget for repairs. Indeed, qualitative data collected at Round I suggested that borehole fee collection and management are intended to allow borehole operators to address small issues, but are hindered by mismanagement and lack of local capacity. There were some changes in borehole fee structures between rounds. As shown in Figure 9, the number of comparison boreholes that did not require any payment decreased from 36 at Round I to 30 at Round II, while fixed tariffs per visit and per month both increased over the same period. Mandera and Meru led this shift to requiring fees for using boreholes; six boreholes and one borehole, respectively, at Round I did not charge fees, while all of the sampled boreholes in these counties charged fees at Round II. West Pokot, in contrast, reported an increase from 13 to 14 boreholes that did not charge fees. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 25 FIGURE 9. COMPARISON COUNTY TARIFFS BY ROUND The power source mix within the comparison counties changed slightly between Round I and Round II. Although these changes were marginal in absolute terms, they have implications for the borehole sensor data, as noted in the Round I summary, and may affect functionality. Round III sensor data collection will provide more insight into whether functionality changed for boreholes that moved to different power sources at Round II. Baringo saw the largest change, with two boreholes switching to solar from utility and generator power. One generator borehole in West Pokot switched to solar power resulting in solar power for all 15 of the sampled boreholes in this county. Drivers for these changes are unclear and may be investigated during Round III qualitative data collection. Borehole asset survey data suggest that reported functionality differs depending on the power source. In Round I, around 82 percent (n=18) of utility powered boreholes were functional, while one utility powered borehole was non-functional at Round II. This stands out compared to the solar powered borehole pumps, 55 percent (n=16) of which were functional at Round I, and generator powered boreholes, 62 percent (n=50) of which were functional at Round I. As shown in the Figure 11, Round II saw an increase in the number of non-functional boreholes, but also saw an increase in the percentage of functional solar powered borehole pumps. Round II also saw an increase in the number of boreholes deemed “partially functional” across all power source types, defined by reduced pump yield. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 26 FIGURE 10. BOREHOLE POWER SOURCES BY ROUND FIGURE 11. COMPARISON COUNTY FUNCTIONALITY BY ROUND AND POWER SOURCE ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 27 At Round I, 38 total boreholes (29%) reported failure in the previous 12 months, while 45 boreholes (34%) reported failure at Round II over the same period. As with Round I, borehole managers that reported failure did so for a variety of reasons. The nature of failures, however, changed across survey rounds in the comparison counties. Figure 12 displays reasons for borehole failure as a portion of total failures reported. Pipeline failure accounted for half of all reported borehole functionality issues at Round I (n = 66), while only 13.3 percent (n=6) of the comparison county boreholes reported this as a reason for failure at Round II. Water source challenges were the main failure reported at Round II, for 33 percent (n=15) of comparison boreholes. Notably, grid power issues did not explain any borehole pump failure at Round I, while one borehole in Laikipia reported grid power failure at Round II. Generator failure was also more frequent at Round II, increasing from six to 10 reports of generator FIGURE 12. COMPARISON BOREHOLE WATER FAILURE REASONS BY ROUND failure between rounds. This has implications for available sensor data, as well as water access for users and broader management and resource concerns. For users and management, the challenge presented by generator pump failure is clear: limited access to water. For the evaluation, generator failure presents a challenge because it either results in no data collected from the sensors, or potentially to sensors being removed in the process of pump repair or replacement. 6.3.3 CONCLUSIONS Although differences exist across survey rounds, average changes across survey rounds are not practically or statistically meaningful. Within counties, there are some notable shifts, including the presence of more solar powered boreholes and increased borehole failures. Borehole use for livestock notably increased in West Pokot and Laikipia, but fell across the other counties; overall, the change in the number of boreholes used for livestock did not rise or fall by more than three boreholes for any county in the comparison sample. This range (i.e. less than four boreholes) was typical for noted differences in most survey measures between Rounds I and II. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 28 Self-reported data is always subject to some reporting error, particularly when respondents—borehole managers in this case—may be biased by their most recent experiences rather than thinking longer term. Similar to the Round I report, the above metrics present a picture of the comparison county boreholes. We see that about two-thirds of the boreholes seem to be largely functional, that generator power is prevalent, but use of solar seems to be increasing, and that tariffs are common. The Round II survey suggests that despite some changes to individual boreholes, the overall comparison context has not altered dramatically, or in ways that cause concern for fidelity to the evaluation design. 6.4 BOREHOLE SENSOR DATA DESCRIPTIVE ANALYSIS As described above, the comparison sensor data provides a detailed measure of borehole pump on-time down to the minute, but comes with the challenge that data have to be physically downloaded at the borehole site. For any given borehole, this generates an extremely large amount of data and, for the purposes of this impact evaluation, provide limited analytical value. The evaluation team exported the comparison sensor data from 114 boreholes at 15 minute and daily increments, while the Kenya RAPID implementation team provided the treatment borehole data in daily increments. Both the comparison and treatment borehole datasets provide a date stamp noting the day, month, and year of the reading and a “percent on” measurement. The on-time data for comparison borehole pumps illustrates usage changes over time. As shown in Figure 13, borehole pumps in each county tend to be used most in the early morning hours and late at night. This trend aligns with qualitative data collected at Round I, as well as the evaluation team’s experience in Kenya. FIGURE 13. AVERAGE COMPARISON BOREHOLE ON-TIME BY HOUR AND MONTH ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 29 The monthly average on-time shows general annual use patterns, but daily use within counties reveals quite a bit of variation within the bimodal use behavior—early in the morning, late at night. As shown in Figure 14, a year of sensor data suggest that some counties, such as Samburu, that recorded very little on-time overall, experienced almost none, on average, in the middle of the day during the Round II recording period. In contrast, West Pokot saw the highest levels of on-time in the early morning and late at night, with some boreholes experiencing more than 19 hours of average on-time. Tana River also stands out given that none of its boreholes reported average hourly on-time per day over 27 percent. FIGURE 14. AVERAGE DAILY ON-TIME PERCENTAGE BY HOUR FOR COMPARISON COUNTIES 2018-2019 As noted in the Round I report, the intervention sensors transmit data allowing for regular data collection throughout the year. This results in the intervention sensor data spanning April 2017 through December 2019 across 69 boreholes, compared to comparison sensor data spanning August 2018 through October 2019 across 114 boreholes, allowing for comparison over a 14-month period. The data produced from each set of sensors appear to be equivalent, but it is worth noting that the number of values for the comparison borehole pump daily on-time percentage vary much more than those for the treatment sensors. The treatment sensors have 37 unique values, ranging from 0 to 1, whereas the comparison sensors reported 999 unique values over the same value range. As shown in Figure 15 below, there are discrete points of on-time percentage for the treatment sensors compared to a more or less continuous range of values for the comparison sensors. For the purposes of analysis, the percent-on data will be treated as continuous data. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 30 FIGURE 15. COUNT OF OBSERVATIONS BY BOREHOLE PUMP DAILY PERCENT ON In looking at the treatment and comparison data together, use patterns within each group are consistent on a daily basis, but vary across assignment groups. As shown in Figure 16, boreholes in both assignment groups pumped with similar frequency each day of the week. It is worth noting that the median (shown as the thin white line) does vary by day of the week, suggesting slightly higher on-time on Wednesdays, Fridays and Saturdays for the Kenya RAPID boreholes. Sensor data across assignment groups shows some seasonal variation across both the treatment and comparison groups. On-times were higher in January through March and again toward the end of the year in November and December. For example, Mandera, in the comparison group, has a visible peak in average daily borehole on-time percentage in March at about 12 hours across all 11 boreholes. On March 16 in Mandera, one borehole ran for just under 24 hours and there were two boreholes that ran for 90 percent or higher average on-time between February and April. Average daily on￾time percentage dropped to zero on April 26 and no individual boreholes had an average daily on￾time above 59 percent in the month of May, with FIGURE 16. DISTRIBUTION OF ON-TIME BY DAY AND ASSIGNMENT GROUP IN 2019 ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 31 the overall county average percentage pump use peaking at 17 percent for that month. In contrast to Mandera, Isiolo experienced low average daily on-time, with the only reported increase in May and August within the Round II recording period. The highest daily use rate for Isiolo was on August 10, 2018, when one borehole ran for more than 15 hours. FIGURE 17. AVERAGE DAILY BOREHOLE ON-TIME BY COUNTY Round I data, as well as the borehole asset survey, suggest that rainfall is a key variable in whether or not borehole managers turn on a borehole pump. In Round I, borehole users described not needing to go to boreholes when rain was more frequent because alternative water collection methods, such as from temporary streams, were available. In looking at average monthly rainfall and borehole pump on￾time, there does appear to be a slightly negative relationship between rainfall and average on-time. This negative relationship is stronger in the comparison counties than in the treatment counties.15 15 The Pearson correlation coefficient between average monthly rainfall and average monthly on-time is -0.26 (p￾value = 0.01) for the comparison counties and -0.08 (p-value = 0.53) for the treatment counties. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 32 FIGURE 18. MONTHLY RAINFALL AND BOREHOLE USE BY ASSIGNMENT In Round I, the evaluation team used county averages from the Climate Hazards Group InfraRed Precipitation with Station (CHIRPS) to compare counties and assignment groups. For Round II, the evaluation team used the same data source to estimate daily rainfall for a 20-meter radius around each borehole from 2017 through 201916 16 Climate Hazards Group InfraRed Precipitation with Station (CHIRPS) is a satellite database of global rainfall providing resolution up to 0.05 degree, . This is a data intensive process, but yields a better picture of precipitation at each borehole and within the counties across both assignment groups. Looking back to 2017 is also helpful for understanding recent weather trends, even though the period of data collection at comparison boreholes did not start until August 2019. During the Round II recording period (August 2018 - October 2019), the average daily rainfall was 1.72 millimeters for comparison boreholes and 0.84 millimeters for the treatment boreholes. The Figure 19 below shows the average daily rainfall across all treatment and comparison counties before and during the Round II recording period. It is not surprising that rainfall was lower, on average, in the treatment counties since this is one reason why Kenya RAPID targeted those counties for intervention. Notably, both assignment groups experienced an increase in rainfall in April 2018, with year-end rainfall for October higher in 2018 than in 2017 or 2019. The daily average rainfall at each borehole site was higher in the treatment counties than in the comparison counties twice between 2017 and 2019, in April 2017 by around 0.10 millimeters and in February 2018 by about 0.15 millimeters, but comparison borehole rainfall was otherwise higher across the data. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 33 FIGURE 19. AVERAGE BOREHOLE SITE RAINFALL BY ASSIGNMENT GROUP ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 34 7 PROPENSITY SCORE MATCHING 7.1 OVERVIEW Key to the evaluation design is the implementation of propensity score matching (PSM) to create a counterfactual (Abadie & Imbens, 2006; Dehejia & Wahba, 1999; Rosenbaum & Rubin, 1983). Under PSM, a select number of observable characteristics are used to generate a predicted probability of being assigned to the treatment group. This predicted probability (propensity score) is used to match treated and comparison units based on the theorem that if they match on the score generated from observable characteristics, they would presumably match on the specific values of the observable characteristics. Unlike covariate matching (i.e., one-to-one pairing for each characteristic), PSM can improve average similarity across characteristics. The evaluation team revisited its PSM approach based on feedback from Round I to further improve the similarity between matched treatment and comparison samples. 7.2 ROUND I RECAP 7.2.1 ROUND I METHOD The Round I report highlighted limitations of PSM, including the critical assumption that similarity of observable characteristics implies similarity across unobservable variables. This assumption is generally violated since it is often known ex ante that interventions are targeted due to both observable (in available data) and unobservable (e.g., political or management characteristics) traits. To address this, PSM often relies on datasets larger than those used in randomized control trials to maximize the number of matches and improve balance. However, a key limitation of this evaluation is that there are a limited number of strategic boreholes in each county from which to sample, thus a limited number of potential matches. As the Round I report notes, matching is often highly sensitive to the parameters of the matching model—the set of observed variables used to estimate the propensity score. The general model for estimating propensity scores at Round I followed that of Rosenbaum and Rubin (1983), where propensity is the conditional probability of treatment subject to a set of key explanatory variables. At Round I, the evaluation team started with a basic model that incorporated the following eleven covariates as key metrics from the borehole asset survey and third-party data sources: • Number of households using borehole scheme • Livestock use (Y/N) • Service area for the scheme (in km2) • Out of service one or more days in past month (Y/N) • Three-year rainfall average (annual, in mm) • Borehole pump power type • Average daily on-time (in hours) • Travel miles from Nairobi • Borehole construction year • Fixed tariff scheme (Y/N) • Recent water test (Y/N) The team selected these variables based on their understanding of the intervention, selection criteria for counties and boreholes, and descriptive analysis. The evaluation team standardized the number of households served, miles, and service area variables by centering and dividing by two standard deviations to make comparisons across parameters easier.17 17 For more on standardizing and rescaling see, Gelman, Andrew. (2008). “Scaling regression inputs by dividing by two standard deviations”. Statistics in Medicine 27: 2865–2873. The Round I report tested multiple model specifications. The goal of matching is to create a pool of similar units based on their propensity scores. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 35 At the end of Round I, the evaluation team found that matching was able to generate a sample of overlapping, i.e. similar, propensity scores across the assignment groups. 7.2.2 ROUND I BALANCE A key consideration for any IE is whether assignment groups are, on average, similar across key baseline characteristics. This is critical for most evaluation designs because it is the basis for the assumption that absent the intervention, the two assignment groups would otherwise achieve the same outcomes. For this IE, we know that the team chose the treatment counties because they are not like the rest of the counties in Kenya, but at the borehole level, characteristics vary, with some similarities across assignment groups. Specifically, rainfall and travel distance are important within the context of the theory of change given that rainfall historically affects borehole use and associated pump breakages.18 18 In modelling the relationship between the key covariates and treatment status, rainfall is the strongest predictor. County officials oversee budget processes that have implications for repairs, the availability of parts, and even personnel. A key feature of Kenya RAPID’s information sharing intervention is to address the challenges remote boreholes face by notifying county officials about pump functionality issues. As detailed in the Round I report, matching achieved balance across the key covariates, with two notable exceptions: mileage and rainfall. While the standardized mean difference for these two variables improved under the various matching approaches, it was only under a parsimonious model (i.e. one with very few control variables) that rainfall and mileage were similar on average across assignment groups, and even then, the values were not within standard thresholds for reliability. FIGURE 20. BALANCE ACROSS KEY VARIABLES WITH ROUND I MATCHING PARAMETERS Figure 20 above shows the main matching approach with baseline rainfall for all seasons and mileage to Nairobi, rather than the revised variable. While matching improves balance for these two key variables, ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 36 they are still far outside of the 0.25 threshold suggesting strong similarity. The imbalance in road miles to Nairobi is not surprising given that the treatment counties are, with the exception of Isiolo, some of the most remote counties in Kenya. 7.3 ROUND II MATCHING 7.3.1 ADJUSTMENTS FROM ROUND I Feedback on Round I suggested two main adjustments to the matching approach: limiting the rainfall variable to those months with the lowest rainfall and adjusting the mileage variable to calculate the distance between each borehole and the county seat rather than between boreholes and Nairobi. The evaluation team considered the travel distance to Nairobi less relevant to borehole management than the travel distance to the local county seat given that immediate resources from the local county officials are critical to repairs. The Kenya RAPID team suggested limiting rainfall to the driest months based on their experience with borehole demand and resource need. Although the overall relationship between rainfall and borehole pump on-time appears to be weak, as noted in Figure 18, this is a critical theoretical consideration for the intervention. With these suggestions in mind, the evaluation team re-ran the matching models. Looking at average daily rainfall for 2017 and 2018, the months of January, February, and September experienced some of the lowest rainfall across both assignment groups. In addition, the team re-ran its travel distance calculations using the Google Maps API based on the above approach. Although matching in Round I provided improved balance, the results shown below show that these changes actually improved the area of overlap between the treatment and comparison groups for Round II, i.e. there were more boreholes with similar propensity scores. The balance across the key variables also improved under the Round II variable specification. 7.3.2 ROUND II METHOD Similar to Round I, the main model used for matching employs an “optimal” matching algorithm, which attempts to minimize the overall distance between propensity scores across the data. This is a way to improve overlap across the entirety of our treatment sample. The evaluation team explored other matching models (i.e. nearest neighbor and many-to-one) and findings are presented in Annex F. As shown in Figure 21 below, without applying matching, only six comparison boreholes have a propensity score above 0.50 and 47 comparison boreholes have a propensity score of zero. The Round II matching model with the new rainfall and mileage variables results in only one comparison borehole receiving an estimated propensity score of zero. As shown in Figure 21, the matching process smooths out the distribution of estimated propensity scores in the comparison group, improving the area of overlap between the treatment and comparison scores overall. When we compare the Round I initial matching approach to the approach applied in Round II, it is also clear that the strong bimodal distribution across treatment and comparison propensity scores diminished with the new parameters. As noted in the Round I report, matching requires a full set of data with no missing values that can diminish the overall sample size. To address the need for a full dataset without missing observations of key variables, such as travel miles, the evaluation team imputed missing values by estimating miles using a basic linear regression with all of the other matching variables as covariates. However, there are still missing values for the borehole asset survey data across both assignment groups, resulting in a total treatment sample of 60 out of a total of 69 strategic boreholes on which to match. Both the Round I and Round II matching approaches result in all 60 treatment boreholes receiving a comparison borehole match, for an overall sample of 120 boreholes. However, as discussed below, the balance across treatment and comparison groups improves across key variables ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 37 with the Round II parameters. The key takeaway of using the revised rainfall and travel variables is that they are more similar, on average, across assignment groups and improve the ability of the comparison group to serve as a convincing counterfactual for the treatment counties. FIGURE 21. UNMATCHED AND MATCHED SAMPLES WITH NEW MODEL INPUTS 7.3.3 ROUND II BALANCE As shown in Figure 22 below, using the new mileage variable, as well as limiting rainfall to the dry months during the pre-intervention period of 2017-2018, achieved overall balance both with optimal matching and a full set of baseline covariates. Balance here is defined by an absolute standardized mean difference below 0.25, which suggests that the average across treatment and comparison groups is statistically similar for the variables of interest. This balance measure provides a scale-free measure of the average difference between assignment groups, which allows for balance across the set of key variables to be easily compared and interpreted. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 38 FIGURE 22. BALANCE ACROSS KEY VARIABLES WITH REVISED MATCHING PARAMETERS It is important to note that before matching, the average dry season rainfall across treatment and comparison counties was, on average, very different: average daily rainfall at comparison county boreholes was 0.62 mm, while it was 0.23 mm at the treatment county boreholes, with a standardized mean difference around 1, suggesting little balance across assignment groups for this metric. After the Round II matching approach is applied, the average comparison county borehole dry season rainfall drops to 0.29 mm, with an absolute standardized mean difference of 0.07 for matched boreholes. As noted in Annex F, the nearest neighbor matching approach yielded improved overlap, but somewhat limited gains in overall balance, while the two-to-one matching approach did not meaningfully improve balance across the treatment and comparison boreholes. Due to the lack of balance, regression analysis at Round II only uses the optimal matching sample. A key takeaway from revisiting the matching process with revised variables is that the theoretical considerations that Kenya RAPID and USAID staff raised after Round I do indeed matter for the construction of a counterfactual group. Based on input from the implementation team, the distance to get to the county seat is more relevant than the travel to Nairobi, even if many funding decisions are made in the capital. Similarly, USAID designed Kenya RAPID to mitigate drought during the months where water access is most critical and substitutes, such as rainfall, are less available, so looking at rainfall during this period is conceptually and practically more important than over a longer period when seasonal trends may get attenuated. 8 INITIAL BOREHOLE FUNCTIONALITY COMPARISONS 8.1 MODEL DEFINITIONS The main models for estimating the effect of Kenya RAPID’s sensor intervention on borehole pump on￾time are based on the model for estimating the propensity scores, with the same set of key baseline variables as described in Section 7.3.2 and in Annex F. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 39 Model 1 contains an indicator variable denoting whether it is one of three identified dry months: January, February, or September. More formally, it is specified as the following: 𝑌𝑌𝑖𝑖𝑖𝑖 = 𝛾𝛾0 + 𝛾𝛾1𝑋𝑋𝑖𝑖 + 𝛽𝛽(𝛿𝛿 ∗ 𝑇𝑇𝑖𝑖) + 𝛾𝛾3𝐶𝐶𝑖𝑖𝑖𝑖 + 𝛾𝛾𝑖𝑖 + 𝜀𝜀𝑖𝑖𝑖𝑖 (1) Where: • Yit is the outcome of interest, daily percentage that borehole pump was on, for borehole i at time t, • Xit is the vector of fixed survey and mileage covariates detailed in the matching section above for borehole i, with the exception of the reported variables on out of service days and estimated on-time since these are directly measured through the dependent variable, • T is an indicator variable equal to 1 for members of the treatment group, • 𝛿𝛿𝑖𝑖 is an indicator variable equal to 1 in the dry months, • Cit is CHIRPS rainfall data for a 20-meter buffer around each borehole i at time t, • γi is a county fixed effects term, • εit is a random error term, • with Y and β as parameters to be estimated. The estimate of Kenya RAPID’s impact is given by β, which reflects the Average Treatment Effect during the months with the lowest rainfall. This model provides a reliable estimate of the treatment effect that takes into account the full variation in rainfall and on-time across all of the available data for the period under review. To further investigate the effect of the intervention solely during the drought season, the evaluation team also employs a second, data-limited model. Model 2 uses a similar specification to model 1, but with the indicator for drought months removed and the data limited to only those specific months, which reduces the number of observations from 67,072 to 18,627. 𝑌𝑌𝑖𝑖𝑖𝑖 = 𝛾𝛾0 + 𝛽𝛽1𝑋𝑋𝑖𝑖 + 𝛽𝛽2𝑇𝑇𝑖𝑖 + 𝛽𝛽3𝐶𝐶𝑖𝑖𝑖𝑖 + 𝛾𝛾𝑖𝑖 + 𝜀𝜀𝑖𝑖𝑖𝑖 (2) As with the descriptive sensor and rainfall analysis, the borehole comparisons are conducted on data from the Round II recording period (August 8, 2018 through October 10, 2019). All continuous independent variables are centered at zero and divided by two standard deviations, which makes them more comparable to the binary, yes/no covariates for the analysis.19 19 See previous reference to Gelman, 2008. Robust standard errors clustered at the county-level are used for both models. 8.2 PRIMARY OUTCOME MEASURE The primary outcome measure of this report and for EQ1 is borehole pump on-time. Notably, on-time for the period under review is consistently higher in the treatment counties. Under the evaluation design, there is no “baseline” measure to capture borehole pump on-time prior to the arrival of the SweetSense sensors; however, as noted in the Round I report and the implementation section, we do know that many of the Kenya RAPID supplementary support activities were delayed or not implemented across all of the counties at the same time. Without additional implementation data, it is difficult to know how much of the on-time increase may be related to Kenya RAPID and how much is simply the result of higher demand, better management, or access to resources. As shown in the figure and table below, the average borehole pump on-time for the matched boreholes is higher overall prior to estimating impact using either of the matching models. What Figure 23 does ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 40 not account for is all of the other borehole characteristics previously discussed, which could account for some of the overall higher on-time rates in the treatment counties. The Round I report provides a detailed overview of the descriptive statistics for the key covariates for the Round I borehole asset survey and the matched boreholes under optimal matching for the Model 1 and Model 2 data. The mean values and standard deviations are provided in Table 4, with the averages on top and the standard deviations below. FIGURE 23. AVERAGE MATCHED BOREHOLE PUMP ON-TIME FROM SEPTEMBER 2018 TO OCTOBER 2019 ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 41 TABLE 3. MODEL DATA DESCRIPTIVE STATISTICS Model 1 Model 2 Comparison n = 132 37,506 obs Treatment n = 69 29,566 obs Comparison n = 60 17,615 obs Treatment n = 60 25,705 obs Comparison n = 132 10,416 obs Treatment n = 69 8,211 obs Comparison n = 60 4,870 obs Treatment n = 60 7,140 obs Unmatched Model 1 Optimal Matching Unmatched Model 1 Optimal Matching # of HHs 857.65 888.05 1064.52 948.12 868.38 888.91 1086.26 949.02 1414.86 1482.37 1739.68 1552.73 1429.26 1483.97 1751.96 1554.34 Borehole Pump On % 0.18 0.25 0.16 0.24 0.20 0.29 0.18 0.28 0.25 0.24 0.22 0.24 0.25 0.26 0.22 0.26 Borehole Rainfall (mm) 1.69 0.85 1.33 0.84 0.71 0.20 0.39 0.19 6.28 4.99 6.02 4.99 3.49 1.36 2.35 1.22 Construction Year 2008 2007 2008 2007 2007.90 2007.93 2008.11 2007.80 13.42 8.74 16.18 8.75 13.57 8.75 16.24 8.76 Fixed Tariff 0.79 0.97 0.87 0.97 0.80 0.97 0.88 0.97 0.41 0.17 0.34 0.18 0.40 0.17 0.33 0.18 Generator Power 0.66 0.39 0.91 0.38 0.67 0.39 0.92 0.38 0.47 0.49 0.29 0.49 0.47 0.49 0.28 0.49 Hybrid Power 0.00 0.49 0.00 0.50 0.00 0.49 0.00 0.50 0.00 0.50 0.00 0.50 0.00 0.50 0.00 0.50 Livestock Use 0.83 0.71 0.89 0.70 0.83 0.71 0.89 0.70 0.38 0.46 0.31 0.46 0.38 0.46 0.31 0.46 ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 42 Model 1 Model 2 Comparison n = 132 37,506 obs Treatment n = 69 29,566 obs Comparison n = 60 17,615 obs Treatment n = 60 25,705 obs Comparison n = 132 10,416 obs Treatment n = 69 8,211 obs Comparison n = 60 4,870 obs Treatment n = 60 7,140 obs Unmatched Model 1 Optimal Matching Unmatched Model 1 Optimal Matching Service Area 3.43 3.81 3.65 3.69 3.46 3.81 3.73 3.68 5.20 4.11 6.25 4.01 5.22 4.11 6.30 4.01 Solar Power 0.15 0.12 0.09 0.12 0.14 0.12 0.08 0.12 0.36 0.32 0.29 0.32 0.35 0.32 0.28 0.32 Utility Power 0.19 0.00 0.00 0.00 0.19 0.00 0.00 0.00 0.39 0.00 0.00 0.00 0.39 0.00 0.00 0.00 Water Test 0.80 0.82 0.71 0.85 0.80 0.82 0.72 0.85 0.40 0.38 0.45 0.36 0.40 0.38 0.45 0.36 ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 43 8.3 INFERENTIAL ANALYSIS The evaluation team conducted regression analyses using multiple matching models to estimate the effect of the Kenya RAPID intervention on borehole on-time, controlling for observable differences between Kenya RAPID and comparison borehole characteristics. Model 1 results suggest that there is no statistically significant effect at Round II of Kenya RAPID on borehole pump on-time during the drought season. It is important to reiterate that this is an initial estimate and that Round III analysis, with more data, will provide a more reliable estimate of the effect of Kenya RAPID’s ICT activities. Figure 24 provides the estimated coefficients with the 95 percent confidence intervals. As shown below, county fixed effects are consistent across each sample (matched and unmatched); for example, Isiolo has a consistently negative correlation with borehole on-time. Both model estimates used robust standard errors clustered at the county-level and are provided in Annex G. The basic linear specification of Model 1 estimates that, holding the other variables constant, the effect of receiving the Kenya RAPID sensor and information sharing intervention during drought periods changes on-time percentage by around 0.04 percentage points (-0.01 0.08 95% CI) compared to non￾Kenya RAPID strategic boreholes during the drought season. In practical terms, this 0.04 estimate means about an hour more pump time compared to non-Kenya RAPID counties, on average and holding all else constant. However, this result should not be taken to imply that this is the causal effect of the intervention. Indeed, the estimate suggests that this effect is not statistically different from zero (i.e., no effect) based on the evaluation data. This finding holds for both the matched and unmatched sample. Restricting the data to only the dry months estimates larger effect sizes for the treatment group, with an average effect of between four and six additional hours of on-time, holding all else constant. The treatment effect has a 90 percent confidence interval of 0.18 to 0.39 for the matched sample, which suggests a more reliable estimate for the data. This result, however, throws out data that may be moderating some of the effect and explaining the variation in borehole pump on-time, which limits the FIGURE 24. MODEL 1 ESTIMATES ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 44 estimates’ reliability. This is useful for understanding the relationship of drought season on-time and Kenya RAPID, particularly the limits of the current model specification. The inferential analysis shown in Figure 25 is fairly consistent with the descriptive statistics, with the average for overall treatment county on-time between five percentage points and eight percentage points higher than the overall comparison county on-time. Notably, the estimated treatment effect when data are limited to only the dry months is much larger in both statistical and practical terms. This is consistent with what is seen descriptively: the mean difference between treatment and comparison counties for the three driest months is between eight and 11 percent more borehole pump on-time, or around two-to-three hours more pump on-time. FIGURE 25: MODEL 2 ESTIMATES A separate consideration is the fact that individual borehole characteristics beyond the state of the pump may be driving some of the treatment effect noted above. While county-level fixed effects do appear to be strong, it may be the case that these individual borehole effects within each county are also driving differences between and even within assignment groups. The proportion of boreholes as a percentage of total water sources for each county may affect overall use and could drive some of the effect observed in both models. This is a variable the evaluation team will attempt to account for at Round III by scraping data from NDMA reports. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 45 9 CONCLUSION This section provides brief conclusions based on the evaluation team’s analysis of Round 1I data and summarizes next steps in the impact evaluation. 9.1 SUMMARY OF CONCLUSIONS This Round II report focuses on the quantitative component of the Kenya RAPID impact evaluation, which answers the evaluation question: “Does the intervention using real-time remote sensing data of water points for strategic borehole management in Kenya RAPID counties lead to increased pump on￾time of strategic boreholes during the drought season?” To answer this question, the evaluation team collected and compiled data from multiple sources, including: 1) data on borehole on-time from two types of sensors in the treatment and comparison counties, respectively; 2) borehole characteristics from survey data collected by implementing partners in treatment counties and evaluation team partners in comparison counties; and 3) secondary data across all counties to capture key contextual variables such as rainfall and borehole remoteness. After merging these datasets, the evaluation team conducted regression analyses using multiple matching models to estimate the effect of the Kenya RAPID intervention on borehole on-time, controlling for observable differences between treatment and comparison borehole characteristics. Across the resulting models, we find evidence that borehole on-time may be higher in treatment counties than in comparison counties. The findings vary in magnitude and reliability across datasets– depending on whether we control for non-dry season use or limit the analysis to only the dry season across all metrics. Out estimation models account for a wide range of factors that differ across intervention and comparison boreholes and may also influence borehole on-time, including power type, populations served (people and livestock), rainfall, location/remoteness, and county fixed effects. The evaluation team carefully selected these control variables through the evaluation design process and consultations with implementing partners, and refined them in Round II based on feedback on the Round I report. Nonetheless, the evaluation team remains cautious about interpreting the observed differences as causal effects of the intervention for two main reasons. First, there may be additional unobserved factors that account for the higher observed on-time in treatment counties relative to comparison counties. In particular, given that intervention counties are by definition the most arid counties in Kenya, it is likely that there are fewer alternative water sources in these areas and that populations are more reliant on boreholes for their water needs. A second caveat to these results is that as of Round II reporting, the evaluation team had limited access to detailed information about the implementation status and fidelity of the Kenya RAPID intervention. Informal communication with Kenya RAPID partners indicates that implementation was somewhat uneven across intervention counties. Learning more about the implementation process and status will greatly aid in interpreting quantitative results in the subsequent round of data analysis. 9.2 NEXT STEPS Looking ahead to Round III, there are several tasks that the evaluation team plans to undertake to strengthen analyses and provide more comprehensive answers to the study’s three evaluation questions. Quantitative data analyses for Round III will closely follow the matching methods developed and presented in this Round II report, examining differences in borehole on-time across treatment and comparison counties controlling for a large set of borehole characteristics and contextual variables. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 46 Given the team’s concern that estimated differences may be due in part to additional unobserved differences between intervention groups, we are working to identify additional data sources that may allow us to capture some of these differences. In particular, we suspect that the reliance on boreholes for water needs is higher in intervention counties, because there are fewer alternative water sources, and that this may be a key factor driving borehole on-time, since we know from qualitative data collection that the decision to run the borehole pump is demand driven. The NDMA monthly water bulletins may be a key resource in adding a measure of substitutes for water boreholes to our analysis, namely ‘boreholes as a % of total water source’, to address a key confounding variable. The evaluation team reviewed these reports as part of the design to populate Tables 1 and 2 with the exact measures that we hope to capture for the entirety of Kenya RAPID implementation. This will entail reviewing each bulletin for each county and entering the data accordingly. The NDMA reporting approach is inconsistent, but the evaluation team will use the interim period between rounds to determine how best to capture this data and manage variation in reporting. A second set of tasks focuses on the second and third evaluation questions, which focus on water managers’ and users’ perceptions of the Kenya RAPID intervention and borehole functionality. As in Round I, in Round III the evaluation team will collect qualitative data through interviews with water managers and focus groups with water users in purposefully selected counties and borehole sites. The evaluation team will continue to coordinate with USAID and Kenya RAPID to incorporate detailed implementation information into the Round III analysis and is awaiting requested implementation data. While awaiting such data, the evaluation team will move forward with selecting sites for Round III qualitative data collection based on borehole functionality and precipitation data. Given that key details related to the local borehole context will still be critical for analyzing qualitative data, the evaluation team will select boreholes, then share these selections with Kenya RAPID to learn what, if any, additional activities they implemented in the selected counties, sub-counties, and at the borehole sites. This will require input from Kenya RAPID to ensure timely selection of borehole sites for Round III qualitative data collection. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 47 10 ANNEXES ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 48 ANNEX A: EVALUATION STATEMENT OF WORK Impact Evaluation of the Kenya Resilient Arid Lands Partnership for Integrated Development Project This Statement of Work is for an impact evaluation commissioned by the Office of Water in the United States Agency for International Development’s Bureau for Economic Growth, Education, and Environment (USAID/E3/Water) that will examine the Kenya Resilient Arid Lands Partnership for Integrated Development (Kenya RAPID) activity. 1. Activity Information Kenya RAPID is a five-year activity that began in October 2015 and is funded by USAID, the Swiss Development Corporation (SDC), and the private sector. Kenya RAPID aims to contribute to sustainable and resilient livelihoods for communities in Kenya’s Arid and Semi-Arid Lands (ASALs) by improving water availability and water services delivery to people and livestock and improve rangelands in the ASALs. Kenya RAPID mobilizes financial and technical resources from development partners, the national government, county governments, and the private sector to address the complex problems created by inadequate water access and poor governance of natural resources in the ASALs. Kenya RAPID targets five northern ASAL counties—Marsabit, Garissa, Isiolo, Wajir, and Turkana. Each has high poverty rates, chronic water shortages and food insecurity, and low access to basic services. Kenya RAPID uses a public-private partnership model to combine the assets and experience of development actors, private and public institutions—leveraging their capital and investments, innovation, and access to markets—to address the complex problems created by inadequate water access and poor governance of natural resources in the ASALs. Kenya RAPID will directly contribute to USAID/Kenya’s Country Development Cooperation Strategy (CDCS) 2015-2018, whose goal is Kenya’s governance and economy sustainably transformed, and the SDC’s Cooperation Strategy for the Horn of Africa goal to contribute to reduction of poverty, improve human security and instability, and address migration challenges. 2. Development Hypothesis USAID/Kenya envisions that building the capacity of relevant private and public stakeholders for improved WASH service provision and improved rangeland management practices will lead to better health and more resilient livelihoods in targeted areas. Kenya RAPID activity components work in concert to promote water access and delivery and enhanced rangeland environments. Access to water for both domestic and livestock use is a critical component to the livelihoods of ASAL communities. Frequently, ASAL communities have limited availability of water resources, which can adversely affect WASH practices; instead of engaging in hygienic practices like handwashing, individuals may choose to use the water for other purposes. Poor water access can also limit livestock growth and inhibit economic growth for individuals. This adversely affects the health and economic wellbeing of communities and individuals. Kenya RAPID will endeavor to add to this growing body of knowledge during the life of the activity by testing appropriate hypotheses that will be specified at a later date. Figure 1 illustrates the causal linkages that USAID/E3/Water and USAID/Kenya envision for translating results under the activities into Kenya RAPID’s intended intermediate and final outcomes that this evaluation will be expected to examine. In this theory of change diagram, the improvement of governance frameworks and WASH coverage leads to improvements in water and sanitation access for individuals, water access for livestock, and rangeland-management practices. FIGURE 1.: KENYA RAPID THEORY OF CHANGE ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 49 3. Existing Performance Information Sources USAID/E3/Water, in coordination with USAID/Kenya, provided the evaluation team with the following documents related to existing performance information: • Kenya RAPID activity documents: • Kenya RAPID Year 1 Work Plan • Kenya RAPID fully executed Task Order • CARE Implementation Activities Progress Presentation The above list, which is non-exhaustive, highlights relevant data sources shared with the evaluation team. The evaluation team did not have access to the following documents, but they will be shared as the evaluation progresses: • All future quarterly project management and progress reports provided by each of the four implementing partners (CARE, Catholic Relief Services, Food for the Hungry, and World Vision) • Documents pertaining to selection and implementation of WASH, water coverage, and rangeland management projects • Annual USAID/Kenya WASH Survey materials In addition to information provided by USAID and each of the implementing partners, the evaluation team will need to access other types of secondary data, including administrative information on the municipalities from a variety of sources. This will likely involve accessing published government sources, or obtaining the information from Kenya RAPID staff who are knowledgeable about existing data for specific municipalities. The evaluation should also collect and analyze information related to WASH, water coverage, and rangeland management in Kenya, other activities to improve WASH services, issues ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 50 that may affect social cohesion and gender inclusion in Kenya, and other factors exogenous to Kenya RAPID that could influence activity impacts or survey responses. 4. Evaluation Purpose, Audience, and Intended Use Purpose This impact evaluation will allow the Agency to learn more about how WASH and rangeland management interventions can lead to improved health and economic outcomes. The results of this evaluation will be made widely available to encourage replication and/or scaling up of pilot activities within and beyond Kenya, as applicable. As such, this evaluation will apply USAID’s Evaluation Policy guidance with respect to using the most rigorous methods possible to demonstrate accountability for achieving results. The evaluation is also designed to capture practical lessons from USAID/Kenya’s experience with regard to increasing sustainable WASH programs and investment in water and rangeland resource management systems. Audience The evaluation is aimed at several audiences. First, the findings are expected to be of value from an accountability and learning standpoint to USAID/E3, particularly in the Office of Water, and USAID/Kenya. Second, findings and lessons learned from this evaluation will also be of interest to MWA, its partners, and other practitioners in the WASH and rangeland management sectors, including the Government of Kenya, which is seeking ways to improve water resource management, WASH coverage and quality of services. Finally, the evaluation will be of interest to donors, implementers, and scholars more generally by making an important contribution to the evidence base on WASH service delivery interventions. Intended Use This evaluation will be used to inform the design of future USAID programming that aims to improve the sustainability of WASH services to increase resilience and sustainable livelihoods for communities. Depending on the intervention/hypotheses USAID elects to examine through an impact evaluation, it may also contribute to a growing body of evidence about WASH effectiveness, to which other USAID evaluations are also contributing as are studies conducted by other institutions. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 51 5. Evaluation Questions The evaluation questions for Kenya RAPID are still in development. Ultimately, they will reflect USAID’s learning priorities for WASH and rangeland management investments and Agency programming for WASH and rangeland management. The evaluation is expected to focus on how ICT solutions can affect water management in drought prone areas. 6. Gender Considerations In line with USAID’s Gender Policy (ADS 203.3.1.5), the evaluation will consider gender-specific and differential effects of Kenya RAPID activities. The evaluation team will disaggregate access and participation data by gender at multiple points along the theory of change diagram to analyze the potential influence it has on pilot activities and outcomes. Data collected through surveys will be gender￾disaggregated to identify gender differences with respect to benefits and outcomes. The evaluation team will conduct further inquiry on gender themes as they emerge during data analysis. 7. Evaluation Methods Impact Evaluation Design Impact evaluations identify activity impact by comparing outcomes between activity beneficiaries to those of a control or comparison group of non-beneficiaries. The control or comparison group is intended to represent the counterfactual, or what would have happened in the absence of the Kenya RAPID intervention. As per the USAID Evaluation Policy, impact evaluations using experimental designs – whereby units are randomly assigned to treatment and control groups – provide the most rigorous evidence of activity impact, and this will be the preferred approach for the Kenya RAPID impact evaluation. Where randomized assignment is not feasible, quasi-experimental impact evaluation designs can be employed as an alternative. The evaluation team responding to this SOW will work with USAID/E3/Water, USAID/Kenya, and the implementing partner to develop a design that suits the objectives, timing, and constraints of Kenya RAPID. The evaluation team will produce an evaluation design proposal to be approved by USAID/E3/Water prior to any site selection or randomization. It is expected that the evaluation questions will be answered using an experimental or, if necessary, quasi-experimental design, and that a mixed-method approach may be suitable to answer the evaluation questions. Data Collection Methods USAID anticipates that data collection for this evaluation will involve the use of household-level surveys that cover all communities targeted for Kenya RAPID. This is likely to include a baseline survey that would be conducted before major interventions commence. The survey would collect information on basic the outcomes of interest that the evaluation will measure. The evaluation team responding to this SOW shall provide further details on data collection methods and the specific survey methodology in the evaluation design proposal, including proposing specific data collection methods on a question-by￾question basis. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 52 8. Data Analysis Methods In its evaluation design proposal, the evaluation team responding to this SOW should propose specific data analysis methods on a question-by-question basis, including the appropriate mix of methods necessary to estimate the impact Kenya RAPID has on the primary outcomes of interest. Potential data analysis methods include difference-in-difference and multivariate regressions. The evaluation design proposal should also explain what statistical tests will be conducted on data collected to address all evaluation questions, how qualitative data will be analyzed, and whether that analysis will allow the evaluation team to transform some data obtained from qualitative into quantitative form. The evaluation design proposal should also indicate and justify the evaluation team’s proposed sequencing of quantitative and qualitative data collection. For example, if key informant qualitative interviews are conducted during the endline data collection process, these lines of data may be collected and analyzed in parallel and only synthesized once data from all other sources are available. 9. Strengths and Limitations The strengths and limitations of the Kenya RAPID impact evaluation will depend on the final design proposed by the evaluation team in consultation with USAID and the implementing partner. The final design should reflect a rigorous approach to answering the evaluation questions and contribute to the global knowledge on water delivery and rangeland management practices. One key contribution of this evaluation is that it is expected to specifically test the impact of private sector engagement on improving access and quality of WASH services. Sample size, activity reach, and implementation fidelity could all create internal validity limitations for this evaluation. Ensuring that the sample size achieves sufficient statistical power will be critical for identifying impact and answering the evaluation questions. In addition, ensuring that randomization is done properly and random assignment, if applied, is systematic will improve the internal validity of the evaluation, but must be done in a transparent manner. Indirect contamination across treatment arms and comparison groups is always a possibility, which is why it is important for the evaluation team and the implementation team to coordinate from the outset. 10. Evaluation Deliverables The evaluation team expects to be responsible for the deliverables listed in Table 1. A final list of proposed deliverables and due dates will be included in the evaluation design proposal for USAID’s approval. TABLE 1. EVALUATION DELIVERABLES Deliverable Estimated Due Date 1. Concept Paper, describing design and methodological options to answer the evaluation questions TBD in consultation with USAID/E3/Water 2. Draft Evaluation Design Proposal TBD in consultation with USAID/E3/Water 3. Final Evaluation Design Proposal, including data collection and analysis methods, evaluation instruments, team composition, proposed timeline, and estimated budget TBD in consultation with USAID/E3/Water 4. Baseline Report o/a 60 days following completion of baseline data collection 5. Draft Evaluation Report o/a 60 days following completion of endline data collection ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 53 Deliverable Estimated Due Date 6. Final Evaluation Report o/a 21 days following receipt of USAID comments on Draft Evaluation Report All documents and reports will be provided electronically to USAID no later than the dates indicated in the approved evaluation design proposal. The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template. 11. Team Composition The evaluation design proposal should describe the specific composition and qualifications of the team members who will be carrying out this evaluation, including CVs for core team members. General qualifications and roles anticipated for core evaluation team are listed below. Local survey research firm(s) with experience in the conduct of household surveys at the village level and/or qualitative data collection may also support the evaluation team, as necessary. Principal Investigator. The Principal Investigator for this impact evaluation should hold a Ph.D. in a relevant economic development field. S/he will have previous experience with WASH programs and will have previously served as a team leader for one or more impact evaluation(s) that include a counterfactual. Familiarity with a range of impact evaluation designs and with USAID evaluation guidance will be sought for this position. Experience in publishing evaluation research in peer-reviewed journals is desirable, as is experience working in East Africa. A demonstrated ability to gather and integrate both quantitative and qualitative findings to answer evaluation questions is expected. Demonstrated experience managing multinational teams and producing highly readable reports for USAID and its developing country partner audiences on a timely basis is expected. This individual will be primarily responsible for the quality of the evaluation design and its execution, particularly with respect to the evidence obtained on questions involving causality and the attribution of outcomes to USAID’s intervention. This is not a full-time position. Evaluation Specialist. The Evaluation Specialist should have a graduate degree in a relevant social science field, and may be a Kenyan national. The individual will have sufficient previous experience with evaluations and other types of studies involving sample surveys to be actively engaged in efforts to oversee and ensure the quality of the evaluation's multiple rounds of household surveys, and ensure that data codebooks are clearly written and all study data prepared by local firms are properly transferred to USAID. Gender analysis experience is also desirable. This is not anticipated to be a full-time position. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 54 12. USAID Participation The desirability of USAID participation in evaluation activities such as data collection will be considered in consultation with USAID and the evaluation team, and any specific roles and responsibilities of USAID staff will be described in the evaluation design proposal. 13. Scheduling and Logistics The following table provides the originally anticipated timeframe for evaluation activities and deliverables. Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Concept Paper Scoping Trip Evaluation Design Proposal Survey Pre-Test Enumerator Training Baseline Data Collection Baseline Data Analysis and Report Oral Presentation of Baseline Findings Kenya RAPID Program Implementation Endline Data Collection and Analysis Endline Report Oral Presentation(s) Final Report Tasks FY 16 FY 17 FY 18 FY 19 FY 20 The evaluation team will be responsible for procuring all logistical needs such as work space, transportation, printing, translation, and any other forms of communication. USAID will offer some assistance in providing introductions to partners and key stakeholders as needed and will ensure the provision of data and supporting documents as possible. 14. Reporting Requirements The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note-preparing-evaluation￾reports). The final version of the evaluation report will be submitted to USAID and should not exceed 30 pages, excluding references and annexes. All members of the evaluation team will be provided with USAID’s mandatory statement of the evaluation standards they are expected to meet, shown in the following text box below, along with USAID’s conflict of interest statement that they sign and return before field work starts. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 55 USAID EVALUATION POLICY, APPENDIX 1 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action 15. Budget The evaluation team responding to this SOW will propose a notional budget in its concept paper for this evaluation, including cost implications of the methodological options proposed. A full detailed budget will then be prepared and included in the evaluation design proposal for USAID’s approval. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 56 ANNEX B: COMPARISON COUNTY FOLLOW-UP BOREHOLE ASSET SURVEY INSTRUMENT SweetSense designed this instrument to collect information on boreholes. The evaluation team used a slightly revised version as part of baseline data collection, which is provided below. Interviewer details: Name of the interviewer ID no. of the interviewer Name of the supervisor ID no. of the supervisor Introduction Good morning/afternoon! I am ____ from Ipsos, a survey and market research company. We are currently conducting a study to better understand water use in this community. Your opinion and knowledge would be incredibly helpful for supporting national efforts to address water management and drought resiliency. Many of the questions I will ask are related to the local water borehole and based on the current context here. There are no right or wrong answers, and please be assured that the information collected from you will be treated completely confidentially. ./ Subax wanaagsan / galab! Waxaan ahay ____ ka socda Ipsos, shirkad cilmi baaris ah iyo suuqayada. Hadda waxaan sameyneynaa daraasad si aan si fiican u fahanno isticmaalka biyaha ee bulshada ama deegankan. Fikraddaada iyo aqoontaadu waxay noqonaysaa mid aad u caawin karta si ay u taageerata dadaalada qaranka ee si wax loga qabta maareynta biyaha iyo wax ka qabadka adkeeysiga abaarta. Su'aalo badan oo aan weydiin doono waxay ku saab san yihin ceelesha biyaha ee dagankan ah oo ku salaysan xaaladda hadda jirta. Ma jiraan jawaabo sax ah ama khaldan, fadlan agaanteed laho in macluumaadka laga soo ururiyey adiga laguula dhaqmi doono si qarsoodi ah. Water supply facilities General Information Water system (distribution scheme linked to this update) A1. Name/description of the water system /Magaca / sharraxaadda nidaamka biyaha ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 57 A2. Unique water system ID (map A1 &A2)/ Nidaamka Biyaha ee Gaarka ah (Khariidad A1 A3. GPS of the borehole location (This is to be done at the borehole exact location.) A4. Take picture of the water system: (INTERVIEWER NOTE: THE PICTURE SHOULD CAPTURE THE WHOLE BOREHOLE SYSTEM PLUS THE SURROUNDINGS [I.E., TANKS, KIOSK] IN ONE CAPTION.) A5. County//Ismaamulka  Baringo  Kitui  Laikipia  Mandera  Meru  Samburu  Tana River  West Pokot A6. Sub-County/ ismaamulka hoose (insert list) A7. Village (insert list)/ Xaafada(( geli liiska)) A8. Rural/Urban (map)/ ) /Baadiyaha/Magaalada(khariidada)  Rural  Urban A9. Local Officer Name ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 58 (INTERVIEWER NOTE: RECORD THE NAME OF THE PERSON RESPONSIBLE FOR BOREHOLE MAINTENANCE/PERSON IN CHARGE.) A10. Local Officer Position  Operator  Water committee chairperson  Water Committee Officials / Saraakiisha Guddiga Biyaha  Other (specify) A11. Local officer telephone number Users/ Hali ya matumizi B1. Total number of households currently served from the scheme? / tirade guud ee iminka isticmasha nidaamkas B6. Is the water scheme used for livestock? / nidaamka biyaha loo isticmaala xoolaha  Yes /haa  No /Maya If Yes/hadii haa tahay B11. Is the water scheme used for water trucking?/ Mashruuca biyaha ma loo isticmaalaa biyo dhaamin?  Yes/haa  No/ maya (SKIP TO C3) ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 59 B13. How many days per week does the water trucking occur? /Meega jeer toddobaadikii ayaa biyo dhaaminaya? Borehole Information C3. Physical state of the well/borehole / xalada guud ee ceelka (INTERVIEWER NOTE: THIS INFORMATION REFERS TO THE PERCEPTION OF THE COMMUNITY ON THE BOREHOLE.)  Function well/ Si ficaan u saqeeya  Poor /hoseesa  Doesn't function / ma shaqeeyo  Unsure / (DO NOT READ OUT) /ma hubo(HAA AKHRIN) C4. Type of Power / Nooca tamarta (MULTIPLE RESPONSE)  Generator / Matoor  Solar / oorahada  Utility Power (Grid power) /isticmaalka tamarta(tamarta roobka) C7. Is there a water meter at the water source? Ma yeela cabirka biyaha goobta biyaha?(WAREESTAHA HA EEGA) (INTERVIEWER TO OBSERVE.)  Yes/haa  No (SKIP TO C11)/Maya(u kac c13) Scheme Functionality / Shaqeeynta nidaamka C11. Is the water scheme currently functional? ?/ nidaamka biyaha iminka ma shaqeeynaya? (INTERVIEWER NOTE: THIS REFERS TO WHETHER THE BOREHOLE IS PRODUCING WATER. IF SO, IS THE AMOUNT OF WATER PRODUCED AS PER DESIGN OR IS IT REDUCED YIELD?)  Functional (producing as designed) (SKIP TO C15) /Shaqeeynta(wax soo sarka sida logu qasdi)  Partially functional (reduced yield) xogaaha shaqeena(wax soo sarka oo is dhimi)  Non-functional/ aan shaqeenin ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 60  Abandoned SKIP TO SECTION E) / laga tagay (u kac E) If the water scheme is partially functional or non- functional /:Hadii nidaamka biyaha ee xoogaha ama aan shaqeenini: C12. Please take a picture illustrating the non-functionality or partial functionality /(INTERVIEWER NOTE: TAKE MORE THAN ONE PICTURE IF MORE THAN ONE PART OF THE SYSTEM IS NOT FUNCTIONING.) fadlan ka quad sawir tusinaaya shageeyn laanta ama xoogaha ka shageeyni If the water scheme is partially functional or non-functional/ Hadii nidaamka biyaha ayan shaqeenin ama xogaha shaqeeyna: C13. Main cause of non-functionality or partial functionality / Sababta ugu weyn keentay shaqeeynta laanta ama uu xogaha ugu shaqeeyna: :JAWAABA BADAN (MULTIPLE RESPONSE)  Insufficient water at source /Biya yari ka jirta isha ceelka  Distribution pipeline or tap failure/ dummoyinka qeeybiya oo fashalmi  No gas for generator / motoorki oo ka dhammaday gaaski  Generator failure / matoorki oo fashalmi  Grid power failure / tamarta griidka oo fashalmay  Solar power failure / tamarta qoraxda oo fashalmay  Submersible pump failure/ doloolkadummoyinka oo fashilmi  Switchboard (electrical) failure / failure/qalabka korontada oo ka fashalmi  Other (please specify) / wax kale(fadlan fahfaahi) If the water scheme is partially functional or non-functional: C14. Number of months since non-functional Idadi ya miezi tangu kituo hiki kufanya kazi (SKIP TO QUESTION D1 IF C11 IS CODED NON-FUNCTIONAL) ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 61 If the water scheme is currently functional or partially functional/ hadii nidaamka biyuhu uu shaqeenin ama xogaha ka shaqeyniya: C15. Was the water scheme out of service one or more days in the last month? Nidaamka biyaha musan shaqeeynin hal malmood iyo wixi ka badan bisha aan soo dhafni ? WARESTAHA:HUBI INUU JAWAAB BIXIYAHA UU FAHMA WAQTIGA INTUSAN KA JAWAABIN (INTERVIEWER: ENSURE THAT RESPONDENT UNDERSTANDS THE TIME FRAME BEFORE RESPONDING [I.E., THE PAST 4 WEEKS FROM THE DAY OF INTERVIEW].)/ Nidaamka biyaha musan shaqeeynin hal malmood iyo wixi ka badan bisha aan soo dhafni ? WARESTAHA:HUBI INUU JAWAAB BIXIYAHA UU FAHMA WAQTIGA INTUSAN KA JAWAABIN  Yes /Haa  No (SKIP TO C17)/ Maya(U Kac C17 C16. If Yes, number of days the scheme was out of service in the last month/ Hadii haa tahay, inta maalmood uu nidaamka aha shaqaa laan bisha laso dhaafay C17. Was the water scheme out of service one or more days in the past 12 months? (INTERVIEWER: ENSURE THAT RESPONDENT UNDERSTANDS THE TIME FRAME BEFORE RESPONDING.)/ Nidaamka biyaha musan shaqeeynin hal malmood iyo wixi ka badan sanadki aan soo dhaafni ? WARESTAHA:HUBI INUU JAWAAB BIXIYAHA UU FAHMA WAQTIGA INTUSAN KA JAWAABIN  Yes /Haa  No La (SKIP TO C20)/Maya (ukac C18. How many times was the water scheme broken in the past 12 months? / imisaa mar ayuu nidaamka biyaha uu jabay sanadki lasoo dhaafay?  Has never broken in the past 12 months / wali ma jabin sanadkii lasoo dhaafay (SKIP TO C20) C18a. What was the nature of the failure? (SELECT MULTIPLE) maxaa uu ahay dhibaatada la xiriira fashilitaanka?(xula jawaaba badan) ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 62  Pump failure /fasilitaanka dhuumaha  Generator/power failure /fashilka matoorka/fashilka tamarta  Pipe failure /fashilka dhuumaha  Tank failure/ fashilka taangiga  Other (please specify) Wax kale C19. Describe the functionality problem/s over the past 12 months. /fahfaahi dhibaatooyinka la xiriira shaqeynta sandki laso dhaafay? C20. Does the scheme have any emerging problems that might lead to non-functionality in the near future?/ nidaamka byaha maleeyhay dhibaatooyin hadaa soo if baxayo oo laga yaabo iney shaqeynta dhib ugeystaan mustaqbalka dhow?  Yes /haa  No (SKIP TO D1 / Maya(SKIP TO D1) C21. If yes, please describe the problems that might lead to non-functionality in the near future. Hadii haa tahay/?/ fadlan fahfaahi dhibaatooyinka laga yaabo iney u hor seedaan shaqeyn laan mustaqbalka dhow Scheme Usage Patterns Mpango wa Matumizi ya Mfumo D1. How many days per week does the pump run on average? imisaa maalin isbuucii D2. How many hours per day does the pump run on average? ?/ imisaa saacadood maalinti ayu bamka biyaha soo jiido socdaa isku celcelis ahaan? D3. Is this a seasonal water scheme that commonly fails in the dry season? / nidaamkani mayahay mid sanadla ah oo inta badan go’a xiliyada roobabka jirin?  Yes/haa  No /Maya ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 63  Don’t know (DO NOT READ OUT)/ GARAN MAYO(HAA KHRIN) D4. Does frequency of use of the pump depend if it is wet/rainy season? / xawaaraha loo isticmaalo bamkan biyaha miyuu ku xiranyahay xiliyada roobabka da’aan misna abaaraha ah?  Yes /Haaa  No ?Maya  Don’t know (DO NOT READ OUT)/ GARAN MAYO(HAA KHRIN) D5. During wet/rainy seasons, how many days a week does the pump run on average? /xiliyada roobabka da’ayaan meeqa maalin ayuu bamka shaqeeyaa isbuuci? D6. During wet/rainy seasons, how many hours per day does the pump run on average? /xiliyada roobabka da’ayaan meeqa sac ayuu bamka shaqeeyaa isbuuci? Pump G1. Pump Controller Manufacturer/ Soo saaraha xakameeyah bambada /dhumooyinka (INTERVIEWER NOTE: SEE TECHNICAL DETAILS FROM THE PLACARDS. IF NOT AVAILABLE, CHECK WITH COUNTY/WATER AUTHORITIES FROM THE COUNTY ENGINEERS, WATER RESOURCE AUTHORITY.) WARESTAHA XASUUSNOW:EEG CALAAMADAHA MACLUMAADKA FARSAMADA /HADII UU YEELIN WAA INAD KA EEGTID MACLUMAADKA MAAMULKA BIYAHA KAUNTIGA.INJINEERKA KAUNTIGA,GUDIGA BIYAHA IYO QEYRAADKA)  Grundfos  Lorentz  Davis & Shirtliff  ABB  Dayliff  Tormak ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 64  Other (please specify) )/Wax kale(fadlan fahfaahi)  Don’t Know/)/ GARAN MAYO(HAA KHRIN) Sensor Information / Macluumaadka Xasaasiga ah I1. Is there a sensor installed at the site? ?/ ma jira qalabka ogaanshaha lagu xirahay goobta(WARESTAHA FADLAN XASUUSNOW TANI TIXRAACA UMA AHAOGSHANSHA DHUUMAHA) (INT: VERIFY IF THE SENSOR IS ACTUALLY CLAMPED TO THE LIVE WIRE )  Yes /Haa  No La >>>>> skip to I7 /MAYA(U KAC 17) I5. Take a picture of the sensor (INT: TAKE A PICTURE OF INSTALLED SENSOR BEFORE UNCLAMPING IT, BROKEN OR VANDALIZED SENSOR AND OPEN PUMP CONTROLLER ON SITE IF THE SENSOR IS NOT ON SITE ) /Sawirka xiritaanka qalabka ogaanshaha I7. If No, what are the reasons to why the sensor is not installed/ Hadday Maya tahay, waa maxay sababaha keenay in qalabka uusan looga rakibin.  Unclamped and set aside :(please ask reason for uninstalling)  Broken  Stolen  Other specify 17.1 For how long has the sensor not been installed? Record in months. If less than 1 month record zero)/ Muddo intee la'eg ayaa qalabka aan la rakibin? Ku diiwan geli bilo. Haddii wax kayar 1 bilood diiwaan geli eber) I8. Was data downloaded from the sensor? / Xogta miyaa laga soo qaaday qalabka? ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 65 (DOWNLOAD DATA FROM SENSOR WHETHER INSTALLED OR NOT)  Yes (SKIP TO I10) / Haa (u kac 110)  No /Maya La  Sensor not at site ( logic check . only applicable if 17 is any other code other than code 1 or 2 ) >>>>>> 113 /qalabka meesha usan yaallin (hubinta caqligal ayaa lagu dabaqi karaa oo keliya haddii 17 uu jiro koodh kale oo aan ka ahayn koodhka 1 ama 2) >>>>>> 113 I9. If No, what are the reasons for not downloading the data./ hadii maya tahay waa maxay sababta aad u helin xogta  Sensor is broken/ qalabka la jabiyay  Sensor malfunctioning. Cannot download data to the computer/ qalabka muu shaqeenayo. Xogta kombuutarka ushan gashani karin  Other (please specify)/ wax kale (fadlan fahfaahi) I10. Was the same sensor installed back?  Yes /Haa  No La (SKIP TO I12)/Maya 111. If Yes, take a Picture of sensor installation. (INT: TAKE A PICTURE OF THE SENSOR IMMEDIATE AFTER REINSTALLING )/ Hadday Haa tahay, qaado Sawir rakibaadda qalabka.(DIG;KA QAADO SAWIR QALABKA KADIIB MARKA LA SAMEEYA IF YES SKIP TO QUESTION J1 112. If No, what is the reason of not installing the sensor. ./hadii maya ,waa maxay sababaha ee loo rakibin qalabka I13. was the sensor replaced?/ ?/qalabka ma la badalay ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 66  Yes / Haa  No (SKIP TO I15) / Maya 114. If Yes, take a Picture of the sensor after replacing. (INT: TAKE A PICTURE OF THE SENSOR IMMEDIATE AFTER INSTALLING )/ haddii haa tahay ka qaado sawir qalabka kadib marka la rakibay(DIG:KA QAADO SAWIR KADIB QALABKA LA SAMEEYA) 115. If No, what is the reason of not replacing the sensor. /haddii maya ,waa maxay sababta qalabka loo badalin Management Usimamizi J1. Management Body  Utility  WASHCO  No management organization  Others (specify)_______________ If management body is utility or WASHCO: J2. Current Management Status  WASHCO or utility is active  WASHCO or utility is not active Tariffs Malipo/Canshuuraha K1. Type of tariff system (most common)nooca canshuurta (Sida badan)  Fixed tariff per visit /canshuur joogto ah mar walboo lasoo booqdo  Fixed tariff per week /canshuur joogto ah isbuuciba  Fixed tariff per month / canshuur joogto ah bishiba  Fixed tariff per half year /canshuur joogto ah sanadki barkiisa  Fixed tariff per year /canshuur joogto ah sandkiba  Tariff per jerrycan (20 litre) )/cashuurta jergan kasta (20 liitar)  Tariff per cubic meter (m3) )/canshuurta miitar sadex lab kasta (m3) ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 67  Ad hoc contributions Malipo maalum inayotozwa kwa dharura (SKIP TO K3)  No payment (SKIP TO K3)/ lacaq laan(U KAC k3)  If type of tariff system (most common) is one of fixed tariff per visit, fixed tariff per week, fixed tariff per month, fixed tariff per half year, fixed tariff per year, tariff per jerrycan (20 litre), or tariff per cubic meter (m3): )/hadii nooca canshuurta (sida badan) uu yahay mid joogto ah mar walboo lasoo booqdo, mid joogto ah isbuuc walbo, mid joogto ah bil walbo, mid joogto ah kala barka sand walbo, mid jooto ah sanad walbo, canshuurta jerganka (20 liitar) canshuurta miitarka sadex laabka (m3) K2. What is the tariff amount (in KES)? /waa imisaa qiimaha canshuurta (KES)? K3. Is there a special tariff for livestock? /ma jirtaa canshuur gaar ah oo laga qaado xoolaha (calaa neef)  Yes /yaa  No (SKIP TO K8) )/Maya (U KAC K8) Other notes/Observations ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 68 ANNEX C: REFERENCES Abadie, A., & Imbens, G. W. (2006). Large sample properties of matching estimators for average treatment effects. econometrica, 74(1), 235-267. doi:doi:10.1111/j.1468-0262.2006.00655.x Altman, D. G. J. T. S. (1985). Comparability of randomised groups. The Statistician: Journal of the Royal Statistical Society. Series D, 34(1), 125-136. Cronk, R., & Bartram, J. (2017). Factors influencing water system functionality in Nigeria and Tanzania: A regression and Bayesian Network Analysis. Environmental Science & Technology, 51(19), 11336- 11345. doi:10.1021/acs.est.7b03287 Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053-1062. doi:10.1080/01621459.1999.10473858 Feinerer, I., Hornik, K., & Meyer, D. (2008). Text mining infrastructure in R. 2008, Journal of Statistical Software, 25(5), 1-54. doi:10.18637/jss.v025.i05 Fisher, M. B., Shields, K. F., Chan, T. U., Christenson, E., Cronk, R. D., Leker, H., ... & Bartram, J. (2015). Understanding handpump sustainability: Determinants of rural water source functionality in the Greater Afram Plains region of Ghana. Water Resources Research, 51(10), 8431-8449. doi:10.1002/2014WR016770 Funk, C. C., Peterson, P. J., Landsfeld, M. F., Pedreros, D. H., Verdin, J. P., Rowland, J. D., . . . Verdin, A. P. (2014). A quasi-global precipitation time series for drought monitoring. US Geological Survey data series, 832(4), 1-12. Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. (2016). Impact evaluation in practice. The World Bank. Harvey, P. A., & Reed, R. A. (2007). Community-managed water supplies in Africa: sustainable or dispensable? Community Development Journal, 42(3), 365-378. Ho, D., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. 2011, Journal of Statistical Software, 42(8), 28. doi:10.18637/jss.v042.i08 Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge: Cambridge University Press. Kenya, R. O. (2014). Ending drought emergencies: Common programme framework for drought risk management. Retrieved from Nairobi. King, G., & Nielsen, R. (2018). Why propensity scores should not be used for matching. Political Analysis. Nagel, C., Beach, J., Iribagiza, C., & Thomas, E. A. (2015). Evaluating cellular instrumentation on rural handpumps to improve service delivery—A longitudinal study in rural Rwanda. Environmental Science & Technology, 49(24), 14292-14300. doi:10.1021/acs.est.5b04077 Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55. doi:10.1093/biomet/70.1.41 Thomas, E. A., Needoba, J., Kaberia, D., Butterworth, J., Adams, E.C., Odur, P., Macharia, D., Mitheu, F., Mugo, R., Nagel, C. (2019). Quantifying increased groundwater demand from prolonged drought in the East African Rift Valley. Submitted to Lancet Planetary Health. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 69 Walters, J. P., & Javernick-Will, A. N. (2015). Long-Term functionality of rural water services in developing countries: A system dynamics approach to understanding the dynamic interaction of factors. Environmental Science & Technology, 49(8), 5035-5043. doi:10.1021/es505975h World Health Organization. (2015). Progress on sanitation and drinking water: 2015 update and MDG assessment. World Health Organization. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 70 ANNEX D: RAINFALL ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 71 ANNEX E: SUMMARY STATISTICS COVARIATE DESCRIPTIVE STATISTICS ACROSS MATCHED SAMPLES ALL PERIODS: MEAN ABOVE AND SD BELOW FOR EACH Comparison n = 132 Treatment n = 69 Comparison n = 60 Treatment n = 60 Comparison n = 17 Treatment n = 29 Comparison n = 120 Treatment n = 60 Unmatched Model 1 Optimal Matching Model 2 Nearest Neighbor with 0.25 Caliper Cutoff Model 3 2-1 Optimal Matching # of HHs 857.65 888.05 1064.52 948.12 587.13 849.84 940.34 948.12 1414.86 1482.37 1739.68 1552.73 451.35 1811.50 1476.00 1552.73 Borehole Pump On % 0.18 0.25 0.16 0.24 0.13 0.18 0.19 0.24 0.25 0.24 0.22 0.24 0.19 0.23 0.25 0.24 Borehole Rainfall (mm) 1.69 0.85 1.33 0.84 1.37 0.71 1.67 0.84 6.28 4.99 6.02 4.99 5.88 4.38 6.24 4.99 Construction Year 2008 2007 2008 2007 2007 2011 2008 2008 13.42 8.74 16.18 8.75 12.20 4.29 13.57 8.75 Fixed Tariff 0.79 0.97 0.87 0.97 1.00 0.97 0.81 0.97 0.41 0.17 0.34 0.18 0.00 0.18 0.40 0.18 Generator Power 0.66 0.39 0.91 0.38 0.90 0.76 0.66 0.38 0.47 0.49 0.29 0.49 0.29 0.43 0.47 0.49 Hybrid Power 0.00 0.49 0.00 0.50 0.00 0.00 0.00 0.50 0.00 0.50 0.00 0.50 0.00 0.00 0.00 0.50 Livestock Use 0.83 0.71 0.89 0.70 0.92 0.69 0.81 0.70 0.38 0.46 0.31 0.46 0.27 0.46 0.39 0.46 Service Area 3.43 3.81 3.65 3.69 2.36 3.93 3.39 3.69 ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 72 Comparison n = 132 Treatment n = 69 Comparison n = 60 Treatment n = 60 Comparison n = 17 Treatment n = 29 Comparison n = 120 Treatment n = 60 Unmatched Model 1 Optimal Matching Model 2 Nearest Neighbor with 0.25 Caliper Cutoff Model 3 2-1 Optimal Matching 5.20 4.11 6.25 4.01 2.62 3.82 5.16 4.01 Solar Power 0.15 0.12 0.09 0.12 0.10 0.24 0.15 0.12 0.36 0.32 0.29 0.32 0.29 0.43 0.36 0.32 Utility Power 0.19 0.00 0.00 0.00 0.00 0.00 0.18 0.00 0.39 0.00 0.00 0.00 0.00 0.00 0.39 0.00 Water Test 0.80 0.82 0.71 0.85 0.77 0.83 0.80 0.85 0.40 0.38 0.45 0.36 0.42 0.38 0.40 0.36 ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 73 ANNEX F: MATCHING REVISITED As noted in the report, the evaluation tested three matching approaches: optimal matching, nearest neighbor with a 0.25 caliper cutoff, and two-to-one matching, using an optimal approach. This section highlights the overlap and balance for the nearest neighbor and two-to-one approaches. The main body of the report contains the selected approach, optimal matching. As shown in the figure below, nearest neighbor matching improved overlap, suggesting a similar probability of receiving treatment based on observed confounders. However, the sample size under this method is quite small at 15 comparison boreholes and 28 treatment boreholes. This sample size is too small to reliably detect an effect. FIGURE 1. NEAREST NEIGHBOR PROPENSITY SCORE DISTRIBUTIONS BEFORE AND AFTER MATCHING The covariate balance for nearest neighbor matching is improved on 10 of 14 variables; however, it is notably worse on the new miles to county seat variable. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 74 FIGURE 2. NEAREST NEIGHBOR COVARIATE BALANCE ACROSS MATCHED AND UNMATCHED SAMPLES Two-to-one matching yields little improvement in overlap or balance. This is due to the small number of available matches from the overall population of 127 comparison and only 60 treatment boreholes. Two-to-one matching would likely be more effective if the number of boreholes overall were much higher. Instead, this approach serves to simply filter out the boreholes in the comparison group that have the lowest propensity scores. As shown below, however, there is very little difference in overlap, with 120 comparison boreholes and 60 treatment boreholes in this sample. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 75 FIGURE 3.TWO-TO-ONE PROPENSITY SCORE DISTRIBUTIONS BEFORE AND AFTER MATCHING This model improves balance across four variables within the 0.25 threshold, but provides limited additional values compared to the unmatched sample. FIGURE 4. TWO-TO-ONE COVARIATE BALANCE ACROSS MATCHED AND UNMATCHED SAMPLES ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 76 ANNEX G: REGRESSION RESULTS Variable Model 1 Model 2 Matched Results Unmatched Results Matched Results Unmatched Results N: 37,335; R² = 0.19 N: 55,959; R² = 0.24 N: 10,456; R² = 0.21 N: 15649; R² = 0.26 Estimate p-value CI Estimate p-value CI Estimate p-value CI Estimate p-value CI Intercept -0.06 0.39 -0.19 0.06 0.07 0.24 -0.03 0.16 -0.12 0.23 -0.28 0.04 0.04 0.59 -0.08 0.15 0.07 0.06 0.10 0.07 Treatment*Drought 0.04 0.18 -0.01 0.08 0.04 0.14 -0.00 0.08 0.03 0.04 Drought 0.02 0.09 0.00 0.05 0.02 0.00 0.01 0.04 0.01 0.01 Treatment 0.17 0.00 0.10 0.25 0.08 0.05 0.01 0.16 0.29 0.00 0.18 0.39 0.20 0.00 0.11 0.30 0.05 0.04 0.06 0.06 # of HHs 0.01 0.52 -0.02 0.05 0.03 0.12 -0.00 0.06 0.01 0.73 -0.03 0.05 0.03 0.18 -0.01 0.07 0.02 0.02 0.03 0.02 Livestock Use 0.04 0.29 -0.02 0.09 0.02 0.47 -0.02 0.05 0.07 0.18 -0.02 0.15 0.02 0.61 -0.04 0.07 0.03 0.02 0.05 0.03 Service Area 0.04 0.10 -0.00 0.08 0.07 0.02 0.02 0.12 0.06 0.05 0.01 0.10 0.08 0.04 0.02 0.14 0.02 0.03 0.03 0.04 Hybrid Power 0.10 0.06 0.01 0.18 0.10 0.06 0.01 0.18 0.04 0.55 -0.07 0.14 0.04 0.52 -0.06 0.15 0.05 0.05 0.06 0.06 Solar Power 0.03 0.42 -0.03 0.09 0.03 0.26 -0.01 0.07 -0.01 0.84 -0.10 0.08 0.01 0.76 -0.05 0.08 0.04 0.02 0.05 0.04 Utility Power 0.21 0.00 0.11 0.32 0.24 0.00 0.14 0.34 0.06 0.06 ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 77 Variable Model 1 Model 2 Matched Results Unmatched Results Matched Results Unmatched Results N: 37,335; R² = 0.19 N: 55,959; R² = 0.24 N: 10,456; R² = 0.21 N: 15649; R² = 0.26 Estimate p-value CI Estimate p-value CI Estimate p-value CI Estimate p-value CI Construction Year -0.03 0.28 -0.06 0.01 0.00 0.81 -0.04 0.03 -0.02 0.51 -0.07 0.03 -0.01 0.79 -0.05 0.04 0.02 0.02 0.03 0.03 Tariff (Y/N) 0.04 0.04 0.01 0.08 0.03 0.30 -0.02 0.09 0.07 0.02 0.02 0.12 0.05 0.15 -0.01 0.10 0.02 0.03 0.03 0.03 Water Testing (Y/N) 0.04 0.40 -0.04 0.11 0.02 0.50 -0.03 0.08 0.06 0.25 -0.03 0.15 0.04 0.32 -0.03 0.11 0.04 0.04 0.05 0.04 Miles to County Seat -0.02 0.27 -0.06 0.01 -0.01 0.41 -0.04 0.01 -0.02 0.38 -0.06 0.02 -0.01 0.50 -0.05 0.02 0.02 0.02 0.03 0.02 Average Rainfall at borehole -0.02 0.00 -0.03 -0.01 -0.02 0.00 -0.03 -0.02 -0.01 0.12 -0.01 0.00 -0.01 0.01 -0.02 -0.00 0.00 0.00 0.00 0.00 Garissa -0.15 0.00 -0.21 -0.08 -0.16 0.00 -0.21 -0.10 -0.24 0.00 -0.33 -0.14 -0.23 0.00 -0.31 -0.15 0.04 0.03 0.06 0.05 Isiolo -0.22 0.00 -0.32 -0.12 -0.22 0.00 -0.30 -0.14 -0.31 0.00 -0.46 -0.17 -0.31 0.00 -0.43 -0.19 0.06 0.05 0.09 0.07 Kitui 0.16 0.00 0.12 0.20 0.07 0.00 0.03 0.10 0.14 0.00 0.09 0.19 0.07 0.00 0.03 0.11 0.03 0.02 0.03 0.02 Laikipia 0.06 0.00 0.03 0.10 -0.11 0.00 -0.13 -0.09 0.08 0.00 0.05 0.12 -0.09 0.00 -0.12 -0.07 0.02 0.01 0.02 0.01 Mandera 0.21 0.00 0.12 0.30 0.11 0.01 0.04 0.17 0.28 0.00 0.18 0.38 0.19 0.00 0.11 0.27 0.05 0.04 0.06 0.05 Marsabit -0.01 0.67 -0.03 0.02 -0.01 0.39 -0.03 0.01 0.00 0.91 -0.03 0.03 0.00 0.98 -0.03 0.02 0.01 0.01 0.02 0.02 ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 78 Variable Model 1 Model 2 Matched Results Unmatched Results Matched Results Unmatched Results N: 37,335; R² = 0.19 N: 55,959; R² = 0.24 N: 10,456; R² = 0.21 N: 15649; R² = 0.26 Estimate p-value CI Estimate p-value CI Estimate p-value CI Estimate p-value CI Meru 0.09 0.00 0.07 0.12 0.01 0.68 -0.03 0.05 0.11 0.00 0.07 0.14 0.04 0.12 -0.00 0.09 0.01 0.03 0.02 0.03 Samburu 0.09 0.01 0.04 0.15 -0.01 0.64 -0.05 0.03 0.11 0.00 0.05 0.17 0.02 0.49 -0.03 0.07 0.03 0.02 0.04 0.03 Tana River 0.01 0.76 -0.04 0.05 -0.06 0.03 -0.11 -0.02 0.01 0.88 -0.05 0.07 -0.04 0.23 -0.10 0.02 0.03 0.03 0.04 0.03 Turkana 0.02 0.27 -0.01 0.06 0.01 0.31 -0.00 0.02 -0.03 0.32 -0.07 0.02 -0.07 0.00 -0.10 -0.04 0.02 0.01 0.03 0.02 West Pokot 0.18 0 0.11 0.25 0.22 0 0.13 0.31 0.04 0.05 ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 79 ANNEX H: IMPLEMENTATION FIDELITY MONITORING QUESTIONNAIRE Quarterly RAPID Intervention Monitoring Checklist Date: County: Please indicate the status of each of the following Kenya RAPID intervention activities: 1. Installation of sensors on strategic / EDE boreholes: a. # sensors installed: b. # sensors functional: c. Comments: 2. Clarification of roles around EDE borehole maintenance: a. Have a set of individuals been identified who are in charge of identifying and responding to functionality problems with EDE boreholes? ___ Yes  See table below ___ No 3. Status of data dashboard: a. Has a customized data dashboard been developed for this county? __ Yes __ No b. Have individuals in charge of EDE borehole maintenance been trained on use of the data dashboard? c. Are the individuals trained on data dashboard or equivalent using the dashboard? 4. Identification of county-level individual(s) in charge of EDE borehole maintenance: Individual Name Organization Position / Job Title Role in EDE maintenance Trained in dashboard use? 5. Designated budget for EDE borehole repairs: a. Has a designated budget been established to support repairs of EDE / strategic boreholes? ___Yes ___ No b. What is the total amount allocated to this fund annually? c. What is the source of these funds? ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 80 d. Who is in charge of managing these funds? 6. Are there non-sensor, non-dashboard related Kenya RAPID activities that have affected borehole repair response, budget allocation, or maintenance? ____Yes _____No Comments: 7. Comments: a. Please provide any additional information on the status of the Kenya RAPID sensor￾based intervention in this county, including successes and challenges. ROUND 1I REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 1 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524