PHOTO CREDIT: JACOB PATTERSON-STEIN ROUND 1 EVALUATION REPORT: IMPACT EVALUATION OF THE KENYA RESILIENT ARID LANDS PARTNERSHIP FOR INTEGRATED DEVELOPMENT ACTIVITY MARCH 22, 2019 This publication was produced for review by the United States Agency for International Development. It was prepared by Tetra Tech. ACKNOWLEDGEMENTS The evaluation team thanks the families, water managers, and local leaders in the counties who shared their time to participate in this study. The National Drought Management Authority and the Kenyan Ministry of Water and Sanitation both deserve praise for their assistance, cooperation, and patience with the evaluation process and team members. Special thanks to the data collection teams from Ipsos Public Affairs Kenya for conducting the water borehole surveys, assembling a competent team to install sensors, and quickly responding to data-related questions, and to the Aquaya Institute for conducting qualitative data collection. The team is grateful to the Kenya Resilient Arid Lands Partnership for Integrated Development Activity staff for their continued collaboration, cooperation, and communication. The team could not have implemented the evaluation without the ongoing and patient cooperation of SweetSense staff, particularly Evan Thomas, Lauren Stover, and Christian Muragijimana. The team also acknowledges Mary Gichuhi for her quality oversight during data collection, translation support, and general guidance as the local coordinator for this evaluation. Finally, the evaluation team appreciates the technical review and support for this report from Tetra Tech’s Siobhan Girling and Morris Israel and Aquaya’s Rachel Peletz and Joyce Kisiangani. The team also thanks Elizabeth Jordan (USAID/E3/Water), Vicky Liyai (USAID/Kenya), Martin Mulongo (USAID/Kenya), Walter Ong’eng’a (USAID/Kenya), and Nicholas Owuor (USAID/Kenya) for their continued support of this study. Prepared by: Dr. Katherine Dickinson (Principal Investigator, University of Colorado) Jacob Patterson-Stein (Evaluation Coordinator, MSI, A Tetra Tech Company) Prepared for the United States Agency for International Development by the Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) project under Task Order number AID-OAA-TO-16-00016 of the Water and Development Indefinite Delivery Indefinite Quantity Contract (WADI IDIQ), contract number AID-OAA-I-14-00068. Tetra Tech Contacts: Morris Israel, Project Director morris.israel@washpals.org Jeff Albert, Deputy Project Director jeff.albert@washpals.org Jonathan Annis, Project Manager jonathan.annis@tetratech.com Tetra Tech 1320 North Courthouse Road, Suite 600, Arlington, VA 22201 Tel: 703 387-2100, Fax: 703-414-5593 www.tetratech.com/intdev Cover Photo: Camels drink from strategic borehole in Kenya, Jacob Patterson-Stein, MSI. ROUND 1 EVALUATION REPORT IMPACT EVALUATION OF THE KENYA RESILIENT ARID LANDS PARTNERSHIP FOR INTEGRATED DEVELOPMENT ACTIVITY MARCH 22, 2019 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY i CONTENTS ACKNOWLEDGEMENTS III ACRONYMS III EXECUTIVE SUMMARY V 1 INTRODUCTION 1 2 KENYA RAPID ACTIVITY BACKGROUND 1 2.1 NATIONAL CONTEXT 1 2.2 KENYA RAPID ACTIVITY DESCRIPTION 2 2.3 DEVELOPMENT HYPOTHESIS AND THEORY OF CHANGE 3 2.4 PROJECT IMPLEMENTATION STATUS 5 3 EVALUATION BACKGROUND PURPOSE, AUDIENCES, AND USES 8 3.1 PURPOSE 8 3.2 AUDIENCE 8 3.3 INTENDED USE 8 4 EVALUATION DESIGN 9 4.1 EVALUATION QUESTIONS 9 4.2 EVALUATION DESIGN 9 4.3 ROUND 1 DATA COLLECTION 11 4.3.1 QUANTITATIVE DATA COLLECTION 11 4.3.2 QUALITATIVE DATA COLLECTION 14 4.3.3 CHALLENGES ENCOUNTERED DURING QUANTITATIVE DATA COLLECTION 17 5 QUANTITATIVE FINDINGS 18 5.1 GEOGRAPHIC AND SERVICE CONTEXT 18 5.2 BOREHOLE CHARACTERISTICS 21 5.3 BOREHOLE TARIFFS AND OTHER MANAGEMENT METRICS 24 6 QUALITATIVE FINDINGS 27 6.1 QUALITATIVE SAMPLE CHARACTERISTICS 27 6.2 QUALITATIVE DATA ANALYSIS METHODS 27 6.3 KEY FINDINGS AND THEMES 27 6.4 BOREHOLE ACCESS AND USE 29 6.4.1 EXCESSIVE DEMAND 30 6.4.2 TRAVEL CONSTRAINTS 32 6.4.3 BOREHOLE RELIABILITY AND FEES 32 6.5 STAFF ROLES AND MANAGEMENT 33 6.6 RESOURCES FOR BOREHOLE REPAIRS 34 6.7 WATER SYSTEM FUNCTIONALITY 37 6.8 INFORMATION SHARING SYSTEMS 37 7 MATCHING 40 7.1 OVERVIEW OF MATCHING METHODS 40 7.2 LIMITATIONS TO MATCHING 40 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY ii 7.3 MATCHING RESULTS 41 7.3.1 MATCHING 42 8 BASELINE EQUIVALENCE 46 8.1 TESTING FOR BALANCE ACROSS TREATMENT AND COMPARISON GROUPS 46 8.1.1 BALANCE ACROSS UNMATCHED AND MATCHED SAMPLES 46 8.2 REVISITING POWER ASSUMPTIONS WITH ROUND 1 DATA 50 9 CONCLUSION AND DISCUSSION 52 9.1 SUMMARY OF CONCLUSIONS 52 9.2 NEXT STEPS 52 10 ANNEXES 54 ANNEX A: EVALUATION STATEMENT OF WORK 55 ANNEX B: BOREHOLE ASSET INVENTORY SURVEY INSTRUMENT 63 ANNEX C: ROUND 1 QUALITATIVE DATA COLLECTION QUESTIONNAIRES 82 ANNEX D: ROUND 1 CONSENT SCRIPTS 89 ANNEX E: INFORMAL GROUP DISCUSSION QUESTIONS 93 ANNEX F: BOREHOLE VERIFICATION CHECKLIST 94 ANNEX G: KEY INFORMANT INTERVIEW LIST 96 ANNEX H: GROUP DISCUSSION PARTICIPANT LIST 97 ANNEX I: REFERENCES 99 ANNEX J: ADDITIONAL BALANCE STATISTICS 101 ANNEX K: ADDITIONAL MATCHING DETAILS 103 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY iii ACRONYMS ADS Automated Directives System (USAID) ASAL Arid and Semi-Arid Land CAPI Computer-Assisted Personalized Interview CHIRPS Climate Hazards Group InfraRed Precipitation with Station E3 Bureau for Economic Growth, Education, and Environment (USAID) EDE Ending Drought Emergency EQ Evaluation Question GD Group Discussion GPS Global Positioning Software HH Household ICT Information and Communication Technology KEMRI Kenya Medical Research Institute KII Key Informant Interview LOWASCO Lodwar Water and Sanitation Company MCA Member of County Assembly MDES Minimum Detectable Effect Size MWA Millennium Water Alliance NACOSTI National Commission for Science, Technology, and Innovation NDMA National Drought Management Authority NDVI Normalized Difference Vegetation Index NGO Nongovernmental Organization PSM Propensity Score Matching RAPID Resilient Arid Lands Partnership for Integrated Development SDC Swiss Development Corporation SDG Sustainable Development Goal SO Strategic Objective SOW Statement of Work WAMSP Water Management as a Service Platform ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY iv WASCO Water and Sanitation Company WASH Water, Sanitation, and Hygiene WASHPaLS Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability USAID United States Agency for International Development ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY v EXECUTIVE SUMMARY This document reports on the findings and results of Round 1 data collection for the impact evaluation of the Kenya Resilient Arid Lands Partnership for Integrated Development Activity (Kenya RAPID). Kenya RAPID is a five-year activity implemented under a Global Development Alliance (USAID agreement number AID-615-A-15-00008) and co-funded by USAID, the Swiss Development Corporation, private sector partners, and the Millennium Water Alliance and its sub-recipients. The evaluation uses a quasi-experimental matching design to rigorously test how remote sensing technology and information-sharing affect water borehole pump runtime and management decisions. This document provides quantitative and qualitative findings from the Round 1 data collection for the evaluation. We provide details on the construction of a counterfactual group through the use of matching methods and related tests of equivalency to assess the similarity of the treatment and comparison groups. We also revisit the power calculations from the evaluation design proposal using parameters from the initial Round 1 dataset. Kenya RAPID aims to contribute to sustainable and resilient livelihoods for communities in Kenya’s arid and semi-arid lands by improving water availability and water service delivery to people and livestock and by improving rangelands in those regions. Kenya RAPID is committed to making data and information and communication technology (ICT) tools available and accessible to improve decision￾making for better water service delivery. The suite of intervention activities includes the installation of approximately 400 sensors to measure flow and water extraction rates, and to detect system failures on water boreholes. Of these 400 sensors, around 70 were installed in areas highlighted as “strategic” by local authorities due to the risk of drought. For these boreholes, the ICT intervention also involves developing a strategic borehole response operations and maintenance team in each county and reaching an agreement with the county governments to provide a dedicated budget for strategic boreholes. These strategic boreholes are the focus of this study. The Kenya RAPID impact evaluation addresses three questions derived from the intervention’s theory of change1: 1. Does the intervention using real-time remote sensing data of water points for strategic borehole management in Kenya RAPID counties lead to increased uptime of strategic boreholes during the drought season? 2. How do water managers perceive the impact of sensor-based systems on their ability to address borehole functionality and how does this compare to perceptions of borehole functionality in non￾Kenya RAPID counties? 3. Do Kenya RAPID’s sensor-based systems affect user perceptions of borehole functionality and access? To answer these questions, the evaluation team designed a quasi-experimental, mixed-methods impact evaluation. Specifically, we use a matching design, in which targeted boreholes in the five Kenya RAPID counties are matched with comparable, non-intervention boreholes in eight neighboring counties. To address the first question, we analyzed quantitative, objective data on borehole functionality collected 1 These questions have been edited slightly from the original statement of work. The boreholes were initially identified as “Ending Drought Emergency,” or EDE, but Kenya RAPID and the evaluation team dropped this label as it became clear that no water authorities or borehole stakeholders used it. In addition, the team has changed references to “drought risk management” to “borehole functionality,” as this more precisely captures the anticipated impact of the sensor-based systems. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY vi using electronic sensors installed on each borehole. The evaluation team addressed the second and third questions using qualitative data collected through key informant interviews with water system managers and operators and group discussions with local water users. Quantitative sensor data will be collected after the drought seasons in 2019 and 2020, with qualitative data collected in 2020. We will compare quantitative sensor data across the second and third rounds of analysis, with borehole asset survey data used as part of the inferential model to estimate impacts. Using qualitative data, the team will compare the first and third rounds of data collection to respond to evaluation questions 2 and 3. The Round 1 data include quantitative information on borehole characteristics across all Kenya RAPID and comparison counties and qualitative information collected from a sampling of both groups. Using quantitative data collected through a borehole asset inventory survey as well as through secondary data sources, the evaluation team assessed similarities and differences in key variables across treatment groups, in both the full sample and subsets derived from applying a variety of matching methods. These quantitative analyses yielded the following key findings: • There are several important differences between Kenya RAPID counties and comparison counties, and between strategic boreholes in these two groups. RAPID counties are drier and farther from Nairobi, and borehole power sources (e.g., diesel generator or solar panels) vary across groups. • At the same time, several characteristics of boreholes are similar across groups, including the number of households served, pipe length, and date of borehole construction. • Propensity score matching algorithms can improve the balance in covariates across Kenya RAPID and comparison boreholes. However, results vary considerably depending on the matching methods used. Methods that maximize the balance across groups come at the expense of sample size reduction, since only the most similar set of Kenya RAPID and comparison boreholes are retained. Qualitative data collection for Round 1 assessed key informant and water users’ perceptions of borehole management and functionality during the 2018 drought season. Key findings from these analyses include: • Users face significant challenges to borehole access and use, including excessive demand during the dry season and travel constraints. Concerns about functionality of strategic boreholes during the drought season are widespread. • Boreholes are managed through a devolved water management system, with primary responsibility falling on local water committees in rural areas. Local operators can manage simple repairs, but major issues must be handled at the sub-county or county level. • Resources for borehole repairs are limited. User fees are the main funding source and are not sufficient to cover maintenance costs; borehole managers face transport constraints, opaque processes for obtaining funds, and difficulty obtaining spare parts for repairs. • Within Kenya RAPID counties, use of the data dashboard to access sensor data and manage boreholes varied during the 2018 drought. The intervention was not yet fully implemented during this period. Based on these preliminary findings, this report offers the following conclusions regarding Round 1 of the Kenya RAPID impact evaluation: • Given observed differences across treatment groups and sensitivity of results to matching methods used, the final impact evaluation should employ a range of approaches and a corresponding range of impact estimates. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY vii • Key components of the Kenya RAPID intervention remain to be implemented, from ensuring that the sensor data are actually accessible via the dashboard and used by water managers to the even more challenging tasks of clarifying management roles and dedicating budgets for repairs. The ultimate success of the intervention will depend on the extent to which the activity team can effectively implement these components prior to the next drought seasons. • A mixed-methods evaluation design is particularly important in this context to assess not only what the Kenya RAPID intervention is able to achieve in terms of increased borehole functionality, but why these impacts (or lack of impacts) are observed. Next steps in the evaluation include: • Monitoring implementation progress of the ICT intervention arm of Kenya RAPID. • Collecting sensor-based data on borehole functionality in Round 2 (2019 drought season). • Refining matching methodology to achieve better balance across treatment groups. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 1 1 INTRODUCTION This Round 1 report corresponds to the impact evaluation of the Kenya Resilient Arid Lands Partnership for Integrated Development (Kenya RAPID) Activity commissioned by the United States Agency for International Development (USAID) Mission in Kenya and the Office of Water in USAID’s Bureau for Economic Growth, Education, and Environment (E3). USAID’s E3 Analytics and Evaluation Project designed the evaluation, and the Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability Project (WASHPaLS) is implementing it. The evaluation incorporates a quasi-experimental matching design to rigorously test how remote sensing technology and information sharing affect water borehole pump runtime and management. Annex A provides USAID’s statement of work (SOW) for the evaluation. This document provides findings from Round 1 data collection, which show a snapshot of key strategic water borehole characteristics and reports initial qualitative themes. The report details the use of matching methods to create a comparable group of non-Kenya RAPID boreholes from the comparison sample. The document also investigates the balance between treatment and comparison groups and revisits the power calculations from the evaluation design proposal using parameters from the baseline dataset. This is not a pure baseline report given that qualitative data collection occurred after the intervention began in the Kenya RAPID counties. However, quantitative data for both treatment and comparison counties were collected before the intervention began, and implementation was in an early phase during the qualitative data collection period, such that the results presented here will provide a useful point of comparison for subsequent rounds of data collection and analysis. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 1 2 KENYA RAPID ACTIVITY BACKGROUND 2.1 NATIONAL CONTEXT Water supply coverage in Kenya has increased from 33 percent in 1990 to 57 percent in 2015 (Organization, Supply, & Programme, 2015). 2 However, reliable and sustained water service delivery in rural areas remains a challenge, particularly in drought-prone areas. Problems maintaining the functionality of water pumps and boreholes result from social, logistical, and technical issues like the breakdown of community management structures, insufficient human resources to provide services and repairs, and lack of spare parts (Harvey & Reed, 2007). In addition, there is a lack of reliable and regular information for monitoring and increasing the responsiveness of maintenance providers. Decisions about the provision of services may depend on having accurate and timely information as well as on political, social, and economic pressures that may influence decision-making within any given local environment. Politically, Kenya is going through a period of major institutional reform including the devolution of authority and resources from the national government to newly elected county governments. County governments now have the political mandate and financial resources to provide water to their communities; however, they operate under new institutions with limited operational capacity. As part of these developments, the Government of Kenya launched its “Common Programme Framework to End Drought Emergencies,” which arose from a series of meetings with development partners between 2013 and 2014.3 As shown in Figure 1, the institutional framework for water management in Kenya consists of multiple stakeholders. The Common Programme includes the Ending Drought Emergencies (EDE) initiative to align stakeholders involved in drought mitigation and water management better across all levels of government. The EDE initiative was created as a framework to improve targeting and coordination with the goal of promoting drought reduction, early warning and response, and institutional capacity for climate resilience (Kenya, 2014). 2 Defined as “piped” or “other improved” water sources. 3 See: http://www.ndma.go.ke/index.php/resource-center/ede-reports/send/43-ending-drought-emergencies/4251-common￾programme-framework ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 2 FIGURE 1: INSTITUTIONAL ROLES IN THE WATER AND SANITATION SECTORS (World Bank, 2011) Challenges to the provision of sustainable and reliable water service, coupled with a changing institutional environment and scarce water resources in Kenya, reinforce the need for stronger and more accountable institutions, enhanced coordination and integration of development programs across sectors, private sector participation, and empowered communities with the knowledge and ability to exercise rights and responsibilities for water resources. Kenya RAPID aims to tackle these challenges. 2.2 KENYA RAPID ACTIVITY DESCRIPTION Kenya RAPID is a five-year activity implemented under a Global Development Alliance (USAID agreement number AID-615-A-15-00008) that is co-funded by USAID, the Swiss Development Corporation (SDC), private-sector partners, and the Millennium Water Alliance (MWA) and sub￾recipients.4 The activity was awarded in 2015 to MWA, a consortium of non-profit organizations in this field, with the aim of building on the successes and lessons learned from USAID’s Kenya Arid Lands Disaster Risk Reduction – WASH Program and the SDC’s Water for Livestock Program—both implemented in northern Kenya from 2012 to 2014.5 4 A total of $35.5 million will be invested through Kenya RAPID: $12.5 million from USAID; $12.5 million in leveraged funds from private sector partners; $7.5 million from SDC; and $3 million in cost share from MWA and its sub-recipients. 5 The MWA members for Kenya RAPID include CARE, Catholic Relief Services, Food for the Hungry, and World Vision. SweetSense and IBM Research are private sector sub-recipients under this award. Other private sector partners include the Coca Cola Foundation, Acacia Water, and KCB Foundation. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 3 Kenya RAPID aims to contribute to sustainable and resilient livelihoods for communities in Kenya’s arid and semi-arid lands (ASALs) by improving water availability and water service delivery to people and livestock and by improving rangelands in those regions. Three strategic objectives (SOs) guide the activity toward the overall goal of sustainable and resilient livelihoods for communities in the ASALs: • SO 1: A responsive and accountable governance framework is in place and operational at the county government level that ensures sustainable provision of water and pasture; • SO 2: Replicable and scalable business models for sustainable water, sanitation, and hygiene (WASH) and livestock service delivery have been developed and operationalized; and • SO 3: Communities have increased access to sustainable WASH services and improved rangeland management. Under these SOs, Kenya RAPID is committed to making data and information and communication technology (ICT) tools available and accessible to improve decision-making for better water service delivery. The activity will install approximately 400 sensors to measure flow rates and water extraction rates and detect system failures on water boreholes. Of these 400 sensors, around 70 have been installed in areas highlighted as strategic by local authorities due to the risk of drought. Ultimately, customized data dashboards will be developed for each county displaying the status of water boreholes in near-real time. This is possible through the facilitation of county coordination units, wherein local county officials lead implementation of the activities with facilitation and support from RAPID partners, such as SweetSense, which developed remote sensor technology to improve service delivery in multiple countries, and IBM, which is developing the data dashboards. SweetSense processes the sensor data, complements it with near-real-time survey information obtained via mobile phone surveys when notable changes to operations are identified, and uses this information to make inferences about causes for usage disruptions and changes. Kenya RAPID will make these data accessible to relevant authorities such as county governments and the appropriate service providers. The sensor data will feed into and inform other core pieces of Kenya RAPID’s support for management processes, specifically the development of operations and management teams in each county with clear roles and responsibilities, and budget support for strategic boreholes, to promote the goal of improving water service delivery. County and sub-county officials will, in theory, be able to use the sensor data to improve their management and deployment of staff and resources—areas that are receiving support through other RAPID interventions. 2.3 DEVELOPMENT HYPOTHESIS AND THEORY OF CHANGE The theory of change envisions that if the Kenya RAPID activity installs sensors on strategic boreholes, shares the data through mobile applications and online dashboards, and provides training on the use of the sensor data; supports the development of county operations and maintenance teams; and facilitates a dedicated budget for strategic borehole repairs, then this will lead to increased strategic borehole functionality, including more borehole pump runtime during critical drought periods and reduced drought impacts on ASAL communities. Kenya RAPID components are intended to work together to promote improved strategic borehole management by addressing key information and resource constraints. Figure 2 illustrates the causal linkages relevant to this evaluation that USAID envisions for translating results under each of the activities into the Kenya RAPID sensor intervention’s intended outcomes. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 4 FIGURE 2: THEORY OF CHANGE FOR THE KENYA RAPID ACTIVITY’S REMOTE SENSOR INTERVENTION Boreholes provide a critical source of water to meet community needs during times of drought. Strategic borehole management in Kenya faces a suite of challenges that often lead to system failures. These challenges mirror those found to limit water service sustainability across a number of contexts and include a “dynamic and systemic interaction of technical, social, financial, institutional, and environmental factors” (Walters & Javernick-Will, 2015). Figure 3 provides a simplified schematic highlighting some of the key challenges facing current strategic borehole management. First, many boreholes designated as key water sources during droughts (i.e., “strategic boreholes”) are in remote areas, such that information on the status of these water points can be slow to reach service providers. Second, in many counties, overlapping roles and responsibilities for borehole maintenance and repairs can cause confusion about who oversees strategic borehole repairs. Finally, service providers may have limited financial FIGURE 3: BASELINE SITUATION FOR STRATEGIC BOREHOLES IN KENYA RAPID AND COMPARISON COUNTIES ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 5 resources to devote to strategic borehole repairs. The theory of change illustrates how the sensor and related support portion of the Kenya RAPID intervention is intended to address these challenges, leading to faster repair times and better strategic borehole functionality. The causal pathways of this theory of change are built on a few key assumptions. The primary assumption is that the absence of timely information, lack of clear roles and responsibilities for strategic borehole repairs, and inadequate budgets for strategic borehole maintenance are the key barriers to better borehole functionality. Another critical assumption for this evaluation is that implementation fidelity holds throughout the entire causal chain. The intervention requires the installation of SweetSense sensors on many boreholes across multiple counties and the development of user-friendly customized dashboards. Water managers must be trained on how to use the data and understand how the constraints they face, the data dashboard, and the available resources all fit together. These dashboards have to be appropriate to the needs and capacity of service providers responsible for maintenance and repairs; if these providers cannot understand or access the database, the intended outcome is unlikely to be affected. Intervention fidelity also includes effective processes to designate and clarify roles and responsibilities for borehole maintenance for each unit, as well as allocation of an appropriate budget to address functionality issues. That is, it must be clear who is responsible for utilizing sensor information, and this person or group must be empowered, both logistically and financially, to act. Sensors alone are not assumed to change outcomes; all these components must be implemented together to achieve improved functionality. 2.4 ACTIVITY IMPLEMENTATION STATUS Kenya RAPID, with its partner SweetSense, has installed 140 sensors, with an average of 14 sensors across the five implementation counties per month and a median of around nine sensors installed on strategic boreholes each month. Figure 4 shows the overall intensity of sensor installation throughout the current implementation period. An important consideration for this evaluation is the temporal difference between the evaluation team’s data collection and sensor installation activities, which took place in September 2018, and SweetSense’s implementation of these activities in the Kenya RAPID counties over a broader timeline. As shown in Figure 4, the bulk of SweetSense’s implementation occurred prior to September 2018 (before comparison county survey commencement), so there may be seasonal variation between the treatment and comparison groups that the evaluation team will need to consider during future rounds of analysis. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 6 FIGURE 4: NUMBER OF KENYA RAPID SENSORS INSTALLED PER MONTH ON STRATEGIC BOREHOLES ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 7 The implementation team has conducted borehole asset surveys and installed sensors across each of the five treatment counties. The original Kenya RAPID work plan called for the installation of SweetSense sensors on 70 strategic boreholes, but the most recent data provided to the evaluation team show that Kenya RAPID has surpassed this goal with 76 sensors installed on strategic boreholes in the treatment counties. As shown in Figure 5, the largest number of strategic borehole sensors was installed in Wajir, while Turkana had the most installed overall.6 FIGURE 5: SWEETSENSE SENSOR INSTALLATION BY KENYA RAPID COUNTY AND BOREHOLE TYPE Sensor installation is only one component of the intervention. Other key components include developing and training users on data dashboards that deliver sensor information, clarifying responsibility for operations and maintenance by developing teams in each county, and establishing dedicated budgets for strategic borehole repairs. Communications with SweetSense staff indicate that the Kenya RAPID counties varied considerably in the extent to which each of these components were implemented prior to the 2018 drought season, which is the focal period for the qualitative Round 1 data collection. Some, but not all, counties did have dedicated teams with access to the sensor data during this period. Few, if any, counties had successfully established dedicated budgets for borehole repairs. The evaluation team will continue to coordinate with Kenya RAPID to monitor implementation, particularly tracking of county-level progress. 6 Due to an implementing partner team data entry error, none of the boreholes in Isiolo were labeled as “strategic.” There were eight boreholes in Isiolo that received sensor installation as of the data transfer to the evaluation team. The evaluation team will update its analysis should relabeled data be received. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 8 3 EVALUATION BACKGROUND PURPOSE, AUDIENCES, AND USES This evaluation comes at an opportune time, as the Sustainable Development Goals (SDGs) have broadened to include not only the provision of safe and affordable drinking water but also support for and strengthening of local community participation in improving water management (SDG 6). As investments shift toward sustainable water provision, drought risk management, and service quality, innovative tools with the potential to improve service delivery, managerial decision-making, and efficient use and allocation of resources need to be evaluated to determine which are appropriate and how to bring them to scale. 3.1 PURPOSE This evaluation will help improve USAID’s understanding of the effectiveness of real-time remote sensing of the functionality of water points during the drought season to improve decision-making for better water service delivery and drought risk management. The results of this evaluation will be made widely available to permit replication or scaling-up of activities within and beyond Kenya, as applicable. As such, this evaluation will apply USAID’s Evaluation Policy guidance with respect to using the most rigorous methods possible to demonstrate accountability for achieving results. The evaluation is also designed to capture practical lessons from USAID/Kenya’s experience to increase sustainability in WASH programs and investment in water resource management systems, specifically in strategic areas. 3.2 AUDIENCE The evaluation is aimed at several audiences. First, the evaluation’s findings are expected to be of value to USAID/Kenya and the USAID/E3 Office of Water, so they can better understand whether decision￾making utilizing data from real-time remote sensing can lead to improved borehole functionality. Second, findings and lessons learned from this evaluation will be of interest to MWA, its partners, and other practitioners in the water sector, including the Government of Kenya, which is seeking to improve water resource management, drought risk management, water coverage, and quality of services. Finally, for donors, implementers, and scholars, the evaluation will make an important contribution to the empirical evidence base on water service delivery and information interventions in drought-prone and at-risk areas. 3.3 INTENDED USE Results from this evaluation will be used to determine whether additional investments should be made on ICT tools for improved borehole functionality in Kenya or beyond. The evaluation’s findings will also inform the design of future USAID programming targeting the sustainability of water service delivery to increase resilience and livelihoods for communities. In addition, the evaluation will add to a growing body of evidence about drought risk management, to which the evaluations and studies conducted by USAID and other institutions also contribute. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 9 4 EVALUATION DESIGN 4.1 EVALUATION QUESTIONS The Kenya RAPID evaluation addresses three questions derived from the theory of change. The evaluation team developed and finalized these evaluation questions (EQs) in collaboration with USAID. 1. Does the intervention using real-time remote sensing data of water points for strategic borehole management in Kenya RAPID counties lead to increased uptime of strategic boreholes during the drought season? 2. How do water managers perceive the impact of sensor-based systems on their ability to address borehole functionality and how does this compare to perceptions of borehole functionality in non￾Kenya RAPID counties? 3. Do Kenya RAPID’s sensor-based systems affect user perceptions of borehole functionality and access? 4.2 EVALUATION DESIGN To answer these questions, the team designed a quasi-experimental, mixed-methods evaluation. The team used a quantitative impact evaluation, using a matching approach, to answer the first question and qualitative data collection and analysis to address the second and third questions. The evaluation design involves two nested units of analysis. Boreholes (and the communities they serve) are the primary units and are nested within counties. To select comparison counties, the evaluation team worked with USAID to identify eight ASAL counties (Table 1) that are nominally comparable to the Kenya RAPID counties based on general information regarding other USAID activities, aridity, and security. This was a purposive process based on USAID staff experience and knowledge of county-level characteristics and use of verifiable county information. After identifying the eight counties, the evaluation team worked with the National Drought Management Authority (NDMA) and each county’s water officer to generate lists of the strategic boreholes within each county. In several counties, the criteria for “strategic” borehole designation were opaque and local authorities appeared unsure whether they should follow FIGURE 6: TREATMENT AND COMPARISON COUNTIES ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 10 specific guidelines. SweetSense staff communicated their own challenges in obtaining a consistent roster of strategic boreholes, and the evaluation team had a similar experience.7 TABLE 1: COMPARISON COUNTIES FOR KENYA RAPID County Arid/Semi-Arid Boreholes as a percentage of total water sources in January 2018 Baringo Arid 9% Kitui Semi-Arid 28% Laikipia Semi-Arid 29.1% Mandera Arid 15% Meru Semi-Arid 43.8% Samburu Arid 25.6% Tana River Arid 14.3% West Pokot Semi-Arid 19.6% (NDMA, n.d.) In contrast, Table 2 shows the treatment borehole characteristics. This table highlights the purposive nature of county selection for implementation—all the Kenya RAPID counties are arid, with a relatively high level of borehole use. TABLE 2: KENYA RAPID COUNTIES County Arid/Semi-Arid Boreholes as a percentage of total water sources in January 2018 Garissa Arid 31.7% Isiolo Arid 28.6% Marsabit Arid 48% Turkana Arid 30% Wajir Arid 33.3% (NDMA, n.d.) 7 Although outside the scope of the evaluation, a full comparison between borehole characteristics across strategic and non￾strategic boreholes in the RAPID counties may provide more insight into determinants of the “strategic” label. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 11 County representatives selected a total of 132 boreholes across the eight comparison counties; these serve as the potential comparison group for the 76 strategic boreholes in the five Kenya RAPID counties. The goal of the evaluation design is to identify a set of boreholes that are as similar as possible to the treatment group, such that we would expect outcomes (in this case, borehole functionality) to be similar in the absence of treatment. If this is achieved, the comparison group’s outcomes can serve as a proxy for the treatment group’s counterfactual, allowing for estimation of the treatment effect. To assess similarity, the evaluation team collected data on the following observable borehole characteristics that are expected to affect functionality during droughts: • Climate and rainfall • Remoteness/distance from county chair or entity responsible for repairs • Type of pump • Populations served (human and livestock) • History of breakages/repair times Each of these variables was measured through the borehole asset inventory survey (described below) or other sources (e.g., meteorological databases). 4.3 ROUND 1 DATA COLLECTION The impact evaluation relies on data from three main primary sources: 1) sensor-based data on borehole functionality; 2) a borehole asset inventory survey on borehole characteristics; and 3) qualitative data on water managers’ and water users’ experiences related to borehole access and functionality.8 The evaluation also uses secondary data sources, such as meteorological variables and travel distance. Below we summarize Round 1 quantitative and qualitative data collection activities. 4.3.1 QUANTITATIVE DATA COLLECTION In Kenya RAPID counties, as part of the intervention process, SweetSense visited each strategic borehole and conducted a borehole asset inventory survey at the time of sensor installation. This survey, which can be found in Annex B, collected observable data on borehole characteristics as well as information on the borehole context through interviews with borehole managers and sub-county and county officials. The survey collects verifiable information, such as the borehole power source, as well as broader contextual information, such as the number of households served and tariff rates. Collected in the Kenya RAPID (treatment) counties between November 2017 and September 2018, the activity subsequently provided these data to the evaluation team.9 For the purposes of the evaluation, the evaluation team conducted a similar survey at each strategic borehole in the comparison counties from August 6 to September 7, 2018. Ipsos Public Affairs, a Kenyan survey research firm with an office in Nairobi, conducted this borehole asset survey and sensor installation. 8 Note: only motor-powered boreholes are included in the intervention and evaluation since the sensors use the electrical current to track functionality. 9 SweetSense has continued to install sensors; however, the treatment data used for this Round 1 report are only through September 20, 2018. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 12 Since SweetSense developed the survey to inform its implementation and not as a data collection tool for use in a quasi-experimental evaluation, the evaluation team encountered some challenges in adapting it. SweetSense is both more familiar with the technical details and motivation behind specific questions and, as the implementing partner, will have an ongoing relationship with local respondents to update any information that may be subject to measurement error. In the absence of deep technical insight and in lieu of repeat visits to the boreholes, the evaluation team adapted the survey to ensure comparability with the SweetSense data and flexibility for enumerators who may not have detailed technical knowledge. The final survey was programmed into iField, a computer-assisted personalized interview (CAPI) application that can be run on a mobile device. The CAPI survey allowed enumerators to take photos, record global positioning software (GPS) stamps, and upload data in near-real time. Given the remote location of many of the boreholes, Ipsos provided each survey team with a separate GPS logger to ensure the most accurate coordinate stamp possible. To assess data quality and fidelity across the borehole asset surveys conducted by SweetSense (in Kenya RAPID counties) and the evaluation team (in comparison counties), the evaluation team measured a subset of key variables upon their visits to boreholes in Kenya RAPID counties for qualitative data collection. Results from these comparisons are presented in Annex F. Differences were observed for a number of households served; other variables show higher agreement. Round 2 data collection will investigate these differences further. The borehole asset survey was implemented in three main stages starting in late July with pre-data collection activities. The evaluation coordinator and field coordinator worked with the survey firm subcontractor to review the borehole survey and address questions and concerns. To understand the borehole context, the evaluation team participated in a training-of-trainers activity with SweetSense to understand how to install the borehole sensors, anticipate challenges in survey implementation, and identify and contact key borehole personnel. The team then incorporated this experience into enumerator training. A key feature in enumerator selection and training for this evaluation was the recruitment of enumeration staff with both data collection and engineering experience. Ipsos structured its enumeration teams to include an electrician who would be able to troubleshoot any issues encountered with the comparison sensors. The survey teams also included enumerators who were ideally from the counties in which they would work or at least spoke the local language to make introductions and data collection easier. The final enumeration team consisted of 24 people, split into eight teams of three with the following make-up: • A supervisor in charge of logistics, survey protocols and procedures, quality control and checks, and follow-up with the county officials; Members of the survey team install a sensor in a borehole control room. JACOB PATTERSON-STEIN ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 13 • An interviewer responsible for recording data in the CAPI program; and • An electrician responsible for sensor installation and assisting the interviewer with data collection. A key challenge with any experimental or quasi-experimental evaluation is communicating the value and role of the counterfactual to people who are themselves part of the comparison group. As is often the case, the evaluation team could not provide any direct benefit to county, sub-county, and borehole-level stakeholders whose knowledge of the water management process was critical to data collection. The evaluation team and its survey subcontractor developed a hierarchical approach to gain the trust of officials at all levels of the water management system. The evaluation team initially contacted NDMA in Nairobi to identify local contacts at the county level. Within the counties, the survey team met with county officials, explained the evaluation, presented the sensor, addressed questions and concerns, and obtained relevant contact information for other officials. From there, each survey team paid a courtesy call to the sub-county officials, if needed, and then held a community sensitization meeting to explain their activities. This was a time-consuming process but, given the broad sensitivity around water in the strategic borehole areas and the trust required to maintain data collection over the two-year evaluation period, building rapport and addressing concerns early were critical. Figure 7 outlines the data collection process described above. FIGURE 7: COMPARISON COUNTY QUANTITATIVE DATA COLLECTION PROCESS Pre-Data Collection July–August Inception Meeting Training of Trainers Sample Review Questionnaire Review Questionnaire Programming Field Briefing Pilot/Debrief Data Collection Activities August–September Entry at the County Entry at the Sub-County Entry at the Community Data Collection Sensor Installation Quality Control Post-Data Collection Activities September–October Data Cleaning By the end of the Round 1 data collection period, sensors had been installed on all treatment and comparison boreholes included in this report; SweetSense installed the former as part of the Kenya RAPID intervention, while Ipsos installed the latter at the time of the comparison borehole asset inventory survey. The Kenya RAPID intervention relies on solar-powered, satellite-connected sensors that transmit data on borehole functionality in real time. Since real-time data are not being used in ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 14 comparison counties, the evaluation team consulted with SweetSense to select lower-cost sensors that collect comparable data to that of the Kenya RAPID sensors but do not transmit data remotely. The evaluation team sensors, which are produced by Dent Instruments, can store up to 32,576 on/off transitions, with data easily accessed through a USB connection and a battery life of five years.10 Evaluation staff will visit comparison boreholes periodically (i.e., at Round 2 and Round 3) to download data. At the same time, the Kenya RAPID team will grant the evaluation team access to raw sensor data for the treatment boreholes. The team will use these data to measure borehole pump run-time (i.e., when the borehole pump is operational and non-operational). This will be used as the main outcome variable for EQ1. 4.3.2 QUALITATIVE DATA COLLECTION Qualitative data collection and analysis play a key role in this evaluation, particularly for EQs 2 and 3. For Round 1, the evaluation team conducted key informant interviews (KIIs) and group discussions (GDs) from November 5 to December 1. Since the intervention is intended to improve borehole functionality during the drought season, the evaluation team timed its qualitative data collection activities to occur shortly after the end of the 2018 drought season. Interview and focus group questions focused on managers’ and users’ experiences during this most recent drought. Prior to commencing data collection, the Colorado Multiple Institutional Review Board (COMIRB, Protocol #18-1438) and the Kenya Medical Research Institute (KEMRI, Protocol #637) reviewed and approved the qualitative study protocol. The KEMRI Institutional Review Board approval was the first step in a process to meet Kenya’s National Commission for Science, Technology, and Innovation’s (NACOSTI) national ethical approval, which is obtained through an affiliation with an approved local organization. For this evaluation, the team partnered with Ipsos, which has a NACOSTI accreditation, to assist with qualitative data collection planning, translation during field work, and quality oversight. Ipsos was able to ensure compliance with NACOSTI standards and build on connections made during the borehole asset survey to coordinate qualitative data collection. 4.3.2.1 SAMPLE SELECTION FOR QUALITATIVE DATA COLLECTION The evaluation team purposefully sampled a total of four counties for inclusion in the qualitative subsample: Turkana and Garissa (treatment) and Tana River and Baringo (comparison). The number of counties was based on the evaluation design’s goal of six to eight group discussions and up to 16 interviews, as well as logistical, time, and thematic saturation considerations. Drawing on the borehole asset survey and rainfall data to select these counties, the team attempted to represent a mix of borehole and community characteristics, including borehole power types, reported rates of non￾functional boreholes, numbers of households and livestock served, tariff schemes, and rainfall. The evaluation team set selection thresholds for boreholes between the twenty-fifth and seventy-fifth percentiles across the aforementioned metrics to ensure selection of “standard” boreholes, but also investigated outlier cases that may be relevant for responding to the evaluation questions. The descriptive review of the borehole asset survey results in Section 5 provides summaries of these key metrics. 10 More formally, the comparison sensor is known as the Dent Instruments TOUCT-4G CT Data Logger. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 15 The evaluation team next sampled two strategic boreholes in each of the four counties (eight boreholes total) for GDs with borehole users. These were selected using asset survey findings on reported past functionality, number of households served, whether the boreholes served livestock, presence/absence of a tariff scheme, and power type (Table 3). TABLE 3: CHARACTERISTICS OF BOREHOLES SELECTED FOR QUALITATIVE DATA COLLECTION # County Power Type Households Served Fixed Tariff Broken in Past Year Used for Livestock 1 Baringo (Comparison) Generator 500 Yes No Yes 2 Baringo (Comparison) Utility 500 Yes Yes Yes 3 Garissa (Treatment) Generator 550 Yes Yes Yes 4 Garissa (Treatment) Generator 480 Yes Yes Yes 5 Tana River (Comparison) Generator 720 Yes No No 6 Tana River (Comparison) Solar 500 Yes No Yes 7 Turkana (Treatment) Hybrid 200 No Yes Yes 8 Turkana (Treatment) Solar 300 Yes Yes No 4.3.2.2 KEY INFORMANT INTERVIEWS The evaluation team selected key informants in the Kenya RAPID counties to include the individuals most closely involved in managing the data dashboard (i.e., the system that conveys information from the SweetSense sensors).11 These informants include leaders in the administration of water and irrigation services, water and sanitation companies (WASCOs), technical managers, and engineers. In comparison counties, the team identified individuals holding similar positions with respect to county-level water management. In addition, the team interviewed sub-county water officers and local pump attendants or borehole operators in each county corresponding to each of the two boreholes selected for qualitative data collection (Table 3). Annex G provides a full, anonymized list of the KII respondents. The interviews followed a semi-structured format, allowing for follow-up questions and flexibility in the evolution of the discussion. The KII protocol, which can be found in Annex C, was structured to gather 11 IBM is implementing the dashboard portion of Kenya RAPID. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 16 background information about the local context, including the local economy, any prior water management or drought mitigation schemes, and implementation of Kenya RAPID interventions from the perspectives and experiences of treatment area respondents. The KII protocol for Round 3 qualitative data collection will include several of the questions from Round 1, as well as questions directly relevant to engagement with the intervention in treatment counties. Given that the study poses minimal risks to participants, the team obtained oral consent for KII subjects. The consent script for qualitative data collection can be found in Annex D. The team recorded the interviews and took detailed notes during each to inform analysis. 4.3.2.3 GROUP DISCUSSIONS To investigate perceptions of borehole functionality further and gain a better understanding of user experiences, the evaluation team conducted GDs with between six and ten water users served by each of the eight selected boreholes. In the study context, the evaluation team determined that having both men and women in mixed discussions would not be acceptable. The team thus developed a procedure for determining whether men or women were the primary users of each borehole and selected single￾sex group discussion participants accordingly. Specifically, women generally collect water for domestic (household) use, while men are often in charge of managing livestock and their water consumption. The team therefore determined whether the primary use of the borehole was for domestic use or for livestock (through the KIIs) and conducted group discussions with women in the former case and men in the latter. Additionally, the evaluation team worked with the chief or the local administration to draw a map of the villages that used the borehole and to help recruit at least one water user from each village for the group discussion. A moderator from the evaluation team facilitated the group discussions in Swahili, or the most prevalent local language, utilizing semi-structured instruments with specific questions to guide the discussion. The evaluation team designed the group discussion protocol, which can be found in Annex C, to obtain participant experiences on the following key issues that are relevant for this evaluation, irrespective of the county: • General livelihoods strategies and borehole access and use • Overall perception of borehole management, functionality, and changes • Usage patterns, perceived challenges to use, and threat of drought • Perception of borehole management-community engagement and accessibility for different community populations (e.g., women and youth) • Prior experiences with water management and preservation programs, perception of program influence, shocks, and mitigation strategies While the evaluation team held each group discussion with either men or women only, discussion questions elicited information about water access patterns and challenges among users of the opposite gender. As part of the planning and facilitation process, the team informally met with whichever gender group was not part of the planned group discussion to explain the evaluation and selection process and collect a few basic details to corroborate with the group discussion data. This informal questionnaire can be found in Annex E. This approach allowed the evaluation team to hold in-depth data collection with the most relevant borehole users while ensuring some level of gender parity. It also served as a way to note divergence in perceptions between gender groups on key factors, such as borehole functionality and management staff availability. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 17 4.3.3 CHALLENGES ENCOUNTERED DURING QUANTITATIVE DATA COLLECTION Data collection using the borehole asset survey encountered two main challenges: 1. Accessibility challenges, such as the inability to access the borehole itself or borehole pump controller; and 2. Technical challenges, such as boreholes that use DC power output, which is not compatible with the comparison sensors, or boreholes that were wind or hand pump powered. In Mandera, security concerns limited the ability of the survey team to travel, so additional strategic boreholes had to be selected. In other cases, such as in West Pokot and Laikipia, boreholes were not appropriate for inclusion because they were hand pump boreholes or pumps had not been fully installed. Advanced planning, repeat visits, and replacement with other strategic boreholes allowed the survey team to overcome these challenges. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 18 5 QUANTITATIVE FINDINGS This section presents Round 1 findings on key borehole survey variables between the two assignment groups. Unless otherwise noted, all data presented were collected through implementation of the borehole asset survey in the treatment and comparison counties. The findings provide a snapshot of strategic borehole characteristics and likely covariates that the evaluation will control for during future stages of analysis. 5.1 GEOGRAPHIC AND SERVICE CONTEXT This evaluation covers 13 of the 47 counties in Kenya and covers the northern part of the country. This geographic coverage is important when considering the external validity of the evaluation or how applicable findings for the sample boreholes are for other boreholes in ASAL counties. The sample of strategic boreholes for this evaluation consists of 68 distinct villages in the treatment group and 131 villages in the comparison group, suggesting some boreholes serve multiple villages. Apart from six boreholes in the comparison group and four boreholes in the treatment group, all boreholes are in rural areas. The evaluation team used Google Maps API to calculate the travel distance, in miles, between each borehole and Nairobi as a measure of remoteness. Boreholes in the comparison counties are an average of 234 miles (377 kilometers) from Nairobi, while those in the treatment counties are 347 miles (558 kilometers) on average from Nairobi. Kenya RAPID was designed to address water needs in arid counties. The initial design process attempted to identify comparison boreholes in similarly arid counties. The comparison counties, on average, do receive more rainfall, both historically and in recent years. According to Climate Hazards Group InfraRed Precipitation with Station (CHIRPS) data (Funk et al., 2014), the monthly average from 2013 to 2018 was about 1.87 millimeters of rainfall in comparison counties and 0.97 millimeters in treatment counties. Of the treatment counties, only Garissa had a higher average monthly rainfall than Mandera, the comparison county with the lowest rainfall (1.04 millimeters and 0.97 millimeters, respectively). Both assignment groups experience similar seasonality in rainfall as shown in Figure 9, but the magnitude in comparison counties is much higher. For example, in 2018 Meru had the highest monthly average rainfall, with about 4 millimeters, while Marsbit had the lowest average rainfall, at about 1.3 millimeters per month for the same period. Figure 9 shows that most of the counties experienced the highest FIGURE 8: MAP OF EVALUATION BOREHOLES ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 19 amount of rainfall between March and July in 2018, although that year does appear to have seen an overall increase in rain, with 2.2 millimeters a month on average compared to 1.4 millimeters from 2008 to 2017. FIGURE 9: AVERAGE MONTHLY 2018 RAINFALL BY COUNTY Data source: CHIRPS Water source access was generally similar in treatment and comparison boreholes, according to the borehole asset survey. The number of households that strategic boreholes serve was similar across assignment groups, with an average of 1,180 households using strategic boreholes in Kenya RAPID and 819 households on average in comparison counties. The distribution of the number of households that strategic boreholes serve across assignment groups is also similar. With 500 households at the median served in the treatment group and 400 at the median in the comparison group, most of the boreholes are in low-density service areas. In the comparison counties, water is accessed through community taps at 90 percent (n = 118) of the boreholes when they are functioning, while only 50 percent (n = 38) of users in the treatment counties access water through community taps. Household taps (10 comparison, 19 treatment), rainfed troughs (2 comparison, 3 treatment), and water kiosks (2 treatment) were a few of the other sources of water access during times when the evaluation boreholes are typically functioning. The borehole survey also asked about water access in times of malfunction, with most (62 percent in the comparison group, 50 percent in the treatment group) using a surface water source, such as a river. The actual service area for the strategic boreholes is also similar across assignment groups as measured by the pipe length between the borehole and the service access point. In the treatment group, the service area was an average of 4.76 kilometers, while it was 3.20 kilometers on average in the comparison group. Round 1 data suggest that 83 percent of the boreholes in the comparison counties and 72 percent of the boreholes in the treatment counties are used for livestock. Turkana, a treatment county, has the lowest ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 20 incidence of livestock use, with an average of 29 percent of boreholes used for this purpose, while all surveyed boreholes in Meru, a comparison county, were reportedly used for livestock (Figure 10). All of the boreholes were also used for human consumption. FIGURE 10: PERCENTAGE OF BOREHOLES USED FOR LIVESTOCK BY COUNTY Of the boreholes used for livestock, goats and sheep were the largest animal group that reportedly use the boreholes across both assignment groups (Figure 11). Estimates of animal use are approximate, with a broad range within and across assignment groups. On average, the treatment group boreholes reported serving more animals per day than the comparison boreholes, but the range of values in the comparison group was higher; for example, a comparison borehole in West Pokot reported serving 6,000 goats and sheep a day.12 Another use captured in the asset survey is water trucking, or utilization of a borehole for broader water distribution. The presence of water trucking implies that the borehole is accessible (i.e., a truck can drive regularly to the location). About 29 percent of the boreholes in the comparison group and 18 percent in the treatment group reported that the borehole was used for water trucking. For boreholes used for trucking, an average of 3.5 trucks visited per day in the comparison group and 3.7 per day in the treatment group. 12 Note: one borehole in West Pokot reported serving 2,000,000 goats and sheep a day. The evaluation team was not able to verify this figure and thus dropped it for creation of the figure that accompanies this discussion. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 21 FIGURE 11: PROPORTION OF BOREHOLE ANIMAL USE BY ASSIGNMENT 5.2 BOREHOLE CHARACTERISTICS The boreholes in the asset survey were largely constructed in the past ten years, with a median construction date of 2011. However, 17 boreholes (4 treatment, 13 comparison) were constructed prior to 2000, which creates the left-skewed distribution seen in Figure 12. There appeared to be no meaningful relationship between borehole construction year and the physical state of the borehole, with a Pearson correlation coefficient of -0.06 (p = 0.37). Indeed, only 2 of the 17 comparison boreholes with a physical state rated “poor” were constructed before 2000. FIGURE 12: DISTRIBUTION OF BOREHOLE CONSTRUCTION YEARS BY ASSIGNMENT ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 22 The asset survey also collected information on the power source for the borehole. The power source could, in theory, affect functionality in cases where power may be affected by local conditions such as sunlight or flooding. The mix of power sources across treatment and comparison counties varies, with none of the comparison counties reporting the use of hybrid borehole pump power sources (e.g., solar power with a generator back-up), compared to 34 hybrid boreholes in the treatment counties. In six of the eight comparison counties, the majority of sampled boreholes had generator-powered borehole pumps, with Tana River reporting 46 percent of its strategic boreholes using generator power. As shown in Figure 13, West Pokot is something of an outlier in this group, with only seven percent of the sampled boreholes powered by generator. Power source is also a limiting factor relevant to the quantitative data collection methods of this evaluation. The sensors used to collect run-time data in the comparison counties only function with AC power sources. During data collection, the evaluation team visited 29 solar-powered boreholes with DC power sources for which the evaluation team sensors were not functional. Round 2 analysis will be able to determine the extent to which the omission of DC power sources may bias estimates. 13 FIGURE 13: STRATEGIC BOREHOLE POWER SOURCES BY COUNTY AND ASSIGNMENT Almost all of the boreholes in the survey were functional or partially functional. Across all of the boreholes visited during Round 1 data collection, only five treatment boreholes were non-functional (two hybrid-powered boreholes, two solar boreholes, and one utility borehole). Partial functionality is defined by reduced yield per the borehole’s design. As shown by the deep red color in the heatmap 13 Discussions with SweetSense field staff and the experience described here led to the conclusion that the comparison sensors do not work with DC power sources. Further discussions with SweetSense leadership, however, suggest that the comparison sensors should have worked with DC-powered boreholes. The sample of borehole types does not appear to be altered due to the omission of DC-powered boreholes, but as noted in the text, the evaluation team will continue to assess whether and how this affects estimates. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 23 (Figure 14), less than 10 percent of each power source was considered non-functional. Darker, redder colors indicate higher response frequency, while blue suggests lower frequency. The qualitative data below notes that users of generator-powered boreholes reported facing frequent issues with functionality. FIGURE 14: HEATMAP OF BOREHOLE FUNCTIONALITY AND POWER TYPES Pipeline failure was the most common reason given for strategic borehole failure, with 19 boreholes reporting this issue. For eight (seven comparison, one treatment) partially functional generator-powered boreholes, generator failure was the reported as the cause of failure. Overall, boreholes that were broken in the 30 days prior to the survey were out of service for about 12 days on average. Wajir County in the treatment group stands out in reporting breakages in 28 of the previous 30 days on average; Tana River had the highest average in the comparison group with 16 days out of service. Despite these service issues, more than 50 percent of the boreholes across both assignment groups were reported as operating eight or more hours a day, with 157 of the 208 boreholes in the sample operating seven days a week. Although median operating hours were the same in both the treatment and comparison counties, average operating hours were slightly higher in the treatment at ten hours a day compared to eight hours a day in the comparison group. Annex J provides a detailed comparison with balance statistics (discussed in the following section). As shown in Figure 15, operating hours vary by power source. Solar-powered boreholes in the treatment group operate for an average of around seven and a half hours per day, while those in the comparison group operate for an average of about eight hours a day. This is lower, on average, than the other power types. In Figure 15, the vertical lines in the middle of each box represent the median (i.e., 50 percent of the values are above that line and 50 percent are below), while the horizontal lines extending to either side of the box show the minimum and maximum ranges. Each dot in the figure is a ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 24 borehole; for example, there is one comparison borehole that uses utility power and operates for 24 hours a day. FIGURE 15: TOTAL BOREHOLE OPERATING HOURS BY POWER TYPE AND ASSIGNMENT The timing of data collection is important for contextualizing these figures. During the rainy season, it is common for borehole users to switch to rainfed water collection methods and for borehole managers to power down borehole pumps. The borehole asset survey data show that 77 percent of comparison borehole managers and 61 percent of treatment borehole managers reported that pump use varies seasonally. Indeed, the median operating hours per day drop from eight hours, as noted above, to four hours a day during the wet season. Boreholes in both the treatment and comparison counties operate for around five hours a day during the wet season. 5.3 BOREHOLE TARIFFS AND OTHER MANAGEMENT METRICS The presence of fixed and formalized tariffs may reflect the management capacity of the borehole operators or their ability to address and budget for repairs. As shown in Figure 16, many of the boreholes have some sort of tariff system in place. Notably, 27 percent (n = 36) of the comparison boreholes have no tariff system in place with four comparison county boreholes reporting an “ad hoc” system, which may not be applied to all users or with the same rates. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 25 FIGURE 16: TARIFF ARRANGEMENT BY ASSIGNMENT There does appear to be a slightly negative, although not statistically significant, relationship between the remoteness of boreholes in the evaluation counties and tariff prevalence with more remote boreholes less likely to include tariffs. This may reflect the difficulty of managing a tariff scheme in remote areas. Figure 17 shows this slightly negative relationship between the average mileage from a county’s boreholes to Nairobi and the presence of tariffs. Inferential analysis in Round 2 and qualitative data collection in Round 3 will be able to assess the degree to which remoteness affects tariff prevalence better. In the comparison group, 36 percent of boreholes (n = 47) have a special tariff for livestock, while 74 percent (n = 56) of the treatment group boreholes have the same. In both assignment groups, around 30 percent of the boreholes in both assignment groups have special tariffs for institutional use (e.g., schools). Notably, in Baringo, a comparison county, 83 percent of the evaluation boreholes have an institutional tariff, while none of the evaluation boreholes in West Pokot, also a comparison county, have an institutional tariff. Another potential measure of management engagement for the evaluation boreholes is whether water testing has ever occurred in the past. In the comparison counties, 81 percent of the boreholes had received a water test in the past, with the last having occurred in 2011 on average. Similarly, 87 percent (n = 66) of the treatment boreholes reported being tested for water quality in the past, with the last test occurring in 2014 on average. In all but three cases (two comparison, one treatment), respondents reported that the tests showed that the strategic borehole water was safe for drinking; in seven of the comparison boreholes, relevant stakeholders were not sure about the final results of the water quality tests. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 26 FIGURE 17: AVERAGE TARIFF PREVALENCE AND DISTANCE ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 27 6 QUALITATIVE FINDINGS EQ2 and EQ3 focus on borehole management and user perceptions, respectively. Round 1 data collection sought to capture the borehole context related to each EQ, as well as to learn more about the constraints, practices, and history of selected boreholes. While Round 1-Round 3 comparisons will serve to highlight potential changes in perceptions over the course of Kenya RAPID’s implementation and provide comparison cases in the non-RAPID counties, there are details from Round 1 that are worth analyzing to better understand perceptions relevant to the EQs. 6.1 QUALITATIVE SAMPLE CHARACTERISTICS The team conducted KIIs with 19 individuals across the four selected counties (Garissa and Turkana in the treatment group, and Tana River and Baringo in the comparison group). Annex G contains information on respondent characteristics. Key informants included water officers, operations and maintenance managers, water engineers and technicians, and borehole operators. The majority of these participants (17/19) were male; the only two female participants were from Turkana County. The group held a total of nine GDs with local users of the selected strategic boreholes, including two group discussions each in Garissa (treatment), Tana River (comparison), and Baringo (comparison), and three in Turkana (treatment). Two of the nine discussions were held with males only (one each in Garissa and Baringo), given that these boreholes were used primarily for livestock, while the remaining seven discussions were held with women. Annex H details age and village of GD participants. In total, 93 people participated in the evaluation GDs, with an average of 10 people per discussion. Group discussion participants were 39 years old on average, with an age range between 18 and 78. 6.2 QUALITATIVE DATA ANALYSIS METHODS The evaluation team analyzed data from the GDs and KIIs using content analysis techniques, in which text was coded according to key themes of interest across the interviewees and discussion participants. Using MaxQDA and Nvivo, the team coded all interview notes. The team applied a mixed coding approach, with thematic codes determined ex ante, followed by open coding to capture themes and broader trends as they emerged during an initial review of the data. The evaluation team exported the codes into Excel and further analyzed them using the R software package to capture counts, frequencies, and relationships between themes and words. The evaluation team also employed text mining methods to the GD notes to analyze further the reported roles of women in borehole use (Feinerer, Hornik, & Meyer, 2008). This analysis included basic topic mapping as well as analysis of what words came up most often within the context of women’s roles in accessing and using water resources to understand better relevant gender themes that may not come out of traditional qualitative coding approaches. 6.3 KEY FINDINGS AND THEMES Five key themes emerged from the Round 1 qualitative data collection. Table 4 briefly summarizes these themes, and we discuss them further in the following sections. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 28 TABLE 4: SUMMARY OF QUALITATIVE THEMES, FINDINGS,AND CONCLUSIONS Round 1 Theme Round 1 Findings Round 1 Conclusions Borehole Access and Use • Excessive demand for borehole water during dry season leads to long lines and overuse. • Travel constraints, including long walking distances and difficulty transporting water, pose barriers to borehole use. • Ability to use borehole is affected by breakages, fuel shortages, and user fees. • Users face significant challenges to borehole use across all areas. • Kenya RAPID intervention may affect some of these (e.g., reduced breakages) but is unlikely to affect others (e.g., excessive demand). Staff Roles and Management • Devolved water management systems with locally appointed water committees exist in rural areas, while utilities manage water in urban areas. • Operators are responsible for turning boreholes on/off and simple repairs. • Major issues are reported to sub-county or county officials. • Variations in water committee quality and effectiveness are likely to affect outcomes (functionality, runtime). • Intervention will be most effective if it succeeds in building capacity and clarifying roles across levels (i.e., from county level down to local operators). Resources for Borehole Repairs • Water users report that user fees are main source of funding for repairs and are not sufficient to cover maintenance costs. • Borehole managers also report limited resources for repairs, as well as transport constraints, opaque processes for receiving funds, and difficulty obtaining spare parts. • There is a strong need for dedicated budgets and clearer processes for accessing resources for borehole repairs. • Similarities in concerns across RAPID and comparison counties indicate that intervention has not yet fixed this problem. Water System Functionality • Most users report that the borehole is at risk of breaking during drought season. • Compared to wet season, breakages are more common during drought but repairs also happen more quickly. • Concerns about functionality are widespread; demand for systems that address these problems is high. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 29 Round 1 Theme Round 1 Findings Round 1 Conclusions Information Sharing Systems • Dashboard use assessed in two Kenya RAPID counties indicated very limited use in Garissa but more use in Turkana. • Both RAPID counties report problems with internet access constraining use of dashboard. • Local borehole managers have limited knowledge of the sensors and their purpose, express desire to learn more. • In comparison counties, key informants believe improved information access would assist with system management. • Intervention will not achieve its intended impacts unless users are willing and able to access data through the dashboard. Evaluation team will need to continue to expand its data collection on implementation progress across Kenya RAPID counties. • Need exists for better outreach and integration of local operators in sensor￾based intervention. A key sub-theme for understanding user habits and perceptions is how gender and age affect perceptions and water use habits. While gender is a crosscutting theme, it largely applies to access and use since, as we report below, women and youth are often responsible for collecting water and are often the first groups to respond to and deal with water-related challenges. 6.4 BOREHOLE ACCESS AND USE KIIs and GDs reported common borehole contexts and experiences across both assignment groups. Users and borehole stakeholders (e.g., operators and sub-county officials) all raised issues around accessibility, safety, and resource distribution and constraints. Use patterns reported in the GDs are similar to those reported in the borehole asset survey, with reliance on pumped borehole water during the dry season and little to no use during the long and short rainy seasons reported in all nine discussions. Across all of the GDs, 70 percent of discussants said they feel they can rely on their local borehole during the dry season. Notably, none of the participants in one Garissa GD felt they could reliably use their local borehole during the dry season. In the rainy season, users reported substituting the strategic borehole for more accessible water points, such as dams (mentioned in four GDs), rivers/streams (five GDs), and temporary streams and shallow wells (four GDs); only one GD (in Turkana) noted using the borehole year-round. A sub-county water officer in Garissa also explained that borehole accessibility becomes an issue during the rainy season due to poor road conditions (KII, Garissa, November 13, 2018). Borehole users and KII respondents noted that during the dry season, when borehole use is highest, the three main issues they encounter are excessive demand, travel constraints, and borehole reliability and fees. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 30 6.4.1 EXCESSIVE DEMAND Qualitative data collection depicts a situation where increased demand during the dry season leads to overuse of boreholes, which then limits functionality, which then slows access and supply. As one borehole operator explained: “During the dry season, there is a high number of both livestock and humans who use the borehole. This high usage results to overworking of the generator, resulting to frequent breakages. The generator is supposed to be operational for eight hours, but in the dry season it is overworked” (KII, Garissa, November 7, 2018). In seven of the nine GDs, discussants mentioned long lines at boreholes as a major constraint to accessing water. As one female discussant explained, “Sometimes we are forced to sleep at the borehole since we can’t get to the borehole water [due to long lines]. We have to sleep here in this village and sometimes we end up not getting water even when we sleep here because the lines are too long” (GD, Tana River, November 13, 2018). The issue of sleeping at the borehole was raised in three of the nine group discussions in Tana River and Turkana. Another challenge mentioned in two Tana River GDs, one Baringo GD, and one Garissa GD was that long lines adversely affect pregnant women who have to queue in the heat to fetch water for their family. As one GD discussant described, “For the pregnant and lactating women, it becomes difficult for them to wait for the long queues at the boreholes” (GD, Garissa, November 8, 2018). A common theme across both treatment and comparison counties was the hardship that water fetching poses to women. In every GD, discussants described how women are largely responsible for obtaining water for their families, regardless of challenges at the borehole. A participant in a Turkana GD summarized: “The women face the most challenges accessing water because even when I go to look for food and come back home to find that there is no water, I have to go and find water so that I can use it to cook. That is difficult because it is difficult to find a place to get the water and I have to carry the water on my back for a long distance and the water is heavy” (GD, Turkana, November 28, 2018). Focus group discussion participants in Turkana and Tana River described substituting borehole access during the dry season with self-dug wells as a means of obtaining water. Focus group participants People gather at the Katumba Borehole in Tana River County. JOYCE KISIANGANI ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 31 described the danger this can pose (including a reported death) as well as the additional time required to dig the new water source and then coordinate collection. As one woman explained, “We had to dig a very deep well on the bed of River Kerio that required up to five women to get in to get water. The five women would line up and pass the water in a bucket from the bottom to the top. The challenge is that because our soils are loose, it was dangerous because the well could have easily collapsed on us” (GD, Turkana, November 28, 2018). One drawback of traditional qualitative coding analysis is that it requires some level of subjective input to capture or perhaps consciously or unconsciously omit text. The team employed text￾mining methods to further assess what themes or topics emerged from the GDs. Figure 18 highlights the words most strongly associated with mentions of “women” in the GDs and KIIs—more formally, the correlation between the mention of women and every other word in the corpus of notes. Respondents invoked “culture” most frequently to explain water collection practices, while the prevalence of the word “pregnant” highlights how, although pregnant women’s challenges were only mentioned in four GDs, it was one of the most common themes when discussing the role of and challenges facing women. This type of analysis can help provide a check on qualitative content analysis by quantifying the themes in the text and minimizing the potential for analyst bias. The strong demand for water during the dry season and lack of alternatives (e.g., streams) reportedly led to conflict at or near the boreholes, which respondents said also affected their access and use patterns. Participants in one GD in Tana River and both GDs in Garissa noted that the long lines have resulted in disputes near the boreholes. As one discussant explained, “There is also conflict during the dry season when a lot of people fight. Some people want to skip the queue and fetch the water before others and so there are fights at the borehole” (GD, Tana River, November 13, 2018). A borehole manager in Tana River provided additional context during a KII: “The borehole serves people from different communities. Some of the conflict issues arise when a member from the host community prevents an outsider from fetching the water” (KII, Tana River, November 13, 2018). A sub-county official in Garissa also noted that boreholes along border areas tend to experience conflict as demand increases. For women, whom respondents said are largely responsible for fetching water within the sampled counties, the combination of high demand and insecurity was reported to be particularly dangerous. As one discussant explained, “In the drought season, accessibility is challenging because the lines at the borehole become longer and you can wait in line to get water until the night and some people get raped at night” (GD, Turkana, November 28, 2018). FIGURE 18: RELATIONSHIP BETWEEN DISCUSSION OF WOMEN’S CHALL OST COMMON TERMS INVOKED ENGES ACCESSING WATER AND M ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 32 6.4.2 TRAVEL CONSTRAINTS Travel constraints were also cited in two GDs in each county, with discussants mentioning walking distances from three to sixty kilometers to access the local boreholes. In addition to the geographic distance travelled, the cumbersome nature of transporting water was specifically mentioned four times in the GDs. A comment from a GD in Baringo highlights the non-monetary costs of long travel and wait times at the strategic water boreholes: “Also, school-going children face a big challenge. They have to fetch water during lunch hour before going back to school. When the water gets exhausted at the storage tank, school children walk for long distances to fetch water from this borehole. This prevents them from going to school sometimes as they will have to spend the whole day to and from this borehole to fetch water” (GD, Baringo, November 22, 2018). 6.4.3 BOREHOLE RELIABILITY AND FEES Users of generator-powered boreholes reported facing frequent issues with functionality (discussed in more detail below), which affected use of the borehole. In Garissa, GD participants noted that increased demand on borehole functionality was responsible for the access issues. For example, “There is frequent breakdown of the generator borehole specifically in the dry season due to long operation hours” (GD, Garissa, November 8, 2018). Users in Garissa (one GD), Tana River (one GD), and Turkana (one GD) noted that demand affects access due to overuse of borehole resources. Borehole use among solar-powered users was reportedly affected by cloud cover, with one borehole operator explaining that, “This borehole depends on solar power. When it is cloudy, and water is not being pumped, people usually ask me why there is no water. I respond by telling them ‘I am not the sun’” (KII, Turkana, November 29, 2018). GD participants mentioned the related issue of diminishing fuel because of overuse at generator-powered boreholes. In Tana River, discussants in one GD noted that they buy their own fuel during the dry season to ensure access to water, including pooling funds among women to purchase the fuel. In Baringo, a GD participant noted that if fuel at their borehole runs out, it may also not be available at the nearest market town, which results in having to purchase from the next￾closest market town around 100 kilometers away. KIIs revealed myriad issues affecting use. As one borehole operator explained, “Boreholes break down a lot because although they are supposed to run for 8 to 10 hours. Where there is electricity, they run them for 18 to 20 hours. So, the borehole is overworked and breaks a lot in the dry season. You get a lot of requests for repairs in the drought season and it worsens when schools open” (KII, Baringo, November 23, 2018). In five GDs (two in Baringo and one GD each in Garissa, Tana River, and Turkana), respondents noted that the borehole fees affected their ability to use the borehole. As one GD respondent explained, “We have to pay water fees to fetch the water. Sometimes we don’t have this cash in the drought season but want to fetch water for our animals and household use” (GD, Baringo, November 22, 2018). The fees in Tana River, where respondents also noted that they purchase fuel themselves, reportedly go directly toward fuel costs rather than toward usage or as part of a maintenance scheme. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 33 6.5 STAFF ROLES AND MANAGEMENT Qualitative data collection provided additional detail into the management structure and staffing of boreholes, including community engagement between users and borehole managers. Both KII and GD respondents across the treatment and comparison counties described a devolved water management system in rural areas, with community associations responsible for appointing borehole managers, managing fees and billing, and conducting maintenance. KII and GD respondents noted that the water associations have between five and twenty people. As a sub-county water officer in Garissa explained, “There are 16 members [in the local water associations] but several members are executive. There is a chairman, vice chairman, secretary, vice secretary, treasurer, vice treasurer, and one ex-official. The rest are members of the water users association. Members are elected by the community. Most communities have their own by-laws that stipulate the number of years the water committees should serve for. Mostly it is two years” (KII, Garissa, November 13, 2018). While this local democratic process was common in all of the counties, there were slight differences. In Baringo County, a borehole operator described a more overtly political process based on broader community leadership: “The management is different [from other borehole areas]. It solely depends on the political realignments after every five years. Currently we have a new youthful Member of County Assembly [MCA] who has selected youthful guys to head the water committee … The management structure is politically aligned such that when a new leadership comes into power, a new management will be selected by the new leadership. The MCA forms the water committee” (KII, Baringo, November 20, 2018). KIIs also revealed that in urban areas, water utility companies manage water access and use. For example, the Lodwar Water and Sanitation Company (LOWASCO) manages 12 Kenya RAPID boreholes with SweetSense sensors that supply water through piped networks in Lodwar town. LOWASCO is responsible for supplying water, collecting and managing water bills, and conducting maintenance and repair of boreholes. At the borehole level, both users and borehole stakeholders described the role of borehole operators who report to the local committee and who are responsible for turning the borehole on and off and maintaining the borehole through simple repairs. These operators were also reportedly the first people A focus group in Baringo County. JOYCE KISIANGANI ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 34 to recognize or receive complaints about borehole functionality issues. Across all of the GDs, 96 percent of respondents reported that they felt confident their local water manager would address borehole issues in a reasonable amount of time. In each of the GDs, users described contacting the borehole operator regarding issues; however, they also noted that they will report issues to the sub￾county- and county-level officials if they feel there is a major borehole problem. As one GD participant described, “Yes, we contact the manager who is located at Dadaab but not the pump operator because the issue is bigger than him and he cannot handle it” (GD, Garissa, November 8, 2018). The county government reportedly has responsibility for major repairs in both treatment and comparison counties, including providing technical assistance and financial support for large-scale replacements. A sub-county water officer in Garissa explained: “The water users association is responsible for maintenance [such as fuel]. Sometimes when their purchasing power goes down, they are assisted by the Department of Water or the stakeholders who are willing to assist in the drought period. In this case, the Department of Water or stakeholders will subsidize the fuel costs. Repairs are done by the county government. There are technicians and a rapid response team that attends to any breakdown that happens at the borehole sites” (KII, Garissa, November 13, 2018). Borehole stakeholders in Baringo and Tana River noted that the current staff arrangement is often limited, with three to four staff for the whole county to address issues, settle disputes, and provide technical support. As a sub-county water officer in Baringo explained, “We also have very few officers we use for mechanical, survey, and electrical issues of boreholes. So, if there is a breakdown in Tiaty, and another in Eldama Ravine, Mogotio, and in other places, it can take quite some time” (KII, Baringo, November 21, 2018). An engineer in Tana River reported that only three staff were available to assist with major repairs in the county (KII, Tana River, November 12, 2018). 6.6 RESOURCES FOR BOREHOLE REPAIRS Qualitative respondents shared experiences of resource constraints that diverge between users and borehole stakeholders, with users largely commenting on the relationship between fees and repairs, while borehole stakeholders noted challenges with transportation, logistics, and processes. In seven of the nine GDs, users reported that water fees are the main source of funding for repairs and maintenance. In one GD, users reported that the fees solely go toward fuel, resulting in a shortfall for repairs: “There are no financial resources set aside to maintain the borehole. Everything from the usage fees goes directly to fuel. Nothing is allocated to maintenance” (GD, Tana River, November 14, 2018). Current fees were viewed as insufficient to maintain the boreholes in each of the seven GDs where users reported that fees go toward maintenance. In Garissa, one GD respondent explained, “We pay the technician depending on the workload. Last time we paid him over 110,000 KES, which we had to raise on our own. This amount [collected through fees] was not enough and each household had to contribute some amount” (GD, Garissa, November 8, 2018). Interviews with borehole stakeholders similarly reported limited resources for borehole repairs. In two KIIs, respondents noted that collection and management of borehole fees were intended to allow borehole operators to address small issues, but mismanagement and lack of local capacity have hindered this goal. An engineer expressed this view, stating: ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 35 “Most of the communities we deal with are unaware. Sometimes they feel that normal servicing of the borehole should be done by the county and not themselves. They are reluctant to do even small things. The water committees collect the money, but they do not want to spend the money. You see, often the committees have a female treasurer to collect the money, but women do not have a say in how the money should be used. So, the men who feel that the repair or borehole equipment servicing is not a priority, the borehole will stay without serving the population” (KII, Tana River, November 12, 2018). For borehole stakeholders, transport constraints were mentioned in six KIIs, specifically the lack of available vehicles or funds for vehicle fuel to access boreholes for repairs. The Kenya RAPID counties (Turkana and Garissa) are among some of the largest in Kenya, and water managers noted that it takes a long time to respond to borehole breakdowns within them. As a water engineer explained, “Turkana County is such a wide county, and available resources to address the borehole challenges are limited. Fulfilling the water needs within the required time is challenging” (KII, Turkana, November 29, 2018). Interview respondents in Turkana and Garissa reported that each county only uses a single vehicle to send technicians out for borehole repairs. In Garissa, a sub-county water officer explained, “If users experience two to three borehole breakdowns, it is usually a challenge to respond because there is one team in the water department and it has only one vehicle. That vehicle is supposed to serve all of Garissa, which is roughly 44,000 km2” (KII, Garissa, November 13, 2018). Although the non-RAPID counties sampled for qualitative data collection are geographically smaller, similar resource constraints were reported in each of them. As one water officer explained, “Transport logistics are a big issue; the car is worn out and even if it is repaired, fueling it will be another hindrance” (KII, Tana River, November 12, 2018). A technician in Tana River also noted that transportation challenges led to reliance on local evaluators: “The previous time your team came to collect quantitative data, we had a problem in Asakone where a borehole had broken down. We had no vehicle or facilitation to go and repair it. But when your team came, I had that opportunity to attend to that particular problem because I used their vehicle to go to the site. That borehole took almost one and a half months before it was repaired … We do not have transport readily available to attend to borehole breakdowns” (KII, Tana River, November 12, 2018). Borehole stakeholders noted that the process for obtaining resources is often opaque or cumbersome. As noted in the background section, the devolved water management system was intended to fast-track decision-making processes within smaller administrative units at the county level. One respondent in Baringo and one respondent in Tana River noted that decentralization had made decision-making and fund acquisition for water system repairs more challenging. As one sub-county officer explained: “Before devolution came, we used to get money straight in the district. I was directly responsible, so I directly took action. If that action was to send a technician to repair a borehole, it was immediate and direct. If it was to purchase equipment, it was direct. But now, if there is a breakdown, I have to report to the county first” (KII, Baringo, November 21, 2018). In the Kenya RAPID counties, devolution was not specifically invoked, but the general bureaucratic process was mentioned in three KIIs with Turkana stakeholders and in one KII in Garissa. A respondent involved in drilling in Garissa explained the process for requesting borehole repairs: ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 36 “We fill a procurement requisition form, then take it to the County Water Director for approval. If approved, it goes to the Chief Officer who ascertains if the funds are available, then contacts a supplier who will make the resources available … The main constraint is the long government bureaucracy. The signatories of the budget might not be around, and this may prolong the procurement process” (KII, Garissa, November 29, 2018). The evaluation team analyzed the qualitative data to see the relationship between mentions of repairs and other maintenance themes and key words to further investigate key ideas in the qualitative data. The darker lines in Figure 19 highlight the terms most frequently mentioned within the context of repairs. Not surprisingly, “repair” is listed in the middle, but the items of interest in the figure below are the words connected by dark lines. “Funds,” “time,” “break,” and “people” are among the most commonly co-occurring terms when KII participants talked about borehole repairs and resources.14 FIGURE 19: NETWORK MAP OF BOREHOLE REPAIR KEYWORDS Water managers in each of the counties described requesting additional funds from county officials or nongovernmental organizations (NGOs). A borehole operator in Garissa gave an example of this situation: “We could not afford to buy the new piston and so the Member of County Assembly for Kumahumato lobbied the sub-county government and an NGO in Dadaab to help purchase it. The technician from the county came and replaced both the piston and the meter” (KII, Garissa, November 8, 2018). In Tana River a borehole repair manager described seeking funds from an NGO to obtain spare parts, while in Garissa a sub-county official explained what happens if no funds come through: “To repair the borehole in circumstances where there are no funds, we go to the merchants and talk to them. They give you the spare parts 14 This figure only includes terms that were mentioned seven or more times. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 37 and you settle the debt later. We do the same for the fuel and the car hire” (KII, Garissa, November 13, 2018). 6.7 WATER SYSTEM FUNCTIONALITY Water system functionality cuts across each of the qualitative themes. Users and stakeholders noted that functionality tends to suffer during the drought season but that boreholes are repaired faster during this period. As noted earlier, borehole users experience long lines and operators report high demand for borehole water during the dry season. Across all GDs, 88 percent of discussants said they felt their borehole was at risk during the next drought season. The issue of limited functionality due to broken borehole power sources, pipes, or pumps was mentioned in three GDs in Turkana, two in Garissa, one in Tana River, and one in Baringo. Group discussion respondents in each of these counties noted issues with generators failing; pipes breaking as a result of the temperature, water pressure, and animals walking on them; and pumps breaking from overuse or environmental factors, such as flooding. In Turkana a borehole user explained, “Most of the pipes are plastic and are exposed to the ground surface when it rains. When the sun is hot, these pipes usually burst, and the water goes to waste. Animals such as goats also step on these pipes causing their breakages. This makes us lose water access for one week as the operator has to look for new pipes to replace the broken ones” (GD, Turkana, November 29, 2018). Users of solar-powered boreholes noted functionality issues due to cloud cover, and stakeholders explained issues that arise at solar boreholes due to high levels of demand: “The borehole is solar powered and pumps very little water, which is not enough to serve the high capacity of both people and animals [during the dry season]” (KII, Tana River, November 13, 2018). Users and stakeholders described shorter repair times during the dry seasons compared to the wet season, during which repairs are said to be infrequent. Users reported repair times ranging from less than one day to two months, with a median of seven days mentioned across all of the GDs. Users and borehole stakeholders reported lodging complaints with the borehole operator, sub-county, and county officials via phone or in person. As a Tana River county-level official explained, “When a strategic borehole breaks, I get information within 30 minutes, because they call immediately” (KII, Tana River, November 12, 2018). Borehole stakeholders noted that the borehole context was a critical factor for how quickly repairs occurred and how well boreholes function. As mentioned above, there were widespread travel constraints, but the local borehole context also reportedly poses an issue. A sub-county water official in Baringo explained, “Sometimes there is a breakdown in a place but due to insecurity you cannot access this place at the time” (KII, Baringo, November 21, 2018). Borehole stakeholders in each of the counties also noted that it can take time to obtain the required parts to fix the boreholes. “Access to spare parts is a challenge, and we often have to wait for spare parts to come from Nairobi” (KII, Garissa, November 13, 2018). Borehole stakeholders also explained that the time to procure resources varies depending on the type of borehole and borehole pump. 6.8 INFORMATION SHARING SYSTEMS Qualitative data collection included questions related to the SweetSense sensors in Kenya RAPID counties to understand borehole stakeholders’ knowledge and use better. The qualitative protocol for Round 1 sought to capture whether dedicated staff were trained or delegated to use the data dashboards. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 38 In Garissa there were no dedicated staff assigned to manage the dashboard because the county reportedly does not have funds allocated toward this task. As one KII participant explained, seven county staff were trained to use the sensor dashboard, but “we do not have a central place where someone can view all the data, which is a major challenge. The sensors are also monitored in Nairobi. CARE is in the process of bringing the whole system so that we can also monitor it here at the county” (KII, Garissa, November 18, 2018). This individual noted that he had tried to download the dashboard phone app but was unsuccessful. The informant also explained that when he has used the dashboard, it was to verify reported borehole breakdowns. In Turkana three key informants reported receiving training and being assigned to monitor the sensor dashboard. As a county engineer explained, “Originally, we had the SweetSense dashboard and now an app has been developed, though it is still under development by IBM … On the app there is WAMSP [Water Management as a Service Platform]. So, on WAMSP, you can get the SweetSense dashboard. Once I log in I can see everything on Turkana County” (KII, Turkana, November 26, 2018). Respondents in both Garissa and Turkana noted that internet access was a major constraint to accessing the sensor-based system. In Garissa a sub-county water officer explained, “The major challenge in using the dashboard is the internet. Sometimes it is slow and you cannot access the information” (KII, Garissa, November 13, 2018). Two of the key informants who were familiar the sensor system reported a similar sentiment in Turkana, with one production officer there suggesting that, “It could be improved by including a satellite network option so that one can manage the boreholes even in areas with no network” (KII, Turkana, November 26, 2018). Neither of the borehole operators in Garissa reported knowing about the sensors. As one borehole operator said, “I am not familiar with the sensor-based system. It was said that the problem can be detected in Nairobi or Garissa, but we do not have knowledge on it. Nobody has ever come to say that the problem is this or that and that they have used the sensor to detect the problem” (KII, Garissa, November 8, 2018). The other borehole operator interviewed in Garissa said, “I do not like the sensor that was placed on the borehole because I always have to make sure that the children do not break it. There is something that I was told lights up when it is not working or working. I do not understand what it does” (KII, Garissa, November 8, 2018). Qualitative data in Turkana are similar, with none of the three borehole operators reporting an understanding of what the sensor is or how it functions. As one borehole operator explained, “I was told by the technician who came to install it that it won’t affect the normal functioning of the borehole and that they can observe it from their offices. I don’t know how it works” (KII, Turkana, November 29, 2018). In both treatment counties, borehole operators expressed a desire to be trained on how the sensor-based system operates. In Garissa one of the water managers noted that the sensor-based system had helped monitor borehole management: “We have involved Kenya RAPID … to help in managing boreholes … We are also able to know of the discharge levels. That is, how much water is being extracted from the borehole in a day in a particular village, and so we are able to know the tariff management because with the data we can know the amount of water generated and … how management is performing” (KII, Garissa, November 5, 2018). Six non-Kenya RAPID KII respondents noted that improved information access would assist in addressing borehole repairs and improving system management. A sub-county water official in Tana ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 39 River put it this way: “A problem known is half solved and so this will assist me to know how to solve problems” (KII, Tana River, November 12, 2018). Although many borehole stakeholders (in both RAPID and non-RAPID counties) reported receipt of borehole issues quickly during the dry season, additional information from a sensor system was seen as a way to improve breakdown responses: “In fact, there would be efficient management of boreholes [with a dashboard or sensor] because I would not need to go to the field to see the problem. The way I am operating now, I have to go to the borehole site, diagnose the problem, then come back, and go back to repair it. It takes so much time” (KII, Tana River, November 12, 2018). None of the treatment county respondents reported using the sensors to detect breakdowns or that the procedures for addressing breakages and obtaining resources had changed since the installation of the sensors.15 15 This aligns with reports from SweetSense and Kenya RAPID regarding the progress of implementation and resource access. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 40 7 MATCHING 7.1 OVERVIEW OF MATCHING METHODS The nature of the Kenya RAPID activity is such that assignment of strategic boreholes to the “treatment” of receiving the sensor-based intervention was not random. Rather, the five Kenya RAPID counties were included in this intervention because of their specific characteristics: namely, their arid to semi-arid climates and associated challenges in reliable water access, particularly during the drought season. Given this reality, the team designed the impact evaluation to select a comparable set of boreholes with similar observable characteristics and controlled for these characteristics through a matching algorithm. The identifying assumption for this design is that, conditional on these observable characteristics, we would expect similar functionality outcomes across boreholes in RAPID and non￾RAPID counties in the absence of the intervention. Selecting a set of characteristics to include in the matching process is a crucial step in this evaluation design. These variables should capture the key factors that influence the outcome—borehole functionality. The evaluation team used a combination of literature review, expert consultation, and empirical data exploration to select variables for use in the preliminary matching routines presented in this report. The few existing studies that have examined determinants of water point functionality highlighted factors including pump age, pump type, tariff collection, use for humans versus livestock, and management structure (Cronk & Bartram, 2017; Fisher et al., 2015; Walters & Javernick-Will, 2015). Rainfall patterns have also been linked with borehole functionality in the study area, with higher functionality observed during the dry season (when alternative sources are less available) (Thomas, 2019). Data sources for these variables included the borehole asset survey, as well as climate and GIS data. Given these data, a number of matching algorithms can be used to select appropriate comparison boreholes for the treatment group. As the number of covariates used in matching becomes larger, finding simple one-to-one matches becomes difficult (i.e., treatment and comparison boreholes with the exact same characteristics). A common alternative method uses propensity score matching (PSM) (Abadie & Imbens, 2006; Dehejia & Wahba, 1999; Rosenbaum & Rubin, 1983). Under PSM, a select number of observable characteristics are used to generate a predicted probability of being assigned to the treatment group. This predicted probability (the propensity score) is then used to match treated and comparison units based on the theorem that if they match on the score generated from observable characteristics, they would presumably match on the specific values of the observable characteristics as well. Unlike covariate matching (i.e., one-to-one pairing for each characteristic), PSM can ensure average similarity across characteristics. 7.2 LIMITATIONS TO MATCHING Although matching provides a more rigorous means of creating a counterfactual than simple pre-post research designs, it has several critical limitations. A few of these are listed below: • Matching relies on observables: Whereas a randomized control trial evaluation will, on average, account for both observable and unobservable factors to minimize bias in estimates, PSM relies on the assumption that unobserved characteristics across assignment groups are similar (Gertler, 2016). This assumption is generally violated since it is often known ex ante that interventions are targeted due to both observable (in available data) and unobservable (e.g., political or management ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 41 characteristics) traits. To address this, PSM often relies on datasets larger than those used in randomized control trials to maximize the number of matches and improve balance. • Matching depends on models: Matching methods generally use a logistic regression to predict the likelihood of assignment to treatment based on observed covariates. The decision-making process in selecting model estimators can heavily bias model results, particularly in cases where pre￾matching balance may already exist (King & Nielsen, 2018). • Matching omits observations: Although the matching process employed in this design attempts to avoid exactly matching on key observables, there are often units that are discarded or pruned from matching models to improve balance on key metrics. While this may make for more reliable causal estimation on average, there may be key features of the discarded units that are important to understand and analyze. • Matching does not account for logistics: The matching process may provide a methodologically rigorous comparison group but may result in a final sample that is largely in difficult-to-access, insecure, or otherwise challenging areas. It may be the case that the final matched sample includes boreholes in areas that are flood-prone or that face other stressors that require ongoing monitoring. This limitation does not affect the implementation of the matching process, but it is important for the ongoing implementation of the evaluation on the ground. The evaluation team will determine what, if any, additional monitoring is needed to ensure a robust matched sample. To address these concerns, the evaluation team will run its final analysis on three sets of boreholes for comparison, as discussed in the next section, namely: the full set of strategic boreholes, matched boreholes, and a subset of high overlap (i.e., very similar) boreholes. This will serve as a sensitivity test to see how biased findings are with respect to variable selection and provide detail on the sub-group effect sizes. 7.3 MATCHING RESULTS The general model for estimating propensity scores for this evaluation follows that of Rosenbaum and Rubin (1983), where propensity is the conditional probability of treatment subject to a set of key explanatory variables. For this evaluation, the evaluation team started with a basic model that incorporated the following ten covariates as key metrics from the borehole asset survey and third-party data sources: • Number of households using the borehole scheme • Livestock use (Y/N) • Service area for the scheme (in km2) • Out of service one or more days in the past month (Y/N) • Three-year rainfall average (annual, in mm) • Borehole pump power type • Average daily runtime (in hours) • Travel miles from Nairobi • Borehole construction year • Fixed tariff scheme (Y/N) The team selected these variables based on their understanding of the intervention, selection criteria for counties and boreholes, and descriptive analysis. Following guidance from (Imbens & Rubin, 2015), the evaluation team conducted additional statistical tests to determine what other variables, interactions, or polynomial covariates to include. A total of 15 models were tested, with a base model including the ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 42 above variables, as well as whether or not a water quality test was conducted and an interaction term between borehole failure in the past month and rainfall. The interaction term implies that the effect of the variables in the bulleted list above differ at different values of the interaction terms. For example, the effect of a borehole being out of service in the past month on the likelihood of being assigned to the treatment group may vary based on the three-year annual rainfall average, which an interaction term captures. As described below, the model specification tests resulted in the base model with the above variables and a limited model specification with fewer variables.16 Annex K provides an overview of the model specification tests used in the matching process. 7.3.1 MATCHING Figure 20 shows the unmatched boreholes across the ten key metrics mentioned above. A key concern in PSM is the area of overlap, sometimes known as the area of common support, which describes the region across propensity scores that both assignment groups share. For example, high overlap would essentially mean that both assignment groups have a similar distribution of propensity scores. The bimodal distribution below is an example of almost no overlap resulting from a simple linear regression, but without employing a matching algorithm. Based on the estimation model using the covariates described above, the average propensity score is about 0.03 in the comparison group and 0.94 in the treatment group. FIGURE 20: UNMATCHED PROPENSITY SCORE DISTRIBUTIONS As noted earlier, matching omits observations to improve the area of overlap and create a counterfactual group that is, on average, more similar to the treatment group than an unmatched sample. Our initial matching model is the same as the model used to estimate the above figure but 16 A key consideration in model specification is how well the variables chosen predict assignment to treatment. Key statistics used in model specification and comparison tests included Akaike’s Information Criteria, adjusted R-squared, and loglikelihood ratios. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 43 applies an optimal matching algorithm. Optimal matching attempts to find the smallest average difference between propensity scores across assignment groups. This matching approach is considered to be well￾suited to data with limited comparison group matches and lack of overlap (Ho, Imai, King, & Stuart, 2011). 17 Figure 21 shows initial results from an optimal matching model. Although this looks similar, there are a few key details to note. This model results in 57 treatment and 57 comparison boreholes in a final matched sample rather than all observations, with an average propensity score of 0.06 in the comparison group, compared to 0.03 in the unmatched model. What this matching model has largely done is cut down on the comparison boreholes that have little similarity with the treatment boreholes across the key matching metrics, as can be seen in the decrease in the left tail of the distribution below. FIGURE 21: FIRST MATCH PROPENSITY DISTRIBUTION (N = 57/57) The area of overlap is still low as can be seen by the bimodal distribution above. This is problematic given that it implies the comparison boreholes serve as a poor counterfactual for the treatment boreholes. As mentioned in the introduction to this section, the evaluation team performed a series of model specification tests, including likelihood ratio tests and iteration across all covariate combinations using stepwise regression to test model fit (i.e., to improve how well the asset variables predict assignment to treatment). 17 Ho et al. note that optimal matching performs similarly to nearest-neighbor approaches, wherein matches are chosen based on the closest similar propensity score between assignment groups. Optimal matching differs in that it minimizes distance between matches and across assignment groups. The evaluation team attempted both optimal and nearest￾neighbor matching, as well as coarsened exact matching, to test for bias in the matching model and trim the sample to improve overlap. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 44 A more parsimonious model selected through stepwise regression, wherein all combinations of model inputs are run and tested through likelihood tests, includes the following five variables to estimate propensity scores: • Miles from Nairobi • Borehole water quality test in the past (Y/N) • Livestock use (Y/N) • Three-year rainfall average (annual, in mm) • Fixed tariff scheme for livestock (Y/N) After model specification tests were performed, the team considered a final approach that trims the sample within some range of the propensity estimates to improve matches. Within this range, comparison boreholes are then randomly selected for each treatment borehole as a match. Standard practice suggests using a cutoff of the 0.25 standard deviation of propensity scores, but the trade-off between overlap and sample size is important to consider. A lower, more conservative matching cutoff may result in boreholes in the treatment and comparison groups that are incredibly similar but potentially with a sample size too low to be statistically reliable for estimating impact. Another way to address lack of overlap is to match multiple comparison boreholes to individual treatment boreholes, also known as many-to-one matching. This approach results in a larger sample size and may improve estimates but may also result in some additional bias due to the inclusion of comparison boreholes that have only limited similarity to treatment boreholes. The evaluation team split these approaches for its final matching model with a two-to-one, comparison￾to-treatment matching approach and a matching threshold of 0.25 of the standard deviation of the estimated propensity scores.18 This cuts the sample size down, leaving only 17 units in the treatment group and 28 units in the comparison group, but also improves the overlap between assignment groups as seen in Figure 22. The next section addresses the implications for statistical power for each model result. In this matching approach, the average comparison group propensity score is about 0.32, which is the highest likelihood of any of the approaches that include rainfall. The key takeaway from our exploration of these approaches using the Round 1 data is that matching is sensitive to key researcher decisions, and thus a variety of approaches should be employed to create a range of estimates. This more nuanced approach will allow the evaluation team to better understand what borehole characteristics affect the impact of the Kenya RAPID intervention and serve as a sensitivity analysis for Round 3 results. 18 The evaluation team also tested each matching model with and without limiting comparison units to a single match with a single treatment unit. Matching with replacement (i.e., matching a single comparison unit to multiple treatment units) improves overlap but severely limits the number of comparison units, since a limited number of “strong” comparison unit matches are repeatedly paired to the treatment sample. For example, in the parsimonious model, matching with replacement resulted in only 11 comparison boreholes, with an average propensity score of 0.36. This is not surprising given the low number of variables on which to match in this model. Matching with replacement on the full model did not change the sample size or general propensity score estimates. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 45 FIGURE 22: TRIMMED SAMPLE WITHTWO-TO-ONE MATCHING (N = 28/17) ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 46 8 BASELINE EQUIVALENCE In addition to providing the descriptive statistics presented in this document, the Round 1 data can also be used to test some of the statistical assumptions related to the evaluation methodology. This section investigates two such assumptions. First, balance tests are used to assess and confirm the comparability of the treatment and comparison groups. Second, the power calculations presented in the evaluation design proposal are revisited using updated parameters from the Round 1 data to assess statistical power, given the actual sample size. 8.1 TESTING FOR BALANCE ACROSS TREATMENT AND COMPARISON GROUPS Round 1 data as outlined in this report largely serve to contextualize the Kenya RAPID intervention and strategic boreholes within the sample. A key to estimating the impact of the ICT intervention of Kenya RAPID on runtime as the main outcome of interest is ensuring that, on average, boreholes are similar prior to fully benefitting from the intervention. If large differences exist between treatment and assignment groups, the comparison boreholes may not serve as a convincing counterfactual. A key role of the matching process detailed in the previous section is to improve balance between the assignment groups by ensuring that boreholes are, on average, similar on the key variables of interest. The evaluation team calculated standardized mean differences to assess the balance between assignment groups for each variable. This statistic is based on the difference between the treatment and control group means, divided by the square root of one-half the sum of the treatment and control group variances. This basically provides a unit neutral and methodologically sound way to assess the similarity of the assignment groups. Absolute standardized difference values greater than 1 raise concerns about extreme imbalance, while absolute values at or below 0.25 suggest strong balance (Imbens & Rubin, 2015). While some researchers have traditionally used t-tests to compare means across assignment groups, there is limited conceptual justification for this practice to assess balance (Altman, 1985). 19 This section provides an overview of general balance across the full, unmatched sample on most of the variables in the asset survey. It then provides a more detailed look at balance on the key matching variables to show how balance may have changed as a result of the PSM process. 8.1.1 BALANCE ACROSS UNMATCHED AND MATCHED SAMPLES To assess initial balance, the evaluation team calculated the standardized mean differences for key metrics from the asset survey, as well as several other relevant variables that could be included in future analyses, such as the normalized difference vegetation index (NDVI), a measure of vegetation coverage. As shown in Figure 23, about half (n = 21) of the key metrics are within the 0.25 threshold suggesting strong balance on these metrics without matching. Annex J provides a detailed overview of the balance statistics. Notably, rainfall and NDVI are extremely imbalanced. It is also worth noting that some values, such as the estimated number of horses/donkeys using the borehole and hours spent collecting water when the borehole is malfunctioning, are the best guesses of borehole staff and have quite a wide range of 19 For a good discussion on this subject see https://blogs.worldbank.org/impactevaluations/should-we-require-balance-t-tests￾baseline-observables-randomized-experiments ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 47 estimates, which makes them noisy and somewhat unreliable metrics. Imbalance on other variables, such as the miles to Nairobi, is expected given the nature of the intervention and the county selection process. The base matching model included ten key metrics, including power source, which have been broken into four separate yes/no variables by type. The initial matching model improves balance for the “hours per day pump runs” variable, reducing the overall mean difference between assignment groups from about 1.43 hours difference to 0.75 hours. Notably, imbalance worsened in the initial match for the livestock use metric, as a result of the proportion of comparison boreholes in the sample with livestock use increasing from 83 to 88 percent. This highlights how balance is sensitive to model specification. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 48 FIGURE 23: BALANCE ACROSS UNMATCHED ASSET SURVEY VARIABLES ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 49 FIGURE 24: BALANCE FOR INITIAL MATCHING MODEL The results of the trimmed, 28-17 sample show improved balance from the initial matching model. On certain key variables, such as rainfall, the imbalance has been significantly reduced, with a standard mean difference of 0.30, which is still within reasonable bounds for comparison. Interestingly, there is less balance for the construction year variable and the generator variable. The average difference in construction year is about four years in this matched sample, which may not actually affect outcome estimates in a meaningful way.20 Generator power is much more common in the comparison group than the treatment counties, as noted in the descriptive section, and thus there is less room to improve balance on this variable absent weighting. 20 Standard t-tests and regression methods for balance do not consider this a significant difference. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 50 FIGURE 25: BALANCE FOR TRIMMED MATCHING MODEL At the analysis stage for Round 2, the evaluation team will likely weight key variables, such as the power source, to improve balance given the strong imbalance in the data. A common practice is to use inverse probability weights (although this can increase error in the estimates) to improve comparability between units in each assignment group that would otherwise not be matched. This section, however, illustrates how the matching process, even without robust weighting, can improve comparability between assignment counties. This is critical for any impact evaluation to ensure that the comparison group is a reliable and convincing counterfactual. 8.2 REVISITING POWER ASSUMPTIONS WITH ROUND 1 DATA As outlined in the evaluation design proposal, there is a trade-off between sample size and the reliability of estimates, with larger sample sizes more likely to detect the causal effect of the activity. While a full review of the underlying power assumptions is not possible until functionality data from the borehole sensors is collected in Round 2, the evaluation team is able to assess whether the initial sample sizes proposed were reasonable for a given set of potential effect sizes. Table 5 shows the original sample size calculations from the evaluation design proposal. TABLE 5: EX ANTE MINIMUM DETECTABLE EFFECT SIZE (MDES) ESTIMATES Sample Size (# of Boreholes) MDES 95% Confidence Interval 150 0.350 0.118, 0.581 175 0.338 0.114, 0.561 190 0.327 0.110, 0.543 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 51 The initial estimates were based on a review of the relevant literature and assumptions regarding potential sample sizes prior to the commencement of the evaluation. The table below (Table 6) is similar to the power table used in the evaluation design proposal, but with updated sample sizes based on the matching options explored in Section 7, the full sample, matched sample, and parsimonious many-to-one matched sample. The minimum detectable effect size (MDES) by sample size is presented based on standard parameters, such as significance level (alpha) of 0.05 and power of 0.8. In the absence of outcome data on functionality, the evaluation team has also reused the conservative 25 percent covariate variance figure from the evaluation design (that is, the assumption that 25 percent of the variance in the outcome can be explained by the covariates, or control variables, such as mileage or the presence of a tariff system). In the matching models, 67 to 90 percent of variance in assignment to treatment is explained by the covariates. As this value increases, the MDES decreases, meaning estimates of impact become more reliable and suggesting that the estimation model has a stronger fit for the data. TABLE 6: MDES ESTIMATES BASED ON BOREHOLE SAMPLE SIZES21 Sample Size (# of Boreholes) Sample Type MDES 95% Confidence Interval 45 Many-to-one with 0.25 cutoff 0.561 0.188, 0.934 114 Matched 0.391 0.132, 0.649 208 Unmatched 0.312 0.105, 0.518 The full, unmatched sample presents a lower MDES than any of the estimates in the design proposal. As expected, with the decrease in raw sample size through matching, the MDES increases. However, as noted above, these calculations are still based on several conservative assumptions that can be re-tested during Round 2. Notably, the matched sample is within the general range of MDES estimates from the design stage but is likely more realistic given that certain assumptions have been replaced with observed data from the asset survey. Overall, the matched and unmatched samples appear to be sufficiently powered for estimation of impact. The trimmed matched sample based on current assumptions may be too low to estimate the impact of the intervention reliably. However, the limited literature in this sector suggests that effect sizes can be as large as 86 percent (Nagel, Beach, Iribagiza, & Thomas, 2015), which could improve our ability to observe an impact in the trimmed sample. These sample sizes will help the evaluation team to test the robustness of results with Round 2 and Round 3 data on borehole functionality and to assess how sensitive impact estimates are to potential researcher biases. 21 Assuming blocked design with the following parameters: alpha = 0.05, power = 0.8, 10 included covariates that explain 25 percent of the variation in the outcome. The percentage assigned to treatment is based on borehole survey data and matching: 37 percent for the unmatched sample; 50 percent for the matched sample; and 38 percent for the trimmed matched sample. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 52 9 CONCLUSION AND DISCUSSION This section provides brief conclusions based on the evaluation team’s analysis of Round 1 data and summarizes next steps in the impact evaluation. 9.1 SUMMARY OF CONCLUSIONS The main conclusion from the quantitative data from Round 1 is that key differences exist between strategic drought boreholes in the Kenya RAPID counties and the comparison counties. RAPID boreholes tend to be in areas that receive less rainfall and are farther from Nairobi. These boreholes are also powered differently; generators are more common in comparison counties, while hybrid (solar/generator) boreholes are quite common in RAPID counties and non-existent in comparison counties. Importantly, many of the characteristics that differ between these two groups of boreholes are likely to affect our key outcome of interest: borehole runtime and functionality during the drought season. This finding has key implications for the impact evaluation design. Specifically, since one cannot reasonably assume that outcomes would be similar across groups in the absence of the intervention, well-designed and robust matching methods are needed to account for observable differences across groups. Our preliminary analyses indicate that some matching methods are able to improve the balance across treatment groups substantially, but at the cost of sample size; boreholes without sufficiently similar matches are dropped, leading to a much smaller set of boreholes on which to estimate program impacts. Given observable differences across groups, and sensitivity of results to methods, it will be important to use a range of matching approaches to generate a range of impact estimates. Meanwhile, qualitative findings indicate that borehole functionality and water access are persistent and pervasive problems across the study area, with disproportionate impacts on women and youth responsible for collecting water. Similarities in reported challenges across Kenya RAPID and comparison counties suggest that the intervention did not result in dramatic changes over the first drought season. This is not surprising in light of results also showing that the intervention was only partially implemented to date. Sensors were installed, but use of the dashboard was limited, roles and responsibilities for operations and maintenance were not fully clarified, and dedicated budgets for repairs were not yet established. Meanwhile, a wide range of other challenges are beyond the scope of this intervention. These include excessive demand for limited water resources, travel constraints, and fuel supply problems. Overall, these early results highlight the importance of using mixed-methods in this evaluation. By applying both quantitative and qualitative methods in this impact evaluation, the team will be well positioned to understand what the RAPID intervention is able to achieve in terms of changes in functionality, as well as why those impacts (or lack of impacts) are observed. 9.2 NEXT STEPS The data and analyses presented in this report lay the groundwork for the impact evaluation of the Kenya RAPID intervention. As noted previously, the intervention has commenced in the Kenya RAPID counties but is in an early phase and likely had limited impacts on borehole management and functionality during the 2018 drought season (corresponding to the period examined by Round 1 qualitative data collection). Next steps in the evaluation include: ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 53 1. Coordinating with Kenya RAPID implementing partners to monitor progress and any changes to implementation. 2. Refining matching methodology, including investigation of different weighting schemes to achieve better balance across treatment groups. 3. Identifying a critical set of comparison boreholes that are included in matched samples and targeting resources to ensure that sensors remain functional for this set of boreholes. 4. Collecting sensor-based data on borehole functionality. These data will be collected in Rounds 2 and 3, following the 2019 and 2020 drought seasons (October and November of each year). In comparison counties, the evaluation team will download data during visits to the boreholes. Kenya RAPID will provide data for the treatment counties. 5. Estimating intervention impacts for the 2019 and 2020 drought seasons using matching methods. 6. Collecting follow-up qualitative data on key informant and user perceptions of borehole management and functionality outcomes. These data will be collected in Round 3 (October and November 2020). 7. Assessing qualitative impacts on borehole management and functionality using the Round 3 data. 8. Integrating qualitative and quantitative results: what happened and why. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 54 10 ANNEXES ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 55 ANNEX A: EVALUATION STATEMENT OF WORK Impact Evaluation of the Kenya Resilient Arid Lands Partnership for Integrated Development Project This Statement of Work is for an impact evaluation commissioned by the Office of Water in the United States Agency for International Development’s Bureau for Economic Growth, Education, and Environment (USAID/E3/Water) that will examine the Kenya Resilient Arid Lands Partnership for Integrated Development (Kenya RAPID) activity. 1. Activity Information Kenya RAPID is a five-year activity that began in October 2015 and is funded by USAID, the Swiss Development Corporation (SDC), and the private sector. Kenya RAPID aims to contribute to sustainable and resilient livelihoods for communities in Kenya’s Arid and Semi-Arid Lands (ASALs) by improving water availability and water services delivery to people and livestock and improve rangelands in the ASALs. Kenya RAPID mobilizes financial and technical resources from development partners, the national government, county governments, and the private sector to address the complex problems created by inadequate water access and poor governance of natural resources in the ASALs. Kenya RAPID targets five northern ASAL counties – Marsabit, Garissa, Isiolo, Wajir, and Turkana. Each has high poverty rates, chronic water shortages and food insecurity, and low access to basic services. Kenya RAPID uses a public-private partnership model to combine the assets and experience of development actors, private and public institutions – leveraging their capital and investments, innovation, and access to markets – to address the complex problems created by inadequate water access and poor governance of natural resources in the ASALs. Kenya RAPID will directly contribute to USAID/Kenya’s Country Development Cooperation Strategy (CDCS) 2015-2018, whose goal is Kenya’s governance and economy sustainably transformed, and the SDC’s Cooperation Strategy for the Horn of Africa goal to contribute to reduction of poverty, improve human security and instability, and address migration challenges. 2. Development Hypothesis USAID/Kenya envisions that building the capacity of relevant private and public stakeholders for improved WASH service provision and improved rangeland management practices will lead to better health and more resilient livelihoods in targeted areas. Kenya RAPID activity components work in concert to promote water access and delivery and enhanced rangeland environments. Access to water for both domestic and livestock use is a critical component to the livelihoods of ASAL communities. Frequently, ASAL communities have limited availability of water resources, which can adversely affect WASH practices; instead of engaging in hygienic practices like handwashing, individuals may choose to use the water for other purposes. Poor water access can also limit livestock growth and inhibit economic growth for individuals. This adversely affects the health and economic wellbeing of communities and individuals. Kenya RAPID will endeavor to add to this growing body of knowledge during the life of the activity by testing appropriate hypotheses that will be specified at a later date. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 56 Figure 1 illustrates the causal linkages that USAID/E3/Water and USAID/Kenya envision for translating results under the activities into Kenya RAPID’s intended intermediate and final outcomes that this evaluation will be expected to examine. In this theory of change diagram, the improvement of governance frameworks and WASH coverage leads to improvements in water and sanitation access for individuals, water access for livestock, and rangeland-management practices. FIGURE 1: KENYA RAPID THEORY OF CHANGE 3. Existing Performance Information Sources USAID/E3/Water, in coordination with USAID/Kenya, has already provided the evaluation team with the following documents related to existing performance information: • Kenya RAPID activity documents: • Kenya RAPID Year 1 Work Plan • Kenya RAPID fully executed Task Order • CARE Implementation Activities Progress Presentation The above list, which is non-exhaustive, highlights relevant data sources that have been shared with the evaluation team. The following additional documents have not yet been provided to the evaluation team, but will be shared as the evaluation progresses: • All future quarterly project management and progress reports provided by each of the four implementing partners (CARE, Catholic Relief Services, Food for the Hungry, and World Vision) • Documents pertaining to selection and implementation of WASH, water coverage, and rangeland management projects ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 57 • Annual USAID/Kenya WASH Survey materials In addition to information provided by USAID and each of the implementing partners, the evaluation team will need to access other types of secondary data, including administrative information on the municipalities from a variety of sources. This will likely involve accessing published government sources, or obtaining the information from Kenya RAPID staff who are knowledgeable about existing data for specific municipalities. The evaluation should also collect and analyze information related to WASH, water coverage, and rangeland management in Kenya, other activities to improve WASH services, issues that may affect social cohesion and gender inclusion in Kenya, and other factors exogenous to Kenya RAPID that could influence activity impacts or survey responses. 4. Evaluation Purpose, Audience, and Intended Use Purpose This impact evaluation will allow the Agency to learn more about how WASH and rangeland management interventions can lead to improved health and economic outcomes. The results of this evaluation will be made widely available to encourage replication and/or scaling up of pilot activities within and beyond Kenya, as applicable. As such, this evaluation will apply USAID’s Evaluation Policy guidance with respect to using the most rigorous methods possible to demonstrate accountability for achieving results. The evaluation is also designed to capture practical lessons from USAID/Kenya’s experience with regard to increasing sustainable WASH programs and investment in water and rangeland resource management systems. Audience The evaluation is aimed at several audiences. First, the findings are expected to be of value from an accountability and learning standpoint to USAID/E3, particularly in the Office of Water, and USAID/Kenya. Second, findings and lessons learned from this evaluation will also be of interest to MWA, its partners, and other practitioners in the WASH and rangeland management sectors, including the Government of Kenya, which is seeking ways to improve water resource management, WASH coverage and quality of services. Finally, the evaluation will be of interest to donors, implementers, and scholars more generally by making an important contribution to the evidence base on WASH service delivery interventions. Intended Use This evaluation will be used to inform the design of future USAID programming that aims to improve the sustainability of WASH services to increase resilience and sustainable livelihoods for communities. Depending on the intervention/hypotheses USAID elects to examine through an impact evaluation, it may also contribute to a growing body of evidence about WASH effectiveness, to which other USAID evaluations are also contributing as are studies conducted by other institutions. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 58 5. Evaluation Questions The evaluation questions for Kenya RAPID are still in development. Ultimately, they will reflect USAID’s learning priorities for WASH and rangeland management investments and Agency programming for WASH and rangeland management. The evaluation is expected to focus on how ICT solutions can affect water management in drought prone areas. 6. Gender Considerations In line with USAID’s Gender Policy (ADS 203.3.1.5), the evaluation will consider gender-specific and differential effects of Kenya RAPID activities. The evaluation team will disaggregate access and participation data by gender at multiple points along the theory of change diagram to analyze the potential influence it has on pilot activities and outcomes. Data collected through surveys will be gender￾disaggregated to identify gender differences with respect to benefits and outcomes. The evaluation team will conduct further inquiry on gender themes as they emerge during data analysis. 7. Evaluation Methods Impact Evaluation Design Impact evaluations identify activity impact by comparing outcomes between activity beneficiaries to those of a control or comparison group of non-beneficiaries. The control or comparison group is intended to represent the counterfactual, or what would have happened in the absence of the Kenya RAPID intervention. As per the USAID Evaluation Policy, impact evaluations using experimental designs – whereby units are randomly assigned to treatment and control groups – provide the most rigorous evidence of activity impact, and this will be the preferred approach for the Kenya RAPID impact evaluation. Where randomized assignment is not feasible, quasi-experimental impact evaluation designs can be employed as an alternative. The evaluation team responding to this SOW will work with USAID/E3/Water, USAID/Kenya, and the implementing partner to develop a design that suits the objectives, timing, and constraints of Kenya RAPID. The evaluation team will produce an evaluation design proposal to be approved by USAID/E3/Water prior to any site selection or randomization. It is expected that the evaluation questions will be answered using an experimental or, if necessary, quasi-experimental design, and that a mixed-method approach may be suitable to answer the evaluation questions. Data Collection Methods USAID anticipates that data collection for this evaluation will involve the use of household-level surveys that cover all communities targeted for Kenya RAPID. This is likely to include a baseline survey that would be conducted before major interventions commence. The survey would collect information on basic the outcomes of interest that the evaluation will measure. The evaluation team responding to this SOW shall provide further details on data collection methods and the specific survey methodology in the evaluation design proposal, including proposing specific data collection methods on a question-by￾question basis. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 59 8. Data Analysis Methods In its evaluation design proposal, the evaluation team responding to this SOW should propose specific data analysis methods on a question-by-question basis, including the appropriate mix of methods necessary to estimate the impact Kenya RAPID has on the primary outcomes of interest. Potential data analysis methods include difference-in-difference and multivariate regressions. The evaluation design proposal should also explain what statistical tests will be conducted on data collected to address all evaluation questions, how qualitative data will be analyzed, and whether that analysis will allow the evaluation team to transform some data obtained from qualitative into quantitative form. The evaluation design proposal should also indicate and justify the evaluation team’s proposed sequencing of quantitative and qualitative data collection. For example, if key informant qualitative interviews are conducted during the endline data collection process, these lines of data may be collected and analyzed in parallel and only synthesized once data from all other sources are available. 9. Strengths and Limitations The strengths and limitations of the Kenya RAPID impact evaluation will depend on the final design proposed by the evaluation team in consultation with USAID and the implementing partner. The final design should reflect a rigorous approach to answering the evaluation questions and contribute to the global knowledge on water delivery and rangeland management practices. One key contribution of this evaluation is that it is expected to specifically test the impact of private sector engagement on improving access and quality of WASH services. Sample size, activity reach, and implementation fidelity could all create internal validity limitations for this evaluation. Ensuring that the sample size achieves sufficient statistical power will be critical for identifying impact and answering the evaluation questions. In addition, ensuring that randomization is done properly and random assignment, if applied, is systematic will improve the internal validity of the evaluation, but must be done in a transparent manner. Indirect contamination across treatment arms and comparison groups is always a possibility, which is why it is important for the evaluation team and the implementation team to coordinate from the outset. 10. Evaluation Deliverables The evaluation team expects to be responsible for the deliverables listed in Table 1. A final list of proposed deliverables and due dates will be included in the evaluation design proposal for USAID’s approval. TABLE 1: EVALUATION DELIVERABLES Deliverable Estimated Due Date 1. Concept Paper, describing design and methodological options to answer the evaluation questions TBD in consultation with USAID/E3/Water 2. Draft Evaluation Design Proposal TBD in consultation with USAID/E3/Water 3. Final Evaluation Design Proposal, including data collection and analysis methods, evaluation instruments, team composition, proposed timeline, and estimated budget TBD in consultation with USAID/E3/Water ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 60 Deliverable Estimated Due Date 4. Baseline Report o/a 60 days following completion of baseline data collection 5. Draft Evaluation Report o/a 60 days following completion of endline data collection 6. Final Evaluation Report o/a 21 days following receipt of USAID comments on Draft Evaluation Report All documents and reports will be provided electronically to USAID no later than the dates indicated in the approved evaluation design proposal. The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template. 11. Team Composition The evaluation design proposal should describe the specific composition and qualifications of the team members who will be carrying out this evaluation, including CVs for core team members. General qualifications and roles anticipated for core evaluation team are listed below. Local survey research firm(s) with experience in the conduct of household surveys at the village level and/or qualitative data collection may also support the evaluation team, as necessary. Principal Investigator. The Principal Investigator for this impact evaluation should hold a Ph.D. in a relevant economic development field. S/he will have previous experience with WASH programs and will have previously served as a team leader for one or more impact evaluation(s) that include a counterfactual. Familiarity with a range of impact evaluation designs and with USAID evaluation guidance will be sought for this position. Experience in publishing evaluation research in peer-reviewed journals is desirable, as is experience working in East Africa. A demonstrated ability to gather and integrate both quantitative and qualitative findings to answer evaluation questions is expected. Demonstrated experience managing multinational teams and producing highly readable reports for USAID and its developing country partner audiences on a timely basis is expected. This individual will be primarily responsible for the quality of the evaluation design and its execution, particularly with respect to the evidence obtained on questions involving causality and the attribution of outcomes to USAID’s intervention. This is not a full-time position. Evaluation Specialist. The Evaluation Specialist should have a graduate degree in a relevant social science field, and may be a Kenyan national. The individual will have sufficient previous experience with evaluations and other types of studies involving sample surveys to be actively engaged in efforts to oversee and ensure the quality of the evaluation's multiple rounds of household surveys, and ensure that data codebooks are clearly written and all study data prepared by local firms are properly transferred to USAID. Gender analysis experience is also desirable. This is not anticipated to be a full-time position. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 61 12. USAID Participation The desirability of USAID participation in evaluation activities such as data collection will be considered in consultation with USAID and the evaluation team, and any specific roles and responsibilities of USAID staff will be described in the evaluation design proposal. 13. Scheduling and Logistics The following table provides the originally anticipated timeframe for evaluation activities and deliverables. Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Concept Paper Scoping Trip Evaluation Design Proposal Survey Pre-Test Enumerator Training Baseline Data Collection Baseline Data Analysis and Report Oral Presentation of Baseline Findings Kenya RAPID Program Implementation Endline Data Collection and Analysis Endline Report Oral Presentation(s) Final Report Tasks FY 16 FY 17 FY 18 FY 19 FY 20 The evaluation team will be responsible for procuring all logistical needs such as work space, transportation, printing, translation, and any other forms of communication. USAID will offer some assistance in providing introductions to partners and key stakeholders as needed and will ensure the provision of data and supporting documents as possible. 14. Reporting Requirements The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note-preparing-evaluation￾reports). The final version of the evaluation report will be submitted to USAID and should not exceed 30 pages, excluding references and annexes. All members of the evaluation team will be provided with USAID’s mandatory statement of the evaluation standards they are expected to meet, shown in the following text box below, along with USAID’s conflict of interest statement that they sign and return before field work starts. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 62 15.Budget The evaluation team responding to this SOW will propose a notional budget in its concept paper for this evaluation, including cost implications of the methodological options proposed. A full detailed budget will then be prepared and included in the evaluation design proposal for USAID’s approval. USAID EVALUATION POLICY, APPENDIX 1 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 63 ANNEX B: BOREHOLE ASSET INVENTORY SURVEY INSTRUMENT SweetSense designed this instrument to collect information on boreholes. The evaluation team used a slightly revised version as part of baseline data collection, which is provided below. Impact Evaluation Asset Inventory Survey Interviewer details: Name of the interviewer ID no. of the interviewer Name of the supervisor ID no. of the supervisor Introduction Good morning/afternoon! I am ____ from Ipsos, a survey and market research company. We are currently conducting a study to better understand water use in this community. Your opinion and knowledge would be incredibly helpful for supporting national efforts to address water management and drought resiliency. Many of the questions I will ask are related to the local water borehole and based on the current context here. There are no right or wrong answers, and please be assured that the information collected from you will be treated completely confidentially. Water supply facilities A. General Information Water system (distribution scheme linked to this update) A1. Name/description of the water system A2. Unique water system ID A3. GPS of the borehole location (This is to be done at the borehole exact location.) A4. Take picture of the water system: ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 64 (INTERVIEWER NOTE: THE PICTURE SHOULD CAPTURE THE WHOLE BOREHOLE SYSTEM PLUS THE SURROUNDINGS [I.E., TANKS, KIOSK] IN ONE CAPTION.) A5. County Baringo Kitui Laikipia Mandera Meru Samburu Tana River West Pokot A6. Sub-County A7. Village A8. Rural/Urban Rural Urban A9. Local Officer Name (INTERVIEWER NOTE: RECORD THE NAME OF THE PERSON RESPONSIBLE FOR BOREHOLE MAINTENANCE/PERSON IN CHARGE.) A10. Local Officer Position Operator Water committee chairperson Other (specify) A11. Local officer telephone number ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 65 B. Users/ Hali ya matumizi B1. Total number of households currently served from the scheme? Jumla ya idadi ya nyumba zinazohudumiwa na mfumo huu wa maji? B2. What is the service area for this scheme? How many kilometers from here do the pipes run? Je, eneo linalohudumiwa na mfumo huu ni kubwa kiasi gani - mifereji imeenea umbali wa kilomita ngapi kutoka hapa? B3. Is this scheme used for crop farming (irrigation) by farmers? Je, mfumo huu unatumika na wakulima katika kilimo cha unyunyiziaji? Yes Ndiyo No La (SKIP TO B6) If Yes: kama ndiyo B4. How many farmers use this scheme? Je, ni wakulima wangapi hutumia huu mfumo wa maji? If Yes: kama ndiyo B5. What is the average sized farm that uses this water (acres)? Je, kwa wastani shamba zinazotumia maji haya zina ukubwa kiasi gani (hekari)? B6. Is the water scheme used for livestock? Je, Huu mfumo wa maji unatumika na mfugo? Yes Ndiyo No La (SKIP TO B11) If Yes: kama ndiyo B7. Estimated number of camels using the scheme (per day) Kadri ya idadi ya ngamia wanaotumia hiki kituo cha maji (kwa siku) If Yes: kama ndiyo B8. Estimated number of cattle using the scheme (per day) Kadri ya idadi ya ngombe wanaotumia kituo hiki cha maji (kwa siku) ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 66 If Yes: kama ndiyo B9. Estimated number of horses/donkeys using the water point (per day) kadri idadi ya farasi/punda wanaotumia kituo hiki cha maji (kwa siku) If Yes: kama ndiyo B10. Estimated number of goats/sheep using the water point (per day) Kadri idadi ya mbuzi /kondoo wanaotumia kituo hiki cha maji (kwa siku) B11. Is the water scheme used for water trucking? Je, kituo hiki cha maji kinatumika na maroli kuteka maji? Yes Ndiyo No La (SKIP TO C1) If Yes: kama ndiyo B12. Number of trucks filling per day Idadi ya malori yanayoteka maji kwa siku B13. How many days per week does the water trucking occur? Ni siku ngapi kwa wiki,maji hutekwa kwa maroli? B14. Any other comments on water trucking details? Ongeza maelezo zaidi kuhusu utekaji wa maji kwa kutumia malori? (INTERVIEWER NOTE: GIVE ADDITIONAL DETAILS SUCH AS SIZES OF TRUCKS, THE AREA THEY COVER, USE OF WATER TRUCKED, ETC.) C. Borehole Information Habari ya Visima Borehole Kisima C1. Construction year of the well/borehole in EFY (or the year of rehabilitation) Mwaka wa ujenzi wa kisima kwa EFY (au mwaka wa urekebishaji) ☐ Don't Know Sijui C2. Borehole or well depth (meters) Kina cha kisima (Mita) (INTERVIEWER NOTE: RECORD 000 FOR DON’T KNOW [VOLUNTEERED]) ☐ Don't Know Sijui (FOR REFERENCE TO THE WATER/COUNTY AUTHORITIES) ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 67 C3. Physical state of the well/borehole Hali ya kisima ilivyo (INTERVIEWER NOTE: THIS INFORMATION REFERS TO THE PERCEPTION OF THE COMMUNITY ON THE BOREHOLE.) Function well Inafanya kazi kwa kawaida Poor Mbovu Doesn't function Haifanyi kazi Unsure Sina uhakika (DO NOT READ OUT) C4. Type of Power Aina ya nguvu za umeme (MULTIPLE RESPONSE) Generator Jenerata Solar Solar Utility Power (Grid power) Matumizi ya nguvu za umeme (umeme kutoka kwa Gridi ya taifa) C5. Production of system: yield or discharge (meters cubed/hour) Hali ya kazi ya Mfumo: utoaji wa maji (ujazo wa mita kila saa) (INTERVIEWER NOTE: RECORD 000 IF RESPONDENT CANNOT ESTIMATE.) C5a. How long does it take to fill the tank (in minutes)? Je, inachukua muda mgani kujaza tanki C5b. What is the volume of the tank? Je, tanki hili linaujazo kiasi gani? (INTERVIEWER NOTE: RECORD IN CUBIC METERS.) Borehole Location Eneo la kisima C6. Please take a picture of the borehole Tafadhali piga picha la kisima C7. Is there a water meter at the water source? Je, kuna mita ya maji katika chanzo cha maji? (INTERVIEWER TO OBSERVE.) Yes Ndiyo No La (SKIP TO C11) If Yes: kama ndiyo C8. Please take a picture of the Borehole Meter Tafadhali piga picha ya mita kwa kisima (INTERVIEWER NOTE: ENSURE CURRENT READING IS VISIBLE IN THE PICTURE.) ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 68 C9. Is the water meter working? Je, mita ya maji inafanya kazi? Yes Ndiyo No La (SKIP TO C11) If Yes: kama ndiyo C10. What is the current reading of the water meter? Je, mita inasoma nukta ngapi kwa wakati huu? Scheme Functionality Hali ya Utendakazi wa mfumo C11. Is the water scheme currently functional? Je, kituo hiki cha maji kinafanya kazi kwa sasa? (INTERVIEWER NOTE: THIS REFERS TO WHETHER THE BOREHOLE IS PRODUCING WATER. IF SO, IS THE AMOUNT OF WATER PRODUCED AS PER DESIGN OR IS IT REDUCED YIELD?) Functional (producing as designed) Kinafanya kazi kama ilivyotarajiwa (SKIP TO C15) Partially functional (reduced yield) Kinafanya kazi kwa kiwango cha chini Non-functional Hakifanyi kazi Abandoned Kimetelekezwa/kiliacha kufanyakazi (SKIP TO SECTION E) If the water scheme is partially functional or non- functional: C12. Please take a picture illustrating the non-functionality or partial functionality Tafadhali piga picha inayoashilia hali ya kutofanya kazi au hali ya kufanya kazi kwa kiwango (INTERVIEWER NOTE: TAKE MORE THAN ONE PICTURE IF MORE THAN ONE PART OF THE SYSTEM IS NOT FUNCTIONING.) If the water scheme is partially functional or non-functional: C13. Main cause of non-functionality or partial functionality Je, ni sababu gani kuu inaeleza utotenda kazi au utenda kazi kiasi wa kituo hiki cha maji (MULTIPLE RESPONSE) Insufficient water at source Hakina maji ya kutosha Distribution pipeline or tap failure Mabomba au mifereji ya usambazaji maji haifanyi kazi. No gas for generator Hakuna mafuta ya jenerata Generator failure Jenerata haifanyi kazi Grid power failure Stima kutoka kwa gridi ya taifa haifanyi kazi Solar power failure Kifaa cha nguvu za umeme (sola) hakifanyi kazi Submersible pump failure Pampu inayopandisha maji haifanyi kazi ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 69 Switchboard (electrical) failure Kifaa kinacho kuunganisha nguvu za umeme hakifanyi kazi Other (please specify) Nyingine (tafadhali taja) If the water scheme is partially functional or non-functional: C14. Number of months since non-functional Idadi ya miezi tangu kituo hiki kufanya kazi (SKIP TO QUESTION D1 IF C11 IS CODED NON-FUNCTIONAL) If the water scheme is currently functional or partially functional: C15. Was the water scheme out of service one or more days in the last month? Je, kituo hiki cha maji hakikufanyi kazi kwa siku moja au zaidi kati ya mwezi mmoja uliopita? (INTERVIEWER: ENSURE THAT RESPONDENT UNDERSTANDS THE TIME FRAME BEFORE RESPONDING [I.E., THE PAST 4 WEEKS FROM THE DAY OF INTERVIEW].) Yes Ndiyo No La (SKIP TO C17) If Yes: kama ndiyo C16. Number of days the scheme was out of service in the last month Idadi ya siku kituo hiki hakikufanya kazi mwezi uliopita? C17. Was the water scheme out of service one or more days in the past 12 months? Je, kituo hiki cha maji hakikufanyi kazi kwa siku moja au zaidi kati ya miezi kumi na mbili iliyopita? (INTERVIEWER: ENSURE THAT RESPONDENT UNDERSTANDS THE TIME FRAME BEFORE RESPONDING.) Yes Ndiyo No La (SKIP TO C20) C18. How many times was the water scheme broken in the past 12 months? Je, ni mara ngapi kituo hiki cha maji hakikufanya kazi (kuharibika) kati ya miezi kumi na mbili iliyopita. Has never broken in the last year Hakijawai haribika kati ya mwaka mmoja uliyopita (SKIP TO C20) ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 70 C18a. What was the nature of the failure? Je, shida ilikuwa ni aina gani? (SELECT MULTIPLE) Pump failure Pampu ilikosa kufanya kazi Generator/power failure Jenerator/Stima ilikosa kufanya kazi Pipe failure Mifereji haikufanya kazi Tank failure Tanki la maji lilikosa kufanya kazi Other (please specify) Nyingine (tafadhali taja) C19. Describe the functionality problem/s over the past year Elezea shida za utotenda kazi wa kituo kwa mwaka mmoja uliopita C20. Does the scheme have any emerging problems that might lead to non-functionality in the near future? Je, mpango umekuwa na matatizo yanayoimbuka ambayo yanaweza sababisha kutotenda kazi kwa siku zijazo? Yes Ndiyo No La (SKIP TO D1) If yes, kama ndiyo C21. Please describe the problems that might lead to non-functionality in the near future Tafadhali elezea shida zinazoweza kuchangia kutofanya kazi kwa siku zijazo D. Scheme Usage Patterns Mpango wa Matumizi ya Mfumo D1. How many days per week does the pump run on average? Je, Kwa kawaida pampu hii hufanya kazi siku ngapi kwa wiki? D2. How many hours per day does the pump run on average? Je, Kwa kawaida pampu hii hufanya kazi masaa mangapi kwa siku? D3. Is this a seasonal water scheme that commonly fails in the dry season? Je, kituo hiki cha maji hutumika kwa msimu, kama mfano, huwa kinakauka wakati wa kiangazi? Yes Ndiyo No La Don’t know Sijui (DO NOT READ OUT) ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 71 D4. Does frequency of use of the pump depend if it is wet/rainy season? Je, hali ya utendakazi ya pampu inaegemea msimu, kama vile wakati wa unyevu au mvua? Yes Ndiyo No La Don’t know Sijui (DO NOT READ OUT) D5. During wet/rainy seasons, how many days a week does the pump run on average? Wakati kuna unyevu/mvua, ni siku ngapi kwa wiki pampu hii huwa inafanya kazi? D6. During wet/rainy seasons, how many hours per day does the pump run on average? Wakati kuna unyevu/mvua, ni masaa mangapi kwa siku pampu hii huwa inafanya kazi? E. Solar Power (INTERVIEWER: ONLY ASK THOSE WHO HAVE MENTIONED SOLAR IN C4) E1. Are solar panels tracking type? Je, paneli sola zinafuatilia miale ya jua kwa kujiegesha/kuzunguka? (INTERVIEWER NOTE: PROBE THE RESPONDENT AND CONFIRM ON THE SOLAR PANELS.) Yes Ndiyo No La E2. Solar panel power rating (watts) Kipimo cha nguvu za umeme ya paneli za sola (wati) (INTERVIEWER NOTE: ASK THE RESPONDENT AND CONFIRM ON THE SOLAR PANELS.) Unknown/Unavailable Haijulikani/Haiko E2a. Number of solar panels Idadi ya paneli za sola E2b. Wattage of single panel Nguvu za umeme (wati) za paneli sola moja E3. Solar Panel condition Hali ya paneli za sola (INTERVIEWER NOTE: FUNCTIONALITY AND PERCEPTION OF EFFICIENCY) Good Nzuri ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 72 Fair Nzuri kiasi Bad Mbaya Unknown Haijulikani E4. Solar panels picture Picha za paneli sola F. Generator Jenerata (ONLY ASK THOSE WHO HAVE MENTIONED GENERATOR IN C4) F1. Year when generator was installed (or year of re-installation) Mwaka ambao jenerata iliwekwa (au mwaka ilipobadilishwa) ☐ Don't know Sijui F2. Physical state of the generator Hali jenerata inavyoonekana kwa sasa? (INTERVIEWER NOTE: FUNCTIONALITY AND PERCEPTION OF EFFICIENCY) Normal Kawaida Poor Mbaya/mbovu Doesn't function Haifanyi kazi Unsure Sina uhakika F3. Generator brand Aina ya jenerata (INTERVIEWER NOTE: ASK TO SEE THE CONTROL PANEL [NORMALLY OUTSIDE THE GENERATOR].) Perkins Lister Petter Cummins Caterpillar Coelmo Green Power Iveco Isuzu Lovol Stamford Other (please specify) Zingine (tafadhali taja) Don’t know Sijui F4. Generator Model Modeli ya jenerata Don’t Know Sijui ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 73 F5. Generator serial number Nambari ya usajili ya jenerata Don’t know Sijui F6. Generator max amps (starting amps) Hali ya utoaji nguvu za umeme ya jenerata (Amps za kuanzia) Don’t Know Sijui F7. Generator alternator serial number Nambali ya usajili ya altanata ya jenerata Don’t know Sijui F8. Generator picture Picha ya jenerata F9. Picture of generator technical details/nameplate Picha ya maelezo ya kiufundi ya jenerata / bati lenye jina Unavailable Hakuna G. Pump G1. Pump Controller Manufacturer Watengenezaji wa kidhibiti pampu (INTERVIEWER NOTE: SEE TECHNICAL DETAILS FROM THE PLACARDS. IF NOT AVAILABLE, CHECK WITH COUNTY/WATER AUTHORITIES FROM THE COUNTY ENGINEERS, WATER RESOURCE AUTHORITY.) Grundfos Lorentz Other (please specify) Nyingine(tafadhali taja) G2. Pump controller model number Nambari ya mfumo ya kuthibiti pampu Unavailable Hakuna G3. Picture of pump controller Picha ya kithibiti pampu ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 74 G5. Pump motor model number Nambari ya mfumo/modeli ya mtambo wa pampu Unavailable Hakuna G6. Pump head model number Nambari ya mfumo/ modeli ya kichwa cha pampu Unavailable Hakuna G7. Well pump remarks Maelezo zaidi kuhusu hali ya pampu G8. Picture of pump controller placards with technical details Picha ya kadi (placards) kithibiti pampu iliyo na maelezo ya kiteknikali Unavailable Hakuna H. Contact Details for Borehole (INTERVIEWER NOTE: WATER SCHEME OPERATOR/ WASHCO CHAIRPERSON/MEMBER) H1. Contact 1 name H2. Contact 1 position WASHCO Chairperson Scheme operator WASHCO member Other (please specify) H3. Contact 1 telephone number H4. Contact 2 name H5. Contact 2 position WASHCO Chairperson Scheme operator WASHCO member ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 75 Other (please specify) H6. Contact 2 telephone number H7. Were additional contacts made? Yes No (Skip to Section I) H8. Contact 3 name H9. Contact 3 position WASHCO Chairperson Scheme operator WASHCO member Other (please specify) H10. Contact 3 telephone number H11. Were additional contacts made? Yes No (Skip to Section I) H12. Contact 4 name H13. Contact 4 position WASHCO Chairperson Scheme operator WASHCO member Other (please specify) H14. Contact 4 telephone number I. Sensor Information Habari ya vipokezi (Sensor) (DO NOT ASK IF C11 IS LISTED AS “ABANDONED.”) ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 76 Sensor Ia. Is there already a sensor installed at the site? Je, kunayo Sensor imewekwa katika kituo cha maji? (INTERVIEWER NOTE: THIS DOES NOT REFER TO THE PUMP SENSOR.) Yes Ndio No La (SKIP TO I1) Ib. Picture of sensor installation Picha ya sensor iliyowekwa Ic. Sensor installation notes Maelezo kuhusu kuwekwa kwa Sensor Id. Type of Sensor Input Aina ya Sensor iliyowekwa Pressac 3 Phase Clamp Pressac 1 Phase Clamp Flow Meter DC Clamp On/Off Clamp VFD Clamp Hardwired Unknown Haijulikani None Hakuna I1. Was a sensor installed at the site? Je, Sensor iliwekwa kwenye kituo cha maji Yes Ndio No La (SKIP TO J1) If Yes: I2. Type of sensor installed Aina ya sensor iliyowekwa Dent Instruments TOUCT-4G CT I3. Date of Installation of Sensor Tarehe sensor iliwekwa I4. Sensor serial number Nambari ya usajili ya sensor I5. Picture of sensor installation Picha ya sensor ilivyowekwa (INTERVIEWER NOTE: TAKE THE PICTURE OF THE INSTALLED SENSOR SHOWING THE CLAMPED WIRE AND THE POSITION OF THE DATA LOGGER.) ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 77 I6. Sensor installation notes Maelezo kuhusu vile sensor iliwekwa I7. Picture of sensor input Picha ya sensor (INTERVIEWER NOTE: TAKE THE PICTURE OF THE DATA LOGGER SCREEN 5 MINUTES AFTER INSTALLING AND SWITCHING ON THE. DATA LOGGER. THE PICTURE SHOULD CAPTURE CURRENT READING OF THE DATA LOGGER.) J. Management Usimamizi J1. Management Body Utility WASHCO No management organization Others (specify)_______________ If management body is utility or WASHCO: J2. Current Management Status WASHCO or utility is active WASHCO or utility is not active K. Tariffs Malipo K1. Type of tariff system (most common) Aina ya mfumo wa malipo inayotozwa (kwa kawaida) Fixed tariff per visit Kitengo maalum cha malipo kinachotozwa kwa ziara moja Fixed tariff per week Kitengo maalum cha malipo kinachotozwa kwa kila wiki Fixed tariff per month Kitengo maalum cha malipo kinachotozwa kwa kila mwezi Fixed tariff per half year Kitengo maalum cha malipo kinachotozwa kwa nusu mwaka Fixed tariff per year Kitengo maalum cha malipo kinachotozwa kwa kila mwaka Tariff per jerrycan (20 litre) Kitengo maalum cha malipo kinachotozwa kwa kila jerrican (lita ishirini) Tariff per cubic meter (m3) Kitengo maalum cha malipo kinachotozwa kwa kila ujazo wa mita Ad hoc contributions Malipo maalum inayotozwa kwa dharura (SKIP TO K3) No payment Hakuna malipo (SKIP TO K3) ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 78 If type of tariff system (most common) is one of fixed tariff per visit, fixed tariff per week, fixed tariff per month, fixed tariff per half year, fixed tariff per year, tariff per jerrycan (20 litre), or tariff per cubic meter (m3): K2. What is the tariff amount (in KES)? Je, ni kiasi gani cha malipo kinatozwa (kwa KES)? K3. Is there a special tariff for livestock? Je, kuna malipo maalum yanayotozwa kwa mifugo (kwa kila mnyama)? Yes Ndiyo No La (SKIP TO K8) If Yes: K4. What is the tariff per camel (KES)? Je, ni kiasi gani cha malipo hutozwa kwa kila ngamia (KES)? Not applicable Sio husika If Yes: K5. What is the tariff per cattle (KES)? Je, ni kiasi gani cha malipo hutozwa kwa kila ng’ombe (KES) ? Not applicable Sio husika If Yes: K6. What is the tariff per horse/donkey (KES)? Je, ni kiasi gani cha malipo hutozwa kwa kila farasi/punda (KES)? Not applicable Sio husika If Yes: kama ndiyo K7. What is the tariff per goat/sheep (KES)? Je, ni kiasi gani cha malipo hutozwa kwa kila mbuzi/kondoo (KES)? Not applicable Sio husika K8. Is there a tariff per institution (schools, health centers, etc.)? Je, ni kiasi gani cha malipo hutozwa kwa kila taasisi(shule,vituo vya afya n.k.)? Yes Ndiyo ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 79 No (but institutions are served by borehole) La (Lakini kituo hiki kinahudumia tasisi) (SKIP TO L1) No institutions are served by borehole La, hakuna taasisi zina hudumiwa na kisima hiki (SKIP TO L1) If Yes: K9. What is the monthly institutional tariff (KES)? Je, ni kiasi gani cha malipo hutozwa taasisi kila mwezi? If Yes: K10. Number of institutions supplied by scheme that pay a monthly institutional tariff Idadi ya taasisi zinazosambaziwa maji na mfumo huu ambazo zinatozwa malipo yaliyotengwa kwa taasisi ya kila mwezi Don’t know Sijui (DO NOT READ OUT.) L. Time Wakati L1. When this borehole is working, from where do households primarily get their water? Wakati hiki kituo hiki cha maji kinafanya kazi, nyumba zinapata maji yao kutoka wapi hasa? Household tap Mfereji nyumbani Community tap Mfereji ya jamii Other (please specify) Nyingine (tafadhali taja) L2. When this borehole is working, how many hours per trip do households typically spend collecting water? Je, kwa kawaida, Wakati hiki kituo kinafanya kazi, inachukua nyumba muda wa masaa mangapi kila siku kuchota maji? (INTERVIEWER NOTE: THIS INCLUDE TO AND FROM AND WAITING TIME AT THE BOREHOLE.) L3. When this borehole is not working, where do households primarily get their water? Je, kwa kawaida ni wapi nyumba hupata maji yao, Wakati hiki kituo cha maji hakifanyi kazi? A different water tap Mfereji tofauti wa maji A surface water source Maji yaliyolea juu ya ardhi Rainwater Maji ya mvua ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 80 Other (please specify) Nyingine (tafadhali taja) Don’t know Sijui (SKIP TO M1) Has never happened Hakijawahi kosa kufanya kazi (SKIP TO M1) L4. When this borehole is not working and households have to use an alternative water source, how many hours per trip do households spend in collecting water? Je, Wakati kituo hiki cha maji hakifanyi kazi na nyumba zinafaa kutumia njia mbadala za kupata maji, kwa kawaida huwa inachukua nyumba muda wa masaa mangapi kila siku kuchota maji? (INTERVIEWER NOTE: THIS INCLUDE TO AND FROM AND WAITING TIME AT THE ALTERNATIVE WATER SOURCE.) Don’t Know Sijui M. Water quality Issues Hali ya ubora wa maji (INTERVIEWER NOTE: MOST INFORMATION IS AVAILABLE AT THE WATER OFFICES AT DISTRICT OR COUNTY LEVEL.) M1. Has there been some water quality testing for this scheme in the past? Je, hapo awali kumekuwa na upimaji wa ubora wa maji uliofanyika katika kituo hiki cha maji? Yes Ndiyo No La (SKIP TO M4) Don’t know Sijui (SKIP TO M4, THIS REQUIRES FOLLOW-UP WITH COUNTY/WATER AUTHORITY.) If Yes: M2. Year of most recent water quality test done Upimaji wa ubora wa maji ulifanyika mwaka gani hivi karibuni Don’t know Sijui If Yes: M3. What was the result? Je, matokeo yalikuwa gani? Water is safe for drinking Maji ni salama kwa kunywa Water is unsafe for drinking Maji sio salama kwa kunywa Don’t know Sijui ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 81 M4. Are there specific contaminants of concern? Je, kuna uchafu maalum wa kutahadharisha? Fluoride Iron Salt (salinity) Other (please specify) Nyingine (tafadhali taja) None Hakuna General Kijumla N1. Please add any other relevant notes on the borehole Tafadhali ongeza maelezo mengine muhimu kuhusu kituo hiki cha maji ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 82 ANNEX C: ROUND 1 QUALITATIVE DATA COLLECTION QUESTIONNAIRES C1 - Key Informant Interview Script Respondent Information Respondent name: Location of interview (county, town/village): Respondent organization: Respondent job title: Date of Interview: Interview Questions (semi-structured interview – use script as a guide) 1. Please describe your involvement in water management. (Have respondent describe their role, and then probe as needed). a. How long have you held this job? b. How long have you worked in water management? c. How many water points do you help manage? (Note: water points are a larger category that includes boreholes. From SweetSense: “The convention we stick to is that ‘boreholes’ are mechanized, powered pumps that distribute to schemes.” Other types of water points include handpumps, elevated tank systems, sand dams, etc.) d. How many boreholes do you help manage? i. Do you know how many of the boreholes you manage are designated as “ending drought emergency (EDE)” or strategic boreholes? (Note: not all water managers will know which boreholes are designated as EDE. If they don’t know, probe for info on which boreholes are critical during the dry season. Throughout the rest of the script, use either “EDE borehole” or “county strategic” borehole to refer to these critical boreholes, depending on how the respondent identifies them.) ii. Do you know how the EDE/strategic boreholes are decided or defined? What makes these boreholes different from other boreholes? e. What is your role in managing these boreholes? f. Are the water points you manage mainly rural, urban, or a mix? 2. What is the management structure of the water committees for the water points that you help manage? a. Are these water points managed by local water committees? WASCOs (utilities)? Other organizations? i. For water committees: how many people are in these committees, and how does someone become a member? b. Who is responsible for operating the site (e.g., turning the borehole on and off)? ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 83 c. Who is responsible for maintaining and repairing boreholes? d. Are fees collected for water use? e. Are fees collected when the water point needs maintenance? f. Who is responsible for collecting and managing fees? g. How do the different actors involved in managing this borehole interact with one another? (Where relevant, probe for: i. Local operator relationship with county government ii. WASCO relationship with county government iii. County government relationship with operations and maintenance organizations like Catholic Diocese in Turkana iv. WASCO relationship with rate payers) 3. What are the main challenges you face in providing reliable water service to users in this area? a. Do you receive any complaints about the borehole from users during the drought season? b. What kind of complaints? c. How do you typically receive these complaints? d. Are you able to address these complaints? If not, why? 4. What are the main challenges that you experience in providing reliable water service during the dry season? a. Do the challenges differ during the dry season from other times of year? If so, how? b. How do you or your team manage or mitigate these challenges? 5. How many of the strategic boreholes that you manage are currently functional? a. What are the most typical issues with these boreholes? b. About how many times did an EDE/country strategic borehole break during the past long dry season? Was this more/less/about the same as usual during this time of year? c. What do you think is the main cause of breakages? Which parts typically break (pump, generator, power source, etc.)? 6. When EDE/county strategic boreholes in your area break, how long does it take… a. For the water office to find out the borehole is broken? b. To get information about the breakages (i.e., specifically what is broken and what repairs are needed)? c. To repair the broken boreholes? d. Do the answers to these questions differ between the dry season and wet season? ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 84 7. Please think about the most recent time when you experienced a broken EDE/county strategic borehole during the drought season. Can you describe what happened? a. How did the borehole break? b. Where was this borehole? c. Who was responsible for repairing the borehole? d. How did the person/organization responsible for fixing the borehole learn that it was broken? (Who reported the information? How did they report it? How long was the borehole broken before it was reported?) e. After the information was reported, what happened? Who repaired it? How long did it take for the borehole to be repaired? f. Looking back on this episode, what worked well? g. Looking back on this episode, what could have worked better? What problems were encountered? 8. What are the main reasons for delays in fixing broken EDE/county strategic boreholes? a. During the long dry season? b. At other times of year? 9. Does the management of EDE/county strategic boreholes differ from management of other boreholes in this area? If so, how? a. Are these boreholes operated year-round, or only during the dry season? b. Does the management structure for these boreholes differ from what was discussed in Question 2? If so, how? 10. I would like to learn more about the resources here to address borehole issues. How often do you need to request funds or resources, such as equipment or skilled technicians, to fix an EDE/county strategic borehole? a. Who do you talk to about the budget for these boreholes? i. What is that process like? b. Are there constraints to obtaining the funds needed to address the management and maintenance of these boreholes? Explain. c. If you can obtain the budget needed, do you feel that you have the right staff capacity for managing these boreholes? 11. What suggestions do you have for ways to improve management of boreholes in this area? a. Who would need to be involved in implementing these suggestions? b. What opportunities do you see for taking these steps? c. What barriers do you see for better drought management? ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 85 For respondents in Kenya RAPID counties only: 12. How familiar are you with the sensor-based systems that have been implemented to manage EDE/strategic boreholes in this county? a. Do you use the Kenya RAPID system? If so, describe how you use it. 13. Who is responsible for reviewing and managing the sensor data in your county? 14. Where would you get resources to repair borehole after a breakdown detected by the sensors? a. Can you describe this process? b. Has the process for requesting resources for repairs changed since Kenya RAPID began implementing here? c. What are the procedures for repair after obtaining resources? 15. Have you personally used the data dashboard to view sensor information on EDE/strategic boreholes? 16. Do you manage or interact with other people who have used the dashboard? 17. Have you encountered any challenges using the sensor/dashboard/phone application? What happened? 18. What do you think about the system with respect to your water borehole management activities? a. Do you feel the information/ data transmitted by the sensor is relevant? Do you feel that it is accurate? Why/why not b. Do you feel that the training you received on the dashboard/phone application is sufficient to help you in managing the boreholes? Why/why not? c. Are there any management challenges that you feel the Kenya RAPID system does not help address with respect to strategic borehole management? d. Do you feel that the county has enough capacity to manage the data dashboards? Why or why not? e. How could this system be improved? For respondents in non-Kenya RAPID counties only: 19. Suppose that you were able to access information very quickly about which boreholes were working/not working. How would you use this information? Do you think this would help with management of these boreholes? If so, how? Explain. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 86 C2 - Group Discussion Guide This guide should be used for all GD respondent categories. Total number of people in GD: ____________ Number of women: ____________ A. Overview of Water Use History 1. Please tell us about this area. Facilitator probes: a. How do most people earn a living here? b. What are the main sources of water that people in this use? i. During the short dry season? ii. During the short rainy season? iii. During the long dry season? iv. During the long rainy season?) 2. How many people here use the local borehole? Refer to the focus borehole. Facilitator probes: a. For what types of activities do people use the borehole? (Probe for: personal use, livestock, irrigation) b. For those that do not use local boreholes (if any): Why don’t you use them? B. Borehole Access and Use 1. Can you access the water from this borehole without any challenges? a. How many people here have personally experienced a challenge accessing water from this borehole? b. What challenges have you faced? (Facilitator probes: Physical access? Any rules imposed by managers?) c. Are there any groups or types of people who face more of these types of challenges, or have greater challenges accessing water? i. Women? Elderly? ii. (If yes): Why? 2. Do people need to pay to get water from this borehole? How much? 3. For (each main challenge listed regarding access), what is the solution? a. Are there any institutions available here to help resolve access challenges? How well do you trust these institutions to assist you? Can you give me an example? 4. Does borehole accessibility vary over the month, or over the year? For example, based on the season or who is in charge? 5. Have people here in this group, or others you are familiar with, contacted the local borehole manager regarding accessibility and service? ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 87 a. What were the reasons for this? b. Is this issue common here, or is it uncommon? c. How often have you contacted the water manager or heard about people contacting him or her? d. When was the last time you or someone you know contacted the local borehole manager? C. Borehole Maintenance and Management 1. Do you know who is in charge of fixing this borehole when it breaks? Who is it? a. Is there someone who is in charge of contacting the water manager when the borehole breaks? How do they contact the water manager? 2. When was the last time this borehole broke? 3. Are there financial resources for the local water borehole managers to maintain the borehole? a. Where do they come from? b. Are they sufficient? Why or why not? Give examples. c. Have the managers or other representatives from the government or aid program talked to borehole users about maintenance? 4. Are there currently any maintenance challenges at this borehole? For example, equipment failure/breakdowns? a. Are there some times of year when maintenance challenges are more likely to occur? b. Do you think the borehole managers plan for seasonal or other known issues that arise each year? 5. Was the management of this borehole always like this? Has it changed over time? a. What has changed? (Ask for examples, such as staff, community outreach, resources.) b. Do you think these changes are positive or negative for accessing water? D. Drought Risk 1. Please think about the most recent time drought season. a. Were you able to access water for you and your household during the drought? b. What water sources did you use during the drought? c. Did you have any trouble accessing water from this borehole? d. Did this borehole break? i. Do you know how it broke? ii. Who was responsible for repairing the borehole? ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 88 iii. How long did it take for the borehole to get repaired? 2. During the dry/drought season, do you think that some people here face larger water challenges than others? a. Do women or the elderly (or other groups) have the same access to water as everyone else? 3. What suggestions do you have for ways to improve management of boreholes in this area? b. Who would need to be involved in implementing these suggestions? c. What opportunities do you see for taking these steps? d. What barriers do you see for better drought management? E. Recorded responses No. Question # of hands YES # of hands for NO 1 How many people here feel they can rely on the local borehole for their water needs during the wet season? 2 How many people here feel they can rely on the local borehole for their water needs during the dry season? 3 How many people here are comfortable contacting the local water manager if they see there is an issue with the borehole? 4 How many people here are confident that the water manager will address an issue in a reasonable amount of time if it is reported by a member of the community? 5 How many people here believe the borehole is at risk during the next drought? F. Conclusion Thank you for discussing these issues with us today. We asked a lot of questions. Is there anything that you want to add, or would like to ask us? ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 89 ANNEX D: ROUND 1 CONSENT SCRIPTS D1 - Key Informant Consent Script Qualitative Evaluation of Kenya Resilient Arid Lands Partnership for Integrated Development INFORMED CONSENT: KEY INFORMANT INTERVIEWS INTRODUCTION Hello, my name is ____________________________________________. I am working with Aquaya, and with the Colorado School of Public Health and WASHPALS in the United Stated. We are doing a study about water services, drought, and other issues in this area. We understand that you are involved in the management of boreholes in this area, and would like to talk to you about your experiences. STUDY PROCEDURE We are interested in learning about how boreholes are managed for water service delivery in this area, especially during times of drought. If you agree to participate in this study, we conduct an interview with you that will take about one hour. If you are willing, we will come back in about one year to interview you again. STUDY OBJECTIVE The objective of this study is to learn about water use in arid and semi-arid lands in Kenya. There are no right or wrong answers, but we do want to understand your perceptions and experiences around water use in this area toward an objective of assessing current and informing future water management programs. RECORDING With your permission, I will audiotape and my colleague will take notes during the discussion. The recording is to accurately capture the information you provide and will be used for transcription purposes only. Excerpts from the recordings/transcripts may be used to illustrate the research findings. This will always be done in a way to protect your identity (that is, your name will not be used). VOLUNTARINESS Taking part of this study is completely voluntary. You have every right to refuse to participate. If you should refuse, you will not suffer any consequences. WITHDRAWAL If you chose to participate in this study, you have the right to withdraw from it at any point in time without any consequences to you. You are free to skip any questions you do not wish to answer or to stop at any time. You may ask the researchers any questions you have at any time. BENEFITS AND RISKS OF PARTICIPATION This study will benefit you and your community by making known the water context here for decision makers at USAID and its Government of Kenya partners. All of the information we collect today will be shared anonymously with these decision makers to help them assess current water programs and understand what future programs may be best suited to communities like yours. There are no direct ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 90 risks in talking with us today, but as we mentioned, you are free to withdraw from the discussion at any time. We are here to listen, record, and share your views, which will eventually benefit both this community and Kenya overall by informing water management practices. COMPENSATION There is no direct benefit from your participation. You will not be paid for your participation in this study. CONFIDENTIALITY We will need to connect your name to the information you give us, but only for a short time—while we are gathering information from many water managers. After that, we will save the information and report what we learn using numbers, not names. Only we, the researchers, will ever see the information with people’s names. QUESTIONS If you have any questions concerning the study, you can contact Mary Gichihi [removed] or KEMRI [0722-205-901]. ORAL CONSENT OF PARTICIPANT Do you agree to participate in this study?  Yes  No WRITTEN CONSENT OF PARTICIPANT Name Phone Number County/ Sub-county Age Gender Consent (Y/N) Signature/Mark ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 91 D2 - Group Discussion Consent Script Qualitative Evaluation of Kenya Resilient Arid Lands Partnership for Integrated Development INFORMED CONSENT: GROUP DISCUSSIONS INTRODUCTION Hello, my name is ____________________________________. I am working with Aquaya, and with the Colorado School of Public Health and WASHPALS in the United Stated. We are doing a study about water services, drought, and other issues in this area. We are not talking to all residents in this area, only some of them. You have been selected as someone who lives in this area and uses the local borehole. STUDY PROCEDURE We are interested in learning about how boreholes are managed for water service delivery in this area, especially during times of drought. If you agree to participate in this study, we conduct a group discussion with you that will take about two hours. If you are willing, we will come back in about one year to conduct a group discussion with you and other users in this area again. STUDY OBJECTIVE The objective of this study is to learn about water use in arid and semi-arid lands in Kenya. There are no right or wrong answers, but we do want to understand your perceptions and experiences around water use in this area toward an objective of assessing current and informing future water management programs. RECORDING With your permission, I will audiotape and my colleague will take notes during the discussion. The recording is to accurately capture the information you provide and will be used for transcription purposes only. Excerpts from the recordings/transcripts may be used to illustrate the research findings. This will always be done in a way to protect your identity (that is, your name will not be used). PHOTOGRAPHY Our team may take photos during this session. We will not publish anyone’s name or the specific location of these photos. The photos taken may be used in our reporting and presentations of the evaluation findings. The role of these photos will be to show the local context and all of the content will be treated respectfully to ensure the protection of your identity. VOLUNTARINESS Taking part of this study is completely voluntary. You have every right to refuse to participate. If you should refuse, you will not suffer any consequences. WITHDRAWAL If you chose to participate in this study, you have the right to withdraw from it at any point in time without any consequences to you. You are free to skip any questions you do not wish to answer or to stop at any time. You may ask the researchers any questions you have at any time. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 92 BENEFITS AND RISKS OF PARTICIPATION This study will benefit you and your community by making known the water context here for decision makers at USAID and its Government of Kenya partners. All of the information we collect today will be shared anonymously with these decision makers to help them assess current water programs and understand what future programs may be best suited to communities like yours. There are no direct risks in talking with us today, but as we mentioned, you are free to withdraw from the discussion at any time. We are here to listen, record, and share your views, which will eventually benefit both this community and Kenya overall by informing water management practices. COMPENSATION There is no direct benefit from your participation. You will not be paid for your participation in this study. CONFIDENTIALITY We will need to connect your name to the information you give us, but only for a short time—while we are gathering information from many households. After that, we will save the information and report what we learn using numbers, not names. Only we, the researchers, will ever see the information with people’s names. QUESTIONS If you have any questions concerning the study, you can contact Mary Gichihi [removed] or KEMRI [0722-205-901]. ORAL CONSENT OF PARTICIPANTS Do you agree to participate in this study?  Yes  No WRITTEN CONSENT OF PARTICIPANTS Name Phone Number Village Age Gender Consent (Y/N) Signature/Mark ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 93 ANNEX E: INFORMAL GROUP DISCUSSION QUESTIONS No. Question # of hands YES # of hands for NO 1 How many people here feel they can rely on the local borehole for their water needs during the wet season? 2 How many people here feel they can rely on the local borehole for their water needs during the dry season? 3 How many people here are comfortable contacting the local water manager if they see there is an issue with the borehole? 4 How many people here are confident that the water manager will address an issue in a reasonable amount of time if it is reported by a member of the community? 5 How many people here in this group, or others you are familiar with, contacted the local borehole manager regarding accessibility and service? 5.1 Only for “Yes” responses to 5: What were the reasons? [Select all that apply] A. Borehole leaking/not functional B. Water quality C. Tariffs/fees D. Usage issues or disputes E. Other 5.2 Only for “Yes” responses to 5: Was the issue resolved? 6 How many people here believe the borehole is at risk during the next drought? ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 94 ANNEX F: BOREHOLE VERIFICATION CHECKLIST In order to assess data quality and fidelity between the borehole asset survey data collected by SweetSense (in Kenya RAPID counties) and the evaluation team (in comparison counties), the evaluation team conducted an abbreviated survey (containing seven key questions, shown below) in the Kenya RAPID counties during qualitative data collection. Verifying Borehole Key Characteristics Questionnaire Geographic location County: Sub-County Village: Respondent Information Respondent first name: Respondent gender: Female / Male No. Question Answer Choices Code Logic Source A Is the sensor working? Yes No Cannot observe (Why) 1 0 3 A1 What is the power type of the borehole? Solar Generator Utility powered Hybrid 1 2 3 4 B1 What is the total number of households currently served from the scheme? B6 Is the water scheme used for livestock? Yes No Don’t know (why) 1 0 99 C Is the borehole functional? Yes, functional Yes, partly functional Non-functional Abandoned 1 2 3 4 >>C17 >>C17 C14 Number of months since the borehole has been non-functional C17 Was the water scheme out of service one or more days in the last year? Yes No Don’t know 1 2 99 D3 Is this a seasonal water scheme that commonly fails in the dry season? Yes No Don’t know 1 0 99 E Is there a formal tariff scheme? Yes No Don’t know 1 0 99 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 95 No. Question Answer Choices Code Logic Source Any comments or notes The table below provides a summary of the differences. One borehole in Baringo was recorded in the asset survey as utility-powered, while the qualitative data collection team recorded it as hybrid. Additional verification during Round 2 data collection will take place to confirm. County HHs Using Scheme Used for Livestock Months Since Non￾Functional Out of Service One or More Days in the Last Year Commonly Fails in the Dry Season Fixed Tariff Difference Type of Power Turkana NA Different 0 Same Same Same Different Turkana NA Same 0 Different Same Same Same Turkana 600 Different 0 Different Same Same Same Tana River 260 Same 0 Same Different Same Same Tana River -700 Same -13 Same Same Same Same Baringo 500 Same 0 Same Same Same Same Baringo -200 Same 0 Different Same Same Different Garissa -50 Same -1 Same Same Same Same Garissa 70 Same 0 Same Same Same Same ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 96 ANNEX G: KEY INFORMANT INTERVIEW LIST KII Age Gender Role of Respondent County Jurisdiction Area Date of interview 1 48 Male Director Garissa Garissa Nov. 5, 2018 2 49 Male Sub-County Water Officer Garissa Dadaab Nov. 7, 2018 3 40 Male Borehole Operator Garissa Dadaab Nov. 8, 2018 4 30 Male Assistant Borehole Operator Garissa Dadaab Nov. 8, 2018 5 54 Male Operations and Maintenance Manager Garissa Garissa Nov. 13, 2018 6 49 Male Electro-Mechanical Engineer & Water and Sanitation Company Technical Manager Tana River Tana-River Nov. 12, 2018 7 58 Male Sub-County Water Officer Tana River Tana-North Nov. 12, 2018 8 42 Male Borehole Operator Tana River Tana-North Nov. 13, 2018 9 24 Male Borehole Operator Tana River Tana-North Nov. 14, 2018 10 30 Male Borehole Operator Baringo Tiaty Nov. 20, 2018 11 56 Male Sub-County Water Officer Baringo Tiaty Nov. 21, 2018 12 48 Male Borehole Operator Baringo Tiaty Nov. 22, 2018 13 52 Male Sub-County Water Officer; County NGO Coordinator; Drought Response Manager Baringo Baringo Nov. 23, 2018 14 28 Female Water Engineer and Acting Sub-County Water Officer Turkana Turkana South Nov. 26, 2018 15 34 Male Production Officer and County Subordinate Technician Turkana Turkana Central Nov. 26, 2018 16 25 Male Borehole Operator Turkana Turkana South Nov. 27, 2018 17 29 Female Borehole Operator Turkana Turkana East Nov. 28, 2018 18 49 Male Borehole Operator Turkana Turkana South Nov. 29, 2018 19 34 Male Chief Drilling Superintendent Turkana Turkana Nov. 29, 2018 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 97 ANNEX H: GROUP DISCUSSION PARTICIPANT LIST GD Borehole County Participant Village Age 1 Kumahumato Garissa Banan 55 1 Kumahumato Garissa Jerin 18 1 Kumahumato Garissa Banan 40 1 Kumahumato Garissa Banan 37 1 Kumahumato Garissa Tinas 48 1 Kumahumato Garissa Jerin 35 1 Kumahumato Garissa Jerin 32 1 Kumahumato Garissa Banan 38 1 Kumahumato Garissa Jerin 18 1 Kumahumato Garissa Hodan 42 1 Kumahumato Garissa Hadle 45 2 Alikune Garissa Abay Qala 45 2 Alikune Garissa Bulla sheikh 70 2 Alikune Garissa Bulla sheikh 63 2 Alikune Garissa Oyussuf 74 2 Alikune Garissa Abay Qala 58 2 Alikune Garissa Daresalam 78 2 Alikune Garissa Biliin 45 2 Alikune Garissa Bulla hagar 40 2 Alikune Garissa Waberi 50 2 Alikune Garissa Biliin 38 2 Alikune Garissa Daresalam 34 2 Alikune Garissa Waberi 60 2 Alikune Garissa Gadud 46 2 Alikune Garissa Daresalam 38 2 Alikune Garissa Gadud 62 3 Katumba Tana River Bisal Hargesa 25 3 Katumba Tana River Bisal Hargesa 40 3 Katumba Tana River Bisal Hargesa 52 3 Katumba Tana River Bisal Hargesa 40 3 Katumba Tana River Kuriti 42 3 Katumba Tana River Kuriti 63 3 Katumba Tana River Kuriti 52 3 Katumba Tana River Kuriti 40 3 Katumba Tana River Kuriti 42 3 Katumba Tana River Kuriti 35 3 Katumba Tana River Kuriti 38 4 Wolesorea Tana River Wolesorea 45 4 Wolesorea Tana River Wolesongumba 18 4 Wolesorea Tana River Wolesorea 50 4 Wolesorea Tana River Wolesongumba 40 4 Wolesorea Tana River Wolesongumba 46 4 Wolesorea Tana River Wolesorea 60 4 Wolesorea Tana River Wolesorea 47 4 Wolesorea Tana River Chifra 40 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 98 GD Borehole County Participant Village Age 4 Wolesorea Tana River Wolesorea 46 4 Wolesorea Tana River Chifra 36 4 Wolesorea Tana River Wolesorea 40 5 Tangulbei Baringo Sowo 30 5 Tangulbei Baringo Bombo 37 5 Tangulbei Baringo Chemakit 36 5 Tangulbei Baringo Panyarit A 35 5 Tangulbei Baringo Panyarit B 30 5 Tangulbei Baringo Koipakwen 32 6 Oro Baringo Cheptaaran 40 6 Oro Baringo Oro 30 6 Oro Baringo Oro 42 6 Oro Baringo Rorok 40 6 Oro Baringo Lokrakow 50 6 Oro Baringo Nginyang 60 6 Oro Baringo Chepkwarkwarian 50 6 Oro Baringo Ngalekan 40 6 Oro Baringo Kalemngorok 40 6 Oro Baringo Oro 22 6 Oro Baringo Oro 55 7 Lokichar (Chief) Turkana Nairobi West Refused to answer 7 Lokichar (Chief) Turkana Nairobi West Refused to answer 7 Lokichar (Chief) Turkana Achukule 23 7 Lokichar (Chief) Turkana Achukule 34 7 Lokichar (Chief) Turkana Achukule 24 7 Lokichar (Chief) Turkana Tonyoutu 46 7 Lokichar (Chief) Turkana Johanesburg 21 7 Lokichar (Chief) Turkana Narengelup 34 7 Lokichar (Chief) Turkana Narengelup Refused to answer 7 Lokichar (Chief) Turkana Achukule 30 7 Lokichar (Chief) Turkana Nalemusekon 27 8 AP-Line Lokori (IDP) Turkana Kalokome 44 8 AP-Line Lokori (IDP) Turkana Kalokome 20 8 AP-Line Lokori (IDP) Turkana Emanman 24 8 AP-Line Lokori (IDP) Turkana Kalokome 31 8 AP-Line Lokori (IDP) Turkana Anyangalim 24 8 AP-Line Lokori (IDP) Turkana Ngikoropua 40 8 AP-Line Lokori (IDP) Turkana Line Moja 37 8 AP-Line Lokori (IDP) Turkana Lokidingos 23 8 AP-Line Lokori (IDP) Turkana Ap-Line 26 8 AP-Line Lokori (IDP) Turkana Anyangalim 21 9 Yericho-B Turkana Yericho-B 30 9 Yericho-B Turkana Ng'deriko-Nyen 28 9 Yericho-B Turkana Ng'deriko-Nyen 23 9 Yericho-B Turkana Abakan 22 9 Yericho-B Turkana Abakan 35 9 Yericho-B Turkana Yericho-B 18 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 99 ANNEX I: REFERENCES Abadie, A., & Imbens, G. W. (2006). Large sample properties of matching estimators for average treatment effects. Journal of the Econometric Society, 74(1), 235-267. doi:10.1111/j.1468- 0262.2006.00655.x Altman, D. G. (1985). Comparability of randomised groups. Journal of the Royal Statistical Society (The Statistician), Series D, 34(1), 125-136. Cronk, R., & Bartram, J. (2017). Factors influencing water system functionality in Nigeria and Tanzania: A regression and Bayesian network analysis. Environmental Science & Technology, 51(19), 11336- 11345. doi:10.1021/acs.est.7b03287 Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053-1062. doi:10.1080/01621459.1999.10473858 Feinerer, I., Hornik, K., & Meyer, D. (2008). Text mining infrastructure in R. Journal of Statistical Software, 25(5), 1-54. doi:10.18637/jss.v025.i05 Fisher, M. B., Shields, K. F., Chan, T. U., Christenson, E., Cronk, R. D., Leker, H., … Bartram, J. (2015). Understanding handpump sustainability: Determinants of rural water source functionality in the Greater Afram Plains region of Ghana. Water Resources Research, 51(10), 8431-8449. doi:10.1002/2014WR016770 Funk, C. C., Peterson, P. J., Landsfeld, M. F., Pedreros, D. H., Verdin, J. P., Rowland, J. D., . . . Verdin, A. P. J. U. G. S. D. S. (2014). A quasi-global precipitation time series for drought monitoring. 832(4), 1-12. Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact evaluation in practice, second edition. Washington, DC: Inter-American Development Bank and World Bank. https://openknowledge.worldbank.org/handle/10986/25030 License: CC BY 3.0 IGO Government of Kenya. (2014). Ending drought emergencies: Common programme framework for drought risk management. Retrieved from https://reliefweb.int/report/kenya/ending-drought-emergencies￾common-programme-framework-drought-risk-management Harvey, P. A., & Reed, R. A. (2007). Community-managed water supplies in Africa: Sustainable or dispensable? Community Development Journal, 42(3), 365-378. Ho, D., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42(8), 1-28. doi:10.18637/jss.v042.i08 Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge: Cambridge University Press. King, G., & Nielsen, R. (2018). Why propensity scores should not be used for matching. Political Analysis. Nagel, C., Beach, J., Iribagiza, C., & Thomas, E. A. (2015). Evaluating cellular instrumentation on rural handpumps to improve service delivery—A longitudinal study in rural Rwanda. Environmental Science & Technology, 49(24), 14292-14300. doi:10.1021/acs.est.5b04077 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 100 NDMA. (n.d.). Early warning bulletins. Retrieved from http://www.ndma.go.ke/index.php/resource￾center/early-warning-reports Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55. doi:10.1093/biomet/70.1.41 Thomas, E. A., Needoba, J., Kaberia, D., Butterworth, J., Adams, E. C., Odur, P., … Nagel, C. (2019). Quantifying increased groundwater demand from prolonged drought in the East African Rift Valley. Science of the Total Environment, 666, 1265-1272. UNICEF & World Health Organization. (2015). Progress on sanitation and drinking water: 2015 update and MDG assessment. Walters, J. P., & Javernick-Will, A. N. (2015). Long-term functionality of rural water services in developing countries: A system dynamics approach to understanding the dynamic interaction of factors. Environmental Science & Technology, 49(8), 5035-5043. doi:10.1021/es505975h World Bank. (2011). Water supply and sanitation in Kenya: Turning finance into services for 2015 and beyond. World Bank, Nairobi. World Bank. https://openknowledge.worldbank.org/handle/10986/12890 License: CC BY 3.0 IGO. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 101 ANNEX J: ADDITIONAL BALANCE STATISTICS Variable Treatment Control Normalized N Mean SD N Mean SD Difference Number of HHs using scheme 76 1180.97 1960.65 132 819.11 1316.17 0.206 Estimated number of goats/sheep using water point (per day) 53 5983.06 8910.38 109 23831.21 191640.8 0.14 Trucking (Y/N) 76 0.18 0.39 132 0.29 0.45 0.25 Number of trucks per day 14 3.86 2.14 38 3.5 3.54 0.063 Trucking days per week 13 4.62 2.72 38 4.13 2.26 0.083 Service area 76 4.76 5.86 132 3.2 4.86 0.284 Used for irrigation 76 0.12 0.33 132 0.02 0.12 0.383 Number of farmers using scheme 9 38.56 63.68 2 5.5 6.36 0.219 Average farm size 7 4.86 6.79 2 0.38 0.18 0.251 Livestock use (Y/N) 76 0.72 0.45 132 0.83 0.38 0.24 Estimated number of camels using scheme (per day) 53 1429.53 2237.39 109 161.52 348.96 0.591 Estimated number of cattle using scheme (per day) 53 1833.49 3508.68 109 1247.94 2233.07 0.16 Estimated number of horses/donkeys using water point (per day) 52 301.35 359.66 109 222.74 406.54 0.179 Construction year of the well/borehole 69 2008.86 7.48 129 2008.81 12.53 0.005 Number of months since non-functional 4 3.75 2.22 48 7.06 6.9 0.32 Out of service one or more days last month (Y/N) 76 0.28 0.45 130 0.2 0.4 0.176 Number of days out of service in the last month 21 19.14 12.87 26 9.77 10.6 0.386 Our of service one or more days last year (Y/N) 76 0.37 0.49 130 0.45 0.5 0.174 Number of times broken in past year 28 6.89 15.58 58 2.76 3.21 0.2 Emerging problems 67 0.1 0.31 130 0.72 0.45 1.607 Days per week pump runs 76 6.53 1.26 132 5.91 1.88 0.407 Hours per day pump runs 76 10.05 4.95 132 8.14 5.17 0.381 Failure during the dry season 76 0.2 0.4 132 0.05 0.22 0.416 Use depends on season (Y/N) 76 0.61 0.49 132 0.77 0.43 0.341 Average days operational during wet season 71 4.37 3.12 132 3.36 2.57 0.334 Average hours per day operational during wet season 71 5.18 4.31 132 4.56 3.92 0.146 Tracking solar panels (Y/N) 9 0.44 0.73 39 0 0 0.264 Year when generator was installed (or year of re-installation) 22 2012.91 3.12 79 2009.85 9.5 0.35 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 102 Variable Treatment Control Normalized N Mean SD N Mean SD Difference Fixed tariff 76 0.95 0.22 132 0.7 0.46 0.756 Functional (Y/N) 76 0.92 0.27 132 0.64 0.48 0.784 Institutional tariff 76 0.3 0.46 132 0.31 0.46 0.017 General use tariff cost (KES) 71 221.01 853.7 95 61.44 229.99 0.221 Livestock tariff (Y/N) 76 0.74 0.44 132 0.36 0.48 0.833 Tariff-camel 46 12.76 9.46 26 13.2 20.4 0.015 Tariff-cattle 48 5.44 4.26 45 4.87 7.43 0.064 Tariff-horse/donkey 23 2.74 4.64 26 5.56 9.93 0.187 Tariff-goat/sheep 51 1.76 1.38 46 5.87 13.82 0.289 Monthly institutional tariff (KES) 22 1550 987.9 41 2646.59 3650.09 0.26 Hours spent collecting water (borehole working) 76 2.74 3.44 132 2.25 1.65 0.169 Hours spent collecting water (borehole not working) 70 2.65 1.85 120 4.3 2.88 0.692 Water quality testing at borehole (Y/N) 76 0.87 0.34 132 0.88 0.33 0.031 Year of most recent water quality test done 49 2013.61 6.87 108 2011.18 8.75 0.271 Miles 65 346.69 81.84 126 234.24 133.2 1.037 Rainfall (five-year avg.) 76 39.16 4.86 132 67.43 18.75 2.36 NDVI (five-year avg.) 76 0.29 0.02 132 0.46 0.07 3.828 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 103 ANNEX K: ADDITIONAL MATCHING DETAILS The table below provides an overview of the descriptive statistics for the key matching variables. Some of this information is reported and contextualized in the previous sections, but is reported here to provide an overview of the matching process. It is worth noting that the number of observations is lower in the table below than what was reported in earlier sections. This is due to the requirement that matching occur on units with full data across all matching variables. If a borehole is, for example, missing information on the construction year, this cannot be used in the matching model since this couldn’t inform estimation of the propensity score. One potential way to address this limitation of matching is to impute missing values; that is, replace missing observations with estimates based on values for similar boreholes. Imputation can be a fraught endeavor because of the assumptions required to ensure reasonable estimates. Given the highly context-specific nature of the borehole survey data, the evaluation team has not replaced missing values with imputed values. DESCRIPTIVE STATISTICS FOR MODEL VARIABLES BY ASSIGNMENT Treatment Comparison Variable n Mean SD Min Max n Mean SD Min Max Number of households using the borehole scheme 57 1,146.04 1,861.91 70 10,000 123 854.73 1,356.41 0 10,000 Service area for the scheme (in km 2) 57 4.91 6.43 0 35 123 3.15 4.88 0 35 Livestock use (Y/N) 57 0.74 0.44 0 1 123 0.83 0.38 0 1 Borehole construction year 57 2008 7.79 1971 2017 123 2009 12.77 1913 2018 Borehole water quality test (Y/N) 57 0.86 0.35 0 1 123 0.82 0.38 0 1 Out of service one or more days (Y/N) 57 0.21 0.41 0 1 123 0.2 0.4 0 1 Average daily runtime (in hours) 57 9.61 4.67 2 20 123 8.19 5.26 1 24 Fixed tariff scheme (Y/N) 57 0.93 0.26 0 1 123 0.67 0.47 0 1 Fixed tariff scheme for livestock (Y/N) 57 0.72 0.45 0 1 123 0.35 0.48 0 1 Miles to Nairobi 57 350.16 80.26 143.21 497.46 123 236.26 134.17 92.4 669.37 Three-year rainfall average (annual, in mm) 57 39.73 4.86 34.38 47.07 123 69.31 18.85 30.7 94.61 Type of power Generator 21 0.37 0.49 0 1 73 0.59 0.49 0 1 Hybrid 28 0.49 0.50 0 1 0 0 0 0 1 Solar 7 0.12 0.33 0 1 29.00 0.24 0.43 0 1 Utility Power 1 0.02 0.13 0 1 21.00 0.17 0.38 0 1 ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 104 A few key variables are important for functionality, the main outcome of interest, and the intervention broadly, for which the mean difference across assignment groups is quite large. The most notable of these is rainfall. The three-year, annual average for rainfall (2012–2014) in the comparison group is 69 millimeters, compared to about 40 millimeters in the treatment group. This is not surprising given Kenya RAPID’s approach of selecting arid counties in need of drought mitigation support. This does, however, mean that there are key underlying contextual factors that will be difficult to mitigate completely through matching. The general model specification for the matching model is a basic logistic regression, as follows: log ( 𝜋𝑡 1 − 𝜋𝑡 ) = 𝛽0 + 𝛽1𝑋𝑖 + 𝜀 Where 𝜋 is the probability of treatment, t, dependent on a set of covariates, X, for borehole, i. The covariates of interest are noted in the matching section of this report. A more parsimonious model selected through stepwise regression, wherein all combinations of model inputs are run, and tested through likelihood tests includes the following five variables to estimate propensity scores: • Miles from Nairobi • Borehole water quality test in the past (Y/N) • Livestock use (Y/N) • Three-year rainfall average (annual, in mm) • Fixed tariff scheme for livestock (Y/N) This results in an improved overlap for 57 comparison boreholes and 57 treatment boreholes, with an average propensity score for the comparison group of 0.21. The bimodal distribution in report Figure 21 remains in the updated figure below, but there is more overlap in the middle of the figure, particularly for scores between 0.3 and 0.7, suggesting an improved match. This does, however, highlight one of the pitfalls of model-based assignment selection. The evaluation team conducted a model stepwise process that included all of the variables in the borehole asset survey, but there could be theoretical or implementation reasons to include the full set of variables from Figure 21 despite the improved matches shown below. For example, when the matching algorithm is run without the inclusion of the rainfall variable, the area of overlap is much larger, but it is perhaps not reasonable based on the Kenya RAPID context. As shown below, the bimodal distribution of propensity scores has disappeared, with an average comparison propensity score of 0.36 and 57 boreholes in each assignment group. The area of overlap is much clearer in the figure below from about 0.20 to 0.75. This may improve balance between assignment groups, but as previously noted a key feature of the Kenya RAPID intervention is to install sensors in drought-prone areas, so the estimates of impact from the matches resulting from this model are likely to omit a key variable that should inform not only analysis but also policy. ROUND 1 REPORT: IMPACT EVALUATION OF THE KENYA RAPID ACTIVITY 105 FIGURE K-1: PARSIMONIOUS MODEL PROPENSITY SCORES (N = 57/57) FIGURE K-2: PARSIMONIOUS MODEL MATCHING WITHOUT RAINFALL INCLUDED (N = 57/57) U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov