WHAT DOES IT TAKE TO SUSTAIN WATER, SANITATION, AND HYGIENE OUTCOMES? Lessons from Six Ex-Post Evaluations i | EX-POST EVALUATION SYNTHESIS USAID.GOV AUTHORITY Prepared for the United States Agency for International Development (USAID) under the Water and Development Indefinite Delivery Indefinite Quantity Contract No. AID-OAA-I-14-00069, Task Order No. AID-OAA-TO-15-00046, awarded September 17, 2015, entitled “Water Communications and Knowledge Management (CKM) Project.” This synthesis report is made possible by the support of the American people through USAID. The contents of this report are the sole responsibility of ECODIT LLC and do not necessarily reflect the views of USAID or the United States Government. The Water CKM project team, comprised of ECODIT LLC and Social Impact Inc., prepared this report under the leadership of Holly Dentz, Senior Technical Specialist; Leslie Hodel, Senior Technical Specialist; Kari Nelson, Technical Director; and Danae Roumis, Technical Director. Cover photo (clockwise from top left): A woman collects water in Zambézia Province, Mozambique. (Photo credit: Forcier Consulting); A local mason examines a SanPlat latrine slab in Amboditafara, Madagascar (photo credit: Water CKM Project); A public community-managed toilet in Sangli, Maharashtra, India (Photo credit: Sujit Mridha/Water CKM Project); Tippy tap handwashing facility (Photo credit: WASHplus). USAID.GOV EX-POST EVALUATION SYNTHESIS | ii FOREWORD The water, sanitation, and hygiene (WASH) Ex-Post Evaluation Series represents a key milestone in USAID’s ongoing quest to unpack the drivers of sustainability within our WASH programs. The commitment, made in the first USAID Water & Development Strategy (2013– 2018),1 came in the wake of headlines about high rates of nonfunctional water systems in partner countries where USAID has long invested. At that time, many governments and development partners were grasping for similar answers, and a number of sustainability assessment tools emerged.2 For its part, the Agency pursued a multi-pronged approach that focused on: 1) measuring progress towards sustainable outcomes (such as through the development and application of the Sustainability Index Tool with Rotary International); 2) gaining a better understanding of the drivers of long-term outcomes through this WASH Ex￾Post Evaluation Series; and 3) testing new ideas, approaches and tools to strengthen the local systems that can deliver WASH service sustainability through the Sustainable WASH Systems (SWS) Learning Partnership.3 The Agency, together with its development partners, has achieved staggering results in terms of delivering first time water and sanitation access to people the world over. Since 2013, USAID has helped roughly 25 million people gain access to at least basic drinking water services and 18 million people gain access to at least basic sanitation.4 Together with partners, USAID has mobilized almost $100 million in new funding for the sector5 and supported nearly 17,000 communities become open defecation free.6 Yet the results of this ex-post series are sobering. Despite tremendous achievements within the life of our programs, they have largely not endured. This is especially the case in countries and communities with the highest levels of poverty at baseline, where the Agency’s resources are needed the most. Rural water systems that, at activity close, delivered safe water to households have fallen into disrepair. Basic latrine ownership and use have dwindled. Communities certified as open-defecation free are backsliding, and gains in handwashing have not been sustained. The series did reveal some programming bright spots. Where USAID invested in providing technical assistance to committed government partners and utilities, gains in service provision and local capacity were sustained, with local actors taking up and expanding upon best practices introduced during activity implementation. Often these successes endured in countries and communities that had higher levels of capacity at the outset. However, the successes in these contexts demonstrate important lessons about investing time and resources into partnering with local institutions and focusing on plans for management of services, not just first-time access. In the course of the roughly 15 intervening years since most of the activities evaluated in this series were designed, the sector has evolved. For instance, in resource-constrained environments, the sector is now coalescing around facilitating the development of professionalized support to community-managed rural water schemes in various forms, rather than expecting voluntary committees to manage essential services alone. Additional approaches beyond community-led total sanitation, including smart subsidies and market-based sanitation, are seen as necessary to move households up the sanitation ladder, and are being applied through USAID programming. And the Agency is shifting its WASH social and behavior change programming to more holistic approaches that address emotional drivers, convenience, and iii | EX-POST EVALUATION SYNTHESIS USAID.GOV social norms to modify intractable behaviors rather than communication or health promotion alone. The WASH Ex-Post Evaluation Series validates why those shifts were essential. All those with a stake in promoting lasting development gains in the sector need to internalize these findings and take a long look in the mirror. USAID has and is seeking to do better. First, under the USAID Water and Development Plan (2018-2022) in support of the Global Water Strategy, USAID codified its commitment to sustainability with the goal of increasing the availability and sustainable management of safe water and sanitation, and an emphasis on improving the underlying governance, finance, and management of water resources that underpin sustainability. Second, the Agency has issued a set of technical briefs that provide new guidance on important topics for developing and implementing WASH activities, as well as recommendations for activity design, implementation and monitoring.7 Third, USAID has launched the Water Security, Sanitation and Hygiene Implementation Research Agenda that identifies and prioritizes sector-specific research questions to close lingering evidence gaps directly related to accomplishing USAID's goal of increasing access to sustainable water and sanitation services. Finally, the Agency is rethinking its approach to sector targets and key performance indicators in its solicitations, recognizing that targets can cause perverse incentives to undermine sustainability from the outset. Doing so underscores the Agency’s commitment to sustainability and its willingness to be held accountable to deliver against those results. Abigail Jones Water & Sanitation Specialist, Center for Water Security, Sanitation, and Hygiene USAID Bureau for Resilience and Food Security Elizabeth Jordan Water & Sanitation Specialist, Center for Water Security, Sanitation, and Hygiene USAID Bureau for Resilience and Food Security USAID.GOV EX-POST EVALUATION SYNTHESIS | iv ACKNOWLEDGMENTS The report authors would especially like to thank our USAID/Washington counterparts Abigail Jones and Elizabeth Jordan, who provided considerable guidance and input throughout this series. We also thank Patricia Mantey and Alison Macalady at USAID for their helpful feedback over the past years. We would like to acknowledge the hard work of former Water CKM colleague Annette Fay to complete evaluations in Madagascar, Indonesia, and Ethiopia; Chief of Party Mary Renwick for her excellent report review contributions; Wendy Putnam for her editorial support; and Grace Tang for literature review support across the series. We would also like to thank all the contributors to the six ex-post evaluations including USAID staff, implementing organizations that provided project documentation, USAID Missions, consultants, and many stakeholders who participated. Specifically: ● Madagascar—Former USAID/Madagascar staff Jacky Ralaiarivony; implementers Catholic Relief Services, CARE, Caritas, and Voahary Salama; and local data collection team Agence Capsule (quantitative data) and Jean Eugene Injerona, Mamy Tiana Rakotoarimanana, Jocelyn Rakotonirina, and Neila Andrianaina (qualitative interviewers) ● Indonesia—USAID/Indonesia staff Trigeany Linggoatmodjo and Endah Shofiani; Former USAID fellow Miriam Kochman, who supported planning and fieldwork; implementer Development Alternatives Inc.; and local data collection team Elena Vijayawati, Enrico Rahadi, Kharisma Prasetyo, Diba Basar, and Amanda Stek ● Ethiopia—USAID/Ethiopia staff Kathrin Tegenfeldt and Mequanent Fentie; implementers Millennium Water Alliance (particularly Mussie Tezazu), CARE, Catholic Relief Services, Food for the Hungry, Lifewater International, Living Water International, Water.org, Hope 2020, World Vision, Relief Society of Tigray, Ethiopian Kale Hiwot Church, Water Action, and Ethiopian Evangelical Church Mekane Yesus-Development and Social Services Commission; and local data collection team Seifu Tilahun and Dessalew Aynalem ● India—USAID/India staff Anand Rudra, RK Srinivasan, and former USAID/India staff Mark Peters; Steve Sena and Sam Huston from USAID Water, Sanitation, and Hygiene Finance (WASH-FIN); and local data collection team Sujit Kumar Mridha, Abhirup Bhunia, Debanjana Das, and Gabrielle Plotkin ● Senegal—USAID/Senegal staff Abdoulaye Boly and Avril Gonzalez; implementer Research Triangle Institute and select implementing partners Caritas, Casades, ENDA, Kabonketor, La Lumiere, and Weswa; local data collection team Atraxis Group and Alioune Watt; and Colleen Brady, who supported planning and desk review ● Mozambique—USAID/Mozambique staff Gastao Mendes, Leonor Domingos, Sheila Zacarias, Clara Dimene and former USAID/Mozambique staff Ken Hassan; USAID/Washington staff Ella Lazarte and Nga Nguyen; Water CKM’s Andrew Chatting, who supported data collection planning and training; and local data collection team Forcier Consulting and Bacelar Muneme v | EX-POST EVALUATION SYNTHESIS USAID.GOV TABLE OF CONTENTS Authority......................................................................................................................................... i Foreword........................................................................................................................................ ii Acknowledgments........................................................................................................................ iv Table of Contents.......................................................................................................................... v Table of Figures............................................................................................................................ vi Table of Tables ............................................................................................................................. vi Acronyms......................................................................................................................................vii EXECUTIVE SUMMARY............................................................................................................viii Ex-post Evaluation Series......................................................................................................................................... viii Financing for Sustainable Services............................................................................................................................ ix Governance of Water and Sanitation Services...................................................................................................... x Long-term Management of Water Services........................................................................................................... xi Hygiene and Sanitation Behavior Change.............................................................................................................. xii Conclusion.................................................................................................................................................................... xii SUSTAINABLE WASH AT USAID............................................................................................ 1 EX-POST EVALUATION SERIES............................................................................................... 2 Key Findings at Ex-post................................................................................................................................................ 4 FINANCING FOR SUSTAINABLE SERVICES ......................................................................... 6 Financial Management................................................................................................................................................... 7 Business Planning........................................................................................................................................................... 7 Cost Recovery ............................................................................................................................................................... 8 Credit Enhancements................................................................................................................................................... 9 Household Finance........................................................................................................................................................ 9 ESTABLISHING EFFECTIVE DRINKING WATER & SANITATION GOVERNANCE..... 11 Government Commitment....................................................................................................................................... 11 Clarity and Capacity In Roles and Responsibilities.............................................................................................. 12 Tariff Policies................................................................................................................................................................ 12 Land Tenure Policies.................................................................................................................................................. 14 EFFECTIVE MANAGEMENT FOR SUSTAINABLE WATER SERVICES............................ 15 Water Service Delivery Management Practices and Models............................................................................ 15 Capacity Building for CBM Entities .................................................................................................................... 16 Capacity Building for Utilities .............................................................................................................................. 16 Establishment of Public-Private Partnerships................................................................................................... 17 Cost Recovery ............................................................................................................................................................. 18 Operations, Maintenance, and Repairs.................................................................................................................. 19 Asset Management ................................................................................................................................................. 19 Water Service and Quality Monitoring ............................................................................................................. 20 Supply Chain Access.............................................................................................................................................. 21 SUSTAINABLE SANITATION AND HANDWASHING DEMAND AND BEHAVIOR CHANGE ..................................................................................................................................... 22 Handwashing Behavior Change................................................................................................................................ 23 Psychosocial Drivers of Behavior....................................................................................................................... 24 Hardware.................................................................................................................................................................. 24 Financial Constraints.............................................................................................................................................. 25 Ongoing Local Behavior Change Communication.......................................................................................... 25 Contextual Factors................................................................................................................................................. 26 Sanitation Behavior Change...................................................................................................................................... 27 Latrine Quality......................................................................................................................................................... 28 USAID.GOV EX-POST EVALUATION SYNTHESIS | vi Household Finance................................................................................................................................................. 28 Professional Construction Support.................................................................................................................... 29 Psychosocial Drivers of Behavior....................................................................................................................... 30 Ongoing Local Follow-up ...................................................................................................................................... 30 Gendered Decision-Making.................................................................................................................................. 30 CONCLUSION............................................................................................................................ 32 ENDNOTES................................................................................................................................. 34 TABLE OF FIGURES Figure 1. Ex-Post WASH Outcomes in Predominantly Rural Evaluation Settings............................................ ix Figure 2. Ex-Post Evaluation Sites, Intervention Components Evaluated, and Methodology.......................... 3 Figure 3. Sustainability of WASH in Predominantly Rural Ex-Post Evaluation Contexts................................. 4 Figure 4. Three-Year Sustainability of Key Madagascar (RANO-HP) Sanitation and Hygiene Outcomes... 5 Figure 5. Water Utility in Indonesia ............................................................................................................................ 16 Figure 6. A Monoblock and Attendant in Madagascar............................................................................................ 17 Figure 7. Water Quality Testing in Senegal............................................................................................................... 20 Figure 8. A Latrine with a Washable Slab and Tippy Tap in Anosy, Madagascar............................................. 23 Figure 9. Mozambiquan Woman Using a Tippy Tap................................................................................................ 25 Figure 10. Mozambique Latrine Quality and Style ................................................................................................... 27 Figure 11. Key Sanitation Outcomes in Senegal, by Intervention Approach..................................................... 29 Figure 12. Madagascar Ex-post WASH Outcomes (observed): Percent Change from Endline by Gender of Household Head ......................................................................................................................................................... 31 TABLE OF TABLES Table 1. Activities Evaluated........................................................................................................................................... ix Table 2. Finance Interventions Evaluated at Ex-post................................................................................................. 6 Table 3. Water Management Interventions Evaluated at Ex-post........................................................................ 15 Table 4. Hygiene and Sanitation Behavior Change Interventions Evaluated at Ex-Post ................................. 23 vii | EX-POST EVALUATION SYNTHESIS USAID.GOV ACRONYMS ASUFOR Association d’Usagers de Forage BCC Behavior Change Communications CBM Community-Based Management CKM Communications and Knowledge Management CLTS Community-Led Total Sanitation DEAAS Double-Entry Accrual-Based Accounting Systems ESP Environmental Services Program ET Evaluation Team FIRE-D Financial Institutions Reform and Expansion—Debt and Infrastructure GoI Government of India HWWS Handwashing With Soap JMP WHO/UNICEF Joint Monitoring Programme for Water Supply and Sanitation MWA-EP Millennium Water Alliance—Ethiopia Program NGO Non-Governmental Organization NRW Non-Revenue Water O&M Operations and Maintenance OD Open Defecation ODF Open Defecation Free PEPAM Programme d’Eau Potable et d’Assainissement du Millénaire PHAST Participatory Hygiene and Sanitation Transformation PPP Public-Private Partnerships RANO-HP Rural Access to New Opportunities for Health and Prosperity SARAR Self-Esteem, Associative Strengths, Resourcefulness, Action Planning and Responsibility SBC Social and Behavior Change SCIP Strengthening Communities through Integrated Programming SDG Sustainable Development Goals UNICEF United Nations Children’s Fund VSLA Village Savings and Loan Associations WASH Water, Sanitation, and Hygiene WHO World Health Organization WP Water Point USAID.GOV EX-POST EVALUATION SYNTHESIS | viii EXECUTIVE SUMMARY While global progress toward achieving Sustainable Development Goals (SDGs) to meet water, sanitation, and hygiene (WASH) needs is notable, the number of people lacking these essential services remains vast, and progress in the world's poorest countries has been sluggish despite decades of development interventions. Since 2000, access to at least basic water services has only increased from 51 percent to 56 percent in the least developed countries, and basic sanitation access has increased from 22 percent to 34 percent. As of 2017, 3 billion people lacked basic handwashing facilities.8 The health and economic implications are critical, as poor WASH conditions are linked to 2 million deaths annually and significant gross domestic product loss due to health and time burdens alone.9 Lack of sustainability is a crucial barrier to progress, with evidence that 25 percent of water points fail within four years and frequent reports of slippage into previous open defecation habits.10 Sustainability has come more into focus in the last decade as WASH sector programming has evolved from a direct service delivery model to a systems approach oriented toward facilitating sustainable service provision through local actors. Through its commitment to identifying sustainable approaches to WASH, USAID commissioned a series of six ex-post evaluations of its WASH activities completed three to 10 years prior. These studies identified what outcomes had been sustained years later and why. This report shares key findings from the series. Findings are summarized according to finance, governance, management, and behavior change. EX-POST EVALUATION SERIES The series examined four rural and two urban WASH activities three to 10 years after the activities ended, as shown in Table 1. All evaluations involved mixed qualitative and quantitative methods. The lack of endline data from the original activities at most ex-post evaluation sites prevented direct measurement of sustainability. However, in Madagascar directly comparable endline household and community data from the original intervention permitted robust conclusions about sustainability. Similarly, ex-post utility performance data in Indonesia proved to be comparable to a metric used during the project. The evaluation team (ET) noted low sustainability of USAID–established rural water points, with functionality ranging from 44 percent to 65 percent at ex-post (Figure 1). In Indonesia, nearly all the eight USAID–supported utilities evaluated had increased water connection coverage in the five years since the activity ended (average change from 35 percent to 43 percent). In India, water access varied widely across the six USAID-supported municipalities evaluated, from a consistent 62 percent for the past five years in one city to a steep three-year increase from 31 percent to 100 percent in another. The ET did not find materials for handwashing with soap in most rural contexts, ranging from none in a very small Ethiopia sample to 31 percent of households observed in Senegal. Access to any type of sanitation at ex-post ranged from 45 percent to 92 percent of households in Senegal, with basic sanitation access much lower in all settings. Madagascar saw statistically significant slippage in latrine use in targeted geographies from 69 percent to 45 percent. Lack of directly comparable endline data prevented confirmation of changes in other settings, but the evaluations confirmed open defecation continued to occur populates targeted by activities in Mozambique, Ethiopia, and Senegal. About This Report This report summarizes key findings from a series of six independent ex-post evaluations, available on Globalwaters.org, that assessed the extent to which USAID–funded WASH activities sustained outcomes three to 10 years after closure and which factors drove those outcomes. These findings and related WASH literature are meant to foster learning and improve evidence-based sustainable development assistance at USAID and across stakeholders in the WASH sector. Topics in this report are generally constrained to former WASH activities studied through the evaluation series; this report is not a comprehensive sector-wide assessment. ix | EX-POST EVALUATION SYNTHESIS USAID.GOV Table 1. Activities Evaluated Country Activity Name Timeframe Activity Focus Ex-Post Follow-Up Madagascar Rural Access to New Opportunities for Health and Prosperity (RANO-HP) 2009–2013 Improve WASH access in primarily rural areas 3 years Indonesia Environmental Services Program (ESP) 2004–2010 Improve and expand safe water access in urban areas with a focus on support to utilities 7 years Ethiopia Millennium Water Alliance Ethiopia Program (MWA-EP) 2004–2009 Improve WASH access in rural areas 8 years India Financial Institutions Reform and Expansion–Debt and Infrastructure (FIRE-D) 1994–2011 Capacitate local stakeholders to plan, manage, and finance urban water and sanitation development 7 years Senegal Programme d’Eau Potable et d’Assainissement du Millénaire (PEPAM) 2009–2014 Improve WASH access in rural and peri￾urban areas 4 years Mozambique Strengthening Communities through Integrated Programming (SCIP) 2009–2015 Improve WASH access in rural and urban areas 4 years Figure 1. Ex-Post WASH Outcomes in Predominantly Rural Evaluation Settings *Ethiopia water point functionality data derive from secondary data in one zone (n=54). Mozambique n=46; Senegal n=169. Madagascar did not include assessment of the water intervention. † Ethiopia n=15; Madagascar n=688; Mozambique n=990; Senegal n=601. ‡ Madagascar n=688; Mozambique n=990; Senegal n=602. Due to the small, purposive sampling approach in Ethiopia, sanitation data are not included here. § Madagascar n=688; Mozambique n=990; Senegal n=540. Due to the small, purposive sampling approach in Ethiopia, sanitation data are not included here. FINANCING FOR SUSTAINABLE SERVICES Adequate and sustainable WASH financing is essential to improve service delivery and meet universal access targets associated with national WASH strategies or the SDGs.11 Sustainability requires efficient financial management practices, strategic planning, and adequate cost recovery or subsidy from public resources to achieve financial stability. Under the right conditions, success in these domains enables service providers to attract private financing for capital investments, including from repayable sources, thereby reducing reliance on public funds or foreign aid. USAID.GOV EX-POST EVALUATION SYNTHESIS | x USAID sought to improve financial viability and performance as a central component of the India FIRE-D (focused on states and municipalities) and Indonesia ESP (focused on utilities) activities. The ET found most approaches that codified improved practices into policies, manuals, or technologies persisted beyond the life of the USAID–funded activity. For example, the ET found evidence of sustainability at nearly all the FIRE-D–supported sites of municipal accounting reforms; cost recovery efforts such as water or energy audits and leak detection repair; tax reform; and “e-governance” initiatives that allowed customers to pay bills, report leaks, and register complaints. These initiatives had also been scaled across many other states and municipalities nationwide since the national government adopted them into its WASH funding schemes—a further demonstration of their value. Similarly, utilities for whom the Indonesia ESP activity provided non-revenue water (NRW) capture protocols and improved meter reading technology still used these resources and credited them with documented reductions to their NRW rate. Despite the sustainability of these efforts, the poor financial performance of evaluated utilities in Indonesia and ongoing financial struggles of some evaluated municipalities in India show that such efforts alone can be insufficient to overcome significant financial challenges. To the extent India and Indonesia sought credit enhancements to facilitate access to market-based repayable financing sources, the ET concluded that the overall enabling environment and financing landscape limited their success. Of three evaluated municipalities where FIRE-D had supported bonds or credit ratings, two maintained a strong enough financial position to continue funding water and sanitation services through bonds. However, FIRE-D’s vision of commercially viable infrastructure did not materialize broadly in India, as abundant public funds for WASH became available, reducing the need for commercial finance. The ex-post series examined the sustainability of some mechanisms to reduce financial barriers for the poor. In Madagascar, three years after USAID support ended, a sanitation microfinance loan product continued to be offered, whereas a water connection microfinance loan in Indonesia failed because the large partner bank applied excessively stringent standards to assess customers’ creditworthiness. To a limited extent—17 percent—members of village savings and loan groups in Madagascar continued to use these mechanisms to fund WASH improvements after RANO-HP ended. GOVERNANCE OF WATER AND SANITATION SERVICES Good governance is an essential foundation that sets the stage for sustainable service delivery in tandem with finance. Strong government commitment to WASH is critical for improvement. Though not designed to measure causal relationships or the role of government policy, the ex-post series revealed that in very general terms, the relative success and sustainability of each activity trended along with the level of government commitment to WASH at the time of USAID activity implementation. The India FIRE-D example illustrated the value of partnering with a committed government to drive sustainability. Most activities examined through the ex-post series did not have explicit governance-focused interventions, but governance-related challenges emerged in all settings. In Ethiopia, the lack of policies and guidelines to clarify roles and responsibilities for water point monitoring, maintenance, and repair created challenges for managing rural water points.12 Ethiopian stakeholders disagreed on who should take responsibility for various rural water point repairs and expressed concerns about each entity’s financial and technical ability to perform their roles. In India, despite the nation’s decentralized governance framework, whereby local entities bore responsibility for planning and managing water and sanitation services, most municipalities lacked the personnel and resources, financial control, or decision-making autonomy to carry out their roles. These challenges illustrate the importance of not only having clear sector policies on the books, but also adequate human resource capacity to implement, and better local planning for sustainability in general. Water service governance also includes tariff setting. While tariffs in utility-served urban areas tend to be clearer, they are less defined for rural water systems, which often follow the community-based xi | EX-POST EVALUATION SYNTHESIS USAID.GOV management (CBM) model. In each of the rural water ex-post evaluations, communities defined their own water fees, with varying levels of guidance and support from implementers or the government. Across the developing world, tariffs for rural water services tend to be set below the point of cost recovery out of concern for the poor; however, evidence suggests even the poor may be willing (and do currently pay) higher rates than are currently set for high-quality services.13 Some countries and their utilities have explored alternative tariff structures that allow for cross-subsidies whereby commercial users and/or wealthier users pay higher rates than poorer users.14, 15 Despite some promising practices, clear and adequate tariff setting remains a challenge in many developing countries. LONG-TERM MANAGEMENT OF WATER SERVICES The ex-post evaluations implicated management challenges as key constraints to the sustainability of water infrastructure at ex-post. However, management of rural and urban water services have different models and trends. The ET examined the influence of various management factors on sustainability, including core management models, treatment of costs, and the handling of operations and maintenance (O&M) and repairs. CBM entities, such as local water committees, dominate management of rural water services. The rural water activities in Ethiopia, Mozambique, and Senegal made efforts to improve the capacity of these entities, such as providing training in basic WP maintenance, setting and collecting water fees, and the frequency of meetings and the sharing of minutes though the ET found mixed results regarding the sustainability of these efforts. The literature also finds that the management and maintenance practices typically taught to CBM entities vary in their effectiveness.16 The training CBM entities received from the USAID implementers regarding water point maintenance was insufficient in many cases to perform regular maintenance, manage funds and address larger repairs, and turnover among trained committee members exacerbated the skills gap. In rural settings, public-private partnerships (PPPs) have had mixed success in improving water service delivery In Senegal, some PPPs failed due to insufficient tariff collection, while in Madagascar (where RANO-HP’s water investments were studied by Villanova concurrently with the ex-post evaluation) some failed due to poor relationships between the private sector entities and local government and communities. This aligns with a 2017 World Bank study, which noted mixed success with private sector management in rural settings, but found promise in successful models that fostered long-term engagement among governments, development organizations, and the private sector, and typically included a clear delineation of the institutional framework to support private sector engagement.17 For urban utilities, capacity building has been a common approach to address service challenges.18 In Indonesia, local water utilities managed to sustain the capacity improvements achieved under ESP and increase access to water connections, in part due to the enduring use of ESP-generated standard operating procedure manuals. In some cases, utility management performance continued to improve after the end of the activity. PPPs are also a common approach to improving urban water service delivery. FIRE-D in India included a small PPP component. The ex-post stakeholder interviews in India emphasized the need to obtain financial stability and address negative public perceptions of private sector involvement for PPPs to be viable options. A 2009 World Bank study of urban utilities found that PPPs can significantly improve operational efficiency when proven approaches are taken.19 Across the ex-post series, the difficulty water service providers had raising sufficient funds to cover their costs contributed to failures, especially in rural settings. Payment collection emerged as a significant challenge as the establishment of appropriate rates. Many users of rural water systems evaluated at ex￾post did not pay any fees, and utilities in some evaluated cities in India and Indonesia had challenges with collection efficiency. Cost recovery remains a significant challenge to sustainability. USAID.GOV EX-POST EVALUATION SYNTHESIS | xii Asset management, water services monitoring, and water quality testing are important components of effective management. The three ex-post evaluations that examined rural water points found weak government asset management and a lack of routine water quality testing—both linked in part to unclear roles and responsibilities. Inadequate maintenance skills played a role in sustainability challenges, but lack of sufficient finances arose as the key barrier. Management of urban utilities fared better. FIRE￾D–supported e-governance tools in some cases facilitated service management, payments, network monitoring, repair requests, and feedback mechanisms through a single portal. Stakeholders noted their contribution to improved efficiency and effectiveness of service management. HYGIENE AND SANITATION BEHAVIOR CHANGE The low prevalence of observed basic handwashing facilities at ex-post, and the very limited improvement within the life of the two activities for which endline data exist, emphasize that achieving sustained habitual handwashing behavior with soap is one of the biggest challenges in the WASH sector. Virtually none of the USAID–promoted tippy taps remained in any of the four evaluation sites. The Participatory Hygiene and Sanitation Transformation (PHAST) and behavior change interventions applied in Ethiopia, Mozambique, and Senegal have known limitations; they tend to focus on knowledge and do not address the multiplicity of complex drivers across technology, psychosocial, and contextual dimensions.20 USAID and other actors have generally replaced or supplemented such approaches with more holistic social and behavior change activities. Regarding sanitation, while Senegal clearly stood out among ex-post evaluations with 92 percent latrine access, the ex-post activities did not meet the long-term community-led total sanitation (CLTS) goal of eliminating open defecation (OD), given low ex-post latrine access in Mozambique, significant slippage in Madagascar, and reports of community OD in 73 percent and 32 percent of surveyed households in Mozambique and Senegal, respectively. Basic sanitation access fared worse; most latrines appeared to either be unimproved or shared with other households. The ET identified latrine quality as a major problem, and stakeholders in three countries attributed low sustainability to poor quality of their original latrines, which had to be rebuilt constantly. These findings are very common regarding CLTS interventions.21 The primary reported barriers to latrine construction, maintenance, and reconstruction in all rural ex￾post evaluations tended to be either financial or material in nature. USAID and others in the sector are increasingly calling for targeted financial support options, such as vouchers, for the poor to facilitate higher quality latrine construction.22 The Senegal ex-post evaluation provides some support for a CLTS– plus-subsidy approach. The professional training of local masons to construct quality latrines showed promise as well as sustainability in the three countries where implemented. CONCLUSION Findings across the ex-post series led to several key takeaways in the areas of finance, governance, management, and behavior change: xiii | EX-POST EVALUATION SYNTHESIS USAID.GOV Key Takeaways Finance 1. Technical assistance to service providers for business planning, improved financial management and cost recovery, and credit enhancements were largely sustainable interventions, particularly when scaled through partnership. However, without an appropriate enabling environment, these strategies proved insufficient on their own to facilitate access to market-based finance, toward the goal of self-sufficiency. 2. Interventions leveraging technology, such as e-governance tools and improved metering and meter-reading, facilitated long-term improvements in financial management and cost recovery, especially through the reduction of NRW. 3. Training CBM entities to establish bank accounts and transparent accounting practices were insufficient to achieve sustainable finance for rural water system operations and maintenance. 4. While some limited microfinance continued to be available for WASH, it did not drive increased access to WASH services. This failure shows the need for appropriate alignment of lender incentives and processes with project goals and beneficiary needs. Governance 1. Unclear roles and responsibilities across actors prevented effective service delivery and discouraged sustainability. Inadequate training and resources to carry out those roles also inhibited sustainability. 2. A committed government is an important driver of activity sustainability, particularly in terms of WASH governance reforms. 3. Unclear tariff policies covering rural areas led to widely varying tariffs across communities, nearly all of which did not attain adequate cost recovery. Management 1. CBM entities did not successfully ensure long-term sustainability of water services in rural areas, regardless of capacity building and training interventions. 2. Cost recovery for water service delivery was a key challenge to sustainability. In both rural and urban areas, setting cost-reflective tariffs and collecting these fees was a challenge, with some rural systems unable to collect fees at all. Operational inefficiencies and high losses also contributed to this challenge in urban areas. 3. The use of PPPs for water service delivery showed mixed success, due in part to the lack of cost recovery, government inexperience with management and oversight of PPPs, and inadequate relationship building among stakeholders. 4. Efforts to strengthen supply chains for water system spare parts to enable maintenance by local technicians largely failed. Hygiene and Behavior Change 1. Handwashing social and behavior change programs focused primarily on knowledge, such as PHAST, did not work, nor did they did address the multidimensional drivers of and structural barriers to behavior change. 2. CLTS did not eliminate open defecation in the long term, and communities relied upon poor quality latrines. Triggered households did not generally progress toward improved or basic sanitation, meaning households missed out on substantial health benefits over the long term. 3. Financial barriers served as the major impediment to sustained latrine quality, maintenance, and use. Targeted subsidies combined with CLTS in Senegal offered modest improvements to sustained outcomes. 1 | EX-POST EVALUATION SYNTHESIS USAID.GOV SUSTAINABLE WASH AT USAID Sustainably addressing water, sanitation, and hygiene (WASH) service needs is fundamental to fighting disease and contributing to human health and dignity. Worldwide, 785 million people lack access to at least basic drinking water services, three billion people lack access to basic sanitation services, and three billion people do not have a basic handwashing facility with soap and water at their home.23 Poor WASH access and behaviors contribute to diarrheal disease, which is linked to two million deaths annually and 800 children’s deaths each day, among other illnesses.24 These conditions also wield a significant economic toll. It is estimated that the health and time-related costs of poor water and sanitation access amount to 4.3 percent of gross domestic product loss in sub-Saharan Africa and 2.9 percent loss in South Asia.25 While global progress toward achieving Sustainable Development Goals (SDGs) to meet WASH needs is notable, the least developed countries in Asia and sub-Saharan Africa have seen little improvement in the past two decades. Between 2000 and 2017, the proportion of people with at least basic water services has only increased from 51 percent to 56 percent in the least developed countries, and basic sanitation access has increased from 22 percent to 34 percent. In 2017, only 28 percent of households in the poorest countries had basic handwashing facilities.26 Many countries, with support from donors, practitioners, and other stakeholders, continue to invest in WASH improvements. However, the limited durability of WASH services in developing countries are well known. The 2016 Water Point Update from the Rural Water Supply Network showed that among 11 countries surveyed an average of 22 percent of water points (WPs) failed to function, with 15 percent failing one year after installation and 25 percent failing within four years.27 In a study of four sub￾Saharan African countries, an average of 13 percent of communities previously declared to be open defecation free (ODF) slipped back into open defecation (OD) status.28 Identifying and scaling sustainable solutions are thus critical to bridging these service gaps to meet the SDGs. In recent years, USAID has reoriented its focus to support partner countries’ commitment and capacity to address their own development needs—a “journey to self-reliance.” This means moving away from direct implementation of services toward system strengthening approaches for local actors. In the WASH sector, USAID aims to increase availability and sustainable management of safe water and sanitation to the underserved and vulnerable in particular. It recognizes that adequate financing, effective governance, effective management, and behavior change are critical elements that must be addressed to support increased sustainable access to and use of sanitation, safe drinking water services, practice of hygiene behaviors, and improved management of water resources.29 Through its commitment to identifying sustainable approaches to WASH, USAID commissioned its Water Communications and Knowledge Management (CKM) activity to complete a series of six independent ex-post evaluations to assess the extent to which USAID–funded activities achieved sustained WASH outcomes three to 10 years after closure and which factors drove those outcomes. This report synthesizes findings across all six evaluations, with additional analysis of WASH literature, to foster learning and improve evidence-based sustainable development assistance across the WASH sector. Findings address topics of finance, governance, water service management, and hygiene and sanitation behavior change. USAID.GOV EX-POST EVALUATION SYNTHESIS | 2 EX-POST EVALUATION SERIES Water CKM conducted its ex-post evaluation series between 2016 and 2019. The evaluation team (ET) took the following criteria into account when determining eligibility for evaluation: activities must have closed three to 10 years prior, possess a minimum budget of $500,000, and not be funded by humanitarian disaster relief or Food for Peace. From this list, the ET purposively selected sites that would provide broad opportunities for learning, with a preference for those with endline data available to enable direct comparison. Through this section process, Water CKM evaluated the following USAID activities, as ordered: ● Madagascar: Rural Access to New Opportunities for Health and Prosperity (RANO-HP), implemented 2009–2013 to improve WASH access in primarily rural areas (the ex-post only evaluated the sanitation and hygiene components). ● Indonesia: Environmental Services Program (ESP), implemented 2004–2010 to improve and expand safe water access in urban areas with a focus on support to utilities. ● Ethiopia: Millennium Water Alliance Ethiopia Program (MWA-EP), implemented 2004–2009 to improve WASH access in rural areas. ● India: Financial Institutions Reform and Expansion–Debt and Infrastructure (FIRE-D), implemented 1994–2011 to capacitate local stakeholders to plan, manage, and finance urban water and sanitation development, with a focus on commercial viability and social inclusion. ● Senegal: Programme d’Eau Potable et d’Assainissement du Millénaire (PEPAM/USAID), implemented 2009–2014 to improve WASH access in rural and peri-urban areas. ● Mozambique: Strengthening Communities through Integrated Programming (SCIP), implemented 2009–2015 to improve WASH access in rural and urban areas. All ex-post findings cited throughout this synthesis are available in the full-length report for each evaluation, which are accessible online along with briefs, blogs, and webinar recordings.30 These evaluations spanned a variety of WASH focus areas and implementation strategies, as shown in Figure 2. Data shared in this report represent only a selection of salient topics evaluated, and readers are encouraged to see full reports for additional findings. All evaluations used a mixed methods design that involved extensive qualitative interviews with a variety of stakeholders; quantitative methods varied for each evaluation as summarized in Figure 2. The ET compared changes since endline directly in Madagascar, with endline survey data replicated in the same sampled villages; somewhat directly in Indonesia, using a modified and replicated utility performance index; and to a limited extent in Mozambique, where the ET compared data from a nearly endline random sample of households to a non-random ex-post sample. At other sites, the ET used measures such as questions about recent latrine maintenance, publicly available data, and reasonable assumptions (e.g., 100 percent of WPs functioned at endline) to infer changes since endline where feasible. None of the study designs had a comparison group to permit causal attribution to the USAID activities. Each report from the series notes specific methodological limitations. 3 | EX-POST EVALUATION SYNTHESIS USAID.GOV Figure 2. Ex-Post Evaluation Sites, Intervention Components Evaluated, and Methodology USAID.GOV EX-POST EVALUATION SYNTHESIS | 4 KEY FINDINGS AT EX-POST Water Access. Across rural evaluation sites, many USAID–constructed WPs had failed, with 44 percent to 65 percent still functioning at ex-post (Figure 3). Urban areas fared somewhat better. In Indonesia, water connections increased in six of the eight utility catchment areas, for an average improvement from 35 percent to 43 percent population coverage over a five-year follow-up period. Two had slightly reduced coverage. In India, two of the USAID–supported municipalities evaluated at ex￾post increased water access in the past four to five years, whereas the four other cities kept relatively constant coverage levels, despite population growth. Coverage rates varied widely, from a consistent 62 percent for the past five years in one city to a steep three-year increase from 31 percent to 100 percent in another. Hygiene Behavior. The ET observed low presence of any handwashing facility with soap and water across all rural sites where USAID-funded implementing partners had intervened. Senegal had the highest lasting coverage since activity closure, at 31 percent. Thirteen percent of Mozambique households and zero out of 15 households visited in Ethiopia possessed handwashing facilities. In Madagascar, which had directly comparable endline data from four of the five targeted regions, the ET measured a significant reduction in those with a handwashing station with soap and water, from 8 to 2 percent (Figure 4). Figure 3. Sustainability of WASH in Predominantly Rural Ex-Post Evaluation Contexts 5 | EX-POST EVALUATION SYNTHESIS USAID.GOV Sanitation Behavior. Among rural ex-post contexts, Senegal had the greatest success in sustaining latrine access. The ET measured use of any type of latrine at 92 percent, but a high prevalence of shared or unimproved latrine models meant lower basic sanitation access, at 47 percent (Figure 3). In Madagascar, the ET measured statistically significant three-year slippage in use of a latrine of any kind from 69 percent to 45 percent at ex-post, though use remained above pre-activity baseline levels (Figure 4). Access to basic sanitation remained unchanged from endline, at 2 percent. Only two out of 69 villages previously declared to be ODF still met ODF criteria at ex-post. In Mozambique, loosely comparable endline data suggested latrine use may have increased from 26 percent at endline to 52 percent four years later, and access to basic sanitation increased from 5 percent to 15 percent.31 In India, only one USAID–supported municipality evaluated at ex-post substantially increased the proportion of households with access to sewerage connections whereas three remained relatively constant over the past few years. Coverage varied widely, with one city having a three-year reduction from 58 percent to 25 percent and another city holding relatively constant at 90 percent. The following chapters of this report describe findings from these ex-post evaluations in more detail and discuss factors that contributed to these changes or lack thereof. The six activities studied are not necessarily reflective of the sustainability potential for all such activities in other settings. However, the report places the ex-post series findings in context of other literature to help draw conclusions about factors and implementation approaches development practitioners must consider moving toward more sustainable WASH outcomes. Figure 4. Three-Year Sustainability of Key Madagascar (RANO-HP) Sanitation and Hygiene Outcomes USAID.GOV EX-POST EVALUATION SYNTHESIS | 6 FINANCING FOR SUSTAINABLE SERVICES Adequate and sustainable financing is a necessary condition for supporting national and subnational WASH strategies, meeting current and future demand, and ensuring sustainability.32 The developing world faces a stark gap in financing its WASH priorities, which is a key barrier that prevents countries from expanding access, improving service delivery, and meeting universal access targets associated with national WASH strategies or the SDGs.33 Factors that dictate the appropriate mix of financing strategies for a given context include: institutional characteristics, geography, population characteristics and growth, local economy, existing governance structures, and others.34 This section summarizes ex-post findings from interventions that addressed some of the conditions necessary to improve sustainable finance for WASH, as shown in Table 2. Table 2. Finance Interventions Evaluated at Ex-post Ethiopia India Indonesia Madagascar Mozambique Senegal Financial management strengthening for utilities, government entities or WASHCOs X X X X X Business planning support X X Cost recovery improvements through technology or reforms X X Credit enhancement to support market-based financing through municipal credit ratings and bonds X X Household finance for WASH products and services strengthened through microfinance and village savings and loan associations X X Key Takeaways: 1. Technical assistance to service providers for business planning, improved financial management and cost recovery, and credit enhancements were largely sustainable interventions, particularly when scaled through partnership. However, without an appropriate enabling environment, these strategies proved insufficient on their own to facilitate access to market-based finance, toward the goal of self-sufficiency. 2. Interventions leveraging technology, such as e-governance tools and improved metering and meter-reading, facilitated long-term improvements in financial management and cost recovery, especially through the reduction of NRW. 3. Training CBM entities to establish bank accounts and transparent accounting practices were insufficient to achieve sustainable finance for rural water system operations and maintenance. 4. While some limited microfinance continued to be available for WASH, it did not drive increased access to WASH services. This failure shows the need for appropriate alignment of lender incentives and processes with project goals and beneficiary needs. 7 | EX-POST EVALUATION SYNTHESIS USAID.GOV FINANCIAL MANAGEMENT Improving financial management among service providers is key to make better use of existing resources and assume a more attractive position for mobilizing repayable finance.35 The ex-post series evaluated several implementation approaches to improve financial management, ranging from establishing double￾entry accrual-based accounting systems to managing simple bank accounts. To strengthen the creditworthiness of selected water utilities in Indonesia, the ESP activity helped develop debt restructuring plans,36 identify sources of finance for new investment, and improve operating ratios and collection. A comparison of ESP–supported urban utilities’ ex-post performance index scores to scores captured at the time of the intervention revealed that most categories of utility performance improved in the seven years since ESP ended (e.g., corporate plans, human resources policies, management information systems, water quality monitoring). However, financial performance (measured through operating ratios, debt service coverage, and debt equity) substantially decreased over time. The evaluation did not pinpoint a particular cause, but it is clear that these interventions proved insufficient to overcome the major financial challenges that utilities faced at the outset. In India, the FIRE-D activity focused in part on improving the financial viability of state and municipal governments as part of a set of urban sector reforms aimed at attracting investments to improve commercially viable urban water and sanitation infrastructure. FIRE-D supported improving municipal financial management practices through adoption of double-entry accrual-based accounting systems (DEAAS). It introduced DEAAS to several municipalities and helped develop a technical manual for their long-term application. It then worked with the Government of India (GoI) to include DEAAS reforms and FIRE-D’s manuals into the GoI’s major urban development funding schemes. The ET found that DEAAS are still being used at most of the targeted cities visited through the evaluation. For the most part, municipalities still used the financial management manuals, even if they had not been updated as often as needed. Partnering with a motivated government facilitated sustainability of financial management reforms, even though these reforms alone did not sufficiently address financial management challenges. Though not an explicit FIRE-D activity at sites visited for the evaluation, management information systems have also facilitated financial stability via digitization of assets, allowing for real-time tracking of water and sanitation usage and improving revenue recovery through collection efficiency, according to stakeholders. In ex-post evaluation settings with a rural water supply component (Ethiopia, Senegal, and Mozambique), implementers typically trained community-based committees to complete detailed and transparent accounting and to establish bank accounts. In Senegal, committee members noted difficulties following through with these best practices years after the activity closed. None of the eight water committees interviewed actively kept transparent records of expenses and revenues. Community-based management (CBM) challenges are discussed later in this report. In the urban contexts, efforts directed at improving financial management of service providers and planners are necessary, and approaches that codify such improvements through policies or manuals can bolster sustainability. However, such efforts alone are insufficient to overcome what are often significant financial challenges. BUSINESS PLANNING Business planning can help utilities or communities improve financial management.37 Business planning aids service providers or other authorities in setting targets for service levels, identifying revenue sources to address financial gaps, prioritizing projects and assessing their costs, and determining revenue allocations.38 Such support may be especially appropriate for entities that are not yet at a stage where they can attract commercial loans or other repayable finance.39 Two utilities studied at ex-post in Indonesia praised ESP’s business planning support, noting its impact years after the activity ended. In that instance, the promise of debt relief from the national government specifically tied to credit enhancements (discussed in Credit Enhancements) rather than the business USAID.GOV EX-POST EVALUATION SYNTHESIS | 8 plans themselves, seemed to be a major motivation to achieve better financial performance. In Madagascar, RANO-HP supported the formation of district-level stakeholder groups that developed Commune Water and Sanitation Business Plans. These plans helped elected officials advocate on behalf of their district to solicit funding or other support for WASH activities from regional governments or other donors. The implementer trained district government stakeholders on how to apply the national water policy and helped them objectively identify priorities for WASH investment. However, local governments found it challenging to realize these plans given the limited regional government funding available for WASH and temporary donor disengagement following a political coup in 2009. Only one of six communes used the Commune Water and Sanitation Business Plans, according to stakeholder interviews. Findings from the ex-post series largely confirmed that business plans play a necessary and helpful role in supporting local government and utilities’ financial sustainability strategies, especially when they are updated regularly, but are only a starting point and must be paired with other strategies and reforms to be most effective in improving financial performance. COST RECOVERY Cost recovery to ensure adequate funding for operations and maintenance (O&M) is essential to achieve financial stability and sustainability, but it remains a perennial challenge in both urban and rural contexts. Virtually none of the water systems evaluated in rural settings had adequate cost recovery, as discussed in the Effective Management for Sustainable Water Services section below. Many urban sites visited in India and Indonesia noted cost recovery as a challenge. Common barriers to cost recovery include inadequate rates and collection efficiency of tariffs or fees, operational inefficiencies, expensive repairs often due to a lack of preventive maintenance, and unrecovered costs, primarily through non-revenue water (NRW). The ex-post series touched on a small number of issues under this larger topic and body of literature. First, cost-reflective tariffs or fees are essential for cost recovery. However, tariffs and fees are commonly not set in such a way as to recover basic O&M costs, which perpetuates dependencies on external sources of funding.40 This may be due to political pressures to keep tariffs low, perceived or actual willingness to pay based on quality of service delivery, and/or issues related to regulatory approval, such as the need for independent regulators to approve tariffs in relevant settings. The ex-post series noted challenges with insufficient tariffs and fees—as well as collection inefficiency—across all ex￾post evaluations with a water component. Given their strong interlinkage with governance issues, tariff setting policies are discussed further in the Establishing Effective Drinking Water & Sanitation Governance section. Reducing NRW is a major lever that water service providers must address to reduce inefficiency, improve cost recovery, and reduce strain on limited resources.41 In Indonesia, where ESP trained utilities to apply NRW reduction techniques, utilities’ performance index scores regarding NRW (e.g. procedures in place to mitigate NRW, functionality and maintenance of water meters, and NRW losses compared to previous year) improved slightly from the end of the activity until ex-post visits. Respondents at three utilities said NRW protocols and improved meter reading technology continue to be used today and have helped reduce their NRW rate. The FIRE-D activity in India included support for expenditure reduction through water or energy audits and leak detection repair, as well as e-governance initiatives that allowed customers to report leaks and register complaints. As with financial reforms described above, the national government adopted these types of reforms into its urban development scheme reform agenda, and the ET learned that other cities continued these practices across India. All three evaluation sites where FIRE-D had supported their development still used the e-governance systems. Respondents in Bangalore, a city with very robust revenue collection and cost efficiency, credited the FIRE-D–supported e-governance tools with facilitating a reduction in NRW from 51 percent to 27 percent. Since the activity ended, the city also 9 | EX-POST EVALUATION SYNTHESIS USAID.GOV adopted more advanced e-management tools, like bulk flow and household meter technology, to further reduce NRW. Reducing NRW continues to be a major challenge for urban utilities, and identifying ways to reduce NRW through the use of technology (e.g., e-governance or metering); collection efficiency; capacity-building; and better resource allocation for preventive maintenance to avoid leaks, bursts, or expensive repairs persists as a major priority for facilitating sustainable financial management. FIRE-D also supported strengthening municipal and state governments’ own-source revenues for the purpose of supporting cost recovery through efforts like property tax reform, which the GoI also adopted into its urban development schemes’ menu of reforms. However, due to lack of comparable quantitative data, the ET could not assess financial stability in the project sites in India as related to these interventions. In an urban context, the ex-post results illustrated the long-term value of interventions that support the application of technology to improve cost recovery. As with other finance interventions, these efforts are not sufficient to achieve financial stability on their own. CREDIT ENHANCEMENTS Attracting adequate financing for capital investments remains a major issue for municipal utilities and rural water services alike, though the issue is more often of interest in the former, considering the more complicated and expensive requirements of networked water systems. Credit enhancement interventions aim to facilitate utilities’ access to repayment-based financing, such as bonds and loans for large infrastructure investments, and reduce their reliance on public funds or foreign aid. In Indonesia, the ESP activity supported credit enhancement for utilities with the goal of providing a credit guarantee for corporate bonds. ESP worked with one utility to first obtain a credit rating to prepare for a corporate bond, but the bond did not materialize because of local reticence about USAID acting as the guarantor. However, this utility did secure a commercial loan to finance a new water treatment plant, which they attributed to the favorable credit rating they received with ESP support. In several India locations, FIRE-D piloted municipal credit ratings and subsequent municipal and state bonds to facilitate access to market-based finance for water and sanitation development. This case is unique given the project’s emphasis on and partnership with state or municipal governments rather than working directly with water service providers. Nonetheless, relevant lessons learned emerge. Intended to expose and address weaknesses in financial stability, the process of developing credit ratings prompts a pathway of reform toward a rating that could leverage market-based finance. All municipalities evaluated at ex-post reported having obtained a credit rating since FIRE-D ended; however, in most cases they undertook the exercise solely to meet a milestone requirement to access grants through GoI urban development schemes, rather than as a pathway to market finance. Therefore, in the case where major national government grant schemes or other donors made readily available, cheaper, and less administratively burdensome capital available, municipalities did not find market-based finance a necessary or attractive prospect at the time of ex-post evaluation, especially given stakeholders’ concerns about the commercial viability of water and sanitation in general. Of three evaluated municipalities where FIRE-D had supported bonds or credit ratings as a step toward market-based finance, two maintained a strong enough financial position to continue funding water and sanitation services through bonds. However, FIRE-D’s vision of commercially viable infrastructure did not materialize broadly in India, as abundant public funds for WASH became available. Where credit enhancements are used to facilitate access to repayment-based financing sources, attention must also be directed toward the overall enabling environment and financing landscape, including service providers’ incentives to seek certain sources of financing over others. HOUSEHOLD FINANCE While municipal utilities in urban contexts work toward cost recovery and self-sufficiency, expanding access to the poor is also a concern and often a mandate.42 This effort can put further financial strain on USAID.GOV EX-POST EVALUATION SYNTHESIS | 10 utilities. Additionally, the appropriate mechanisms for extending service access are not always clear. Several studies in the literature document potential positive impacts of microfinance in aiding poor households to invest in household water or sewer network connections.43 However, such efforts must be aligned with the local context to be effective or sustainable. Findings from the Indonesia ex-post evaluation indicate that the partner bank’s large size and lack of experience lending at such a small scale led it to set standards too stringently when assessing customers’ creditworthiness. A lack of advertising on the part of both parties to increase awareness about the program compounded the problem. Ultimately, the ET found no evidence that microloans for connection had been sustained at sites selected for evaluation in Indonesia. Microfinance can also be applied to sanitation. The implementer in Madagascar introduced a microfinance loan product to support construction of improved household latrines. Three years later, the ex-post evaluation found this loan product still in use, though typically only in urban areas and small towns where purchasing power is stronger and slab transport is easier. Alternatives to microfinance for the purpose of extending access to the poor include subsidies,44 finance directly from a utility (interest-free connection loans), communities pooling resources to pay for water connection or latrine construction costs, discounts to poor customers, and others. Indonesia utilities attempted discounts to low-income customers for short periods of time, but these discounts ultimately did not result in sustainable increases in access. The ET found that customers connecting through these campaigns frequently defaulted after a few months of their monthly payments. In Madagascar, the implementer supported Village Savings and Loan Associations (VSLAs) to help community members save capital for a variety of needs, including WASH, and as a venue for delivering WASH behavior change messages. The evaluation found that 17 percent of VSLA members took loans to finance latrine construction or improvements after the activity ended. Financing strategies that lower the initial steep barrier to obtaining a connection or constructing a household latrine, including microfinance and others, must be appropriately oriented toward the needs of the targeted customers and contextual realities in which service providers operate. The ex-post found that both the microfinance and utility discounts evaluated in Indonesia largely failed, because they did not take into consideration the context or consider long-term financial viability on the part of the utility or customers. 11 | EX-POST EVALUATION SYNTHESIS USAID.GOV ESTABLISHING EFFECTIVE DRINKING WATER & SANITATION GOVERNANCE No work on WASH sector finance is possible without simultaneously addressing governance, which sets the stage for sustainable WASH service delivery. Improving sector governance is a key USAID priority in WASH.45 This section discusses the core service delivery governance issues raised in the ex-post series and contextualizes them within the broader literature. Governance-related issues, including government commitment to the sector, clarification of roles and responsibilities between actors, tariff policies, and water-related land tenure policies surfaced as key enablers or inhibitors of sustainability. None of the ex-post evaluations focused on water resource management,46 and thus this aspect is not covered in this report. Of the activities covered in the ex-post series, FIRE-D in India had the strongest focus on policy reform and governance. This activity focused on improving the capacity of national, state, and local government actors to better govern and enact their roles and responsibilities for newly decentralized water and sanitation service delivery. As discussed above, FIRE-D partnered closely with the GoI to incorporate successful governance reforms, tools, and approaches into a major GoI urban development scheme. After FIRE-D ended, many of the same reform incentives were carried over into a subsequent GoI urban development funding scheme.47 FIRE-D also established capacity training networks. The findings regarding FIRE-D’s efforts are highlighted below. Though the other activities in the ex-post series did not intentionally intervene at the governance level, the evaluations nonetheless found governance to be a key factor affecting sustainability. Thus, these examples are also referenced below. GOVERNMENT COMMITMENT In all aspects of WASH, a strong government commitment is important. India serves as a notable example of a committed government willing to dedicate significant resources toward WASH and particularly for sanitation. One illustration of the impact of such a commitment is the Swachh Bharat Mission to end open defecation. After its establishment, the annual rate of decline in OD accelerated from 3 percent to 12 percent.48 Donor support for major government WASH initiatives is a valuable way to support sustainability. The FIRE-D ET concluded that the activity’s partnership with the GoI to integrate its work into its urban development schemes served as the key driver of sustainability and was a model worth replicating in the sector. Though the other ex-post evaluations were not designed to examine the influence of government engagement or policy on sustainability, particularly amid a constellation of other factors, it is conceivable that the political commitment and climate impacted sustainability in other locations too. For example, it is worth noting that Senegal, which had the highest latrine coverage, also had a substantial government commitment to WASH SDGs. The Government of Senegal established PEPAM as a framework through which various donors, including USAID, would address unified WASH development goals. In contrast, Madagascar experienced a coup d’état as RANO-HP began, which caused global donor disengagement, Key Takeaways: 1. Unclear roles and responsibilities across actors prevented effective service delivery and discouraged sustainability. Inadequate training and resources to carry out those roles also inhibited sustainability. 2. A committed government is an important driver of activity sustainability, particularly in terms of WASH governance reforms. 3. Unclear tariff policies covering rural areas led to widely varying tariffs across communities, nearly all of which did not attain adequate cost recovery. USAID.GOV EX-POST EVALUATION SYNTHESIS | 12 prohibitions on direct support to the national government, and general political volatility and uncertainty—a climate that likely impacted the WASH sector broadly at that time. CLARITY AND CAPACITY IN ROLES AND RESPONSIBILITIES National and local entities must have policies to help clarify roles and responsibilities for each aspect of service delivery, as well as the capacity to enact those roles to ensure service delivery is consistent, timely, and safe. It must be clear which party bears each responsibility, including building or rehabilitating water infrastructure, installing new connections, managing water infrastructure, maintaining and repairing infrastructure, and conducting and tracking water quality testing. In India, most municipalities did not have sufficient capacity to carry out their roles to plan and manage water and sanitation services at ex-post, which some respondents attributed to lack of personnel and resources to support the intended decentralization effort, lack of financial control, or lack of decision￾making autonomy. These challenges illustrate the importance of adequate implementation of governance policies by national and local authorities. FIRE-D also established capacity training networks, none of which remained active (for unknown reasons), as well as City Manager Associations, some of which remained active and effectively engaged in peer capacity building, while others failed due to deficiencies in governance, such as lack of clear mandates or bylaws. Generally, though insufficient to address all capacity or financial challenges, stakeholders viewed FIRE-D’s capacity support through DEAAS training and other financial stability mechanisms as valuable in the long term. The activity’s focus on governance in partnership with the national government enabled most of these capacity initiatives and reforms to be sustained and scaled through the GoI’s national WASH development schemes. The Ethiopia MWA-EP evaluation illustrated a situation in which institutional arrangements did not successfully clarify roles and responsibilities between different actors in rural settings. In interviews for the ex-post evaluation, respondents provided a wide variety of answers regarding who took responsibility for which types of maintenance and repairs of rural WPs. Some thought the sole responsibility lay with the water committee; others thought the water committee should respond to simple O&M whereas the government should take responsibility for more major repairs; and still others thought non-governmental organizations (NGOs) should support major repairs and rehabilitation. These inconsistencies existed not only between different types of stakeholders, such as water committees and government officials, but also within stakeholder groups, such as between different government entities. Though inconsistently reported, the most common understanding of roles and responsibilities between the government and communities under MWA-EP was the government handling “major” repairs and the community handling “minor” repairs. The SCIP evaluation in Mozambique found similar distinctions. Though the “major” versus “minor” repair distinction provides a modicum of clarity in roles and responsibilities, the specific repairs that constituted these two categories were not clear in any evaluation. To the extent known via project documents, neither the MWA-EP nor SCIP activities included specific efforts to clarify the roles and responsibilities between water actors. This lack of clarity regarding roles and responsibilities in rural Ethiopia and Mozambique are in line with broader findings regarding rural water service.49 However, larger-scale water services face slightly different issues: roles and responsibilities tend somewhat clearer, particularly where utilities are involved, but concerns remain regarding each entity’s financial and technical ability to effectively and efficiently perform their roles, as seen in the India FIRE-D example.50 TARIFF POLICIES The ability of service providers, whether community-level water committees or large utilities, to sustainably deliver services requires them to be able to cover their costs.51 A key component of cost recovery is cost-reflective tariffs (i.e., tariffs that reflect actual costs of service). Commonly, tariffs and fees are not set in such a way as to recover even basic O&M costs, which perpetuates dependencies on 13 | EX-POST EVALUATION SYNTHESIS USAID.GOV external sources of funding and often results in foregone maintenance and repairs.52 Sustainability of WP functionality in all rural settings depended upon the ability to cover O&M costs. These cost recovery concerns surfaced in both the India FIRE-D and the Indonesia ESP evaluations as well. In the urban areas utilities typically serve, tariff setting involves government-level policy and regulatory oversight, where regulators exist. However, in most rural areas community-based committees are frequently left to define their own water fees.53 The PEPAM/USAID ex-post evaluation in Senegal clearly illustrated the distinction between rural and urban tariff setting. PEPAM/USAID installed multi-point water systems in urban and peri-urban areas. Semi-professional water user organizations called Association d’Usagers de Forage (ASUFORs) managed these systems, while community water committees managed the small, manual water pumps. Government policy dictated the water fees that must be paid at the larger, ASUFOR–managed systems, but not for the small, community-managed WPs. Among the rural water activities in Ethiopia, Senegal, and Mozambique, tariff setting appeared largely ad hoc. PEPAM/USAID did extensive life cycle costing studies and advised water committees on how to set tariffs, but they left establishing the actual tariffs to the communities and water committees. In most cases, the water fees did not allow for adequate cost recovery, though it is unclear how many committees used PEPAM/USAID’s guidance initially. In Mozambique, government officials noted that they provided guidance to water committees on setting water fees but did not have official guidance or policies on concerning the level of fees. Water tariffs in developing countries are often set well below what would allow for full cost recovery.54 An inherent tension exists between establishing cost-reflective tariffs and ensuring adequate water access to lower income and vulnerable populations.55 In setting tariff structures and levels, governments must balance these divergent needs. Substantial political pressure is in place to keep tariffs low and affordable. At the same time, utilities tend to advocate for higher tariffs that will allow them to recover more of their costs and make adequate investments in the system.56 The pressures of keeping tariffs low are often based on the assumption that people are unable or unwilling to pay more for water. However, studies have found that willingness to pay is not solely a function of incomes, even among the poor.57 Willingness is also closely linked to service quality and perceived water quality. In the ex-post series, customers of all eight utilities studied in Indonesia complained of service outages or poor-quality water during qualitative interviews, such that many households reportedly turned to free, alternative sources of water as soon as they became seasonally available. Some countries and their utilities have explored alternative tariff structures that allow for cross￾subsidies whereby commercial users and/or wealthier users pay higher rates than poorer users. For example, Increasing Block Tariffs (IBTs) have been one of the most common cross-subsidizing models. Typically, under an IBT, different rates are set for different levels of consumption. With the assumption that poorer households consume less water, the rates at the lowest end of the scale are heavily subsidized while rates at the high end of the scale (and often for commercial or industrial users) are much higher, helping to subsidize the lower rates offered at the bottom of the scale. Some successes have been found with such pro-poor tariff policies, though not in all cases. Some studies have also found that efforts to subsidize consumption of the poor through cross-subsidies benefits the poor less than anticipated.58,59,60 Misaligned assumptions about willingness to pay can result in tariffs that are less cost reflective than necessary.61 Some progress can be made to change the understanding about how tariffs and fees are set, improve pro-poor tariff policies, improve service quality to attract and keep more customers, and correct perceptions about water quality. USAID.GOV EX-POST EVALUATION SYNTHESIS | 14 LAND TENURE POLICIES Land tenure governance is a broad topic with implications well outside the WASH sector. Such issues affected water access in both the India and Ethiopia ex-post evaluations, albeit in very different ways. The India FIRE-D evaluation discovered land tenure inhibited the extension of drinking water and sanitation access to the poor who lived in informal settlements without property rights. This arose in several interviews as a primary barrier to utilities expanding network access in these communities. Fully addressing this barrier may be complex, given a study in Peru that found securing a household’s land tenure rights led to only marginal improvement to water access but greater improvements in sanitation and electricity.62 In the Ethiopia MWA-EP evaluation, land tenure-related conflicts threatened sustainable long-term WP access in two cases. In one case, MWA-EP supported construction of a protected spring on a farmer’s land. Conflicts arose between the farmer and the community over the selection of the management committee. The issue was particularly contentious because attendees were paid to attend committee training. When the WP broke down, the farmer refused to let it be fixed and refused to allow a new WP to be built on his land. In another case, MWA-EP constructed a multi-village piped water system. The service pipes traversed one man’s land, but distance prevented him from accessing the system’s water. In response, the landowner sabotaged the water pipes so that he could collect water from them. The breaks and leaks caused WPs downstream to fail. Both cases illustrate that the failure to take into consideration land tenure when installing water or sanitation infrastructure can result in barriers to sustainable and equitable access. 15 | EX-POST EVALUATION SYNTHESIS USAID.GOV EFFECTIVE MANAGEMENT FOR SUSTAINABLE WATER SERVICES Many factors affect the long-term sustainability of water infrastructure, but each of the ex-post evaluations that included a water component found the service provider’s management practices to be a key factor. This section discusses what did and did not work for each in terms of core management models, treatment of costs, and the handling of O&M and repairs. Findings related to management interventions across the ex-post series (Table 3) are discussed. Table 3. Water Management Interventions Evaluated at Ex-post Ethiopia India Indonesia Madagascar Mozambique Senegal Capacity Building for CBM entities X X X Capacity building for utilities X X Establishment of public-private partnerships X X X Cost recovery training or tools for service delivery entities in fee/ tariff collection X X X X X Water service and quality monitoring training or tools X X Supply chain access interventions for spare parts X X WATER SERVICE DELIVERY MANAGEMENT PRACTICES AND MODELS Water services are managed in multiple ways. The ex-post series evaluated examples of CBM, utility￾based services, and private sector contracts. Key Takeaways: 1. CBM entities did not successfully ensure long-term sustainability of water services in rural areas, regardless of capacity building and training interventions. 2. Cost recovery for water service delivery was a key challenge to sustainability. In both rural and urban areas, setting cost-reflective tariffs and collecting these fees was a challenge, with some rural systems unable to collect fees at all. Operational inefficiencies and high losses also contributed to this challenge in urban areas. 3. The use of PPPs for water service delivery showed mixed success, due in part to the lack of cost recovery, government inexperience with management and oversight of PPPs, and inadequate relationship building among stakeholders. 4. Efforts to strengthen supply chains for spare parts to enable maintenance by local technicians largely failed. USAID.GOV EX-POST EVALUATION SYNTHESIS | 16 CAPACITY BUILDING FOR CBM ENTITIES CBM structures have historically dominated the management of rural water services, including all three rural water activities studied in the ex-post series. The ET found significant proportions of the activity￾supported WPs at these sites no longer functioned, suggesting universal shortcomings in the CBM approach: 56 percent in Ethiopia (eight years post-activity), 37 percent in Senegal (four years post￾activity), and 35 percent in Mozambique (four years post-activity) were non-functioning.63 Broader reviews of rural WPs have found similar rates of non-functionality as the ex-post series.64 Many CBM entities evaluated in the ex-post series received basic training on how to maintain WPs and conduct minor repairs, but typically did not have the skills necessary for more significant repairs. Additionally, skills taught during the activity were sometimes be lost when committee membership changed. Frequently, the government, donors, and NGOs were forced to play a large role to ensure water access in communities. Some specific challenges CBM entities face in managing WPs are discussed later in this section. CAPACITY BUILDING FOR UTILITIES Many urban utilities in developing countries underperform and fail to provide effective or efficient service to their customers due to complex factors. Thus, donor-funded urban water projects often aim to improve utility effectiveness and efficiency and expand service to underserved segments of the population. Capacity building efforts are common approaches.65 The ESP activity in Indonesia included capacity building for utilities. ESP’s efforts to strengthen these water utilities and increase access to water connections appeared to be largely sustainable. The ex-post evaluation found that almost all evaluated utilities had maintained and even continued to improve their management capacity scores proceeding the activity. The largest gains came in the areas of corporate planning, tariff setting, and human resource policies. ESP’s development of standard operating procedures over the course of a long and committed engagement with the utilities appeared to be a key driver of this success. However, as discussed above regarding tariff policy governance, customer reports of dissatisfaction with the utilities’ quality of service highlight a gap between the utility performance Figure 5. Water Utility in Indonesia 17 | EX-POST EVALUATION SYNTHESIS USAID.GOV metrics used and actual quality of service levels, showing that the necessity of improvements to sustain consistent consumer demand and hence revenue streams. One component of the FIRE-D activity included the corporatization of the public water utility in Bhubaneswar, Odisha State, with the goal of improving the business orientation of the utility. The changes included revising tariffs, establishing regulatory frameworks for contracts, and improving capacity for managing performance-based contracts. The ex-post evaluation found that the corporatization took substantially longer than anticipated due to political and administrative hurdles, but six years after the end of FIRE-D a corporatized entity had been successfully established, in part through continued support from other donors. ESTABLISHMENT OF PUBLIC-PRIVATE PARTNERSHIPS Engagement of the private sector in water service delivery is substantially more prevalent in urban areas than in rural areas, creating a longer track record in urban areas regarding what does and does not work.66 A 2009 World Bank study found that PPPs achieved significant operational efficiency gains, particularly regarding staff productivity and bill collection.67 FIRE-D included a PPP component in Tiruppur, Tamil Nadu State for water supply and sewerage. Unfortunately, it was largely unsuccessful and failed after the government passed new environmental legislation that caused demand to plummet. Across visited sites in India, stakeholders expressed skepticism about the future of water and sewerage PPPs, due to concerns over commercial viability and negative public opinions of privatization efforts. The commercial viability perception ties back to the importance of financial stability. Given low user fee collection in many cities in India and political incentives to keep tariffs low, the future success of PPPs will depend on improved revenue collection efficiency and tariff rationalization, among other factors. Some rural activities in the ex-post series explored combining CBM with private sector models, such as outsourcing O&M to private operators. Both the Senegal and Madagascar evaluations included a private sector component for water management. In Senegal, PEPAM/USAID helped create and train local drillers and maintenance/repair companies. It also established contracts between water committees and the maintenance/repair companies. None of these contracts still existed at the ex-post evaluation, although some of the companies remained in business and continued to work with local communities. Figure 6. A Monoblock and Attendant in Madagascar USAID.GOV EX-POST EVALUATION SYNTHESIS | 18 Several factors contributed to the unsustainability of contracts: some communities failed to honor the payments required for their service contracts; PEPAM/USAID-supported firms had competition from similar repair companies; finding necessary parts made it difficult to complete work; and no one took ownership for oversight and enforcement of the contracts. Though contracting for services ceased, the companies reported continued demand for services, albeit on an ad hoc basis rather than through signed contracts. The ex-post evaluation in Madagascar examined RANO-HP “monoblocks” established to provide water access along with latrine, showering, and laundry facilities. Through a PPP arrangement, private contractors operated, maintained, and repaired the monoblocks. The evaluation found mixed results on sustainability. Of the five monoblocks in two communities visited in the evaluation, all still provided water. However, the two communities had vastly different experiences. In one community, the PPP model largely operated as intended and maintained all WASH service components. However, in the other community, the private firm had disappeared and only the water services still functioned, albeit with deferred maintenance and repairs. In the latter case, the ex-post evaluation found that poor relationships between the private sector entities and the local government and communities contributed to the poor performance. Though not covered directly in the ex-post series, Villanova University conducted a separate study of 10 PPP-managed RANO-HP water systems in Madagascar at approximately the same time as the Water CKM ex-post evaluation. It found that all 10 of the studied water systems still functioned, though reliability scores ranged from 63 percent to 100 percent (mostly due to poor water quality).68 In addition, inadequate management practices and below-target demand, given alternative sources, threatened profitability and sustainability. A 2017 World Bank study found similarly mixed results for private sector approaches in rural areas, though the authors believed the approaches held significant promise. The study mostly captured rural private sector approaches in the pilot stage or being implemented at small scales. The most successful models resulted from long-term engagement among governments, development organizations, and the private sector, and typically included a clear delineation of the institutional framework to support private sector engagement.69 Not all PPPs are successful, leading some to doubt their role in the sector. However, several key factors undergird successful PPPs, including having staff experienced in PPP implementation within the utilities, banks, and private sector companies. Political will is needed to engage in these partnerships and create a clear regulatory and institutional framework; such frameworks are also linked with PPP success. Successful PPPs, as with other types of utility reforms, depend upon financial and technical experts and managers to drive long-term planning, rather than politics.70 COST RECOVERY In both the ex-post series and the broader literature, financial challenges and the inability to raise sufficient funding to support O&M and repairs were core challenge for the ability of rural water committees and urban water utilities to sustain their infrastructure.71 In rural areas a division of responsibility for costs is often lacking, even in cases where roles and responsibilities are clear. This tends to be clearer for utilities, however.72 Across water activities in the ex-post series, management entities had difficulty raising sufficient funds to cover their costs, which contributed to high rates of infrastructure failure. In Ethiopia, only six of 12 visited water committees collected any water fees at the time of the evaluation. In Senegal, only 33 percent of water users reported paying water fees. Though Mozambique was markedly better, only 60 percent of households reported paying water fees. Other studies find similar difficulties in collecting water fees.73 In part, the challenges in collecting sufficient funds can be attributed to the water fee rates (discussed in more depth in the Establishing Effective Drinking Water & Sanitation Governance section), but also to low collection rates. The ex-post series identified two key factors contributing to the collection of 19 | EX-POST EVALUATION SYNTHESIS USAID.GOV water fees. First, poverty levels can affect how much households report paying for water, as seen in the Mozambique study. It is not clear from the ex-post evidence whether the differences between households are due to different rates being charged to poorer households or their lower ability or willingness to pay. This may be due in part to poorer households paying less or paying less frequently. Given the lack of clarity on how to set tariffs for CBM-managed water services and that committees and communities are often left to set rates, lower collection rates may also be due to poorer communities setting lower tariff rates. Second, the Ethiopia evaluation found that access to alternative, particularly free, water sources can hinder willingness to pay, as can a lack of prior experience with paying for water. In that evaluation, more households paid water fees, and the ET heard less skepticism about paying water fees in regions where users had been doing so for many years as compared to areas new to the practice. Similarly, many communities across the ex-post series did not collect water fees on an ongoing basis, but rather did so only when they needed the funds for repairs. The Indonesia ESP ex-post evaluation found that customers’ often unfavorable perceptions of utility water quality and service reliability affected their willingness to connect to or pay for it and the extent to which they opted for alternative water sources. A World Bank study found that the commercially￾focused nature of PPPs helped utilities significantly improve bill collection, resulting in higher utility revenues. However, the study found inconclusive results on cost-reflective tariffs, given the wide number of factors involved in tariff setting, including political economy issues, pro-poor policies, and other factors.74 OPERATIONS, MAINTENANCE, AND REPAIRS The pervasive challenges in collecting sufficient funds to pay for O&M and repairs at times led to foregone maintenance and consequently more costly delayed repairs or replacement. In urban areas, maintenance can also be neglected in favor of network expansions, further exacerbating the cycle of poor maintenance. Some communities leave their WP unrepaired if they are unable to raise the necessary funds.75 This “fix on failure” approach has often become the de facto model for community managed water supplies. This trend—identified across all three of the rural water ex-post evaluations— is exacerbated when communities only collect water fees once a specific repair is needed. All three rural water ex-post evaluations experienced this scenario when breakdowns led to nonfunctioning WPs, as opposed to wells drying up or other concerns. However, beyond adequate finance, effective operations, maintenance, and repair of water systems also requires an understanding of the water system. This includes the location of infrastructure, its ownership, and its performance status (including functionality and water quality). Asset management, water service monitoring, and water quality testing combine to address these needs, with an adequate supply chain serving as another essential component. ASSET MANAGEMENT A 2017 World Bank study found that rural areas often lacked asset information as well as systems for monitoring water services.76 Unclear ownership and responsibility for infrastructure can lead to poor maintenance and repairs. Particularly for areas with CBM structures, monitoring systems tend to be nascent and often struggle to go beyond occasional water system inventories or one-off studies.77 This is in line with the findings of the ex-post series, which documented the difficulty of obtaining government￾level information regarding the status of WPs that often did not exist. The team could not obtain WP information from national or regional government offices in any of its rural water evaluations. At the local level, only in Ethiopia did any local water authorities have standardized lists or accounting of water infrastructure, and this information tended to be outdated or unavailable in all districts. Even with the government lists (when available), the ET had significant difficulties locating many of the WPs installed as part of the MWA-EP activity. The team often had to USAID.GOV EX-POST EVALUATION SYNTHESIS | 20 rely on the recall of local water office officials and local leaders to locate activity-supported infrastructure. The literature shows that urban utilities have had better and clearer asset management and roles and responsibilities in place than rural providers.78 None of the urban ex-post evaluations specifically focused on improving asset management, though FIRE-D included a mapping exercise in selected municipalities that included mapping of water infrastructure. Local stakeholders hailed FIRE-D’s technical assistance in mapping slums as helpful in promoting inclusive slum development; however, local entities failed to update the maps over time, leaving a notable information gap. WATER SERVICE AND QUALITY MONITORING Wealthier countries tend to have more sophisticated monitoring systems, as do utilities, which serve more urban areas.79 In India, FIRE-D supported municipal infrastructure monitoring systems in some areas as a component of the e-governance systems that integrated numerous governance and management tasks.80 The ex-post evaluation assessed the water monitoring systems that remained functional after FIRE-D ended and found stakeholders held them in high regard. They considered the frequent data obtained through these platforms to be invaluable to the local utilities that used them to better manage their services. In terms of water quality, in all three of the rural water activities studied in the ex-post series, the national governments (Ethiopia, Senegal, and Mozambique) set their own standards for water quality and had policies in place regarding water quality testing. Despite these policies, none of the three evaluations found regular water quality testing of rural WPs. In most cases, water quality testing took place at the time of construction and then infrequently, if ever, after that. The primary constraints to fully implementing water quality testing policies, based on the ex-post series findings, included a lack of human capacity (both in terms of numbers and skills) and financial resources. Additionally, Ethiopia stakeholders’ confusion about roles and responsibilities extended to water quality testing and related remediation. Different stakeholders espoused differing views on who should perform water quality testing and remediation. In Indonesia, where ESP supported improved water quality monitoring at utilities, an organizational performance assessment index showed improvement in water quality monitoring (measured to include frequency and number of samples, laboratories and percentage of results in compliance with standards, and actions taken for non-compliance) between the end of the activity and the ex-post evaluation. Figure 7. Water Quality Testing in Senegal 21 | EX-POST EVALUATION SYNTHESIS USAID.GOV SUPPLY CHAIN ACCESS Access to necessary spare parts is frequently cited as an additional challenge to sustaining water service delivery in rural areas.81 Many recent WASH projects have included activities aimed at ensuring sustained access to spare parts for the WPs, including the Senegal PEPAM/USAID and Mozambique SCIP activities. The PEPAM/USAID ex-post evaluation did not cover the two parts suppliers created as a part of the activity, and thus little is known about its potential success. In Mozambique, the activity helped establish and train new businesses to sell spare WP parts. The evaluation found these businesses no longer operated, potentially due to a lack of demand, as the activities focused mainly on ensuring adequate supply of spare parts and not on ensuring sufficient demand to make the businesses viable in the long term. This is similar to the finding regarding the repair and maintenance companies created in Senegal. As discussed above, the ET found that finances trumped all other concerns, including spare part availability. It is perhaps unsurprising, then, that despite the focus on ensuring a supply of spare parts in these activities, the long-term sustainability of the WPs did not depend on these supply chains. USAID.GOV EX-POST EVALUATION SYNTHESIS | 22 SUSTAINABLE SANITATION AND HANDWASHING DEMAND AND BEHAVIOR CHANGE Handwashing with soap (HWWS) at critical times and the use of sanitation for safe feces disposal are essential behaviors to reduce the risk of diseases that are the leading cause of child mortality in most developing countries.82 Like safe drinking water, hygiene and sanitation are also inextricably linked with nutrition, environmental, and broader health considerations.83 Decades of sluggish improvement in HWWS and sanitation behaviors in developing countries, and the commonality of slippage away from open defecation free behavior over time, require more user￾responsive WASH products and services and more effective social and behavior change (SBC) approaches. With such critical health implications, it is imperative that governments and citizens, with the support of donors and practitioners, hasten progress toward sustainable change. SBC is multidimensional, and practitioners draw on several behavioral frameworks in an effort to prompt both habitual and normative HWWS and sanitation practices. However, consensus in the WASH SBC sector is lacking when it comes to which approaches and behavioral determinants should be employed to foster sustained change. A 2020 meta-analysis by White et al. codified that knowledge alone is not sufficient to impact lasting handwashing behavior change.84 SBC messaging and behavioral determinants occur within a much larger and complex enabling environment that includes governance, finance, structural, and market-based factors. The complexity of human behavior and the constrained environments in which handwashing and sanitation are practiced in the developing world create a significant challenge to fostering lasting behaviors. Using a multidimensional SBC orientation, this section discusses ex-post findings regarding handwashing and sanitation behavior change within four rural evaluation settings and addresses how various factors affected these outcomes. The ex-post series addressed a variety of common WASH interventions, shown in Table 4. Key Takeaways: 1. Handwashing social and behavior change programs focused primarily on knowledge, such as PHAST, did not work, nor did they did address the multidimensional drivers of and structural barriers to behavior change. 2. CLTS did not eliminate open defecation in the long term, and communities relied upon poor quality latrines. Triggered households did not generally progress toward improved or basic sanitation, meaning households missed out on substantial health benefits over the long term. 3. Financial barriers served as the major impediment to sustained latrine quality, maintenance, and use. Targeted subsidies combined with CLTS in Senegal offered modest improvements to sustained outcomes. 23 | EX-POST EVALUATION SYNTHESIS USAID.GOV Table 4. Hygiene and Sanitation Behavior Change Interventions Evaluated at Ex-Post Ethiopia India Indonesia Madagascar Mozambique Senegal Handwashing behavior change promotion primarily through PHAST, SARAR X X X X Promotion of tippy taps as fixed handwashing hardware X X X X Sanitation behavior change primarily through CLTS X X X X Household finance strengthening through sanitation subsidies (combined with social marketing or CLTS), microfinance, or village savings and loans X X Professional construction support through training of local masons X X X Ongoing local behavior change communication or follow-up through training local agents and committees X X X X HANDWASHING BEHAVIOR CHANGE The ex-post series examined the sustainability of household-level HWWS practices at sites that received past interventions based on PHAST in Ethiopia, Senegal, and Mozambique. Senegal used SARAR in conjunction with PHAST. In Madagascar, the implementer simply promoted three key WASH behavior messages that included HWWS. All sites trained local behavior change agents85 and used CLTS strategies.86 Tippy taps87 constructed from locally available materials served as the predominant handwashing hardware promoted in Senegal, Mozambique, and Madagascar. Figure 8. A Latrine with a Washable Slab and Tippy Tap in Anosy, Madagascar USAID.GOV EX-POST EVALUATION SYNTHESIS | 24 The ET measured a low prevalence of basic handwashing facilities (those with both soap and water) across all sites where it conducted household observations years after HWWS promotion interventions ended (Figure 3). Observations at the USAID–supported sites in Senegal exceeded JMP national and rural coverage estimates from the prior year, suggesting above average behavior.88 However, without the benefit of endline data for direct comparison, it is unclear whether 31 percent with hand washing facilities reflects slippage, sustained levels, or improvement from project close. Endline data available for both Madagascar—handwashing practices improved from 1 percent of intervention communities at baseline to 8 percent at project close—and Mozambique—the endline documented no significant change in self-reported handwashing—revealed that neither project achieved substantial improvement in HWWS indicators during the course of the activity, making the low coverage at ex-post unsurprising. This aligns with a systematic review of handwashing behavior that found only four of 30 HWWS interventions demonstrated lasting behavior change one year after the intervention.89 Similarly, another ex-post evaluation of a USAID urban WASH activity in Ghana found only 17 percent of households had basic handwashing facilities four years after the handwashing promotion activity ended.90 Achieving HWWS behavior change during an activity, let alone sustaining it several years later, is one of the biggest challenges in the WASH sector. Determinants of handwashing behavior are not well understood in the WASH sector, and much of the research and programming have targeted a relatively narrow range of determinants, such as knowledge, health, perceived risk, and personal characteristics, rather than more complex drivers across technology, psychosocial, and contextual dimensions.91 Ultimately, the lack of substantial HWWS behavior change at the end of the interventions pre-empted the ET’s ability to assess sustainability. PSYCHOSOCIAL DRIVERS OF BEHAVIOR All hygiene interventions studied shared a common focus on knowledge about diseases and preventative behaviors to improve health. Studies have found such WASH messaging interventions are unable to prompt sustained behavior change.92 Other weaknesses are commonly associated with these approaches, including skepticism related to social biases and the challenges of participatory processes where identifying “correct” behavioral drivers may not actually align with actual behavioral drivers.93 In sum, these behavior change approaches did not appear to contribute to sustained behavior change, which is in line with current sector views. The ex-post findings ranged from a majority of respondents self-reporting HWWS behaviors at critical times in Senegal and Mozambique to very low levels in Madagascar.94 However, regardless of self-reporting, observable indicators of HWWS practice, such as low levels of available water and soap for handwashing, are aligned with the literature, which indicates that knowledge alone is not effective to sustain behavior change.95 Actors in the sector have largely replaced or supplemented PHAST and SARAR with other interventions that address the sector’s evolving understanding of other behavioral drivers such as aspiration, convenience, and nurture.96 HARDWARE The 2020 White et al. systematic review concluded that one of the most effective interventions to improve HWWS may be to ensure that a fixed, conveniently located handwashing station with soap and water is available in an observable setting. This acts as a cue to action at critical times to overcome psychological barriers, such as perceived difficulty or lack of time, and to enhance positive social pressure.97 While tippy taps have been popular due to their fixed placement and simple design using local materials, the ET found virtually none at ex-post in the communities mobilized to use tippy taps in Senegal, Ethiopia, Mozambique, or Madagascar. Respondents in Senegal and Mozambique widely reported the lack of durability of their tippy taps. Most households in Senegal and Mozambique replaced their defunct tippy tap with mobile objects like a pitcher and basin. Selecting appropriate hardware to facilitate behavior is often dependent on the local context. The WASH sector is increasingly interested in applying human-centered design to hardware like handwashing stations to avoid such challenges.98 25 | EX-POST EVALUATION SYNTHESIS USAID.GOV Given the observation that few households had both water and soap available, even in alternative devices like mobile kettles, the root concern about the lack of sufficient motivation or reduction in barriers to change HWWS behavior is also very important and mutually reinforcing with fixed hardware. It has been shown that when water sources are distant from the home, which is often the case in these rural settings, use of water for hygiene purposes in particular declines.99 Strong shifts in maintenance of basic handwashing materials may not occur without substantial improvement in on-premises water access. FINANCIAL CONSTRAINTS Respondents in Ethiopia and Mozambique cited financial constraints as a barrier to HWWS. In Mozambique, 78 percent of all survey respondents said lack of finance prevented consistent HWWS practice. Poorer households more commonly noted financial constraints. In impoverished communities, families constantly make difficult decisions about how to allocate their sometimes extremely limited finances. As a relatively low-cost behavior, lack of financial allocation to maintain soap and water is perhaps more reflective of low prioritization relative to other needs. Overall, limited HWWS does not necessarily indicate a lack of value or will to carry out the behavior and should be viewed within the larger enabling environment. The ex-posts found the same could be said of financial constraints related to sanitation and water. ONGOING LOCAL BEHAVIOR CHANGE COMMUNICATION The WASH sector has been debating the value of ongoing, consistent interpersonal WASH behavior change promotion in shifting norms and helping to enshrine behavior. Qualitative interviews in Ethiopia and Senegal called for more follow-up as one solution to the poor hygiene practices observed at ex￾post. The effect of ongoing behavior change communication (BCC) activities is in some ways illustrated through the observation that communities in Madagascar and Senegal had better HWWS outcomes if they received additional WASH interventions from other donors after the USAID activity ended. However, ongoing donor engagement is not a sustainable or desired outcome. Host governments and local community members must be the ones to champion WASH goals and related communication in their journey to self-reliance. According to qualitative interviews, parties in Madagascar, Ethiopia, and Mozambique that had been trained to conduct ongoing house-to-house WASH BCC after the activities ended continued these activities through the time of ex-post data collection, though the frequency, content, and rigor of those efforts remains unknown. The very low prevalence of observed basic handwashing facilities at all three sites indicates this traditional BCC strategy has not been sufficient to move households from knowledge Figure 9. Mozambiquan Woman Using a Tippy Tap USAID.GOV EX-POST EVALUATION SYNTHESIS | 26 to action. In Senegal, where the activity capacitated local BCC agents and used social marketing during the intervention, the higher prevalence of basic handwashing facilities relative to nationwide survey data from the prior year provides an interesting contrast, though the lack of endline data and other information limitations prevent conclusions about the direction and drivers of change. Beyond potential root inadequacies of BCC approaches, logistical constraints for local BCC agents may also be at play. Insufficient staffing and resources in Ethiopia hampered health extension workers’ efforts to cover large catchment areas. In addition, the government tasks health extension workers with numerous health promotion topics in addition to WASH, which limits their ability to address WASH behaviors in depth. The ET had limited information on ongoing BCC efforts, but other sites may have faced similar challenges. Effective promotion of behavior change is complex, and beyond the theoretical framework used, operational factors can have a significant impact on performance and eventual behavioral uptake. Suboptimal implementation, such as overburdened BCC agents, insufficient training and retraining opportunities, minimal or no supportive supervision, and limited resources and renumeration, plays a significant role in the potential for success of community behavior change agents.100 Ultimately, because community BCC agents tend to focus on reinforcing knowledge and, to some degree, values and norms, this is not sufficient to address more important underlying drivers of HWWS behavior. CONTEXTUAL FACTORS The contextual characteristics of an individual, environment, or setting can influence behavior. For example, the level of water scarcity in one’s area can lead one to economize water use for hygiene.101 The broader literature supports that intrahousehold dynamics, typically reflected in gendered decision￾making related to prioritization and financial decisions, can stymie consistent access to handwashing materials, especially soap.102 Findings in Madagascar may reflect such gendered prioritization, as the possession of a basic handwashing facility declined more sharply over time among male-headed households (81 percent decline since endline) than among female-headed households (28 percent decline). Such dynamics may necessitate changes to programmatic approaches that have traditionally excluded men from WASH behavior change activities.103 Though not a component of any activities evaluated, behavioral “nudges” or environmental manipulations are attracting increasing interest as a way of prompting subconscious behavioral choices. A randomized controlled trial found that environmental nudges at schools, such as painting shoe prints on a path from the latrines to the handwashing station, improved student handwashing practices by 20 percentage points—as effective as a high-intensity hygiene education intervention.104 These innovations hold some promise and could be applied as part of an arsenal of behavior change tools.105 Beyond potential root inadequacies of BCC approaches, logistical constraints for local BCC agents may also be at play. Insufficient staffing and resources in Ethiopia hampered health extension workers’ efforts to cover large catchment areas. In addition, the government tasks health extension workers with numerous health promotion topics in addition to WASH, which limits their ability to address WASH behaviors in depth. The ET had limited information on ongoing BCC efforts, but other sites may have faced similar challenges. Effective promotion of behavior change is complex, and beyond the theoretical framework used, operational factors can have a significant impact on performance and eventual behavioral uptake. Suboptimal implementation, such as overburdened BCC agents, insufficient training and retraining opportunities, minimal or no supportive supervision, and limited resources and renumeration, plays a significant role in the potential for success of community behavior change agents.106 Ultimately, because community BCC agents tend to focus on reinforcing knowledge and, to some degree, values and norms, this is not sufficient to address more important underlying drivers of HWWS behavior. 27 | EX-POST EVALUATION SYNTHESIS USAID.GOV SANITATION BEHAVIOR CHANGE The ex-post series examined household-level sanitation hardware and behaviors at three sites that received CLTS interventions: Madagascar, Senegal, and Mozambique. All three countries trained local masons to provide ongoing skilled construction and repair. The Senegal intervention allowed comparison of communities that received traditional CLTS, a hybrid CLTS-plus-subsidy, and a subsidized sanitation marketing approach (hereafter referenced as the Subsidy approach).107 PHAST appeared to be the dominant sanitation promotion approach in Ethiopia,108 and it was used alongside CLTS in Senegal and Mozambique. As described earlier, evaluation sites in Senegal had nearly full latrine coverage at ex-post, and approximately half of observed households in Madagascar and Mozambique had access to any latrine. In the long term, CLTS did not succeed in eliminating open defecation in the ex-post villages. Nearly all previously declared ODF villages in Madagascar and 73 percent and 32 percent of surveyed households in Mozambique and Senegal, respectively, reported that they knew someone who still practiced OD. In Senegal, the percentage of people reporting OD behavior differed significantly based on what sanitation approaches had been put in place: 14 percent at CLTS sites, 31 percent at Subsidy sites, and 43 percent at hybrid CLTS-plus subsidy sites. At ex-post, the ET observed basic sanitation (improved, nonshared latrines) access at 15 percent and 47 percent of households in Mozambique and Senegal, respectively. Access remained extremely low in Madagascar at 2 percent. Rural household latrines are expected to require periodic maintenance and occasional pit evacuation or complete replacement after they fill. It is hoped that households would add additional features of privacy, safety, and quality to their latrines over time so they transform from unimproved to basic sanitation —known as climbing the “sanitation ladder.”109 Across all rural contexts evaluated, many households that constructed latrines at the time of the activity did maintain and replace them as needed over time, but the latrines generally remained of poor quality, many with safety concerns. The ET observed a few examples of households making quality improvements up the sanitation ladder and many shared latrines between households. The majority of latrines lacked slabs, a roof, and walls. As with handwashing, the factors that drive sustained use of latrines and adoption of basic sanitation services are multifaceted. The sections below address various factors that either improved or served as barriers to sustained maintenance, use, and improvement of latrines and how the factors fit into broader evidence about CLTS in particular, as it is the predominant approach studied in this series. Figure 10. Mozambique Latrine Quality and Style Mozambique latrine quality and style ranged from simple unstable open pit holes without a superstructure (top left) to concrete slabs with superstructure and a handwashing station (bottom right). Photo credit: Forcier Consulting USAID.GOV EX-POST EVALUATION SYNTHESIS | 28 LATRINE QUALITY The state of knowledge and experience with CLTS has been well documented in a comprehensive review that USAID commissioned in 2018.110 Readers are highly encouraged to review this resource for more in-depth information about the impacts and sustainability of CLTS. Ex-post findings that latrine quality generally remained poor and deteriorated after the interventions ended align with a consensus view, noted in this report, that CLTS is not designed or well-suited to aid households in progressing up the sanitation ladder. Rather, the goal of CLTS is to achieve rapid and sustained elimination of OD. In qualitative interviews, community members in Senegal and Mozambique attributed the lack of latrine sustainability to poor quality; the Madagascar study came to the same conclusion. Respondents noted frustration with the constant process of rebuilding poor-quality latrines that did not last, and some gave up on rebuilding latrines altogether. In some places, physical and environmental constraints, such as poor soil quality, flooding, high-frequency weather events, and a scarcity of wood or other local materials, exacerbated the issue, contributing to frequent latrine collapse or an inability to rebuild. Furthermore, most households did not consider pit evacuation to be feasible or available. For example, in Madagascar, only 3 percent of households evacuated their pit after it filled. By and large, households had to reconstruct new latrines each time the pit filled. The primarily financial barrier to this behavior is discussed in the section below, including variations across subsidy models in Senegal. Poor quality or poorly maintained latrines—especially those that lack privacy—are known to have a negative impact on use, potentially reducing health impacts.111 Several robust studies have observed that poor quality latrines or low community levels of latrine usage do not confer health benefits associated with sanitation improvements. Under such conditions, substantial environmental fecal exposure may still occur.112 This puts CLTS, at least as it has historically been implemented, in question as a standalone strategy, given its failure to prompt adoption of at least basic sanitation (by design) and its inability to sustain widespread ODF behavior in many contexts, including those examined in this ex-post series. If health benefits from unimproved and non-ubiquitous latrines are indeed tenuous, sanitation programming must drive harder toward the goal of widespread basic sanitation, whether through a modified CLTS modality, another approach, or a combination. HOUSEHOLD FINANCE In all the rural ex-post evaluations, financial or material constraints served as the primary barriers to latrine construction, maintenance, and reconstruction, as found in another CLTS review.113 Poorer and more vulnerable households typically owned fewer latrines. The sustained use of microfinance and VSLAs in Madagascar are discussed in the Reduced Financial Barriers for the Poor section. Several qualitative interview respondents in Madagascar, Mozambique, and Senegal, including local government officials in the latter two countries, pointed to the need for financial subsidy support to construct higher quality latrines. Among all evaluation sites with household sanitation interventions (CLTS/PHAST), the ET identified the ability to afford cement as a critical difference between households that had a latrine that lasted and households that did not. A member of a Senegal community that participated in a traditional CLTS intervention noted, “[PEPAM/USAID] should have supported us financially so that we could build modern toilets. Because the ones we build with our own means don’t last and we have to dig every year.” In an urban Ghana setting, an ET observed recurrent costs to be a key barrier to sustained latrine use four years after a USAID intervention supported construction of latrines. Shared latrines in particular required frequent fecal sludge removal, a process that became more costly as the pits filled.114 The intervention in Senegal allowed a comparison of subsidized and nonsubsidized CLTS approaches and found a tradeoff between quality and use. Communities that received the traditional, nonsubsidized CLTS intervention had the highest evidence of use but the lowest quality latrines four years later, while the subsidized sanitation marketing and hybrid CLTS-plus-subsidy approach had lower evidence of use, but better-quality latrines that more frequently met basic sanitation service standards (Figure 11). 29 | EX-POST EVALUATION SYNTHESIS USAID.GOV Figure 11. Key Sanitation Outcomes in Senegal, by Intervention Approach Ultimately, given aspirations to move households up the sanitation ladder to increase the likelihood of health benefits, the ET and a variety of stakeholders concluded the Hybrid CLTS -plus -subsidy approach in Senegal struck the best balance in improving access to basic sanitation service and establishing norms. Households that received this e Hybrid approach also attended to repair issues more complete repairs more frequently. While a subsidy does not help fund ongoing repairs or reconstruction years later, it is possible that constructing a higher quality latrine at the outset might bolster other behavioral drivers, such as prestige, privacy, and convenience, that reinforce habits and norms. The finding that Hybrid recipients that received a subsidy and together with CLTS triggering more commonly repaired their latrine and more commonly had an improved latrine four years later provides some evidence that subsidies can provide long-lasting effects. Subsidies have traditionally been considered antithetical to the founding principles of CLTS, and some studies have found decreased latrine adoption where prior subsidies have been offered.115 However, USAID’s recent review concluded that the disconnect between consumer demand and price must be addressed before households in Africa can progress up the sanitation ladder and that targeted output￾based subsidies might be the most viable approach.116 PROFESSIONAL CONSTRUCTION SUPPORT Respondents in Senegal and Madagascar cited a limited ability to access help or construction expertise as a barrier to initially building, repairing, or reconstructing latrines. This can be of particular concern for the elderly, widows, and single women, or other vulnerable households who are unable to dig a pit. According to the ex-post findings in Madagascar, Mozambique, and Senegal, training local masons to properly construct latrines, including improved models, can create sustainable businesses—although the intervention did not appear to substantively affect the prevalence of quality latrines observed at the time of the evaluation. The ET found trained masons functioning at all ex-post sites, and some qualitative respondents touted their usefulness. In Mozambique, 31 percent of latrine owners had used a skilled mason for construction. In Madagascar, 45 percent paid for skilled labor to construct their latrines and 35 percent paid for skilled labor to improve latrines since the activity ended. However, the study did not specify whether this referred specifically to the activity-trained masons. In Senegal, interviewed masons reported their services to be in high demand, though only 1 percent of surveyed households reported hiring skilled labor to make a repair. This discrepancy suggests that masons may have been solicited more for construction than repairs, though it is possible the community did not consider their repair rates to be affordable. According to surveys and qualitative interviews alike, most people found that the quality materials needed to construct improved latrines with concrete slabs to be either unaffordable or unavailable. Having skilled support without proper materials, or ability to pay, is of little use in moving up the sanitation ladder. A 2018 literature review identified adequate local supply chains to be a critical USAID.GOV EX-POST EVALUATION SYNTHESIS | 30 component to the success of mason training interventions.117 The review also identified common challenges with masons being unable to expand or sustain their businesses, often due to constrained local market demand. Overall, while the ex-post series found efforts to equip a local skilled workforce can facilitate ongoing adoption of higher quality improved latrines, issues of supplies, finance, and market demand must be addressed to achieve substantial results. PSYCHOSOCIAL DRIVERS OF BEHAVIOR As with hygiene, drivers of sanitation behavior are multidimensional, and sectoral approaches have addressed a variety of psychological determinants. The PHAST or SARAR strategies used in Senegal, Ethiopia, Mozambique, and the CLTS strategy used in Senegal, Ethiopia, Mozambique, and Madagascar share a common participatory method that targets health knowledge and preventative behaviors. CLTS goes further to invoke other behavioral drivers like disgust, dignity, pride, and shame.118 Qualitative and quantitative findings indicated that households across evaluation sites understood the importance of sanitation but did not necessarily internalize that knowledge to change habits. The sanitation findings above clearly indicate that OD continues to occur at high rates in some locations, like Madagascar and Mozambique. Findings appeared more mixed in Senegal, depending on what type of sanitation intervention took place. The consistent use of latrines did not appear to be normative for the majority of intervention communities, signaling that other drivers of behavior have not been adequately addressed. CLTS is designed to trigger various psychosocial drivers and prompt quick collective adoption of community norms to stop OD. The strength of the norms that are developed are a critical factor in sustainability. In Madagascar, the region with the highest level of slippage in latrine use also had the lowest pre-intervention baseline latrine coverage. Conversely, the lowest level of slippage occurred in a region that had substantially higher baseline levels of latrine coverage. This finding might reflect an acceptability and normalcy of latrine use that laid the groundwork for greater sustainability in the long term. Twenty-eight percent of those without a latrine at ex-post noted latrines were “not common” in their communities. The persistence of OD in most settings bolsters current sector views that PHAST, SARAR, and CLTS approaches are not sufficient to drive sustained, widespread normative behavior change. ONGOING LOCAL FOLLOW-UP Studies consistently point to follow-up as the key solution to mitigate slippage in latrine use and reach behavior change “maturity.”119 As USAID’s CLTS review notes, follow-up is only a sustainable solution if the host government or other local parties take it on and fully absorb the costs. The ex-post series did not examine the agents, frequency, or nature of CLTS follow-up done. However, the significant latrine use slippage in Madagascar suggests that ongoing WASH BCC and safe sanitation promotion, to the extent it was done, did not improve uptake. It is unclear to what extent increases in latrine coverage at ex-post in Mozambique could be attributed to the ongoing behavior change promotion work of community health committees and activists as compared to other factors. One assessment across three countries noted governments are more effective at follow-up than village volunteers, but lack of funding and incentives are a problem.120 Qualitative interviews in Ethiopia confirmed that health extension worker activities faced funding challenges. Implementers could seek ways to better support governance systems for sanitation oversight and behavior change; however, such efforts are not likely to substantially impact long-term sanitation behavior if the underlying drivers of normative and habitual change and other aspects of the enabling environment are not addressed. GENDERED DECISION-MAKING Another contextual factor known to affect WASH behavior is the way household roles influence behavior decision-making.121 While respondents in Mozambique and Senegal noted the challenges 31 | EX-POST EVALUATION SYNTHESIS USAID.GOV widows and single women faced in digging a latrine pit, the ET found a surprising trend in Madagascar, where female-headed households had lower rates of slippage in latrine use and handwashing stations over time when compared to male-headed households. Female-headed households also opted for higher quality latrines over time by a small margin (Figure 12). This might reflect differing priorities when women are the locus of household decision-making. One study found a positive association between women’s decision-making power for significant household purchases and better household sanitation.122 It is possible that women place greater value on the privacy, safety, convenience, health benefits, and cleanliness of a latrine.123 It is also possible that having access to trained skilled masons in this setting lowered barriers to latrine use for these women. While the reasons for the Madagascar findings are not certain, they point to a need for implementers to understand the potentially different behavioral drivers of male and female decision-makers and tailor SBC strategies accordingly. Figure 12. Madagascar Ex-post WASH Outcomes (observed): Percent Change from Endline by Gender of Household Head USAID.GOV EX-POST EVALUATION SYNTHESIS | 32 CONCLUSION This report presents the findings from six ex-post evaluations and highlights the factors that may have impacted outcomes and ultimately sustainability. The ET observed poor sustainability at USAID– established rural water points, with functionality ranging from 44 percent to 65 percent at ex-post. The evaluations found that rural households did not commonly practice handwashing with soap, and open defecation continued to be widespread in CLTS–triggered communities. In contrast, most Indonesian utilities supported through USAID technical assistance increased water access in the years since the activity ended. Water and sanitation access varied widely across Indian municipalities evaluated. The ET identified a variety of factors that supported or impaired long-term sustainability. Sustainable finance for WASH is critical to ensure that high quality services can be delivered, and assets maintained. Sustainable finance for WASH also facilitates expansion of access to the poorest segments of the population, who face numerous barriers, including cost, to access adequate WASH services. Findings from the ex-post series demonstrate the potential promise of various approaches to improve financial management, cost recovery, business planning, and to undertake credit enhancements, while also highlighting the ultimate importance of the enabling environment. Activities that involved the introduction of new tools, methods, or financing mechanisms showed mixed results in the longer term, often because other factors related to service providers’ operating environment or demand-side barriers had not changed. This includes findings related to credit enhancements in India and microfinance in Indonesia and Madagascar. Findings from the series highlight the importance of considering the total enabling and operating environment when planning and implementing project-based interventions to improve WASH financing. In tandem with financing, water and sanitation governance provides the broader enabling environment for sustainable service delivery. National government commitment to improving and expanding water and sanitation service delivery is a critical foundation for a strong governance framework. USAID implementation approaches that provide rollout capacity support for national governance initiatives show significant potential to sustain results, as the national scaling of FIRE-D–supported initiatives demonstrated. Lack of clarity on roles and responsibilities emerged as a challenge, as well as resources and capacity to carry out those roles—both issues that could be improved through stronger governance frameworks and implementation thereof. The major challenges associated with inadequate cost recovery highlight the need for stronger and more consistent governance around tariff setting to ensure rates are cost-reflective and based on actual rather than assumed willingness to pay. Also of note, although none of the activities in the ex-post series focused on land tenure and water infrastructure interventions, issues arose in both India and Ethiopia that indicated lack of attention to land ownership in water service plans or management could impede access and sustainability. Effective management of water services in rural and urban settings plays a pivotal role in sector efforts to reach long-term sustainability, but remains a significant challenge. The rural ex-post evaluations indicated that CBM entities failed in managing water services in the long term. In these rural settings, management of assets and consistent monitoring of water systems and water quality were pervasive challenges. Urban utilities fared better, though capacity challenges remained. While many point to the private sector as a solution to this challenge, this series found that PPP-managed water systems failed in some cases and remained active in others, although not always operating as envisioned. Successful PPPs require strong relationships between stakeholders and a strong enabling environment for managing contracts between users and private sector service providers. In rural and urban systems alike, the ET found that cost recovery constrained sustainability, expansion, and improvement of water service delivery. Inefficient collection practices, customer ability to pay, quality of service, and availability of alternative water sources all contributed to inadequate fee or tariff recovery (e-governance initiatives in select India sites improved this somewhat). 33 | EX-POST EVALUATION SYNTHESIS USAID.GOV The low prevalence of both hygiene and sanitation behavior indicators across all rural sites and persistence of open defecation indicate the primarily PHAST– and CLTS–based implementation approaches did not achieve sustained, widespread normative behavior change. For handwashing, traditional BCC strategies like PHAST that focus on knowledge as a motivator (and the engagement of local people to continue to deliver PHAST messages) did not sufficiently move households from knowledge to action. In addition, the long-term failure of tippy taps highlights the importance of identifying more durable fixed hardware that is appealing for users. For sanitation, beyond limited sustainability of OD elimination—the primary focus of CLTS—the persistence of poor-quality latrines likely jeopardized health benefits. Financial barriers to latrine quality maintenance and use persist; however, targeted subsidies combined with CLTS in Senegal provided modest improvements to sustainability. Local masons that USAID–funded activities trained to construct improved latrines remained active and valued, indicating a sustainable approach to quality latrine adoption; however, financial barriers limited widespread use. Better sustainability patterns for female-headed households in Madagascar highlighted the need for implementers to consider the potential influence of gendered household decision-making around handwashing and sanitation and design activities that best address unique SBC drivers for males and females alike. Ultimately, ex-post findings supported other WASH literature in concluding that, to achieve normative behavior change, social and behavior change strategies must go beyond knowledge or traditional implementation of CLTS to address multidimensional drivers and structural barriers. Overall, the ex-post evaluation series illuminates challenges and successes and provides evidence within specific contexts on whether and how particular approaches achieved sustained outcomes. While this report addresses specific activities and contexts, the findings should resonate with governments, donors, and practitioners and reinforce shifts already underway within the WASH sector. It is envisioned that the ex-post series findings, within the context of the broader literature, will provide additional insight and evidence for USAID and the WASH sector to further collective goals of improving sustainable WASH service delivery. USAID.GOV EX-POST EVALUATION SYNTHESIS | 34 ENDNOTES 1 USAID Water Strategy. https://www.usaid.gov/sites/default/files/documents/1865/USAID_Water_Strategy_3.pdf. 2 Schweitzer, R., C. Grayson, and H. Lockwood. 2014. Mapping of Water, Sanitation and Hygiene Sustainability Tools. Triple-S Working Paper. https://www.ircwash.org/resources/mapping-water-sanitation-and-hygiene￾sustainability-tools. 3 Sustainable WASH Systems (SWS) Learning Partnership at GlobalWaters.org. https://www.globalwaters.org/sws. 4 USAID Performance Plan and Report (PPR) data pulled 8/25/2020. 5 USAID Dollars to Results: https://results.usaid.gov/results. 6 USAID PPR. 7 For the latest guidance on preferred Agency approaches to address WASH challenges, please see https://www.globalwaters.org/water-and-development-technical-series. 8 According to the United Nations Joint Monitoring Programme, basic sanitation is use of non-shared, improved facilities (i.e. flush/pour flush to piped sewer system, septic tanks or pit latrines, ventilated improved pit latrines, composting toilets or pit latrines with slabs). Basic handwashing facility is the availability of either a fixed or mobile handwashing facility on the premises with soap and water. UNICEF and WHO. 2019. Progress on Household Drinking Water, Sanitation and Hygiene 2000–2017. Special Focus on Inequalities. New York: UNICEF and WHO. https://www.who.int/water_sanitation_health/publications/jmp-report-2019/en/. 9 UN. 2018. Clean Water and Sanitation: Why It Matters. https://www.un.org/sustainabledevelopment/wp￾content/uploads/2016/08/6.pdf.; WHO. 2012. Global Costs and Benefits of Drinking-Water Supply and Sanitation Interventions to Reach the MDG Target and Universal Coverage. Geneva: WHO https://www.who.int/water_sanitation_health/publications/2012/globalcosts.pdf. 10 Banks, B. and S. Furey. 2016. “What’s Working, Where, and for How Long: A 2016 Water Point Update.” Poster session presented at the 7th Rural Water Supply Network Forum, Abidjan, Cote d’Ivoire. https://rwsnforum7.files.wordpress.com/2016/11/full_paper_0150_submitter_0239_banks_brian.pdf.; Tyndale￾Biscoe, P., M. Bond, and R. Kidd. 2013. ODF Sustainability Study. FH Designs for DFID and DFAT. http://www.communityledtotalsanitation.org/sites/communityledtotalsanitation.org/files/Plan_International_ODF_S ustainability_Study.pdf. 11 U.S. Government. 2017. Global Water Strategy. https://www.usaid.gov/sites/default/files/documents/1865/Global_Water_Strategy_2017_final_508v2.pdf.; Runde, D. and C. Metzger. 2019. Channeling Financial Flows for Urban Water and Sanitation. Center for Strategic and International Studies. https://www.csis.org/analysis/channeling-financial-flows-urban-water-and-sanitation.; Trémolet, S., P. Börkey, C. Kauffmann, and A. Martoussevitch. 2011. Meeting the Challenge of Financing Water and Sanitation. OECD. https://www.pseau.org/outils/ouvrages/ocde_meeting_the_challenge_of_financing_water_and_sanitation_2013.pdf. 12 Lockwood, H. 2019. Sustaining Rural Water: A Comparative Study of Maintenance Models for Community￾Managed Schemes. USAID. https://www.ircwash.org/sites/default/files/sustaining_rural_water_- _a_comparative_study_of_maintenance_models_for_community-managed_schemes.pdf; World Bank. 2017. Sustainability Assessment of Rural Water Service Delivery Models: Findings of a Multi-Country Review. Washington DC: World Bank. 13 Null et al. May 2012. “Willingness to pay for cleaner water in less developed countries: systematic review of experimental evidence.” International Initiative for Impact Evaluation Systematic Review 006. https://www.3ieimpact.org/sites/default/files/2017-11/sr006.pdf 14 Fuente et al. June 2016. “Water and sanitation service delivery, pricing, and the poor: An empirical estimate of subsidy incidence in Nairobi, Kenya.” Water Resources Research, 52, 4845-4862. https://doi.org/10.1002/2015WR018375. 15 Boland, J. and D. Whittington. 2000. “Water tariff design in developing countries: Disadvantages of increasing block tariffs (IBTs) and advantages of uniform price with rebate (UPR) designs.” 16 Whaley, L., and F. Cleaver. June 2017. “Can ‘functionality’ save the community management model of rural water supply?” Water Resources and Rural Development 9: 56-66. https://doi.org/10.1016/j.wrr.2017.04.001. 17 World Bank. 2017. 35 | EX-POST EVALUATION SYNTHESIS USAID.GOV 18 Fisher, J., and S. Kayaga. January 2006. “Improving water utility management and performance in developing countries.” Water Utility Management International 1, no. 4: 6-8. 19 Marin, Philippe. 2009. Public-Private Partnerships for Urban Water Utilities: A Review of Experiences in Developing Countries. Trends and Policy Options: no. 8. World Bank. https://openknowledge.worldbank.org/handle/10986/2703. 20 White, S., A. H. Thorseth, R. Dreibelbis, and V. Curtis. June 2020. “The determinants of handwashing behaviour in domestic settings: An integrative systematic review.” International Journal of Hygiene and Environmental Health 227. https://doi.org/10.1016/j.ijheh.2020.113512. 21 Albert et al. April 2018. An Examination of CLTS’s Contributions toward Universal Sanitation. USAID. https://www.issuelab.org/resources/30547/30547.pdf?download=true. 22 Albert et al. 2018.; USAID. May 2020. Rural Sanitation. USAID Water and Development Technical Series, Technical Brief 2. https://www.globalwaters.org/sites/default/files/usaid_rural_sanitation_tech_brief_2.pdf. 23 According to the United Nations Joint Monitoring Programme, basic water service is that which comes from an improved source (i.e. piped water, boreholes, tubewells, protected dug wells, protected springs, rainwater, or packaged/delivered water) that requires less than 30 minutes round-trip to collect. Basic sanitation is use of non￾shared, improved facilities (i.e. flush/pour flush to piped sewer system, septic tanks or pit latrines, ventilated improved pit latrines, composting toilets or pit latrines with slabs). Basic handwashing facility is the availability of either a fixed or mobile handwashing facility on the premises with soap and water.; UNICEF and WHO. 2019. Progress on Household Drinking Water, Sanitation and Hygiene 2000-2017. Special focus on inequalities. New York: UNICEF and WHO. https://www.who.int/water_sanitation_health/publications/jmp-report-2019/en/. 24 UN. 2018. Clean Water and Sanitation: Why It Matters. https://www.un.org/sustainabledevelopment/wp￾content/uploads/2016/08/6.pdf 25 WHO. 2012. Global costs and benefits of drinking-water supply and sanitation interventions to reach the MDG target and universal coverage. Geneva: WHO. https://www.who.int/water_sanitation_health/publications/2012/globalcosts.pdf. 26 UNICEF and WHO. 2019. Progress on household drinking water, sanitation and hygiene 2000-2017. Special focus on inequalities. New York: UNICEF and WHO. https://www.who.int/water_sanitation_health/publications/jmp-report-2019/en/. 27 Banks, B. and S. Furey. 2016. “What’s Working, Where, and for How Long: A 2016 Water Point Update”. Poster session presented at the 7th Rural Water Supply Network Forum, Abidjan, Cote d’Ivoire. https://rwsnforum7.files.wordpress.com/2016/11/full_paper_0150_submitter_0239_banks_brian.pdf. 28 Tyndale-Biscoe, P., M. Bond, and R. Kidd. December 2013. ODF Sustainability Study. FH Designs for DFID and DFAT. http://www.communityledtotalsanitation.org/sites/communityledtotalsanitation.org/files/Plan_International_ODF_S ustainability_Study.pdf. 29 U.S. Government. 2017. Global Water Strategy. https://www.usaid.gov/sites/default/files/documents/1865/Global_Water_Strategy_2017_final_508v2.pdf. 30 All six ex-post evaluation reports as well as webinar recordings, briefs, and blog posts are available at https://www.globalwaters.org/resources/ExPostEvaluations. 31 Ethiopia data are not discussed here, as its purposive sample of 15 households thought to have constructed a latrine under MWA-EP was not designed to represent overall community conditions. 32 U.S. Government. 2017; Runde, D. and C. Metzger. November 2019. “Channeling Financial Flows for Urban Water and Sanitation.” Center for Strategic and International Studies. https://www.csis.org/analysis/channeling￾financial-flows-urban-water-and-sanitation; Trémolet, S. et al. October 2011. Meeting the Challenge of Financing Water and Sanitation. OECD Studies on Water. https://www.pseau.org/outils/ouvrages/ocde_meeting_the_challenge_of_financing_water_and_sanitation_2013.pdf. 33 Pories, L., C. Fonseca, and V. Delmon. March 2019. Mobilising finance for WASH: getting the foundations right. IRC, Water.org, and The World Bank. https://www.ircwash.org/resources/mobilising-finance-wash-getting￾foundations-right; World Bank Group and UNICEF. March 2017. Sanitation and Water for All: How Can the Financing Gap Be Filled? A Discussion Paper. Washington DC: World Bank. https://openknowledge.worldbank.org/bitstream/handle/10986/26458/114545-WP-P157523-PUBLIC-SWA￾Country-Preparatory-Process-Discussion-Paper-8-Mar-17.pdf?sequence=1. USAID.GOV EX-POST EVALUATION SYNTHESIS | 36 34 Pories et al. 2019.; Trémolet, S. et al. 2015. UN-Water GLASS TrackFin Initiative: Tracking financing to sanitation, hygiene and drinking-water at the national level. WHO. https://apps.who.int/iris/bitstream/handle/10665/204861/WHO_FWC_WSH_15.23_eng.pdf;jsessionid=94403BAAB 8FA1E1CA9FA98CC2FD8C4AD?sequence=1; World Bank Group and UNICEF. 2017. 35 World Bank Group and UNICEF. 2017. 36 By the time the ESP activity began, many utilities were struggling with significant debt due in part to interest and fines from decades-old multilateral development bank loans, and in part to poor management practices. 37 Pories et al. 2019. 38 UN Water and WHO. 2017. “Financing Universal Water, Sanitation and Hygiene under the Sustainable Development Goals: UN-Water Global Analysis and Assessment of Sanitation and Drinking-Water.” GLAAS 2017 Report. https://apps.who.int/iris/bitstream/handle/10665/254999/9789241512190- eng.pdf;jsessionid=8DDFCE14D9A05CCF91324852920695A0?sequence=1. 39 Nagpal, T., A. Malik, and M. Eldridge. September 2018. Mobilizing Additional Funds for Pro-Poor Water Services: An Exploration of Potential Models to Finance Safe Water Access in Support of Sustainable Development Goal 6.1. Center on International Development and Governance Research. https://www.oecd.org/water/Background￾Document-OECD-GIZ-Conference-Urban-Institute-GIZ-Innovative-Financing-Mechanisms-final.pdf. 40 Pories et al. 2019; UN Water and WHO. 2017; Leigland, J., S. Trémolet, and J. Ikeda. 2016. Achieving Universal Access to Water and Sanitation by 2030: The Role of Blended Finance. World Bank. http://documents.worldbank.org/curated/en/978521472029369304/Achieving-universal-access-to-water-and￾sanitation-by-2030-therole-of-blended-finance. 41 UN Water and WHO. 2017. 42 Fonseca, C. and L. Pories. April 2017. “Financing WASH: how to increase funds for the sector while reducing inequalities.” Position Paper for the Sanitation and Water for All Finance Ministers Meeting. https://www.ircwash.org/resources/financing-wash-how-increase-funds-sector-while-reducing-inequalities-position￾paper; Nagpal et al. 2018. 43 Davis, J., G. White, S. Damodaron, and R. Thorsten. 2018. “Improving Access to Water Supply and Sanitation in Urban India: Microfinance for Water and Sanitation Infrastructure Development.” Water Science and Technology. 58, no. 4: 887-91. https://pubmed.ncbi.nlm.nih.gov/18776626/; Afrane, S. and B. Adjei-Poku. 2014. “Expanding the Frontiers of Microfinance in the Service of the Poor: Experiment with Water and Sanitation.” International Journal of Academic Research in Business and Social Sciences 4, no. 6: 133-145. https://ideas.repec.org/a/hur/ijarbs/v4y2014i6p133-145.html. 44 Household latrine construction subsidies are discussed in the Sustaining Sanitation and Handwashing Demand and Behavior Change section. 45 U.S. Government. 2017. 46 Water resource management was a key component of the Indonesia ESP activity; however, this component was not within the scope of the ex-post evaluation. 47 The GoI first adopted several FIRE-D reforms into its Jawaharlal Nehru National Urban Renewal Mission. This was subsequently overhauled and relaunched as the Atal Mission for Rejuvenation and Urban Transformation, which adopted many of the same reform incentives as JNNURM and was one of the most influential funders of urban water and sanitation infrastructure in India at the time of the ex-post evaluation. 48 UNICEF and World Health Organization 2019. 49 Lockwood. 2019; World Bank. 2017. 50 World Bank. 2017. 51 Olaerts et al. July 2019. “Factors Influencing Revenue Collection for Preventative Maintenance of Community Water Systems: A Fuzzy-Set Qualitative Comparative Analysis.” Sustainability 11, no. 13. https://doi.org/10.3390/su11133726. 52 Pories et al. 2019; Leigland et al. 2016; UN Water and WHO. 2017. 53 This report distinguishes urban and rural water services. Small towns fall between these two extremes but are often grouped together with urban water services in that small town water services are more often managed by utilities or utility-like structures (like water boards). There is emerging literature on the specific dynamics in small communities (see Tutusausa, M. and K. Schwartz. 2018. "Water services in small towns in developing countries: at 37 | EX-POST EVALUATION SYNTHESIS USAID.GOV the tail end of development." Water Policy 20:1–11., for example), but these nuances were not captured in the ex￾post series and thus are not a focus of this report.; World Bank. 2017. 54 World Bank. 2017. 55 Pories et al. 2019. 56 Banerjee et al. 2010. Cost Recovery, Equity, and Efficiency in Water Tariffs: Evidence from African Utilities. World Bank Policy Research working paper no. WPS 5384. https://openknowledge.worldbank.org/handle/10986/3868. 57 Null et al. May 2012. 58 Fuente et al. June 2016. “Water and sanitation service delivery, pricing, and the poor: An empirical estimate of subsidy incidence in Nairobi, Kenya.” Water Resources Research, 52, 4845-4862. https://doi.org/10.1002/2015WR018375. 59 Boland, J. and D. Whittington. 2000. “Water tariff design in developing countries: Disadvantages of increasing block tariffs (IBTs) and advantages of uniform price with rebate (UPR) designs.” 60 Collingnon, B. and R. Schmid. Jan 2008. “Price of water- Working paper on water costs, tariffs, and subsidies.” DOI: 10.13140/RG.2.2.29493.73441. 61 Kayaga et al. February 2019. “Towards sustainable urban water services in developing countries: tariffs based on willingness-to-pay studies.” Urban Water Journal 15, no. 10: 974-984. https://doi.org/10.1080/1573062X.2019.1581234. 62 Meeks, R. 2018. “Property Rights and Water Access: Evidence from Land Titling in Rural Peru.” World Development 102:345-57. https://www.sciencedirect.com/science/article/abs/pii/S0305750X16301723 63 Based on the available data, one of the primary drivers of the varied functionality rates between Ethiopia and Senegal/Mozambique is likely the relative age of the infrastructure. Eight years had passed after the end of MWA￾EP when the ex-post was conducted while only around 4 years had passed in the PEPAM/USAID and SCIP cases. 64 Lockwood. 2019. 65 Fisher, J., and S. Kayaga. January 2006. “Improving water utility management and performance in developing countries.” Water Utility Management International 1, no. 4: 6-8. 66 World Bank. 2017. 67 Marin. 2009. 68 Villanova University College of Engineering. 2017. Sustainable WASH Research Initiative Madagascar Project Report. https://www.globalwaters.org/sites/default/files/Sustainable-WASH-Madagascar-Jordan-Emilio.pdf 69World Bank. 2017. 70 Kruger et al. June 2019. “How can Alternative Service Delivery improve water services?” African EU Water Partnership Project. https://www.siwi.org/wp-content/uploads/2019/07/AEWPP_PPP_2019_WEBB-FINAL￾11072019.pdf. 71 Lockwood. 2019; Olaerts et al. 2019. 72 World Bank. 2017. 73 Carter, R.C.; Harvey, E.; Casey, V. 2010. “User financing of rural handpump water services.” In Proceedings of the IRC Symposium 2010: Pumps, Pipes and Promises, Hague, The Netherlands,; Foster, T. 2013. “Predictors of Sustainability for Community-Managed Handpumps in Sub-Saharan Africa: Evidence from Liberia, Sierra Leone, and Uganda.” Environ. Sci. Technol. 2013, 47, 12037–12046. 74 Marin. 2009. 75 Lockwood. 2019. 76 World Bank. 2017. 77 Ibid. 78 World Bank. 2017. 79 Ibid. 80 Project documents did not contain detailed information about the exact scope of the e-governance activities, nor about which components were implemented where, though it was clear that different utilities implemented different sets of e-governance components. USAID.GOV EX-POST EVALUATION SYNTHESIS | 38 81 Harvey, P. February 2011. “Sustainable Supply Chains for Rural Water Services: Linking local procurement of handpumps and spare parts supply.” Rural Water Supply Network. https://www.ircwash.org/sites/default/files/Harvey-2011-Sustainable.pdf. 82 This review focuses on community hygiene and handwashing practices and does not address school, hospital, or other institutional settings.; Cairncross et al. 2010. “Water, Sanitation and Hygiene for the Prevention of Diarrhoea.” International Journal of Epidemiology 39, Suppl. I: 193-205. https://pubmed.ncbi.nlm.nih.gov/20348121/; Freeman et al. August 2014. “Systematic review: Hygiene and health: systematic review of handwashing practices worldwide and update of health effects.” Tropical Medicine & International Health 19, no. 8: 906-916. https://onlinelibrary.wiley.com/doi/full/10.1111/tmi.12339; Wolf et al. May 2018. “Impact of Drinking Water, Sanitation and Handwashing With Soap on Childhood Diarrhoeal Disease: Updated Meta-Analysis and Meta￾Regression.” Tropical Medicine & International Health 23, no. 5: 508-524. https://pubmed.ncbi.nlm.nih.gov/29537671/; Rabie, T., and V. Curtis. March 2006. “Handwashing and Risk of Respiratory Infections: A Quantitative Systematic Review.” Tropical Medicine & International Health 11, no. 3: 258-267. https://pubmed.ncbi.nlm.nih.gov/16553905/; Aiello et al. August 2008. “Effect of Hand Hygiene on Infectious Disease Risk in the Community Setting: A Meta￾Analysis.” American Journal of Public Health 98, no. 8: 1372-81. https://pubmed.ncbi.nlm.nih.gov/18556606/.; Strunz et al. March 2014. “Water, Sanitation, Hygiene, and Soil-Transmitted Helminth Infection: A Systematic Review and Meta-Analysis.” PLoS Medicine 11, no. 3. https://pubmed.ncbi.nlm.nih.gov/24667810/; Pickering et al. June 2019. “Effects of single and integrated water, sanitation, handwashing, and nutrition interventions on child soil￾transmitted helminth and Giardia infections: A cluster-randomized controlled trial in rural Kenya.” PLoS Medicine 16, no. 6 : e1002841. https://doi.org/10.1371/journal.pmed.1002841. 83 UNICEF, WHO and USAID. 2015. Improving Nutrition Outcomes with Better Water, Sanitation and Hygiene: Practical Solutions for Policies and Programmes. https://www.unicef.org/media/files/IntegratingWASHandNut_WHO_UNICEF_USAID_Nov2015.pdf. 84 White et al. June 2020. “The determinants of handwashing behaviour in domestic settings: An integrative systematic review.” International Journal of Hygiene and Environmental Health 227: 113512. https://doi.org/10.1016/j.ijheh.2020.113512. 85 “Behavior change agents” is a generic term, and each activity evaluated had specific names for ongoing local WASH behavior change promoters (e.g., Community Health Workers in Madagascar, Community Health Committees and WASH Activists in Mozambique, local leaders in Senegal, and Health Extension Workers in Ethiopia). 86 CLTS is a participatory method that mobilizes communities to eliminate open defecation. Traditionally, it focuses on ending open defecation and does not promote specific hardware. It also includes hygiene and water contamination awareness centered around motivators such as disgust. 87 A tippy tap is a simple handwashing device comprised of a plastic jug of water suspended by a rope, often with a foot-operated lever to dispense water. 88 UNICEF and WHO. 2019. 89 Vindigni, S. M., P. L. Riley, and M. Jhung. April 2011. “Systematic review: handwashing behaviour in low‐ to middle‐income countries: outcome measures and behaviour maintenance.” Tropical Medicine & International Health 16, no. 4: 466-477. https://doi.org/10.1111/j.1365-3156.2010.02720.x. 90 USAID. 2018. Ex-Post Evaluation of the Water Access, Sanitation, and Hygiene for Urban Poor Activity in Ghana. https://www.globalwaters.org/sites/default/files/ghana-wash-up-expost-evaluation.pdf. 91 White et al. 2020. 92 De Buck et al. June 2017. Promoting handwashing and sanitation behaviour change in low- and middle-income countries: a mixed-method systematic review. International Initiative for Impact Evaluation. https://www.3ieimpact.org/sites/default/files/2019-01/SR%2036-Behaviour-change-sanitation_2.pdf. 93 Mosler, H.-J. 2012. “A systematic approach to behavior change interventions for the water and sanitation sector in developing countries: A conceptual model, a review, and a guideline.” International Journal Of Environmental Health Research 22, no. 5: 431-49. https://doi.org/10.1080/09603123.2011.650156. 94 Ethiopia data are not referenced in this section, due to small sample size of 11 qualitative interviews. Other evaluations had larger, more representative quantitative samples to assess HWWS. 95 Mosler, H.-J. Fishbein, M., and I. Ajzen. 2011. Predicting and Changing Behavior: The Reasoned Action Approach. New York: Psychology Press (Taylor & Francis). https://doi.org/10.4324/9780203838020. 39 | EX-POST EVALUATION SYNTHESIS USAID.GOV 96 Safari et al. December 2019. “Lessons learned from the national sanitation campaign in Njombe district, Tanzania.” Journal of Water, Sanitation and Hygiene for Development 9, no. 4: 754–764. https://doi.org/10.2166/washdev.2019.274. 97 White et al. 2020. 98 Human-centered design starts with extensive user preference inquiry to develop designs that best address needs, followed by testing prototypes and ensuring market viability. See examples of WASH applications at https://www.globalwaters.org/resources/assets/water-currents-wash-human-centered-design. 99 Pickering, A., and J. Davis. 2012. “Fresh Water Availability and Water Fetching Distance Affect Child Health in Sub-Saharan Africa”. Environmental Science & Technology. 46.: 2391-7. 100 Bhattacharyya, K., P. Winch, LeBan, K. and M. Tien. October 2001. Community Health Worker Incentives and Disincentives: How They Affect Motivation, Retention and Sustainability. USAID. https://chwcentral.org/resources/community-health-worker-incentives-and-disincentives-how-they-affect￾motivation-retention-and-sustainability/. 101 Hoque, S. F., and R. Hope. “The water diary method – proof-of-concept and policy implications for monitoring water use behaviour in rural Kenya.” Water Policy 20, no. 4 (April 2018): 725–743. https://doi.org/10.2166/wp.2018.179. 102 Koita, S. N. June 2010. Involving Men in Handwashing Behavior Change Interventions in Senegal. Water and Sanitation Program. https://www.wsp.org/sites/wsp.org/files/publications/WSP_InvolvingMen_HWWS.pdf. 103 Ibid. 104 Grover et al. January 2018. “Comparing the behavioural impact of a nudge-basedhandwashing intervention to high-intensity hygiene education: a cluster-randomised trial in rural Bangladesh.” Tropical Medicine & International Health 23, no. 1: 10-24. https://onlinelibrary.wiley.com/doi/epdf/10.1111/tmi.12999. 105 Putnam, W. and N. Nguyen. October 2019. Are Nudges the Answer to Improving Handwashing Behavior? USAID. https://www.globalwaters.org/resources/blogs/are-nudges-answer-improving-handwashing￾behavior?utm_source=twitter&utm_medium=social&utm_campaign=gw-blogs. 106 Bhattacharyya, K., P. Winch, LeBan, K. and M. Tien. October 2001. Community Health Worker Incentives and Disincentives: How They Affect Motivation, Retention and Sustainability. USAID. https://chwcentral.org/resources/community-health-worker-incentives-and-disincentives-how-they-affect￾motivation-retention-and-sustainability/. 107 Dreibelbis et al. 2013. “The Integrated Behavioural Model for Water, Sanitation, and Hygiene: a systematic review of behavioural models and a framework for designing and evaluating behaviour change interventions in infrastructure-restricted settings.” BMC Public Health 13: 1015. https://doi.org/10.1186/1471-2458-13-1015. 108 Details about intervention modalities in Ethiopia were lacking. Some project reports discussed demonstration latrines while others simply noted sanitation promotion and PHAST. This missing detail as well as limitations in the non-representative, small selection of households visited at ex-post prohibits extensive discussion of Ethiopia results in this section. 109 According to JMP, improved sanitation refers to facilities designed to hygienically separate excreta from human contact (e.g., flush/pour flush to piped sewer system, septic tank, or pit latrines; ventilated improved pit latrines, composting toilets, or pit latrines with slabs). Those with improved sanitation who also do not share with other households are considered to have “basic sanitation” access (JMP 2018). 110 Albert et al. An Examination of CLTS’s Contributions toward Universal Sanitation.” Tetra Tech for USAID. April 2018. https://www.issuelab.org/resources/30547/30547.pdf?download=true. 111 Garn et al. April 2017. “The impact of sanitation interventions on latrine coverage and latrine use: A systematic review and meta-analysis.” International Journal of Hygiene and Environmental Health 220, no. 2 part B: 329-340. https://doi.org/10.1016/j.ijheh.2016.10.001; Venkataramanan, V., et al. February 2018. “Community-Led Total Sanitation: A Mixed-Methods Systematic Review of Evidence and Its Quality." Environmental Health Perspectives 126, no. 2. https://ehp.niehs.nih.gov/doi/10.1289/EHP1965. 112 Patil et al. August 2014. “The Effect of India's Total Sanitation Campaign on Defecation Behaviors and Child Health in Rural Madhya Pradesh: A Cluster Randomized Controlled Trial.” PloS Medicine. https://doi.org/10.1371/journal.pmed.1001709; Clasen, T. et al. November 2014. “Effectiveness of a rural sanitation programme on diarrhoea, soil-transmitted helminth infection, and child malnutrition in Odisha, India: a cluster- USAID.GOV EX-POST EVALUATION SYNTHESIS | 40 randomised trial.” The Lancet Global Health 2, no. 11: e645–53. https://pubmed.ncbi.nlm.nih.gov/25442689/; Pickering, A. J., H. Djebbari, C. Lopez, M. Coulibaly, and M. L. Alzua. November 2015. “Effect of a community-led sanitation intervention on child diarrhoea and child growth in rural Mali: a cluster-randomised controlled trial.” The Lancet Global Health 3, no. 11: e701–11. https://doi.org/10.1016/S2214-109X(15)00144-8.; Wolf et al. 2018.; Jung, Y. T., R. J. Hum, W. Lou, and Y. Cheng. March 2017. "Effects of neighbourhood and household sanitation conditions on diarrhea morbidity: systematic review and meta-analysis.” PLoS One 12, no. 3: e0173808. https://pubmed.ncbi.nlm.nih.gov/28296946/. 113Venkataramanan et al. 2018. 114 Venkataramanan et al. 2018. 115 Garn et al. 2017. 116 Albert et al. 2018. 117 USAID. 2018. Scaling Market Based Sanitation: Desk review on market-based rural sanitation development programs. Washington, DC., USAID Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) Project. https://www.globalwaters.org/sites/default/files/Scaling%20Market%20Based%20Sanitation%20JUNE2018.pdf 118 Venkataramanan et al. 2018. 119 Albert et al. 2018; Jerneck, M., C. Voorden, and C. Rudholm. 2016. Sanitation and Hygiene Behaviour Change at Scale: Understanding Slippage. Water Supply and Sanitation Collaborative Council. https://www.mcd.org/docs/sheets/Sanitation-and-Hygiene-Behaviour-Change-at-Scale-Understanding-Slippage￾Reflections-Paper-GSF.pdf. 120 Crocker, J., and J. Bartram. 2015. Creating Better, Evidence-Based Policies and Plans for Sanitation Interventions: Policy Findings from the Testing CLTS Approaches for Scalability Program. Plan International and UNC Water Institute. https://waterinstitute.unc.edu/files/2015/02/Creating-Better-Evidence-based-Policies-and￾Plans-for-Sanitation-Interventions-Policy-Findings-20160925.pdf. 121 Dreibelbis et al. 2013. 122 Hirai, M., J. P. Graham, J. Sandberg. 2016. “Understanding women’s decision making power and its link to improved household sanitation: the case of Kenya.” Journal of Water, Sanitation and Hygiene for Development 6, no. 1: 151–160. https://doi.org/10.2166/washdev.2016.128. 123 Jenkins, M. W., and B. Scott. April 2017. “Behavioral indicators of household decision-making and demand for sanitation and potential gains from social marketing in Ghana.” Social Science & Medicine 64, no. 12: 2427-2442. doi:10.1016/j.socscimed.2007.03.010.