LSHTM PHOTO CREDIT: JEAN EUGÈNE INJERONA MID-TERM PERFORMANCE EVALUATION OF MADAGASCAR RURAL ACCESS TO NEW OPPORTUNITIES IN WATER, SANITATION, AND HYGIENE (RANO WASH) ACTIVITY October 2021 This report was produced for review by the United States Agency for International Development. It was prepared by Tetra Tech. Prepared for the United States Agency for International Development by the Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) project under Task Order Number AID-OAA-TO-16-00016 of the Water and Development Indefinite Delivery Indefinite Quantity Contract (WADI), contract number AID-OAA-I-14-00068. Tetra Tech Contacts: Morris Israel, Project Director morris.israel@washpals.org Jeff Albert, Deputy Project Director jeff.albert@washpals.org Lucia Henry, Project Manager lucia.henry@tetratech.com Tetra Tech 1320 N. Courthouse Road, Suite 600, Arlington VA 22201 Tel: 703-387-2100 Fax: 703-414-5593 www.tetratech.com/intdev ACKNOWLEDGEMENTS This performance evaluation was conducted by Carolien van der Voorden (Team Leader, Independent), Ridjanirainy Randrianarisoa (WASH Governance Specialist, Independent), Jean Eugène Injerona (WASH Specialist, Independent), Nirina Rado Randrianavoson (WASH Monitoring and Evaluation Specialist, Independent), Ny Aina Ratsimba (Monitoring and Evaluation and Social Audit Specialist, Independent – joined the team after the inception phase). In the inception phase of the evaluation, Lalaina Andrianamelasoa (WASH Programming Specialist, Independent) also supported the team. Mr. Andrianamelasoa left the project in May 2021 for another assignment. Each team member made meaningful contributions to this report. The Evaluation Team gratefully acknowledges the support given by Clinton Sears (Tetra Tech) and Michelle Kolb (Tetra Tech) who provided guidance and support for the use of the Dedoose platform; Morris Israel (WASHPaLS Project Director, Tetra Tech) and Ken Caplan (WASHPaLS Senior Advisor, Independent) who peer reviewed earlier versions and the final draft report and provided invaluable guidance to the Evaluation Team overall; and Siobhan Girling and Emily Dreyer (WASHPaLS) who provided support throughout the evaluation and helped to organize successful field missions despite challenges presented by the COVID-19 pandemic. We extend a sincere thanks to the RANO WASH staff, particularly the Project Coordination Team (PCT) and the regional and commune-level staff in the regions visited by the Evaluation Team, for their close collaboration and cooperation in the planning and execution of the evaluation. The team would also like to thank these staff for their assistance in organizing the field visit components. And we are particularly grateful to the many informants at national, regional, communal and community levels who participated in Key Informant Interviews or Focus Group Discussions, and who provided open, transparent, and reflective responses to our many questions. Lastly, we are grateful for the support, insights, and guidance provided by the USAID/Madagascar team, notably Falihery Rabetaliana, Patricia Norolalao, and Daniel Nover. MID-TERM PERFORMANCE EVALUATION OF MADAGASCAR RURAL ACCESS TO NEW OPPORTUNITIES IN WATER, SANITATION, AND HYGIENE (RANO WASH) ACTIVITY OCTOBER 2021 DISCLAIMER The author’s’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS ACKNOWLEDGEMENTS............................................................................................................................................. II ACRONYMS AND ABBREVIATIONS ....................................................................................................................... II EXECUTIVE SUMMARY.................................................................................................................................................V BACKGROUND .....................................................................................................................................................V EVALUATION QUESTIONS AND METHODOLOGY................................................................................V FINDINGS................................................................................................................................................................VI 1.0 RANO WASH PROGRAM DESCRIPTION..................................................................................................... 1 1.1 BACKGROUND ......................................................................................................................................... 1 1.2 DEVELOPMENT HYPOTHESIS............................................................................................................... 1 2.0 EVALUATION PURPOSE, METHODS AND LIMITATIONS ...................................................................... 3 2.1 EVALUATION PURPOSE ......................................................................................................................... 3 2.2 AUDIENCE AND INTENDED USES ..................................................................................................... 3 2.3 EVALUATION QUESTIONS ................................................................................................................... 3 2.4 EVALUATION METHODOLOGY ......................................................................................................... 3 2.5 SECONDARY DATA COLLECTION ................................................................................................... 5 2.6 PRIMARY DATA COLLECTION............................................................................................................ 5 2.7 DATA ANALYSIS METHODS ............................................................................................................... 12 2.8 CAVEATS AND LIMITATIONS ............................................................................................................ 12 3.0 FINDINGS............................................................................................................................................................... 14 3.1 EVALUATION QUESTION 1 - RANO WASH DESIGN ............................................................... 14 3.2 EVALUATION QUESTION 2 - GOVERNANCE CAPACITY....................................................... 23 3.3 EVALUATION QUESTION 3 - PRIVATE SECTOR APPROACHES ........................................... 33 3.4 EVALUATION QUESTION 4 - DEMAND AND USE OF WASH PRODUCTS AND SERVICES..................................................................................................................................................... 38 4.0 RECOMMENDATIONS AND PRIORITIZATION....................................................................................... 49 4.1 RECOMMENDATIONS FOR FUTURE DESIGN OR PROGRAM EXTENSION ..................... 49 4.2 RECOMMENDATIONS FOR THE REMAINDER OF THE PROGRAM ..................................... 50 ANNEX I: EVALUATION SOW................................................................................................................................ 54 ANNEX II: EVALUATION METHODS AND BACKGROUND........................................................................ 62 ANNEX III: DATA COLLECTION INSTRUMENTS............................................................................................. 65 ANNEX IV: DATA SOURCES .................................................................................................................................... 97 ANNEX V: EVALUATOR BIOGRAPHIES AND CONFLICT OF INTEREST FORMS...............................100 MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT i ACRONYMS AND ABBREVIATIONS AOPDEM Association des Opérateurs Privés et Distributeurs d’Eau de Madagascar (Association of Water Operators and Distributors of Madagascar) ASUREP Association des Usagers du Réseau d’Eau Potable (Water User Association) ATEAH Agent Technique de l’Eau, Assainissement et l’Hygiène (Water, Sanitation and Hygiene Technical Officer) BCD Behavior Centered Design BPON Budget Programme par Objectif National (National Program Budget for Objective) BPOR Budget Programme par Objectif Régional (Regional Program Budget for Objective) CHV Community Health Volunteers CLTS Community-Led Total Sanitation CRS Catholic Relief Services CSO Civil Society Organization DIREAH Directeur Régional de l’Eau, de l’Assainissement et de l’Hygiène (Regional Director for Water, Sanitation and Hygiene) DREAH Direction Régionale de l’Eau, de l’Assainissement et de l’Hygiène (Regional Directorate for Water, Sanitation and Hygiene) DSP Dalle Sanitation Platform EQ Evaluation Question EQND Equality and Non-Discrimination ESA External Support Agency FAA Fonds d’Appui pour l’Assainissement (Global Sanitation Fund) FGD Focus Group Discussion FUM Follow Up Mandona GOM Government of Madagascar GUS Grow-up Sticker HCF Health Care Facility iDE International Development Enterprises IP Implementing Partner IR Intermediate Result JMP UNICEF-WHO Joint Monitoring Program JSR Joint Sector Review MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT ii KII Key Informant Interview LSHTM London School of Hygiene and Tropical Medicine MEAL Monitoring, Evaluation, and Learning MHM Menstrual Hygiene Management MoWASH Ministry of Water, Sanitation, and Hygiene NPP-WSH National Platform for the Promotion of Water, Sanitation, and Hygiene ODF Open Defecation Free OSCEAH Organisation des Sociétés Civiles de l’Eau, Assainissement et l’Hygiène (Organization of Civil Societies for Water, Sanitation, and Hygiene) PCDEAH Plan Communal de Développement Eau, Assainissement et Hygiène (Commune Level WASH Development Plans) PC-EAH Politique Sectorièl en Eau, Assainissement et Hygiène (WASH Sector Policy) PCT Project Coordination Team PMP Performance Monitoring Plan PPP Public-Private Partnership RANO WASH Rural Access to New Opportunities in Water, Sanitation, and Hygiene SE&AM Suivi Eau et Assainissement de Madagascar (Madagascar Water and Sanitation Monitoring) SLC Structure locale de Concertation (Local Structures for Cooperation) SO Strategic Objective SOW Statement of Work SRMO Structure de Mise en oeuvre de la Coordination Régionale (Regional Implementation Coordination Mechanism) STEAH Service Technique de l’Eau, Assainissement et l’Hygiène (Water, Sanitation and Hygiene Technical Service) SWA Sanitation and Water for All TA Technicien d'Appui (Support Technician) TOC Theory of Change USAID United States Agency for International Development USG United States Government VSLA Village Saving and Loans Association WASH BAT Water Sanitation, and Hygiene Bottleneck Analysis Tool WASH Water, Sanitation, and Hygiene WASHPaLS Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT iii WSP Water Service Provider WSSCC Water Supply and Sanitation Collaborative Council MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT iv EXECUTIVE SUMMARY BACKGROUND Rural Access to New Opportunities in Water, Sanitation, and Hygiene (RANO WASH) is a five-year $30 million bilateral United States Agency for International Development (USAID) water, sanitation, and hygiene (WASH) activity in Madagascar. Its period of performance is June 2017 to June 2022. CARE International leads the consortium implementing the project, and sub-awardees include Catholic Relief Services (CRS), WaterAid, BushProof, and Sandandrano. The program aims to reach 250 rural communes in six high-priority regions of Madagascar: Vatovavy Fitovinany, Atsinanana, Alaotra Mangoro, Amoron’i Mania, Haute Matsiatra, and Vakinankaratra. RANO WASH is built around three interconnected strategic objectives (SOs): • SO 1: Strengthening the governance and monitoring of water and sanitation • SO 2: Increasing the engagement of the private sector in the delivery of WASH services • SO 3: Accelerating the adoption of healthy behaviors and the use of WASH service EVALUATION QUESTIONS AND METHODOLOGY The Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) project completed a mid-term evaluation between April and August 2021 to assess whether the approaches employed and activities undertaken are successfully contributing to the program’s goal of increasing equitable and sustainable access to WASH services. The evaluation sought to answer the five following evaluation questions (EQs): 1. How appropriate is the RANO WASH design to the WASH challenges in the target regions and communes? 2. To what extent has RANO WASH built governance capacity to improve sustainable service delivery at multiple levels? 3. To what extent have the different RANO WASH private sector approaches for water service delivery and sanitation access expansion been successful? 4. To what extent have RANO WASH activities been successful at building demand, activating demand, and ensuring use of WASH products and services in intervention regions and communes? 5. What implementation approaches should be prioritized in future for RANO WASH’s final two years and by the WASH sector? For each EQ, the Evaluation Team devised sub-questions to guide more detailed analysis. Four principal data collection methods were employed: (a) a review of contractual deliverables and documents produced by RANO WASH, the Government of Madagascar, and partners; (b) key informant interviews (KIIs); (c) focus group discussions (FGDs); and (d) site observations. The Evaluation Team took care to triangulate and validate data, reduce bias, and increase the depth and balance of the findings and analysis. Using KII and FGD topic guides for each respondent type, the Evaluation Team visited 15 communes across four regions, and conducted 103 interviews. Twenty-four interviews were FGDs with between 3 and 20 participants. Interviewees included RANO WASH program staff and senior management staff from the various Ministry of Water, Sanitation, and Hygiene (MoWASH) directorates, local government stakeholders, development partners, and national civil society representatives, private sector service MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT v providers and commune and community-level stakeholders who engaged in RANO WASH supported activities or received direct technical assistance. All qualitative texts from primary data collection were uploaded, coded, and analyzed using Dedoose, a secure web-based application. The team also conducted a desk review and collected secondary data to contextualize background and triangulate findings. FINDINGS Overall, the Evaluation Team assessed the program to be complex, ambitious, and innovative; perceived positively by a large majority of stakeholders; and seen as a sector leader. While water supply provision through the public private partnership (PPP) model is substantially off-track, there are plausible reasons for these delays, and important lessons to be learned. Sanitation results, particularly regarding Open Defecation Free (ODF) status, are above target. RANO WASH design. In a sector with frequent ministry changes, high levels of staff turnover and low scores on most sector building blocks (see, for example, WASH BAT 2014, 2018), the program’s focus and design is highly relevant and deemed appropriate by the vast majority of informants. The program was welcomed specifically for its emphasis on governance and private sector engagement. However, the design underestimated the magnitude of challenges facing the sector. The complexity of working along three pillars (the three SOs) was not initially supported by a clear strategy and framework, leading to some confusion and slow take-off. The adoption and operationalization of a systems approach into nine building blocks in 2019 helped clarify direction and prioritization of actions, but the approach is still considered highly complex and not yet universally understood among program stakeholders, including implementing partners. The program design’s strong focus on programmatic and joint sector learning has not yet fully come to fruition. RANO WASH design covered an ambitious spread of activities and themes, and the program therefore set targets of people to be reached by WASH services quite conservatively. Given the need to strengthen governance capacity, innovate and demonstrate scalable approaches for the sector, and build ownership to ensure long-term sustainability, the focus on range over scale was the appropriate choice. However, the Evaluation Team finds that it might have been better to take a commune-wide approach in a smaller number of communes. This would have supported more inclusive planning, as the need to demonstrate the potential of the chosen PPP Construct-Invest-Manage model required selection of “viable” schemes in terms of location and population size. This left behind many of the smaller, more remote communities within intervention communes. Governance capacity. RANO WASH actively engaged in sector policy and strategy processes, including work on a new Sector WASH Policy. Specifically, the team supported MoWASH to formalize its PPP model contract template and improve its planning and monitoring processes. RANO WASH invested heavily in supporting roll-out and capacity building for the use of the Suivi Eau et Assainissement de Madagascar (Madagascar Water and Sanitation Monitoring [SE&AM]) tool, particularly at regional and communal levels. While informants widely appreciated this support, many pointed out that large challenges remain in the development of a harmonized sector monitoring and evaluation system, and that RANO WASH can further support the required assessments and strategy development to arrive at such a system. With regard to institutional arrangements, the Evaluation Team considers the support to commune-level Technical WASH Services (STEAH) and Technical WASH Assistants (ATEAH) as a cornerstone of the RANO WASH strategy and one of its key sector contributions. Possible risks relate to the high level of expectations and skills required, the lack of commune governance stability, and insufficient commune budgets to cover ATEAH salaries and STEAH services. STEAH require backstopping by Regional Directorates of WASH (DREAH), who are responsible for regional coordination. RANO WASH support to strengthen DREAH capacity, provide operational resources, and co-lead Structures de Mise en oeuvre de la Coordination Régionale (Regional Implementation Coordination Mechanisms), was widely acknowledged and appreciated by all informants. However, the MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT vi role of ATEAH should be formalized, which will help DREAH and STEAH structures to remain stable WASH resources for their communities. Most national informants recognized RANO WASH for having made substantial contributions to improved sector collaboration and coordination in recent years, despite the delayed establishment of a new government-driven WASH coordination mechanism. At the commune level, RANO WASH invested heavily in the establishment and/or strengthening of a range of community engagement and accountability platforms. While local informants overall expressed appreciation and perceived that this support contributed to local WASH prioritization and ownership, the Evaluation Team noted some concerns regarding the overlap and confusion in roles between the various bodies, their dependence on RANO WASH, and their self-professed roles in sensitization without necessarily having been trained properly. Private sector approaches. For rural water supply, informants cite RANO WASH as a key contributor to a paradigm shift that is now noticeably underway, from a situation based mainly on community management and the principle of “free” water services to one of private sector management of paid for (and therefore higher quality) and sustainable water supply services. However, given the likely slow pace of transition and the need to cover entire communes with water supply services, the program could build on its PPP+ model and further explore possible transitional models of private and community-managed services operating side-by-side under commune leadership, with a progressive move towards private sector management, where this can be viably done. The program delays in meeting its original PPP targets were mainly due to underestimating the time needed to build relationships, mutual understanding, buy-in, and PPP-related capacity among all partners involved as well as the time needed to develop procedures that work and deal with complex and often very lengthy approval systems, including those of the donor. Overall, the Evaluation Team found that the program has managed to demonstrate an approach that works; that there is sufficient interest from private sector partners to get involved; and that households are by and large willing and able to pay the tariffs, including households in vulnerable situations thanks to the diversified product lines. Divergent opinions exist concerning the selection of water service providers (WSPs), pointing mainly to the need to ensure strong and customer-oriented local interfaces, no matter the type of WSP. So far, WSPs have not shown autonomous financing capacity to cover, for example, scheme extensions, and there have been apparent weaknesses in marketing and diversified targeting of services. The ‘100 first connections’ scheme was devised as a marketing scheme only, but has the potential to be reshaped into a tool for inclusion of more vulnerable households. More can also be done with the MoWASH to explore ways to subsidize or reduce household connection costs, which are seen to be the largest obstacle. In the area of market-based sanitation, RANO WASH’s engagement with local masons (and with seamstresses marketing menstrual hygiene materials) was seen to be more of a ‘social actor’ rather than a business approach. Masons indicated to the Evaluation Team that they were “selected” or “appointed” for their roles, they set prices based on RANO WASH’s guidance, they are dependent on materials provided by the program, and that, in most cases, they were doing this “for the good of their community,” rather than to make money. Some concerns also emerged about the role of mayors, the relatively limited choice of toilet options, and generally low household ability to pay. While noting the work done by RANO WASH with International Development Enterprises (iDE), the potential (and limitations) of a sanitation market-based approach in the context of Madagascar should be studied and elaborated further in collaboration with development partners and the MoWASH. Demand and use. Demand for water is omnipresent in Madagascar, and as a program that has ‘water’ in its name, RANO WASH could communicate and manage expectations of intervention communes more transparently regarding possible program support to rehabilitation, construction, or extension of water supply infrastructure. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT vii With regard to demand and use of improved sanitation facilities, the program devised a thorough sanitation and hygiene behavior change strategy built around five components, with community-led total sanitation (CLTS) at its core and incorporating the Grow-Up Sticker (GUS) method (the result of extensive research and behavior-centered design with the London School of Hygiene and Tropical Medicine [LSHTM]). While the Evaluation Team endorses any move to support open defecation free (ODF) communes, the evaluation highlighted a number of points for continued learning and reflection. These include potential risks linked to conducting sensitization activities by large numbers of actors and community groups with mixed capacity and understanding of WASH and behavior change processes; a potential “watering down” of the CLTS principles of facilitation and self-efficacy, versus a more didactic, messaging approach; limited potential for scale beyond RANO WASH and limited evidence of the spillover effect of the GUS method; the potential adverse effect of “rewarding” ODF communities with water supply; the risk of community-sanctioned use of force and coercion affecting people in vulnerable situations hardest; the perceived pressure to buy while facing limited options; and the complexities of facing more and more “non-strategic” villages. Many interviewees cited RANO WASH’s programming on menstrual hygiene management (MHM) as a key example of the program’s gender-inclusive approach. The Evaluation Team found the approach, however, to be largely limited to promoting body washing and the marketing and use of reusable pads. Of some concern was the spread of misinformation illustrated in informant interviews. Overall, the program’s focus on sustainability was found to be evident throughout its work and at the forefront of much of its systems building work. Even so, the Evaluation Team proposes a concerted and continued focus on exit planning, and on balancing the program’s desire to drive change with the need to ensure local ownership. RECOMMENDATIONS For future design/extension: Consider extending the timeframe for a program of this complexity and ambition. Continue to operationalize and familiarize people with the systems building blocks approach, but explore handing elements off to other programs or sector actors to keep the program focus manageable. Move toward a district or commune-wide, inclusive planning and design approach and • Consider reducing the number of target communes; • With the MoWASH, identify and start planning for the first districts and/or communes most able to demonstrate this approach; • Add a layer of longer term and business planning to the Plan Communal de Développement Eau, Assainissement et Hygiène (Commune-Level WASH Development Plans [PCDEAH]) to gradually cover entire communes and invite WSP inputs into related planning and consultation processes. Design for scale and scalability, exploring private sector or hybrid models that can include larger as well as smaller communities. Focus on scalable approaches to behavior change programming with community ownership and a clearer exit strategy. With MoWASH and government partners, develop different subsidy approaches and WSP financial support mechanisms to offer affordable services to more vulnerable households. Continue to focus on increasing the sector funding base, including through private sector engagement, but leave work on increased local tax collection to other (USAID) programs. Negotiate government co-financing at the start of a program to cover, at minimum, the salaries and operating expenses of regional and communal structures, specifically DREAH and STEAH. Carefully consider whether extension of private sector WASH service delivery in urban areas (beyond commune ‘capitals’) should be included in the same program or done separately. Future process and structure Start work in all intervention regions at the same time to allow for contextualization and maximize available time for all. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT viii Review the internal consortium set -up and seek ways to reduce procedural load . Future process and structure For the remainder of the program: MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT ix Institutional arrangements Advocate for an increase in DREAH budgets, explore ways to strengthen lines of engagement and dialogue between DREAH and STEAHs, and progressively hand over responsibility for STEAH training to DREAHs . Work with government to formalize status of STEAH and ATEAH and to ensure inclusion in communal budget lines, but undertake a review of the spread of responsibilities and required skills of ATEAHs to ensure realistic load and sustained continuity. Support the phased establishment of local dialogue and input structures, particularly structures locales de concertation [local structures for cooperation, SLCs], Association des Usagers des Réseaux d’Adduction en Eau Potable [Water User Association, ASUREP], Organisations des Société Civile de l’Eau, Assainissement et de l’hygiène [platform of WASH civil society organizations, OSCEAH], along the needs of the commune, to avoid confusion and redundancy Encourage mayors to ensure more pro -active sharing of information with commune stakeholders. Planning, monitoring and evaluation Engage in a joint assessment, with sector partners , of the root causes hindering successful implementation of national sector monitoring and planning, beyond the SE&AM tool, including an assessment of capacities required for the establishment of a solid sector-wide monitoring, evaluation and planning syste m. Private sector engagement in WASH Develop clear criteria to guide decision making on the process of transition towards private sector management from community management , and any transitional measures required. Design and develop a package of activities intended to support communes as they manage private and community management methods at the same time in their territories. Continue to operationalize PPP+ components and build diversified marketing and customer relationship management skills to increase coverage . Consider setting targets or bonuses for inclusive , possibly phased, service delivery to larger numbers of vulnerable households per scheme. Continue working with partners toward a common vision on private sector engagement in WASH in Madagascar and continue to explain and promote the PPP mode l. Work with communes to develop longer -term plans and strategies for universal drinking water supply at the level of a commune, its capital (chef lieu) and villages of all fokontany. Invite private sector WSPs to engage in this commune-level planning, in terms to be specified. PPP contracting and roll -out Simplify the PPP contract and ensure its availability in Malagasy language at the commune level. Review the WSP selection process and/or ensure full knowledge among PPP stakeholders, particularly around WSP contracting for extensions or within geographical regions, to allow for economies of scale . - Coordination and dialogue Continue engaging in and, where relevant, driving sector dialogue, learning and sharing, including advocating for and supporting a Joint Sector Review. Communication & clarification Clarify and clearly communicate which communes/communities and institutions will still receive what support for water system construction/rehabilitation/PPP establishment between now and the end of the program. Market based Sanitation Continue to gain understanding of the potential (and limitations) of market-based approaches in the context of rural Madagascar, and work with sector partners to jointly develop a shared sector roadmap towards market-based sanitation. Revisit the mason and seamstress selection process, and further support and strengthen marketing and business planning training, supply chains, and access to finance. WASH in schools and health care facilities Continue joint advocacy with sector partners, particularly UNICEF, to promote a more effective and scalable approach for institutional WASH beyond the ‘WASH Friendly’ institutions approach. Equity and inclusion Train implementing partners and commune stakeholders, particularly mayors, ATEAH and SLC, on Do No Harm principles and the potential consequences of using force or penalties as a means of behavior change motivation. Behavior change programming Continue to target ODF communes, covering both ODF and non-ODF communities with a contextualized package of behavior change strategies. Review menstrual hygiene management (MHM) training and messaging provided. Learning and documentation Refine program learning and sharing on the systems approach and building blocks. Prioritize documenting good practices on governance, private sector contracting, and STEAH strengthening, and offer peer-to-peer exchanges between communes. Ensure continued learning and documentation of the RANO WASH behavior change strategy and approaches used, focusing on scalability, sustainability, and the effective combination of approaches. Conduct a program end-line survey, including a focus on habit creation, equity and inclusion, and service delivery. (USAID to) facilitate exchange and joint learning between USAID-funded rural water supply programs with private sector engagement in different countries. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT x 1.0 RANO WASH PROGRAM DESCRIPTION 1.1 BACKGROUND Rural Access to New Opportunities in Water, Sanitation, and Hygiene (RANO WASH) is a five-year US$30 million bilateral USAID WASH activity investment in Madagascar, scheduled to run from June 2017 to June 2022. CARE International leads the consortium implementing the program. Sub-awardees include Catholic Relief Services (CRS), WaterAid, BushProof, and Sandandrano. The latter two are private sector enterprises. The program addresses WASH challenges at local, regional, and national scales and aims to reach 250 rural communes in six high-priority regions of Madagascar: Vatovavy Fitovinany, Atsinanana, Alaotra Mangoro, Amoron’i Mania, Haute Matsiatra, and Vakinankaratra. To do so, the consortium partners have developed a systematic partnership with national and regional governments, water and sanitation institutions, communities, private sector actors, civil society organizations (CSOs), and beneficiaries. The program has been implemented in phases and with different consortium leads. Regional activities started in 2018 in Vatovavy Fitovinany and Atsinanana, led by CRS, and Alaotra Mangoro, led by WaterAid. CARE began leading implementation in Vakinankaratra in 2019, expanding to reach Amoron’i Mania and Haute Matsiatra in 2020. To ensure sustainability and local buy-in, the program is based on a cost share arrangement, with community partners and WASH service providers expected to contribute to the cost of service provision. Through public-private partnership (PPP) construction management contract models, private operators have contributed between five and 22 percent of the cost of construction or rehabilitation, depending on the systems already in place. 1.2 DEVELOPMENT HYPOTHESIS The ultimate goal of RANO WASH is to increase equitable and sustainable access to WASH services to maximize the impact on human health and nutrition and preserve the environment in 250 rural communes in six regions. Toward this end, the program posited the following development hypothesis: “(1) If WASH governance and systems and capacities to manage WASH services accountably are strengthened; (2) if quality WASH products and services are reliably available and affordable for all; and (3) if demand for improved WASH behaviors and services grows in an expanded consumer market; then the incidence and prevalence of childhood diarrhea, exposure to environmental enteropathy, and under five mortality rates will decrease.” This hypothesis was further elaborated into a program theory of change (TOC), summarized in Figure 1 below. As such, the program focuses its efforts on three interlinked Strategic Objectives (SOs): SO1: Strengthening the governance and monitoring of water and sanitation. The focus lies on national leadership, commitment, and accountability for sector performance; planning, monitoring, and learning to influence policy; regional and local government tools, resources, and management capacities for WASH service delivery; and increased community and civil society control over services used (Final Technical Proposal, Section 2 A.1). SO2: Increasing the engagement of the private sector in the delivery of WASH services. The program takes a predominantly private sector-oriented approach to WASH product and service delivery, aiming to facilitate structured engagement of private sector actors to provide market-based WASH products and services, and looking to take evidence-based approaches, products, and business￾models to scale (Technical Proposal, Section 2 A.2). MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 1 SO3: Accelerating the adoption of healthy behaviors and the use of WASH services. The program posited that to achieve this objective, it must do three things: better understand and address behavior drivers that influence demand for adoption of and adherence to sanitation and hygiene behaviors among target populations; develop and embed specific behavior change strategies that target the key determinants of sustained adoption; and inform national policies and programs that shape the “macro-level determinants” or enabling environment of sustained sanitation and hygiene behaviors (Technical Proposal, Section 2 C.3). The three SOs were expected to be inter-dependent, as illustrated in Figure 1. Running throughout the program is a focus on “community dialogue and engagement for a gender transformative approach to empower all people, especially women and girls” (TOC). To take best advantage of the three international consortium members’ existing operations and expertise in the different regions and areas of engagement, RANO WASH was designed around a division of leadership. CARE took overall management lead and led the behavior change and gender work under Strategic Objective 3, CRS took the thematic lead on private sector engagement under Strategic Objective 2 and on the project’s monitoring, evaluation and learning (MEAL) system, and WaterAid took thematic lead on governance under Strategic Objective 1. The private sector partners Sandandrano and BushProof engaged mainly on Strategic Objective 2 and on the governance aspects of private sector engagement covered under Strategic Objective 1. FIGURE 1: RANO WASH THEORY OF CHANGE MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 2 2.0 EVALUATION PURPOSE, METHODS AND LIMITATIONS 2.1 EVALUATION PURPOSE The evaluation is designed to inform the implementing partners and USAID/Madagascar if the approaches employed by the CARE-led consortium are successfully meeting the program’s goal of increasing equitable and sustainable access to WASH services. The evaluation results are expected to inform the need for any course corrections or review of program priorities in its final year of implementation. The evaluation may also be a resource for future USAID/Madagascar WASH programming. 2.2 AUDIENCE AND INTENDED USES The audiences for the evaluation report include: the USAID/Madagascar mission and its Health, Population, and Nutrition team; CARE International and the consortium partners; the Government of Madagascar (GOM), and WASH sector collaborators. To a lesser degree, the evaluation findings may be used by USAID/Madagascar and Washington D.C. technical staff to inform global WASH sector programming. The Evaluation Team intends for the findings of the performance evaluation to be disseminated widely. 2.3 EVALUATION QUESTIONS The evaluation statement of work (SOW) included the following Evaluation Questions (EQs, see Annex I): 1. How appropriate is the RANO WASH design to the WASH challenges in the target regions and communes? 2. To what extent has RANO WASH built governance capacity to improve sustainable service delivery at multiple levels? 3. To what extent have the different RANO WASH private sector approaches for water service delivery and sanitation access expansion been successful? 4. To what extent have RANO WASH activities been successful at building demand, activating demand, and ensuring use of WASH products and services in intervention regions and communes? 5. What implementation approaches should be prioritized in future for RANO WASH’s final two years and by the WASH sector? Based on an initial desk review and discussions with key informants from USAID/Madagascar, and as discussed in detail in the Inception Report, a set of sub-questions was devised for each of the five EQs. These sub-questions guided the analysis presented in Section 3. 2.4 EVALUATION METHODOLOGY The approach and data analysis methods described herein are generally consistent with the methodology outlined in the Inception Report approved by USAID prior to the start of data collection. The Evaluation Team employed four principal data collection methods: (a) review of contractual deliverables and other documents produced by RANO WASH, the GOM, and other third parties (development partners); (b) MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 3 key informant interviews (KIIs); (c) focus group discussions (FGDs); and (d) site observations. The data sources, data collection, and analysis methods used to answer each EQ are provided in Table 1. In developing the methodology, the Evaluation Team employed three measures to triangulate and validate data, reduce bias, and increase the depth and balance of the findings and analysis: 1. Methodological triangulation (i.e., consulting data from multiple sources: interview data, RANO WASH reports, and government documents); 2. Investigator triangulation (i.e., involving teams of at least two data collectors per interview wherever possible, and having multiple team members involved in data analysis and identification of emerging findings); and 3. Data triangulation (i.e., obtaining the perspectives of multiple similar types of respondents across the sampled regions and communes). TABLE 1: DATA SOURCES, DATA COLLECTION AND ANALYSIS METHODS EVALUATION QUESTIONS EMPLOYED DATA SOURCES DATA COLLECTION METHODS DATA ANALYSIS METHODS EQ 1: How appropriate Sector documents such as the KIIs with Ministry of Water, Qualitative data analysis, is the RANO WASH Code de l’Eau, the ‘Madagasikara Sanitation, and Hygiene including context and design to the WASH Madio 2025’ roadmap and (MoWASH) representatives pattern analysis, data challenges in the target WASH Bottleneck Analysis Tool (current and at time of disaggregation by location, regions and communes? (WASH BAT) reports; program documents such as the RANO WASH Technical Proposal, MEAL Plan, annual work plans, activity-specific planning, strategy or implementation descriptions covered in RANO WASH progress report annexes; selected sector documents such as on systems strengthening, gender and EQND. program conception), USAID and RANO WASH staff, program managers from USAID partner projects, Fonds d’Appui pour l’Assainissement (FAA), UNICEF and others; country reps and senior staff of the consortium partner agencies, regional Directorates of Water, Sanitation and Hygiene (DIREAH), representatives and local government staff, private sector partners. gender, and role, as appropriate. EQ 2: To what extent National policy documents, KIIs and/or FGDs at national, Qualitative data analysis, has RANO WASH built ‘Madagasikara Madio 2025’ regional and communal level, including context and governance capacity to roadmap, national budget including previous and current pattern analysis, data improve sustainable allocations, Suivi Eau et staff from MoWASH and disaggregation by location, service delivery at Assainissement de Madagascar regional DIREAH, key sector gender, and role as multiple levels? (SE&AM) documents, WASH BAT reports. Annual and quarterly program reports, activity-specific plans, and documentation. Documents about regional coordination, dialogue, and capacity building exercises. partners at national and local level, RANO WASH staff, implementing partners, project stakeholders including private sector and civil society representatives. appropriate. EQ 3: To what extent have the different RANO WASH private sector approaches for water service delivery and sanitation access expansion been successful? WASH market assessments and WASH market development plans; micro-finance/loan product documentation; PPP contracting documentation; regional reporting into SE&AM; annual and quarterly program reports, learning products, case study documents. Water service KIIs with project staff, private sector service providers (individual and FGD), national and regional government staff, CSO representatives. Regional and site visits to WSP-managed/PPP schemes; sanitation entrepreneurs; project fokontany, schools and Qualitative data analysis, including context and pattern analysis, data disaggregation by location, gender, and role as appropriate. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 4 EVALUATION QUESTIONS EMPLOYED DATA SOURCES DATA COLLECTION METHODS DATA ANALYSIS METHODS provider (WSP) reports, service tracking and expenditure data, as available. health facilities. EQ 4: To what extent have RANO WASH activities been successful at building demand, activating demand, and ensuring use of WASH products and services in intervention regions and communes? Program annual plans and reports, quarterly reports and activity-specific plans and updates, baseline survey reports, assessments, workshop reports, best practice case studies, research reports linked to the London School of Hygiene and Tropical Medicine (LSHTM) study, reporting into SE&AM. Key Informant Interviews with national government representatives, RANO WASH staff, program managers from FAA, UNICEF and other sector programs, regional, commune, and community-level project stakeholders and government partners. Site visits and FGDs with promoters/natural leaders and community representatives, CSO representatives. Qualitative data analysis, including context and pattern analysis, data disaggregation by location, gender, and role as appropriate. EQ 5: What implementation approaches should be prioritized in future for RANO WASH’s final two years and by the WASH sector? Summary of data source analysis conducted for EQs 1 to 4. KIIs with MoWASH representatives, USAID and RANO WASH staff, program managers from USAID partner projects, FAA, UNICEF and others; heads of the consortium partner agencies, regional DIREAH representatives and local government staff, private sector partners. Qualitative data analysis, including context and pattern analysis, data disaggregation by location, gender, and role as appropriate. 2.5 SECONDARY DATA COLLECTION The Evaluation Team began the exercise by conducting a desk review of key project documents provided by RANO WASH and USAID, notably the quarterly and annual progress reports and associated annexes, along with other GOM policy and strategy documents useful for placing RANO WASH and the evaluation in context (see Annex IV for a complete list of documents reviewed). The Team reviewed some documents in their entirety, but in most cases, members limited their reviews to documents or sections of documents related to their evaluation focus. The Evaluation Team integrated findings from the desk review in the primary data collection process (most notably by informing the KII respondent list and KII and FGD guides). At the same time, desk review and secondary data collection was an ongoing and iterative process, where findings from KIIs and field work prompted further exploration of key documents. 2.6 PRIMARY DATA COLLECTION The Evaluation Team used three primary data collection approaches: KIIs, FGDs, and site observations where appropriate. 2.6.1 SELECTION OF REGIONS AND COMMUNES FOR PRIMARY DATA COLLECTION Per the Inception Report, the Evaluation Team was expected to visit and undertake primary data collection in four of the six RANO WASH program regions. Primary factors for regional and commune selection initially discussed with USAID included the following: MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 5 ● Start date/year of RANO WASH activities: visit sites representing different start dates of project interventions; ● Mix between CARE, WaterAid and CRS-led regions; ● Mix of sites that have been successful and those that have encountered challenges/blockages; and ● Sites representing the results of the different components of WASH in relation: (i) to CLTS and sanitation and hygiene behavior change, the scaling-up strategy and ODF status; (ii) to the PPP model; (iii) to sanitation marketing; and (iv) to the development of local governance. The final selection of field visit sites covered four regions and 15 out of 250 program communes (see Figure 2). The selected regions included Alaotra Mangoro, Atsinanana, Amoron’i Mania, and Vatovavy Fitovinany. A more detailed description of the characteristics of each region and RANO WASH program interventions is provided in Annex II. Table 2 provides more information on the selected communes and their sanitation and hygiene situation. Information regarding water system activities and communal WASH planning is based on data gathered during the field visits. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 6 FIGURE 2: MAP OF INTERVENTION AREAS TABLE 2: KEY INFORMATION FOR SELECTED COMMUNES REGION DISTRICT COMMUNE RANO SE&AM RANO PCDEAH IN WASH START SANITATION COVERAGE WASH WATER PLACE (KIIS) RATE SYSTEM ACTIVITIES Alaotra Mangoro Moramanga Anosibe Ifody 2018 49,4% Yes Under validation Alaotra Mangoro Moramanga Beforona 2018 16,86% Yes In progress Alaotra Mangoro Moramanga Ampasimpotsy Gara 2019 16,06% No No Atsinanana Brickaville Ranomafana Est 2018 54,80% Yes Yes Atsinanana Brickaville Antsampanana 2019 2,2% No In progress Atsinanana Vatomandry Niarovana Caroline 2018 10,28% Yes In progress Amoron’i Mania Ambositra Ilaka Centre 2020 13,06% Yes In progress Amoron’i Mania Ambositra Ambositra II 2020 65,96% Discussions ongoing In progress Fitovinany Ifanadiana Ambiabe 2019 13,04% No Yes REGION DISTRICT COMMUNE RANO WASH START SE&AM SANITATION COVERAGE RATE RANO WASH WATER SYSTEM ACTIVITIES PCDEAH IN PLACE (KIIS) Fitovinany Ifanadiana Ranomafana 2020 17,38% Discussions ongoing Under validation Fitovinany Manakara Atsimo Marofarihy 2018 4,50% No No Fitovinany Vohipeno Andemaka 2020 0,93% Discussions ongoing Done but not seen Fitovinany Vohipeno Vohitrindry 2018 0,93% Yes Yes Fitovinany Vohipeno Nato 2019 7,61% No In progress 2.6.2 KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSIONS All national and regional level interviews were conducted in French. Audio recordings were made, and interview notes were transcribed by one of the team members, then reviewed by another. At the commune level, most interviews were undertaken in Malagasy. The team members then used the audio recordings to transcribe and finalize notes in French. Interviewees included RANO WASH program staff and senior staff from the various MoWASH Directorates, local government stakeholders, development partners and national civil society representatives, private sector service providers and commune and community-level stakeholders who engaged in RANO WASH supported activities or received direct technical assistance. Informants were selected based on their (expected) familiarity with RANO WASH either through engagement at the design stage or direct engagement/interaction at the national, regional, or communal level, and/or based on their expert status reflective of their role and responsibilities. The identification of informants within the communes was conducted jointly by the Evaluation Team and the RANO WASH program support technician (TA). Within each commune, the mayor/deputy mayor and the technical agent responsible for water, sanitation, and hygiene (Technical WASH Assistant [ATEAH]) were systematically interviewed. In Andemaka and Ambiabe communes, the team did not interview the mayors due to absence or illness. The team held discussions with the members of the various structures set up within the communes, including the Structure Locale de Concertation (Local Structure for Cooperation [SLC]), the Organization of Civil Societies for Water, Sanitation, and Hygiene (OSCEAH), Association des Usagers des Réseaux d’adduction en Eau Potable (Water User Association [ASUREP]), members of the Voamamy Association/ village savings and loans association (VSLA), and some school and health care facility staff. Private sector actors interviewed at the local level included local agents of the private sector water service provider, local masons, and local seamstresses. Lastly, the teams carried out informal discussions with community members in the visited villages to collect more information and to discuss observations about the WASH infrastructure. The Evaluation Team conducted and captured interview notes for a total of 103 KIIs and FGDs covering an estimated 330 people.1 Of the 103 captured interviews, 66 took place during the 15 commune visits, and the team held 37 interviews with national or regional stakeholders. Of the 66 commune interviews, 23 were FGDs, with numbers of participants ranging from three to over 20 people, and two were double interviews. Of the 37 national and regional interviews, three were small group interviews and 1 Estimated because some of the larger FGDs saw some movement of people coming in and out of the meeting. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 7 one was a small FGD with three private sector providers. As noted in Table 3, most of the larger FGDs were held with commune civil society groups and multi-stakeholder platforms (OSC/SLC), and VSLA women’s groups, with three smaller group discussions held with commune-level private sector actors. While it was originally foreseen that more private sector actors might be grouped in FGDs, the actual number of WSP representatives, masons, or seamstresses per commune was quite limited, resulting in more one-on-one interviews with this target group instead. The division of gender and roles of informants is visualized in Figure 3. Importantly, a majority of informants, particularly in government roles, are male. This imbalanced gender representation is thought to reflect the reality of Madagascar’s WASH sector rather than a shortcoming of the evaluation. With regard to different types of community members (commune and community structures as described above), the gender balance was more equal. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 8 TABLE 3: INTERVIEWS BY TYPE OF RESPONDENT COMMUNE TOTAL CAPTURED INTERVIEWS (NO. OF PEOPLE INVOLVED) COMMUNITY MIXED CSO/ SLC VSLA SCHOOL HCF COMMUNE STAFF PRIVATE SECTOR Anosibe Ifody 6 (34) 1 (10) 1 (20) 2 2 Beforona 6 (29) 1 1(12) 1 (13) 1 1 1 Ampasimpotsy Gara 5 (31) 1 (5) 1 (23) 1 2 Ranomafana Est 2 (26) 1 (7) 1 (15) 1 2 1 Antsampanana 3 (18) 1 (15) 2 1 Niarovana Caroline 6 (6) 1 1 1 2 1 Ilaka Centre 3 (17) 1 (15) 1 1 Ambositra II 2 (6) 1 (2) 1 (4) Kelilalina 3 (6) 1 (4) 1 1 Ambiabe 3 (22) 1 (18) 1 1 (3) Ranomafana 2 (11) 1 (10) 1 Marofarihy 6 (17) 1 (3) 2 (10) 2 1 (2) Vohitrindry 5 (29) 1 1 (15) 1 (9) 2 (4) Nato 8 (16) 1 2 (10) 3 2 Andemaka 6 (17) 1 2 (12) 1 2 (3) Sub-Total (commune level) 66 (285) Sub-Total (national and regional) 37 (45) Total 103 (330) MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 9 Gender Role Private sector partner Implementing partner Government Consortium member RANO WASH staff community member FGD equal FGD majority female FGD majority male KII Male FIGURE 3: INFORMANTS BY GENDER AND ROLE Sessions with national and regional stakeholders included ten interviews with GOM staff including MoWASH and the Directeur Régional de l’Eau, de l’Assainissement et de l’Hygiène (Regional Director for Water, Sanitation and Hygiene [DREAH]); eight interviews with development partners; ten interviews with 12 RANO WASH staff; seven interviews with nine RANO WASH consortium partners; and one interview and one FGD with private sector partners. A detailed list of national and regional informants is provided in Annex IV. 2.6.3 KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSION GUIDES KIIs, FGDs, community engagement, and community observation guides are attached as Annex III. The KII topic guides are structured to cover key themes of relevance to answer the EQs, as appropriate to the respondent, combining more general questions such as progress toward targets, lessons learned, and internal and external success factors with more specific questions related to the SOs, sector and programmatic learning, collaboration and harmonization, gender and inclusion, and sustainability considerations and results. The KII guides combine questions on the different EQs and were prepared for each respondent type (RANO WASH staff, national level government staff, regional government staff, collaborating/sector partners, private sector service providers, civil society/community representatives). While the KII guides vary based on respondent categories and the EQs, most include some variation of the following: • Basic demographic and context data, including interviewee’s sex, age, location (as well as some indication of “believability” in case there is reason to believe that the respondent may not respond truthfully); • Relationship with other activities, government, or stakeholders and other factors that could affect responses; MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 10 • Respondent’s relationship to RANO WASH as a staff member, policymaker, government partner, sector or project partner, beneficiary, stakeholder, as applicable; • Project and activity discussion based on respondent relationship to RANO WASH; • Perception of successes and failures, challenges, and opportunities; • People’s perceptions and expectations with regard to sustainability of program results and the necessary mechanisms to achieve it; • RANO WASH adaptive capacity, collaboration/interaction with, and accountability among partners and key stakeholders; and • Promising practices that should be retained and why, as applicable. To collect data from specific groups of actors during the field visits, topic guides were developed for four different categories of actors (private sector WASH providers, regional and local civil society representatives, women and girls, and community members, schoolteachers, and health facility staff). These topic guides were designed to either be used to guide FGDs or in individual conversations with local actors in these categories. The content of the guide covers largely the same themes as the KII guides, but the number of questions is more limited and focused on people’s direct interaction with the program. After completing the first phase of field visits in Alaotra Mangoro and Atsinanana, the Evaluation Team decided to add a questionnaire specific to mayors and STEAH (WASH Technical Service) staff. Largely built on the original KII guide for regional government staff, the team simplified and further localized the questions to ensure they focused on mayors’ engagement with the RANO WASH program and their exposure to and understanding of WASH issues. In addition, the Evaluation Team produced a field observation guide and template to take note of the community environment and WASH facilities. The team used the guide to survey the villages and log the types of latrines and water infrastructure used. With regard to sanitation, the team encountered the same type of latrines in almost all settings and due to lack of time noted key details of observed systems or infrastructure and took photographs rather than completing the templates in detail. Early interviews followed the flow of the guides closely. As the evaluation progressed, patterns began to emerge, leading the Evaluation Team to explore certain issues more closely. The evaluation is framed as a formative rather than summative evaluation, nor is it strictly a piece of research. Therefore, the team considered it appropriate to diverge from the KII guides to probe deeper on some issues as needed. 2.6.4 SITE OBSERVATIONS The Evaluation Team visited a sample of the communities that have worked with RANO WASH and as suggested by the TAs. Community site visits often scheduled before or after an FGD entailed a short walk through the community to inspect the status of latrines, handwashing stations, and water infrastructure, as well as looking out for signs of behavior change communications, Grow-Up Stickers (GUS), and accountability mechanisms. Site visits to schools and health centers provided opportunities to inspect institutional latrines and boreholes and were usually accompanied by KIIs with head teachers, or health attendants (mostly mid-wives). These visits enhanced the team’s understanding of the nature and justification for RANO WASH types of interventions discussed during the KIIs and FGDs. The visits allowed the Evaluation Team to discuss issues one-on-one with a wider range of community members. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 11 2.7 DATA ANALYSIS METHODS Using the primary and secondary data collected, the Evaluation Team analyzed and synthesized information about RANO WASH’s performance to date in relation to the EQs. Using the Dedoose application (a secure online package to conduct qualitative data analysis), the team uploaded all qualitative texts from primary data collection for coding and analysis. The team then systematically analyzed the data from KIIs and FGDs to identify patterns that might highlight convergence or divergence of opinion across regions, activity types (e.g., water vs. sanitation service provision) and stakeholder groups. More information on the coding process is provided in Annex II. In addition to the analysis of interview data, the team also used secondary data from the desk review and RANO WASH monitoring, evaluation, and learning data to help explain the results and to triangulate findings from the qualitative analysis. 2.8 CAVEATS AND LIMITATIONS Team composition and field work. Due to several factors, the Evaluation Team consisted of three members instead of four. One of the local Evaluation Team members was replaced mid-way due to competing work commitments. In addition, fieldwork was postponed due to a COVID-19-enforced quarantine of one team member. Finally, because of the delay, another team member became unavailable. As this composition did not allow for the planned two-by-two interview teams, the three￾person team undertook many of the interviews together and most of the FGDs in tandem. Some interviews were undertaken by one team member in order to cover the full range of interviews and informants. Interviews with local masons and seamstresses and the local agents of the private sector water service provider were led by one team member with one exception: all team members were present in Anosibe Ifody. In the cases of Marofarihy, Nato, Vohitrindry, and Kelilalina FGDs, representative members of the different commune engagement platforms met one team member at the commune office. In all cases, the team produced audio recordings and finalized (and translated) notes based on the recording. Possible risk of investigator bias was mitigated by interviewing the same types of actors across multiple regions and communes, allowing for triangulation of actor viewpoints. Because the team planned to interview geographically disparate informants, they made regular tradeoffs between visiting village sites and undertaking stakeholder interviews at the commune level. Due to bad road conditions, they made small adjustments. On one occasion the team could not visit a small water system as planned, so the key informants met the Evaluation Team at a more central location in the commune. Scope and time limitations. As agreed in the inception phase, the evaluation focused on four of the six RANO WASH program regions, and 15 out of 250 program communes. This selection was not designed to be representative. The Evaluation Team notes that the narrow set of experiences likely omits some participant and stakeholder views and that any findings and reflections from the evaluation cannot in all cases be extrapolated or generalized across the RANO WASH program. The Evaluation Team sought insight on key issues and activities from as many people as possible, and supplemented it with information reported in RANO WASH quarterly and annual reports and other program documents. The Evaluation Team did not interview any staff from the Ministries of Health, Education, Decentralization, or other ministries with which the program has reported collaboration. While the team did initially foresee engagement with the Ministries of Health and Education, it was difficult to find people sufficiently aware of the ministry’s collaboration with RANO WASH. The difficulty is attributed to high turnover and reshuffling within the government. Ultimately, due to time constraints, the team decided to move forward and focus its energy on other parts of the evaluation. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 12 Interaction. Phone and online interactions presented data collection challenges. Due to COVID-19 restrictions, all national-level and selected regional-level KIIs were conducted virtually, generally using Microsoft Teams or a similar platform. Where feasible, the team encouraged the use of video-feeds for remote interviews. When internet bandwidth was low, the interview started with a video-feed during introductions and then continued with cameras off. In addition, when the Evaluation Team noticed that informants were struggling to express themselves sufficiently in French, the interview language was switched to Malagasy. All notes and transcriptions were then translated back to French. The risk of bias was counter-acted by extensive triangulation of facts and statements obtained from interviewees. Coding. The Team Leader read all interview transcripts in their entirety and coded them in Dedoose. Given time constraints and the volume of data generated from 103 interviews, the team members were unable to validate all of the coding individually. Interpretation of the coded excerpts, however, came at the drafting stage where the Team Leader tasked team members to draft sections using interview excerpts and data sourced from the desk review. Members were thus able to frame the findings as per their expertise. Having only one coder helped with consistency of code application across the data set, but limited interaction across the whole team during this stage. All team members have reviewed this document and contributed to the analysis. Scope. This review is not intended to substitute for a Data Quality Assessment exercise to inspect the accuracy of the RANO WASH monitoring and evaluation system. Achievements reported in annual and quarterly reports were taken largely at face value. The Evaluation Team looked for evidence that the types of interventions described in the documents were of sound design and implementation and contributed to RANO WASH’s wider objectives. The team did not verify if the numbers reported by RANO WASH were accurate or substantiated. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 13 3.0 FINDINGS This section presents findings and conclusions for the five EQs. Recommendations are embedded in the narrative and provided in list format in Section 4. First, the Evaluation Team’s assessment of the RANO WASH program is generally positive. The program is ambitious and innovative, and largely on track, given the complexities of what it set out to do. While water supply provision through the PPP model is off-track, there are plausible reasons for these delays, as discussed below. There are also important lessons to be learned. Mostly, the program is perceived positively by sector stakeholders, including national, regional, and local government representatives. RANO WASH is considered one of the leaders in the sector, particularly for its work on private sector engagement in water supply, and on issues of governance. Informants also noted that the RANO WASH team is highly knowledgeable and has wide ranging expertise in Madagascar’s WASH sector. This was also evident in the quality of the program’s reporting. The Evaluation Team recommends that the findings be considered as opportunities to reflect, learn, and sharpen the program’s direction as it moves into its final year, and not in any way as points of failure. 3.1 EVALUATION QUESTION 1 - RANO WASH DESIGN EQ1 - HOW APPROPRIATE IS THE RANO WASH DESIGN TO THE WASH CHALLENGES IN THE TARGET REGIONS AND COMMUNES? SECTOR CONTEXT AND RANO WASH RESPONSE. The WASH sector in Madagascar is strongly affected by the seemingly permanent state of flux at the highest levels of government. The WASH portfolio alternates between the Ministry of Energy, Mines, and Hydrocarbon and operating as a stand-alone MoWASH. Since January 2020, it has been a singular ministry. In general, as stated by a senior member of government, the WASH sector is important, but “somewhat neglected” by the other sectors, and as a result, “the place of the WASH Ministry as a lead in itself is not very comfortable, and its coordination not very visible.” The shuffle in government corresponds with high turnover, affecting RANO WASH capacity building and advocacy work. Similarly, at the regional and communal levels, the program’s ability to coordinate with local government was affected by the mayoral elections. On occasion, new local leadership sought to replace technical staff or service providers which caused complications. Yet, the program’s focus and design are highly relevant and deemed appropriate by the vast majority of informants. At the same time, informants acknowledge that the program may have initially underestimated sector challenges. The initial design had a clear objective: strengthen core aspects of Madagascar’s WASH sector including finance, coordination, governance, and dialogue. Reports including the Sanitation and Water for All (SWA) country profile and WASH BAT 2014, published around the time of the program’s launch, clearly outline these needs. As one informant said, “USAID’s design of the program (the bidding process) held up well to framework documents and aligned with departmental expectations ...” The program also set out to make significant contributions to improving WASH services and to align itself with sectoral WASH objectives, not only at the population level (percent of populations with access to WASH services, in Performance Monitoring Plans [PMPs]), but also at the institutional level (number of institutions achieving WASH-friendly status, in PMP plans). The 2014 WASH BAT report highlighted that the WASH sector in Madagascar was in a critical situation and presented relatively low scores in several areas, in particular, on sector governance, participation by MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 14 private sector, communities, and decentralized government institutions, and the overall ability of the sector to achieve its objectives. In its initial design, the RANO WASH program described several actions that it intended to implement and grouped these into three pillars: governance, the private sector, and the promotion of healthy behaviors. As outlined in the theory of change, the program recognized these three pillars as the entry points for changing sector dynamics and the main levers for improving WASH services. Compared to previous USAID projects which were narrower in scope, this ambitious project aimed to meet the sector’s range of challenges, and, in particular, support improved sector governance. As phrased by one sector development partner, “the advantage of the RANO WASH program is its institutional support component that will allow fairly consistent assistance to the MoWASH and other ministries.” In the area of private sector engagement, RANO WASH’s design aligns with the MoWASH. As one MoWASH informant put it: Currently the Ministry plans to do private management for all systems in Madagascar… The project meets this expectation of the Ministry… evolves with the Ministry to achieve this objective of professionalizing the management of water, sanitation and hygiene systems. The Evaluation Team felt this alignment is partly the result of RANO WASH’s own efforts, having worked hard to place private sector engagement at the forefront of implementation, and with it, ministry engagement. As a result, the Evaluation Team observed that, while acknowledging inputs and engagement by other sector partners on this theme as well, the ministry leans heavily on RANO WASH to inform its private sector policies and strategies, taking into account the unique nature of the type of PPP model promoted by RANO WASH. RELEVANCE AND INTERPLAY OF THE THREE PILLARS. While all those interviewed confirmed that the design of the program with three interrelated pillars was innovative and relevant, some questioned the timeframe, noting that five years was perhaps not enough to achieve real, lasting change, or paradigm shifts, particularly around the first two pillars, governance and private sector engagement. It is also clear that the initial conception was not sufficiently documented or clarified for interventions at municipal and regional levels. Several informant responses, especially at the local level, indicated an (initial) lack of understanding of the program’s approach, even among program staff and particularly in its early years. To some extent this may be linked to early internal program communications and induction of regional and local staff. (Section 3.1.3 on the consortium provides more information.) From a design point of view, the more important point is that the Evaluation Team could not discern from early program documents that references to activities that might be implemented jointly between the three pillars of the program were supported by a clear strategy or framework for harmonizing these joint actions. This made it difficult for program teams to assess the interactions planned between the pillars and even less so the sequence of actions to be carried out to achieve the expected results. The initial design clearly mobilized all the expertise developed by the members of the consortium, roughly split across the three pillars, but it also highlighted the need for these members to build common approaches to program implementation. The consortium members needed to produce detailed approaches and strategies that could translate the TOC into action and results. One key mechanism to do this was the consortium’s planned Learning Hub. Besides a broader focus on sector learning and informing sector policy, the monitoring, evaluation, and learning (MEAL) plan (revised version, May 2019, page 27) specified that common learning would be a tool to help in the coordination of activities between the three pillars and between the members of the consortium, with a learning plan that details the key issues to be addressed by the program. However, as discussed in Section 3.4, implementation of MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 15 the learning plans was delayed in the first two years of the program, also affecting the early operationalization of the three pillars. However, during the third year of implementation, the program initiated an internal reflection to organize its strategies around a holistic model. Systems thinking and the nine system building blocks were introduced as a tool to clarify approaches and to help decision-making on where to focus efforts. The program articulated activities and approaches for each building block, and regularly reviewed their progress as illustrated in the RANO WASH Mid-Term Review Report. At first glance, the building blocks approach is indeed an operationalization of the three-pillar approach and has the potential to clarify and help direct actions. However, as discussed in more detail in Section 3.2, it is still considered highly complex and not yet universally understood. FIGURE 4: RANO WASH BUILDING BLOCKS APPROACH TARGET SETTING AND AMBITION OF SCALE. The program team set its original targets, and the 2018 revisions, conservatively as compared to the total population of 1.5 million people in the intervention regions. It is understood that the rationale for this was to undertake a very ambitious spread of activities to ensure long-term sustainability of results and cover additional themes related to nutrition, water resources management, institutional WASH, etc. But given the scale of the population still to be covered in Madagascar and the size of USAID’s investment, the Evaluation Team felt that one of the key dilemmas in the design of this program could be summarized as ‘Range versus Scale’. Here, it is important to reemphasize that most of the respondents recognized the pertinence of the targeted changes and the themes covered by the program, in light of the state of the sector. Moreover, at the local level, the communities tend to demand even more from RANO WASH (specifically on drinking water), as illustrated by the types of questions collected via the program’s “Green Line” (a phone line set up to strengthen accountability and transparency) and reported in the program’s annual reports. These elements just confirmed the multi-dimensional character of WASH issues and solutions, the latter deemed to be multisectoral as well. What’s more, the Code de l’Eau (Water Code) designates local institutions with the responsibility for WASH service delivery, quality, and equity. In the face of their capacity gaps, a classic project approach MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 16 would seek to deliver the much-needed services, sometimes, in spite of institutions and at the expense of community ownership and scalability of results. Here, the RANO WASH program is notably innovative in fostering ownership and strengthening capacity of local institutions through participatory governance processes, to deliver services that last. This model is recognized as innovative and unprecedented for the WASH sector in Madagascar, as quoted by one of the consortium members. Rather than scale, the program’s value lies in its innovation and ability to drive learning in the sector. It serves as a bridge to several WASH sub-sectors and its connections lead to interaction between them. That said, as illustrated by UNICEF-WHO Joint Monitoring Program (JMP) data for Madagascar, the sector is in dire need of accelerating its performance, mainly on WASH service coverage. This aspect is then pushing the sector and RANO WASH to learn, and to learn quickly to go to scale by further strengthening institutions (commune, regional, and national), communities and private actors to take ownership. With this, however, also needs to come increased national government ownership. Programs such as RANO WASH are in essence designed to demonstrate viable approaches implemented at reasonable scale (such as at commune level), that can be further scaled under leadership of (and with investment by) national stakeholders, notably the GOM. The Evaluation Team noted that several senior government representatives interviewed directly requested or suggested the need for USAID/RANO WASH investment in other zones beyond the six regions and 250 intervention communes, without an explicit or implicit indication of buy-in or increased government spending. Currently RANO WASH only works in a few regions, so it lacks the other regions which are a little neglected. RANO WASH should expand its area of intervention if possible. – Director at MoWASH USAID planned a budget of 100 million dollars per year for Madagascar for the next 5 years, but only 9 million per year for WASH. – GOM Representative With this in mind, particularly given the unstable nature of (MoWASH) leadership, future program designs might do well to push for conditional government co- or counter-financing of WASH sector efforts, covering at the very least basic running costs and salaries of national and subnational staff, but ideally also contributing to costs related to WASH service delivery. COMMUNE SELECTION. With 250 intervention communities but limited budget to reach all, particularly with water supply and institutional infrastructure, choices had to be made. USAID prescribed initial region selection, and RANO WASH documents indicated that commune selection took place in close coordination with the DREAH in particular, using documents such as the Budget Programme par Objectif Régional (Regional Program Budget for Objective [BPOR]). The regional directors with longer tenure confirmed that their offices participated in RANO WASH commune selection, as was expected in line with their regional coordination role. But they also acknowledged that the presence of data and planning documents to inform such selection and intervention coordination, were still weak at the time of design—an admission reconfirming the relevance of RANO WASH’s engagement in government systems strengthening, particularly around coordination, planning and data. Furthermore, while RANO WASH staff were clear on the process and criteria used in the demand￾based approach to commune selection later in the program, this clarity was not necessarily reflected in the interviews with commune stakeholders. Some commune staff indicated they had been approached by RANO WASH and asked if they were interested, others did recall a meeting where they were asked to submit a type of application. The Evaluation Team observed that the information did not always reach other stakeholders in the commune. In particular, several informants expressed ignorance on why some communes and communities had been selected for infrastructure or rehabilitation interventions, and others had not. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 17 No infrastructure built by RANO WASH yet. This is one of the concerns of the mayor who raises this subject each time with RANO WASH, the project will soon come to an end but there is still no question of water supply infrastructure for us. Their response has always been let’s see. We have also requested the rehabilitation of two of our existing AEPGs. – ATEAH, Commune of Ambiabe Some municipalities have received drinking water supply infrastructure from RANO WASH. We wonder why the Nato Municipality has not received it when we really need it: we are very far from water, and the population is suffering from it. What are the criteria for selecting municipalities for the construction of drinking water supply infrastructure? – ATEAH, Commune of Nato Given the high level of local government staff turnover (more detail provided in Section 3.2), the confusion may be caused by the fact that informants were relatively new to the job. But, particularly with a demand-driven approach—where commune inclusion and decisions on the intervention type are directly linked to the initiative shown by that commune—the program needs to proactively respond to the risks of stakeholder staff attrition. The program also needs to ensure sufficient repeat conversations and communications on intervention scope and activities conducted. A side-point with regard to selection of intervention regions and communes, was a point of process: Some RANO WASH consortium partners indicated that in hindsight, a better strategy might have been to start work in all intervention regions at the same time, albeit in selected communes, rather than in waves. Not only would each region and regional team have had more time to work through the very complex and time-consuming processes of sector strengthening, shifting paradigms, building local ownership and establishing partnerships and contracts, they would have also been able to better contextualize interventions from the start, rather than transposing experiences and strategies developed in one region on other regions later. The Evaluation Team does not assume that work could have started simultaneously in all 250 communes, but asserts that starting regional teams at the same time might have been both feasible and smart. The Evaluation Team recommends that future implementers explore this approach during the next round of programming. 3.1.1 COLLABORATION From its original Technical Proposal, it was clear that the RANO WASH program foresaw a lot of complementary collaboration with a range of sector partners, including with other USAID-funded programs. Successful coordination and collaboration require upfront intent and deliberate design, so the Evaluation Team reviewed the program’s design from this angle. At the time of RANO WASH design, the Diorano-WASH Coalition, a sector coordinating mechanism established by the GOM in 2003 with Water Supply and Sanitation Collaborative Council (WSSCC) support and with a Secretariat hosted by WaterAid, was still active and the program expected to engage with the mechanism. However, the then minister in August 2017 disbanded Diorano-WASH, instead proposing a government-driven coordinating mechanism. Support to establishing and shaping this WASH Sector Promotion National Platform was written into the 2018 Annual Work Plan, but the mechanism was never formalized following further shifts at the ministry level. Nonetheless, the clear intent of RANO WASH to strengthen sector coordination and collaboration was visible in its Results Framework, particularly linked to Output 1.1.1 Sector coordination and learning mechanisms operating effectively under strong national leadership, and Output 3.2.1 WASH behavior change program coordination improved in RANO WASH regions. In addition, RANO WASH was designed with a clear focus on contributing to sector learning and advancement of policies and practices. This is clearly illustrated in Output 1.2.2 Learning agenda implemented to increase and better regulate private sector engagement in WASH, and Output 3.3.1 National-level networks, policies and programs engaged for sustainable WASH behavior change. Whether and how these learning activities were successfully implemented is discussed further in Section 3.4. The Evaluation Team’s view is that the MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 18 design did show a clear intent to collaborate and coordinate with all sector stakeholders, including other External Support Agencies (ESAs) and CSOs, which is especially important and relevant in a context where governance instability can easily disrupt sector progress and direction. At the regional level, RANO WASH set out to invest in the Regional Implementation Coordination Mechanism (Structure de mise en oeuvre de la coordination Régionale [SRMO]) as a key coordination mechanism, and, where relevant, perform the role of co-lead with the DREAH, building the latter’s capacity and intent to ensure continued regional sector coordination. This strategy was also plagued by frequent turnover. One RANO WASH implementing partner in Alaotra Mangoro, remarked that they saw the arrival of four DIREAH in as many years. Nevertheless, the Evaluation Team assesses that the intent to continue to focus on strengthened sector coordination was the right strategy to take. With regard to direct collaboration with other sector partners, while there is evidence of early engagement with other USAID programs and the Fonds d’Appui pour l’Assainissement (FAA – Global Sanitation Fund program) around joint learning to inform program strategy (2018 Annual Report), various KIIs pointed out that in practice, not all these collaborations took shape as planned or were limited in scope. For example, the collaboration with the USAID-funded SHOPS PLUS program regarding capacity strengthening and facilitation of access to finance for private sector enterprises was said to not have seen any real activity since 2018/19. The collaboration with FAA regarding CLTS and Follow Up Mandona (FUM) capacity building, mentoring and implementation initially took off with FAA for example training many of the RANO WASH implementing partners on CLTS, but then each program went its own way for reasons not quite explained to the evaluators, but seemingly related to a difference in approaches adopted. Conversely, a number of KIIs remarked that, particularly thanks to the active role played by the various DREAHs, the program had avoided overlap with other WASH programs in intervention communes and communities. However, it also became clear, particularly from interviews with many of the commune and community-level stakeholders, that RANO WASH intervened in many communities that had seen prior engagement from other programs, for example on sanitation behavior change but also on infrastructure provision. In this respect the Evaluation Team found it difficult to confirm the extent to which the sanitation results reported by RANO WASH were informed by or built on prior efforts by other partners. Particularly at the national level, but consistent overall, RANO WASH clearly took a flexible, responsive, yet proactive approach to sector collaboration and coordination, which was acknowledged and commended by many of the KIIs. This is further discussed in Section 3.2. In the eyes of the Ministry, the RANO WASH program is somewhat of a leader among the other actors, even though there is collaboration of other stakeholders such as UNICEF. There is an institutional arrangement within the framework of collaboration…the project is a bit distinguished. – Senior Government Official 3.1.2 GENDER, SOCIAL INCLUSION, AND VULNERABILITY From the earliest program documents, it is clear that the consortium, and CARE as the lead agency, planned to take a deliberate and considered approach to gender mainstreaming and inclusion of women, girls, boys, and men. As summarized in the 2020 Annual Report, for example, the program undertook a range of exercises to inform its strategies and activities, particularly with regard to SO1, including developing a gender strategy, undertaking gender equity analysis of RANO WASH stakeholders, gender staff mapping, a mid-term self-evaluation of gender mainstreaming and social inclusion, a focus on strengthening and increasing participation in decision-making processes for men, women, and youth, social accountability mechanisms to be used by all, and training for WASH service providers. There is MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 19 ample evidence that these strategies were built on a solid understanding of Madagascar gender relations, traditions and power dynamics. However, in its sanitation behavior change programming, the Evaluation Team found less evidence of a deliberate inclusive approach going beyond gender, age, and physical disability. For example, it did not find signs of explicit identification, targeting, and support for different types of potentially disadvantaged and vulnerable populations within the context of CLTS, nor of careful consideration and explicit guidance on potential harmful effects. Section 3.4 will go into more detail on the practical application of gender and inclusion concepts in the program’s work, as well as introducing some further sector thinking on the subject. The Evaluation Team also noted an absence of specific gender or social inclusion indicators in the RANO WASH MEAL Plan (Revised May 2019), despite a set of indicators proposed in the April 2019 RANO WASH Gender and Social Inclusion Mainstreaming Strategy. While particular key outcome indicators (e.g. the access to water and sanitation indicators 2.2.1 to 2.2.4) were said to be disaggregated by sex and wealth quintile and USAID confirmed these data are provided to them, this disaggregation is not included as a standard practice in Quarterly or Annual Reports (narrative or in PMP annexes, with the exception of the FY21 Q2 report), but was discussed in the RANO WASH Mid-Term Review Report. Similarly, while disaggregated process indicators have not been included as standard practice in Annual or Progress reports, it is acknowledged that such data is shared with USAID. The most pertinent question the Evaluation Team asks with regard to equity and inclusion concerns the balance between the program’s focus on private sector management to improve service quality and sustainability, and the model’s ability to reach and include vulnerable households – particularly acknowledging the high levels of poverty and vulnerability in Madagascar’s rural areas. Choice of communes and intervention sites was informed by location and size of serviceable populations, to permit the private sector enterprises to run a viable scheme and demonstrate the potential of the Construct-Invest-Manage model. Given the very dispersed nature and small size of villages or hamlets in much of rural Madagascar, servicing such villages is not (yet) interesting or viable for the private sector, which left many such areas outside program intervention. The Evaluation Team acknowledges that the “PPP+” model partially seeks to address this issue, by focusing on extensions to villages and hamlets beyond the initial scheme perimeter. Document review pointed out that the current diversification of types of social connections was developed later on in the program to cater for more vulnerable households. Similarly, the KIIs pointed out that in the large majority of cases, the program’s promotion of the first 100 connections was used as a marketing drive rather than an opportunity to focus specifically on connecting most vulnerable households to the scheme. While it is fully understood that the program set out to shift paradigms on private sector engagement in the WASH sector in Madagascar, it is also acknowledged that the ultimate aim of the program was to ensure sustainable and equitable access to WASH services for a growing part of the population, therewith contributing to national goals. In that light, the Evaluation Team suggests considering the following suggestions, summarized into future design recommendations in Section 4: ● Consider a district or commune-wide approach to water service provision to ensure inclusion, acknowledging also that this is in line with government aspirations and was suggested by several MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 20 senior government and ESA key informants.2 This would need to start from business planning, possibly even before the Market Assessment Studies and before identifying a WSP, to ensure that an exclusive focus on profitability does not increase the risk to be non-inclusive. While such an approach could continue building on the sector drive—and RANO WASH experience—to increase private sector engagement and investment in the sector and ensure private sector efficiencies in water system management and operations, such planning would need to start from the desired goal of providing water services to all villages and fokontany within a district or commune, those that could be profitable and those that likely could not, and explore ways of how this could be done. This may, initially, also mean an agreed mix of private sector and community managed approaches, with many possible ways for the private sector provider to engage with, partially oversee, or progressively take-over the community-managed schemes—some already being explored within the current PPP+ strategy (see Section 3.2 for further discussion of some of the options). ● The Evaluation Team would suggest that the program try to capitalize on its achievements to date and, still within the current program timeframe, start supporting MoWASH in identifying the first districts most able to demonstrate this approach. The team acknowledges that the development of such multi-year and multi-dimensional business plans would likely require considerate technical support at the start, for example around planning and financial modeling to build economic scenarios, developing technical solutions based on existing data in the BPOR and Budget Programme par Objectif National (National Program Budget for Objective [BPON]), and building on business plan models such as those developed by Sandandrano. To have a greater impact, RANO WASH would have to work at the district level… that is in the objectives of the Ministry. – Director, MoWASH ● Explore and advocate for different subsidies or financial support to WSPs to offer more affordable connections to vulnerable households. From the KIIs undertaken at community and commune-level, the highest cost barrier for households to connect to the water scheme was often said to be the water meter (compteur), rather than the recurring tariffs. RANO WASH together with the MoWASH could look for ways to subsidize the water meter and/or connection cost, or through a reduction of tax at source on these types of materials, allow WSPs to offer the meters/connections at lower cost. Another option to explore could be an annually decreasing subsidy to WSPs to allow and encourage them to connect vulnerable households early on, possibly setting annual household connection targets for them to achieve to receive the subsidy (a form of results-based funding). 3.1.3 THE CONSORTIUM While not originally foreseen as an evaluation sub-question, during the KIIs, the Evaluation Team took note of a number of reflections and concerns on the shape and functioning of the RANO WASH Consortium itself, which it considered relevant for an overall assessment of the design of the program. The collaboration of three key sector agencies in this consortium was seen mostly as a positive, particularly denoting each agency’s complementary expertise in one or some of the three program pillars, their long-term experience in the sector in Madagascar, and the additional support received from their international head offices. The inclusion of private sector partners in the core-team was also mostly seen as a positive. The long-term engagement and sector experience of each partner clearly contributed to the constructive relationship with the MoWASH and Regional Directorates, as well as 2 The “Approche District” was mentioned in these terms by several informants including senior government representatives. While it is acknowledged that in the Malagasy setting, districts may not be the most appropriate level of focus (with WASH technical structures either positioned at Regional or Communal level), in this report we will continue to use the term ‘district or commune-wide approach’ in line with the going sector language. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 21 facilitating easy collaboration and coordination with many of the other development partners. Ministry stakeholders also commended the level of expertise, passion and commitment of the individual RANO WASH team members. However, some concerns were also noted—listing only those mentioned by more than one consortium member or implementing partner: ● The three-layered set-up, with the PCT in the center, the Regional RANO WASH Team and the implementing partners in the field, is reflected to cause delays, with references to a top-down attitude of the PCT often requesting changes in workplans already underway and agreed with the regional lead partner, or causing delays due to required approvals from the PCT. This was also said to refer to fund management and sign off on use of funds. In this context, the efficiency of the ‘double’ regional set-up with regional offices of the regional lead partner (CRS, WaterAid or Care) and separate regional RANO WASH teams was also pulled into question. ● KIIs suggested that there is scope for improvement of communications and the relationship between the PCT and the consortium partners. In this respect, where some of the PCT members for example commented on the fact that all RANO WASH actors currently share the same vision of the implementation of the program and are all convinced about the application of the systems approach at all levels, this was pulled into question by some statements made by regional or local RANO WASH partners indicating either a lack of understanding of or comfort with operationalizing the systems approach or, in one case, with the focus on private sector engagement itself. With regard to coordination, we feel like the implementers of the approaches of the PCT, without being able to include our own experiences from other projects. – Implementing Partner The Evaluation Team however also acknowledges this concern is known by the PCT leadership, with a KII citing for example a “mosaic of activities where even the field teams have difficulty articulating them, defining the best sequence of events or understanding the various related implementation risks.” ● The structure is considered “heavy,” particularly with regard to procedures for validation, approval, and shifting of funds. This, combined with cited “leadership challenges” at the start of the program, was said to have contributed to some of the delays in delivery of planned program results. Several KIIs, including government partners, suggested that the consortium should seek to lighten some of its procedures, both within the consortium and with regard to ways of working with the various public and private partners. With regard to the latter, some of the enterprises currently involved in the PPPs mentioned a level of demotivation, for example citing the prolonged selection and contracting procedures. In this respect, continued dialogue and reflection between partners is essential. It was also suggested by a senior government stakeholder that some such procedures “should be discussed with MoWASH before deciding.” The execution price has increased from the forecast price in the offer without any price revision. The slowness of the process can cause the company to go bankrupt. At the time of the offer the price for a bag of cement is 24,000 Ariary, at the time of execution it is 29,000 Ariary. The 90-day offer validity is largely exceeded (by almost two times). This processing time should be improved, and the requested surety of 15% further demotivates companies. The most financially powerful are favored. – Private Sector WSP MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 22 3.2 EVALUATION QUESTION 2 - GOVERNANCE CAPACITY EQ 2 - TO WHAT EXTENT HAS RANO WASH BUILT GOVERNANCE CAPACITY TO IMPROVE SUSTAINABLE SERVICE DELIVERY AT MULTIPLE LEVELS? 3.2.1 STRENGTHENING NATIONAL GOVERNANCE CAPACITY Policy and strategy. Since 2018, RANO WASH has provided continuous support to MoWASH at the national level, specifically on writing sector policies and strategies and fostering sector dialogue. The program has been part of significant sector opportunities to influence the Ministry through policy and sector wide reflections, including the WASH BAT process, the Sector WASH Policy, BPORs, and the Madagasikara Madio 2025 roadmap, coordinating its efforts with other key contributors. The program provided significant support in consultation processes around the PS-EAH and started to realign its approach within the newly crafted building blocks for a strong WASH system. In 2020, following onset of the COVID-19 outbreak and subsequent restrictions, the program also provided technical support for the MoWASH to effectively organize cluster WASH meetings and other sector dialogues via video conference. These efforts were acknowledged by sector stakeholders and raised the visibility of the program as a key player in the sector. As discussed throughout this report, the program faced FIGURE 5: GOVERNMENT challenges during its first three years of implementation, DEPARTMENTS IN CHARGE OF specifically in the areas of governance and policy support. WASH SINCE 2008 Problems originated from three factors: 1) frequent July 2008: creation of Ministry of institutional changes of the department charged with Water overseeing WASH activities; 2) sector activities shifted focused as the institutions leading activities changed; and 3) March 2009: fusion with Ministry of national policymaking processes slowed down as new Environment ministerial staff were appointed, stalling policymaking May 2009: second birth of Ministry of processes. Water; Period of relative stability from 2007 until 2017 Though RANO WASH invested continuous support in the WASH sector policy process, the government never April 2017: fusion with Ministry of released the policy document and further policymaking HydroCarbon and Energy stalled with the most recent iteration of the ministry. The June 2018: third birth of Ministry of COVID-19 pandemic also stalled the processes. WASH Respondents mostly recognized RANO WASH for its February 2019: re-fusion with Ministry successful work on the PPP contract model implemented of HydroCarbon and Energy under SO2. Several respondents acknowledged RANO January 2020: fourth rebirth of the WASH has helped the ministry to craft a clear vision for Ministry of WASH WASH PPP programming. Additionally, informants noted that the ministry is already seeking to capitalize on this success on new challenges around urban settings. In parallel, the program also supported several initiatives implemented by the Ministry of Education and the Ministry of Health concerning the WASH-friendly institutions approach, by sharing program experiences related to its implementation and application. The Evaluation Team found several mentions of WASH components in Education and Health Policy documents from 2017. Similar to the problems created by MoWASH’s turnover and institutional changes, the Evaluation Team observed programming challenges within these ministries as well, due in large part to reorganization and attrition over the last two years. The Ministry of Education was reorganized most recently in September 2020 and split into two departments. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 23 Budgets and finance. The Evaluation Team found varied evidence that RANO WASH has continuously advocated for improved sector financing. Among others, the program helped to draft the TrackFin 2020 report by providing logistical and technical support to the national consultations launched in December 2020 as part of this report. But most of the program’s contributions in this area focused on the commune-level and are further detailed in Section 3.2.2.. Planning, monitoring, and SE&AM. The program provides substantial technical support to the SE&AM system as well as to BPOR planning tool, two critical components of MoWASH’s monitoring, evaluation, and learning system. Notably, these tools were adopted by the ministry following WaterAid’s consistent advocacy efforts over a period of 10 years. Key informants appreciated the program’s efforts to improve the quality of the SE&AM system. In addition, ministry informants welcomed the program’s involvement in MoWASH’s monitoring, evaluation, and learning portfolio and said the program’s impact was visible, as described in Section 3.2.2. That said, challenges remain. We don’t have a simple tool. It was well known that the SE&AM database is really complicated. We need to find a way to build capacity of MoWASH teams and make sure that this capacity remains within the Ministry and is less and less dependent on external consultants. We also need to reinforce the IT infrastructure with servers, computers for the Monitoring and Evaluation department. – National Government Representative In this same vein, several capacity-building activities (evaluations, technical support) as well as dialogues were organized with the support of RANO WASH to help improve the BPOR/BPON planning tools. Respondents recognize the program’s significant technical support to the ministry, but they also raise the fact that the complexity of current systems is preventing actors from using them systematically. Here too in 2020, the program’s plans to provide support to SE&AM and BPOR planning encountered institutional hurdles due to government staff turnover. COVID-19 restrictions compounded the issues as all national discussions and initiatives ground to a halt. National level respondents, however, praised the program’s effort to turn this situation into an opportunity to demonstrate the use of a simpler tool to collect and process data around the pandemic. Several partners indicated there is no common vision and strategy for sector planning and a monitoring and evaluation system. With this in mind, the Evaluation Team recommends that the program engage in a more in-depth assessment alongside MoWASH and sector partners. The assessment should include a review of the root causes that hinder successful implementation of sector monitoring and planning, beyond challenges related to the SE&AM tool. It should also investigate the capacities and resources needed to set up a solid sector-wide monitoring, evaluation, and planning system that could be used by all as a data source to support decision making, and longer-term planning and sector focus. As an aside, the Evaluation Team noted that the program has supported a reflection around financial planning tools such as life-cycle costs or WASH costing tools. However, very few respondents mentioned the utility or even the existence of these tools, highlighting the need for the program to share more information about the importance of such tools. National Institutional Arrangements. Though the Code de l’Eau (Water Code) designates local institutions with the responsibility for WASH service delivery, quality, and equity, several respondents at the national level mentioned the difficulty for the sector to comply with these arrangements, especially around the effective delegation of the ‘maîtrise d’oeuvrage’ (‘project manager’) role to the communes. While concerns often linked to local capacity levels, some respondents noted that it is rare for people to know the Code or have mastery of its components which, in turn, leads to confusion for those trying to implement it on all levels. These problems have contributed to inaction in revising or updating it. In this context, the program drew on expertise from the consortium members to initiate discussions with the MoWASH to strengthen the role of the private sector and consolidate the vision of the MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 24 MoWASH on the PPP model. Given that the State has published a policy letter on decentralization which ultimately aims to effectively transfer the responsibility and management of social services to the commune level, this presents an opportunity to contribute significantly to strengthening the skills of the decentralized bodies and therefore the sustainability of the achievements of RANO WASH. The program could capitalize on its experience working with communes in the areas of governance, private sector engagement, and STEAH strengthening to deliver a set of best practices, lessons learned, and risks to the Ministry of Decentralization and MoWASH. The program could also produce or update a guide for mayors, STEAH, and DIREAH around WASH service delivery. That said, there are still many aspects provided for by the water code that remain to be exploited, such as the integrated management of watersheds, which has the potential to both sustain water resources and open up avenues of rehabilitation, collaboration with environment stakeholders and the management of climate hazards. RANO WASH’s early engagement with the People, Health, and Environment network is acknowledged in this regard. However, such an enterprise cannot likely be implemented in the current program, even with an extension. The Evaluation Team recommends that USAID and the Ministries of WASH and Environment jointly explore future programming in this area. Rather than implement along this workstream, RANO WASH could help facilitate conversations that might shape the design. The program could do so by leveraging its experience working with the BPOR/BPON tools as well as joint work on forecasting the installation of electric dams with the Ministry of Energy and Mines. National coordination mechanisms and dialogue. All informants welcomed the example set by RANO WASH in supporting the MoWASH to play its role of sector lead. This was also highlighted by ministry respondents themselves, and includes technical support in the facilitation of sectoral dialogues linked to initiatives such as WASH BAT, TrackFin, or the SWA high-level meeting. The Evaluation Team recommends that the program continue serving as a facilitator at the national level. In addition, given that national sector reviews have not been carried out for more than two years, the program is well￾positioned to propose a joint sector review for 2021 or early 2022, and to help in its organization. We asked, insisted, that from this year there should also be a sector review. It has been 2-3 years since there has been a sector review. – Civil Society Partner All the investments made to facilitate sectoral dialogues, whether face-to-face or remotely, will gain added value by capitalizing on them in improving coordination. The Ministry has expressly formulated the need for long-term technical support to form a national coordination mechanism based on the lessons learned by the sector. In that respect, the minister suggested, “In general, there is better collaboration and consolidation between sector partners now, than before.” Several other sector partners echoed this sentiment in interviews saying, “what we have seen, although it also depends on who heads the ministry, is that there has been an improvement in the inclusiveness of partners, in discussions and decisions.” Notably, the majority of national sector partners and many MoWASH representatives credited RANO WASH with contributing to this shift in some form or other: In general, there has been a 180 degree turn in the atmosphere between the partners. Before there was a lot of conflict between the partners, but now that has changed and the partners are more aware that they are ‘forced to work together’. Especially there is much more facilitation and collaboration with regard to the support to the ministry. RANO WASH took a lot of initiative for that, they invited others, created and contributed to a better space to share. – Sector Partner Involvement of private sector actors. The Evaluation Team found the actual involvement of private sector actors in national-level WASH sector dialogue, policy, and strategy formulation; planning and monitoring; and finance to be limited. While the private sector partners of the RANO WASH MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 25 Consortium themselves have been directly engaged in the work with the MoWASH on the standard PPP modality contract (the “contract type” in French), engagement of other WSPs in sector dialogue, or joint sector learning events, is case based. As confirmed by RANO WASH management and by the financial institutions interviewed, there is also an urgent need to engage in further, more concrete dialogue on the possible role of financial institutions in supporting private sector enterprises who want to invest in the WASH sector, and the ways in which the GOM could support and facilitate this better, particularly given that the public sector is the ultimate owner of any infrastructure put in place. When the Evaluation Team talked to bank representatives, they expressed willingness and interest in working with private sector providers on WASH, but pointed out that government leadership is the key to any success. One concrete example of collaboration given by the banks is that of a government-backed guarantee fund, as currently exists in the agricultural sector through the presidential program fihariana. Notably, both financial institutions the Evaluation Team spoke with indicated that initial conversations with RANO WASH had not led to any further engagement beyond some joint trainings, and that more discussion was needed to understand incentives, key rules of engagement (including banks’ possible preference for an exclusive relationship with the program, as was expressed by one of them), and the most appropriate instruments to use. They felt the ball was largely in RANO WASH’s court to discuss any next steps. The institutions also suggested they should be involved earlier in the selection of private enterprises to ensure that the enterprises involved would qualify for investment by the financial institutions. Given the program’s limited time remaining and its work with the ministry on its private sector engagement policy framework and model contract, and with a view to continued sector engagement in PPPs, the Evaluation Team recommends that the program restart its conversations with the banks. At a minimum, the program should document its conversations to-date, including incentives for mutual engagement, limitations and concerns, and recommended next steps for further collaboration. RANO WASH could also facilitate a discussion between banks and the MoWASH or other relevant Ministries, to explore the idea of a government-backed guarantee fund for WASH. 3.2.2 STRENGTHENING SUBNATIONAL GOVERNANCE CAPACITY RANO WASH invested heavily in building regional and commune-level governance capacity. The program also worked hard to establish and support structures that could sustain that capacity and the WASH outcomes achieved during implementation. While work is ongoing across all nine systems building blocks, most visible are the efforts to ensure commune-level WASH technical capacity. These include building dialogue, advocacy, and feedback structures, strengthening planning and monitoring, and increasing WASH budgets. Technical capacity and human resources. RANO WASH has invested heavily in training mayors, and this was gratefully acknowledged by some of the mayors, particularly those of the first or second ‘waves’ of training provided. However, an obvious problem has been the high turnover of mayors, particularly after the most recent mayoral elections. The Evaluation Team spoke to some newer mayors not yet fully acquainted with or trained by the program. They shared their skepticism which the team attributes to poor understanding of program objectives. If the RANO WASH program wants to make its mark here, it would have to put in place at least some infrastructure, even a garbage bin or public toilet for the market. It is really contrary to the works of the Muslim Associations here which really help the communities. – Newly Elected Mayor To counterbalance some of the political fluctuations, RANO WASH has also focused on building up the capacity of STEAH and ATEAH within the communes. The program layered its approach in the MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 26 communes this way assuming that even as elected officials rotate out of office, the ATEAH will remain constant. ATEAH are key actors in RANO WASH’s strategy. Per the STEAH Training Cycle (Annex 7, Annual Report 2019), and confirmed by the ATEAHs interviewed, they have a wide range of responsibilities. Their positions are technical and require extensive trainings, coaching, and mentoring to be fulfilled. Among others, ATEAHs are responsible for overseeing water services; managing the relationships with service providers; monitoring and collecting data; protecting the watershed; implementing CLTS and FUM co-facilitation; and providing overall sensitization, follow-up, and engagement with all the local and communal structures. The majority of ATEAHs interviewed balance this work with a second job and many interviewees had not been paid, or paid sufficiently for their work. Thanks to RANO WASH’s insistence, nearly all interviewees held an annually renewable contract with the commune, and in all cases were considered commune staff. Here as well, turnover is high and the Evaluation Team learned that several ATEAH had been replaced with new ATEAH upon the arrival of new commune leadership. Most of the ATEAH interviewed indicated that they think the employment is unstable and dependent on the favor of the mayor. Most perform the role from of a sense of duty or personal aspiration without a clear framework for their retention. Since April 2021, there were two ATEAHs before me who left the program . . . At first, it was our TA who explained to me my duties and what I should do … It was the mayor who contacted me directly. He explained the situation to me and that he wanted me to replace the person who was no longer there. It was decided very quickly, the town had no people. I started on a Saturday and the following Monday I had to attend training already. – ATEAH Commune de Nato No, I don’t get compensation, I don’t even get a salary. – ATEAH Commune of Ampasipotsy Gara Many of the interviews conducted with mayors, adjunct-mayors, and ATEAHs, show that the communes do not (yet) see the clear participation of the Regional WASH Directorates and the Ministry of Decentralization in the STEAH program of work. The commune has not yet seen DIREAH support so far. Two reports have already been sent to DIREAH over one year but it is still RANO WASH who supports the municipality. DIREAH is expected for construction, training and awareness…. but only controls the quality of the water. – Mayor of Beforona In addition, there was a strong sense that the RANO WASH program itself was driving a lot of the activities, along the way building capacity of the communes and DREAH, but perceived as largely pulling the cart from upfront rather pushing from behind. This is noteworthy given the program’s intention to build local ownership and capacity to ensure long term sustained sector management once the program closes, but does need to be nuanced by the fact that most people spoke positively of the importance of the roles they now played. It does however point toward the need for a well-balanced exit strategy, possibly withdrawing support in stages. The municipality chose me when RANO WASH asked to have someone from the municipality as ATEAH to set up the technical service. – ATEAH Commune d’Antsapanana When RANO WASH arrived, they took people by surprise. They spoke of putting in place committees, structures. They spoke of putting in place committees for the infrastructure that RANO WASH had not put in place. – ATEAH Commune de Marofarihy. While the Evaluation Team fully agrees with RANO WASH’s strategy of reinforcing the STEAH as the most local layer of WASH technical service, the Team sees important risks with the approach. Risks fall into two categories: assumptions placed on the ATEAHs’ capacity, and sustainability plans. For the STEAH to provide a long-term service, the STEAH need to be formalized within the government’s institutional set-up at the commune level. Continued advocacy to formalize their role with MoWASH MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 27 and the Ministry of Decentralization is crucial. STEAH should also be included as a line item in communal budgets. One alternative suggestion from an informant is to explore linking the STEAH directly to the DREAH, so the DREAH would employ and oversee the multiple commune-level STEAH in their region, easing the STEAH/ATEAH dependency on commune-level politics and leadership turnover. However, it would have to be explored how and at which level the ATEAH salaries would be secured in such a scenario, and high turnovers experienced at the DREAH could further negate the plan’s goal. It may be more appropriate for the program to instead explore other mechanisms to strengthen lines of engagement and dialogue between the DREAH and STEAH. An additional finding from the interviews with local stakeholders was that mayors seemed to hold a lot of information to themselves, and that even with the existence of an SLC and presence of an ATEAH, there was a lot of information these stakeholders were not privy to. Examples included the status of the Commune-Level WASH Development Plans, (the PCDEAH), or the discussions with RANO WASH on possible program interventions. Planning, monitoring and review. With regard to planning, RANO WASH has focused its support mainly on the development of the PCDEAH. The program sees this as step that is equally useful in all regions it works in. This includes communes where RANO WASH has supported water supply infrastructure, and in communes where it has not. The PCDEAH would provide the communes with a ready-made tool to seek further investment from other development partners or government. Yet in reality, only three of the 15 communes visited confirmed that the PCDEAH is final and available at the commune. The remaining 12 communes report that the plan is in various stages of drafting, existent in draft form but still to be validated, or ready but never seen by local actors. Further, for communes not eligible for water infrastructure support under RANO WASH and those with extensions remaining to be done, it is not clear what the actual scope and perspective is for communes to successfully fundraise for their plans, without systemic support of and inclusion in regional and national planning processes. If the program believes that there are funding opportunities for these communes to be explored, even if limited, then the program should consider providing training on negotiation techniques and fundraising/ resource mobilization together with its PCDEAH development support. Then, the plans are more likely to be realized. I do not really know the usefulness of this PDEAH, but it is perhaps like the other PCDs, that we could show to donors to raise funding. – Mayor Ampasipotsy Gara With regard to monitoring, RANO WASH has made concerted effort to build regional and commune￾level capacity to use the SE&AM system, including training the ATEAH to collect data for reporting to the system every three months. All ATEAH interviewed confirmed this to be one of their tasks. The majority enter the data manually at the commune level and send the data to the RANO WASH TA and/or submit it directly to the DREAH. ATEAH said that their monitoring and reporting focuses mainly on the quality of water, measured by sight and smell, latrine coverage, hygiene status of villages, and checking fees paid by the WSP. Some ATEAH reported that regular checks on the status of WASH, particularly with regard to latrine coverage in communes striving to become ODF, can be useful. However, the Evaluation Team found that SE&AM data currently only has limited statistical purpose, if any, at the commune level. Also, while DREAH have computer equipment and an internet connection, the data fed into their systems is collected by hand, increasing the likelihood of transcription errors, as recounted by the DREAH of Vatovavy Fitovinany. The fact that SE&AM does not work very well at the regional level because everything is centralized, was also mentioned as a limitation. Regional and communal budgets, expenditure, finance tracking. Despite evidence of RANO WASH efforts and advocacy, DREAH financial capacity is still very limited. For example, while one DREAH indicated that they now budget for SRMO meetings without help from RANO WASH or other partners, that same DREAH pointed out a large shortfall of funds to properly carry out its regional MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 28 oversight role. It subsequently requested a larger consideration and contribution from RANO WASH. Continued work with national and regional governments to increase DREAH budget remains a priority. As reported in the FY2020 Annual Report, 27 communes reported an increase in their WASH budget, and 97 communes reported they had a WASH budget line. Annual WASH budgets were said to, “vary from MGA 170,000 to MGA 25,000,000, ranging from the salary and indemnity of the STEAHs to the activities of repairing works or construction of latrines and water points.” The Q2 2021 report stated, “an analysis of 151/250 communes that submitted information on their 2021 primary budgets showed that 138 communes have a WASH budget, and 111 communes have increased their WASH budget for 2021.” At the same time, “only 45 of the communes that increased their WASH budgets were able to transfer their budgets to RANO WASH in time to be included in our tracking system.” Importantly, this reporting pertains to budget information rather than expenditure tracking. Based on field interviews, the Evaluation Team saw limited evidence that new funding was mobilized beyond RANO WASH funding and household contributions. The examples of ATEAH salary shortages illustrates this point. The registered commune budget is 280,000,000 Ariary (operation and investment combined), but the commune only has 100,000,000 Ariary. – Mayor It has been six months since the staff of the commune received a salary. – ATEAH In all communes the team visited where a WSP serves under a PPP scheme, the fees (‘redevance’) are being paid to the commune by the private provider as planned. These funds are being used, among other things, to cover the ATEAH salary, at least in part. The team confirmed that some communes have been able to set money aside to cover expenses such as extensions of the water supply system. This aspect of the PPP modality shows clear potential to contribute to sustainable financing of communal WASH budgets. In addition, the Evaluation Team agrees that the RANO WASH strategy to train commune officials on tax collection as a means to increase the municipal funding base for public expenses is an important part of the systems building approach. But many of the communes visited by the team had not yet managed to collect taxes from local taxpayers and economic actors. Learning and transfer of good practices from more advanced communes deserve to be further exploited; for example, the experience of Niarovana Caroline and Ranomafana Est. This is also an area where RANO WASH should define its exit strategy, and identify and engage with actors beyond the WASH sector or identify other programs that could ensure continuation of the tax collection efforts set in motion. Coordination, dialogue and advocacy structures. Multiple national and regional actors commended RANO WASH’s support to reviving or setting up SRMOs. Overall, interviewees said that regional SRMOs that have received support now work effectively, and that they would likely continue to do so when the program exits. RANO WASH has left its mark on local consultation and dialogue structures. The program has spent time and energy to form and strengthen these important community WASH building blocks, and the impact was visible during regional visits – there were operational SLCs and/or OSCs in all 15 communes. This support was one of the elements most often mentioned by informants at the commune level as a key contribution of RANO WASH (after the provision of water infrastructure but often before its work on sanitation behavior change). The most common structures said to be put in place or strengthened with RANO WASH support were the SLC, OSC, ASUREP, and WASH committees, as well as systematic engagement with Voamami/VSLA. Yes, it was RANO WASH who set up the OSCEAH, the program told us about the importance of setting up this structure. We invited all the actors of the Commune and those who are motivated were appointed as members of the board. – OSCEAH Beforona MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 29 Despite the positive reviews, the interviews nonetheless gave rise to some risks or concerns: • The structures put in place, particularly the OSC and SLC are often ignored by the commune leadership, or their suggestions are not taken seriously. This was more evident in communes where there had been a recent change in leadership. This finding points to the continued need to sensitize new leaders to the uses and, in the case of the SLC, obligations of communes vis-a-vis these coordination and dialogue mechanisms. • In communities with a private sector WSP in place, the structures occupy an oversight role and represent the communities in discussions regarding, for example, poor service or water quality complaints. Structure interviewees indicated that they were not privy to or had no reference documents to frame their role regarding monitoring of the management contract. It needs to be noted though that in the majority of communes with a PPP in place, the relationship between the OSC, SLC and WSP was described as constructive, if not always clearly defined. • Respondents repeatedly asked for more resources from RANO WASH or from the commune to carry out their roles. While SLCs should receive funding from commune budgets, OSCs and VSLAs are expected to operate more as voluntary associations. Calls for external support pose a potential sustainability risk as external financial revenue streams are unlikely to materialize. • Particularly between the OSC (a civil society grouping with a strong advocacy purpose) and the SLC (a government decreed multi-actor consultation mechanism to suggest, discuss, and agree on commune-level development efforts), there is a strong overlap in membership, and a clear confusion of roles and responsibilities. This was evident in the OSC/SLC or community focus group discussions, when asked to describe functions of the roles of their different structures and who was in them. Below are quotes from a joint interview with OSC and SLC representatives from the same commune: I am a fokontany chief and member of the SLC, which is a structure set up within the municipality to sensitize the population on sanitation and hygiene. – SLC Representative I am a member of the CSO which brings together all the driving forces of the municipality. We support the municipality in improving sanitation…. We also apply DINA for people who defecate in the open. – OSC Representative The role of the CSO is also unclear for us, but we work with this structure. – SLC Representative Notwithstanding the above, OSC and SLC members showed conviction of the importance and relevance of the structures they were part of, and there were definite examples of a clear advocacy or input role. However, when asked to describe their activities, most interviewed SLC and OSC members leaned towards their role in sensitizing the communities (‘sensibilisation’) with regards to WASH behaviors, construction of latrines, payment of water services, etc. Emphasizing sensitization instead of the other components of their work may further indicate that they are confused about their roles and responsibilities. As is further discussed in Section 3.4, equating these positions to behavior change roles raises some separate concerns as the OSC/SLC members are not necessarily trained in this respect. Accountability mechanisms. Various informants highlighted the ‘ideas box’ at the commune level. They also liked that the SLC and OSC raise concerns and provide space for suggestions. Overall, the majority of communes reported that RANO WASH impacted the space for dialogue about WASH issues positively. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 30 The SLC is a good thing because it monitors the municipal budget. It relays the voices of the communities, and it also collects the opinions of the population, through the suggestion boxes. The mayor, councilors and the SLC attend the opening and reading of the contents of the suggestion boxes. The subjects that are the subject of multiple complaints are then prioritized and followed up. – Mayor However, some informants said that boxes were opened irregularly. In addition, some community members complained about disputes with WASH service providers, and “non-responsive” mayors. Similarly, RANO WASH’s reporting on the efficacy of its “Green Line program,” the telephone hot line set up for all beneficiaries and community members to express their views and provide feedback on the RANO WASH activities being implemented in their communities, shows low numbers of use so far. Yet, on the whole, RANO WASH’s focus on establishing community structures and insistence on local accountability mechanisms has noticeably increased commune-level WASH activities and awareness. Overall, the Evaluation Team felt that the program leaned a bit on a “cookie cutter” approach, following the same process in each commune, putting the same structures in place, and providing the same range of trainings, despite obvious differences in context. This is understandable from a program intervention perspective, particularly in a consortium setup where implementing partners said they feel like “executors” of the PCT strategy and approaches, and so will carry out the activities they have been told to carry out. And standardization of systems and structures is a prerequisite for scale. But this design risks local ownership when carrying out a set of activities not felt to be the most appropriate or relevant by local stakeholders who implement them because “the program said so.” This point is revisited in Section 4, Recommendations. Involvement of private sector actors. Many communes are convinced that the private sector is a better manager of WASH infrastructure and systems. Some because it is an orientation proposed and supported by the program or directed by the regional or national government. Others because they associate community management with failing systems and think the private sector can do better. They also see it as a way to “split” some of the costs related to management and operations of water supply systems and extension of services. Realistically speaking however, particularly given institutional, capacity, and financial impediments— transition to nationwide private sector provision of water supply services will always be a slow process. The Evaluation Team proposes the following considerations regarding such a transition: • Respondents repeatedly inferred that RANO WASH suggested transitioning to a private sector service provision model despite the presence of a community managed system. Though the existing service may have been poor, the Evaluation Team found that emotions ran high and not all members of the community wanted to transition to the private sector. The biggest challenge arises in the switch from community management to private management for communes that already have free or fixed-cost water systems, especially when the system is still functional and the procedure is to cut off the waterpoints. – Implementing Partner Continued advocacy will be needed for the attention of communes (councils, mayors and SLC) to explain the potential advantages of a shift to private sector management. The Evaluation Team understood that a key stumbling block is the perceived transition from a “free” to a paid service, thus advocacy should include more evidence that even “free schemes” are rarely free, as even in those cases, households have likely made occasional contributions to repair or rehabilitate the systems. The main shift for households is therefore from occasional to regular payments, rather than from free to fees. • More importantly, it is likely that even in communes willing to shift, there will always be a transition period in which private sector management and community management exist side-by-side. The MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 31 Evaluation Team recommends developing a clear set of criteria for evaluating PPP over community management accompanied by a transparent commune-wide consultation or decision-making process to guide the transition. While some criteria will be more serious than others, and may result in a quick shift (for example where existing water supply systems have fallen completely in disrepair), there may be other criteria that could lead the community to monitor the current state before enacting change. RANO WASH would then encourage the municipality and DREAH to monitor these sites under community management, and to take action when the observation of poor management is made. If management structures becoming less and less functional, then it will be clear to the community that they need to change systems. For Marozevo, community management is a favor given to the community in return for their support to infrastructure rehabilitation. The Management Committee volunteers to manage the system. We will switch directly to private management if community management fails. – Mayor, Beforona • RANO WASH could also design and develop a package of activities intended to support communes having to manage several management methods at the same time in their territories during the phase of transition. In this light, the case of Marozevo described by the mayor of Beforona illustrates a potential transitional measure, where even within a community-managed scheme, households are expected to start paying monthly fees: A Management Committee has been elected by the general assembly of the Fokotany. Each household pays 600 ariary per month, of which 200 ariary is for maintenance, 200 ariary paid to the commune for the construction of infrastructure for the other fokotany and 200 ariary for the Management Committee. – Mayor, Beforona • The program should also consider collaborating with MoWASH on development of a strategy for construction and management of drinking water supply systems in the most remote villages with fewer inhabitants. It is recognized that the Water Code decrees private management of water systems, but that in Article 41 it also specifies that “communities and/or the “Fokontany” may, at their request, exercise delegated project management of small rural drinking water supply systems located on their territory with the agreement of the Regulatory Body referred to in Section IV of this chapter and the commune to which they are attached.” Given that the national reality is that many such commune- or community-managed local schemes still exist, such a national strategy could possibly include transitional hybrid options between private management and community caretaking. For example, water user committees might ensure daily operations and perform routine maintenance and repairs, while a contract with a WSP running a PPP elsewhere in the commune covers backup support, covered through the regular fees collected by the water user committee. Another interesting model piloted in Kenya3 is that of an insurance scheme to boost sustainability by covering repair or rehabilitation expenses in exchange for small monthly premium charges. • With support from the program, communes should develop longer-term strategies to work towards universal drinking water supply at the level of the entire commune, its capital (chef lieu) and all villages of all fokontany. Private sector WSPs should be invited to engage in such commune-level planning of WASH interventions—in terms to be specified—to understand the issues, give input, and possibly make commitments toward management options they favor. In this regard, WSPs themselves mentioned their interest in reviewing the WSP selection process, or ensuring its proper MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 32 3 See this project with World Vision: https://www.wvi.org/newsroom/kenya/new-insurance-innovation-aims-boost￾sustainability-rural-water-projects-kenya; and a discussion of a slightly different scheme in India particularly focused on covering damages from extreme weather events: https://www.ircwash.org/blog/insurance-rural-water-supplies-good-idea￾revisited. application, to ensure that incumbent WSPs would automatically be favored in tender-processes for extensions or construction/rehabilitation of additional systems within the same commune. 3.3 EVALUATION QUESTION 3 - PRIVATE SECTOR APPROACHES EQ 3 - TO WHAT EXTENT HAVE THE DIFFERENT RANO WASH PRIVATE SECTOR APPROACHES FOR WATER SERVICE DELIVERY AND SANITATION ACCESS EXPANSION BEEN SUCCESSFUL? 3.3.1 PRIVATE SECTOR APPROACHES TO WATER SERVICE DELIVERY Building on prior work of RANO HP, RANO WASH has made undeniable progress in popularizing and operationalizing the PPP approach. Specifically, the program has elevated the Constructeur-Investisseur￾Gestionnaire (Construct, Invest, Manage) model, which shows potential in most of the contexts used. Available literature on successful, at scale, and sustained private sector engagement in rural water supply is still scarce and largely case based. Though not recent, one of the few overview studies noted that: …few large-scale examples of private sector provision of rural water services. Even scarcer were instances of private investment in rural water scheme installation where upfront costs are recouped through user fees. 4 This highlights the need for RANO WASH to not only continue to ensure and (internationally) share documentation of the Madagascar experiences, but also the potential for USAID to facilitate further exchanges between USAID-funded rural, small town, and peri-urban water programs with private sector engagement components, such as with stakeholders from the former Kenya KIWASH program. The Evaluation Team considers one of the biggest risks in the current PPP approach to be the long duration of the contracts and the time it takes for WSPs to break even. Some WSPs perceive risks related to political instability and frequent changes of leadership. These are legitimate concerns. If successful, RANO WASH’s work at the national level to ensure a clear policy and legal framework will hopefully help to counteract this. This may also be one area where international exchange around risk-reducing measures will be of interest. Inclusive design is also critical, and the program should explore more varied models of management and construction investment approaches building on the PPP+ model to cover entire geographical areas. We recognize the move by RANO WASH towards the establishment of water kiosks by the WSPs in some of the remote communities, as an in-between step and one example of exploring transitional models. With regard to the actual PPP and PPP+ models, the Evaluation Team took note of the Mid-Term Review Report indicating that: The project has 24 operational water supply systems under Public-Private Partnership (PPP) management (out of 152 originally planned) with an overall potential coverage of over 127,000 water users. However, access to water has not improved as quickly as anticipated, with a total of 13,046 people currently with access to safely managed water services and 42,543 with access to basic water services –representing only 14% and 20% of the project’s overall targets, respectively. – Mid-Term Review Report The Evaluation Team believes that the program’s inability to meet targets in the available timeframe is the result of many contributing factors. From conversations with informants, two points stand out: the initial under-estimation of the time it takes to build relationships, mutual understanding, and PPP-related capacity among all partners involved and, particularly, to ensure full ownership of local government and 4 See: https://www.ircwash.org/sites/default/files/foster-2012-private-sector-provison-of-rural-water-services.pdf MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 33 buy-in from communities; and the time taken to develop procedures that work and deal with complex approval systems, including those of the donor. From its interaction with local stakeholders and private managers, the Evaluation Team noted the following: ● Managers selected by the program, even without much prior system management experience, have all, with seemingly one exception, managed to supply services as provided for in their contracts with the communes. • The program was able to mobilize operators from the private sector. - It has raised interest of players in the private sector, if we look at the number of proposals received following calls for expressions of interest for systems management. - The method of selecting candidates for water system managers made it possible to entrust the systems to operators with extensive experience in systems management, but also, to novice operators. This point is encouraging because it confirms the existence of potential in terms of capacities that can be mobilized in case of need. That said, in practice it was felt by several informants that more could be done to attract local, smaller-sized enterprises to enter the market. - A lot of these remarks seemed to stem particularly from a sentiment that the larger, non-local providers either did not understand local context sufficiently, or did not care enough about the local dynamics to find ways to attract and connect more vulnerable households to the schemes, linked for example to seasonal inability to raise cash for the connections or tariffs. But it might actually be these larger WSPs who have the economies of scale and cash flow to allow for more flexible solutions in such cases. The Evaluation Team suggests that at a minimum, no matter the size or location of the WSP, it is important to ensure that there is a local interface, the person who is the day-to-day contact point for the households, who is either local, or local enough to understand the community context and customs, and who can inspire household trust and confidence in the WSP and the services provided. There are already selection criteria for managers. Our criteria select the rich managers… there are a lot of criteria but they are harsh and eliminate small private managers, but often the rich manager is not interested in small systems. – RANO WASH Implementing Partner ● Less palpable are the results of the program in terms of strengthening the capacity of these private sector operators. The reinforcement was essentially or even exclusively in the form of multiple trainings on a vast range of topics given to managers. The difference in capacity levels of these operators before collaboration with the program and after, has not been sufficiently documented. One suggestion in this regard would be to try and better “personalize” trainings to the background and situation of the service providers, for example concerning size and experience of enterprise. ● The results of the market creation efforts to date are difficult to assess. The field visits showed that the extensions carried out and envisaged to existing supply systems did not demonstrate the private managers’ autonomous financing capacities; RANO WASH still stepped in to contribute to funding them. ● Much of the work to diversify products, that is, the option of social connections and standpoints, as well as the marketing done to increase the number of subscribers and the work to strengthen customer service, have been actively supported by the program, as private manager skills were said to be lacking. The Evaluation Team does acknowledge that the need to build better private manager MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 34 skills and capacity on marketing, market segmentation, ‘customer attraction’ and customer relations has been recognized by RANO WASH as well, and is part of the PPP+ focus. ● The fact that many households, including those considered vulnerable, are willing and able to cover the water fees is encouraging and holds potential for future connections. ● Only in one case did we find a commune planning to use the RANO WASH ‘100 first connections’ marketing scheme, offering connections at a discounted rate, for the specific purpose of connecting most vulnerable households to social connections. While it is understood that at the start, and in order to familiarize people with the new model, the scheme was needed as a pure marketing tool, the scheme raises concerns of fairness and reach. It needs to be considered that many households will face difficulty in covering the connection and water meter costs, even if they are willing customers. ● These last two points taken together, further support the Evaluation Team’s recommendation to explore ways to possibly subsidize or facilitate cheaper connection and water meter fees across the board, therewith abandoning the 100 first connections scheme. Or if the promotion scheme of the 100 first connections is maintained, it could be continued mainly as a marketing scheme in situations where a pre-existent ‘free’ service already exists and the purpose really is to get people interested in switching to the improved service delivery that comes with the PPP. But in cases where no pre￾existing services exist, it should be considered to reform the scheme so that it benefits particularly the most vulnerable households. In conclusion, the Evaluation Team judges that it is too soon to tell whether PPPs and private managers will be able to provide services without external support, and even more so whether commune leadership could engage in PPP contracting processes and long-term relationships without substantive engagement of development partners. But the Team also finds that with the positive results achieved so far, and given the time-intensive nature of the paradigm shift involved in setting up the PPPs, it is also too soon to abandon the model. A caveat to this is our earlier point about considering a commune- or district wide approach consisting of a mix of possible management approaches, with a possible progressive move toward more and more private sector management, where this can be viably done. 3.3.2 PRIVATE SECTOR APPROACHES FOR SANITATION AND MENSTRUAL HYGIENE SERVICE DELIVERY With regard to market strengthening for sanitation, the Evaluation Team considered that the situation was perhaps more concerning than for water. From the interviews with local stakeholders and masons and seamstresses, the Evaluation Team felt that the approach used to mobilize and support these entrepreneurs was more of a ‘social actor’ rather than a business approach: ● Masons and seamstresses all indicated that they had been selected or even “appointed” for this role, and that for most of them, this was not a business they had previously been involved in. There were also repeated instances where the masons were the “second wave,” because the first wave of masons selected for the program had abandoned the position or had not been found to be active enough. More troubling were the repeated inferences to mayors personally selecting the mason, creating a power dynamic between the community, the mayor, and the favored mason. ● Only some masons and seamstresses indicated that they themselves had set the price for the toilet products or menstrual hygiene materials they were producing and marketing. Most stated that RANO WASH had suggested the price range. They also repeatedly stated that price levels were still too high for the households in their target regions to be able to afford, and they were struggling to market their products. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 35 ● Some, although not all, masons and seamstresses were provided with materials such as molds, tools, and sewing machines by the program. Those who did not receive material support, particularly the second wave of masons, were visibly frustrated. The Evaluation Team also did not see signs of microfinance use by the masons; for example, to buy raw products or produce some stock upfront. ● By and large, the attitude of masons and seamstresses was that they were doing this “for the good of their community,” rather than to make money. In fact, most of them indicated they would continue to do this, but that it was not a viable model and they were struggling to break even— although there were a few exceptions where a mason and seamstress had successfully marketed their product. Some seamstresses said that the program had purchased their products directly. …It’s just to take part in the development of our commune and especially to eliminate the practice of open defecation. – Mason We earn 500 Ariary per sanitary towel. Our life has changed but the sale does not go very well and it is not yet profitable. But as local seamstresses we continue to make them even if there is no longer a buyer because RANO WASH still buys around Global Menstruation Day each year. – Seamstress The Evaluation Team strongly sensed, based on a number of KIIs with community stakeholders, that households are at times pushed into purchase (including by mayors and through local sanctions such as DINA – see Section 3.4). Further, households seem to have limited choice of products in an affordable price-range and there seems to be limited emphasis on community initiative and community innovation beyond using the masons, or innovation by the masons themselves. These last two points were also mentioned in KIIs with other development partners, while discussing market-based sanitation in the context of Madagascar. Considering that poverty levels are extremely high and that in many villages the team visited, there are seasons where households barely have cash on hand, purchasing ability needs to be a careful consideration in trying to establish a sanitation marketing approach. Prior sector studies on market-based sanitation, including some undertaken by WASHPaLS5, have indicated that a singular focus on mason training will not activate or grow a long-term market, as their business model is too uniform to be viable and that at the very least, business models would need to be more diversified. The Evaluation Team finds that beyond RANO WASH’s work with iDE, the potential (or not) of a sanitation market-based approach in the context of Madagascar has not been studied and explored extensively, and that programs and organizations are all doing slightly different things in this space. The program may consider working with development partners and the MoWASH to learn from experiences to date, including the iDE work, and jointly develop a shared sector roadmap or strategy for market-based sanitation. Such a strategy should then cover the full sanitation service chain, even in rural areas, rather than focus singularly on latrine construction. 3.3.3 WASH IN SCHOOLS AND HEALTH CARE FACILITIES Sanitation and water supply results in schools and health care facilities were incorporated in SO2 in the original program design. As such, the inception report for this evaluation proposed a sub-question to assess the successes and challenges of including the WASH in Schools and Health Care Facilities (HCFs) component. The program planned to work with private sector actors to provide these institutional services. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 36 5 See ‘Scaling Market-Based Sanitation – Desk review on Market-Based Rural Sanitation Development Programs’: https://www.globalwaters.org/sites/default/files/Scaling%20Market%20Based%20Sanitation%20JUNE2018.pdf The program’s work in institutions aimed to focus both on hardware and software by implementing a WASH-friendly schools and HCF approach. In pure numbers, the program was far from its year over year targets by the end of the 2020 financial year. Indicators include increased access to basic sanitation facilities and basic water supply and institutions being certified as WASH-friendly. With regard to the WASH-friendly institutions program, many of the reasons for this delay were well documented by RANO WASH and were largely outside the program’s control. As reported in the 2020 Annual Report: Targets were missed for WASH friendly institutions, especially for health centers. As previously reported to USAID (…), the organization of training events and the certification process itself is not simple and is heavily centralized at the MoH and MoEd levels. As it currently stands, the process is costly, not scalable, and does not encourage regional and local government responsibility. The program noted the relationship with the Ministry of Education has been conducive to programming, among others because it is committed to decentralizing WASH activities and to delegating responsibilities to regional directorates. Overall, though, RANO WASH planned to work with development partners to jointly promote a more scalable, decentralized, and sustainable model Instead, and partly because of the COVID-19 pandemic, the program decided to focus on institutional triggering in health facilities and schools, helping them to develop agendas for action and undertake small, immediately doable actions. It also continued sensitization work with institutions in some of the target communes, and distribution of handwashing stations and soap. Altogether, the Evaluation Team fully agrees with RANO WASH’s self-assessment that it needs to coordinate more closely with partners at the national level to advocate for a different model that can exponentially increase the number of schools and HCFs with sufficient, sustainable WASH infrastructure and behaviors. This requires transversal action and coordination across the lead ministries, including MoWASH, Ministry of Education, Ministry of Health, and Ministry of Decentralization, which is also noticeably absent at local levels. In this regard, the interviews with HCF staff illustrated the near impossible position many find themselves in because of limited budgets that often do not include provision for WASH investments or recurrent costs. While the institutions connected to a PPP scheme expressed satisfaction and voiced only positive or neutral feedback about the WSPs, staff did say the tariff payment is problematic. For example, the school in Ranomafana became a WASH-friendly school with RANO WASH support and is connected to the PPP scheme, but tariff payments account for 30 percent of the school budget. The tariff stresses an already thin budget, which was insufficient to cover all operational costs to begin with. Some staff say they pay out of pocket to keep WASH services going. We ask those who come to give birth to buy soap. But sometimes we still have to pay out of pocket. When there is no sponge too, we have to pay out of pocket. – Sage Femme In some cases, the Evaluation Team observed that communes have stepped in to cover part of the institutional WASH budgets. But the financial plan needs to be more sustainable. Systemic change should come from the ministerial level, and where possible, be buttressed by concerted advocacy from RANO WASH, UNICEF, and other partners. Regarding the program’s support for installation or rehabilitation of WASH infrastructure, the team noted a lack of clarity among local stakeholders around the RANO WASH process to decide which facilities would receive interventions and which would not. Interviews with commune staff and school and HCF staff showed they were not always clear on why they had or had not been selected for infrastructure improvements. The Evaluation Team recommends that RANO WASH improve the MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 37 transparency and (repeat) communication on these decisions. Where institutions did gain access to new or rehabilitated services this seemed largely linked to the implementation of a water PPP. Going forward, the Evaluation Team recommends including WASH infrastructure and service delivery in schools and HCFs as part of a holistic district-wide approach. However, even a district-wide strategy should include additional programming at the national and regional levels to ensure that institutions have sufficient resources to ensure long-term access to services. Access may be contingent on ability to pay tariffs, finance unforeseen maintenance or repair work, or purchase soap and supplies as need be. The facilities have not yet experienced an outage. But I think that RANO WASH will bear the cost of repair if we need it because the maintenance of these installations is not yet integrated in the budgeting of the PTA 2020. – Schoolteacher 3.4 EVALUATION QUESTION 4 - DEMAND AND USE OF WASH PRODUCTS AND SERVICES EQ 4 - TO WHAT EXTENT HAVE RANO WASH ACTIVITIES BEEN SUCCESSFUL AT BUILDING DEMAND, ACTIVATING DEMAND, AND ENSURING USE OF WASH PRODUCTS AND SERVICES IN INTERVENTION REGIONS AND COMMUNES? In general, there is no doubt that RANO WASH’s efforts have led to an increase in access to water supply and sanitation services, particularly with regard to water supply; this was mentioned by a majority on informants as a key contribution by the program. However, the point was made repeatedly that with a program that has the word “RANO” in its name (Malagasy for water), many intervention communities expected that RANO WASH would support them in either rehabilitating, constructing, or extending water supply infrastructure; yet this was not part of the intervention package for the majority of the 250 intervention communes. When it was, the interventions only covered part of the communes. As per one of the implementing partners: All communes need capacity strengthening, and the RANO WASH approach is an opportunity for communes to better respond to their own needs, for example by helping them to develop communal WASH development plans (PCDEAH) that they can then ‘market’ with other donors. But RANO means water in Malagasy and the program name implies access to water systems. The communes always hope that the project’s main purpose will be to provide water supply infrastructure. Some communes, for example Ambiabe, mentioned that initial market studies had been undertaken and that RANO WASH had ‘promised’ to provide or rehabilitate water infrastructure, but that they were still waiting to hear back. Given the remaining timeframe of the program, the Evaluation Team recommends that all intervention communes that had at any point been involved in conversations with RANO WASH on the provision of water supply infrastructure but where such activities have not yet started, are given swift clarity to manage expectations for the remainder of the program. With regards to sanitation behavior change, field observations by the Evaluation Team suggested that only three villages among those visited met the ODF criteria (see Table 4). The information in Table 4 is based solely on the observations made by the team, which included checks against all criteria used for ODF verification in Madagascar: no signs of open defecation, presence of latrines that cut the oral-fecal chain (including use of drophole covers), and presence of handwashing facilities close to the latrines. From the many conversations had, the team concluded that for most local leaders ODF status is solely based on the number of latrines a village has, rather than on their use. While some villages declared that they were ODF or were close to ODF status, the Evaluation Team encountered active open defecation zones, and toilets that remained intact, clean and unused. The use of latrines was not commonplace in the villages despite an observation on the development of the sale of sanitation platforms among some local masons. Importantly, these observations align with the SE&AM data MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 38 reported in Table 2, which show that the visited communes have sanitation coverage rates ranging from 0,93 percent in Andemaka, to 65,96 percent in Ambositra II. TABLE 4: ODF VILLAGES AMONG SITES VISITED COMMUNE VILLAGE VISITED ODF CRITERIA MET Anosibe Ifody Anosibe Ifody No Beforona Marozevo kely No Ampasimpotsy Gara Ambohimarina No Ranomafana Est Andekaleka No Antsampanana Chef lieu de la Commune No Niarovana Caroline Mangarivotra Yes Ilaka Centre Andohafatsakana Yes Ambositra II Ampila Yes Kelilalina Antanambao No Ambiabe Chef lieu de la Commune No Marofarihy Fenosoa No Vohitrindry Tsarinetso No Nato Croisement Nato No Andemaka Tanambao Vohitromby No Total Village ODF 3/14 3.4.1 SANITATION AND HYGIENE BEHAVIOR CHANGE AND DEMAND CREATION Many informants pointed out that “demand for water” did not need to be created as it is a key priority in intervention communities. The key behavior change around water supply therefore had more to do with habits around (non) payment for water and types of water service management (see Section 3.3), than it did with habits around use of services themselves. This section focuses on the range of approaches used by RANO WASH to instill or reinforce sanitation, hygiene, and menstrual hygiene behaviors. As described in Figure 6 and summarized by one of the RANO WASH implementing partners, the program built its sanitation and hygiene behavior change strategy on five main components: 1. CLTS and Follow Up Mandona (FUM) 2. GUS 3. Voamamy/VSLA 4. Work with schools and HCF 5. Sanitation marketing The Evaluation Team recognizes and commends the RANO WASH team on the thorough way it has gone about its behavior change work. Sustainably improving hygiene and sanitation behaviors at scale is one of the key challenges in the WASH sector, and one where no silver bullet exists. Working with LSHTM afforded the program access to some of the sector’s most advanced thinking and expertise on sanitation and behavior change. From the reports provided by RANO WASH, it is clear that the GUS approach is the result of a solid research and Behavior-Centered Design (BCD) process. The team also acknowledges the program’s intention to pull together elements of different approaches to combine them into something even stronger and more likely to lead to sustainable results. Therefore, the points of attention listed hereunder are an encouragement for continued learning and reflection on this matter. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 39 FIGURE 6: THE CONTINUUM OF BEHAVIOR CHANGE AS DESCRIBED BY RANO WASH Actors and capacity. A large number of actors are involved in community- and household-level behavior change. While the RANO WASH TA takes overall lead and facilitates many of the CLTS triggering sessions, they are supported by numerous actors: 1) the ATEAH for co-facilitation of triggering, post-triggering follow-up, and facilitation of FUM sessions; 2) local promoters to facilitate the GUS approach and engage with the VSLAs; 3) the OSC, SLC, ASUREP, mayor and VSLAs for general sensitization, and 4) the local masons and local seamstresses who combine marketing of their products with sensitization as well. With so many actors involved, not only is there a risk of misinformation and confusion, (e.g., about toilets built with local materials not being 'improved’, or with regards to menstrual hygiene as discussed below) there also may be a risk of message fatigue among households being overwhelmed with too many messages. It is also hard to ensure that all of these individuals and structures are well trained. For example, the majority of ATEAH interviewed indicated that they were only trained on CLTS in 2020, after much of the initial CLTS triggering and follow-up had already taken place. They reported trying to learn on the job from their TA. Mobilizing so many actors and conducting so many different activities can lessen oversight, and risk messages and activities contradicting each other, rather than being reinforced and taken up as designed. The Evaluation Team understands that the mobilization of multiple actors and structures around sanitation and hygiene behavior change is a strategy to scale and that reduced fidelity to particular behavior change strategies or messages is likely, possibly even desired, as people will start promoting WASH in their own way. But it will be important to examine risks and impact, including some discussed below. CLTS vs. sensitization. While the approach is said to have CLTS at its base, some of the core principles of CLTS were not evident in the communities visited. Many actors in the field described habits that align more with a sensitization and education approach—linking to the point above. This was clear even from the words used, with CLTS really emphasizing the importance of ‘facilitation’ (Fanamoràna). MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 40 This takes communities through a process where they themselves realize the implications of their bad hygiene habits and decide to act and become ODF – ‘Community problems, community solutions’. Instead, most of the local respondents involved in behavior change activities, described that they were involved in ‘sensitization’ (Fanentanana), and ‘giving messages’ on latrine use, handwashing with soap, water treatment, etc. RANO WASH identified this risk in relation to the GUS approach, as reflected in Annex 16 of the Q1 2020 Report: While the project did not plan to use one-sided delivery of sensitization and education as a means to promote changes, the evaluation reveals that it is still difficult to remove this habit of educating people from local promoters, as some of them still use education and simple messaging to promote changes. This affects adherence to the project’s prescribed strategy. One small but illustrative example of this messaging approach might be the installation and use of showers. The large majority of local informants said this was an intervention suggested by RANO WASH, even in communities without an evident source of water. This was questioned by some, and mentioned by others as a difficult behavior to sustain in their context. While it is acknowledged by the Evaluation Team that stimulating this particular behavior may be as much driven by dignity and privacy as by hygiene motives, in some such cases it might be more appropriate to introduce the construction of showers after water supply arrangements have been improved. Each household built latrines and showers at the same time. But the villagers are not used to using the showers and they are becoming worn and broken without being used…We do not have a water connection at home yet, we have to take water a kilometer from the village before taking a shower. – VSLA Potential for scale. The idea of the spillover effect, or “oil stain”(tache d’huile), by which households and communities start influencing each other to adopt hygienic behaviors, is well-documented by RANO WASH, and a key part of the GUS and model household approaches. Informants familiar with the topic indicated that while they do think the oil stain effect is there, it is still very modest and quite slow. Currently, the GUS approach is implemented by local promoters directly supported by the program, who are each responsible for 12-16 households over a three-month period, before moving on to a new set of households. This approach is human resource intensive and requires significant training and materials (training materials, materials for use by promoters, stickers and petals, etc.), all of which are currently supported by RANO WASH, raising raise concerns about scalability and sustainability. The Evaluation Team also observed that the program does not explicitly focus on working with natural leaders but rather with ATEAH and with program-appointed local promoters in GUS communities. In other programs (including FAA, Mahefa) natural leaders are a key element in the strategic approach to exponential scale as it grows the “army of facilitators.” Facilitators who could then potentially work with and support the ATEAH. The Evaluation Team felt their absence is a potential additional challenge in the drive for scale, which warrants further learning and understanding. Sanitation and water. Community ODF status was an early selection criterion for RANO WASH water supply interventions. The program assumed that in communities with ODF status, the water system could reinforce already adopted sanitation and hygiene behaviors, and, to an extent, serve as a reward or incentive to reach and maintain ODF status. This is an approach applied in different countries and the Evaluation Team learned it was also encouraged by other sector partners such as FAA. However, the Evaluation Team found a risk in implementing this approach. Some communes indicated that their drive for ODF status was very much based on their larger desire for a water supply system and their understanding that they would qualify for a RANO WASH water supply intervention upon achieving this status. Rather than choosing to eradicate open defecation for the sake of it as inspired through quality CLTS facilitation, GUS etc., there is then an element of coercion to make sure all households start using latrines as quickly as possible, and a sense of disappointment when a community MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 41 realizes they might not (yet) be in line for a water supply intervention. This risk was denied by RANO WASH implementing partners, but implied in interviews with local stakeholders in different communes visited and by community members in informal one-on-one conversations. The big challenge is to become an ODF municipality because RANO WASH cannot put in place the infrastructure until the villages are completely clean. – Mayor The program bears the name “RANO,” and all the inhabitants are waiting for the water supply which is not yet effective. … In this case everyone steps away from sanitation because the inhabitants are disappointed. – Mayor The biggest risk is in sustaining the sanitation and hygiene behaviors adopted where they have not yet become habits and where people built the toilets merely to qualify for water supply. At the same time, hygiene practices such as handwashing with soap and use of showers become much easier with a reliable water supply. Even if the motive of these communities was informed by the improved water supply, it is possible that a concerted behavior change strategy promoting the different key behaviors might still result in sustained latrine use and yield improved hygiene practices. The Evaluation Team does however feel this is an area of attention that warrants further investigation and learning. Use of force and penalties. When discussing behavior change approaches, the majority of communes visited by the Evaluation Team mentioned various uses of force to persuade ‘non-adopters’ to construct and use latrines. Particular practices mentioned were the use of DINA (local social sanctions, including fines) or the ‘Carte WC.’ The Carte WC is a card proving that households have a toilet at their homestead enabling them to access other social and public services in the commune that would otherwise be denied to them. It is acknowledged that these practices are generally developed and put in place by the communes and communities themselves, rather than promoted by RANO WASH. But while the use of local bylaws and methods of locally agreed enforcement are not necessarily problematic and part of encouraging local ownership, care needs to be taken to make sure behavior change isn’t just based on force. The change in the behavior of community members is not so much the result of awareness but the constraint of adopting the WC card. – Local Promoter Moreover, the use of force also carries with it a large risk of unintentionally affecting vulnerable people and households hardest. 6 RANO WASH has placed strong emphasis on local leadership driving sanitation behavior change, and the positive effect of a committed mayor and committed commune leadership has been described in its success stories; this was observed by the Evaluation Team during its field visits. It is clearly a key driver to change, and a prerequisite to scale. However, it is this same leadership that can enforce potentially harmful practices. Therefore, people involved in community behavior change activities, be they TA, ATEAH, local promoter or others, should take special care to understand the drivers behind household non-adoption. Beyond a lack of means, there may be many other motives and reasons why households do not adopt the new community norm, including mental health issues7, exclusion, or marginalization. RANO WASH could potentially do more to train these WASH stakeholders on this aspect of identifying and understanding potential disadvantage, or at minimum on the potential risk of occurrence of unintended consequences and the importance of some basic Do No Harm principles. 6 See for example: House, S., Ferron, S., Cavill, S. (2017). Scoping and Diagnosis of the Global Sanitation Fund’s Approach to Equality and Non-Discrimination. Geneva, Switzerland: WSSCC 7 See for example: 40th WEDC International Conference, Loughborough, UK, 2017. Understanding, respecting and including people with mental health conditions as part of the CLTS process. S. Cavill, P. England, S. House & S. Ferron (UK) PAPER 2604 MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 42 Sanitation Marketing and informed choice. The commune-level interviews pointed to a strong focus on a very limited set of sanitation technologies and materials. In addition, many community stakeholders hold a strong conviction that latrines built with local materials are simply not good enough and that households should be persuaded to invest in improved technologies. This, combined with the fact that all the masons with whom we spoke seemed to almost exclusively sell the Dalle sanitation platforms (DSPs), implies a level of limited choice, rather than ‘informed choice’. The Evaluation Team understands that this drive to market better quality latrines/latrine components stems from the desire to ensure more durable and higher quality latrines. But it can become another element of force, and in combination with the earlier point on poor training quality and messaging issues, the narrow market presents a risk factor, particularly for people who cannot afford the DSPs. The Evaluation Team did not find many examples of masons innovating or diversifying, a finding that reflects the challenges presented by the market and also by the lack of business training discussed in Section 3.3. Lastly, in this context it may also be pointed out that recent sector work on ensuring safely management sanitation in rural settings concluded that simple pit latrines are in many cases the safest option with least pathogen transmission risks, posing some interesting questions for the continued push ‘up the sanitation ladder’.8 At the same time, it is clear that latrines need to be of sufficiently durable to avoid the need for frequent repairs or replacement, and that a slab that is easily cleanable will help further reduce pathogen transmission risks. In addition to continued work to develop affordable yet aspirational models, another strategy to explore would be to introduce more flexible payment options for households unable to afford the current choices on offer. Village Savings and Loan Schemes. The Evaluation Team spoke to a range of voamamys or VSLAs, as the engagement of these largely women’s groups is a key component of the RANO WASH strategy. As stated in the RANO WASH 2020 Annual Report, VSLA members have better financial means to afford WASH services and products, and they play a key role in achieving ODF status. They are also a key platform to discuss and re-define social and gender norms around sanitation and are proven to enhance social capital, which is key to fostering collective action around sanitation. In practice, while the VSLAs interviewed were mostly positive about the RANO WASH intervention and spoke of the importance of the messages they had received and the activities they had engaged in, there had so far been only minor investment of VSLA funds in WASH. A number of groups indicated investments in kitchen utensils (in line with RANO WASH findings, Annual Report 2020). One discussed use of the funds to cover some of the water system connection fees, and only one mentioned the third “caisse” for WASH investments. There may be ways to further incentivize WASH investments via VSLAs, such as economies of scales purchasing. A latrine costs around 30,000 Ariary. The money for the VSLA is only used for our urgent needs but not for the latrines, there is no WASH fund (‘Caisse EAH’). – VSLA Vohitindry No, we have not yet used the VSLA money for the construction of the latrines. We use the money to buy pigs or rice to increase our income. – VSLA Ampila I have already repaired my toilet by borrowing 30,000 Ariary from the Voamamy cash register. This loan is dedicated to repairing my latrine and making tippy taps. – VSLA Beforona Activities vs. Strategy. Some implementing partners spoke of non-strategic villages, that RANO WASH is therefore not prioritizing. Criteria to declare a village as non-strategic include the number of olsky et. al, 2019; referenced in this report based on a study by Andy Robinson and Andy Peal: WSSCC (2020). Safely ged Sanitation Services in the Global Sanitation Fund. Geneva, Switzerland: WSSCC. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 43 8 See K Mana households, the extent to which they are on the road to ODF status, and whether they have already been touched, unsuccessfully, by CLTS. The criteria also include an assessment of the extent to which there was the potential for a spillover effect from this community to those around it. As was stated by the informants, the problem was that “most villages are not strategic,” mainly because by now almost all villages have been touched by some kind of intervention and so many of these villages were now “left to the side.” But this was said to be one of the reasons for the program to focus more on an ODF Commune approach, which requires a more strategic and concerted behavior change strategy contextualized to each commune, albeit it built on the full RANO WASH package of behavior change approaches. The shift to ODF communes aligns with a recommendation made throughout this report: the Evaluation Team suggests a district/commune-wide approach. Nevertheless, these interviews highlight the balance behavior change programs need to manage; inspiring inherent motivation and autonomous adoption on the one hand, and using an appropriate or reasonable level of peer pressure for the common good on the other hand. Overall, the Evaluation Team recommends that the program continues to document its behavior change strategy and approaches taken carefully, possibly prioritizing that over embarking on any new research with the LSHTM. We would recommend focusing documentation particularly on scalability of the approaches, and the reality of combining CLTS and other strategies—including addressing some of the concerns raised above. 3.4.2 MENSTRUAL HYGIENE MANAGEMENT Most menstrual health and hygiene promotion strategies revolve around at least three or four pillars covering social support, knowledge and skills, facilities and services, and materials. 9 Within RANO WASH, the Evaluation Team did find elements of ‘breaking the silence’ or addressing taboos, including as part of the ‘engaging men and boys’ approach. But from the many interviews with community actors, what stood out is an almost exclusive focus on two elements: the importance of washing and keeping the body clean during menstruation (particularly the use of showers), and the use of reusable menstrual hygiene materials (see for example the Grow Up Sticker Implementation Guide). Nonetheless, in the majority of the discussions with OSC/SLCs and with commune staff, and particularly VSLAs, informants felt that the “silence had been broken,” and that there was an ability and willingness to talk about the topic among women and men—often considered the first step in MHM programming. But many of the interviewees simply equated MHM to use of washable pads, and in some cases the presence and use of showers. When asked about MHM, interviewees frequently referred to the reusable pads produced by local seamstresses. This finding might link to the marketing techniques used by the seamstresses, or it might reflect a general trend. It is possible that the Evaluation Team asked questions too narrow to determine the levels of understanding of menstrual cycles and menstrual health and hygiene. Regardless, MHM messaging should broaden awareness about possible MHM resources, whether home-made washable fabrics or disposable materials. With clear and objective messaging, people are better equipped to independently make informed choices rather than acting solely on suggestion. The Evaluation Team sees the current messaging approach as a risk; interviewees were clearly confused about their MHM options. The Evaluation Team recommends refresher training and facilitated discussions with community and women’s groups. Those who have more means, use disposable napkins which are not hygienic. This is why I insist on making washable sanitary napkins even if they are not cost effective. – Local Seamstress Disposable napkins give you cancer. – ATEAH 9 See for example the UNICEF ‘Guidance on Menstrual Health and Hygiene’: https://www.unicef.org/media/91341/file/UNICEF-Guidance-menstrual-health-hygiene-2019.pdf MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 44 3.4.3 APPROACHES TO HARMONIZATION, RESEARCH AND LEARNING All national and regional informants were asked whether they felt there was a growing sense of harmonization of approaches and strategies in the sector, particularly in the RANO WASH intervention regions. The majority of informants did not see great progress in this space, with responses varying from there is no harmonization in the sector at all to a sense that even while adhering to the Mada Madio roadmap, and with a growing sense of coordination through the SRMOs, there is still a lot of variation in practical programming approaches and strategies being used around, for example, behavior change, private sector engagement, and, noticeably, the use of handouts in some programs, such as the free provision of sanitation platforms. The latter was mentioned by several local informants, with an expectation or suggestion that for RANO WASH to really ‘support’ the communities, the program should provide sanitation platforms. The fact that such practices co-exist with strong non-subsidy approaches, all while a national roadmap and strategy is in place, confirms once more the urgent need to establish a functional learning, dialogue, and coordination mechanism at the national level. Not to necessarily ban one approach over another, but to share, learn and agree on best practices that work for Madagascar. Informants report inconsistently on RANO WASH’s impact on sector coordination. For example, informants recognized some of RANO WASH’s webinars and could describe its role in supporting sector dialogues. At the same time, informants did not highlight RANO WASH’s contributions to sector-wide learning, sharing, and the drive toward harmonization. The absence of a national learning mechanism, or a platform that would boost RANO WASH’s visibility as a coordinator, may be at fault here. RANO WASH made attempts to catalyze the efforts of learning but it can still be improved. – Development Partner As discussed in Section 3.1, internal and programmatic learning was built into the original design, including to help operationalize the three pillars. Program staff described in interviews that the program experienced initial delays in fully engaging in learning activities, which likely contributed to some of the confusion among implementing partners and consortium members. This issue started to be addressed in 2019 with strong leadership moves to emphasize learning in the program, as illustrated by the increasing number of learning documents attached to the annual reports. Around the same time, project leadership articulated the need to identify and focus on sector building blocks as a practical translation of the program theory of change. However, based on interviews, it is clear that the comprehension of this systems approach is currently much stronger among program personnel (particularly those in the PCT). Ministry staff at a strategic level are not as familiar, and the Director-level ministry staff and other sector stakeholders even less so. This fact highlights the need for further investments in refining program learning and sharing and to use these learning products for sector dialogue, policy making, and advocacy. A notable exception in this may be the PPP model which, as mentioned by several informants, has already garnered real interest from partners. On this aspect of programming, people are eager for more information and engagement, and for in-depth sector dialogue beyond just sharing of success stories. The Evaluation Team noted what seemed to be an increased focus and production rate of documentation and learning activities in recent years, and the program documentation shows a high level of internal program analysis, including the very thorough Mid-Term Review carried out earlier in 2021. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 45 When asked about learning practices within the program, some implementing partners indicated that they felt quite low engagement in these processes (we are only asked to send success stories). At the same time, the Evaluation Team concluded that this sentiment may have varied depending on the regional consortium lead partner. As discussed in Section 3.2, the Evaluation Team feels there is real scope for more documentation and, perhaps more importantly, peer-to-peer engagement and learning between the program communes. RANO WASH’s design included an ambitious research agenda via several planned studies regarding, in particular, BCD and nutrition. The program reported the various reasons why some of these studies faced delays. Building research into implementation-oriented programs is complex and does not often result in real application of research findings. The Evaluation Team commends the program’s success of bringing its BCD-resulting GUS approach to substantial scale. The CLTS Research Protocol developed by LSHTM in January 2020 can also bring useful insights, many related to points raised by the Evaluation Team. However, given the other program goals still to be achieved, the Evaluation Team suggests that the remaining research pieces be deprioritized. 3.4.4 SUSTAINABILITY OF ACCESS AND USE For the PPPs currently in place, the Evaluation Team considers it likely that WSPs will be able to sustainably provide services to the target population, especially with continued or, in some cases, improved focus on customer satisfaction and flexible options to manage fluctuating ability of households to pay the tariffs. Continued work at the national and local levels to ensure long term adherence to contractual obligations will also help ensure the success of the PPPs. As for sanitation, RANO WASH has been transparent in its reporting on ODF slippage, (e.g. 25% slippage reported in the FY2020 Annual Report). While this is substantial, it is not surprising, and it shows substantial progress against ODF sustainability rates reported in prior Madagascar programs, including RANO HP.10 The concerted program focus on sustainability is evident both in design and implementation across the three pillars. RANO WASH has undoubtedly tried to build sustainability into all its activities, and the building blocks approach is an important element herein. Using the WASH Alliance ‘FIETS’ model of sustainability,11 the Evaluation Team noted the following: • Financial sustainability has been a key focus of many of the activities of the program, described in prior sections. • Institutional sustainability, particularly at regional and local levels is linked to capacity and ownership, but almost more so to financial sustainability. While much of the work with DREAHs, mayors, and STEAHs has been directed at strengthening capacity and ownership and on formalizing arrangements for the long term, there is still a high dependence on the program and risks identified, for example around the STEAH. But program strategies in this respect do show progress, particularly with the PPP model. 10 References to international ODF slippage experiences including RANO HP can be found in: USAID, 2018. An Examination of CLTS’s Contributions toward Universal Sanitation. Washington, DC., USAID Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) Project. https://www.globalwaters.org/sites/default/files/clts_desk_review_final.pdf 11 See: https://wash-alliance.org/our-approach/sustainability/ MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 46 • Environmental sustainability was mentioned as an under-emphasized area by some informants, and program staff and partners seemed particularly preoccupied with the complexities and time taken by environmental compliance procedures. • Technical sustainability is a clear objective of the PPP model and a driver for behavior change programming. The program team needs to take care to ensure that it is mutually reinforceable with behavioral sustainability. The latter will benefit from a program end-survey, including a suggested focus on habit creation, and links between behavior and vulnerability/ inclusion. • Social sustainability is addressed particularly through the focus on local structures, some risks cited in Section 3.2, and on gender, youth, and physical disability, but with some scope to do more to ensure inclusion and equity of all groups (see the next section). In general, several program staff said that the program has been working on the various exit strategies it will need to ensure sustained results beyond the program duration. In this light, the majority of informants felt that further reinforcement of capacity, local structures, and local ownership should be a key priority for the remainder of the program. 3.4.5 EQUITY AND INCLUSION IN PRACTICE The Evaluation Team’s primary reflection on inclusive planning is captured in previous sections on design and the PPP model. With regard to SO3, recent sector studies have shown that CLTS is not automatically inclusive unless special measures are taken to understand and address the multiple types of disadvantage and vulnerability at play, which go far beyond gender.12 Yet, multiple KIIs still confirmed the sentiment among informants that “everybody was included.” As for non-adopters, the desk review and KIIs did not reveal that implementing partners and local actors had been trained or supported to better understand so-called clusters of disadvantage or possible vulnerabilities underlying people’s inability or decision to not build, use, or maintain a latrine. The team also heard little if any emphasis on so-called Do No Harm principles.13 As pointed out above, these principles are particularly important given the widespread application of practices like DINA, fines, and threats of exclusion through for example the Carte WC. The Evaluation Team did note one of the DINA schemes discussed in interviews, which sets aside the funds collected through fines to assist some of the vulnerable households, though these same households would still need to first pay the fines. In the 2020 Annual Report, the program describes advising communities to be aware of vulnerable households as they set and carry out sanctions and fines. It is possible the team only observed a fraction of the program’s Do No Harm strategy. There is also a question on how well the extensive gender and inclusion work done under PCT leadership concretely affected efforts on the ground. As noted in one KII, consortium partners commended the large amount of work done on gender, but that the approach had been very time and data-intensive and that it wasn’t clear how all that information was ultimately used to inform approaches and activities. With something as transversal and potentially complex as gender, care needs to be taken that any proposed approach is ‘doable’ in practice. It is important to prioritize perhaps initially, a relatively small set of ‘simple, immediately doable actions.’ In this context, the team noted for example the RANO WASH Gender and Social Inclusion Mainstreaming Strategy (April 2019) outlined 35 different sets of 12 See for example: House, S., Ferron, S., Cavill, S. (2017). Scoping and Diagnosis of the Global Sanitation Fund’s Approach to Equality and Non-Discrimination. Geneva, Switzerland: WSSCC 13 See for example: Water for Women Fund, SNV and DFAT (2020). ‘Do No Harm’ for inclusive WASH: working toward a shared understanding’, Learning brief – Systems Strengthening / Leave No One Behind. Melbourne, Water for Women Fund MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 47 activities across three strategic objectives. While critical, the Evaluation Team questions whether this was appropriate given the already highly complex program design. An important question is whether all the RANO WASH staff and partners are really conscious of the gender and inclusion thematic and whether they really take ownership and know how to integrate these themes into their work. Perhaps we need more workshops on how to concretely do this, also at commune level. There is lots of talk of social inclusion, but it is not clear what is retained by staff. – Consortium Partner The RANO WASH team itself has identified some of its challenges regarding implementing a gender and socially inclusive approach. As discussed in the Rapid Gender Analysis for COVID-19 in Madagascar conducted in 2020, concrete recommendations include better “understanding schedules” to facilitate different people’s ability to participate in activities or interventions. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 48 4.0 RECOMMENDATIONS AND PRIORITIZATION 4.1 RECOMMENDATIONS FOR FUTURE DESIGN OR PROGRAM EXTENSION • Consider extending the timeframe for a program of this complexity and ambition. The kind of systems change envisaged in this program could easily span a decade. • Continue to work with the building blocks approach, taking time and care to familiarize people and help them operationalize it, and continue along the three pillars. But equally, consider whether there are elements that may be better handled in other programs or by other sector actors to keep the program focus manageable, for instance, the work on integrated management of watersheds with the MoWASH and Ministry of Environment. • Move toward a district or commune-wide, inclusive planning and design approach to progressively cover WASH service delivery in all types of communities: – Consider a reduction in target communes, but with a ‘total service delivery’ approach under commune leadership. In other words, an approach that would cover WASH services all fokontanys and communities; households, institutions, and public spaces. – Capitalize on RANO WASH’s PPP achievements and rapport with MoWASH to support the ministry in identifying the first districts or communes most able to demonstrate a district or commune-wide approach with private sector engagement; invest in specific expertise in planning and financial modeling to build economic scenarios, technical solutions, and business plan models. – Add a layer of longer term and business planning to the PCDEAH, acknowledging this will initially need program technical support, including economic analysis and multi-year planning to gradually cover entire communes, including institutions, possibly through a transitional mix of private and non-private management approaches. Invite input from WSPs into this planning process and commune discussions. • Design for scale and scalability: – Explore private sector or transitional private—community management models that can include larger as well as smaller communities. – With regard to governance and behavior change programming, focus on scalable approaches with community ownership and a clearer program exit strategy. • Work with MoWASH and government partners to develop different subsidy approaches or financial support to WSPs so they can offer affordable services to higher numbers of vulnerable households. Examples include a water meter subsidy, tax rebates for private sector providers, or results-based funding to connect vulnerable households. • Continue to focus on increasing the sector funding base, including through private sector engagement and facilitation of dialogue with financial institutions. One useful outcome could be a Guarantee Fund for WASH. However, explore the possibilities of leaving an intervention like increasing local tax collection to other programs. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 49 • Negotiate government co-financing at the start of a program to cover, at a minimum, DREAH and STEAH salaries and operating expenses. • Carefully consider whether extension of private sector WASH service delivery in urban areas (beyond commune capitals) should be included in the same program or done separately. Future process and structure • Rather than a phased approach, start engaging in all intervention regions at the same time, accounting for context and maximizing time for all stakeholders to build relationships and mutual understanding • Review the internal consortium set-up and seek ways to reduce procedural load. The USAID Partnership Incubator may be a good resource for the consortium. 4.2 RECOMMENDATIONS FOR THE REMAINDER OF THE PROGRAM 4.2.1 INSTITUTIONAL ARRANGEMENTS • Advocate with the MoWASH and Ministry of Decentralization for an increase in DREAH budgets to cover their full range of responsibilities, including oversight and verification. • Progressively move toward making DREAH responsible for ATEAH training to prepare for the program’s exit. Explore ways to strengthen lines of engagement and dialogue between DREAH and STEAH. • Work with MoWASH and DREAH and advocate more directly with the Ministry of Decentralization to formalize the status of STEAH and ATEAH and ensure their inclusion in communal budget lines. • Undertake a review of the breadth of responsibilities and required skills of ATEAH, to ensure effective delivery, a realistic workload and sustained continuity of the model • Support phased establishment of local input and dialogue structures, along the needs of the commune, to avoid confusion and overlap. Start with the SLC in line with government stipulations – and advocate for the required inclusion in commune budgets. Establish and support ASUREPs where appropriate. Encourage community decision making on OSC establishment and the identification of discrete roles and responsibilities. Encourage mayors to share information more proactively with commune stakeholders. 4.2.2 PLANNING, MONITORING AND EVALUATION ● In collaboration with the MoWASH and sector partners, engage in a more in-depth assessment of the root causes that hinder successful implementation of national sector monitoring and planning, beyond challenges with the SE&AM tool. Include an assessment of the capacities required for the establishment of a solid monitoring, evaluation and planning system for the sector. 4.2.3 PRIVATE SECTOR ENGAGEMENT IN WASH • Develop clear criteria to guide decision making on the process of transitioning from community management to private sector management, communities to prioritize in this transition, and any transitional measures required. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 50 • Design and develop a package of support materials and activities for communes that simultaneously oversee private and community management methods in their territories. These tools will help communes more consistently monitor all services and costs. • Continue to operationalize PPP+ components and build diversified marketing and customer relationship management skills to increase coverage. Consider setting targets or bonuses for inclusive, possibly phased, service delivery to more vulnerable households per scheme. • Work with MoWASH and partners, including AOPDEM, to mold a common vision of the role of the private sector in Madagascar’s WASH sector. Continue to explain and promote the RANO WASH PPP model as one useful option. • Work with communes to develop longer-term plans and strategies for universal drinking water supply at the level of a commune, its chef lieu and villages of all fokontany. Invite private sector WSPs to engage in this commune-level planning, in terms to be specified. 4.2.4 PPP CONTRACTING AND ROLL-OUT • Simplify the PPP contract and make it available in Malagasy at the commune level so that commune staff, ATEAH and SLC, OSC and ASUREP members can better understand the content and mutual responsibilities laid out in the contract. • Review the WSP selection process and/or ensure full knowledge and understanding of relevant passages among PPP stakeholders, particularly around WSP contracting for extensions or within geographical regions, to allow for economies of scale. 4.2.5 COORDINATION AND DIALOGUE • Continue to play the role of sector facilitator at the national level, actively engaging in and, where appropriate, driving sector dialogue, learning, and sharing, and joint sector review. 4.2.6 COMMUNICATION AND CLARIFICATION WITH PROGRAM BENEFICIARIES • Clarify and clearly communicate which communes, communities, and institutions will receive what support for water system construction or rehabilitation and/or PPP establishment between now and the end of the program. • Ensure equal distribution of tools to masons and seamstresses, especially where promises were made. 4.2.7 MARKET-BASED SANITATION • Build on the iDE study to better understand the potential and limitations of market-based approaches, including the population’s ability and willingness to pay, seasonality factors, access to finance and potential flexible payment options. • Work with development partners and the MoWASH to learn from experiences and jointly develop a shared sector roadmap or strategy toward market-based sanitation. • Review support to masons within the context of broader professionalization of sanitation business models: – Review the mason and seamstress selection model and avoid selection or appointment by mayors. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 51 – Increase marketing and business planning training to enable entrepreneurs to set appropriate price models and effectively market their products. – Continue to explore ways to strengthen supply chains and to allow entrepreneurs to buy raw products and prepare stock upfront. Examples include offering micro-finance solutions or tax￾rebates. 4.2.8 WASH IN SCHOOLS AND HEALTH CARE FACILITIES • Continue joint advocacy with sector partners, particularly UNICEF, to promote a better model or improvements to the WASH Friendly institutions approach to increase the number of schools and HCFs with sufficient, sustainable WASH infrastructure and practices. 4.2.9 EQUITY AND INCLUSION • Elaborate on existing training efforts and processes and share good practices to strengthen understanding of implementing partners and commune stakeholders, particularly mayors, ATEAH and SLC, on Do No Harm principles and the potential unintended consequences of using force or penalties as a means of behavior change motivation. 4.2.10 BEHAVIOR CHANGE PROGRAMMING • Continue to target ODF communes, covering both ODF and non-ODF communities with a tailored package of RANO WASH behavior change strategies. • Review MHM training and messaging provided, to ensure correct understanding of safe management and facilitate informed choice. 4.2.11 LEARNING AND DOCUMENTATION • Refine program learning and sharing on the systems approach and building blocks. The program should prioritize documenting good practices on governance, private sector contracting, and STEAH strengthening, and offer peer-to-peer exchanges between communes. • Ensure continued learning and documentation of the RANO WASH behavior change strategy and approaches used, focusing on scalability and sustainability, and recommend effective combinations of approaches. • Conduct a program end-line or outcome survey, including a focus on habit creation, equity and inclusion, and service delivery. If possible, consider this to be part of a MoWASH-led outcome or sector survey covering intervention zones of several programs. • The Evaluation Team recommends that USAID facilitate exchange and joint learning between USAID-funded rural water supply programs with private sector engagement in different countries, particularly around PPP elements, risk identification and mitigation, and inclusive private sector service delivery models. 4.2.12 SUGGESTIONS ON PRIORITIZATION Below the Evaluation Team suggests some activity areas the program should prioritize and deprioritize in the time remaining. The implementers and USAID should review whether the vast range of activities undertaken by the program under the three pillars is possibly too much for one program to produce solid results across the board. Some of the suggestions for deprioritization do not concern relevance of the activities as such, but rather a determination of the best-placed programs or actors to take them further. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 52 TABLE 5: RECOMMENDATIONS OF PROGRAM AREAS TO PRIORITIZE/DEPRIORITIZE Theme Prioritize Deprioritize Systems approach and Continue capacity building on the Deprioritize the tax collection building blocks systems approach and building blocks, but be realistic in the remaining time left. Prioritize national MoWASH staff. training and support. The Evaluation Team recommends USAID explore adding this workstream to another program. Systems approach and Start discussions with the MoWASH Deprioritize the additional work building blocks and key sector partners on the selection of districts and communes to demonstrate the district-wide approach. planned to integrate water resource management. Consider removing this component or shifting it to a separate program in the next iteration of WASH programming. Document reflections based on work to date. Governance Prioritize work with Ministries to formalize the role of STEAH and ATEAH and ensure continuity of their work. Prioritize supporting sustainable resourcing of the STEAH and DREAH structures. Governance Prioritize building local capacity and ownership to ensure sustainability of structures and roles in preparation for RANO WASH exit Water PPPs Prioritize finalizing the PPP contract processes that are underway. Manage the expectations of communes/ communities that won’t be covered during the remainder of the program. Demand and use of Ensure an independent outcome Deprioritize efforts to support the WASH services survey of sanitation and hygiene related access, use, sustainability and equality outcomes before program end. Where possible, secure MoWASH leadership to conduct a larger survey incorporating several programs into the scope. roll-out of WASH-Friendly schools and healthcare facilities, instead focusing on small doable actions in institutions within the scope of ongoing behavior change and PPP programming. Demand and use of WASH services Address MHM misinformation and review training and messaging provided. Research and learning Prioritize documentation of commune successes and facilitation of peer-to￾peer learning between successful and ‘lagging’ communes. Deprioritize embarking on any new studies that had been originally planned, including with LSHTM. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 53 Mid-Term Performance Evaluation of Madagascar Rural Access to New Opportunities in Water, Sanitation, and Hygiene (RANO WASH) Under Cooperative Agreement Number: AID-687-A-17-0002 ANNEX I: EVALUATION SOW Identifying Information Program: Health, Population, and Nutrition (HPN) Office Activity Name: Rural Access to New Opportunities in Water, Sanitation, and Hygiene (RANO WASH) Implementing Partner: CARE-led Consortium Award Number: AID-687-A-17-002 Activity COR/AOR: Patricia Norolalao Award Dates: June 15, 2017 to June 15, 2022 Activity Funding: $30 million Period to be Evaluated: June 15, 2017 to Present Date Completed Evaluation is Needed: TBD Estimated Evaluation Budget: $200,000 BACKGROUND RANO WASH is USAID/Madagascar’s largest bilateral WASH investment with a five-year duration and scheduled end date of June 2022. RANO WASH will reach 250 rural communes in six high-priority regions of Madagascar: Vatovavy Fitovinany, Atsinanana, Alaotra Mangoro, Amoron'i Mania, Haute Matsiatra, and Vakinankaratra. The project addresses Water, Sanitation, and Hygiene (WASH) challenges at local, regional and national scales. To accomplish this goal, the CARE International-led consortium, including sub-awardees Catholic Relief Services (CRS), WaterAid, and local partners BushProof, and Sandandrano, has developed a systematic partnership with national and regional governments, water and sanitation institutions, communities, private sector actors, civil society organizations, and beneficiaries. Through its activities, the project implements a strategic set of mutually supporting activities that contribute to three interlinked strategic objectives: (1) Strengthening the governance and monitoring of water and sanitation; (2) Increasing the engagement of the private sector in the delivery of WASH services; and (3) Accelerating the adoption of healthy behaviors and the use of WASH services (See Theory of Change Figure below). MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 54 FIGURE 1: RANO WASH THEORY OF CHANGE AND ILLUSTRATIVE ACTIVITIES EVALUATION PURPOSE USAID seeks the services of a contractor to evaluate the performance of the USAID/Madagascar RANO WASH Activity. The primary purpose of the evaluation is to assess the activity’s design and implementation to inform future USAID/Madagascar and sector WASH programming. The evaluation will inform any midcourse corrections for the final two years of implementation. The primary audience for the evaluation report will be the HPN Office, current implementing partners, Government of Madagascar, and water-sector working groups and collaborators. To a lesser degree, the evaluation findings may be used by USAID/Washington technical staff to inform efforts in the WASH sector. DISSEMINATION AND UTILIZATION PLAN USAID/Madagascar will disseminate the report widely with relevant stakeholders and the implementing partner, and it will also be made publicly available for other interested parties. The findings will be used by each of the relevant audiences to influence design and implementation of current and future activities. KEY EVALUATION QUESTIONS 1. How appropriate is the RANO WASH design to the WASH challenges in the target regions and communes? 2. To what extent has RANO WASH built governance capacity to improve sustainable service delivery at multiple levels? 3. To what extent have the different RANO WASH private sector approaches for water service delivery and sanitation access expansion been successful? 4. To what extent have RANO WASH activities been successful at building demand, activating demand, MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 55 and ensuring use of WASH products and services in intervention regions and communes? 5. What implementation approaches should be prioritized in future for RANO WASH’s final 2 years and by the WASH sector? In responding to these questions, the Contractor should consider gender as a cross-cutting theme, as this is a key priority for RANO WASH and USAID/Madagascar. METHODOLOGY The performance evaluation methodology will be undertaken at the highest degree of rigor to ensure credible findings and recommendations. A mixed-method approach will draw on the following key data gathering techniques. Desk Review: A detailed desk review of relevant project documents and reports including Government of Madagascar (GOM) documents, donor-funded programs, and published WASH literature. Key internal sources of information include but are not limited to: • RANO WASH Program Description • CARE consortium’s proposal to USAID • Approved Monitoring, Evaluation and Learning (MEL) Plan • Annual work plans • Quarterly progress reports to USAID/Madagascar • Annual reports • Project success stories, case studies, presentations, snapshots, and other communication material such as videos • Project deliverables and activity reports • MEL data, including GIS information, cartography, and knowledge products created for MEL purposes • List of contact information for local authorities. Key Informant Interviews: Interviews with USAID/Madagascar, relevant GOM staff at national and regional levels, RANO WASH, private sector actors, civil society organizations, and knowledgeable beneficiaries. A list of proposed interviewees will be submitted with the inception report. The final list of interviewees shall be agreed upon in consultation with USAID and the RANO WASH team. Focus Group Discussions: Focus Group Discussions (FGDs) to obtain the required data from project beneficiaries where applicable. The groups shall be constituted in consultation with USAID and the RANO WASH team. Site Visits: To further compliment information/data generated through key informant interviews and literature review, field visits should be done in up to all six implementation regions. At a minimum, it is important to include samples from the high plateau and the coastal regions to represent the diverse geography, cultural dynamics, and thematic interventions. Sampling must be balanced with logistical and cost implications. USAID/Madagascar will review and approve a data analysis plan that details how data generated will be transcribed and analyzed. This plan should enumerate the procedures that will be used to analyze MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 56 qualitative data from key informants and other stakeholder interviews prior to the start of data collection. OPERATING CONSTRAINTS The offeror should outline any anticipated constraints in designing the methodology to be used or the operations and logistics of implementation. Specifically, this should include any specific challenges related to site visits, security concerns, and, of course, COVID restrictions. PARTICIPATION The primary point of contact for this evaluation will be the RANO WASH AOR, Patricia Norolalao, who, with WASH specialist Daniel Nover, will serve as a resource person to the Evaluation Team. She, with other USAID staff, will provide technical guidance pertaining to questions the Evaluation Team might have during the implementation of the SOW, and approve the evaluation methodology prior to the start of any fieldwork. The USAID team will assist the Evaluation Team in their work by reviewing draft deliverables, responding to questions and resolving administrative or logistical obstacles. USAID staff may accompany the Evaluation Team to selected site visits as observers but will not be full members of the Evaluation Team. Implementing partner staff will also support the Evaluation Team by providing project documentation and information about project implementation. RANO WASH staff will also be available to facilitate connections with key informants and beneficiaries for interviews or focus group discussions. STANDARDS The work plan proposal, including methods, and the final evaluation must be consistent with and meet the standards of USAID’s Evaluation Policy. EVALUATION TEAM The offeror shall propose a team, including a team leader, to successfully evaluate RANO WASH, answer the evaluation questions, and complete deliverables in the specified timeline. The Evaluation Team shall consist of international and national staff and must include the following expertise: • Water, sanitation, and hygiene (WASH) programming in Madagascar • Familiarity with Public Private Partnerships in WASH service delivery • Familiarity with market-based approaches to WASH service delivery • Extensive knowledge in Community Led Total Sanitation (CLTS) • Analysis of WASH infrastructure design, functionality, and sustainability • Social behavior change and behavior change communication • Experience with donor-funded activities, especially USAID • Experience in evaluation design and management • Field-based monitoring The Team Leader will serve as the primary point of contact between the USAID and Evaluation Team. The incumbent must: • Be able to communicate effectively with senior U.S. and host country officials and other leaders. • Demonstrate 10+ years record of leadership, coordination, and evaluation delivery for development projects and programs. • Have excellent writing/organizational/management skills and proven ability to deliver a quality written product (Evaluation Report and PowerPoint). MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 57 Please include CVs of the proposed team lead and primary technical staff in the proposed workplan and/or inception report for review by USAID. The Evaluation Team should plan on identifying and funding a small local team of enumerators to provide support for field work for easy access and communication in the local languages. NOTIONAL SCHEDULE AND LOGISTICS Contractor to estimate LOE and schedule. Note: Logistics & Timeline (assumes Saturdays as workdays in Madagascar) The following provides a notional presentation of a prospective allocation of level of effort for the Evaluation: FIGURE 2: EVALUATION SCHEDULE ACTIVITY EXPECTED DURATION LOCATION Preparation – Document review. Finalization of evaluation methodological approach and proposed field schedule. Development of questionnaires and/or other tools to be used in conducting surveys and fieldwork. To be approved by USAID/Madagascar before initiation of field work. 5 days U.S. Field Work – In-brief with USAID/Madagascar, interview of USAID staff and IPs, partners, and beneficiaries to confirm results, identify activity achievements and constraints, as well as stakeholder views on how assistance could more effectively achieve desired results. 30 days Madagascar Follow-up and synthesis – In-country teamwork and delivery of draft report/findings to USAID/Madagascar Mission and stakeholders. Additional meetings and interviews may also be scheduled to validate/confirm findings and debrief. 7 days Madagascar Revision and refinement – In response to comments from USAID, team will incorporate feedback and other input into analysis, report, and presentation. 3 days U.S. Final report production and presentation – Completion and delivery of final evaluation report and summary power-point presentation. 1 day U.S. PERIOD AND PLACE OF PERFORMANCE 1) The evaluation to be conducted between dates to be proposed by WASHPaLS. 2) The evaluator will randomly select districts from each region where the data will be collected. RANO WASH is implemented in the following Regions: Vatovavy Fitovinany, Atsinanana, Alaotra Mangoro, Amoron'i Mania, Haute Matsiatra, and Vakinankaratra. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 58 DELIVERABLES The following is an illustrative description and timeline for deliverables. Contractor to propose timeline, considering that the rainy season (historically Jan - March; more sporadic in contemporary times) should be avoided and can hinder on-the-ground activities. FIGURE 3: DELIVERABLES NO. DELIVERABLE DESCRIPTION DUE FORMAT NOTES 1 Initial meeting USAID, contractor, and evaluation team to clarify evaluation questions, proposed methodology, roles and responsibilities, logistical issues, and timelines. Day Two of evaluation activities meeting 2 Inception Report Inception Report: This includes a workplan that provides: • detailed evaluation methodology • final list of proposed key informants, including IP staff • draft data collection tools • limitations and mitigation measures • timeline 14 days after #1 Word document 3 Weekly updates Weekly update of progress. Monday OOB Email During field work only 4 Debriefing/Presenta tion of preliminary findings Debrief/Presentation of preliminary findings and conclusions: • Validation workshop with IPs • USAID/HPN debrief, possibly including key stakeholders 5 days after preliminary report submitted PowerPoint presentation Submit the day prior 5 Preliminary Draft Evaluation Report Draft report of preliminary findings. 2 weeks after field work completed Word document 6 Final Report Final evaluation report and a short brief due to USAID: submitted to USAID along with all raw data on a hard device. Submitted 10 days after feedback on Draft received Word Document 7 Upload to DEC Final report uploaded to the DEC. 5 days after #6 8 Dissemination meeting Possible dissemination meeting with the IP and other stakeholders. To be discussed and agreed upon with USAID. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 59 Final Report: Final evaluation report is due to USAID/Madagascar within 10 workdays of receiving USAID and IP’s comments on the revised draft evaluation report and based on the provisions of the USAID evaluation policy. The final report incorporates all team responses to comments and suggestions. The report shall include an executive summary (highlighting key lessons learned), table of contents, list of acronyms, evaluation design and methodology, limitations, findings, conclusions, recommendations, and lessons learned as shown below. The report shall be submitted in English, in both electronic and written copies. The final report must not exceed 25 pages excluding annexes. A summary of this report, not exceeding 4 pages, excluding any potentially procurement-sensitive information, shall be submitted (also electronically, in English) for dissemination among implementing partners and stakeholders. The report must meet standards outlined in the evaluation policy. The final evaluation report must conform to the standards set forth in the Agency Evaluation Policy as follows: Reporting Requirements Executive Summary The Executive Summary will state the development objectives of the program/project evaluated; purpose of the evaluation; study method; findings; conclusions, lessons learned and future design implications. Table of Contents Introduction The context of what is evaluated including the relevant history demography socioeconomic and basic political arrangements. Body of the Paper 1. The purpose and study questions of the evaluation. Include a brief description of the project activities in Madagascar. 2. Methodology 3. Evidence, findings, and analysis of the study questions. 4. Conclusions drawn from the analysis of findings stated succinctly. 5. Recommendations. Appendices shall include: 1. Evaluation scope of work 2. List of relevant project targets and results 3. List of documents consulted 4. List of individuals and agencies contacted 5. Technical topics including study methodology if necessary 6. Schedule of activities in an Excel format. The Final Evaluation Report and PowerPoint addressing the USAID's comments should be submitted in both Word and PDF formats. Once the PDF format has been approved by USAID, the Team will submit MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 60 the Final Evaluation Report to the Development Experience Clearinghouse (DEC) for archiving￾www.dec.usaid.gov. In addition, the contractor must submit a one-page abstract of the findings/recommendations considered most important for USAID senior managers. If the report contains any potentially procurement-sensitive information, a second version report excluding this information shall be submitted (also electronically, in English). All primary source data, both quantitative and qualitative, generated during the course of evaluation must be provided to USAID in an electronic file in an easily readable format; organized and fully documented for use by those not fully familiar with the activity or the evaluation. In addition, all background documents collected for this evaluation must be provided to USAID on CDs, organized by implementing mechanism, along with the final report. The report and the data should also be submitted to the Development Experience Clearinghouse (DEC) and the Development Data Library (DDL). MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 61 - ANNEX II: EVALUATION METHODS AND BACKGROUND EVALUATION SUB-QUESTION SUMMARY Below evaluation questions are presented along with their sub-questions. FIGURE 1: EVALUATION SUB-QUESTION SUMMARY EVALUATION QUESTIONS EVALUATION SUB QUESTIONS EQ1: How appropriate is the RANO WASH Sub-questions deal with national policies, strategies and sector design to the WASH challenges in the target dynamics; collaborative design; interplay between the SOs; and regions and communes? the extent to which equality and inclusion were designed into the program. EQ2: To what extent has RANO WASH built governance capacity to improve sustainable service delivery at multiple levels? Sub-questions deal with governance capacity at the national level as well as at regional and communal levels; and community and civil society engagement and control over WASH services. EQ3: To what extent have the different RANO Sub-questions deal with market shaping; private sector vs. non￾WASH private sector approaches for water private sector delivered program results; the different private service delivery and sanitation access expansion sector approaches for water service delivery and for sanitation been successful? and hygiene delivery deployed by the program; as well as WASH in Schools and in Health Care Facilities. EQ4: To what extent have RANO WASH Sub-questions deal with the activities undertaken to change activities been successful at building demand, behaviors and build demand; harmonization of approaches and activating demand, and ensuring use of WASH learning and research activities; the extent to which sustainable products and services in intervention regions and results have been achieved; and impact on women, girls and communes? people in vulnerable situations. EQ5: What implementation approaches should be prioritized in future for RANO WASH’s final 2 years and by the WASH sector? Sub-questions deal with prioritization, adaptation, and reprioritization. DESCRIPTION OF SELECTED REGIONS FOR FIELD WORK Alaotra Mangoro. Alaotra comes from the name of the large lake Alaotra located in the middle of the districts of Ambatondrazaka and Amparafaravola (the largest lake in Madagascar) while Mangoro is the name of the main river crossing Moramanga and Anosibe An’Ala from north to south. The rivers of the Alaotra Mangoro Region are mostly non-navigable because they are cut by dangerous waterfalls and rapids. The flow of water is strongly linked to rainfall and the rivers are very sensitive, causing sudden and violent floods during the rainy season, with implications for WASH services. The Alaotra Mangoro Region has 79 municipalities spread over its five districts including Andilamena, Amparafaravola, Ambatondrazaka, Moramanga and Anosibe An’Ala. Of RANO WASH’s target communes, three communes in the district of Moramanga and two communes in the district of Amparafaravola have benefited from water infrastructure. The implementation of activities in this region is ensured by the Technical Partner SAF FJKM Unit Moramanga, under the supervision of WaterAid. The region was part of the first wave of programming. Atsinanana: The Atsinanana region is located in the province of Toamasina. It is made up of seven districts namely Toamasina I, Toamasina II, Brickaville, Vatomandry, Antanambao, Manampotsy, Mahanoro, Marolambo, jointly comprising 85 communes. The Atsinanana region has several rivers, most of them fast flowing. Water flow is strongly linked to rainfall, and floods are sudden and violent during the rainy season. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 62 The entire coastal zone includes a multitude of ferries and bridges, often washed away or destroyed during floods. RANO WASH operates in 51 communes spread over four districts. The charitable organization Organe de Développement du Diocèse de Toamasin plays the role of technical implementation partner, with the support of CRS as consortium member. Amoron’i Mania: The Amoron’i Mania Region is located in the central part of the southern highlands (province of Fianarantsoa) and is made up of four districts. The hydrology of the Amoron’i Mania Region is dominated by the Tsiribihina watershed, which originates in the region and flows into the Mozambique Channel, once recovered by the Tsiribihina river. RANO WASH operates in 30 communes in three districts of the Region. The Amoron’i Mania Region has not yet benefited from program water infrastructure. However, procedures to install water infrastructure have started. Program implementation in this region did not start until 2020, which justifies the Evaluation Team’s decision to evaluate activity progress against how long implementation has been ongoing. The NGO, Act Intercooperation Madagascar ensures the implementation of activities in this region, especially those under SO3, supported by CARE. Vatovavy Fitovinany: The Vatovavy Fitovinany region is part of the province of Fianarantsoa and is located in the south-eastern part of Madagascar. The main rivers in the region are, from north to south, the Sakaleona, the Mananjary, the Namorona, the Faraony and the Matitanana. These are short rivers which originate in the central highlands and have torrential and rapid courses and waterfalls. The region is subdivided into six districts, with the town of Manakara as its capital. It has 139 municipalities. RANO WASH operates in 65 communes. The field activities have been carried out by the NGO NY TANINTSIKA since 2018. Vatovavy Fitovinany is one of the program’s first intervention regions and was intentionally chosen for its long history collaborating with NGOs and donor-funded projects on WASH activities. The commune is also notable because of a socio-cultural taboo held by the community about the use of latrines. DATA CODING AND ANALYSIS IN DEDOOSE The Team Lead uploaded transcripts to Dedoose as they were completed. Coding on June 15 and ended on the July 22, 2021. To minimize bias and increase consistency in the application of the codes, the Team Leader was the exclusive coder of all data sets collected during the exercise, though all team members reviewed and analyzed coded transcript text related to their area of expertise. In total, 103 transcripts were imported into Dedoose and 54 codes were applied 1697 times across the data sets. As would be expected, some codes entered in Dedoose at the outset of the exercise were applied infrequently, and in some cases parent codes were used more frequently because the chosen sub-codes (‘child codes’) were not felt to fully cover the load. The word cloud hereunder gives a sense of how frequently which codes were applied. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 63 FIGURE 2: DEDOOSE WORD CLOUD After the team coded most of the data, they isolated and downloaded coded transcript text related to each EQ to undertake the analysis. Four members of the Evaluation Team analyzed the data, in line with their areas of expertise. Analysis focused on identifying patterns, comparisons and substantive quotations; eventually generating an initial outline of a response for each EQ. While analyzing the data, the team reviewed the secondary data from the desk review and project monitoring data again to provide further contextual background, fill in gaps in understanding, and to help to triangulate findings from the qualitative analysis. The team also looked for patterns in the data to highlight convergence or divergence of opinion across activity types and stakeholder groups. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 64 ANNEX III: DATA COLLECTION INSTRUMENTS The examples below are illustrative but have been informed by a preliminary set of interviews with key stakeholders at the national level. Further work is needed to test the regional and local level guides. Referenced numbers (in brackets) refer to the evaluation sub-questions and are tools for the Evaluation Team to assess completeness of the interview guides. INTERVIEW TOPIC GUIDES Key Informant Interview Topic Guide – RANO WASH Staff This Topic Guide is intended for use in terms of data collection from RANO WASH staff. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide key informant interviews with these respondents. There are a few questions where a specific set of responses are provided to assist in the collection of data that can be more easily quantified. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. I. Introduction (~10 minutes) a. Thank the respondent for taking the time to participate in the interview b. Introduction to the researcher and the research i. Introduce yourself: I am a consultant residing in ______. I represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. ii. As part of its own planning for the next few years, USAID has asked us to conduct a mid-term evaluation of the RANO WASH program to assess its progress toward achieving its objectives. So, what we are trying to assess are the strengths and weaknesses of the program, its accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondent would like to remain anonymous, and that the assessment team will ask permission if would like to use a quote from the respondent in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondent is willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. e. Check whether respondents have any questions. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 65 II. RANO WASH’s approaches and wider contribution to the sector (~45 minutes) Transition: I would like to spend some time speaking with you about your knowledge of the USAID-supported RANO WASH program. a. (1.1) Do you recall the program design and target-setting phase of RANO WASH? In this process, to what extent were key policy documents, governance realities and the viewpoints of national, regional, and communal actors taken into account? Can you describe the processes used? What was your engagement or role in this process? Or: b. (2.1) According to you, what were the main challenges facing the MoWASH and the sector at the beginning of the RANO WASH program? Is the RANO WASH program desi gn in line with the needs to meet these institutional challenges? c. (1.2) What do you think of the three pillars of the program - governance improvement, increase of private sector service provision and behavior change? Is this design relevant for Madagascar? Is it appropriate? How have these three pillars interacted or affected each other and what, if any, have been the challenges in inter-connecting the three objectives? d. (2.1) According to you, what are the most significant contributions of RANO WASH in shaping the WASH sector in Madagascar? e. (2.2/3.3) For water service delivery, which RANO WASH regions or communes do you think are making most significant progress? What is driving the observed good progress? How has RANO WASH helped to build regional/communal capacity and/or service provider capacity in the region(s) in which you are working (or) in which the program is working? Explore the difference between rehabilitation of existing infrastructure vs. construction of new infrastructure in this regard. Probe for evidence of effectiveness f. (2.2/3.4) For sanitation service delivery and hygiene, which RANO WASH regions or communes do you think are making most significant progress? What is driving the observed good progress? How has RANO WASH helped to build regional/communal capacity and/or service provider capacity in the region(s) in which you are working (or) in which the program is working? Probe for evidence of effectiveness g. (3.3) What has been your experience with the PPP contract modality? What has worked well, and what have been some of the challenges, for you and the private sector partners? Please highlight any differences (different approaches, challenges) between managing rehabilitation of existing water services and newly built water systems. h. (4.1) Which of the approaches used by the program to increase demand and use of WASH products and services do you consider to be the most effective and why? What suggestions for improvement do you have to further increase demand and use of WASH products and services in target communities? i. (3.2) According to you, has the program been able to create, strengthen or shape markets for sanitation products and services? Can you give concrete examples? What have been the success factors and the challenges? j. (2/4.4) What is the likelihood of RANO WASH achievements being sustained? Why? Probe for achievements at the project/local level, as well as in terms or governance, and influencing stakeholders at the regional or national levels k. (1.4/4.5) RANO WASH’s goal is to contribute to equitable services and the program emphasizes the need for inclusion of particularly women and girls. In your knowledge, how was the program designed to ensure a gender and inclusion focus? In practice, what did the program do around gender and inclusion (of women, girls, and people in vulnerable situations) and how do you think this has affected the program outcomes? What could have been done differently or better? l. (general) Are there any internal or external factors that you think are undermining or affecting your efforts and performance? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 66 m. (general) What internal or external factors do you think have enabled RANO WASH to be successful in its efforts? n. (5.1) What would you like to see RANO WASH build on in its final year to have greatest impact at regional/communal level? And at national level? o. (5.1) What recommendations, if any, do you have for USAID with regard to the design and delivery of future programs to strengthen the WASH sector to ensure and expand access to sustainable WASH services? Additional optional questions depending on the role of the informant or staff member: p. (1.3/4.3) Beyond the consortium members, can you comment on the extent to which the RANO WASH program has collaborated with other sector partners? What kind of collaborative practices or processes did you use? (if need be, nudge on joint planning, coordination, implementation, verification, learning.) What has been the result of this collaboration? What in your opinion have been some of the challenges in collaborating with multiple partners? q. (4.3) What adaptations have you made following the action research on behavior change? What impact have these changes had on demand generation for water, sanitation, and hygiene? As appropriate, specific questions to emerge for each region based on the document review. Conclusion: ● Thank the respondent for their time ● Tell the respondent they are welcome to contact you to ask questions at a later date. Ask permission of the respondent to use their name in the report if you might use a quote. Note their response: ______________. __________________________________________________________________________ Key Informant Interview Topic Guide – National Government Staff This Topic Guide is intended for use in terms of data collection from national government staff or representatives. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide key informant interviews with these respondents. There are a few questions where a specific set of responses are provided to assist in the collection of data that can be more easily quantified. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. I. Introduction (~10 minutes) a. Thank the respondent for taking the time to participate in the interview b. Introduction to the researcher and the research i. Introduce yourself: I am a consultant residing in ______. I represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. ii. As part of its own planning for the next few years, USAID has asked us to conduct a mid-term evaluation of the RANO WASH program to assess its progress toward achieving its objectives. So, what we are trying to assess are the strengths and weaknesses of the program, its MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 67 accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondent would like to remain anonymous, and that the assessment team will ask permission if would like to use a quote from the respondent in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondent is willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. e. Check whether respondents have any questions. II. RANO WASH’s approaches and wider contribution to the sector (~45 minutes) Transition: I would like to spend some time speaking with you about your knowledge of the USAID-supported RANO WASH program. a. (general) What is your relationship with the RANO WASH program? In what ways do you engage with the program? b. (1.1) Do you recall the program design and target-setting phase of RANO WASH? In this process, to what extent were key policy documents, governance realities and the viewpoints of national, regional, and communal actors taken into account? Can you describe the processes used? What was your engagement or role in this process? c. (2.1) What were the main challenges facing the MoWASH and the sector at the beginning of the RANO WASH program? Is the RANO WASH program design in line with the needs to meet these institutional challenges? d. (1.2) What do you think of the three pillars of the program - governance improvement, increase of private sector service provision and behavior chance? Is this design relevant for Madagascar? Is it appropriate? How have these three pillars interacted or affected each other and what, if any, have been the challenges in inter-connecting the three objectives? e. (2.1) According to you, what are the most significant contributions of RANO WASH in shaping the WASH sector in Madagascar? f. (3.2) According to you, has the program been able to create, strengthen or shape markets for water and sanitation products and services? Can you give concrete examples? What have been the success factors and the challenges? g. (4.1) In your opinion, what are the contributions of RANO WASH in improving approaches aimed at changing behaviors and increasing demand and use of WASH products and services in Madagascar? What policies and strategies have changed as a result? Are there tangible results in one or more regions that stand out to you? Can you describe them? h. (4.1) In your opinion, does private sector involvement promote demand creation and use of WASH products and services? What are the limits and scope of their involvement in the sector? i. (1.3/4.3) Can you comment on the extent to which the RANO WASH program has collaborated with other sector partners? What kind of collaborative practices or processes has the program used? What has been the result of this collaboration? What have you observed as some of the challenges in this respect? j. (1.3/4.3) What kind of learning activities has the sector engaged in in recent years and what has been the result or impact of these learning processes? What has been RANO WASH’s contribution to sector learning? Are you aware of the RANO WASH applied research on behavior change and on nutrition and WASH.? How have lessons from these studies been applied in the sector? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 68 k. (1.4/4.5) RANO WASH’s goal is to contribute to equitable services and the program emphasizes the need for inclusion of particularly women and girls. To your knowledge, how was the program designed to ensure a gender and inclusion focus? In practice, what did the program do around gender and inclusion (of women, girls, and people in vulnerable situations) and how do you think this has affected the program outcomes? What could they have done differently? l. (4.4) Do you think the program results are sustainable? Why or why not? What could they do differently to ensure sustainability? m. (3.6) Specific question for Health and Education Government staff representatives: how has the RANO WASH program contributed to improving Education and Health sectors and Health and Education sector goals? n. (5.1/1.1) What aspects should be fostered and developed in the way the program was designed and implemented? What would you like to see RANO WASH build on in its final year to have greatest impact at regional/communal level? And at national level? o. (5.2/1.1) If you could change something in the way the program was designed, what would be your three main suggestions? p. (5.1) What recommendations, if any, do you have for USAID with regard to the design and delivery of future programs to strengthen the WASH sector to ensure and expand access to sustainable WASH services? (As appropriate, specific questions to emerge for each region based on the document review) Conclusion: ● Thank the respondent for their time ● Tell the respondent they are welcome to contact you to ask questions at a later date. Ask permission of the respondent to use their name in the report if you might use a quote. Note their response: ______________. _________________________________________________________________________________ Key Informant Interview Topic Guide – Regional Government S taff This Topic Guide is intended for use in terms of data collection from regional/local government staff. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide key informant interviews with these respondents. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. I. Introduction (~10 minutes) a. Thank the respondent for taking the time to participate in the interview b. Introduction to the researcher and the research i. Introduce yourself: I am a consultant residing in ______. I represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 69 ii. As part of its own planning for the next few years, USAID has asked us to conduct a mid-term evaluation of the RANO WASH program to assess its progress toward achieving its objectives. So, what we are trying to assess are the strengths and weaknesses of the program, its accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondent would like to remain anonymous, and that the assessment team will ask permission if would like to use a quote from the respondent in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondent is willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. e. Check whether respondents have any questions. II. RANO WASH’s approaches and wider contribution to the sector (~45 minutes) Transition: I would like to spend some time speaking with you about your knowledge of the USAID-supported RANO WASH program. a. (General) What is your relationship with the RANO WASH program? In what ways do you engage with the program? b. (1.1) Do you recall the program design and target-setting phase of RANO WASH? In this process, to what extent were the viewpoints of national, regional, and communal actors taken into account? Can you describe the processes used? What was your engagement or role in this process? c. (2.2) What were the main challenges facing the Regional MoWASH department and the sector at the beginning of the RANO WASH program, for example in terms of policies and strategies, coordination, monitoring and evaluation, budget allocation? Is the RANO WASH program design in line with the needs to meet these institutional challenges? d. (2.1) According to you, what are the most significant contributions of RANO WASH in your region/in shaping the WASH sector in Madagascar? e. 1.2) How have RANO WASH program st aff and its consortium members influenced regional and local level governance dynamics and sector coordination, including national – regional coordination? f. (1.3/4.3) Can you comment on the extent to which the RANO WASH program has collaborated with other sector partners in your region? What has been the result of this collaboration? What in your opinion have been some of the challenges? How did RANO WASH ensure complementarity of the activities within the program and with other programs in the same commune/region, and what was your role in coordinating this? g. (3.3) What has been your experience with the PPP contract modality? What has worked well, and what have been some of the challenges, for you and the private sector partners? Please highlight any differences (different approaches, challenges) between managing rehabilitation of existing water services and newly built water systems. h. (4.2) How has it been ensured that the designs of infrastructure implemented correspond to the expectations/demands of the target populations? How much influence do community members have in construction and management decisions and what mechanisms are in place for this? i. (4.1) Can you talk about the scope and limitations of the approaches applied by RANO WASH to increase demand and use of sanitation and hygiene products and services? (CLTS, VSLA, schools approach, sensitization, BCD approach, use of local radio, involvement of private operators). Which have you felt to be most effective, and why? What have been some of the challenges? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 70 j. (3.1) According to you, has the program been able to create, strengthen or shape markets for sanitation? Can you give concrete examples? What have been the success factors and the challenges? k. (4.1) Are there specific tangible results in your region or commune as a result of using the behavior change approaches adopted by the program? What are the results and through which approaches? l. (4.1) In your opinion, is the involvement of the private sector in the provision of WASH products and services adapted to the needs of the target population? Why or why not? How can this be improved? m. (4.5) In practice, what did the program do around gender and inclusion (of women, girls, and people in vulnerable situations) and how do you think this has affected the program outcomes? What could they have done differently? n. (4.4) Can you talk about the sustainability of the use of and access to WASH products and services installed or accessed as a result of RANO WASH program activities at the level of target communities? How long have results been sustained so far? What is being done to ensure long term sustainability? By whom? o. (5.2/1.1) If you can change something in the way the program was designed and implemented, what would be your three main suggestions? p. (5.1/1.1) What aspects should be fostered and developed in the way the program was designed and implemented? q. (3.6) Specific question for regional/local Health and Education Government staff representatives: how has the project contributed in improving Education and Health sector processes, outcomes or targets? As appropriate, specific questions to emerge for each region based on the document review. Conclusion: ● Thank the respondent for their time ● Tell the respondent they are welcome to contact you to ask questions at a later date. Ask permission of the respondent to use their name in the report if you might use a quote. Note their response: ______________. _________________________________________________________________________________ Key Informant Interview Topic Guide – Collaborating/Sector Partners Questions may be differentiated for each partner. This Topic Guide is intended for use in terms of data collection from RANO WASH collaborating partners and other sector partners. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide key informant interviews with these respondents, but some differentiation of questions is foreseen based on the type of partner. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. I. Introduction (~10 minutes) a. Thank the respondent for taking the time to participate in the interview b. Introduction to the researcher and the research MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 71 i. Introduce yourself: I am a Consultant residing in ______. I represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. ii. As part of its own planning for the next few years, USAID has asked us to conduct a mid-term evaluation of the RANO WASH program to assess its progress toward achieving its objectives. So, what we are trying to assess are the strengths and weaknesses of the program, its accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondent would like to remain anonymous, and that the assessment team will ask permission if would like to use a quote from the respondent in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondent is willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. e. Check whether respondents have any questions. II. RANO WASH’s approaches and wider contribution to the sector (~45 minutes) Transition: I would like to spend some time speaking with you about your knowledge of the USAID-supported RANO WASH program. a. (General) What is the professional link between your work and the RANO WASH program? Can you cite examples of collaboration? b. (1.1) Do you recall the program design and target-setting phase of RANO WASH? In this process, to what extent were key policy documents, governance realities and the viewpoints of sector partners taken into account? Can you describe the processes used? What was your engagement or role in this process? c. (2.1) What were the main challenges facing the MoWASH and the sector at the beginning of the RANO WASH program? Is the RANO WASH program design in line with the needs to meet these institutional challenges? d. (1.2) What do you think of the three pillars of the program - governance improvement, increase of private sector service provision and behavior change? Is this design relevant for Madagascar? Is it appropriate? How have these three pillars interacted or affected each other and what, if any, have been the challenges in inter-connecting the three objectives? e. (2.1) According to you, what are the most significant contributions of RANO WASH in shaping the WASH sector in Madagascar? f. (IF NOT ADDRESSED UNDER e) How have RANO WASH program staff and its consortium members influenced national, regional, and local level governance dynamics and sector coordination, including national – regional coordination? g. (3.3) From your observations and experience, what are your thoughts on the PPP contract modality? What has worked well, and what have been some of the challenges, for the public and private partners and for the recipient communities? h. (4.1) Can you talk about the different approaches the program takes to increase demand and use of WASH products and services? Which of the approaches do you consider to be the most effective and why? What suggestions for improvement do you have to further increase demand and use of WASH products and services in target communities? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 72 _________________________________________________________________________________ i. 3.1) According to you, has the program been able to create, strengthen or shape markets for sanitation? Can you give concrete examples? What have been the success factors and the challenges? j. (4.1) In your opinion, is the involvement of the private sector in the provision of WASH products and services adapted to the needs of the target population? Why or why not? How can this be improved? k. (4.5) In practice, what did the program do around gender and inclusion (of women, girls, and people in vulnerable situations) and how do you think this has affected the program outcomes? What could they have done differently? l. (4.4) Do you think the program results are sustainable? Why or why not? What could they do differently to ensure sustainability? m. (1.3/4.3) Can you comment on the extent to which the RANO WASH program has collaborated with other sector partners? How successful or challenging were these collaborations? What has been the result? Can you give specific examples of collaboration successes, challenges, and accomplishments, for example related to governance, private sector and CSO involvement, behavior change activities? n. (4.4) Overall, and particularly in the 6 RANO WASH program regi ons, would you say there is increased harmonization of behavior change or demand generation approaches? Why or why not? What have been the challenges around harmonization? o. (4.1) In your opinion, what roles should local civil society structures play to increase the demand and use of WASH products and services? How can these roles be strengthened, and what is RANO WASH doing in this regard? p. (General) If you could change something in the way the program was designed and implemented, what would be your three main suggestions? q. (5.1) What aspects should be fostered and developed in the way the program was designed and implemented? What would you like to see RANO WASH build on in its final year to have greatest impact at regional/communal level? And at national level? r. (5.1) What recommendations, if any, do you have for USAID with regard to the design and delivery of future programs to strengthen the WASH sector to ensure and expand access to sustainable WASH services? Additional optional questions depending on the role of the informant / staff member: s. (4.1/4.5) According to you, what are the implications of the involvement of private sector actors in improving the demand and use of products and services? For water supply? For sanitation and hygiene? Who is most affected by their involvement? Who benefits most? t. (4.3) Are you aware of the RANO WASH applied research on behavior change and on nutrition and WASH? How have lessons from these studies been applied in the sector? As appropriate, specific questions to emerge for each region based on the document review. Conclusion: ● Thank the respondent for their time ● Tell the respondent they are welcome to contact you to ask questions at a later date. Ask permission of the respondent to use their name in the report if you might use a quote. Note their response: ______________. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 73 Key Informant Interview Topic Guide – Private Sector Service Providers This Topic Guide is intended for use in terms of data collection from private sector water or sanitation service providers collaborating with RANO WASH. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide key informant interviews with these respondents. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. I. Introduction (~10 minutes) a. Thank the respondent for taking the time to participate in the interview b. Introduction to the researcher and the research i. Introduce yourself: I am a consultant residing in ______. I represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. ii. As part of its own planning for the next few years, USAID has asked us to conduct a mid-term evaluation of the RANO WASH program to assess its progress toward achieving its objectives. So, what we are trying to assess are the strengths and weaknesses of the program, its accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondent would like to remain anonymous, and that the assessment team will ask permission if would like to use a quote from the respondent in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondent is willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. e. Check whether respondents have any questions. II. General information ● Name of Company ● Type of company –(e.g. National Level Manufacturer, Regional Level Distributor, District level Distributor, small enterprise or artisan in community) ● Location: ● Name of Interviewee: ● Contact: ● Gender: ● Position in the company: ● How long has informant been part of the project? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 74 III. RANO WASH’s approaches and wider contribution to the sector (~45 minutes) Transition: I would like to spend some time speaking with you about your knowledge of the USAID-supported RANO WASH program. a. (General) What is your relationship with the RANO WASH program? In what ways do you engage with the program? b. (1.3/3) In your own words, can you describe the key elements of RANO WASH’s program for increasing private sector engagement in the WASH sector in Madagascar? c. (1.1/3) Which element was decisive for you to get involved in the partnership with RANO WASH? Does a similar thing exist in Madagascar? What blockage has this lifted? (For private sector partners /finance institutions who chose not to engage; which element(s) was/were decisive for you not to engage? Does a similar project exist in Madagascar?) d. (3.3/3.4) What has RANO WASH action enabled you to do? For example, what technical or business skills have you been able to expand thanks to RANO WASH support? What services have you added? e. (3.3) In your opinion, how appropriate is the RANO WASH private sector recruitment and contracting process? What types of actors does it attract? Can it be applied at the speed and scale required to reach BPOR/national objectives? What suggestions for improvement would you have? f. (3.3) What has been your experience with the PPP contract modality? What has worked well, and what have been some of the challenges, for you and the public sector partners? g. (3.1) According to you, has the program been able to create, strengthen or shape markets for water products and services? And for sanitation? Can you give concrete examples? What have been the success factors and the challenges? h. (4.2) How has it been ensured that the designs of infrastructure implemented correspond to the expectations/demands of the target populations? How much influence do community members have in construction and management decisions and what mechanisms are in place for this? How would you describe your relationship with the community-structures and community-members benefiting from your services? If applicable, nudge on the difference (challenges, approaches) between rehabilitation of existing services and construction of new systems. i. (4.1) How can you ensure that the products and services provided by your business are used by your targets? Which of the approaches adopted by the RANO WASH program best promote the creation of demand for your WASH products and services? What obstacles or challenges are there with regard to people accessing your products and services? j. (4.1) What are your suggestions for improving program approaches to further increase demand and use of WASH products and services in target communities? k. (4.1) In your opinion, is the involvement of the private sector in the provision of WASH products and services adapted to the needs of the target population? Why or why not? How can this be improved? l. (4.5) In practice, what did the program do / what did you do around gender and inclusion (of women, girls, and people in vulnerable situations) and how do you think this has affected the program outcomes? What could they have done differently? m. (4.4) Do you think your business/water scheme or the products you provide are or have the potential to be viable and sustainable in the long term, without RANO WASH support? What would need to change to ensure viability and/or sustainability? n. (1.4/4.5) RANO WASH’s goal is to contribute to equitable services and the program emphasizes the need for inclusion of particularly women and girls. In the way you collaborate with the program, what has been done or how have you been supported to build consideration of gender and inclusion of people in vulnerable situations into your activities, and how do you think this has affected the program outcomes? What could have been done differently? As appropriate, specific questions to emerge for each region based on the document review. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 75 Conclusion: ● Thank the respondent for their time ● Tell the respondent they are welcome to contact you to ask questions at a later date. Ask permission of the respondent to use their name in the report if you might use a quote. Note their response: ______________. _________________________________________________________________________________ Key Informant Interview Topic Guide – Civil Society/Community Representatives This Topic Guide is intended for use in terms of data collection from commune - or community -based stakeholders and civil society representatives. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide key informant interviews with these respondents. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. I. Introduction (~10 minutes) a. Thank the respondent for taking the time to participate in the interview b. Introduction to the researcher and the research i. Introduce yourself: I am a consultant residing in ______. I represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. ii. As part of its own planning for the next few years, USAID has asked us to conduct a mid-term evaluation of the RANO WASH program to assess its progress toward achieving its objectives. So, what we are trying to assess are the strengths and weaknesses of the program, its accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondent would like to remain anonymous, and that the assessment team will ask permission if would like to use a quote from the respondent in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondent is willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. e. Check whether respondents have any questions. II. RANO WASH’s approaches and wider contribution to the sector (~45 minutes) Transition: I would like to spend some time speaking with you about your knowledge of the USAID-supported RANO WASH program. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 76 a. (General) What is your relationship with the RANO WASH program? In what ways do you engage with the program? What benefits or support have you received from the program? b. (General) In your own words, can you describe the key elements of the RANO WASH program? What activities have been undertaken in your community as a result of RANO WASH? c. (4.1) Specifically with regards to activities that aimed to increase demand and change people’s WASH behaviors, which approaches/activities have you felt to be most effective? d. (4.1) What action has the community undertaken to change use of and access to water supply, sanitation, and hygiene facilities? What has been your role in this? e. (4.1) What are your suggestions for improving program approaches to further increase demand and use of WASH products and services in your commune/community? f. (4.1 and 4.5) According to you, what are the implications of the involvement of private sector actors in provision of WASH products and services? Who is most affected by their involvement? Who benefits most? g. (4.1) In your opinion, what roles should local structures play to increase the demand and use of WASH products and services? h. (2.3) What kind of accountability mechanisms have been put in place with support of RANO WASH, and what is your role herein? Do you feel there are sufficient mechanisms to hold local government and/or WSPs to account? If not what more could be done? i. (2.3) How likely do you think it is that the accountability mechanisms will continue to function after the end of the project? How likely is it that you will continue to engage on WASH issues and the accountability mechanisms? Why or why not? j. (4.4) Since the start of RANO WASH program activities, what changes have you witnessed in terms of demand and use for WASH products and services? Do you think the program results are sustainable? Why or why not? k. (1.4/4.5) How do you feel the needs and perspectives of women, girls, and people in vulnerable situations, for example people living with a disability, were taken into account in the design and provision of WASH services? How do you think this has affected the program outcomes? What could have been done differently? l. (5.2/1.1) If you could change something in the way the water scheme in your community was designed, constructed, and managed, what would be your three main suggestions? And what about sanitation and hygiene? More questions to be added based on local context/type of community stakeholder. As appropriate, specific questions to emerge for each region based on the document review. Conclusion: ● Thank the respondent for their time ● Tell the respondent they are welcome to contact you to ask questions at a later date. Ask permission of the respondent to use their name in the report if you might use a quote. Note their response: ______________. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 77 Key Informant Interview Topic Guide – Commune Mayors This Topic Guide is intended for use in terms of data collection from commune mayors or deputy mayors. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide key informant interviews with these respondents. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. I. Introduction (~10 minutes) a. Thank the respondent for taking the time to participate in the interview b. Introduction to the researcher and the research i. Introduce yourself: I am a Consultant residing in ______. I represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. ii. As part of its own planning for the next few years, USAID has asked us to conduct a mid-term evaluation of the RANO WASH program to assess its progress toward achieving its objectives. So, what we are trying to assess are the strengths and weaknesses of the program, its accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondent would like to remain anonymous, and that the assessment team will ask permission if would like to use a quote from the respondent in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondent is willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. e. Check whether respondents have any questions. II. RANO WASH’s approaches and wider contribution to the sector (~45 minutes) Transition: I would like to spend some time speaking with you about your knowledge of the USAID-supported RANO WASH program. a. Do you recall the selection phase for your commune to become part of the RANO WASH program? What were the selection criteria? How did you go about it? Who contacted you? b. What are the roles of the commune in the implementation of the program? What is the role of the mayor? c. What, if any, were the obstacles or challenges encountered by the commune at the start and during the program implementation? d. Are these problems currently resolved? If yes, how were they resolved? e. Do the objectives of the RANO WASH program respond to the needs of your commune? If yes, in what way? If no, why not? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 78 f. Capacity strengthening a. What types of capacity strengthening have been carried out by RANO WASH? b. Who in the commune has benefited from these capacity strengthening efforts? c. What are the products or results of this capacity strengthening?) (PCDEAH, establishment of various structures, training, logistics support, etc.) d. Which of these support efforts do you judge to be most important in order to develop your implementation capacity or to assist you in achieving your objectives? e. What is the importance /the interest of each of the structures put in place to achieve the commune’s WASH objectives (in order of priority)? g. Sanitation and hygiene a. What support has been provided by RANO WASH to promote sanitation and hygiene in your commune? b. What, if any, were the challenges in implementing these activities? c. In your opinion, which of the approaches to increase demand and use of sanitation and hygiene services are most effective? (CLTS, VSLA, schools approach, sensitization, BCD approach, use of local radio, involvement of private operators) Why? What have been some of the challenges? d. What are the results that are visible at present? (sanitation market in place, ODF commune, etc.) e. Are there tangible results that are specific to your region or commune following the implementation of the behaviour change approaches adopted by the program? What are the results, and through which approaches? h. Water supply i. What has been the role of RANO WASH with regards to water supply policy/governance in your commune? j. Can you describe the steps followed from the start up to contracting of a private sector provider? (Describe any differences between rehabilitation of existing systems vs. construction and management of new infrastructure.) k. What is the role of the commune in the PPP? l. What, if any, have been the major challenges encountered? m. What is your opinion of the PPP approach used? n. Does the water supply infrastructure implemented through the program respond to the needs of the community? o. What could still be improved? What would be your suggestions? p. In your opinion, is the involvement of the private sector in the provision of WASH products and services adapted to the needs of the target population? Why or why not? How can this be improved? q. Can you talk about the sustainability of the use of and access to WASH products and services installed or accessed as a result of RANO WASH program activities at the level of target communities? How long have results been sustained so far? What is being done to ensure long term sustainability? By whom? r. In practice, what did the program do around gender and inclusion (of women, girls, and people in vulnerable situations) and how do you think this has affected the program outcomes? What could they have done differently? s. If you can change something in the way the program was designed and implemented, what would be your three main suggestions? t. What aspects should be fostered and developed in the way the program was designed and implemented? u. Specific question for regional/local Health and Education Government staff representatives: how has the project contributed in improving Education and Health sector processes, outcomes or targets? As appropriate, specific questions to emerge for each region based on the document review. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 79 Conclusion: ● Thank the respondent for their time ● Tell the respondent they are welcome to contact you to ask questions at a later date. Ask permission of the respondent to use their name in the report if you might use a quote. Note their response: ______________. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 80 Focus Group Discussion and Community Engagement Topic Guides In order to collect data during the field visits and from specific groups of actors, the evaluation team has developed topic guides for 4 different categories of actors. These topic guides can be used either to guide FGDs, or in individual conversations with local actors in these categories. 1. Private sector WASH service providers 2. Regional and local Civil Society Representatives 3. Women and girls 4. Community members, school teachers, health facility staff For the first three categories, targets of each discussion are described in more detail in the Discussion Guides below. Questions are considered guiding questions, and depending on the exact composition of the Focus Group or the background of the individual being interviewed, and the time available, the evaluation team may select the most appropriate questions. For the Community guide, questions are grouped in such a way that they can guide either collective community discussions or individual interviews / conversations with community members. For example, the guide can be used at the start of a community visit where various community members are gathered or for smaller community FGDs. The individual guide can be used during a community walk or while visiting households, schools or health centers. Focus Group Discussion Guide – Private Sector WASH Service Providers This Topic Guide is intended for use in terms of data collection from private sector WASH service providers, through FGDs at the communal level. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide FGDs with these respondents. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. Target: Private-sector water service providers or sanitation sector providers. Expected to be either one or the other, not mixed. I. Introduction (~10 minutes) a. Thank the respondents for taking the time to participate in the interview b. Introduction to the researchers and the research i. Introduce yourselves: We are consultants residing in Antananarivo. We represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. ii. We have been asked to assess the RANO WASH program’s progress toward achieving its objectives, and the strengths and weaknesses of the program, its accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 81 and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondents would like to remain anonymous, and that the assessment team will ask permission if it would like to quote respondents by name in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondents are willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. e. Check whether respondents have any questions. II. Questions Introduction question • Tell me about yourself/yourselves and the community/ies you serve. What kind of services do you provide, and how many households / clients / communities do you reach with your products or services? – question to be answered by each individual in the focus group. Engagement with RANO WASH • In your own words, can you describe the key elements of RANO WASH’s program for increasing private sector engagement in the WASH sector in Madagascar? • Which element was decisive for you to get involved in the partnership with RANO WASH? Does a similar thing exist in Madagascar besides RANO WASH? What blockage(s) has this program lifted? • What was/were your objectives when you started engaging in the project? Did you succeed in achieving this/these objective(s)? Water Service Providers • Do you think that the private sector is well positioned to provide rural water supply services in this region/ commune? Why or why not? • How do you feel about the build/construct/manage model and the relationship with the commune and with the clients? What is working well and what isn’t? Could the model be improved? How? • In the community/ies you serve, do all households receive the service? If not, why not? What can be done to improve service delivery to currently excluded households? • Do you see what you’ve started as something that will last and become a standard activity of your business? Or just a “one shot” activity? • Do you think the water service you are providing is sustainable in the long term? Why or why not, and what can be done to increase sustainability? • What are the factors constraining the private sector from further involvement and investment in rural water supply? Who should address these constraints? And how? • Would you have any advice for RANO WASH to improve the way they work? Sanitation Service Providers • Do you think that the private sector is well positioned to provide sanitation services in this region/ commune? Why or why not? • What is working well and what isn’t with regards to the sanitation products or services you are offering? Is your business model successful? Could it be improved? How? • What activities do you undertake to increase demand for your sanitation (and hygiene) related products and services? Do you feel you have been successful in increasing demand? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 82 • In the community/ies you serve, do all households demand your products or services? Can they all access them equally? If not, why not? What can be done to improve sanitation service delivery to currently excluded households? • Do you see what you’ve started as something that will last and become a standard activity of your business? Or just a “one shot” activity? • Do you think the sanitation services people currently have access to are sustainable in the long term? Why or why not, and what can be done to increase sustainability? By whom? (if need be, prod for roles of local government, private sector providers like themselves, RANO WASH program) • What are the factors constraining the private sector from further involvement and investment in rural sanitation? Who should address these constraints? And how? • Would you have any advice for RANO WASH to improve the way they work? Focus Group Discussion Guide – Regional and local Civil Society representatives This Topic Guide is intended for use in terms of data collection through regional or commune level FGDs with CSO representatives, regarding the role of civil society in strengthening local governance, accountability mechanisms, and shifting the power balance. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide FGDs with these respondents. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. Target Civil society representatives active at regional, communal or community level, such as those representing OSCEAH, women’s groups, youth groups, water user associations, etc. I. Introduction (~10 minutes) a. Thank the respondents for taking the time to participate in the interview b. Introduction to the researchers and the research i. Introduce yourselves: We are consultants residing in Antananarivo. We represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. ii. We have been asked to assess the RANO WASH program’s progress toward achieving its objectives, and the strengths and weaknesses of the program, its accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondents would like to remain anonymous, and that the assessment team will ask permission if it would like to quote respondents by name in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondents are willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 83 __________________________________________________________________________ e. Check whether respondents have any questions. II. Questions Introductory question • Tell us about yourselves. What organization or group of people do you represent? How long have you been active in/ have you been a member of this organization or group? Question to be answered by each individual in the focus group. Engagement in WASH and RANO WASH • What is/are your (organizations’) role(s) in ensuring access to water, sanitation and/or hygiene in your commune/community and how do you implement this role? (for example oversight, grievance mechanisms, demand creation, request of services...)? • Are you engaging with the RANO WASH program in this context, and if so, what is your relationship with the RANO WASH program? • What is the interest of civil society organizations to collaborate with the program? What benefits or support have civil society / your organization(s) received from the program? • Do you think that civil society is more involved in WASH issues as a result of the program? Do you think CSO involvement benefits the program? Why and how? • Can the mechanisms put in place and activities carried out by the program help the local government and the community to reach its local WASH goals? If not, what should be changed / what should the program do differently? Collaboration • How do you find the interactions between civil society organizations and public institutions (municipality), technical partners (like RANO WASH), and private sector WASH providers? Can you describe the successes and challenges for each of these relationships? • Why do civil society organizations in your municipality / region maintain relationships with each these institutions? For what purpose? (Request/advocate/grievance) Gender and inclusion • Do you feel the WASH activities undertaken in your municipality/ies have benefited everybody equally? Have they benefited women and girls, and vulnerable people such as people living with a disability, or people that are marginalized or excluded from community life? • What has been the role of civil society in supporting inclusive processes and results? What have been some of the challenges? What could have been done differently / better? Impact and sustainability: • What changes have you witnessed in terms of demand and use for WASH products and services as a result of RANO WASH? Do you think the program results are sustainable? Why or why not? • In your opinion, do the current structures and mechanisms put in place make it possible to sustain access to water supply, sanitation and hygiene even after the end or the exit of the RANO WASH program? If not, what else could be done to ensure sustainability? • When RANO WASH withdraws from your municipality, will you maintain your role / your activities? Why or why not? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 84 Focus Group Discussion Guide – Women and Girls This Topic Guide is intended for use in terms of data collection through commune level FGDs from women and girls, regarding the program’s activities on gender and inclusion. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide FGDs with these respondents. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. Target Regional/communal/community representatives of women's groups or associations. High school girls, women, the elderly, illiterate women or girls, women or girls and inhabitants of remote villages. I. Introduction (~10 minutes) a. Thank the respondents for taking the time to participate in the interview b. Introduction to the researchers and the research i. Introduce yourselves: We are consultants residing in Antananarivo. We represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. ii. We have been asked to assess the RANO WASH program’s progress toward achieving its objectives, and the strengths and weaknesses of the program, its accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondents would like to remain anonymous, and that the assessment team will ask permission if it would like to quote respondents by name in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondents are willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. e. Check whether respondents have any questions. II. Questions Introductory question • Tell me about yourselves. What do you know about the RANO WASH program? Conception/activities (questions to group or to the lead of women’s association) • How, if at all, have the activities implemented by the program impacted on your lives? And on the lives of other women, girls and people in vulnerable situations in your community? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 85 __________________________________________________________________________ • Do the WASH services provided in your community answer the specific needs of women and girls of different ages and with different backgrounds? If yes, which needs are met? And how? If not, which needs are not met? • How do you find the WASH situation in your schools and health facilities? Have you noticed any changes as a result of the program? • Were you involved in any of the activities to improve your WASH services? What did the program do to increase involvement of women and girls? • What do you like about the program, and what could be improved? Access to water services • Can you describe your access to water services? How easy is it to access water, how long does it take you to get water for the household? How long have you had these services? • Did you have any say in the type or location of water service your family now has? Who asked you? • Are certain people or households in the community excluded from using the services provided by the program/ the Water Service Provider, or do they have only limited access? If yes, who are these households, and why do you think they are excluded? Access to latrines and bathroom • Do you have a toilet at home or near your home? If not, where do you defecate? If so, what type of toilet do you use? Do you always use it? • Are any of you still defecating in the open? What do you think about this? • Has anybody come to talk to the community about ending open defecation? Who came, and can you describe the intervention? Were you part of it? How did you react? • Did the community make a plan to stop open defecation? What was the role of women and girls in this process? • Were you involved in the decision to build a toilet, and the type and location of the toilet? What were your contributions to the process? • How did your family pay for the toilet? Are you a member of a VSLA and if so, did you use any of that money for WASH services? • Are you satisfied with the toilet? What do you like about it? What do you not like about it? • What could have been done differently or better? Hygiene • What do you know about hand washing? Why do you wash your hands? At what occasions? • Where do you hear about the need to wash your hands? Or any other hygiene practices? • Where do you hear about the need to take care of your menstrual health and hygiene? • What, if any, has changed thanks to the RANO WASH project with regards to MHH? • Do you have all the facilities required to ensure handwashing with soap? And menstrual hygiene? (water, place, sanitary napkins….)? What could be improved? Empowerment of women • In which ways has the program helped women to develop their decision-making power, or improve daily and financial life? Could you give some examples to illustrate the fact? • What more could the program do in this respect? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 86 Community (FGD and household/individual KII guide) This Topic Guide is intended for use in terms of data collection from community members, through FGDs or individual conversations at the household, community or (if relevant) institution level. This evaluation will assess the relevance and effectiveness of RANO WASH’s technical assistance activities to strengthen WASH governance, increase and improve private sector engagement in WASH service delivery, accelerate the adoption of healthy behaviors and use of WASH services, and contribute to a more collaborative, harmonized sector in Madagascar. It should be followed as closely as possible to guide key informant interviews with these respondents. Instructions to the interviewer are in red. Interviewers will be trained to be compliant with USAID policy in regard to the “USG Common Rule” for the protection of human subjects. Target Mixed community-members of target communities, considering status, gender, age, people with reduced mobility. Including teachers / health center staff if present in the community. I. Introduction (~10 minutes) a. Thank the respondents for taking the time to participate in the interview b. Introduction to the researchers and the research i. Introduce yourselves: We are consultants residing in Antananarivo. We represent an Evaluation Team fielded by Tetra Tech ARD, a Washington DC based firm that has been contracted by the U.S. Agency for International Development (USAID) to conduct an independent evaluation of the USAID RANO WASH program. ii. We have been asked to assess the RANO WASH program’s progress toward achieving its objectives, and the strengths and weaknesses of the program, its accomplishments and best practices, but also any obstacles and shortcomings faced and how it could be more effective. iii. Ultimately the research will be used to ascertain the extent to which RANO WASH activities have improved lives through the development and management of sustainable water, sanitation, and hygiene services in Madagascar. It will also be used to propose recommendations based on the findings to inform future WASH programming by USAID. c. We will follow privacy protocols to protect your anonymity: i. Explain confidentiality and anonymity and note whether the respondents would like to remain anonymous, and that the assessment team will ask permission if it would like to quote respondents by name in the final report. ii. Explain how collected data will be stored without identifying information. iii. Ask if the respondents are willing to be recorded and note their response. d. Explain recording, length, and nature of discussion. e. Check whether respondents have any questions. II. Questions: For community members (in collective discussion or individual) • Tell me about yourselves and the community Managing the Water Supply (FGD) • How does the community manage the water facility? • Are you aware of a group/individual who is responsible for managing the water system? Do you feel they are managing it well? If not, why not? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 87 • When was the last time your water facility broke down and how long did it take to be repaired? Who repaired it and where did the spare parts come from? For communities benefiting from a large or small water system / PPP arrangement • How much did the community pay for the construction / rehabilitation of the water facility and what was the source of the money? • Did any of you benefit from the subsidy for the first 100 households to connect to the water system? How was it decided who would benefit? Did you think that was fair? What could be improved in that process? • Who of you have branchements particuliers et who have branchements sociaux? Which ones do you prefer? Why? • How does the community pay for tariffs? Does the community keep collected tariffs in a bank account or how else is tariff collection organized? • Has there been special attention paid to poorer or more vulnerable members of the community to ensure they have access to water supply facilities? How were these families identified? By whom? What role did the community play in supporting the poor and vulnerable to have access to the water supply? • Does your family manage to pay the water bills? How much are you paying? For communities not yet benefiting from a water system / PPP arrangement • What kind of water system would you prefer to have access to? • Would you be willing to pay for a water service? To be connected? Regular fees? How much would you be willing and able to pay for such a service? Sanitation and Toilet Provision • Does the community have toilet facilities in/by the house? Does everyone have a toilet? If no, where do people defecate? If yes, what types of facilities do households have? • Does anybody in this community defecate in the open? What does the community think of this? Does the community undertake actions to stop open defecation? If so, what kinds of actions? Who leads? • For declared/certified ODF villages: – Is this community ODF? Since how long? Can you describe some of the differences you have noticed since everybody started using a toilet? • How much did people pay for these latrines? How long did it take to pay for these latrines or undertake installation (if they built it themselves)? How did they raise funds for latrine construction or purchase of products? • Did most people build by themselves or did you use artisans? How easy is it go get an artisan, how affordable are their charges? • If you built by yourself, did any of you buy any parts for latrine installation and if so, where did you buy them? What kind of latrine products are available? • Any follow up from RANO WASH or the commune since you installed? • How long do you hope your latrines will last? What will you do if there is a problem, like your latrine collapses or becomes full? Has anybody from RANO WASH or the commune spoken to you about this? • Have you seen a shift in attitudes towards investing in sanitation facilities in the home? If funds are available, do community members prioritize sanitation as compared to other products? Is it affordable for poorer households to invest in a toilet? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 88 • Who decides on where and how latrines are constructed? Do women and girls have a say? Elderly / less able-bodied household members? • Has there been special attention paid to poorer or more vulnerable members of the community to ensure they have access to sanitation facilities? What role did the community play in supporting the poor and vulnerable to have a toilet? Is there a process to identify families that may require extra support to access, build or maintain sanitation facilities? Hygiene (household level questions) • What do you know about hand washing? Why do you wash your hands? At what occasions? • Besides water supply and toilets, has anybody of the programme spoken to you about hygiene? How did that go, what did you discuss? (if Grow-Up sticker household, discuss how they like the approach • How often do you wash your hands? • Have you seen a shift in attitudes about hand washing and hygiene? • Who is supporting community members to maintain the practice of hand washing? Looking to the Future • What will the community do after the project has ended and there are new people who need support? • Apart from providing further financing, what advice would you have for RANO WASH about how to strengthen these projects and make sure that the results are sustainable? For Natural Leaders • What has been your role in the RANO WASH project? • What type of support have you received from RANO WASH project to be a natural leader? • How have you created demand in the community? • How have you facilitated access to sanitation products for community members? • What has been the level of acceptability for the sanitation products? Which products or options are mostly used by the households? • What kinds of support do the household members need and how do you provide support? • Beyond the projects, how will you take forward your role as natural leader (explore if there are any project-based incentives that might not be available after the project) • How do you think the sanitation facilities and behaviours in your community can be sustained? What is needed for that, and who should be involved? For Community Health volunteers • What has been your role in the RANO WASH project? • What type of support have you received from RANO WASH project to be a VHC? • Are there other related interventions in the community from other Development Partners? How do these differ from what RANO WASH is trying to do? • How have you changed behaviors in the community? What kind of behaviors? What has been the result? Can you give examples? • Have you seen changes in the health status of the community? What kind of changes? • How have you facilitated access to sanitation or hygiene products for community members? What kind of products? • What has been the level of acceptability for the sanitation or hygiene products? Which products or options are mostly used by the households? • What kinds of support do the household members need and how do you provide support? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 89 • Beyond the projects, how will you take forward your role as CHV? • How do you think the water supply and sanitation facilities and hygiene behaviors in your community can be sustained? What is needed for that, and who should be involved? For School headmasters / teachers / school health club members • Does the school have water supply and handwashing facilities? Toilets? Mixed, or separated for boys and girls? Are there toilets for the teachers? • How many toilets are there, for how many students? Are there toilets that are adapted to the needs of students/teachers living with disabilities? • Do boys and girls have equal access to WASH facilities in the school? Are there changing rooms for girls with full view mirror? Bins? Washing facilities? • How long have these facilities been in place? Were they constructed / rehabilitated with support from the RANO WASH project? • Tell me about the RANO WASH interventions being implemented: How was your school selected to benefit from the project? When were facilities provided? What kinds of behavior change activities have been put in place? • Are there other related interventions in the school from other Development Partners? How do these differ from what RANO WASH is trying to do? • What does the school do to educate the pupils of different ages on healthy hygiene behaviors? Which behaviors? School health clubs? Hygiene educators? Other initiatives? • Who maintains the water and sanitation facilities? Who cleans? Who decides and how is it financed? • What will happen when there are breakdowns or major repairs needed? Who will finance this? Is there support from the commune / Ministry of Education/PTA/other sources? • How many times have your water facilities broken down and how long did it take to be repaired? • Who repaired it and did the school find it difficult to have a pump mechanic to repair it? • How much did the school pay? • Was the school trained to undertake preventive maintenance and does the school have the tools to enable them do t hat? • Do you have a Facility Management Plan (MP) and to what extent are you following the FMP? • What role does the PTA/SMC play in the school’s FMPs? • Where/how do you get soap for hand washing services? Is there a budget allocated for soap? • Have you seen a sustained shift in behaviors (social norms) around hand washing and hygiene? What factors would continue to support this sustained change in behavior? • Are there particular challenges with regard to sustaining the gains made by the RANO WASH activity? Please suggest how the initiative can be sustained. • [Apart from financing] Do you have any advice for RANO WASH moving forward? How can your collaboration with the project be further improved? For Health Care Facilities (if present in the community) • Tell me about your health care facility –How many patients do you attend to in a day? (Male/female) • How many staff do you have on roll? What are the three topmost diseases you treat? • Tell me about the RANO WASH interventions being implemented / How was your HCF selected to benefit from the project? When were facilities provided? What kinds of behavior change activities have been put in place? • Are there other related interventions in the HCF from the Ministry of Health or other Development Partners? How do these differ from what RANO WASH is trying to do? • Is there equal access for WASH facilities in the HCF for both sexes including children? • How do you maintain your latrines and water points? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 90 • How do you finance preventive maintenance and major repairs? Is there a budget allocated in the HCF for O&M of WASH facilities? • How many times have your water facilities broken down and how long did it take to be repaired? Who repaired it and was it difficult to find a pump mechanic to repair it? How much did the HCF pay? • Was the HCF trained to undertake preventive maintenance and does the HCF have the tools to enable them do that? • Do you have a Facility Management Plan (MP) and to what extent are you following the FMP? • How are sanitation and hygiene practices promoted for patients/clients? Please give examples. • Where/how do you get soap for hand washing services? Is there a budget allocated for soap? • Have you seen a sustained shift in behaviors (social norms) around hand washing and hygiene? What factors would continue to support this sustained change in behavior? • Are there particular challenges with regard to sustaining the gains made by the RANO WASH activity? Please suggest how the initiative can be sustained • [Apart from financing] Do you have any advice for RANO WASH moving forward? How can your collaboration with the project be further improved? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 91 FIELD OBSERVATION GUIDE AND TEMPLATE – COMMUNITY ENVIRONMENT AND WASH FACILITIES This observation guide and template is intended to assist the evaluation team in undertaking quick but systematic spot checks of selected facilities during their community visits. It is expected that in each community the Evaluation Team will observe a key water point. The team will inspect a minimum of three household or collective toilet and handwashing facilities and enter data in the forms. The template incudes questions regarding general type and quality of the facilities, questions related to gender and inclusion, and questions related to safely managed sanitation. Region District Commune Fokontany Village General environment QUESTION ANSWER OBSERVATIONS Does the general environment of the village look clean? (solid waste management, roaming animals, etc.) Yes/No Are there signs of open defecation / human faeces lying around the village? Yes/No Water Supply system (household level) QUESTION ANSWER / OBSERVATIONS Is there a community water system, or individual water points? General description of water system/point(s), including materials, technology used What type of connection (branchement) does the visited household have? – Particulier – Social Is the waterpoint working? Are there signs of regular use? Are the waterpoints and surroundings clean? Are the waterpoints easily accessible? At night? For people with reduced mobility? (see accessibility checklist) MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 92 Sanitation facilities (household level) QUESTION ANSWER / OBSERVATIONS General description of latrine type: Dry toilet with a single pit Describe materials the toilet is made of, including material of the slab – Dry toilet with a double pit – (poor)flush toilet with a single pit – (poor)flush toilet with twin pits – (poor)flush toilet with septic tank or soak pit – Urine diverting toilet with cartridge or storage tank – Flush toilet connection to simplified or conventional sewerage system Is the toilet accessible for all intended users? e.g. clear and secure access path, added features for those with special needs/reduced mobility such as access ramp, handrail, etc. Is the toilet superstructure in good condition, complete, damaged and/or does not provide privacy and security to the intended users? Does it have a do or and a roof? Is the slab whole and does it properly close off the pit(no cracks or holes)? Is there a squat-hole cover? Is the toilet clean or dirty ? e.g. visible excreta on surfaces Is there evidence of solid waste being deposited in the pit? Are there excreta overflowing from the squat hole, pan or pedestal; and/or are there ponds of effluent visible on the ground outside the toilet? Is the toilet and pit located within 15 meters of a well or handpump that is used for drinking? If yes, roughly how close to the waterpoint? Is the pit/septic tank located on higher ground from the drinking water source? Is the container/pit/septic tank accessible for emptying? Has the pit been emptied before? Ask latrine owner If yes, where were the contents taken? How were they disposed of? Were they treated? How? Who emptied and disposed of the pit contents? Was this a paid service? Ask latrine owner Source: Adapted from WHO Sanitation Inspection Form, draft May 2019 MID-TERM PERFORMANCE EVALUATION OF RANO WASH : FINAL REPORT 93 Handwashing facilities (household level) QUESTION ANSWER/ OBSERVATIONS Are handwashing facilities visible inside or next to the toilet? (Handwashing facilities consist of the presence of water and soap. They may be fixed or mobile and include a sink with tap water, buckets with taps, tippy-taps, and jugs or basins designated for handwashing. Soap includes bar soap, liquid soap, power detergent, and soapy water. ) Are there signs of frequent use of the handwashing facility? e.g. presence of water and soap / well-worn path to the facility, drainage /vegetation underneath Menstrual hygiene facilities (household or school/health center level) QUESTION ANSWER/ OBSERVATIONS Are there bins inside the toilet for disposal or cleaning of menstrual hygiene materials? Is there water to for women and girls to clean the body and/or reusable menstrual pads? Is there a private space to wash and dry reusable pads? Communal or institutional sanitation facilities (school, health center) QUESTION ANSWER / OBSERVATIONS General description of latrine type: – Dry toilet with a single pit – Dry toilet with a double pit – (poor)flush toilet with a single pit – (poor)flush toilet with twin pits – (poor)flush toilet with septic tank or soak pit – Urine diverting toilet with cartridge or storage tank – Flush toilet connection to simplified or conventional sewerage system Describe materials the toilet is made of, including material of the slabs. How many toilet cubicles are there for how many users? Are there separate facilities for men and women / boys and girls? How many? Are the toilets accessible for all intended users? MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 94 QUESTION ANSWER / OBSERVATIONS e.g. clear and secure access path, added features for those with special needs/reduced mobility such as access ramp, handrail, etc. Does the toilet superstructure provide sufficient privacy and safety for the users? (e.g. lockable doors, solid roof structure, working lights) Is the toilet block well maintained and in good condition? e.g. condition of slabs, walls, roof structures, pits. Are the toilets clean or dirty? e.g. visible excreta on surfaces Are there excreta overflowing from the squat hole, pan or pedestal; and/or are there ponds of effluent visible on the ground outside the toilet? Is the toilet and pit located within 15 meters of a well or handpump that is used for drinking? Is the pit/septic tank located on higher ground from the drinking water source? Is the container/pit/septic tank accessible for emptying? Have the pits been emptied before? If yes, where were the contents taken? How were they disposed of? Were they treated? How? Ask latrine owner Who emptied and disposed of the pit contents? Was this a paid service? Ask latrine owner Are handwashing facilities visible inside or next to the toilets? (Handwashing facilities consist of the presence of water and soap. They may be fixed or mobile and include a sink with tap water, buckets with taps, tippy-taps, and jugs or basins designated for handwashing. Soap includes bar soap, liquid soap, power detergent, and soapy water.) Is there evidence of frequent use of the handwashing facilities? e.g. ready presence of water and soap / well￾worn path to the facility, drainage /vegetation underneath Source: Adapted from WHO Sanitation Inspection Form, draft May 2019 MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 95 WATER POINT ACCESSIBILITY CHECKLIST Distance: How far is it from home(s)/classroom to water point? Access route: • Is the route outside or inside? • If used at night, is the path lit? • What is the path/ access route made of? • Is the path wide enough for all users? (recommended minimum width 90cm) • Is the path level and firm, with nothing to trip up? Is the surface of the path slippery when either dry or wet? Are there obstacles that make it difficult to get past, or easy to trip up, especially for visually impaired people? (e.g., vegetation, rubbish, up to 2m above floor level). • Are there any parts of the path which make women or children feel unsafe when using it? If so, why? • Are there landmarks that a blind/visually impaired person can follow, e.g., clear surface texture, landmarks or guide rail? • If there is a slope or ramp, how steep is it? (Recommended maximum 1 in 10) Is the surface of the slope slippery or non-slip? From: Violence, Gender & WASH: A Practitioner's Toolkit – Making water, sanitation and hygiene safer through improved programming and services. House S. et al (2014) London, UK: WaterAid/SHARE. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 96 ANNEX IV: DATA SOURCES LIST OF KEY INFORMANTS – NATIONAL AND REGIONAL INSTITUTION POSITION NAME National government stakeholders MoWASH The Minister Voahary Rakotovelomanantsoa Secretary General Simon Randriantsiferana Director of information systems, monitoring and evaluation Andrinirina Razafindrakoto Director of Sanitation and Hygiene Voahirana Rasoamanantena Director of Research and Development Fanantenana Rakotonirina Director of Water Provision Tsirimiarantsoa Andriamanga Director of Communication and Partnership Julie Resampa Director of Legal Affairs Rado Lalaina Andrianjanaka Collaborating programs/consortium partners WaterAid Program manager RANO WASH Heritiana CRS Country Representative Carla Fajardo CRS Deputy Head of Programs Lanto Rafanomezantsoa CRS Head of Programs Tanja Englberger BushProof General Manager Serge Ranaivojaona Sandandrano General Manager Gerald Razafinjato ODDIT Implementation Coordinator Santoné Agesilas Ny Tanintsika National Coordinator Eugenie Raharisoa SAF FJKM Coordinator Dr Fanja Andrianjatovo Other development partners UNICEF Madagascar WASH chief Brigitte Pedro HELVETAS WASH Coordinator Davy Robson MAHEFA Chief of Party Dr Yvette USAID Shops Plus Project Chief of Party Sarindra Ramanitrivonony MCDI FAA program manager Rija Fanomeza RAN’EAU Executive Director Patrick Rasolofo GRET Deputy representative Mamisoa Andriamihaja OSC EAH OSC EAH National Coordinator Arsène Raveloson RANO WASH program staff Program Coordination Team Chief of Party Sebastien Fesneau Deputy Chief of Party Avo Ratoarijaona Director of Monitoring and Evaluation Berenger Tchatchou Gender and Social Inclusion Advisor Hanitra Njatonirina Senior WASH Governance Advisor Rodolphe Rakoto-Harisoa Senior Private Sector Advisor Lalaina Razanajatovo Construction and Environmental Compliance Specialist Tianazo Rakotoarisoa Marketing and Enterprise Specialist Ntsoa Ranaivoson Senior WASH Behavior Change Advisor Harisoa Rasamoelina RANO WASH regional staff Regional Coordinator Atsinanana Solofomalala Ravelomanantsoa Regional Coordinator Alaotra Mangoro Andry Tianarivelo Regional Coordinator Vatovavy Fitovinany Lahatra Randriamifidy Private sector service providers MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 97 INSTITUTION POSITION NAME ACCES Bank Head of Customer Acquisition Department Lucia Masy BNI Madagascar Deputy CEO Adrian Chindris Lova Velu Rado Rasoanaivo 2ADH Simon Rakotonirina APR Alain Pierre Regional government staff DREAH Atsinanana VOAVY Jackson Willy DREAH Alaotra Mangoro RAKOTOARINOSY Damascène Hajandrianony DREAH Amoron’i Mania RAKOTOVAO AndrianjafimaharavoTantelisoa MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 98 PRIMARY DOCUMENTS CONSULTED IN DESK REVIEW RANO WASH Program Documents ● RANO WASH Technical Proposal ● RANO WASH Theory of Change ● RANO WASH MEAL Plan – revised May 2019 ● Annual work plan 2017 ● Annual work plan 2018 ● Annual work plan 2019 ● Annual work plan 2020 + Annexes ● RANO WASH revised PMP – March 2019 ● Approved Water Quality Assessment Report ● Data Quality Assessment Report - FY19 ● Rapid Gender Analysis for COVID-19 in Madagascar ● RANO WASH Joint Site Visit Report May 2019 ● RANO WASH Mid-Term Review Report RANO WASH Program Reports • FY2017 Annual Report • FY2018 Q1 Report • FY2018 Q2 Report • FY2018 Q3 Report • FY2018 Annual Report • FY2019 Q1 Report + Annexes • FY2019 Q2 Report + Annexes • FY2019 Q3 Report + Annexes • FY2019 Annual report + Annexes • FY2020 Q1 Report + Annexes • FY2020 Q2 Report + Annexes • FY2020 Q3 Report + Annexes • FY2020 Annual Report + Annexes • FY2021 Q1Report + Annexes • FY2021 Q2 Report Narrative Madagascar Sector Reports ● Madagaskira Madio 2025 ● National Policy of Water, Sanitation and Hygiene (July 2020) ● WASH Bottleneck Analysis Tool Report 2014 ● WASH Bottleneck Analysis Tool Report 2018 ● TRACKFIN Initiative Final Report, May 2018 MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 99 ANNEX V: EVALUATOR BIOGRAPHIES AND CONFLICT OF INTEREST FORMS Team Leader: Ms. Carolien van der Voorden Ms. van der Voorden is a senior WASH professional with 20 years of experience and demonstrated skills in organizational leadership, strategy development, designing and providing technical support to programs combining behavior change/CLTS and supply/market-based approaches; she has worked on public-private partnerships in WASH, and has done substantial work on sanitation financing and WASH systems strengthening. She is currently working as an independent consultant on WASH, health and development. Most recently, as Head of the Water Supply and Sanitation Collaborative Council (WSSCC) Technical Support Unit and prior to that, as Senior Program Officer in WSSCC and the Global Sanitation Fund (GSF), Ms. van der Voorden conceptualized, led and oversaw a wide range of studies, assessments and learning exercises, particularly on themes related to programming at scale, sustainability and equality. She was involved in designing and reviewing program evaluations in Togo and Benin; led a 2019 review of WSSCC’s country engagement modalities; and managed a large inter-country Equality and Non￾Discrimination (EQND) scoping and diagnosis study of GSF programs in 2015/16. More recently she managed a GSF-wide study on safely managed sanitation services in a rural context, designed to inform the newly established Sanitation and Hygiene Fund, for which she co-authored the strategy and multiple policies, and wrote the programming guidelines. In Madagascar, Ms. van der Voorden supported the WSSCC National Coordinator and Diorano-WASH coalition partners from the launch of the coalition in 2002 throughout the Diorano-WASH life-course, amidst the institutional and political turbulence of Madagascar’s governance system. She helped design and develop the GSF-funded Fonds d’Appui pour l’Assainissement (FAA) program, provided technical support throughout its 2010-2020 life-course, and organized a 2016 Global Learning Event reflecting on the Madagascar experience with global sector partners and GSF implementers from across Africa and Asia. Her responsibility on FAA was to critically reflect and facilitate important learning and adaptive management processes, among others facilitating and co-authoring the 2016 WSSCC report Sanitation and Hygiene Behaviour Change at Scale: Understanding Slippage based almost entirely on the Madagascar experience, as well as reviewing the programmatic choices made, lessons learned and adaptations implemented around local governance, institutional triggering, and programming for sustainability. She holds a Master’s Degree in Development Studies and an undergraduate degree in Cultural Anthropology. WASH Governance Specialist: Mr. Ridjanirainy Randrianarisoa An experienced social development policy maker and program manager, Mr. Randrianarisoa’s professional experience spans several programmatic areas including WASH, Education, and ICT. From 2011 to 2018, he served as Senior Manager for Advocacy and Research for WaterAid, where he was involved in numerous activities in support of WASH sector planning and policy development objectives. He supported the country program identify and undertake research and learning activities that helped deliver more sustainable, innovative, equitable and cost-effective solutions for the targeted communities, including research activities on financial sustainability of WASH services. Mr. Randrianarisoa was involved in the formulation of the Madagascar WASH Sector Strategy and sector level long term financial planning; he also provided technical expertise to the Ministry of Water in preparing the WASH Sector Roadmap 2015- 2019. More recently, he was involved in the formulation of the Madagascar WASH Sector detailed planning document based on SWA principles, and in 2020, he undertook a sector review to inform a possible new national sanitation program, and is therefore well-aware of the most recent sector dynamics, plans and programs. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 100 In addition to working with national and international not-for profit organizations, his experience includes work with national government across multiple ministries, academia, as well as the private sector. Specialized in strategic planning, policy formulation, program design, implementation and evaluation and research design, he has worked with partners such as USAID, UNICEF, and French Aid. While having served as one of the co-authors of the original RANO-WASH program proposal while at WaterAid, in recent years he has taken a number of positions outside of the WASH sector thereby creating some distance, and his prior engagement with and knowledge of the program is seen as a benefit rather than as a potential conflict of interest. Mr. Randrianarisoa has a Bachelor of Science and a Master’s degree in Applied Computer Science and Statistics. WASH Specialist: Mr. Jean Eugène Injerona A development programming specialist with extensive experience in WASH, health, and communications, Mr. Injerona has a Doctorate diploma in Health and an MBA, and has worked with a number of international organizations in Madagascar, including over 10 years with WaterAid in various positions and Sustainability and Management Expert with UNICEF. At WaterAid, Mr. Injerona was involved in numerous WASH advocacy and implementation activities including in areas such as local governance, WASH as a business opportunity, and sanitation marketing. In 2016, he served on the evaluation team for the RANO WASH predecessor project and has conducted numerous internal project evaluations that have included development and application of survey instruments. While at UNICEF, he worked closely with private water operators on issues of capacity building, financial monitoring, and sustainability. He has worked with private operators, local authorities, and civil society, to strengthen multi-stakeholder partnerships, engagement, and service delivery models, and strengthen private sector engagement. Mr. Injerona is well aware of the challenges and barriers faced by private operators and small entrepreneurs in the WASH sector. Most recently he has served as Head of Project for Studio Sifaka, a communications outlet and broadcaster focusing on youth and development, his experience includes acting as Country Focal Point for GAVI, working as Senior Program Manager on a USAID-funded Maternal and Child Health Survival program. WASH Monitoring and Evaluation specialist: Mr. Nirina Rado Randrianavoson A Malagasy geographer and M&E specialist with extensive experience both as head of M&E for large programs/international NGOs and as consultant leading various evaluations, studies, and data collection exercises. From 2013 to 2020 he served as Head of M&E for the Global Sanitation Fund-supported FAA program, where he developed extensive expertise in CLTS and behavior change programming, monitoring and measurement. Concurrently with his role with FAA, he undertook several consulting assignments for clients such as USAID (situational analysis for ACCESS program), Handicap International (KAP survey) and Catholic Relief Services (end evaluation of RANON’ALA program). Mr. Randrianavoson was trained on CLTS facilitation, Equality and Non-Discrimination, and has served as a trainer and mentor in international program exchanges. With extensive experience in study and survey design, tool development, data collection and reporting combined with his knowledge of WASH programming, Mr. Randrianavoson functioned as the M&E specialist on the team, including development of questionnaires and tools for field-based data collection, and quality assurance. Monitoring and Evaluation Specialist: Ms. Ny Aina Ratsimba (May-September 2021) A Malagasy monitoring and evaluation expert with broad experience in undertaking evaluation assignments in a range of social sectors. Evaluation Specialist with a focus on social and gender and inclusion related issues. With a master’s in business finance and administration, her professional experience saw her focus initially on education, with positions at the Ministry of Education and the World Bank, where she worked as M&E Officer from 2006 to 2015, monitoring projects implemented MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 101 by the Technical Support Unit of the World Bank financed Education for All program. She undertook social audits of education projects across Madagascar, from where she grew her expertise in undertaking social audits for a range of private sector companies with supply chains in Madagascar, now as an independent consultant. These audits touched strongly on a range of gender, inclusion and human rights issues. A versatile M&E specialist, she brings particular experience in undertaking field missions and conducting KIIs and FGDs with a range of national and local stakeholders, including women’s groups and civil society platforms. She also has strong analytical and report writing experience. WASH Programming Specialist: Mr. Lalaina Andrianamelasoa (April-May 2021) A Malagasy WASH expert with experience in WASH program delivery as well as sector governance, Mr. Andrianamelasoa also has international experience as a WASH consultant through UNICEF placements in Chad and Mali. He has 14 years’ experience working in progressively senior roles, including a solid footing in WASH programming. Starting off working with an engineering consulting firm, he later joined WaterAid where he worked as a project and program officer, gaining important first-hand experience in managing and implementing regional WASH projects, including acting as gender and equity & inclusion focal point; establishing local networks of technicians and spare parts selling points, and ensuring data collection, technical monitoring, and evaluation of project outcomes. At UNICEF, he worked as WASH Field Coordinator and WASH in Institutions Officer for three years, increasingly supporting and working with government partners, as well as further elaborating his M&E skills and experience, including coordinating a Sustainability Check. After gaining some first international experience with UNICEF Mali, he returned to Madagascar to serve as Secretary General of the Ministry of Water, Sanitation and Hygiene for 10 months, until the Ministry was once more absorbed into the Ministry of Energy, Water and Hydrocarbons, where he then served as the Principal WASH Technical Adviser, until again joining UNICEF for a field position in Chad. During his time at the Ministry/ies, he among others led the conception and elaboration of the Madagascar ODF roadmap (Madagasikara Madio 2025), served as SWA focal point, led WASH-Bat and GLAAS data￾collection processes, as well as administering a range of national programs and supervising WASH infrastructure building. Mr. Andrianamelasoa brings first-hand WASH program implementation experience, including CTLS, water service provision, and institutional WASH. He combines this with solid experience in strengthening governance and systems, and local, regional and national government capacity building. His M&E experience includes development of data collection tools and conducting key informant interviews, coordinating sample surveys, leading ongoing program monitoring, and facilitating consultations and participatory reviews. Mr. Andrianamelasoa is a hydraulic engineer with a Master’s degree in Environmental Analysis and Policy. MID-TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 102 MID-TERM PERFORMANCE EVALUATION OF RANO WASH : FINAL REPORT 103 MID-TERM PERFORMANCE EVALUATION OF RANO WASH : FINAL REPORT 104 MID-TERM PERFORMANCE EVALUATION OF RANO WASH : FINAL REPORT 105 MID-TERM PERFORMANCE EVALUATION OF RANO WASH : FINAL REPORT 106 MID-TERM PERFORMANCE EVALUATION OF RANO WASH : FINAL REPORT 107 MID-TERM PERFORMANCE EVALUATION OF RANO WASH : FINAL REPORT 108 MID-TERM PERFORMANCE EVALUATION OF RANO WASH : FINAL REPORT 109 MID TERM PERFORMANCE EVALUATION OF RANO WASH: FINAL REPORT 110 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 - www.usaid.gov