PERFORMANCE EVALUATION OF THE USAID/ETHIOPIA TRANSFORM WATER, SANITATION, AND HYGIENE ACTIVITY Final Report JANUARY 2023 This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The author’s views expressed in this report do not necessarily reflect the views of USAID or the United States Government. ACKNOWLEDGEMENTS The performance evaluation was conducted by Ken Caplan (Team Leader, Independent), Desta Dimtse Assefa (Water, Sanitation, and Hygiene [WASH] Specialist), Mohammed Ebrahim Mussa (WASH Governance Specialist), Meskerem Mulatu Legesse (Gender Equality and Female Empowerment Specialist), Addis Meleskachew (Market-Based Sanitation Specialist), Yared Legesse (Monitoring and Evaluation Specialist), and Benyam Shebru (Team Coordinator). Each team member made meaningful contributions to the evaluation and this report. The Evaluation Team would like to acknowledge the invaluable contributions of Caleb Cord (WASHPaLS #2) for support throughout the evaluation particularly with the design and analysis of the household survey; Erica Goldberg (WASHPaLS #2) for support with contracting, logistics, document preparation, and related tasks; Lucia Henry and Carolien van der Voorden (WASHPaLS #2) for careful review of the draft report; Subhash Chennuri (FSG) for constructive insights on the Transform WASH business models; and Morris Israel (WASHPaLS #2) for support throughout and peer review of the final report. A special thank you to PRIN Consulting who undertook a significant household survey with professionalism and care. We extend a sincere thank you to Transform WASH staff for their close collaboration and cooperation in the planning and execution of the evaluation as well as their important insights. The team would also like to thank the numerous key informants at the national and sub-national levels, who participated openly and thoughtfully in the interviews, and USAID/Ethiopia for providing oversight and guidance throughout and helpful comments on this report in draft versions. Preferred citation: USAID. (2023). Performance Evaluation of the USAID/Ethiopia Transform Water, Sanitation, and Hygiene (WASH) Activity. Washington, DC, USAID Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) #2 Activity. Cover photo credit: Retailer in Dodola Woreda, Oromia Region (Photo taken by Desta Dimtse Assefa) Prepared for the United States Agency for International Development by the Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS #2) project, contract number 7200AA21C00079. Tetra Tech Contacts: Morris Israel, Chief of Party morris.israel@tetratech.com Carolien van der Voorden, Deputy Chief of Party c.vandervoorden@tetratech.com Lucia Henry, Project Manager lucia.henry@tetratech.com Tetra Tech 1320 N. Courthouse Road, Suite 600, Arlington VA 22201 Tel: 703-387-2100 Fax: 703-414-5593 www.globalwaters.org/washpals-2; www.tetratech.com/intdev FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | i ABSTRACT This report provides the findings from a performance evaluation of the Transform WASH Activity to understand key contributions to advancing market-based sanitation (MBS) approaches in Ethiopia. The performance evaluation sought to understand the effectiveness of demand creation activities, approaches to advancing MBS, Transform WASH contributions to good governance, and how gender equality and female empowerment were incorporated into the design and implementation of the activity. A household survey brought quantitative data to inform the qualitative assessment. The evaluation findings confirmed that the USAID-funded activity has made much appreciated and significant contributions to advancing MBS as a bona fide approach in Ethiopia. Through advocacy and convening efforts, a close working relationship with government, and shared constructive research and analysis, Transform WASH has shifted skeptical attitudes toward MBS and helped shape MBS-related Government of Ethiopia policies and approaches. Through learning by doing, Transform WASH has provided a practical demonstration of the National MBS Implementation Guidelines. In particular, the activity has facilitated the manufacturing and distribution of a wide range of sanitation products that are appreciated by consumers and created greater linkages across the service delivery supply chain. It has enhanced mechanisms to increase access to sanitation products and services including for low-income households. Through Transform WASH, business partners have developed knowledge and skills to expand their customer base, diversify income sources, provide job opportunities for others in the community, and develop into profitable enterprises. The activity has demonstrated that women can be successful in this conventionally male-dominated construction sector. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | ii TABLE OF CONTENTS ABSTRACT ............................................................................................................... i TABLE OF CONTENTS.........................................................................................ii ACRONYMS............................................................................................................iii EXECUTIVE SUMMARY ........................................................................................ v 1.0 EVALUATION PURPOSE AND AUDIENCE............................................. 1 1.1 INTRODUCTION 1 1.2 PERFORMANCE EVALUATION PURPOSE AND INTENDED USES 1 1.3 AUDIENCE 1 1.4 BRIEF EXPLANATION OF EVALUATION METHODOLOGY 2 1.5 LIMITATIONS OF THE METHODOLOGY 3 1.6 STRUCTURE OF THE REPORT 3 2.0 PUTTING TRANSFORM WASH INTO CONTEXT................................. 4 2.1 SANITATION CONTEXT IN ETHIOPIA 4 2.2 GOE SANITATION POLICIES AND STRUCTURES 4 2.3 MBS AS A TOOL AND THE USAID/WASHPALS MBS FRAMEWORK 5 2.4 THE TRANSFORM WASH ACTIVITY’S RESPONSE 6 3.0 EVALUATION QUESTIONS: KEY FINDINGS........................................ 10 3.1 EVALUATION QUESTIONS 10 3.2 EQ KEY FINDINGS 10 3.2.1 EQ1: Effectiveness of Transform WASH Demand Creation Approaches.............................. 10 3.2.2 EQ2: Market-Based Sanitation Results............................................................................................. 22 3.2.3 EQ3: Effectiveness of Transform WASH Governance and Learning Activities ..................... 29 3.2.4 EQ4: Gender Equality and Female Empowerment........................................................................ 35 3.3 CONSORTIUM EFFECTIVENESS AS A CROSS-CUTTING ASPECT 41 3.4 OBSERVATIONS ON SPILLOVER EFFECT 42 4.0 CONCLUSIONS AND RECOMMENDATIONS ...................................... 46 4.1 RECOMMENDATIONS 46 4.1.1 Recommendations for Transform WASH and/or Future Programming.................................. 46 4.1.2 Recommendations for USAID............................................................................................................ 48 5.0 PRIMARY REFERENCES ............................................................................. 50 ANNEXES submitted under separate cover ANNEX A: USAID EVALUATION STATEMENT OF WORK (SOW) ANNEX B: EVALUATION METHODOLOGY ANNEX C: DATA COLLECTION INSTRUMENTS ANNEX D: HOUSEHOLD SURVEY FINDINGS ANNEX E: LIST OF KIIS AND FGDS ANNEX F: FULL LIST OF TRANSFORM WASH WOREDAS AND SELECTION OF WOREDAS FOR PE TEAM VISIT FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | iii ACRONYMS CAP Consolidated WASH Annual Plan CLTSH Community-Led Total Sanitation and Hygiene COVID-19 Coronavirus Disease 2019 CWA Consolidated WASH Account EQ Evaluation Question ETB Ethiopian Birr FGD Focus Group Discussion FMOA Federal Ministry of Agriculture FMOH Federal Ministry of Health FY Fiscal Year GEFE Gender Equality and Female Empowerment GESI Gender Equality and Social Inclusion GOE Government of Ethiopia GtN Growth through Nutrition HCF Healthcare Facility HEW Health Extension Worker HH Household HHS Household Survey IRC International Water and Sanitation Centre ISI Institutional Strengthening Index JMP Joint Monitoring Program JSS Joint Supportive Supervision KII Key Informant Interview M&E Monitoring and Evaluation MBS Market-Based Sanitation MFI Microfinance Institution MHH Menstrual Health and Hygiene MOU Memorandum of Understanding MSP Multi-Stakeholder Platform MWSA Ministry of Women and Social Affairs NGO Nongovernmental Organization NWCO National WASH Coordination Office NWSC National WASH Steering Committee FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | iv OMI Omo Microfinance Institute OWNP One WASH National Program PSI Population Services International PSNP Productive Safety Net Program RBF Results-Based Financing SACCO Savings and Credit Cooperative SBCC Social and Behavioral Change Communication SDG Sustainable Development Goal SNNPR Southern Nations, Nationalities, and Peoples’ Region SOW Statement of Work SWZHB South Wollo Zonal Health Bureau Transform WASH Transform Water, Sanitation, and Hygiene Activity TVET Technical and Vocational Training and Education UNICEF United Children’s Emergency Fund USAID United States Agency for International Development USG United States Government VSLA Village Savings and Loan Association WASH Water, Sanitation, and Hygiene WASHPaLS Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability WHDA Women’s Health Development Army WHO World Health Organization WWSC Woreda WASH Steering Committee WWTT Woreda WASH Technical Team FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | v EXECUTIVE SUMMARY Transform Water, Sanitation, and Hygiene (Transform WASH) is a six-year, US$27.5M activity awarded to Population Services International (PSI) and consortium partners SNV, International Water and Sanitation Centre (IRC WASH), and Plan International to test and develop market-based models in Ethiopia that increase demand for and supply of quality and affordable water, sanitation, and hygiene (WASH) products and services. With a target to reach over 400,000 people in 41 woredas gaining access to a basic sanitation service, the phased implementation approach started with nine woredas in one region in the first year, with expansion to 41 woredas across all regions by Year 3. With activity support, business partners have sold over 100,000 basic sanitation products from 2017 to 2022 (PSI/Ethiopia, 2022), reaching well beyond the activity’s targets. USAID/Ethiopia requested that the USAID Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability #2 (WASHPaLS#2) project conduct a final performance evaluation of Transform WASH between April and October 2022. The evaluation sought to understand the effectiveness of demand creation activities, approaches to advancing market-based sanitation (MBS), Transform WASH contributions to good governance, and how the activity incorporated gender equality and female empowerment into design and implementation. The Evaluation Team employed multiple data collection methods including: (a) a desk review of activity documents and data; (b) over 160 key informant interviews with national and subnational stakeholders from across the public, private, and civil society sectors and with development partners; (c) 46 focus group discussions with communities, entrepreneurs, woreda administration officials, and consortium members; (d) 16 woreda visits (12 Transform WASH intervention woredas and 4 non-intervention woredas) including observations at the community level; and (e) a household survey that collected responses from 3,076 randomly selected participants (2,308 from intervention and 768 from non-intervention areas). The team chose non￾intervention woredas primarily to understand how MBS approaches are reaching non-activity areas. A key limitation was the lack of baseline data across the whole Transform WASH area to compare progress since the start of the activity.1 Brief Sanitation Context – With an estimated 2020 population of 101 million, and just under 80 percent living in rural areas (Government of Ethiopia [GOE] Federal Ministry of Health [FMOH], 2021), the GOE has seemingly made solid progress increasing rural sanitation coverage nationally from 11 percent in 2000 to 71 percent in 2017. According to Joint Monitoring Program (JMP) 2020 data, however, almost 21 percent of rural households still practice open defecation and those with access mostly use unimproved toilets. The household survey conducted for this evaluation confirms that over two-thirds of respondents (70.9 percent) have a latrine, of which 16.1 percent share that facility with others. Thus, Ethiopia remains significantly behind in attaining the 2030 GOE national goal of 82 percent basic sanitation coverage. A key mechanism to achieving this goal is deployment of an effective and sustainable MBS system (GOE FMOH, 2020). MBS is defined as the development of a sanitation market in which the user makes a full or partial monetary contribution to the private sector towards the purchase, construction, upgrade, and/or maintenance of a toilet. Building on the concept of “sanitation marketing” (largely a demand creation approach), MBS encourages entry and supports private sector capacity to provide products and services (USAID, 2018). Effectiveness of Transform WASH Demand Creation Approaches – The Transform WASH strategy aims to increase access to improved sanitation that hygienically separates human excreta from 1 Baseline data was collected across a range of health-related intervention issues for only the nine woredas in SNNP in which Transform WASH began initial implementation. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | vi human contact. Demand creation refers to activities carried out to drive awareness of and interest in WASH behaviors, products, and services. This needs to be supported by demand activation to persuade customers to convert their intent into a decision to purchase and install. The tools for demand creation include standard social and behavior change communication materials, such as health extension worker (HEW) counseling cards, village savings and loan association (VSLA) promotional efforts, business marketing activities (radio ads, market-day promotions, etc.), and other outreach activities. Strengthening public sector and community structures, Plan International generally followed a more standard approach, working through HEWs, the Women’s Health Development Army, VSLAs, and influential community leaders. Although HEWs reach most households, their visits focus on an 18- component package of health messages, thus entrenching intended behaviors and encouraging investments in and use of improved sanitation is not a sole focus. To refine messaging, Transform WASH designed a Decision Intelligence approach as a house-to-house counseling tool to encourage households to install and upgrade latrines using higher-quality sanitation products. From the private sector angle, PSI provides business facilitators who build capacity of businesses to offer sanitation products and services and support regional and woreda health officials to connect household demand to suppliers. PSI has also trained dedicated sales agents brought on by manufacturers, retailers, and mason/installers to generate sales on a commission basis. Increasingly seen as an effective investment, roughly 25 percent of Transform WASH business partners now directly contract sales agents to generate sales. Sales agents and mason installers largely focus on the Decision Intelligence sales approach and pictorial site seller at the household level, which is seen as more effective at generating product sales than the behavior change materials, by combining health messaging with a “problem-led approach that helps households make purchase decisions that address their own challenges” (as per Transform WASH staff). At a broader level, marketing campaigns supported “business partners’ demand creation activities with the slogan ‘My pride, My health’ [which is] meant to associate a sense of pride with owning a basic sanitation facility.” (PSI/Ethiopia, 2022). The public sector/community approach is aimed at poorer households to shift them from open defecation up the sanitation ladder to fixed point defecation, and then basic latrines. The private sector starting point is generally those who can afford it, aimed at reaching scale quickly. The tactics and tools used differ across the two approaches. The emergence of the Transform WASH business models begins to flesh this out. By Plan International’s own reckoning, sales conversion rates from messaging to sale and then sale to installation using more traditional social and behavior change communications tools have been less effective than the sharper, more innovative private sector approaches. Furthermore, sales agents receive a commission for every sale and thus have a direct and tangible incentive in a way that HEWs do not. After over a decade of public health messaging, Ethiopian households are aware of the benefits of investing in sanitation. Interviewees noted that household demand for sanitation products has increased dramatically over recent years. For customers who can afford to pay, a key strategy has been to ensure sufficient visibility of the options and then ease of purchase and installation. Interviewees with a range of suppliers noted that more could have been done to support product promotion and product placement, but, as Transform WASH staff understand, purchasing decisions at the household level are complicated by a variety of factors so a range of strategies need to be integrated effectively. For consumers willing but temporarily constrained in terms of cashflow, Transform WASH brings a wealth of experience and expertise to inform national and local-level microfinance institutions and savings and credit cooperatives about the role they could play in developing consumer lending products that meet these customers’ needs. With a subsidy protocol recently issued by FMOH supported by Transform WASH, greater recognition of sector financing bottlenecks, and more institutions experimenting in this space, there remains significant scope to explore these options in more depth. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | vii For those unable to pay, Transform WASH has experimented by incorporating sanitation loans into the VSLA approach in nine woredas across four regions. Over 450 VSLAs with more than 8,900 household members have received initial support, of which almost 3,900 (roughly 43 percent) have taken sanitation loans to date. In the absence of other channels of consumer finance or a clear government strategy until recently to increase access through targeted subsidies, VSLAs have given poorer households the opportunity to invest. For VSLA members, loan amounts are small, and several borrowings may be needed to install a latrine. While a sound investment strategy for poorer consumers, it may not provide the pace and scale needed by businesspeople to invest in the market. In terms of subsidies, discussions are underway to determine how to promote sanitation through the Productive Safety Net Program. Transform WASH has been piloting a subsidy scheme with results forthcoming. Understandable for an MBS activity, Transform WASH is widely perceived to be “more focused on the commercial part” of the equation with a steep learning curve among the activity implementers in understanding and working through a commercial marketing approach. The activity has operated in environments where there is an expectation of free product distribution, households are strapped for cash, and there is an absence of consumer lending or a targeted subsidy scheme for sanitation home improvements. The market is far from saturated, though, and so with some perception of market take￾off, the private sector is most likely to drive demand creation. Integrated efforts between healthcare officials and private sector actors with engagement of financial institutions and judicious use of financial mechanisms (subsidies, loans, taxation, etc.) will be critical going forward. Market-Based Sanitation Results – The series of actors involved in the supply chain include the importers/manufacturers of products like the SATO pan and AIM slab; wholesalers who make these products available through a distribution channel to distributors, or directly to the retailers and nongovernmental organizations; and slab manufacturers and mason installers. Along with HEWs, sales agents serve as an interface between households and product and service suppliers. These actors have formed the basis of three somewhat overlapping Transform WASH business models that provide installation or upgrading options for different consumer segments. A fourth business model is through retailers who offer plastic product sales and support to households who will then self-install. Transform WASH has analyzed and documented these four approaches well, explaining to potential new entrepreneurs what is required in terms of financial investment, space, equipment, and skill levels to offer a particular type of service. This information also translates into what entrepreneurs can expect in terms of average monthly profitability based on a set number of sales. Of the more than 530 business partners with whom Transform WASH has worked, over 430 remain active sanitation product and service providers. To strengthen these partners, the activity has offered a wide range of support (e.g., technical and business training, coaching, marketing and promotional materials, registration, data management, and connections to finance). Transform WASH now presumably understands which markets are most likely to take off and under what circumstances, but with aspects to explore further in terms of expectations around margins. Retailers may allow for lower margins for stock that sells quickly but may expect higher margins or discontinue selling the product if sales are slow. There is still some work to do to align the demand creation approaches; the market segmentation; the work on finance, pricing, and subsidy design and targeting; and related aspects to refine the business models further to understand how best to speed up the process. By their own admission, Transform WASH consortium interviewees noted a greater focus on encouraging and capacitating business partners and building supply chains than on business financing. As the sale of products increases, more formalized credit schemes will be needed, either within the supply chain or through externally developed credit lines. Transform WASH has experimented with a small revolving fund for lending to business partners through microfinance institutions, but a misconception by enterprises that the loans are supported by development assistance programs and need not be repaid. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | viii Stakeholders continue to call on Transform WASH to share its evidence and findings more effectively around how best to foster and support the market. The business models provide an ideal starting point for these discussions about what works best, in which geographies, and for which customer segments. The Evaluation Team saw thriving, profitable sanitation businesses at the local level that are adapting their business models and product options and services to increase sales—even if not at the pace needed to encourage a significant expansion of the market. Effectiveness of Governance and Learning Activities – Transform WASH has dedicated significant energy to activities aimed at ensuring that the enabling environment is open to and prepared for a viable MBS approach that will contribute to households’ improved access to sanitation services. The activity is widely appreciated as having advanced MBS through its contributions to important national-level policy and guidance documents as well as research and practical demonstration. Transform WASH led significant capacity building and training efforts and introduced several tools (e.g., the Institutional Strengthening Index, multi-stakeholder platforms, and joint supportive supervision exercises) to strengthen WASH and MBS subnational teams and share information. A key challenge has been to ensure that the material covered, skills reinforced, and tools put in place have become institutionalized, particularly in a context of very high turnover of regional and woreda leadership. Appreciation for these efforts notwithstanding, interviewees revealed that the consortium should have more deliberately integrated and sequenced these efforts with more intentional follow-up to identify: 1) the impact of these efforts; 2) any challenges faced in applying the learning and tools; and 3) any further gaps that may warrant support from Transform WASH consortium members. Overall, capacity building efforts have resulted in a change in officials’ thinking regarding sanitation broadly and MBS more specifically—as evidenced by the inclusion of sanitation more systematically in woreda strategic plans. Transform WASH has developed important research and learning publications and activities. While helpful, the strategy around developing the research agenda could have been clearer and the findings more effectively communicated at the subnational level. Such a learning strategy, owned by the whole consortium, would define which topics would have benefited from longer-term multi-year research efforts versus the more “quick and dirty” think pieces aimed at advancing debates and bringing new entrants into the sector up to speed quickly. Gender Equality and Female Empowerment – As in many countries, the manufacturing and construction sector in Ethiopia is male dominated, with participation of women quite limited. To its credit, Transform WASH interventions have systematically sought to alter social norms that marginalize women in the WASH value chain, creating opportunities for rural women by strategically identifying and encouraging them to enter manufacturing and construction businesses. Continuous head hunting by Transform WASH of women in construction material retailing businesses and energy saving cooking stove producers has been carried out to find and encourage them to expand their business operations to include sanitation products and services. Of Transform WASH business partners, women’s participation as sales agents of sanitation products is 85 percent, as manufacturers 14 percent, and as mason installers 4 percent. Women involved in MBS activities have practically demonstrated they can successfully lead profitable construction and manufacturing businesses. Despite numerous challenges that are embedded in the socio-cultural and socio-economic values of the community in relation to gender roles and social norms, the Evaluation Team found that female entrepreneurs and women-owned enterprises have been successful in promoting and selling their products and convincing customers to pay for their services. While needing further verification, stakeholders anecdotally report that women sales agents are more trusted than men and thereby achieve higher conversion rates in terms of sales. Strategies Transform WASH has employed to advance women’s participation and access have been successful. At the community level, the activity has also been instrumental in advocating for and supporting women to engage in VSLAs, which supports their economic empowerment and ability to afford to pay for their basic sanitation needs. With Oromia and Somali regions as outliers, the target of FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | ix 25 percent women’s participation in the MBS supply chain has been exceeded with a 28 percent aggregated contribution. This has broken the stereotypes in many communities that have held women back from certain business activities. While these advances are impressive and important, women remain less active in higher-level decision-making forums. The absence of the Ministry of Health’s Women, Children and Youth Affairs Directorate in national and regional level sanitation discussions and programming, including in the National Gender Equality and Social Inclusion Task Force, has been understood to be largely a capacity issue, with insufficient staffing to cover all incoming requests for engagement. This has structurally limited the ability of woreda- and kebele-level Women’s Affairs staff to support MBS activities that are targeting women-owned businesses. Consortium Effectiveness – Consisting of a series of interventions delivered through a consortium of four institutions, the coordination involving the staging and sequencing of activities becomes critical to ensure that policy work, demand creation and supply-side efforts join up effectively. The team clearly designated deliverables and targets for each consortium partner with regular exchange and reporting at the national level. The communications do not appear to have always translated down to the woreda level, however. While Transform WASH has been promoting this integration among regional and woreda stakeholders, subnational-level interviewees noted a lack of a joint annual plan from Transform WASH. Woreda staff “see separate organizations rather than one Transform WASH integrated approach.” As acknowledged in the Transform WASH Reflection Workshop, the learning agenda was also too siloed with one consortium partner and could have been more effectively designed as a cross-cutting activity. Observations on Spillover Effects – The differences between intervention and non-intervention woredas were somewhat stark. As expected, the Evaluation Team saw limited systematic demand creation activities in non-intervention woredas, no organized sanitation businesses, and emerging (but minimal) availability and awareness around improved sanitation products. In the intervention woredas, latrines seen during community visits were of higher quality, One WASH National Program structures were better established, and local authorities understandably had better awareness around MBS approaches. If Transform WASH coordinates with other projects, an MBS approach could expand quickly as rural consumers are aware of latrines and the benefits they provide. That said, there is a need to ensure that standards of product installation are of sufficient quality. Efforts are needed to ensure that MBS is understood and agreed as a strategy, particularly in areas where products continue to be distributed for free. Forums to share best practices across and within regions will support these efforts. Transform WASH has been quite careful and methodical in its experimentation. As this performance evaluation has shown, to be effective, an MBS approach needs to be designed with all the pieces of demand creation, supply side ecosystems, and governance and oversight well thought through. Highlights of Relevant Data from the Household Survey – The household survey revealed that being located in an activity intervention area was not significantly associated with whether or not a household had a toilet. That said, the number of households reporting no open defecation was significantly higher in intervention areas (51 percent) than those in non-intervention areas (25.7 percent). Of those who had a latrine, 17.6 percent in intervention areas and 11.3 percent in non￾intervention areas had recently upgraded (13.4 percent vs 6.9 percent in the last year). A fairly equal and significantly high percentage (73.2 in intervention woredas versus 68.4 in non-intervention woredas) of households believed that a toilet reduces the risk of diarrhea, whether they had a toilet or not. Conclusions – Overall, the evaluation findings confirmed that the USAID-funded activity has made many appreciated and significant contributions to advancing MBS as a valid approach with the potential to reach greater numbers with sanitation services in Ethiopia. Through advocacy and convening efforts, direct communications and a close working relationship with government, and the sharing of critical research on topics relevant to the sector, Transform WASH has shifted more skeptical attitudes toward MBS and helped shape MBS-related GOE policies and approaches. Shifting the discourse on key issues and supporting the revision of key GOE policies and guidelines have been significant contributions of the FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | x USAID-funded activity. Through learning by doing, Transform WASH has provided a practical demonstration of the National MBS Implementation Guidelines that brings ready-made lessons for non￾Transform WASH woredas to adopt. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 1 1.0 EVALUATION PURPOSE AND AUDIENCE 1.1 INTRODUCTION The United States Agency for International Development Mission in Ethiopia (USAID/Ethiopia) requested that the USAID Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) #2 project conduct a final performance evaluation of the Transform Water, Sanitation, and Hygiene (Transform WASH) Activity. This document provides the findings from the evaluation together with a set of short- and long-term recommendations for consortium partners and the funder. The evaluation started in April 2022, initial national-level interviews took place in May, field work occurred from June to August, and the team submitted the final report in October 2022. The evaluation statement of work (SOW) provided by USAID/Ethiopia is included as Annex A. 1.2 PERFORMANCE EVALUATION PURPOSE AND INTENDED USES As stated in the SOW, USAID sought quality evidence from this final performance evaluation on major achievements and lessons learned from the Transform WASH activity interventions. The overarching purpose of this evaluation was to examine the validity of the overall Transform WASH development hypothesis(es), validate the effectiveness and likely sustainability of approaches, and generate quality data for the decision-making process regarding related future United States Government (USG) investment and programming in the WASH sector. Specific objectives of this final performance evaluation were to: a) Understand the effectiveness of USAID/Ethiopia’s Transform WASH approaches and strategies to stated objectives and results, and assess factors influencing achievement or non-achievement of intended results; b) Estimate the broader outcomes of USAID/Ethiopia’s Transform WASH activity including the overall change in sanitation markets, changes in sanitation coverage, the viability of supported enterprises, ability of the market to meet the needs of the underserved, and the likely sustainability of these changes; and c) Identify and document key learning and recommendations to inform future programming. The evaluation sought to inform both USAID/Ethiopia and the implementing partners if the approaches employed are successfully meeting the activity’s goal of expanding and ensuring sustained access to market-based sanitation (MBS) services. The evaluation results are intended to inform decisions on any minor course corrections as time remaining in the activity allows or reemphasize activity priorities in its final months of implementation. 1.3 AUDIENCE This performance evaluation report lays out the findings, conclusions, and recommendations for each of the evaluation questions (EQs) laid out in the SOW (Annex A). USAID/Ethiopia is the primary intended audience for the report. The Mission will use the findings and recommendations to review its investment Brief Description of Transform WASH Transform WASH is a six-year (originally five, with a one￾year extension, January 2017–December 2022), US$27.5M activity awarded to Population Services International (PSI) and consortium partners SNV, International Water and Sanitation Centre (IRC) WASH, and Plan International to test and develop market-based models that increase demand for and supply of quality and affordable WASH products and services. Through the development and testing of scalable and replicable business models, this first￾of-its-kind activity sought to support and increase the capacity of the One WASH National Program (OWNP) to market, promote, and expand utilization of improved specific WASH products (e.g., the SATO pan, water filters, menstrual hygiene materials, and other products) and services (e.g., installation and repair). Transform WASH operates in all nine regions of Ethiopia targeting 41 woredas and Dire Dawa City Administration, where one of two other USAID activities (Transform: Primary Health Care or Transform: Health in Developing Regions) have also implemented interventions. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 2 and future programming in the WASH sector. Pending discussion on the types of information and how best to share these with the Government of Ethiopia (GOE) and other stakeholders, the GOE Ministry of Water and Electricity, One WASH National Program (OWNP) Coordination Office, and the Federal Ministry of Health (FMOH) would then be expected to use the findings and recommendations for planning and monitoring WASH activities and future design. 1.4 BRIEF EXPLANATION OF EVALUATION METHODOLOGY The Evaluation Team employed a mixed-methods approach to conducting the evaluation, gathering both qualitative and quantitative data through multiple data collection methods, including: (a) a desk review of performance reports and other project documents and data; (b) key informant interviews (KIIs); (c) focus group discussions (FGDs); (d) site visits; and (e) a household survey (HHS). Where feasible and available, the Evaluation Team used external third-party data to provide additional context on the WASH operating environment in Ethiopia and other related or complementary WASH programs. The Evaluation Team used methodological triangulation (i.e., consulting data from multiple sources such as interviews, Transform WASH reports, government documents, etc.), investigator triangulation (i.e., involving teams of two data collectors per interview and having multiple team members involved in data analysis and identification of emerging findings), and data triangulation (i.e., obtaining the perspectives of multiple similar types of respondents across woredas in which Transform WASH operates). In total, the Evaluation Team conducted over 160 KIIs at the national, regional, zonal, woreda, kebele, and community levels with stakeholders from across the public, private, and civil society sectors, and development partners. The team held 46 FGDs with communities, entrepreneurs, woreda administration officials, and consortium embers. As noted in Table 1, of the 16 oredas visited, 12 were Transform WASH ervention woredas (from a total of 41) and 4 were non-intervention woredas. The Evaluation Team chose control or non-intervention woredas primarily to understand the spillover effects for two woredas adjacent to intervention woredas, and to place these in contrast to two woredas that are non-adjacent to derive some understanding of how MBS approaches are reaching non-Transform WASH areas more broadly. To complement the qualitative findings, a household survey, designed by the Evaluation Team and reviewed by USAID/Ethiopia, was conducted by an Ethiopian firm (PRIN Consulting). Data were collected from 3,076 randomly selected study participants across the same 16 woredas, including 2,308 participants from Transform WASH intervention areas and 768 from non-intervention areas. The response rate was 99.9 percent. The team collected data through the Kobo Toolbox platform and exported it to SPSS v20 for analysis. The team cleaned data and removed outliers, and computed descriptive statistics such as means, frequencies, and percentages. They used logistic regression to m int Table 1: Intervention and Control Woredas Selected for Primary Data Collection Control woredas noted in grey (*Dire Dawa was visited at the administration level. It was replaced by Gursum in Oromia Region by w the household survey team.) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 3 determine associations between independent and dependent variables; results were considered significant for p<0.05. Thus, the team used quantitative data through the household survey and the Transform WASH dashboard and reporting to establish correlation. The Evaluation Team used qualitative data from KIIs and FGDs to provide some explanation of causation and give a sense of the magnitude of outcomes and changes. Map 1: Transform WASH and Performance Evaluation Woredas 1.5 LIMITATIONS OF THE METHODOLOGY A key limitation for the Evaluation Team in assessing overall progress over the life of the Transform WASH activity was the lack of baseline data across the 16 woredas against which to compare progress. A wider baseline collection effort was conducted in Southern Nations, Nationalities, and Peoples’ Region (SNNPR) but with limited questions focused specifically on WASH aspects. Thus, the Evaluation Team was largely reliant on scant data and interviewee perception of conditions at the start of the activity. This was emphasized through questions in the household survey seeking to understand whether WASH-related circumstances have changed over time. Due to language limitations, the Evaluation Team used an interpreter for 2 of 16 woreda visits. The team conducted field work during the rainy season when households had limited disposable cash on hand to buy sanitation products. The timing may have influenced the responses of all local-level stakeholders, even though households may purchase sanitation products in the dry season. The timeframe pressures meant that there was less synergy in scheduling between the Evaluation Team and the team carrying out the household surveys. The initial aim was to ensure that there was sufficient overlap to ensure that definitions were consistently understood and being applied and to jointly reflect on emerging findings to tease out further some aspects of correlation and causation. 1.6 STRUCTURE OF THE REPORT This final report of the performance evaluation of Transform WASH includes a brief overview of the Ethiopian context with regard to sanitation coverage in Section 2; the findings and brief conclusions for each of the evaluation questions plus brief analysis of the effectiveness of the Transform WASH consortium, findings on the spillover effects of Transform WASH in non-intervention woredas, and overarching findings from the household survey in Section 3; conclusions and recommendations for the activity consortium and the funder in Section 4; and primary references in Section 5. Annexes include the USAID/Ethiopia Scope of Work (Annex A) and provide further detail on the evaluation methodology (B), the data collection instruments (C), the household survey findings (D), the list of participants in KIIs and FGDs (E), and the full list of Transform WASH woredas (F). FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 4 2.0 PUTTING TRANSFORM WASH INTO CONTEXT 2.1 BRIEF SANITATION CONTEXT IN ETHIOPIA With an estimated 2020 population of 101 million, and just under 80 percent living in rural areas (GOE FMOH, 2021), the GOE has seemingly made solid progress increasing rural sanitation coverage nationally from 11 percent in 2000 to 71 percent in 2017. According to Joint Monitoring Program (JMP) 2020 data, however, almost 21 percent of rural households still practice open defecation and those with access mostly use unimproved toilets. The Revised National MBS Guidelines (GOE FMOH, 2020) suggest that 93 percent of the population requires strategic support from federal, regional, and local governments (with assistance from development partners) to increase access to basic sanitation and hygiene facilities. Thus, Ethiopia remains significantly behind in attaining the 2030 GOE national goal (urban and rural) of 82 percent basic sanitation coverage. Very much in line with the national statistics, the household survey confirms that over two-thirds of the 3,076 respondents (70.9 percent) have access to a latrine, of which 19.2 percent can be categorized as reaching basic sanitation standards and 16.1 percent share that facility with others. Latrine ownership, however, is not uniform within and across regions; Sidama region had the highest (96.3 percent) and Somali region the lowest (24 percent) number of households using a latrine. Failure to convert households to basic/improved sanitation facilities means that the waste is still open to the environment and is a potential source of disease-causing pathogens.2 Diarrhea alone remains the third leading cause of under-five mortality in Ethiopia, indicating the presence of unmanaged risks (GOE FMOH, 2020). 2.2 GOE SANITATION POLICIES AND STRUCTURES A key mechanism to achieve the GOE strategy goal is by deploying an effective and sustainable market￾based system for sanitation products and services. (GOE FMOH, 2020). According to the National Hygiene and Environmental Health Strategy (2016-2020), all households are responsible for building their own basic sanitation and hygiene facilities, while attainment of total coverage depends on the government’s political commitment, as well as administrative, programmatic, and budget support. To change the hygiene and sanitation status of the population, the government has developed and implemented strategies and plans, and capacity building and community mobilization interventions using the Community-Led Total Sanitation and Hygiene (CLTSH) approach (GOE FMOH, 2020). In terms of structures, the National WASH Steering Committee (NWSC) is chaired by the Minister of Water and Energy and includes ministers and state ministries from the ministries of Water and Energy, Health, Education, and Finance. For sanitation specifically, under the guise of the NWSC, the FMOH takes a primary leadership and coordination role by devising policies, strategies, and protocols and delivering them directly through subnational offices, enabling the delivery by other stakeholders, and monitoring the implementation of these key government objectives. Under the FMOH, the Regional Health Bureaus support the delivery of these strategies through the woreda and primary health care units and health posts down the chain to the healthcare extension workers (HEWs). A range of other government offices and development partners also make important contributions through awareness raising, implementation activities, research and evidence gathering, and other related activities. With support from development partners, efforts like that of Transform WASH are increasingly being required to engage the private sector in sanitation efforts. 2 Adopted in August 2021, the GOE definition of improved sanitation is “Sanitation facilities that are designed to hygienically separate human excreta from human contact. These include wet sanitation technologies such as flush and pour flush toilets connected to sewers, septic tanks or pit latrines, and dry sanitation technologies such as dry pit latrines with slabs and composting toilets.” Basic sanitation is “Use of improved sanitation facilities that are not shared with other households” (GOE FMOH, 2021b). FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 5 Governed by the NWSC, the OWNP coordinates planning, implementation, and reporting efforts across these various government and nongovernment stakeholders. The OWNP oversees the Consolidated WASH Annual Plan (CAP), which reflects the strategies and implementation activities of all stakeholders contributing to WASH activities through the budget mechanism of the Consolidated WASH Account (CWA). This process is replicated down to the region, zone, woreda, and kebele levels. The OWNP coordinates monitoring and evaluation (M&E), organizes periodic program reviews, and stimulates learning and knowledge management. Ultimately, there is a recognition in government that “successful sanitation change does not solely depend on the supply of sanitation infrastructure, but requires changes to the political, economic, social, cultural, and environmental underpinnings of sanitation, and a systemic behavior change at individual, household, and community levels” (Novotný et al., 2018). Efforts by government and development partners are very much aimed at holistically understanding and overcoming barriers across these angles. The general sense (not specific to Ethiopia) is that the prevalence of unimproved sanitation facilities is due to the lack of affordable or desirable options to construct an improved toilet (USAID, 2018). The data in Ethiopia suggests that lagging growth in improved sanitation coverage for many households is not caused primarily by affordability issues or a lack of willingness to pay for improved latrines, but a lack of availability of viable options in the market. As such and as discussed below, the Transform WASH activity objectives to increase latrine access through an MBS approach is seen as consistent with the strategic GOE objectives as set out in the FMOH Hygiene and Environmental Health Strategy (FMOH, 2016-2020) and the OWNP (Phase II – 2018-2020). 2.3 MBS AS A TOOL AND THE USAID/WASHPALS MBS FRAMEWORK MBS is defined as the development of a sanitation market in which the user makes a full or partial monetary contribution (with savings and/or cash equivalents) toward the purchase, construction, upgrade, and/or maintenance of a toilet from the private sector. Building on the concept of “sanitation marketing” (largely a demand creation approach), MBS strengthens supply by encouraging entry and supporting capacity of the private sector to provide such products and services (USAID, 2018). While the focus of interventions tends to be the sanitation market (the interaction between buyers and sellers), successful MBS interventions also seek to bring about change in the broader sanitation market system (e.g., associated supply chains and supporting functions like financial services infrastructure). To apply this systems lens, the USAID/WASHPaLS framework for MBS interventions specifies the various levels at which stakeholders should intervene to bring about systems change (see Figure 1). According to the USAID/WASHPaLS Scaling MBS report (USAID, 2018), “The framework identifies three distinct domains: (1) broader context such as social norms, economic environment, and geographical conditions, which interveners must seek to understand but typically cannot influence; (2) business environment shaped by government policy or the availability of raw materials and financial services, which interveners can potentially influence, depending on complexity and resources available; and (3) the core sanitation market, which is comprised of customers, enterprises, and entrepreneurs; each of which large-scale interventions can address.” FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 6 Figure 1: USAID’s Conceptual Framework for the Sanitation Market (USAID, 2018) The business environment is further shaped by factors such as the availability of non-excludable public goods (e.g., market information on product designs in the public domain); the state of associated supply chains (e.g., availability and price of construction raw materials used to build toilets); the state of financial services infrastructure, which affects the availability of credit for customers and entrepreneurs; and business-related laws, regulations, and policies (e.g., government programs to provide in-kind hardware subsidies) (USAID, 2018). In theory, MBS should provide customers with products that they want and for which they are willing to pay, be financially sustainable (a sanitation enterprise earns profits by delivering products), be cost￾effective and scalable, and make households more likely to derive the benefits of sanitation (e.g., improved health, privacy, dignity) through the use and maintenance of toilets that they value (Cairncross, 2004, as quoted in USAID, 2018) 2.4 THE TRANSFORM WASH ACTIVITY’S RESPONSE3 3 Adapted from Transform WASH documentation. USAID Transform WASH aims to influence, where feasible, the broader context in which MBS is operating, advance the specific business environment through government policy and institutional support to the sector, and develop a sanitation market that is capable of responding to effective demand creation activities. While multi-layered, a key focus of Transform WASH is to test and develop market￾based models that are aimed at increasing demand for and supply of quality, affordable WASH products and services. Through development and testing of these scalable and replicable business models, the activity supports and increases the capacity of the GOE OWNP. This is achieved through: 1) increased WASH governance and management capacity at subnational level; 2) increased demand for low-cost, quality WASH products and services, with a primary focus on sanitation; 3) increased supply of such products and services, with a focus on sanitation; and 4) an increased knowledge base to bring WASH innovations to scale (Population Services International/Ethiopia, 2022). The target is to reach over 400,000 people in 41 woredas gaining access to a basic sanitation service as a result of USG assistance. 1 BROADER CONTEXT SANITATION MARKET BUSINESS ENVIRONMENT CUSTOMER Affordability Availability ENTREPRENEUR Target Market Delivery Model Product System Sales & Marketing ENTERPRISE FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 7 The goal of Transform WASH is to reduce preventable death and illness in Ethiopia due to diarrheal disease, particularly among children under five years of age. The activity’s results framework is presented in Figure 2. R1.1 Woreda WASH teams capacitated to provide sustainable WASH services R1.2 Regional/zonal levels capacitated to support OWNP activities R1.3 USAID/Ethiopia supported strengthening sector policies R2.1 Increased household demand for WASH products and services through the HEW platform R2.2 Increased household demand for WASH products and services through private sector enterprises R3.1 Value chains to supply WASH products and services strengthened R3.2 Capacity of the local private sector to engage in WASH sector built R3.3 Availability of financing for WASH products and services increased R4.1 Current models for increasing demand tested and documented R4.2 Models for increasing household access to financing for WASH products to increase update of WASH products tested and documented R4.3 Partnerships between the private sector and government to scale WASH products and services developed Objective: Increased Use of Improved WASH Products and Services R1: Increased WASH governance and management capacity at sub￾national levels R2: Increased demand for low-cost quality WASH products and services, with a focus on sanitation R3: Increased supply of low-cost quality WASH products and services, with a focus on sanitation R4: Increased knowledge base to bring innovations to scale Goal: To reduce preventable death and illness in Ethiopia due to diarrheal disease, particularly among children under five Figure 2: Transform WASH Development Hypothesis Results Framework While the USAID/Ethiopia health office has worked for several decades supporting sanitation marketing of key life-saving health products such as contraceptives, oral rehydration salts, water treatment tablets, and various nutritional supplements and nutritious foods for women and children, “the Transform WASH Activity represents a first-of-its-kind pilot program to harness the power of the market to build commercially viable suppliers of sanitation products.” (USAID, 2016) The Transform WASH activity is comprised of four results areas: • Enhancing governance and management capacity at the sub-national level; • Increased demand through social and behavior change communications (SBCC) that seeks to catalyze the existing cadre of HEWs to promote WASH products to households; and to promote models for community financing to increase households’ ability to access sanitation-related products and services. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 8 • Increased supply of sanitation products with a focus on building the business and marketing skills of sanitation product suppliers and partners by testing four business; and • Developing an increased knowledge base to bring innovations to scale. Developed by the Evaluation Team, Figure 3 shows the entire demand creation and supply side response ecosystem. Transform WASH has been implemented in three phases. Phase 1 started with 9 woredas in SNNPR in the first year and expanded by 11 woredas in Year 2 (Phase 2) to include Amhara (4), Oromia (6), and Benishangul Gumuz (1) and by 21 woredas in Year 3 (Phase 3) to include Amhara (5), Oromia (7), Afar (1), Somali (3), Gambella (1), Tigray (3), and Dire Dawa Administrative City (1), totaling 41 woredas over the three phases. In Year 6, Transform WASH expanded its total coverage to 63 woredas by adding 22 from the Growth through Nutrition (GtN) Activity after its close in December 2021. (Annex F provides a full list of Transform WASH intervention woredas.) The focus for these late joining woredas has largely been around capacity building and sharing best practice. With support from Transform WASH, business partners have sold over 100,000 basic sanitation products between 2017 and 2022 (Population Services International/Ethiopia, 2022), reaching well beyond the targets set at the outset of the activity considering an average household size of 5+ members. In its simplest sense, Plan International and PSI have focused on demand creation, PSI has taken the lead on business development, SNV has focused on capacity building efforts at the subnational level, and IRC has focused on knowledge generation, management, and sharing. Among Transform WASH consortium staff members working at the central level, PSI has nine (Chief of Party, Deputy Chief of Party, Program Director, Gender Advisor, Business Development Team, Marketing Team, and M&E Manager), SNV has two (Transform WASH Project Manager/Institutional Development Advisor and Capacity Building Advisor), Plan International has three (Project Manager, Economic Empowerment Coordinator, and SBCC Advisor), and IRC has four (Transform WASH Team Leader, Research and Policy Manager, Learning Alliance Coordinator, and Publication and Knowledge Management Officer). Not all positions are full time. At the regional level, PSI has 22 field staff members (Business Development Team and M&E Advisors), SNV has seven Capacity Building Advisors, and Plan International has seven SBCC Coordinators. In addition, PSI has 18 Business Facilitators at the woreda level (Population Services International/Ethiopia, 2022). Transform WASH is most active at the woreda level. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 9 Figure 3: Full Sanitation Service Delivery Ecosystem FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 10 3.0 EVALUATION QUESTIONS: KEY FINDINGS 3.1 EVALUATION QUESTIONS While Transform WASH is implementing a range of sanitation-based activities with clear deliverables and targets, USAID’s EQs for this performance evaluation fall under four key themes: fostering demand (EQ1), ensuring supply (EQ2), the broader enabling environment and governance (EQ3), and gender equality and female empowerment (GEFE) (EQ4) as follows: 1. How effective have the Transform WASH approaches to demand creation been at contributing to improving sanitation uptake through an MBS approach and thereby sanitation outcomes more broadly? 2. How effective have the Transform WASH approaches to market-based sanitation been at contributing to improving sanitation uptake and thereby sanitation outcomes more broadly? 3. To what extent has Transform WASH contributed to good governance in the sanitation sector particularly with regard to MBS? 4. How did Transform WASH consider gender equality and female empowerment in activity design and implementation? The Evaluation Team modified the EQs somewhat from the original provided in the SOW from USAID/Ethiopia (see Annex A). They proposed these to sharpen the focus and reduce minor duplication and overlap amongst EQs. While difficult to predict with high levels of confidence, the Evaluation Team sought evidence that project components are likely to be sustained by other actors upon completion of the Transform WASH activity, and, if not, what elements could have been strengthened or should be emphasized in any future work. A further aspect is around how Transform WASH staff have learned from their own M&E practices to sharpen their approaches. Addressing the EQs does not require a determination on whether Transform WASH is on track to meet its targets or to validate or verify the specific achievement of reported targets. A categorical determination validating that the targets across the 41 woredas have been met was not possible with the time and resources available. The Evaluation Team’s approach was to understand the factors that are most likely to hinder sustainable sanitation service provision and then to determine how well the Transform WASH project has been at addressing these through an MBS approach. The subsequent sections provide a summary of the key findings for each EQ. Recommendations are embedded in the narrative but also provided in a separate section at the end of the report (Section 4.0). 3.2 EQ KEY FINDINGS 3.2.1 EQ1: EFFECTIVENESS OF TRANSFORM WASH DEMAND CREATION APPROACHES How effective have the Transform WASH approaches to demand creation been at contributing to improving sanitation uptake through an MBS approach and thereby to sanitation outcomes more broadly? Performance Evaluation EQ1 Data Sources Sources of information for the demand creation line of inquiry included project documents triangulated against the Evaluation Team’s own data collection efforts. GOE reports helped to validate some of the key challenges of demand creation approaches in the Ethiopian context. KIIs and FGDs with HEWs, local opinion leaders, and key WASH sector actors, including from the private sector, also provided a key source of information to understand perceptions around the likelihood of product uptake and sustained behavior adoption. The household survey (see Annex D) also provided a critical source of information on, for example, perceptions of the importance of sanitation investments and how such decisions are made at the household level. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 11 The Transform WASH demand creation and SBCC strategy aims to increase access to improved sanitation that, according to the GOE definition, is designed to hygienically separate human excreta from human contact. Demand creation is understood to mean activities carried out to drive awareness of and interest in WASH behaviors, products, and services. This needs to be supported by demand activation with activities carried out to persuade customers to convert their intent to purchase into a decision to purchase and install (see Figure 4). The tools for demand creation include standard SBCC materials, such as HEW counseling cards, combined with village savings and loan association (VSLA) promotional efforts, business marketing activities (radio ads, market-day promotions, retailer umbrella branding, etc), and other similar outreach activities. Transform WASH developed the Decision Intelligence approach to further refine messaging at the household level. The DQ is a house-to-house, problem-led counseling approach to encourage households to invest in or upgrade their latrines using higher-quality sanitation products (e.g., concrete slabs, concrete slabs fitted with SATO, retrofitting and installation of medium size plastic slabs). All demand creation actors at the project level present sanitation products to customers using a picture catalog. EQ1 – Key Findings. Using the USAID/WASHPaLS framework as the starting point for the Evaluation Team’s analysis, demand strategies pivot around three key elements relating to the customer base: latent demand, affordability (willingness to pay and price elasticity), and household liquidity (including the role of subsidies and availability of microfinance). Transform WASH’s two-pronged demand creation approach is shown in Figure 5 with a public sector/community approach (blue and grey elements in the graphic) and a private sector approach (red elements). Figure 4: From Demand Generation to Demand Fulfilment (USAID, 2018) EQ1 Key Findings Customer segmentation strategies for demand creation could be more explicit in relation to MBS. The business models suggest further refinements. The VSLA approach reaches poor families, but not at a speed that accelerates the market. Household financing options could have been explored further and remain critical – the subsidy pilots will be important going forward. Evidence of marketing materials and product visibility was limited – business partners expressed that more support could be provided in this area. Sales agents are an effective strategy and business partners are beginning to hire them directly. Sales catalogues that focus on pride of place appear to be more effective than health-related behavior change campaigns. Efforts to contextualize products and services has helped to accelerate sales. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 12 Strengthening public sector and community structures, Plan International has generally followed a more traditional approach, working through HEWs and the Women’s Health Development Army (WHDA), as well as VSLAs and influential community leaders. HEWs and health center supervisors provide counseling at the household level on options for investing in sanitation. Local-level formal and informal leaders, like kebele managers and administrators, gott (village) leaders, and development groups (e.g., Iqub, Idir, and WHDA), and religious leaders, disseminate messages. Training materials carefully follow national SBCC protocols, supported by public service announcements at the community level. The installation of model latrines at health care facilities (HCFs) has also been included to support the dissemination of messages. Figure 5: Demand Creation Ecosystem Although HEWs reach most households in the intervention kebeles, their visits focus on a range of health messages, and thus entrenching the intended behaviors and encouraging investments in and use of improved sanitation is not a primary focus. All government stakeholder respondents as well as sales agents and masons confirmed that they primarily use the counseling cards and DQ approach. Key message displays, billboards and sign boards, or other marketing materials for product visibility or public service announcements were not widely seen during the field visits. The installation of model latrines at the HCFs was scaled back as a Transform WASH intervention (from latrines at 40 centers to latrines at 20 centers) due to construction and management concerns. Those that the Evaluation Team saw were often not open for use by patients or their families. It was unclear to the team what the strategy was to integrate these into the messaging and how the locations had been chosen. It is understood that these initial investments when Transform WASH entered a Sales agent and mason at Hoda (Picture credit: Muhammed Ebrahim Mussa) Door-to-door counselling booklet (Picture credit: Muhammed Ebrahim Mussa) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 13 woreda were seen as quick wins to get subnational authorities on board with the activity. Indeed, subnational authorities were initially expecting that Transform WASH would make significant direct investments in infrastructure. It took some effort by consortium members to create awareness around a kind of development partner intervention that was more geared to building up a local stakeholder response than to the more traditional approach of direct provision through international assistance. From the private sector angle, PSI provides business advisors/facilitators who support two or three woredas to work with regional and woreda health officials to build capacity of sanitation product and service providers and connect household demand to suppliers. Working on a commission basis, PSI has also trained dedicated sales agents who act on behalf of manufacturers, retailers, and installers to generate sales. Sales agents, and mason installers where sales agents have not been deployed, then do their own marketing. This has increasingly been seen as an effective investment by businesses as, according to PSI’s own surveys, roughly 25 percent of Transform WASH sanitation enterprises are now directly contracting sales agents on a commission basis to generate sales, with some of these deploying two or more sales agents. Sales agents and mason installers primarily use the Decision Intelligence sales catalog, which is found to be more effective at eliciting product sales, than the behavior change materials. The Decision Intelligence approach at the household level is distinct from wider marketing activities. Marketing campaigns supported “business partners’ demand creation activities with the slogan ‘My pride, My health’ [which is] meant to associate a sense of pride with owning a basic sanitation facility. This was used as the basis for developing all marketing and communication materials” (PSI/Ethiopia, 2022). This aligns with the literature conveying that “messaging around health benefits alone rarely catalyzes sanitation demand, [and as such] marketing should focus on the functional and emotional attributes of the specific products being sold, such as convenience, safety, durability, status, and privacy.” (Multiple sources as referenced in USAID, 2018). That said, wholesalers and retailers suggested that more support could be given by Transform WASH in terms of how best to promote the product and at what touchpoints for different target customer segments. This includes more of a focus on superstructure models. In terms of sustainability, PSI is developing a plan around who will continue to train sales agents and what are the best ways to do so after the Transform WASH activity ends. A training manual, already developed and soon to be launched, will be rolled into the Transform WASH capacity building and facilitation guidelines being developed at the national level (discussed under EQ3 below). The role of technical and vocational education and training (TVET) and the job creation centers (discussed under EQ2) could still be strengthened to incorporate this aspect. Understanding issues around household awareness – Interviews at the community level suggest that, after over a decade of CLTSH efforts, there is good awareness of the benefits of investing and a willingness to invest (over time) in sanitation infrastructure at the household level.4 A greater understanding of consumer behavior, religious factors, and other market dynamics have supported uptake in several implementation areas. Contextualizing the installation of SATO pans to prevent the issues raised by Muslim users, for example, around technologies that most easily foster cleanliness and hygiene, has resulted in wide acceptance in the Somali region, even if the overall access in the region is 4 HEWs were identified in the household survey as the main source of information about the benefits of having a toilet for 90 percent of Transform WASH intervention and 94 percent of non-intervention areas. 54 percent of intervention and 63 percent of non-intervention area households had attended an SBCC event. “Nowadays, the demand for the latrine products, especially SATO and AIM, is progressively increasing. However, we still need to have some promotional support to increase our reach to the far kebeles …” (Wholesaler, Debremarkos￾Gozamen Woreda) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 14 among the lowest.5 Interviewees noted that further contextualization (of messaging and promotional materials, including around technologies for different soil types) could result in still higher numbers of households investing in improved sanitation. According to Ethiopia-specific literature and as confirmed by the household survey data, “[while] potential effects on human health were the most commonly reported motivations for various aspects of the sanitation process… intuitively, such a widespread focus on health-related reasons does not correspond to the fact that people can hardly trace cause-and-effect links between latrine use and their health from their experience” (Novotný et al., 2018). As noted, Transform WASH (primarily through the private sector approach) has helpfully sought to combine this health messaging with messages around pride of place. As per the literature, a key function of demand generation activities is to overcome households’ overestimation of the cost of installing a latrine; to bring greater awareness of latrine designs, models and other sanitary options; and to work with households to overcome space limitations (USAID, 2018). In its work with business partners, Transform WASH has clearly approached marketing in this way. In terms of awareness and preference for specific technologies, while the SATO pan is appreciated, the AIM product appears to be overtaking the SATO pan in terms of sales. This is largely because the AIM product needs less or no water for cleaning (particularly helpful in areas with less access to water)6 and is easier to install and lighter to transport, but most importantly, the cost of construction materials needed to install the SATO pan with the reinforced concrete slab is increasing. The SATO pan combined with the reinforced concrete slab is, nonetheless, seen as more durable and easier to clean, and its self-closing design more readily limits odors and flies. (A key caveat is that the AIM slab has only been on the market for two years, compared to SATO that has been available for several years longer.) The Evaluation Team understands that technology preference (as a function of price and other factors) is an area that Transform WASH continues to monitor closely (nationally and by region) in an effort to support business partners to respond appropriately. Indeed, the business models (discussed below) offer a range of options for households, from skirting (with a smaller platform in which the pan sits) and retrofits to more complete installations with full slabs for households that can afford it. Understanding issues around household affordability – A key element of the USAID/WASHPaLS MBS framework is understanding issues of affordability at the household level. Interviewees noted that household demand for sanitation products has increased dramatically over recent years. The key challenge for scaling uptake, as stated repeatedly in KIIs, is to ensure that products and services are affordable. Access to finance at the household level remains a key bottleneck to reaching the goals around uptake. The increasing cost of living as reflected in rising prices of construction materials (noted as increasing three- and four-fold over the past two to three years) and other inputs, like labor, has resulted in households putting off the purchase or purchasing and installing/upgrading latrines over a prolonged period when incomes or access to finance allows (e.g., after the harvest season). As reported in household interviews, a further complication is the expectation that products will be made available for free or at highly reduced prices as part of development partner activities. A few household-level interviewees were quite explicit about this, stating that “the project gives products to the retailer for free and then they sell it and take all the profit.” Numerous cases of free distribution by other development partners in several woredas does appear anecdotally to have influenced potential customers’ purchasing decisions (although only 104 respondents in the household survey suggested that 5 With support from the regional health bureau, PSI and TVETs, using a high pedal retrofit and lowering the SATO pan somewhat below floor level has ameliorated the problem of urine splash back on clothing or parts of the body, which is forbidden in the Muslim religious tradition. Masons at Dege Habour were trained on this new installation technique with positive results from the community. 6 Only 12.4% of households (17 percent in non-intervention areas and 11 percent in Transform WASH intervention woredas) had access to more than 25 liters of water per day. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 15 they were waiting for free distribution). Thus, the interest in installing latrines is not reflected necessarily in a dramatic increase in sales for a range of reasons (e.g., drought influencing incomes, escalation of construction material prices and labor costs, and conflict, especially in the northern part of the country). Studies supported by WASHPaLS and reaffirmed by Transform WASH’s own analysis suggest that purchasing decisions at the household level are very price sensitive. For most Ethiopian households, demand sharply declines once the price of plastic pans passes ETB 200 (US$3.85 at August 2022 exchange rate). For instance, “at the price point of ETB 400 (US$7.69) for toilet plastic pans, the demand from the bottom 40 percent of households is almost zero while only around 5 percent of the middle 40 percent of households reported being able to afford it” (GOE FMOH, 2021c). Given that evidence, sanitation subsidies for the poorest seem necessary. Ethiopia is unlikely to reach all households with improved sanitation products and services by 2030 without a targeted and smart subsidy scheme (GOE FMOH, 2021c). Thus, a key objective is to keep prices low in an environment where plastic products are largely imported. At present, SATO pans are manufactured outside of the country and brought in on a tax-exempt status as part of development partner initiatives. Once such programs end and the private sector assumes the role of directly importing and distributing sanitation products, the price will increase by at least 35 percent under the current tax regime (USAID, 2021). Thus, this price elasticity challenge makes such investments even more unlikely for income￾poor Ethiopian households in particular, unless (as discussed under EQ2) some success can be had at advancing domestic manufacturing. Increased household access to microfinance/subsidy – With household interest in installing or upgrading toilets growing (as based on Transform WASH’s analysis and Evaluation Team KIIs and FGDs), Transform WASH has been seeking to understand affordability, recognizing that consumers can be categorized into three sub-groups: • Those who can afford to directly purchase basic sanitation products and services; • Those who are willing to purchase the products and services, but face financial liquidity constraints during non-harvest seasons and/or are not able to pay at once but can regularly save some amount of cash into a saving and credit scheme; and • Those who cannot afford sanitation products and services due to lack of means or lack of livelihood or other chronic social problems.7 Transform WASH noted that the rough breakdown across these categories is that 30 percent can afford to purchase directly, 30 percent can pay but with some cashflow constraints, and 40 percent cannot 7 This is increasingly recognized as a standard approach in the WASH sector to segmenting the market. Household Survey Data Only 12.8% of respondents reported that it was affordable to maintain their latrine, roughly the same proportion in both intervention and non-intervention areas. Almost half (47%) of respondents cited a lack of disposable cash as a challenge to toilet upgrading (45% in intervention, 52% in non￾intervention). Almost 75% of households from both Transform WASH intervention and non￾intervention areas reported having no money to purchase latrine upgrading products in the market. Household Survey Data Participants who were currently working in paid employment were 70% more likely to have a toilet than those who were not. As expected, higher economic status of study participants was significantly associated with having a toilet. Households that were in the wealthiest quartile were more likely to have a toilet than those in the other quartiles. Being the beneficiary of Transform WASH projects was not significantly associated with having a toilet. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 16 afford such investments and require support. Strategies are needed to address the needs of each of these groups. Reaching those who can afford to pay – For the first category of consumer, door-to-door sales and promotion is the main strategy of businesses with different options offered to the household from a completely new installation of an improved latrine to a significantly cheaper retrofit of an existing one. The key strategy has been to ensure sufficient visibility of the options available in the market, and then ensure ease of purchase and installation. From discussions with a wide range of suppliers, the Evaluation Team noted that more could have been done around product promotion and product placement. The literature also suggests that a focus on early adopters may help “drive acceptance of new sanitation enterprises and toilet designs by the wider market, whose preferences and aspirations are often shaped by observing usage by neighbors and receiving word-of-mouth recommendations” (USAID, 2018). The Transform WASH dashboard, which aggregates data collected across the activity, bears this out with word-of-mouth from early adopters seen by retailers and mason/installers as being a primary conduit for new sales. Reaching those who are willing but have a temporary constrained cashflow – As the Revised National Market-based Sanitation Implementation Guideline (2020) suggests, responding to the needs of the second category, those who are willing but temporarily cash poor, the options include lowering the price of the products, ensuring low costs of installation in part by positioning service providers nearby, and providing access to some form of micro-credit service or supplier provided credit arrangements through instalment payments. Transform WASH has little control over the rising cost of materials and labor for installation, but has worked extensively with government to ensure that prices of imported sanitation products are affordable, at least until such time as there are domestic producers in place. As discussed under EQ2, Transform WASH has also been very successful in engaging business partners from across the supply chain to tailor their offers to meet the needs of different kinds of consumer. Through numerous stakeholder conversations, the Evaluation Team notes that more could still be done for consumers by facilitating access to untapped or underutilized financing options at both woreda and kebele level. The Revised National MBS Guideline discusses this issue in some depth noting the need to engage microfinance institutions (MFIs) and commercial/development banks on the household product opportunities in this emerging sanitation space. Having helped shape the MBS guideline, Transform WASH can now bring a wealth of experience and expertise from working with implementers to inform such institutions about the role that they could play in developing consumer lending products that meet the needs of Category 2 consumers who face temporary cash flow constraints, and perhaps also to Category 3 consumers. The Evaluation Team sees savings and credit cooperatives (SACCOs) as an untapped resource. Some SACCOs, like in Bahirdar-Gojjam, have incredible reach and could serve as an appropriate vehicle for mobilizing significant uptake in sanitation products and services. In rural kebeles, 80 percent of households are members of a cooperative that is administered under Union Bank. These cooperatives offer different services, and, though they mainly focus on agricultural product loans, some have developed a sanitation loan as part of home maintenance and improvement lending. If this is more widely implemented and supported with a fund injection through Union Bank, key informants suggested that the lending for sanitation and home improvements more generally will likely increase dramatically. Recognizing the need for affordable household financing for WASH, FMOH is working with commercial lending institutions to establish a class of bank loans targeting household sanitation, similar to what was done to increase the availability of credit for fertilizer purchases (GOE FMOH, 2022). It is worth tapping Dire Dawa Wholesaler (Picture credit: Muhammed Ebrahim Mussa) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 17 into the experience of the FINISH Mondial project led by WASTE, Amref, and Bole Baptist Bible Church in Ethiopia (see https://finishmondial.org/countries/finish-in-ethiopia/) that is implementing a similar approach with notable success in Kenya. According to the Ethio-MBS newsletter (GOE FMOH, 2022), “Water.org also provides a good example of advancing this issue. Their WaterCredit program in Ethiopia has successfully worked with several MFIs to develop affordable household loans for WASH product purchases. The loans have proven very popular; in less than one year, these MFIs disbursed loans totaling nearly ETB 19 million (US$ 365k), reaching around 22,500 individuals.” While an important development, it is unclear how much of this lending was for sanitation purchases. The Vision Fund provides another emerging example. All of its 99 branches accept sanitation-related applications with the interest rate set at 14.6 percent per year, as determined by the National Bank of Ethiopia. Payback has been recorded at 91 percent with a 6.5 percent default rate. A key challenge noted by interviewees is the repayment modality. For agriculture-related loans, payments can be scheduled in accordance with the harvest season. For “consumption loans” like for sanitation investments, the expectation is that payments would begin a month after the loan has been recorded, even if income is seasonal. That said, according to the Vision Fund interviewee, solar loans used to be the same as WASH loans, but external funding allowed the organization to amend the repayment modalities. He further noted that “…there is some experience in being flexible in terms of revising payment rates, installment rates, and interest rates …” The Vision Fund has also established processes to make sure that sanitation loans are not used for unrelated purposes. It works closely with HEWs to verify that pits have been dug before releasing the money. To date, the Vision Fund has made over ETB 60 million (US$1.15M) in WASH-related loans – this is still only reaching a small segment of their active clients, but certainly a sizeable and growing financial resource for households. Recognizing that households wanted to borrow for a wide range of home improvement projects, as part of their experiment, the Vision Fund reduced the requirement around the 80-20 rule whereby 80 percent of the loan had to be for WASH-related investments and 20 percent could be for other home improvements. As part of this relaxation of the rules, they are now seeing more of an emphasis by households to combine WASH and energy loans. Given Transform WASH’s emerging interest in similar activities particularly for farm-based income earners, this combination could prove helpful in moving the needle further with regard to sanitation lending.8 Beyond the commercial and larger national MFIs, the project has missed opportunities to work more closely with instruments at the local level, like the Somali MFI. Almost 95 percent of rural Somalis are Muslims and require interest-free banking services. The Somali MFI has experience providing interest free loans to households (as well as small-scale enterprises) in the form of procuring and providing the product directly and then offering an extended term of repayment by households. While this performance evaluation was not designed to go into significant depth on this issue and cannot vouch for the effectiveness of these models and approaches, MFIs, cooperatives, and even commercial banks are keen to increase their share of the market by capturing households who are not yet part of the formal banking system. Transform WASH has been busy seeking to understand and determine how best to develop the market both from a demand activation and supply side angle. While Transform 8 Consortium partner SNV has a plan to increase the link between MBS and its biogas project, with 600 households already having installed SATO latrines connected to biogas producing equipment. The Evaluation Team observed this in Tijit Kebele (Guzamen Woreda, Amhara Region) where, with direct support from SNV through a pilot project, all 168 households connected their SATO pit latrines to biogas structures. As noted by Transform WASH in the Reflections Workshop, “an integrated approach appears to have expanded results and satisfied consumer demand for sanitation, energy, and agricultural options (e.g., organic rather than chemical fertilizer).” The evaluation team agrees that this intervention shows promise and could be explored further. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 18 WASH produced a detailed learning note on sanitation financing for both businesses and consumers, this more comprehensive exploration of how best to activate financing options appears to have been a gap in later stage programming. This gap is partially explained by the relatively nascent nature of such lending for sanitation, the perceived bureaucracy of lending institutions, and different views on how to structure the products and what requirements (in the form of collateral or otherwise) should be put in place. Albeit from the supply side, low loan repayment rates by sanitation enterprises for some lenders (e.g., 62 percent in Oromia, 57 percent in Sidama, and 49 percent in SNNPR, from earlier phases of the activity) resulted in some caution around these financial instruments and perhaps Transform WASH’s approach.9 Particularly with a subsidy protocol recently issued by the GOE, greater recognition of the financing bottlenecks across the sector, and more institutions experimenting in this space, there is significant scope to explore financing options in more depth going forward. Options could include social/development impact bonds, loan guarantee funds through design of approaches akin to the USAID Microfinance Development Credit Authority, and revolving funds that operate at the woreda level. The MBS newsletter also suggests the promotion of “an aspirational percentage of funding by Development Bank of Ethiopia (DBE)’s Rural Finance Intermediation Program to target financial institutions for provision of low￾interest, sanitation-oriented loans” (GOE FMOH, 2022). Again, market segmentation (working with Category 2 consumers) will be the likely focus for such instruments.10 Reaching those who are unable to pay – For the third category of consumer, Transform WASH has experimented by incorporating sanitation loans into the VSLA approach in nine woredas across four regions. Over 450 VSLAs with more than 8,900 household members have received initial support, of which almost 3,900 (roughly 43 percent) have taken sanitation loans to date, repaid through continued monthly contributions. In the absence of other channels of consumer finance or a clear government strategy to increase access through targeted subsidies,11 VSLAs have proven critical to giving poorer households the opportunity to invest. This involves community members saving money on a monthly basis that is then provided to members generally at one loan per month. While an important instrument, more evidence is needed to determine whether the VSLA approach meaningfully accelerates and scales the market. The key challenge for VSLAs is that the amounts of money in the system are too small to allow a quick scale up and may require several borrowings per household to complete the process of installing an improved latrine.12 In other words, in a first VSLA borrowing, a household is likely to buy the substructure and flooring products, a second round is needed to buy the roof and superstructure, and a third round then pays for the installation. Again, while this may be a sound investment strategy for poorer consumers, it may not provide the pace and scale needed by businesspeople to invest in the market. 9 A key challenge for MFIs that are linked to nongovernmental organizations (NGOs) is the community perception that NGOs will underwrite loans and, thereby, do not necessarily need to be repaid. According to the Vision Fund representative, recent efforts to distance these institutions from both NGOs and governments have led to higher repayment rates. Further analysis is needed as a full picture of repayment across all regions was not available. 10 There is an increasing sense that mobile banking will become a game changer, reshaping consumer finance and household savings and financial transfers across the country. This would allow for easier access and repayments, making rural customers “more bankable” with quicker response times from lenders and without requiring the borrower to make multiple trips to the financial institution to set up the loan. Mobile banking can also provide for smaller, more frequent payments that might be easier for poorer households to manage. 11 Although recently issued, implementation mechanisms for the subsidies protocol have not yet been determined. 12 This approach requires an average of 20 members with monthly contributions of ETB 20–30 (US$ 0.38-0.58). FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 19 While it has worked to generate community capital for household investment in sanitation, the VSLA approach can only reach scale with government programming and budget reserved for this activity. The Revised National MBS Guidelines note that a wide range of development partners are also deploying VSLAs or similar models for households to access basic sanitation products at affordable prices. USAID should support further study of these more local approaches to understand what has worked well in terms of reaching target consumers, repayment rates, and the scalability of such models and how outside investment could best leverage further scale. It may also require some acknowledgement that markets may never reach poor remote rural areas without significant subsidy. Similarly, as noted in the USAID/WASHPaLS Scaling MBS document referring to work done by IRC and Centre Régional pour l’Eau Potable et l’Assainissement à faible coût published in 2006, such approaches may also require exploration and understanding of the land tenure constraints for poorer households to invest (USAID, 2018).13 Interviewees generally viewed it unlikely that the VSLA approach will be strengthened for sanitation in the absence of a development partner driving it. Government partners would need the knowledge, funds, and mandate to continue to expand the methodology and Transform WASH is working specifically on encouraging government programming and reservation of budget for these activities. The hope is that regions and woredas will see the value in reaching poor households in this way and will thus take on the community training activities required to expand such initiatives. That said, in Sidama and SNNP, 13 additional VSLAs have been created directly by sanitation entrepreneurs, but this requires initiative from local non￾government stakeholders. Subsidies for sanitation investments are a nascent space in the Ethiopian context. As mentioned, the Sanitation Subsidy Protocol was recently launched, providing guidance on how such a scheme should be developed and rolled out. A key aspect is around targeting, likely through the Federal Ministry of Agriculture’s (FMOA) Productive Safety Net Program (PSNP), which is expected to reach over nine million food-insecure people over the next five years (US Embassy in Ethiopia, 2021). The Evaluation Team understands that discussions are underway between the FMOH, development partners, and the FMOA to determine how best to promote sanitation through the PSNP program (GOE FMOH, 2022). (Otherwise, establishing new identification and verification systems proves an expensive proposition.) In response, in recent months Transform WASH has been piloting a subsidy scheme in Oromia and Sidama. The targeting, as defined in the protocol, is based on whether the household is exempted from paying health insurance charges and has access only to lower levels of service – including no latrine or latrine but no slab or latrine with poorly constructed slab. A 25 percent initial contribution is made by 13 “Another area where the lack of local political commitment was demonstrated is on the issue of land ownership. In Ethiopia, individuals must possess the land they want to construct a latrine on. Tenants or short-term residents are unable to build latrines on their own and would be liable if landlords take legal action for the defacement of property. Until such time comes where the legislation favors non-landowners in the community, the national policy and sanitation strategy should consider the promotion of suitable latrine options based on societal circumstances, including land ownership” (Tamene & Afework, 2021). Considerations for a sanitation subsidy for vulnerable HHs • Identify specific population group (screen using the eligibility criteria including geological challenges) • Clarify latrine component to be subsidized: sub-structure, slab, or superstructure • Determine form of subsidy (cash or in￾kind) • Determine timing of subsidy (product acquisition, after verification, etc.) • Identify channel of subsidy (i.e., whether through the household or the enterprise) • Determine amount of subsidy (fixed or percentage of the total, or partial cost of the sanitation product and/or services). (Adapted from Revised National MBS Guidelines, 2020) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 20 the household and then the remaining 75 percent is covered by the subsidy. The analysis is pending but this will provide an important input to FMOH and the sector more broadly. Transform WASH has been active in advocating for increased financing options for consumers, particularly at the national level. Transform WASH staff have been instrumental in supporting the government in drafting both the National MBS Guidelines, which provides sufficient detail at this stage about the roles that finance institutions could play, and also the Sanitation Subsidy Protocol. It is understood that public finance is and shall remain quite limited, particularly at the woreda level, to support such activities. Transform WASH colleagues acknowledge that consumer finance and partnerships with finance providers could have featured more effectively and strategically in the Transform WASH activity. Consistency of approach – The public sector/community approach is aimed at poorer households to shift them from open defecation up the sanitation ladder to fixed and then basic latrines (noted in Figure 6 as the Traditional Target Group). This is certainly appropriate for the third and perhaps parts of the second segments of the market. The private sector starting point is generally those who can afford it, aimed at reaching scale quickly (the MBS Primary Target Group). The starting point for the private sector strategy appears to be aimed more at reaching scale quickly and thus focused on the first segment and parts of the second. The tactics and tools used differ across the two approaches. The emergence of the business models (discussed below) begins to flesh this out. By Plan International’s own reckoning, the sales conversion rates from messaging to sale and then sale to installation using more traditional SBCC tools have been less effective than the sharper, more innovative private sector approaches. While sales agents receive a commission for every sale, HEWs see the sanitation messaging as one part of an 18-component package of health messaging relayed at the household level. It would be helpful to understand more about the customer category in relation to the demand creation approach used, and whether buyers are moving from open defecation or had been using unimproved latrines. Indeed, different dynamics need to be acknowledged in that sales agents receive a commission for every sale whereas HEWs see the sanitation messaging as one part of an 18-component package of health messaging relayed at the household level. Also, in practical terms, the HEW has no direct incentive to move from demand generation to demand activation to demand fulfilment. A further challenge is the geographic area expected to be covered by HEWs with two HEWs suggested per health post with each supporting over 500 households with limited transport capacity and budget. While it does not resolve the incentive challenge for HEWs and proves quite labor intensive as an approach, there is anecdotal evidence that HEWs and sales agents going to households together generates trust around the value of investing, the price points, and the expected quality of the installation. Transform WASH’s own dashboard confirms this. Conclusions – In general, the Evaluation Team found that the different public/community and private sector strategies to generate demand have not always walked in lock step, but perhaps they need not. The Evaluation Team notes that more effort (and budget) went into understanding and building the supply side of the market as opposed to developing the demand side. As a result, and understandably as an MBS initiative, Transform WASH is widely perceived to be “more focused on the commercial part” of the equation. Numerous interviewees noted that more concerted effort towards “a joint action plan and integration among consortium organizations and… a clearer understanding of the project approaches” may have better linked the different demand creation activities and also supported an understanding of when to sequence activities on demand creation once the supply chain was more able to respond. (See Section 3.3 below for more on consortium effectiveness) “All kebeles in Biyo Awale woreda are supported by the Safety Net program. HHs are poor though with little capacity to build improved latrines, so they are expecting PSI to provide the plastic slab for free.” (Dire Dawa Microfinance Operation Department Head) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 21 Figure 6: Sanitation Ladder Unlike many development actor activities, Transform WASH is not directly delivering products and services at the household level, but working with and through service providers and interlocutors at along the value chain. The consortium is thus reliant on the message being consistent across a wide range of actors who may be operating from different priorities on moving households from open defecation to open defecation free, delivering the sale of products, accommodating cash-strapped households who see the price of materials increasing dramatically, etc. This is also in a context of very high turnover of government officials at all levels (suggested to be 25-30 percent for HEWs) and instability and insecurity (as a function of both conflict and drought) in several project geographies. With these challenges noted, Transform WASH has put in place a monitoring protocol to understand who has purchased and installed sanitation products and to what level of quality. This could help to overcome the inconsistent collection and recording of data at the local level, which in turn will help sharpen demand creation activities. Thus, various factors have influenced the Transform WASH activity from a demand creation perspective, particularly in relation to conversion of customer interest to sales. Such factors include the lack of sufficient staff dedicated to sanitation at the woreda level to support demand creation and a lack of access to vehicles to monitor demand creation interventions and to support the VSLAs. There has also been a steep learning curve among the activity implementers in understanding and working through a commercial marketing approach. The activity has operated in environments where there is an expectation of free product distribution, many households are strapped for cash, and there is an absence of consumer lending and a targeted subsidy scheme for sanitation home improvements. While VSLAs have made an important contribution, given the relatively small amount of funds involved, this approach does not aggregate demand at a scale that is appealing to the private sector. Indeed, while the VSLA approach was largely designed as a pilot activity, there is not at present a clear government counterpart prepared to take this activity forward and scale it. That said, some sanitation product manufacturers (in FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 22 Sidama and SNNP regions) are continuing to form additional VSLAs in other kebeles and creating demand among VSLA members and selling concrete slabs and retrofitting SATO pans. In terms of sustainability of demand creation activities, the GOE is keen to scale up access with policies and supporting documents in place (in many instances developed with significant support from Transform WASH). Public health officials have the responsibility to continue to advance latrine coverage, but are stretched with a heavy mandate and limited resources (and no specific sanitation budget line at the local level) to drive demand activities. Strong coordination across the sector is still needed as well as robust M&E to feed into future design of demand creation activities. Towards this end, Transform WASH has generated a more structured conversation amongst regional, zonal, woreda and kebele healthcare professionals and provided an effective tool for HEWs through the DQ technique for house-to-house visits. The market is far from saturated, and households are keen to gain access to sanitation products and services. Interviewees noted that with sufficient scale and some perception of potential market take-off, the private sector is most likely to drive demand creation activities to ensure there is progression across the demand generation, activation, and fulfilment spectrum. They may still need support (and clearer evidence) in terms of sales techniques that best promote sanitation products and services. Integration between healthcare officials (HEWs primarily) and private sector actors and engaging with financial institutions and addressing financial mechanisms (like subsidies, loans, taxation, etc.) will be critical going forward. 3.2.2 EQ2: MARKET-BASED SANITATION RESULTS How effective have the Transform WASH approaches to market-based sanitation been at contributing to improving sanitation uptake and thereby sanitation outcomes more broadly? The GOE recognizes that to achieve its ambitious goals around sanitation, the private sector must play an important contribution and has been supportive of MBS projects like Transform WASH to enable and strengthen a private sector response. As discussed under EQ3, these efforts have been advanced through the development of the Revised National MBS Guidelines issued in 2020 by the FMOH in collaboration with development partners. Transform WASH played an important role in providing evidence and experience to revise the MBS guidelines originally launched in 2013 and then to advancing their implementation. The Evaluation Team notes that Transform WASH has made significant contributions at all levels from policy to enterprise development to the household level, including the introduction of new technologies. While Transform WASH’s contributions have been significant, the Evaluation Team notes the following provisos and caveats to the analysis: • Transform WASH has not been involved in direct implementation of MBS activities, i.e., Transform WASH does not sell and install directly at the household level but rather supports sanitation business partners to initiate/expand their sanitation-related business operations. The activity was established as a learning program, designed to test how best to support MBS approaches in Ethiopia over a five-year period. Performance Evaluation Data Sources for EQ2 Sources of information for this MBS line of inquiry included project documents, especially sales data, promotion, and result reports, as well as the evaluation team’s own data collection efforts through KIIs with relevant national-level stakeholders and woreda-level FGDs and KIIs with sanitation-based enterprises, representatives of financing institutions, HEWs, and local opinion leaders. The Household Survey (see Annex D) also provided an important source of information around perceived ease of access to products and services (including repairs) and to gauge market penetration of MBS-based products and services. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 23 • The USAID/WASHPaLS global assessment based on the literature around scaling MBS suggests that, through an analysis of year-on-year sales (based on data availability), sales are only expected to rise four to five years after the initiation of the MBS activity (USAID, 2018). While Transform WASH has seen significant growth in sales with more people gaining access, particularly in years 3-5,14 factors beyond the activity’s control, like the Coronavirus Disease 2019 (COVID-19) pandemic, sweeping political change, widespread conflict and war, and the rising cost of materials and labor, has no doubt had an impact on the activity’s more recent efforts to scale. The USAID WASHPaLS MBS Framework unpacks two key components of a viable sanitation market system: the enterprise and the entrepreneur. The enterprise requires a “critical mass” of a clear target market of customers to serve profitably, and thus (cost) effective sales and marketing are needed to generate and then convert demand into sales. Product systems (technologies) need to be acceptable to customers and suppliers. The supply chain needs to be run in such a way as to ensure that customer orders can be met efficiently. The entrepreneur needs to see sufficient profitability to make investing financial and other resources worthwhile and have access to sufficient capital to expand the business as desired. As explored in the sections below, Transform WASH has indeed sought to address each of these areas. In general, the Evaluation Team would agree with the Transform WASH partner statement that, Transform WASH has “practically demonstrated market-based sanitation as an approach and increased government confidence, commitment, and buy-in to scale up into other woredas.... [as a result] tens of thousands of households have got access to and utilization of improved sanitation products both in project intervention and non-intervention kebeles.” The supply-side ecosystem graphic (Figure 7) suggests, there are two channels to reach households with products and services: one through the public sector (noted in grey) which has been a more traditional approach for some development partners, and the other through a private sector supply chain (noted in red). The public sector approach channels guidance and assistance in accessing sanitation products and services through the health system, with the connection at the household level mainly through HEWs. While having some impact on the market-based approach as discussed below, the public sector channel has not been a primary focus of the Transform WASH activity and thereby its general effectiveness is not a subject of this evaluation. As Figure 7 shows, a series of private sector actors are involved in the sanitation supply chain including the importers/manufacturers of products like the SATO pan and AIM slab, the wholesalers who make 14 Using an average household size of 4.6 and accounting for some slight regional differences, “people gaining access to basic sanitation” as reported by Transform WASH totalled 66 in Fiscal Year (FY) 2017, 2,325 in FY2018, 9,635 in FY2019, 25,581 in FY2020, 32,891 in FY2021, and 47,910 in FY2022 (as of August 2022). Figure 7: Supply Side Sanitation Ecosystem FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 24 these available through a distribution channel to distributors or directly to retailers and NGOs, then the slab manufacturers and mason/installers. Along with HEWs, sales agents serve as an interface between households and retailers, slab manufacturers and mason installers. These different actors form the basis of three somewhat overlapping Transform WASH business models that provide installation or upgrading options for different consumer segments. A fourth business model centers on retailers who offer plastic product sales and support to households who will then self-install (see Figure 8). In terms of the installation and upgrading approaches, the Advanced Product Manufacturing and Construction (“Advanced Manufacturers” or “Contractors”) business model largely targets the first customer segment, i.e., those who can afford to invest in the upgrading or new construction of an improved toilet facility. This service includes the superstructure, pit lining, and related components combined with a circular or rectangular slab with SATO pan or lid. The Basic Product Manufacturing and Installation business model (“Basic Manufacturers”) is aimed largely at upgrading with a small precast reinforced slab with SATO pan. The third business model, Door-to-Door Simple Upgrades (“Mason/Installers”), involves the sale and installation of a SATO pan with concrete plastering of the floor (referred to as “skirting” if installed on wood/mud latrine floor, or “retrofit” if on a concrete slab). Figure 8: Breakdown of Business Models (PSI/Ethiopia, 2022) Transform WASH has analyzed and documented these four approaches well. This has helped PSI business facilitators to explain to potential new entrant entrepreneurs what is required of each level in terms of financial investment for space and equipment as well as skill levels to offer a particular type of service. Transform WASH also has estimated what entrepreneurs can expect in terms of average monthly profitability based on projections from a set number of sales.15 The skill levels, space, and 15 Transform WASH has calculated the different profit expectations of the four business models. The enterprise is expected to have an average monthly net profit of ETB 15,548 (US$299) for selling 200 products under the advanced product manufacturing & construction model. The enterprise is expected to have an average monthly net profit of ETB 3,162 (US$61) for selling 70 SATO pan and precast slabs in the basic product manufacturing and installation business model. The mason/installer is expected to have an average monthly net profit of up to ETB 4,500 (US$87) for installing improved latrines at 50 HHs in the door-to-door simple upgrade business model. For plastic products sales, the retailer is expected to have an average monthly net profit of ETB 4,600 (US$88) for selling 50 AIM and ETB 5,250 (US$101) for selling 150 SATO pans. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 25 equipment of providers of the more advanced model allows them to tailor their offering to meet the needs of customers looking for a lower end service. Transform WASH has also worked with retailers to develop their approach to reaching customers by deploying sales agents, marketing and promotions, and other strategies, like linking more closely with the HEWs, who present a trusted source of information for households.16 (The assumption is that the HEWs are not working on a sales commission from the businesses.) Starting with more experienced businesses and then engaging with start￾up entrepreneurs, Transform WASH has worked with over 530 business partners, many already established, offering construction services, or are retailing construction and home improvement products. (The Transform WASH dashboard stored on its own DHIS2 platform and mWater compiles an impressive and important range of data on key aspects of these businesses.) To introduce them to sanitation and strengthen these partners, Transform WASH has offered a wide range of support including technical and business training, coaching, marketing and promotional materials, registration, data management, and connections to finance. Specific courses have been provided on: 1) the basics of entrepreneurship; 2) business planning, management and preparation of business plans; 3) customer handling in sanitation businesses; 4) how to lead markets in your territory; 5) market assessment and customer classification; and 6) financial management, recording and profit calculation. The activity also has worked with differing degrees of success with TVETs and Job Creation Centers to train entrepreneurs and sales agents across the regions. Interviews with TVET officials revealed challenges around how best to attract and recruit participants, how to incorporate quite specific courses into wider modules, and how best to connect students to business operators. EQ2 – Key Findings. Of the 530+ business partners with which Transform WASH has worked, over 430 remain active sanitation product and service providers. The vast majority of these (nearly 300) are mason / installers, around 120 are retailers, and a handful are distributors. (Mason /installers may also be slab manufacturers.) Notwithstanding much appreciated support from Transform WASH, the operating environment has proven a challenge for some enterprises. Interviewees noted that some have stopped largely as a function of not being able to develop a clear target market or marketing strategy, the inability to invest to grow the business, or due to the rising costs of operating and procuring materials, which has reduced profit margins or even made breaking even challenging. 16 It is interesting to note this finding from Transform WASH’s own business model analysis from June 2022: “Almost 20 percent of customers report having procured the products from a health extension worker. This is surprising because the HEWs were not expected to sell products directly but rather to refer customers to business partners.” Factors that limit new entrants to sanitation market • Lack of access to start-up capital, manufacturing and sales premises • Escalation of costs for raw materials • Weak support from regional Enterprise Development Office • Underdeveloped marketing/demand creation and business management skills • Slow moving nature of the business associated with latent demand but limited cashflow • Sluggish sales during non-harvest seasons of the year • Lack of prior experience in similar business • Entrepreneurs or their employees drop out due to inadequate revenue generated from a standalone sanitation business, which does not provide enough revenue for survival or growth, and preference of graduates for a permanent job in the public sector • High profit expectation and financial return within short period of time. (GOE FMOH, 2020; National MBS Guidelines, 2020 – Slightly modified) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 26 The Evaluation Team interviewed around 20 active (and one currently inactive) business partners17 and their feedback was generally very positive on the support received from Transform WASH. One area that emerged in several interviews was an expectation that Transform WASH would provide more marketing and promotional support, either in the form of materials or advertising strategies. Transform WASH sees its role as articulating the skillsets and mindsets needed to advance a sanitation￾related business, rather than taking on these more operational roles and creating dependence around material giveaways, “which has been a source of friction with other development partner MBS approaches,” according to Transform WASH interviewees. As a result of these expectations not being met, some business partners will step up and others may drop out. Developing effective sales and marketing approaches – Transform WASH supported businesses in developing the sales agent model. This sees “door-to-door promotion through a “decision intelligence” (DQ), problem-led sales approach” (PSI/Ethiopia, 2022). The Evaluation Team notes and, as a result of inquiry at the local level, does not question Transform WASH’s own analysis that: “implementation of the decision intelligence (problem-led) sales approach resulted in a remarkable increase in monthly sanitation product sales, which continued to be maintained into the ‘low’ season (post-harvest). From February to March 2021, total sales increased by more than 1,400, a 90 percent increase in one month” (PSI/Ethiopia, 2022). Thus, door-to-door sales of simple upgrades (business model 3 – mason/installers) have expanded significantly in the later years of the project as a mechanism to reach customers. The Evaluation Team recognizes the promise of this approach, particularly as businesses themselves, rather than Transform WASH or other external parties, are deploying sales agents directly through a commission arrangement. Given the experimentation that has taken place with this approach, Transform WASH now presumably understands which markets are most likely to take off and under what circumstances or criteria. Given the three customer segments noted in response to EQ1,18 Transform WASH has created the set of business models to support entrepreneurs to respond to different kinds of customers. The Evaluation Team questions whether there is more to tease out from the Transform WASH data around which are the easiest markets to reach and how. For example, the Advanced and Basic manufacturing models target customers who are “already convinced about the need and have planned a purchase.” It would be helpful to understand how and when sales agents target households that are less convinced or are further behind in their planning. There are also aspects to explore in terms of expectations around margins. Retailers may allow for lower margins for stock that sells quickly but may expect higher margins or discontinue 17 It is unclear if inactive business partners will return to the market. 18 I.e., those who can afford sanitation products and services, those who need some intermediate support due to intermittent cashflow, and those who are unlikely to be able to afford the products and services any time soon. SANITATION MARKET Target Market Delivery Model Product System Sales & Marketing ENTERPRISE EQ2 Key Findings Significant progress has been made on MBS under challenging circumstances in relation to conflict in the country, conflict abroad pushing prices up, as well as the COVID-19 pandemic. Emerging Transform WASH thinking around MBS business models will be critical in guiding business partners and the sector generally. These can be refined further integrating demand creation approaches, market segmentation, and financing options, and work is now needed to understand how best to speed up the process. Further work is needed to develop domestic producers of sanitation products. More attention is needed to business finance options. A trust-based cashflow system across the supply chain will need revisiting as the system expands. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 27 selling the product if sales are slow. (This may explain the relatively lower business partner satisfaction rates for retailers as shown in the Transform WASH dashboard and as based on interviews.) As discussed in the above, there appears to be some work still to do to align the demand creation approaches; the market segmentation; the work on finance, pricing and subsidy design and targeting; and related aspects to refine the business models further. While the approaches are clear, there may still be some gaps in understanding how best to speed up the process. KIIs with retailers conveyed their belief that greater emphasis on marketing and promotions might help. As part of its exit strategy but also as part of investing in their own business interests, Transform WASH is keen for retailers and mason / installers to absorb marketing and promotion (and other aspects) directly. Based on the KIIs at the woreda and kebele level, there is still some way to go in terms of equipping businesses to be more self￾standing. A further complication likely to be impacting actual versus expected sales is the free distribution of SATO pans (nearly 5,000 in Dire Dawa alone), delivered through a development partner to Regional Health Bureaus. This has an impact on the market by dissuading households from buying the products, and, as was the case in Somali Region, causing a wholesaler to stop selling the product, even though he was holding 1,000 SATO pans in stock. The design of a well-considered strategy for distribution of free products to avoid a negative impact on the existing sanitation market is being supported by Transform WASH going forward. Product systems (technologies and processes) – With keen attention to price points and price elasticity, the response from the market has been positive to the range of products and service packages introduced by Transform WASH. Suppliers and customers appreciate the products for their ease of transport, price point, and effectiveness at reducing odor and flies. Transform WASH notes that “‘skirting’ and retrofits are very attractive low-cost options to upgrade traditional latrines to improved status and better consumer experiences” (PSI/Ethiopia, 2022). A further challenge has been access to imported molds to manufacture plastic sanitation products in the country. As noted in the Ethiopia Tax Exemptions Report, foreign technology providers are “reluctant to provide molds to third-party domestic manufacturers because mechanisms to remit royalties or licensing fees from Ethiopia were unclear” (USAID, 2021). The Evaluation Team interviewed one plastics manufacturer who was unaware of the market for sanitation products but was intrigued by the potential business opportunity if sufficient demand could be guaranteed. The company was willing to explore a costed business plan and may have the machinery to produce the SATO pan or similar types of products if a mold can be procured. The representative noted that procuring the raw materials from the Middle East was less of an issue (even if foreign exchange is difficult to come by for such purchases). Manufacturing in the country would help keep prices down by removing the need for foreign exchange to purchase and import the product, payment of import taxes, and remove the need to repatriate profits. Recognizing the importance, Transform WASH has put much energy and effort into this aspect and is facilitating local manufacture of plastic slabs by Silafrica. A primary aspect that needs attention is to ensure that the supply chain from importers to distributors to retail sales is run in such a way as to ensure that customer orders can be met efficiently. In terms of logistics for distribution, Transform WASH has been working with Silafrica to establish a more conveniently located wholesale outlet in Merkato. Transform WASH’s own analysis suggests that, SATO Retailer at Mille (Picture credit: Muhammed Ebrahim Mussa) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 28 “The supply chains established by Transform WASH have proven to be sustainable and resilient. Regional distributors have proven this through regular procurement of WASH products, and some retailers have entered the wholesale business, selling to other nearby retailers. Furthermore, business partners are using their own funds to further distribute and promote the products” PSI/Ethiopia, 2022). As evidence, the Evaluation Team observed stocks of products and un-installed new concrete slabs being prepared for sale indicating an allocation of own resources by distributers and retailers. Wholesalers and retailers, however, expressed some challenge with the lack of access to credit. With relatively low product turnover, a trust-based cash flow system can and generally seems to be working up and down the supply chain at present. This suggests that the outer rings of the private sector ecosystem (as shown in Figure 7) can afford to on-lend the product to the inner parts of the supply chain. This may likely change as the system expands. A further consideration is that the rising costs of inputs may be discouraging customers and suppliers alike. The Revised National MBS Guidelines note this conundrum quite clearly, stating that: “construction and raw material cost escalation has forced [enterprises] to suspend production and sales. It also resulted in an increase of unit costs of products, which in turn affected affordability” (GOE FMOH, 2020). One way to overcome the supply challenge may be to organize slab manufacturers into associations that can then buy raw materials in greater quantities hopefully at lower prices. The Evaluation Team notes that other development actors have been experimenting with this modality with some success. By their own admission, Transform WASH consortium interviewees noted, however, that they were more focused on encouraging and capacitating the business partners and building the supply chains with less attention paid to financing. As the sale of products increases, there will then be increasing need to develop more formalized credit schemes either within the supply chain or through externally developed credit lines. Transform WASH has experimented with a small revolving fund for lending to business partners through MFIs. However, loan disbursement to sanitation businesses in the Oromia region, for example, was only at 27 percent due to Sinke Bank, the loan administrator’s, “bureaucratic processes” and, perhaps more importantly, the need for loan collateral. Thus, the track record of the relatively small number of sanitation businesses given access to a loan for business expansion through grant agreements channeled through cooperatives and MFIs needs greater scrutiny. (At the time of drafting this report, the Evaluation Team understands that such lending in SNNP and Sidama has a 41 percent default rate. In Kalu, Amhara Region, of the ETB 40,000 (US$769) allocated for enterprise loans, ETB 35,000 (US$673) had been disbursed but only ETB 18,000 (US$346) repaid.) Even though Transform WASH has designated full authority to the lending institution, there does appear to be a misconception by enterprises that the loans are supported by development assistance programs and thereby need not be repaid. Interviewees suggested that a further problem is insufficient follow-up to ensure loan repayment, further reinforcing these perceptions. The Evaluation Team agrees with one development partner’s summary of this situation in stating that there are clear roles for government at all levels to “practically demonstrate its commitment and leadership support to set mechanisms/a system through which entrepreneurs or enterprises can access construction materials … at a reasonable price.” The Evaluation Team would go a step further to suggest that there is also scope for the government to support the design of financial modalities that support sanitation businesses. Transform WASH has played an important role in bringing together suppliers and distributors. As noted in the Rural Sanitation Call to Action’s recent work, “Donor capital channeled as revolving funds or credit default guarantees to financial institutions demonstrated a possible market opportunity for consumer and enterprise sanitation loan products (USAID, 2018; USAID, 2021), but much more evidence is required on successful credit strategies” (USAID et al., 2021). Availability ENTREPRENEUR FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 29 The final element of the MBS framework relates to the entrepreneur directly. The entrepreneur needs to see sufficient profitability that makesinvesting time and money viable and have access to sufficient capital to invest effectively. The USAID WASHPaLS Scaling MBS document notes that “sanitation is rarely viable as a standalone, full-time business, especially in dispersed communities” (USAID, 2018). Indeed, while committed to selling sanitation products, like the SATO pan and AIM slab, all wholesale and retail outlets visited by the Evaluation Team are engaged in other businesses with sanitation product sales as only a relatively small and seasonal part of their operations. The Evaluation Team also interviewed several mason/installers who are increasingly happy with the growing income derived from the sanitation business. This is allowing them to further invest in the business but also to extract income for household expenses. At this level, the training and capacity building activities provided by Transform WASH have helped to develop both the commercial and customer-facing skills needed to be successful (USAID, 2018). Conclusions – Transform WASH has worked across the value chain to understand and enhance the market and enabling environments that facilitate and foster investments by both businesses and consumers alike. Efforts to join up the different component parts are now very much needed (starting with marketing that drives demand, that increases sales turnover in conjunction with lending to suppliers, combined with purchasing consortia/associations to counter the increasing cost of construction materials). There remains a call from stakeholders for Transform WASH to share its evidence and findings more effectively around how best to foster and support the market. The business models provide an ideal starting point for these discussions about what works best, in which geographies and for which customer segments. Ultimately, Transform WASH has only been in operation a short time and so expectations need to be measured. That said, significant progress has been made under challenging circumstances in relation to conflict in the country, conflict abroad pushing prices up, as well as the COVID-19 pandemic. The Evaluation Team has seen thriving, profitable sanitation businesses at the local level that are adapting their business models and product options and services to increase sales, even if not at the pace needed to encourage a significant expansion of the market. An area for further exploration is whether there are any significant differences between the service-based and product-based businesses. The business model analysis is starting to develop this angle. A proxy indicator going forward will be whether new funding sources enter the market, keen to lend to or invest in sanitation businesses. This will require evidence of well-run businesses (using data and record-keeping, an emphasis on customer service, etc.) that are successfully investing their own funds and/or have the potential to apply for funding. Transform WASH has an emerging story to tell in this regard. 3.2.3 EQ3: EFFECTIVENESS OF TRANSFORM WASH GOVERNANCE AND LEARNING ACTIVITIES To what extent has Transform WASH contributed to good governance in the sanitation sector particularly with regard to MBS? “Ethiopia’s political system is known for its traditionally centralized, vertically stratified organization… State representatives at all levels have strong authority and discretionary powers,” which leads to significant influence as well as responsibility for what occurs at the local level (Novotný et al., 2018). Governance for sustainable WASH services, the overarching focus of this EQ, includes all the relationships, mechanisms, processes, and institutions through which stakeholders can negotiate their interests, exercise their rights Performance Evaluation Data Sources for EQ3 Sources of information for this governance and learning line of inquiry included project documents that detail strategy towards the enabling environment, enhancing governance, and fostering learning. Data collection efforts through KIIs with relevant stakeholders (including non-affiliated sanitation sector stakeholders) at national, regional and woreda levels also informed the evaluation team’s understanding of Transform WASH’s contributions in this area. Local government staff and local opinion leaders served as a key source of information to understand perceptions around the robustness of governance and learning mechanisms for the sector. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 30 and obligations, and make decisions for the delivery and provision of services. A supportive enabling environment and good governance have been identified broadly as critical elements of the organization of robust WASH program activities. Strengthening governance and leadership are priority issues of the transformation agenda in the second GOE Health Sector Transformation Plan. The Transform WASH activity is seen as making a critical contribution to achieving these objectives. Transform WASH has dedicated significant energy to governance-related activities aimed at ensuring that the enabling environment is open to and prepared for a viable MBS approach that will contribute to households’ improved access to sanitation services. Such activities focus on ensuring a functional OWNP structure at sub-national level by: 1) promoting and influencing policies and strategies related to sanitation; 2) strengthening government capacity; and 3) supporting learning and sharing of MBS challenges and solutions across the sector. The 2020 Revised National MBS Guidelines acknowledge the need for support in these areas. EQ3 – Key Findings Promoting and influencing government policies and approaches related to sanitation – With support from development partners including Transform WASH, the GOE has developed the overarching policies and guidelines that enable a market-based sanitation approach to support the expansion of coverage of sanitation services across the country. That said, there is a general sense that MBS is still a relatively new approach that needs time to develop. Alongside practical efforts on the ground working with a wide range of stakeholders to advance access to sanitation, Transform WASH is widely appreciated by stakeholders as having contributed significantly to advancing MBS through its advocacy, research efforts, and practical demonstration on aspects related to: 1) tax exemption for sanitation products (with a focus on plastic and fiberglass slabs and pans as well as raw materials to manufacture them in Ethiopia); 2) increased access to sanitation loans for households and enterprises from the loan portfolios of financial institutions (including injection of additional capital by the National Bank of Ethiopia dedicated for sanitation and other WASH loans); and 3) introduction of smart and targeted subsidies financed through a proposed national sanitation fund. Transform WASH’s experimentation and sharing of learning has helped to advance national and regional level conversations around how best to reach customers through market-based approaches. Indeed, Transform WASH has contributed substantially to revisions and subsequent publication in 2020 of the National MBS Guidelines, the Sanitation Subsidy Protocol issued in 2022 (with piloting undertaken by Transform WASH), and revisions to the FMOH national definition for “improved sanitation” to align with global definitions. A key gap, however, is the lack of a dedicated budget line for sanitation in local budgeting processes. Sanitation falls under a general health budget line which is largely understood to be apportioned to HEWs and this affects how priority is given and budget tracking undertaken for sanitation-related activities. With encouragement from Transform WASH through the establishment and strengthening of strong Woreda WASH Steering Committees (WWSCs) and Woreda WASH Technical Teams (WWTTs), some efforts are being made to resolve this issue. EQ3 Key Findings Transform WASH has made significant contributions to the overarching policy framework, enabling environment, and general attitudes and awareness around MBS as an approach. While capacity building efforts and tools introduced have been appreciated, more intentional follow-up has been needed to understand and identify: 1) the impact of these efforts, 2) any challenges faced in applying the learning and tools, and 3) any further gaps that may warrant support from Transform WASH consortium members. Research and learning efforts have been welcomed, but have not yet been seen to have meaningfully reached the sub-national level. The learning strategy could have been clearer and needed to be owned by the entire consortium. This would have better supported the integration of the different consortium member specialisms and contributions. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 31 At the woreda level, the WWTT is a mandated structure composed of government employees. The role of the WWTT is to prepare and manage a Woreda WASH Program, integrating and coordinating the inputs of the Sector Offices and those of participating NGOs. The WWTT has both a coordinating and a management function. The WWT is accountable to the Woreda Council, through the Woreda Cabinet, for the achievement of WASH targeted outputs. Established and chaired by the woreda administration, the WWTT includes the heads of the Woreda Health Office, the Water Resource Development Office, the Education Office, the Women and Child Affairs Office, the Finance and Economic Development Office, and the TVET/Job Creation and Enterprise Development Office. With the support of Transform WASH, 37 of the 41 intervention woredas have established and revitalized the WWTT with Transform WASH-provided training on different WASH-related topics including MBS. Woreda-level structures have been supported by Transform WASH but with somewhat mixed results. As an example, according to numerous interviewees, the woreda five-year strategic plans have not been seen as generally fit for purpose given their length, the perceived likelihood that they will be used effectively, and the fact that they are not aligned with actual expected budgets. As noted, an important contribution from Transform WASH came through the research undertaken on sanitation related taxes, particularly around the tax charged against imported plastic sanitation products. With the Ministry of Finance reluctant to lose revenue from taxation on sanitation products, some preliminary even cursory analysis on the income from such products in relation to the costs to the health service of households that do not have access to sanitation could be instructive and help inform the government’s decision-making process. Initial Transform WASH analyses of the price elasticity of sanitation products, how subsidies could be applied in the sanitation sector, as well as lending modalities for households and sanitation businesses alike have also been of great value to the FMOH. Strengthening government capacity – The Revised National MBS Guidelines (2020) note that “…design and implementation of customized capacity building programs are important for stakeholders at federal ministries, regional bureaus, zonal and district offices, and for service providers including sanitation small and medium enterprises where the capacity building interventions are mainly tailored to their mandates, roles and responsibilities.” Thus, there is wide recognition of the importance of ensuring the right skills (and mindsets) are in place across all levels of government and other stakeholders. Transform WASH consortium partners led capacity building and training efforts to strengthen WASH and MBS teams at all levels. To date, 6,616 people have been trained out of a life of project target of 8,685 (a 76 percent achievement). Alongside capacity building for the private sector, public sector capacity building has focused on leadership and management, procurement management, contract administration, financial management and strengthening OWNP itself. Training of Trainers at the national level has been cascaded down through regional and then woreda offices. The WWSC and WWTT have been structured to facilitate both vertical and horizontal integration of discussions of best practice. Transform WASH in collaboration with the FMOH prepared the comprehensive national MBS training manual and facilitators’ guide for the private sector and worked with the National WASH Coordination Office on a national WASH sector capacity building guideline which is under final stages for government approval. Interviewees confirmed that the wide range of Transform WASH capacity building efforts have been much appreciated by stakeholders. These have allowed the Transform WASH team to bring together sector offices at region, zone, woreda and kebele levels. However, the Evaluation Team observed and heard that capacity building efforts have not always fully translated into action. As an example, several project management training efforts at the woreda level (on planning, monitoring, data management, documentation, etc.) have been delivered. But there is huge inconsistency across woredas in terms of how the tools and techniques have been applied. The Evaluation Team noted data inconsistency regarding sanitation product sales, installation, purchased but not installed, household access to FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 32 improved sanitation and related factors, and a lack of integrated annual planning, project performance reporting and sharing in some woredas visited. A key challenge has been to ensure that the material covered, the skills reinforced, and the tools put in place have become institutionalized, particularly in a context of very high turnover of leadership at the regional and woreda levels. In one intervention woreda, key leadership positions changed three times during the intervention period. This largely means going back to basics to bring new leadership up to speed on the MBS approach, how Transform WASH contributes, and what has been achieved thus far. Appreciation notwithstanding, interviewees also revealed that such efforts should have been more deliberately integrated and sequenced by the consortium with more intentional follow-up to understand and identify: 1) the impact of these efforts; 2) any challenges faced in terms of applying the learning and tools; and 3) any further gaps that may warrant support from Transform WASH consortium members. Related to this, some support to the GOE in determining what kind of staffing is required at all levels would be instrumental in taking MBS forward as an approach. Overall, Transform WASH government capacity building efforts have resulted in a change in officials’ thinking with regards to sanitation broadly and MBS more specifically; decision makers are more aware and attentive to issues of sanitation service provision as evidenced by the inclusion of sanitation more systematically in woreda strategic plans, and regular interaction among decision makers has led to the scaling up of MBS in some non-Transform WASH woredas. Additionally, in alignment with the GOE’s established systems for enabling MBS, Transform WASH partnered with TVETs, including colleges and national and regional offices, and other local institutions for business capacity building. The Labor and Skills Development Office is one of the project memorandum of understanding (MOU) signatories at the woreda level and are part of the WWTT. Although not considered directly within the OWNP structure, this office’s involvement has helped to mobilize budget for MBS activities to establish, train and follow up with sanitation enterprises. Transform WASH facilitated woreda level TVET instructors training that includes installation of stabilized soil-making machines and production of soil cement blocks, basic masonry and construction of superstructures, construction of various sizes of cement rings for pit linings, construction of in-situ and off-site toilet floors, and development of quality standards and safety measures. The TVET training has been further cascaded to masons, sales agents and artisans. The Labor and Skills Development Office supported youth and women to initiate sanitation business activities including development of business plans and linking and facilitating loans for small and medium enterprises from MFIs. While the OWNP structure participates in regional review and multi-stakeholder platform meetings, the coordination structure at national and regional levels has no legal authority (i.e., is not established by proclamation), is thus not a signatory to agreements between stakeholders, and has no strong influence on implementing ministries/bureaus and/or WASH project implementing partners. Individual sector ministries, regional sector bureaus and the WWTTs have the mandate to influence project planning and implementation. Hence, OWNP coordination offices main recourse is to seek to influence WASH￾related decisions through sector ministries and bureaus. Transform WASH recognizes this and thus deploys a three-pronged strategy engaging OWNP, ministries and regional bureaus. Dodola Focus Group Discussion (Picture credit: Desta Dimtse Assefa) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 33 A key instrument introduced by Transform WASH, the Institutional Strengthening Index (ISI) has provided a way for woreda-level decision makers to improve direction, clarity, and transparency in WASH programming. Aimed at tracking changes in government capacity year-on-year, the tool introduced 21 indicators for self-assessment of aspects around planning, coordination, implementation capacity, financing, monitoring, and reporting. Initial use of the ISI was informative with woredas putting a range of elements in place as a result of helpful Transform WASH training. This prompted woredas to consider their approach more holistically. Beyond those early stages, interviewees at the woreda level suggested that the tool became largely for Transform WASH as it has not been embedded effectively in woreda processes. Woredas suggested that feedback was not forthcoming from Transform WASH and the ISI reports are not shared with the Regional Health Bureaus. As a result, it has generally become a tick-box exercise that may not reflect progress year-on-year or serve as the discussion tool it once was. If seen as helpful, discussions should be held with FMOH/National WASH Coordination Office (NWCO) to re-engage them in the ISI tool and to establish ownership beyond the Transform WASH partners, and then, as appropriate, making it a federal level requirement to be used as a tool at all levels of government monitoring and reporting. At that point, it may also be helpful to redesign the tool to be more on a Likert scale with, say, 1-5 responses for each category that showed a progressive realization of the ideal scenario. At present, responses are on a yes-no basis which seems to suggest that many woredas are further along than is perhaps the case. Indeed, 90 percent of the 41 Transform WASH intervention woredas have scored equal to or greater than 75 percent on the ISI, with finance and a few other categories being problematic for all to answer in the positive. A primary gap noted by numerous stakeholders including Transform WASH staff is the need for a more robust management information system that includes more consistent details on sales, installations and other aspects of market intelligence collected at the kebele and woreda levels. The Evaluation Team noted that data from Transform WASH (collected from business partners) did not always align with that collected at the woreda level. Again, this is perhaps understandable as Transform WASH are not direct implementers and are thus reliant on different sources including woreda and kebele-level officials, retailers, and mason installers to relay what is happening on the ground. That said, the Transform WASH data collection efforts are helpfully stored on the activity dashboard in a concise, compelling format for sharing information with government and other institutions in the sector. A further mechanism for enhancing capacity at the region and sub-regional level has been through sub￾grants to support the regions in a range of activities from supplying hand washing facilities, to conducting training at the local level to maintenance for existing water schemes. While the project initially planned to provide direct financial support for interventions at the regional level through a results-based financing (RBF) modality, the plan was not implemented in all regions as intended. This is due primarily to the lack of RBF experience by some regions (combined with low disbursements to regions in conflict). There were also some delays due to administrative and sequencing issues related to the utilization of the CWA budget. Supporting learning and sharing of MBS challenges and solutions across the sector – A key challenge in the sanitation sector is around government and stakeholder coordination. Donors and development partners continue to play a key driving role. Transform WASH has sought to ensure that all important national government actors have been involved in different components of its work with mixed success. Recent years have been challenging with other pressing issues like the conflict in the northern regions and COVID-19 understandably dominating government agendas. This has limited sector efforts like those of Transform WASH to galvanize a national conversation that included non￾traditional actors such as parliamentarians, the “As research and knowledge management are key components of Transform WASH, creating a focal point at the center of the project management team would create more cross-consortium synergies while improving and speeding up project learning.” (Transform WASH Reflections Workshop) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 34 Ministry of Labor and other government agencies more involved in encouraging and strengthening entrepreneurship through the TVET and business associations such as the Ethiopian Chamber of Commerce and the Ethiopian Plastic and Rubber Manufacturers Association, and others. While MBS as a strategy appears to be well entrenched in the system with the MBS guideline, subsidy protocol endorsed and being piloted, basic sanitation definition revised with MBS explicitly noted as a means of access to basic sanitation, further institutionalization will continue to be necessary to convince different stakeholders to make the contributions expected of them as noted in the Revised National MBS Guidelines. Beyond the practical demonstration of MBS approaches and the direct influencing on GOE policy, the Transform WASH activity has expended significant energy in developing research and learning publications and activities on a range of subjects, including SBCC effectiveness, piloting of the subsidy protocol, sanitation taxation, and other topics. While helpful in advancing relevant topics, it has been unclear to the Evaluation Team what the strategy around developing these products has been. Transform WASH partners suggest that the process has admittedly been rather ad hoc with somewhat uncertain impact. Website visits and downloads are helpfully captured but it is not clear who is accessing the materials. (Statistics can be collected on where people are viewing and downloading the documents, how long they are staying on the site, and other aspects.) Interviews at the subnational level suggest that the findings of these research activities could be better communicated. A more concerted learning strategy could have usefully articulated which topics would have benefited from longer-term multi-year action research efforts based on specific hypotheses (around the topics of finance and subsidy, for example) versus the more “quick and dirty” thought pieces aimed at advancing the debates in a more immediate way and bringing new entrants into the sector up to speed quickly. This gap of wider ownership is acknowledged by Transform WASH in the Reflections Workshop held in March 2022.19 Operating under the auspices of the National Hygiene and Environmental Health Task Force in support of the OWNP, a key mechanism for learning and sharing efforts has been through the creation of multi￾stakeholder platforms (MSPs) at the national and regional levels. According to the National MSP Terms of Reference (Government of Ethiopia Ministry of Health, 2018), MSPs play a neutral coordination function to “document and disseminate emerging lessons for evidence-based learning across the sector”, and at the regional level “to promote grassroots civil society and community-based organizations mobilization and participation on related policy debates and program designs with relevant government stakeholders.” Supported by Transform WASH, the national level MSP allowed for significant input into the MBS Guidelines revised in 2020. While recognizing that these dialogues are important to develop the sector, some stakeholders noted that, without a clear target like shaping the MBS Guidelines, a high￾level coordinated ongoing agenda on MBS has waned somewhat. The Transform WASH MSPs have had mixed success at the regional level with most interviewees seeing the ownership and responsibility placed squarely with Transform WASH. KIIs revealed that there is insufficient engagement of enterprises and community-based organizations that affects the impact and continuity of the MSPs at this level. Consistent and continuous participation of stakeholders in the MSPs has been a challenge. MSPs have suffered from leadership turnover in their membership base; 72 percent of all participants in the MSPs were one-time attendees who did not continue their engagement. Thus, without continuous participation, it is difficult to build, maintain, and transfer institutional knowledge relevant to MBS challenges and solutions. Encouraging women’s participation in the MSPs has been noted as needing more attention. 19 As another means of sharing information, Transform WASH developed a curriculum and training materials (MBS modules) and uploaded on the WASH System Academy along with other WASH System Academy courses. IRC piloted the blended course for government staff and launched the online course. The system allows for easy download of documents and tracks registered, unfinished and completed participant coursework. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 35 While action points are established during MSP meetings, follow-up on action plans and consistent reporting of progress among participants has remained a challenge. The Regional Health Bureaus, as chair of the MSPs, should ensure that follow-through occurs and provide any needed support to achieve objectives. As the Transform WASH team acknowledges, the MSP should set mutually agreed, measurable goals and objectives with a monitoring protocol in place to assess progress. With the advent of the COVID-19 pandemic limiting face￾to-face meetings, some shifts in approach were clearly needed. The social media platform, Telegram, provided a useful vehicle for strengthening MSPs by facilitating activity coordination and sharing of knowledge products. Suffice to say that some Telegram groups have been more active than others and, in some cases, woredas have created their own MSPs that appear to be more vibrant, positively reflecting the kind of local ownership that an activity like Transform WASH is keen to engender. In addition to the formation of the MSPs at the regional level for general learning and sharing, Transform WASH has also initiated Joint Supportive Supervision (JSS) activities at the local level. This consists of a sub-woreda level monitoring platform viewed as practical and problem-solving with joint multi￾stakeholder site visits followed by feedback sessions to debate and discuss what was observed and assign responsibilities for resolving issues that were encountered. While the MSPs are viewed as general learning and sharing platforms, the JSS are more experiential with specific agendas helpfully created around a particular challenge. As such, they are appreciated and thus more likely to be sustained after the Transform WASH activity ends. Conclusions – Overall, the Transform WASH activity has been highly influential in advancing policies and GOE instruments to incorporate an MBS approach into sanitation programming. Capacity-building efforts have been very much appreciated at all levels, but there is a sense that they could have been more integrated and sequenced by the different Transform WASH consortium partners with more follow up that ensures these are translated into action. The different mechanisms put in place to advance learning and sharing need to be designed in such a way as to more clearly engender local ownership. While difficult, bringing in relevant non-WASH actors into these activities and conversations will become increasingly critical to address some of the emerging bottlenecks (like sector finance) explained in other parts of this report. The MSPs should involve more relevant actors beyond the line offices of FMOH, such as MFIs and TVETs established and supported as part of the GOE’s efforts to further enable MBS. This would allow for proof of concept and benefit demonstration to actor groups that play a crucial role in MBS approaches. Aleta Wondo Focus Group Discussion (Picture credit: Desta Dimtse Assefa) 3.2.4 EQ4: GENDER EQUALITY AND FEMALE EMPOWERMENT How did Transform WASH consider gender equality and female empowerment in activity design and implementation? EQ4 focused on assessing the relevance, sustainability, effectiveness, and efficiency of Transform WASH implementation from a GEFE perspective. This evaluation investigated GEFE in areas such as engagement of women in sanitation businesses and services, female-headed households’ access to improved/basic sanitation services, participation of women in savings and credit associations/cooperatives, and enhanced women's role in decision-making and governance. The Evaluation Team assessed how the Transform WASH activity incorporated GEFE in the project objectives and considered GEFE in the results framework, and how the activity addressed the needs and interests of women and girls in the project planning, implementing, reporting, monitoring and evaluation practices. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 36 Equitable access to sanitation services is essential in realizing the sexual and reproductive health and rights of women and girls. It frees up more of their time to participate in public sphere activities like self-employment, reduces their vulnerability to gender-based violence, and provides privacy and dignity. It enhances women and girls’ success and increases their voice, agency, and economic empowerment. Promoting women’s participation in the MBS value chain creates an economic and social empowerment opportunity for women. With these aspects in mind, the Evaluation Team identified key GEFE challenges faced by Transform WASH during implementation, as well as lessons learned and recommendations to strengthen GEFE aspects in future projects in the WASH sector. This goes beyond basic rights to sanitation and access of women and girls to sanitation products and services to include evidence of empowerment and women as drivers of change as well. The USAID framework for gender analysis was used to analyze the results (USAID, 2017). While reference is made to the household survey, indeed without a baseline, it is difficult to determine if there have been improvements over time. Ideally then, the household survey findings become the baseline for further study. Performance Evaluation Data Sources for EQ4 Evaluation data was collected through a comprehensive document review followed by FGDs and KIIs. The sources of data included key Transform WASH and other implementing agencies, responsible government offices, private sector partners, and development partners at the federal, regional and woreda levels and beneficiaries of the project at the community level (women, men, girls, boys, people with disabilities, the elderly, etc.). The household survey was carefully designed to capture GEFE-orientated data. The evaluation team members involved in the data collection activities had an orientation and discussion session to ensure all team members shared a common understanding of the data collection tools from a GEFE lens. EQ4 – Key Findings Fostering women’s participation during project implementation – Efforts to ensure the operationalization of GEFE objectives have been analyzed through the lens of laws, policies, regulations, and institutional practices. The Evaluation Team confirmed that Transform WASH made considerable effort to ensure gender equality and female empowerment issues are integrated throughout the overall project design and implementation. During the FGDs and interviews with national-level consortium staff, it was explained that at the launch of the project, with the support of the gender specialist, the technical team and the management team incorporated GEFE in the project results framework. Based on this, indicators for the intermediate level results were outlined to monitor progress of gender specific activities. To ensure GEFE had been systematically integrated in project activities, a training on gender responsive planning, implementation, monitoring, and reporting was provided for key Transform WASH technical implementation staff (at the national and regional levels). On-the-job support and supervision were provided by the gender specialist. In addition, a gender planning and reporting checklist was developed and used as a reference during the planning and reporting process. In collaboration with the GOE National Gender Equality and Social Inclusion (GESI) Task Force, and woreda and kebele level health and Women’s Affairs offices, continuous training was given for woreda level staff who directly support and supervise implementation at the community/beneficiary level. According to a Transform WASH report, a training of trainers on gender in WASH was rolled out for EQ4 Key Findings Transform WASH has made significant commitments to and advances in women’s participation in the sanitation supply chain, a traditionally male dominated vocation. Albeit with regional differences that warrant further analysis, the activity has exceeded the targets agreed of 25 percent female business partners. While further study is needed, VSLAs appear to have been an important mechanism to empower women in terms of decision making on investments in home improvements and more generally to provide economic independence. Transform WASH appears to have had less impact in advancing women’s participation at higher decision-making levels. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 37 72 WASH gender focal persons (31 female and 41 male) and the same training was cascaded to 588 staff (59 women and 529 men) at the WWTT level. Demand creation activities are conducted by HEWs (the majority of whom are women), Women’s Development Army leaders and staff, and sales agents trained on sustainable behavioral change and communication approaches. Although involved at the woreda and kebele levels, one of the major challenges mentioned by Transform WASH staff was the absence of the Ministry of Women and Social Affairs (MWSA) at the national and regional level, including in the National GESI Task Force. This has been understood to be largely a capacity issue for MWSA, with insufficient staffing to cover all incoming requests for engagement. This structurally limits the ability of woreda and kebele level Women’s Affairs staff to work with MBS activities to target women-owned businesses and women beneficiaries with particular support. Although MWSA have a high-level mandate on policy and strategy formulation of gender equality and women’s empowerment in all development sectors, its limited involvement in the WASH sector poses a significant barrier to support, promote and advocate for equitable women’s participation and empowerment in MBS activities and more broadly in WASH/sanitation related activities at the national level. Discussions with stakeholders, including leaders of sanitation product manufacturers as well as a review of VSLA records, confirms that the Transform WASH activity has considered gender issues and women’s empowerment in demand creation and capacity-building interventions. To ensure inclusion of all community members and access to improved sanitation services, the project advanced the VSLA mechanism and created opportunities for low-income households to buy and use improved sanitation services, benefiting women and girls in particular. As an example, in the Oromia region, Transform WASH organized 145 VSLAs in 27 kebeles of four intervention woredas. Of the 2,513 VSLA members encouraged by Transform WASH, women account for 63 percent (1,593). This is consistent with the make-up of women’s participation in VSLAs in other parts of the country. In Amhara, women’s participation in VSLAs is still higher at 72 percent (828 of 1,141 members at the time of the visit to the region). More could be explored by Transform WASH around the gender aspects of membership, i.e., how membership in VSLAs explicitly increases female participation more broadly and how membership influences household decisions to invest in sanitation. Focusing on women as private business partners – In relation to gender roles, responsibilities and time used as well as access to and control over assets and resources, Transform WASH has GEFE-Related Responses from the HHS More than half of all participants from both Transform WASH intervention areas (61.3%) and non￾interventions areas (64.2%) reported that decisions regarding toilet construction was the responsibility of the man. Regarding purchasing of WASH projects, 43.4% of households in intervention areas and 47.1% of households in non-intervention areas reported that husbands and wives made decisions jointly. Fewer reported that the man of the household (23.4% and 21%, respectively) or the woman of the household (31.3% and 31%, respectively) were solely responsible for these decisions. These figures generally align with reported statistics regarding purchasing decisions for all products (WASH products and others) as well. Male-headed households were approximately 33% less likely to own a latrine than female-headed households, on average across both intervention and non￾intervention woredas. However, female-headed households were 54% less likely to have upgraded their toilet than male-headed households. Woman installing SATO Pan (Picture Credit: PSI) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 38 also performed well. It has advocated for GEFE and involved women as business partners at all levels, strategically challenging the traditional gender-based division of labor and ensuring women’s participation in the public sphere. This has translated into providing opportunities in MBS as well as advocating and supporting women to be members of VSLAs to be economically empowered and able to afford to pay for their basic sanitation needs as well as other needs in their household. The Evaluation Team identified that from the design stage, the Transform WASH team agreed on a target of 25 percent of activity beneficiaries to be women and girls, including women as users of improved sanitation as well as MBS partners. Efforts have been made by all consortium members in terms of advocating for women’s participation in the overall MBS value chain, albeit with some important distinctions by role. Women’s participation as sales agents of sanitation products is 85 percent, as manufacturers 14 percent, and as mason installers four percent. SNNP (48 percent), Sidama (36 percent) and Amhara (35 percent) have exceeded the overall target percentage of women’s participation in the supply chain. In Oromia and Somali regions, only 8 and 10 percent of Transform WASH business partners across all categories are women respectively (see Figures 9 and 10). Transform WASH sought to identify women-run construction material retailing businesses and energy saving cooking stove producers to encourage them to expand their business operations to include sanitation products and services.20 Also, particular attention and advisory support has been given to ensure women who continued to work with Transform WASH are sustainably running their business. In terms of access to finance, 50 percent of loans for MBS businesses provided by the Omo Microfinance Institute (OMI) in Mierab Bedewacho Woreda went to women. An FGD in Mierab Bedewacho in SNNP Region identified that out of four sanitation enterprises, two are owned and led by women. Local Transform WASH staff were committed to engaging women and 20 The strategy has been particularly successful in SNNP Region with 2 of 6 retailers, 2 of 10 manufacturers, and 4 of 20 mason/installers being women; and in Sidama Region, with four of 12 manufacturers and three of 15 mason/installers. Figure 9: Percentage of Female Entrepreneurs Supported by Transform WASH Figure 10: Percentage of Male/Female Sales Agents (Not Introduced in Oromia or Somali) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 39 realizing the untapped potential and capacity to lead their own businesses. Transform WASH could analyze this further to provide guidance to other woredas. In addition, discussions with members of VSLAs indicate that the savings and credit associations are not only creating an opportunity for poor households to access improved latrines, but also providing an opportunity to meet the financial needs of women who face cash constraints. They mentioned that they took loans from their respective VSLAs to buy and install improved latrines at very low-interest rates of 10 percent compared to the MFI interest rate of 17 percent and higher. They explained that before they became members of the VSLA, they were entirely economically dependent on their husbands to fulfill all the needs of their family. This situation has changed since they started to save money and take loans from the VSLA to improve their housing, and pay for clothing, school fees and student uniforms, medication and health insurance, purchasing agricultural inputs, and other household requirements.21 The findings of the evaluation in Aleta Wondo Woreda of Sidama Region indicate that women’s participation in the sanitation market is very promising. Three of six sanitation enterprises are led and owned by women, and one of three mason/installers is a woman. It was also noted that 33 percent of loans provided by OMI in Aleta Wondo Woreda were for women in sanitation businesses. In Afar and Somali regions, the absence of female sanitation manufacturers brings the average number of women across all roles (sales agents, manufacturers, mason/installers, retailers) to 14 percent. (FGDs in Somali Region indicated that women are actively discouraged from participating in labor/construction work on cultural and religious grounds. Their traditional dress is also not seen as conducive or comfortable for this kind of work.) Further analysis could be conducted to understand the constraints and to see how they might be overcome. Transform WASH has recently set a target of having at least one female mason/installer in business per woreda. The findings of the action research conducted in 2020 by Transform WASH consortium members22 show that challenges reported by female retailers and sales agents included limited ability to travel and often relying on support from others. Opportunities to socialize and network with other actors across the value chain were also limited, especially for retailers as such activities are often associated with sexual infidelity in the case of women. No male entrepreneurs reported these challenges. Enhancing women’s role in decision-making across the sanitation sector – Analyzed in relation to cultural norms and beliefs and patterns of power and decision-making, strategic Transform WASH interventions have systematically sought to alter social norms that marginalize women in the WASH value chain. Such interventions have challenged the status quo and brought women to decision making power by strengthening their agency and status in their community due to their socio-economic empowerment. As in many countries, the manufacturing/construction sector (related to WASH or otherwise) in Ethiopia is male dominated with participation of women quite limited. The sector is traditionally labelled as blue collar and the associated socio-cultural assumptions represent significant barriers for women’s participation. Some misconceptions and assumed arguments are related to women’s physiological makeup and technical skills. To its credit, Transform WASH has created opportunities for rural women by strategically identifying and encouraging them to enter manufacturing and construction businesses. This is a practice of paradigm shift, showcasing women’s capacity and ability if the enabling environment is supportive. It also has an impact on the cultural bias and perception of the community since women 21 As part of improved sanitation products, Transform WASH has also promoted greater accessibility to improved menstrual health and hygiene (MHH) products through retailers, manufacturers, and sales agents in some regions. MHH key messages were incorporated in SBCC materials used for demand creation activities. 22 Women as Business Leaders, Opportunities, Challenges and Strategic Intervention: Action Research Report 2020 FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 40 involved in MBS activities have demonstrated they can successfully lead profitable construction and manufacturing businesses. KIIs revealed that women are seen as trustworthy and reliable business owners. While this needs to be verified further, stakeholders anecdotally report that women sales agents are more trusted than men and thereby achieve higher conversion rates in terms of sales. Despite numerous challenges that are embedded in the socio-cultural and socio-economic values of the community in relation to gender roles and social norms, the Evaluation Team found that female entrepreneurs and women-owned enterprises have been successful in reaching out to target communities, promoting and selling their products, and convincing their customers to pay for their services (see Case Study: Yeshi Arage’s Plastic Slab Enterprise in text box below). As a starting point, overall percentage of participation by women is promising and if continuous efforts are made, additional results can still be achieved. Overall, the strategies employed by Transform WASH to advance women’s participation and women’s access have been successful. With Oromia and Somali regions as outliers, the target of 25 percent women’s participation in the MBS supply chain has been exceeded with a 28 percent aggregated contribution. This has broken the stereotypes in many communities that have held women back from certain business activities. While these advances are impressive and important, it should be noted that women remain less active in higher-level decision-making forums. Ansha Ahmed, Kalu Messon (Picture credit: Muhammed Ebrahim Mussa) CASE STUDY: YESHI ARAGE’S PLASTIC SLAB ENTERPRISE (RETAILER AT WEREILU, CHISA KEBELE, AMHARA REGION) Yeshi with her three friends (one man and two women) established an enterprise to manufacture smokeless stoves in 2011. At the end of 2011, they were offered training on toilet slab production in Dessie W/ro Sihin College. They started production of concrete slabs and sales of plastic sanitation products in 2012. Yeshi dropped out of school when she was in grade 8 because she was given for marriage. She has six children and has been a single mother since her husband passed away 16 years ago. She took all the responsibilities of supporting her family and raising her children without any support from anyone. She sees herself as a strong woman running her own business to support her children. Currently, all her children are running their own businesses, except her last born 18- year-old daughter who is living with her and supporting her mother in manufacturing and installing smokeless stoves and concrete slab, which she does outside of school hours. Yeshi created a job opportunity for three employees (one of whom is an orphan). Yeshi installed a Sato-fitted latrine first for herself to be a model for the community and started house-to-house promotion to introduce concrete and plastic slabs and create a market for her products. The health extension workers also highly supported her in her business promotion. Beside door-to-door visits, she uses different promotion and marketing strategies at market day and bazaars. Yeshi said that “we struggled a lot while introducing the products house-to-house but now the households are coming to our place and purchasing the slab.” The products that she is selling include hand washing equipment, water filters, water treatment tablets, soap, and toilet brushes. She is very happy with the work that she is doing, although there are misconceptions and unfavorable attitudes among some community members regarding women who are involved in mason work and toilet installation. Sometimes on her way home after finishing household latrine installation, people show her a sign of holding their nose and keep a distance from her. She felt sad and struggled a lot to the level of thinking to quit her business. But she persisted and decided to continue after reporting the issue to the kebele chair and setting it as an agenda item during one of the community meetings. The issue has been discussed among the community and she has seen a positive response. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 41 CASE STUDY, CONTINUED The business is running with a trust-based loan without any interest and guarantee. She takes the product from Dessie (EMA retailer shop), and she also gives similar loans to sales agents working under her. Further access to interest-free or low-interest loans would help her to expand her business. Her job has created an opportunity to be known by all the kebele’s residents and helped to create strong social bonds. She has provided free installation service for four poor households in the community. She is also facilitating a VSLA to solve the financial problem for poor households. She is very committed to her work, even working late in the evening during the Ramadan fasting period. Her total sales at the time of the interview in August 2022 were 933 sanitation product sales. 3.3 CONSORTIUM EFFECTIVENESS AS A CROSS-CUTTING ASPECT The Evaluation Team recognizes USAID/Ethiopia’s keen interest in understanding the performance of the Transform WASH activity through efficiency and effectiveness lenses. Rather than undertaking the complicated and intensive process of comparing Transform WASH with other MBS or general sanitation programs in Ethiopia (or elsewhere) or indeed by comparing Transform WASH with other consortia, the Evaluation Team presents some observations reflecting the internal workings of the activity. Consisting of a series of interventions delivered through a consortium of four institutions (PSI, SNV, Plan International, and IRC WASH), the coordination involving the staging and sequencing of activities across the consortium becomes critical to ensure that policy work, supply-side efforts, and demand creation join up effectively and efficiently. This suggests, as one example noted elsewhere in this report, that demand creation activities be closely aligned and sequenced with the business model approaches for supplying customers. There are clearly designated deliverables and targets for each consortium partner with regular reporting. The Evaluation Team notes that there is significant and regular exchange among the consortium partners at the national level. The communications do not appear to have always translated down to the woreda level, however. FGDs and KIIs at the subnational level often mentioned that consortium partners could work more effectively in an integrated way, noting insufficient sequencing and coordination of interventions. While Transform WASH has been promoting this integration among the region and woreda level stakeholders, subnational level interviewees noted a lack of a joint annual plan from Transform WASH. Woreda staff “see separate organizations rather than one Transform WASH integrated approach.” Transform WASH encouraged government partner offices at the woreda level to sign a joint MOU, however, the role and expected length of contribution of Transform WASH (as the facilitator of this process) is not mentioned, so government officials expressed less clarity on what to expect from the activity. While this has improved in later stages of the activity, interviewees expressed that local Transform WASH staff did not always reflect the same understanding of the overarching MBS approach at the outset of the activity. Speaking with one voice from the outset and showing more of an integrated plan would have given more legitimacy to efforts to encourage integration of local level partners. While the Evaluation Team did not review budget or staffing allocations, subnational level interviewees suggested that the demand creation support was under-resourced. The Transform WASH Reflection Workshop Report acknowledges this challenge of “stretched resources across implementation areas, with some woredas being too dispersed, and time and resource-intensive.” This also helped to give the impression that Transform WASH was more about sales and the supply side than demand creation. As noted under the response to EQ3 above and also acknowledged in the Reflection Workshop, the learning agenda (especially M&E) was too siloed in one consortium partner and could have been more effectively designed as a cross-cutting activity. Indeed, the diversity of the implementation areas provided a wealth of opportunities to understand the application of an MBS approach across different contexts. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 42 The Evaluation Team perceives that Transform WASH could contribute still more to this contextualization across the country. The supply side activities and the incorporation of gender considerations appeared to evolve significantly over the period. For both areas, Transform WASH took advantage of the “flexibility to pivot quickly and try new approaches that are responsive to local market conditions” (Population Services International/Ethiopia, 2022). It is unclear that the approach to demand creation and governance sufficiently incorporated lessons from phase one to phase two and then from those two phases to phase three. Government counterparts from later stage woredas expressed this gap directly, noting that Transform WASH could have had still greater impact and told a more compelling story to subsequent woreda counterparts had the partners referenced the learning from phase one more directly with stakeholders. 3.4 OBSERVATIONS ON SPILLOVER EFFECT The Evaluation Team’s primary objectives in expanding the Transform WASH performance assessment to four “control” or non-intervention woredas was to understand horizontal diffusion of the activity models to adjacent and more remote woredas and to explore differences in performance of MBS, sanitation demand and enabling environments. The hypothesis was that there would be a natural progression towards MBS through a general increase in awareness of the need for and benefits of improved sanitation and general shifts in the functionality of WASH governance and institutional arrangements related to sanitation provision. (Non-intervention woredas were chosen by the Evaluation Team where there were understood to be no other MBS activities undertaken by development partners or NGOs.) In some senses, the differences between the intervention and non-intervention woredas are fairly stark. There is limited systematic demand creation activities in non-intervention woredas, no organized sanitation businesses, and emerging but minimal availability and awareness around improved sanitation products. Latrines that were seen during the community visits in intervention areas were of higher quality. The OWNP structures were better established in the intervention woredas and there was understandably a clearer understanding of and awareness around MBS approaches by local authorities. This is translating into expansion of the approach into non-intervention kebeles by Regional Health Bureaus (in SNNP, Sidama, and Amhara in particular) and woredas (Sidama Region is expanding the approach to eight other woredas, SNNP to one other woreda, and Amhara to other woredas through the South Wollo Zone Health Bureau). Specific non-intervention woreda findings – In comparison to the other three non-intervention woredas visited, product spillover and market penetration in Albuko Woreda (South Wollo Zone, Amhara Region) seems to be working well. According to the woreda health bureau and PSI officers, product uptake is increasing in this non-Transform WASH woreda and the retailer is already established, buying WASH products and initiating sales directly. While business operations and promotional activities are mainly focused on selected intervention kebeles, the business and health information also reach people in adjacent physical and social proximity. Spillover is evident through product availability, product awareness, and wholesalers and retailers’ interest in stocking the products. South Wollo Zonal Health Bureau (SWZHB) learned from the experience of Transform WASH woredas. After the Were Elllu woreda experience visit, SWZHB discussed how best to scale up the Transform WASH learnings and MBS approach with non-Transform WASH woredas. In 2022, SWZHB selected 89 kebeles from five woredas (of a total of 205 kebeles). ETB 1.8 million (almost US$35k) was allocated from the zone health bureau and seven representatives from each of the five woredas were trained on the MBS approach by Transform WASH. Government stakeholders, especially the health bureaus, are engaged in promoting SATO pans and training and organizing youth to establish businesses to install and construct latrines. A quarterly performance review workshop is conducted with participation of all woreda and kebele Technical Teams members to evaluate their plans and create an opportunity to share ideas. (The seven woreda representatives included the Kebele Administrator and FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 43 deputy, general manager, two HEWs, Women and Child Affairs representative, and school principals.) A Telegram group was created for each woreda to share ideas and learnings. An enterprise was then established in each woreda to sell SATO pans and manufacture cement slabs. SWZHB linked those enterprises with the wholesaler at Dessie for plastic slabs. Regarding demand creation, woreda and kebele administrations took the responsibility to facilitate. Many SATO pans have already been sold and installed in the scale up woredas of South Wollo. Small business owners are keen to stock SATO products. SATO and AIM wholesalers and retailers in South Wollo and Gozamen Woredas of Amhara region stated that they have retailers from outside of Transform WASH woredas, like Dima Woreda, buying in bulk for resale to churches and health posts. All these transactions have been made without the Transform WASH market facilitation and support. The cooperative office in Albuko has confidence in sanitation marketing taking off as households are aware about latrines and have the capacity to purchase the products and engage installers. The sense is that the market will definitely respond. In further support of this, W/ro Sihin college has been working with Transform WASH since 2020 aiming to contribute to advancing the community and creating job opportunities. Staff from the college participated in a Training of Trainers at the national level in sanitation enterprise development. The college then provided training on MBS for enterprises and individuals at the request of the Transform WASH Dessie cluster office, SWZHB and Kutaber, Legehida and Kelala woredas. Were Ellu and Tehuledere TVET also benefited from the training, which is not only focused on technical skills but also includes business development and entrepreneurship. Based on the demand, the college is ready to provide the training which will eventually strengthen the market in these non-intervention woredas. In contrast to Albuko, Angolella Tera Woreda (also in Amhara Region and more remote from Transform WASH woredas) shows no sign of WASH business activities or knowledge or interest amongst business owners. In this former USAID GtN intervention area, the woreda was one of the targets with a package of WASH interventions. Although some of the KII respondents were familiar with the SATO products, visited households were not aware of them. The sanitation coverage in the woreda is below 26 percent as per the woreda health bureau officer. People use traditional pit latrines and open defecation is practiced in the community as per the KII. With 21 rural kebeles and three primary HCFs, Mirab Abaya Woreda is one of the OWNP/CWA woredas (under a World Bank lending program to enhance WASH services in over 300 selected woredas) in the Gamo Zone of the SNNP Region. The head and deputy head of the woreda health office stated that CLTSH and follow-up Mandona tools (an action-orientated approach post triggering) are combined with sanitation marketing approaches to increase household access to improved sanitation and increase the number of model households that could serve as examples. Most households (about 85 percent) in the woreda use latrines. The woreda is also working on constructing WASH facilities for HCFs. As part of the OWNP/CWA intervention, the woreda had very recently (literally weeks before the visit) begun work to strengthen the enabling environment for sanitation enterprises. This included finalizing construction of a shed to be used for storage, provision of SATO pans and AIM slabs, providing training on slab manufacturing and installation for sanitation enterprises, and other activities. To verify the adoption of improved sanitation products and construction workmanship, the Evaluation Team visited seven household latrines and observed that the products were installed (retrofitted) onto an existing concrete floor. In informal discussions with households, they noted that they were satisfied with the product and mentioned that, as it automatically closes after use, it prevents odors and flies. In addition, it is cleaned with a small amount of water. To its credit, Mirab Abaya Woreda is seeking to adopt best practices but without necessarily following the step-by-step procedures Transform WASH put in place or the national MBS guidelines suggest. The FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 44 woreda initiated the process quickly without strengthening the enabling environment and capacity building on demand creation approaches and provision of SBCC tools, etc. Furthermore, from the very onset, the local government (woreda health office) and regional health bureau have provided significant in-kind subsidies for the enterprises, including 100 SATO pans estimated to be ETB 25,000 (US$481) and 60 AIM plastic slabs at ETB 30,000 (US$577), with large quantities of sand, gravel, and cement as start-up contributions. Additional information from the woreda health office and HEWs shows that the regional health bureau gave direction to install 20 of 80 SATO pans for poor households. In general, even if the interest of the woreda to initiate the MBS approach is encouraging, course correction is required to establish a viable and sustainable sanitation market in the woreda based on the national MBS guidelines. The Evaluation Team also visited Gedeb Assessa (Oromia Region) to determine if there was any spill over occurring. In essence, Gedeb Asessa officials showed little awareness of MBS or Transform WASH activities and there was little evidence in the market of a nascent private sector response to any need for sanitation products and services. In the adjacent Transform WASH intervention woreda of Dodola, the health office supports kebeles on household latrine inventory during the first quarter of each year. Based on the inventory findings, primary health care units plan and implement sanitation activities. Regarding horizontal communication between neighboring woreda health offices, interviewees stated that, with the exception of quarterly review meetings organized by the West Arsi Zonal Health Office, there is no formally established exchange between woredas, including Dodola Woreda. However, the head of the Gedeb Assessa health office said he had heard about the marketing of improved sanitation products in the Dodola Woreda. Following discussion with the Woreda Health Office, the Evaluation Team observed seven household latrines and found that all were unimproved facilities (with either wooden riprap or mud plastered floor). Open defecation was also noted in the village visited. With the exception of installing a SATO pan (with skirting), the Evaluation Team observed that the status of household latrine facilities in Gedeb Assessa is similar to latrines in Dodola. Conclusions regarding the spillover effects – If Transform WASH coordinates with other projects including OWNP/CWA in non-intervention woredas where households have the resources to purchase the products and services, an MBS approach could expand quite quickly as rural consumers are aware of latrines and the benefits they provide. That said, there is a need to ensure that standards of product installation are of sufficient quality. Mechanisms also need to be put in place to track who is stocking the products, who they are selling to, and when these products are being installed. There is also a need for coordination among stakeholders to ensure that MBS is understood and agreed as a strategy, particularly in areas where products continue to be distributed for free. In such cases, clear targeting mechanisms need to be put in place for those that would receive a free product. Some outreach is also needed to attract the private sector to this market. Regional WASH coordination offices and Zonal WASH teams will need training and guidance to provide technical support and to ensure that the enabling environment is appropriate for an MBS approach. Some criteria should be established with the FMOH to determine which woredas are best placed to make quick progress (due to easy access to markets, relatively higher income areas where households are more likely and willing to invest, etc.). A coaching and mentoring process could be put in place from more successful woredas to those just initiating MBS strategies on an as-needed basis. Forums to share best practices across and within regions will also be needed. Transform WASH has been quite careful and methodical in its experimentation. As this performance evaluation has shown, an MBS approach needs to be designed with all the pieces of demand creation, supply side ecosystems, and governance and oversight well thought through. The National MBS Guidelines provide aspects of this but a further fleshed out strategy may still be needed. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 45 Highlights of relevant data from the household survey – In support of the Transform WASH qualitative assessment, PRIN conducted a household survey in the same intervention and non￾intervention woredas to determine if any significant differences can be detected in sanitation coverage (see Annex D for full report). Without a significant baseline survey from which to compare progress over time, the household survey provides some clues about WASH coverage based on a total sample size of 3,076 households (2,308 in intervention areas and 768 in non-intervention areas). Several interesting findings emerged from the survey. The household survey revealed that being located in a Transform WASH intervention area was not significantly associated with whether or not a household had a toilet. That said, households reporting no open defecation was significantly higher in intervention areas (51 percent) than in non-intervention areas (25.7 percent). Of those who had a latrine, 17.6 percent in intervention areas and 11.3 percent in non-intervention areas had recently upgraded (13.4 percent vs 6.9 percent in the last year). Interestingly, a higher percentage of households in non-intervention areas (75.4 percent) were prepared to repair their latrines if needed as compared to households in intervention areas (65.7 percent). The slabs in over 20 percent of toilets (equal in both intervention and non-intervention areas) were smeared by feces. A fairly equal and significantly high percentage (73.2 in intervention woredas versus 68.4 in non-intervention woredas) of households believed that having a toilet reduces the risk of diarrhea regardless of whether they have a toilet or not. In terms of access to water, roughly equal numbers around 80 percent had access to limited or basic (improved) water sources in intervention and non-intervention areas with SNNP being the best served at 91.5 percent and Somali having the least access at 56.9 percent; 56.5 percent versus 43.6 percent had access within a 30-minute walk, 27.8 percent versus 38 percent had access within 30-60 minute walk, and 15.8 percent compared to 18.4 percent had over an hour trip to fetch water. A higher percentage (53.7 percent versus 46 percent) pay for water in intervention woredas versus non-intervention woredas. The vast majority (around 90 percent) are not treating water in the home. The household survey data is available for USAID/Ethiopia and related organizations to use as a baseline, as appropriate, for future programming and analysis. Table 2 below provides a few key findings. Table 2: Key Household Survey Findings Metric Intervention Woredas – % (Respondents) Non-Intervention Woredas – % (Respondents) Have some form of a toilet facility 68.9% (1590) 77.2% (593) Report practicing open defecation 38.1% (548) 30.9% (102) Observation reveals latrine appears to be unused 7.8% (124) 7.4% (44) Have latrine in own yard/compound or home 87.1% (1382) 83.3% (484) Using unimproved facility with no slab/open pit 41.7% (662) 55.8% (331) Of those with a latrine, have access to basic latrine facility 22.2% (131) 10% (59) Toilet shared with neighbors 15.2% (241) 18.6% (109) Households accessing improved water source (denoted as “basic” on the water service ladder) 37.7% (870) 27.7% (213) Households treating water at home 12.1% (279) 7.9% (61) Report attending behavior change education 53.9% (1242) 63% (484) Report a belief that having a toilet reduces the risk of diarrhea 73.2% (1598) 68.4% (611) Households with a handwashing facility 9.4% (217) 9.5% (73) Households with soap and detergent 5.7% (132) 6.4% (49) Report having no idea where to purchase sanitation products 10.8% (101) 12.7% (61) Report more difficult to find parts to repair the latrine 79.6% (1264) 74.5% (442) Report purchasing a toilet slab as not a priority 9.4% (88) 12.0% (58) Report having “no money to buy sanitation products” 74.3% (695) 73.2% (353) Construction decision made by the male of the household 61.3% (785) 64.2% (315) Report paying to construct their current toilet 42.0% (667) 32.5% (193) FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 46 4.0 CONCLUSIONS AND RECOMMENDATIONS The Evaluation Team finds that the Transform WASH activity has made much appreciated and significant contributions to advancing MBS as a bona fide approach in Ethiopia. Through advocacy and convening efforts, direct communications and a close working relationship with government, and the sharing of critical research findings on topics relevant to the sector in a timely way, Transform WASH has shifted earlier more skeptical attitudes toward MBS and helped shape MBS-related GOE policies and approaches. Shifting the discourse on taxation issues and supporting the revision of key GOE policies and guidelines have been significant contributions of the USAID-funded activity. Through learning by doing, Transform WASH has provided a practical demonstration of the National MBS Guidelines that are starting to be adopted in some non-Transform WASH woredas. In particular, the activity has facilitated the manufacturing and distribution of a wide range of sanitation products that are in demand by consumers and created greater linkages across the service delivery supply chain. It has enhanced mechanisms to increase access to sanitation products and services including for low-income households. Transform WASH’s experimentation with and careful documentation of business models has encouraged new entrants into the sector. Fostering opportunities with existing businesses, especially those engaged in retailing construction materials, has shown to be more successful than recruiting new people and forming new businesses at that supplier level. Retailers engaged with Transform WASH as business partners see sanitation products and rural customers as having the potential to generate revenues and have brought on sales agents to expand their marketing efforts. Through Transform WASH, business partners have developed knowledge and skills to expand their customer base, diversify their income sources, provide job opportunities for others in the community, and develop into profitable enterprises. The activity also has demonstrated that women can be successful in this conventionally male-dominated construction sector. 4.1 RECOMMENDATIONS While Transform WASH’s achievements are many, some gaps in the approach as highlighted in this report suggest aspects that could be strengthened going forward. As noted elsewhere, these are offered in the context of and recognizing that Transform WASH is not a direct implementer nor directly engaged in the business of selling products and services at the household level. 4.1.1 RECOMMENDATIONS FOR TRANSFORM WASH AND/OR FUTURE PROGRAMMING Demand Creation • Identify and document more deliberate strategies that unpack the three customer segments from a marketing perspective that traverse from demand generation to demand activity to demand fulfillment. • Ensure that demand creation (behavior change) activities and commercial promotions are appropriately synchronized. • Pursue more wide-reaching financial institutions, like cooperatives, to engage rural households. • In conjunction with demand creation and promotional activities, work with MFIs to determine whether a bridging arrangement could be created for household “consumption loans” (where repayments would currently be expected to begin a month after the loan has been recorded) that allow for household lending to occur throughout the year, but household-level repayment modalities can be geared around the harvest season as is done for agriculture loans. • In line with the aspirational pride and prestige selling points, ensure sufficient emphasis on the superstructure together with the emphasis on the latrine floor. • Continue to explore how practical demonstrations including public latrine installations can help increase consumer aspirations. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 47 • Continue to explore the further integration of MBS interventions and energy/biogas production for farm-based income earners as further incentive for sanitation investments. Supply-Side Interventions • Continue to shape the curriculum contribution of the TVET platform to build knowledge and local design efforts that are less costly, acceptable to consumers, and suited for specific geographic contexts. • Continue to develop a how-to guide and advocacy activities to engage microfinance and finance institutions and cooperative associations around innovative ways to structure lending products for sanitation to small and medium-sized enterprises (as well as to consumers). • Encourage USAID and other funders to engage with finance institutions more directly to build up their awareness of the sanitation sector and the opportunities to leverage their products and services to expand their customer base. • Provide (at least) cursory analysis on GOE income from sanitation product taxation in relation to the costs to the health service of households not having access to sanitation. • To overcome the supply challenge, explore expanded wholesale bulk supply options and consider organizing slab manufacturers into associations to buy raw materials in greater quantities which may help to lower prices. Governance • Continue to advocate for sanitation to have a dedicated budget line at national, regional and woreda levels – once that has been achieved, continue to advocate for necessary allocations • Conduct more deliberate and purposeful follow-up to understand and identify: 1) the impact of capacity building and training efforts; 2) any challenges faced in terms of applying the learning and tools; and 3) any further gaps that may warrant support from Transform WASH consortium members. • Support the GOE to determine the staffing required at all levels, which will be instrumental in moving forward with an MBS approach. While at present, budget, salary scale and trained human resources are in short supply, GOE needs support in understanding minimum, satisfactory and optimal staffing levels to drive MBS from national to local levels. • Continue to forge national-level dialogue and coordination with a stronger emphasis on engaging those stakeholders who have a role to play but who are currently unaware of the sanitation sector’s MBS-related opportunities (e.g., municipalities at all levels, cooperatives, commercial banks with an interest in financial inclusion, and plastic-manufacturing companies in Ethiopia). • Promote government-owned, regional-level MSPs as an association of the JSS process to ensure greater local ownership through which to engage stakeholders, share information, and solve problems. • Re-engage FMOH/NWCO in the ISI tool to establish ownership beyond the Transform WASH partners, making it a federal requirement to be used at all levels of government monitoring and reporting. (Determine whether a redesign is required, for example, to implement a Likert scale with 1–5 responses for each category to show progressive realization of the ideal scenario.) • While MSPs should continue in their current format for at least a few more years, determine a longer-term home for them by potentially integrating into other hygiene and environmental health platforms for long-term knowledge-sharing and strengthened coordination among MBS actors. • MSP design should include clear metrics to track progress and measure their success, which could galvanize greater engagement across stakeholders. • As a research area, explore any negative consequences for vulnerable populations (including female￾headed households or women more generally) of the enforcement of bylaws requiring household investments in latrines. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 48 GEFE • Ensure involvement, support, and commitment of relevant GOE ministry offices, like the federal and regional level MWSA offices. This is aimed at supporting policy formulation that sustainably guides the systematic participation of women and girls in decision-making at the local level. Consider secondments into relevant offices for this purpose. • Further explore the gender aspects of VSLA membership (how membership in VSLAs explicitly increases female participation more broadly, how membership influences household decisions to invest in sanitation, etc.). • Further verify stakeholders’ anecdotal reporting that women sales agents are more trusted than men and thereby achieve higher conversion rates in terms of sales. • Conduct further in-depth study and analysis as to the factors constraining women’s participation in the sanitation supply chain in, for example, the Oromia region. • Further consider how to support TVET and enterprise development/job creation offices specifically with the recruitment of women in training and business development support activities. • Support the development of awareness creation sessions for private sector actors like MFIs that result in flexible and smooth loan processes for women in sanitation businesses. While also a good business investment, this could be explored as falling within MFIs’ corporate social responsibility initiatives. • Emphasize the whole latrine structure with improvements to the superstructure design to ensure privacy and security for women and girls.23 Operational Matters • Ensure that the research and advocacy agendas are jointly owned by all consortium members with a clear strategy for different stages of the activity. • Revisit whether FMOH/NWCO should be a signatory with the federal level MBS MOU signatories on behalf of OWNP (otherwise, support to OWNP, which has no direct authority and legal mandate on its own, proves sub-optimal). This will take the issue of MBS to the NWCO and Technical Team and allow for greater integration into their respective regional and woreda WASH structures. • Include the Transform WASH consortium as a woreda signatory in local-level MOUs (that are already developed by the Activity) to foster transparency and clarity about the Activity’s expected role and contributions.24 • Ensure more dedicated in-house expertise at the senior consortium level on finance aspects and how to engage financial institutions. 4.1.2 RECOMMENDATIONS FOR USAID • Support further study across the USAID MBS portfolio (including Kenya, Ghana, and Uganda, etc.) of the more local approaches like VSLAs to understand what has worked well in targeted consumer outreach, repayment rates, GEFE considerations, scalability, and outside investment to leverage further scale. • Clarify the scope of the USAID-Lixil partnership to understand the barriers to supporting local manufacture with Lixil (producer of the SATO pan) seeking subsidiary relationships in Ethiopia. • Explore how sanitation funds (National Sanitation Account), loan guarantee funds (akin to the USAID Microfinance Development Credit Authority), social/development impact bonds, and 23 This could include supporting HEWs, mason/installers, or others on advice for households on small, immediate, doable actions as per the Follow-Up Mandona approach the Global Sanitation Fund introduced in Ethiopia. 24 This recommendation is endorsed by the Evaluation Team but comes directly from woreda officials. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 49 revolving funds have been deployed across the MBS portfolio or elsewhere to expand access to credit for suppliers, service providers, and households. • Reference experience in other countries (like Kenya) regarding mobile banking, mobile lending, etc. to get ahead of likely changes in the near future and determine how these could positively (or negatively) impact Ethiopian consumer access to finance and financial inclusion more broadly. • Continue and strengthen implementing partner efforts to shape the GOE WASH data management information system to ensure consistent WASH data can be used at all levels for planning, monitoring, and reporting. (This may require convening all USAID implementing partners to explore the issue further to understand current gaps and challenges.) • Given the wealth of USAID experience in designing and funding these types of consortia, create a learning approach to consortium design across MBS countries. FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 50 5.0 PRIMARY REFERENCES GOE FMOH (Government of Ethiopia Ministry of Health). (2016). National Hygiene and Environmental Health Strategy (2016-2020). Government of Ethiopia. GOE FMOH. (2018). Terms of Reference for Sanitation Marketing Multi-stakeholder Platform. Government of Ethiopia. GOE FMOH. (2020). National Market-based Sanitation Implementation Guideline. Government of Ethiopia. GOE FMOH. (2021a). Health Sector Transformation Plan II. Government of Ethiopia. GOE FMOH. (2021b). National Standard Definitions for Sanitation, Hygiene, and Water. Government of Ethiopia. GOE FMOH. (2021c). Sanitation and Hygiene Smart and Targeted Subsidies Protocol. Government of Ethiopia. GOE FMOH. (2022). Ethio-MBS, Ethiopia’s National Market-Based Sanitation Newsletter, Issue 11. Government of Ethiopia National WASH Coordination Office (NWCO). (2018). One WASH National Programme (OWNP) Phase 11 - 2018-2020. Government of Ethiopia. Novotný, J., Humňalová, H., & Kolomazníková, J. (2018). The social and political construction of latrines in rural Ethiopia. Journal of Rural Studies, 63, 157–167. https://doi.org/10.1016/j.jrurstud.2018.08.003 Population Services International/Ethiopia. (2022). USAID Transform WASH Reflections Workshop Report: Five Years of Learning and What Comes Next to Scale Market-Based Sanitation. Tamene, A., & Afework, A. (2021). Exploring barriers to the adoption and utilization of improved latrine facilities in rural Ethiopia: An Integrated Behavioral Model for Water, Sanitation and Hygiene (IBM-WASH) approach. PLoS ONE, 16(1), e0245289. https://doi.org/10.1371/journal.pone.0245289 Transform WASH. (2021). Transform WASH Annual Progress Report FY2021. US Embassy in Ethiopia. (2021). U.S. and Ethiopia Launch New $2.2 Billion Phase of the Productive Safety Net Program. USAID. (2016). USAID Transform WASH Activity Design Document, 2016 _NFO no. RFA-663-16- 000007 Transform/Water, Sanitation, Hygiene (WASH). USAID. (2017). ADS Chapter 205: Integrating Gender Equality and Female Empowerment in USAID's Program Cycle. (Partial Revision Date: 01/22/2021). US Government. USAID. (2018). Scaling Market-Based Sanitation: Desk Review on Market-based Rural Sanitation Development Programs. USAID Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) Project. USAID. (2019). USAID Country Development Cooperation Strategy, July 2019-July 2024. USAID. (2021). Tax Exemptions: A Catalyst for Demand and Supply of Plastic Sanitation Products: Impact Assessment—Ethiopia. USAID Water, Sanitation, and Hygiene Partnerships and Learning for Sustainability (WASHPaLS) Project. USAID, UNICEF, SNV, WaterAid, World Bank, & The Sanitation Learning Hub. (2021). Area-wide Rural Sustainable Sanitation: Learning Event—Summary of Discussions. USAID/WASHPaLS. (2021). Area-wide Rural Sustainable Sanitation: Learning Event - Summary of Discussions. (Published on Global Waters, 31 January 2022. (https://www.globalwaters.org/resources/assets/area-wide-rural-sustainable-sanitation-learning￾event-summary-discussions) World Health Organization/United Nations Children’s Fund Joint Monitoring Program (JMP) 2021 Data, (https://washdata.org/data) (accessed September 2022). FINAL REPORT – PERFORMANCE EVALUATION OF ETHIOPIA TRANSFORM WASH ACTIVITY | 51 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov