EVALUATION REPORT Isuku Iwacu Mid-term Evaluation Final Report Limited Internal Distribution September 18, 2019 This publication was produced for review by the United States Agency for International Development (USAID). It was prepared by WASH for life. TABLE OF CONTENTS Acronyms………………………………………………………………………… ii Executive Summary…………………………………………………………….. iv Glossary…………………………………………………………………………… ix Introduction……………………………………………………………………… 1 The Intervention………………………………………………………………… 1 Purpose and Scope of the Evaluation………………………………………… 2 Methodology……………………………………………………………………… 2 Findings and Conclusions………………………………………………………. 4 1. Appropriateness of the Activity design………………………………………… 4 2. Effectiveness of the Activity……………………………………………………. 16 3. Performance, Impact and Sustainability Prospects…………………………….. 22 4. Building Local Capacity…………………………………………………………. 25 5. Gender and Equity……………………………………………………………… 29 Learning and Capitalization…………………………………………………… 32 Recommendations………………………………………………………………. 34 Utilization Plan…………………………………………………………………… 37 List of Annexes…………………………………………………………………… 37 ii Acronyms ACB Amasezerano Community Bank AMIR Association of Microfinance Institutions in Rwanda AOR Agreement Officer’s Representative BCC Behavior Change Communication CBEHPP Community-based Environmental Health Promotion Programme CHAIN Community Health and Improved Nutrition CHC Community Health Club CSO Civil Society Organization DBC Designing for Behavior Change (Framework) DSC District Sanitation Center EQ Evaluation Question FGD Focus Group Discussion FSM Fecal Sludge Management FY Fiscal Year (October 1 – September 31) GOR Government of Rwanda HQ Headquarters IR Intermediate Result JMP Joint Monitoring Program KII Key Informant Interview LCSO Local Civil Society Organization M&E Monitoring and Evaluation MEL Monitoring, Evaluation and Learning MFI Microfinance Institution MHM Menstrual Hygiene Management MINALOC Ministry of Local Government MININFRA Ministry of Infrastructure MOH Ministry of Health MOU Memorandum of Understanding NGO Non‐governmental Organization NISR National Institute of Statistics Rwanda NSP National Sanitation Policy ODF Open Defecation Free PD Program Description PPP Public-Private Partnership PWD Person / People with Disabilities Q Quarter (i.e. 3 months of Fiscal Year) RWF Rwandan Francs SACCO Savings and Credit Cooperative SDGs Sustainable Development Goals SGF Sanitation Guarantee Fund iii SILC Saving and Internal Lending Community SO Strategic Objective U Ubudehe (a quintile of social class) USAID United States Agency for International Development VSLAs Village Savings and Loan Associations WASAC Water and Sanitation Corporation WASH Water, Sanitation and Hygiene ZOD Zero Open Defecation iv Executive Summary Background Isuku Iwacu is a four‐year rural sanitation Activity in Rwanda, funded by the United States Agency for International Development (USAID) and executed by a consortium of international non‐ governmental organizations (NGOs), headed by SNV USA and with World Vision International and Water for People. The Activity’s overall goal is to ‘increase local ownership and capacity to deliver sustainable, high quality sanitation and hygiene services in order to decrease childhood stunting.’ The Activity is being implemented in eight districts in Rwanda, where it will help an estimated 500,000 people gain access to improved household sanitation, as defined by the Joint Monitoring Program (JMP), while ensuring that Rwandans living in 137 targeted villages across the eight districts are using, at a minimum, JMP‐defined improved sanitation and living in open defecation free (ODF) environments (USAID/Rwanda refers to this ODF approach as Zero Open Defecation–ZOD). Evaluation purpose and scope The four-year Activity started in September 2016 and has passed the mid-term. WASH for life has consequently been contracted to conduct a Mid-term Evaluation of the Activity. This evaluation assessed the appropriateness of the Activity’s design and basic assumptions and its progress and performance to date. WASH for life also identified lessons learned and successes to inform the remainder of this Activity. The scope of evaluation encompasses the implementation of the Activity from its start date, September 2, 2016, through Fiscal Year 2019, Quarter 1 (FY19Q1), December 31, 2018. The Mid-term Evaluation addresses the following broad evaluation questions (EQs): 1. How appropriate was Isuku Iwacu’s initial activity design of building capacity and applying market‐based approaches to increase demand for sanitation products and access to sanitation services in the targeted districts and villages? 2. How effective are Isuku Iwacu’s current implementation strategies to successfully expand access to sanitation services? What changes could be made to increase effectiveness in the remaining two years? 3. To what extent have the different implementation approaches been successful? Which of them appear to be sustainable and should be prioritized going forward? 4. To what extent has Isuku Iwacu been successful in building capacity to improve sustainable access to sanitation? 5. How has Isuku Iwacu’s approach taken into consideration issues of gender and equity? Methodology WASH for life undertook this evaluation between April and August 2019, applying a tailored qualitative approach. Data collection was conducted through: comprehensive document review of internal reports (for information and quantitative data) and relevant external literature; 50 key informant interviews and where necessary, group informant interviews; 6 community focus group discussions (FGDs); and 13 site visits for direct observations. v Key findings As the first project of its kind in Rwanda, Isuku Iwacu was designed to place SNV (and the consortium members) as facilitators for promoting market-based sanitation – rather than exclusively as implementers. As such, there is a need for learning throughout implementation, and furthermore, a need to document capitalization for internal processing and external sharing. 1. Appropriateness of the Activity design Conceptually, Isuku Iwacu had a good Activity design. It was meant to be complementary to other USAID WASH, Health and Nutrition Activities, although there is also an associated confusion that might cause overlapping efforts by USAID Activities (Isuku Iwacu and Gikuriro), or even worse, gaps in coverage. In terms of appropriateness to national goals, Isuku Iwacu shares goals with the Governmentof Rwanda’s 2016 National Sanitation Policy (NSP). Thus, Isuku Iwacu gained local buy-in and ownership rather quickly (despite few challenges), and demonstrated a good working relationship. The Activity’s work to translate and help disseminate the NSP was very appreciated by local stakeholders, although deeper sanitation concepts (i.e. goals beyond building latrines) did not trickle down from the national level. This may impact local stakeholders’ abilities to take over and scale up sanitation outreach and financial approaches. The Activity’s first intermediate result aims to increase demand for sanitation products and services. Although, given that the vast majority of households (around 96%) had latrines (improved and unimproved), there was very little need to generate demand for basic access to sanitation. Hence, Isuku Iwacu shaped its efforts to its BCC strategy, appropriately developed using the Designing for Behavior Change (DBC) Framework (though the timing was delayed and strategic development less effective than it could have been). The Activity’s second intermediate result aims to improve the supply and availability of sanitation products and services in the private sector. Through a series of local studies and international workshops, and trial-and-error with the private sector, Isuku Iwacu realized its original design was not relevant to the context. Isuku Iwacu needed to study more and readjust its strategy. The New Private Sector Implementation Strategy was developed in March 2018 (though well into the implementation phase), which readjusted the Activity’s direction as an ‘anticipated sanitation market facilitator.’ Though even with the expected engagement of local Civil Society Organizations (LCSOs) in FY20 and the pending functionality of all District Sanitation Centers, this evaluation finds that there isn’t enough time left in the Activity to effectively test and sustainably carry out the Strategy’s remaining phases (sanitation loan testing and scale up). 2. Effectiveness of the Activity This ‘learning by doing and adapting’ learning curve explains many of the delays in working towards IR1 and IR2, and the Mid-Term Evaluation recognizes valuable efforts by Isuku Iwacu to improve sanitation by engaging the various actors of the private sector. The evaluation also offers several recommendations for strengthening such engagement. Though generally, the Activity would have been more effective if there had been early installation of the District Sanitation Centers (DSCs), links with LCSOs, and engagement with Microfinance Institutions (MFIs) and local savings groups to promote sanitation loans from the onset of the Activity. Capacity issues of LCSOs in regard to the private sector led to delays; LCSOs’ capacity building measures have been put in place to mitigate shortcomings. vi Building on the findings from EQ1, strengthening collaboration between Gikuriro and Isuku Iwacu (especially for BCC) is an identified key room for improvement in order to support the Activity’s overall effectiveness and measurable impact. Implementation delays and inconsistent sequencing of the DBC Framework might deteriorate behavior change effects of the Activity, leading to lower outcomes. Finally, the evaluation uncovered some inconsistences in the ZOD definition and resulting technical definitions. This is discussed further under EQ2. Inconsistencies identified by this evaluation in the ZOD definition, technical criteria, and monitoring process must be addressed. Furthermore, as LCSOs will work with and eventually replace SNV in this Activity to continue sanitation collaborations with the districts and communities, there is the need for strengthened monitoring tools and processes to share and hand over. 3. Performance, Impact and Sustainability Prospects Data indicate an estimated overall progress of 20-25% of the outcome indicators by FY19Q2, i.e. after 31 months of project corresponding to 60% of completed project timeline. Considering the delayed implementation, Isuku Iwacu may consider the opportunity of an extension to have more time to work towards achieving its targets with high quality. With an extension, the Activity can also meet the need for further learning throughout implementation, readjustment of financial approaches as needed, and document capitalization. The two financial approaches, vouchers and loans, are diverging, taking a different form in each district, even in each cell and village, due to factors linked to socio-economic class, distance from the DSC, amount of BCC received, and mobilization generated. Considering that the loan approach encourages community members to self-finance their own improved latrines, without Isuku Iwacu subsidies, the preceding voucher approach risked creating an aura of assistance. With two distinct financial approaches, the Activity cannot expect the results of these diverging approaches to be uniform nationwide. The Activity did not pause long enough to learn from the voucher pilot phase and readapt the approach, and it could have benefited from doing so. The challenges and successes vary incredibly by district – even by cell and village. Considering this, the Activity may prefer to adjust slightly the sanitation loan approach to the specific differences and needs of the sector and cell levels – as opposed to carrying out a nationwide approach. 4. Building Local Capacity As part of IR3, Isuku Iwacu provided hands on support to the government. The Activity steered away from traditional capacity building and focused on facilitating sustained access to sanitation products and services by partnering with and strengthening local stakeholders. These stakeholders include local governments, LCSOs, construction companies, and financial institutions. The Mid-term Evaluation finds that deeper sanitation concepts like adjusting/scaling up financing approaches, ensuring coverage (vs. access), inclusion (EQ5), and verifiable monitoring haven’t transferred down from the national level. These concepts are currently lost at the sector, cell and village levels. There, just the goal of building latrines remains – which is partly effective – however, building latrines and counting numbers doesn’t take into consideration how to reach the most the vulnerable, monitor accurately, and promote coverage. Therefore, as Isuku Iwacu adapts and tests new approaches – such as the forthcoming loan approach – it must continue to engage with national and district-level officials to build their capacity in durable approaches. This is in fact incorporated in the Activity’s FY19 workplan and should be added to the FY20 workplan. vii Furthermore, with CHCs being used as primary focal points for community health, this evaluation highlights the importance of strengthening CHC capacity. On the same note, Isuku Iwacu should strengthen the capacity of local leaders, savings groups (i.e. VSLAs) and Amasibo to encourage sanitation coverage. 5. Gender and Equity The Activity conformed to USAID’s Policy on Gender Equality and Women’s Empowerment and was in line with the Government of Rwanda’s National Gender Policy. Women and girls have been targeted because the Activity targeted all households in the selected villages (thus, females were not excluded) – though Isuku Iwacu did not take additional steps to design programming around the specific needs of women and girls. In terms of equity, the Activity offers no distinct definition; still, it took steps to address the specific sanitation needs of people with disabilities, the elderly and children under five. While relevant and inclusive, these efforts have only recently been launched. Other equity efforts targeted the poorest households (those from Ubudehe 1 category, considered the most vulnerable). However, concerns about the beneficiary selection process and social class categorization highlight a risk for Isuku Iwacu in achieving sanitation access for the most vulnerable. In all, Isuku Iwacu needs to modify or expand its approach to be more inclusive and attentive to the specific needs of all population groups. Learning and Capitalization Isuku Iwacu encountered many challenges and delays, but it also took informed steps to mitigate them, learn from them and adapt. This evaluation was able to identify several key areas from which the Activity can further learn: Lessons learned to date 1. It is relevant and critical to take time to pilot the various financial approaches on a smaller scale prior to scaling up nationwide. 2. Due to consequential delays largely from undertaking an innovative sanitation approach in Rwanda, the Activity is not on track to meet its ambitious targets, which could have been reduced. Simply evaluating Isuku Iwacu by its quantitative indicators does not reflect its learning curve. 3. This evaluation finds various evidences that highlight the low level of consistency within the WASH sector in Rwanda, suggesting the need for Isuku Iwacu to take part in learning within the WASH sector for improved program delivery nationwide. Topics to assess for further learning and advocacy 4. Isuku Iwacu may consider conducting an evidence-based survey during the last months of implementation to collect primary data on health outcomes to show the relationship between the Activity and its nutritional Strategic Objective. 5. USAID/Rwanda can explore the possibility of an Ex-Post Impact Evaluation 1-3 years after implementation ends in order to strengthen learning for future programming and advocacy. viii Recommendations Based on the key findings and upon further discussion with SNV and USAID/Rwanda, this Mid￾term Evaluation offers several recommendations to help improve the final year and a half ofthe Isuku Iwacu Activity as well as few recommendations for future WASH programming: Quality completion of the Isuku Iwacu Activity 1. Continue BCC/outreach activities, including further training and encouraging CHCs, local leaders and Amasibo to mobilize their communities andfollow up. 2. Explore the relevance of other business model options available in the country (such as cell-level traders) to expand local access to sanitation products and services at sector/cell/village levels via deeper supply chain mapping. 3. Reaffirm and strengthen the process to reach and pre-declare ZOD status in the targeted villages, and strengthen local/district monitoring and verification capacity. 4. Consider strengthening monitoring and conducting more monitoring activities, given the few but serious discrepancies and issues reported. 5. Act quickly to pilot the sanitation loan approach via AMIR and SACCOs. 6. Carry out more capacity building efforts with LCSOs, MFIs, local leaders and community groups, service providers, and local authorities in order to leave a solid framework in place after the Activity ends. 7. Encourage districts to fully operationalize District Sanitation Centers as soon as possible to increase access to sanitation products and overall Activity effectiveness. 8. Further review the effectiveness and impact of the Isuku Iwacu innovative approaches and develop best practices. 9. Consider requesting a project extension from USAID. Future WASH programming 10. Address water supply if not secured to gain health and nutrition-related outcomes. 11. As a nexus of SNV programming, promote safe practices for emptying latrine pits in villages and explore the potential for integrated Sanitation and Livelihoods programs. ix Glossary The following definitions have been provided by WASH for life for the sake of clarity in this Mid￾Term Evaluation. They are either based on globally recognized definitions that have been adapted for this evaluation, or are quoted/adapted from relevant Activity documents. Sources are specified. Access to sanitation: Ability to acquire affordable services and improved (private) sanitation facilities, as well as safe on- or off-site treatment and disposal of wastewater and sludge. [NSP, 2016] Coverage: The use of improved (private) sanitation facilities [NSP, 2016] – Including operation and maintenance services, shifting the focus from ‘having latrine facilities’ to ‘safely and properly using one in a way that promotes good health.’ Equity: Consideration of and actions to address different needs of different population groups, including vulnerable people, like the elderly, poorest, and people with disabilities who have special needs – rather than applying a blanket approach to assistance and development programming that doesn’t consider vulnerabilities and specific needs. [Adapted from UNICEF] Gender: Consideration of and actions to address contextual differences in gender roles, responsibilities, associated barriers, and specific needs – not just of women andadolescent girls, but also men, boys and young girls (also known as gender equity). [European Institute for Gender Equality] Sanitation marketing: Sharing information with current and potential consumers about the availability of sanitation products and services, including shop locations, costs and varieties available as well as options for sanitation financing. [As understood by the Isuku Iwacu Activity implementation and goals] Sanitation products: Products and materials needed in sanitation, such as construction materials for improved latrines, cleaning products, sanitary pads (for women), etc. [As understood by the Isuku Iwacu Activity implementation and goals] Sanitation services: Services related to sanitation, such as the construction, service for emptying pits, cleaning, availability of information at sanitation businesses, etc. [As understood by the Isuku Iwacu Activity implementation and goals] Zero Open Defecation Pre-Declaration*: Completed by Isuku Iwacu. Term used when all houses in a village have at least an on-site toilet which separates human excreta from human contact, no matter the standard/ quality. Zero Open Defecation Declaration*: Completed by and at the discretion of each district and all the local authorities, including MININFRA, MINALOC, and MOH. Although not yet known, likely to follow the SDG definition, focusing on getting all households to “basic sanitation services.” *This evaluation found that a common definition in Rwanda is disputed; thus, an analysis of ZOD is presented in the report body. 1 1. Introduction Isuku Iwacu is a four‐yearruralsanitation Activity in Rwanda with a budget of nearly USD$10.7 million dollars, funded by the United States Agency for International Development (USAID) and executed by a consortium of international non‐governmental organizations (NGOs). The Activity’s overall goal is to ‘increase local ownership and capacity to deliver sustainable, high quality sanitation and hygiene services in order to decrease childhood stunting.’ The Activity is being implemented in eight districts in Rwanda, where it will help an estimated 500,000 people gain access to improved household sanitation. The four-year Activity started in September 2016 and has passed the mid-term. WASH for life has consequently been contracted to conduct a mid-term evaluation of the Activity. This evaluation assessed the appropriateness of the Activity’s design and basic assumptions and its progress and performance to date. It also identified lessons learned and successes to inform the remainder of this Activity. The Mission will use this evaluation to improve quality implementation of the Activity and inform future Water, Sanitation and Hygiene (WASH) programming in Rwanda. 2. The Intervention Isuku Iwacu is a four‐year rural sanitation Activity, funded by USAID, awarded on September 2, 2016, and estimated to be completed by November 9, 2020. The Activity is being executed by a consortium of international NGOs headed by SNV USA with World Vision International and Water for People. Through Isuku Iwacu, USAID/Rwanda will contribute to improvements in sustainable access to WASH services, with an emphasis on the lowest poverty quintile. Isuku Iwacu is being implemented in eight districts: Kayonza, Kicukiro, Ngoma, Nyabihu, Nyanza, Nyarugenge, Ruhango, and Rwamagana. The Activity will help an estimated 500,000 people gain access to improved household sanitation, as defined by the Joint Monitoring Program (JMP) for the Millennium Development Goals, while ensuring that Rwandans living in 137 targeted villages across the eight districts are using, at a minimum, JMP‐defined improved sanitation and living in open defecation free (ODF) environments (USAID/Rwanda refers to this ODF approach as Zero Open Defecation). The overall goal of Isuku Iwacu is to increase local ownership and capacity to deliver sustainable, high quality sanitation and hygiene services in order to decrease childhood stunting. Over the life of the Activity, to foster local ownership and strengthen capacity building, the Isuku Iwacu Activity has defined the following Strategic Objective (SO): • Improve access to and encourage correct, consistent use of household sanitation and hygiene facilities in order to decrease childhood stunting. The Activity will achieve this SO by: • Directly supporting and contributing to the Government of Rwanda (GOR)-led efforts to improve access to sanitation in Rwanda; and • Promoting districtwide, private sector‐driven household sanitation and hygiene interventions and district‐ and national‐level capacity development. Three intermediate results (IRs) will contribute directly to achieving this SO: • IR1: Increased demand for sanitation products and services; • IR2: Improved supply and availability of sanitation products and services in the private sector; • IR3: Improved governance for sustained access to sanitation and hygiene products and services. 2 3. Purpose and Scope of the Evaluation The purpose of this mid‐term evaluation is to: 1. Assess the appropriateness of the Activity’s design and basic assumptions; 2. Assess the progress and performance to date; and 3. Inform the remainder of this Activity and future Mission WASH programming. The scope of evaluation encompasses the implementation of the Activity from its start date, September 2, 2016, through Fiscal Year 2019, Quarter 1 (FY19Q1), December 31, 2018, thus slightly over mid-term. Due to delays in the field visit for data collection, this evaluation also considered new activities, villages and approaches from FY19Q2 for a more robust analysis. Evaluation Questions The evaluation addresses the following broad evaluation questions (EQs) for the aforementioned time period: 1. How appropriate was Isuku Iwacu’s initial activity design of building capacity and applying market‐based approaches to increase demand for sanitation products and access to sanitation services in the targeted districts and villages? 2. How effective are Isuku Iwacu’s current implementation strategies to successfully expand access to sanitation services? What changes could be made to increase effectiveness in the remaining two years?1 3. To what extent have the different implementation approaches been successful? Which of them appear to be sustainable and should be prioritized going forward? 4. To what extent has Isuku Iwacu been successful in building capacity to improve sustainable access to sanitation? 5. How has Isuku Iwacu’s approach taken into consideration issues of gender and equity? The detailed sub-questions and applied approach to answering each EQ, including data sources and analysis plan, are elaborated in the Evaluation Design Matrix (Annex A). 4. Methodology To answer each of the evaluation questions described above, WASH for life applied a tailored qualitative approach. Data collection was conducted through the following methods: • Comprehensive document review of internal reports (for information and quantitative data) and relevant external literature • 50 key informant interviews (KIIs) and where necessary, group informant interviews • 6 community focus group discussions (FGDs) • 13 site visits for direct observations 1 Considering the delay in implementing the Mid-Term Evaluation, the remaining period is accordingly reduced to 15 months. 3 On an ongoing basis, primary and secondary data were entered into the evaluation team’s data collection tool (Excel matrix), coded and analyzed, with preliminary findings drawn out early on as well as key queries highlighted for further investigation. The Evaluation Design Matrix (Annex A) expands on the EQs and informs the data collection tools, sources and analysis process by illustrating in detail how the different data collection methods will feed into the analysis plan to answer each evaluation question. The full methodology detailing the above data collection methods, evaluation approach, site selection, inception and analysis phases, and preliminary findings workshop is presented in Annex B: MTE Methodology. Limitations and Assumptions The evaluation team noted that there are few differences in the Activity’s outputs in Kigali area and in the districts within Eastern and Southern provinces. Based on Activity reports, available data and inputs from key SNV staff to highlight differences, site selection was made to be representative for qualitative analysis. All quantitative data come from Activity planning and reporting documents. A key part of this evaluation (EQ5) investigates aspects related to equity, which is understood as the inclusion of vulnerable groups in the Activity, including people with disabilities (PWD) and the elderly (refer to Glossary for definition). While the evaluation team managed to speak with key stakeholders on this topic, they were unable to identify PWD in the villages to visit, and further, no FGD contained PWD, though efforts were made to invite them. The evaluation team did manage to have formal and informal talks with elderly people in the villages, who are also considered vulnerable when it comes to constructing and accessing household latrines. Through these efforts, the evaluation team has worked to circumnavigate this limitation. All key stakeholders have been interviewed, except the SNV WASH Sector Lead for Rwanda. The position is crucial for providing with adequate technical support to Isuku Iwacu partners. The person was out of country during the Mid-Term Evaluation (in-country investigations were postponed due to delayed Government approval). This is important for the External Evaluation to indicate the limitations in cross-checking data collection. As a matter of fact, direct evidence of proper in-house technical support could not be verified as expected (no WASH Sector Lead Skype Interview, no access to all technical support notes/reports), but from confirmation of key informants interviews only. As a mitigation measure, it was suggested to proceed to a Skype interview, but it was not feasible for some reasons. Further, two field visit reports from SNV USA/HQ were also collected. As a consequence, the Mid-Term Evaluation could not include a detailed review of in-house technical support and associated support. Nonetheless, the evaluation received confirmation from key informants that in-house technical support had been provided via meetings with the SNV WASH Sector Lead for Rwanda and other stakeholders, and analysis is based on this information. 4 5. Findings and Conclusions The main goal of the four-year Isuku Iwacu rural sanitation Activity transcends sanitation as a standalone topic, stating an objective “to improve access to and encourage correct, consistent use of household sanitation and hygiene facilities in order to decrease childhood stunting and support the Government of Rwanda’s (GOR) work in sanitation.”2 As the first project of its kind in Rwanda, Isuku Iwacu was designed to place SNV (and the consortium members) as facilitators for promoting market-based sanitation – rather than exclusively as implementers. Hence, this Activity was designed, implemented and adapted using innovative approaches. Considering this, some may view Isuku Iwacu as an experimental Activity – a program not intended to meet the quantitative targets outlined, but rather a program that is testing market￾based theories in Rwanda, trying to find the most appropriate, effective, efficient, and inclusive approach to ensuring Sanitation for All.3 Altogether, there is a need for learning throughout implementation, and furthermore, a need to document capitalization for internal processing and external sharing. Key informants highlighted that, internally, SNV has taken many steps towards learning, such as by commissioning several consultants to lead research in order to adapt Activity implementation. There are also plans to document capitalization in 2020 during the Activity’s last year of implementation – a beneficial step. Also supporting learning and quality programmatic adjustments is this Mid-Term Evaluation, which has interrogated the Activity’s three intermediate results through the five evaluation questions. Significant findings and conclusions of this analysis are presented in five related sub-chapters that refer to the Evaluation Questions. 1. Appropriateness of the Activity design Relation to other USAID Activities The Isuku Iwacu Activity was designed to be complementary to other USAID WASH, Health and Nutrition Activities (ongoing and planned at the time). While conceptually, this complementary function seemed relevant and with a good design, its implementation is less clear. The Mid-Term Evaluation finds an associated confusion that might cause overlapping efforts by USAID Activities (namely Isuku Iwacu and Gikuriro), or even worse, gaps in coverage on the ground. CHAIN and Gikuriro Given that this Activity ultimately has a health and nutrition-related objective, it falls under the overall umbrella of USAID/Rwanda’s Community Health and Improved Nutrition (CHAIN) Project Activities. One of those – Gikuriro, the Integrated Nutrition and Water, Sanitation, and Hygiene Activity – is “designed to operationalize USAID/Rwanda’s strategic goal of ending childhood stunting.”4 2 Cooperative Agreement: AID-696-A-16-00008/SNV, Isuku Iwacu Activity, Attachment B – Program Description, p.25 of 95 3 Rwanda’s Economic Development and Poverty Reduction Strategy II (EDPRS II, 2013-2018) includes the goal to ensure universal access to water and sanitation by 2018 (http://sanitationandwaterforall.org/) 4 Isuku Iwacu Program Description, p.30 of 95 EQ1: How appropriate was Isuku Iwacu’s initial activity design of building capacity and applying market‐based approaches to increase demand for sanitation products and access to sanitation services in the targeted districts and villages? 5 Therefore, “Gikuriro serves as the coordinator of all USAID nutrition and WASH investments authorized under the CHAIN project in its eight focus districts; as a result, the implementer of the Isuku Iwacu [SNV and the NGO consortium] should expect to work in close collaboration with Gikuriro to ensure that WASH results are achieved in the target districts via complementary and closely-coordinated interventions.”5 The Isuku Iwacu Activity determined its eight districts for intervention by picking from areas where Gikuriro was also working, allowing Isuku Iwacu – on paper – to build on Gikuriro’s work and objectives, essentially to promote an integrated, cost-effective, and more impactful rural sanitation Activity. Isuku Iwacu’s interactions with Gikuriro during the design phase were more plentiful; however, on the ground, it is unclear how well this collaboration worked over time (refer to EQ2). Three key components coming out of this integration are Gender, Behavior Change Communication (BCC) and Grants/Group savings synergy. 1. Gender Practically, this means that part of the initial design of the Isuku Iwacu Activity built on existing CHAIN and Gikuriro actions, such as the CHAIN Gender Analysis (2014) that provided the first basis of gender attention in the Activity. The Gender Analysis informed some of the gender and equity-related assumptions feeding into the Activity (see further EQ5). 2. Behavior Change Communication (BCC) It is clear that Isuku Iwacu, like Gikuriro, is following the GOR’s National Social Behavior Change Communication Strategy that focuses on a Community-based Environmental Health Promotion Programme (CBEHPP) approach. Gikuriro promotes CBEHPP through BCC outreach to raise awareness in the communities including through mass media campaigns, such as radio and theater – and Isuku Iwacu builds its activities on that existing momentum. Isuku Iwacu also works with Gikuriro-supported Community Health Clubs (CHCs) to channel messages of sanitation and hygiene into the communities and motivate members to improve their sanitation facilities. This collaboration demonstrates a good design approach to integrate Isuku Iwacu’s objectives with Gikuriro’s objectives, yet community members, village leaders and local authorities are unclear about who is doing BCC outreach (see further EQ2). 3. Grants/Group savings synergy The two USAID Activities are meant to work together on improving the sanitation supply chain, specifically through the “set-up of supply chain distribution systems (graduated [village savings and loan associations (VSLAs) / saving and internal lending communities (SILCs)] to acquire sanitation products from rural service centers for distribution in villages).”6 According to the Isuku Iwacu Program Description (PD), Isuku Iwacu meant to partner with Gikuriro “to strengthen financial links between graduated CHCs and SILCs/VSLAs to increase community￾based lending for local service providers.”7 They have not been targeted as intended so far (see further EQ2). Relevance to national policies and goals Isuku Iwacu is an innovative sanitation Activity that includes new market-based approaches and scaling up perspectives to reach total sanitation in rural areas of Rwanda. It started around the time that the Government of Rwanda released its National Sanitation Policy (NSP) in 2016, and due to shared goals, Isuku Iwacu gained local buy-in and ownership rather quickly (despite few challenges). 5 Ibid, p.31 of 95 6 Isuku Iwacu Activity, Collaboration Plan with Gikuriro Program, p.4 7 Isuku Iwacu Program Description, p.43 of 95 6 8 Reported in the FY17 AnnualReport 9 According to the NISR Census 2012 and the 5th Rwanda Demographic and Health survey (DHS–2014/15), as cited in the National Sanitation Policy, p.9-10 The Activity’s work to translate and help disseminate the NSP was a good output, very much appreciated by local stakeholders. Nonetheless, the Mid-Term Evaluation finds that key sanitation concepts did not trickle down from the national level. The revised Ministry of Infrastructure’s (MININFRA’s) 2016 NSP reflects the nuances associated with measuring and achieving Sanitation for All. It clearly explains the differences between sanitation ‘access’ and ‘coverage’ (refer to Glossary) and their link to affordability, market services, knowledge, and behavior change. However, while the NSP clearly expresses a solid understanding of sanitation access and coverage, in general, those concepts are lost further down at the sector, cell and village levels. There, just the goal of building latrines remains, although the NSP articulates the difference between building and having a latrine facility, and safely and properly using one in a way that promotes good health (i.e. coverage). In any case, the Isuku Iwacu Activity design and purpose relies heavily on the ownership and engagement of local stakeholders, including national and district governments and CHCs. The implications of their interest, awareness and capacity on sustainability are further discussed in EQ4. The Isuku Iwacu Activity defines itself, in part, as helping to push the NSP into action. As the largest sanitation program in the country, with ambitious objectives, Isuku Iwacu is therefore expected to bring a significant contribution to the GOR-led efforts to improve access to sanitation in rural areas of Rwanda. The Activity’s goals are in line with the NSP, and therefore, there was quick buy-in from the GOR. Despite few bumps with MININFRA in the planning process,8 national and district level authorities were on board from the beginning. In addition, performance contracts between the national, district, sector, and cell levels promote achievement of these national goals. All levels of the concerned local authorities/bodies interviewed perceive the Activity as a great supportive initiative to achieve sanitation progress, i.e. closing the sanitation gap. In all, this leads to a good working relationship with the national government, ministries and district officials, as they too have a vested interest in the Activity. Their buy-in (and the push from the GOR performance contracts) means they will most likely continue working to close the sanitation gap even after Isuku Iwacu ends their intervention. Finally, while the Isuku Iwacu Activity follows the revised JMP Sanitation Ladder (see Annex I), there are consequential discrepancies when comparing JMP to the NSP’s definitions of ‘improved’ and ‘unimproved’ latrines. These discrepancies might be due to the shift in the sanitation ladder that occurred when the MDGs transitioned and updated to the SDGs. The JMP definition follows the MDGs, whereas the NSP’s definition follows the UN Agenda 2030 SDGs. The implications of this on village level mapping and ZOD declarations are presented in EQ2. Activity design and assumptions This sub-section refers to analysis of the IR1 and IR2 Activity design components, while IR3 (Governance) is further reviewed in its own section, ‘Building Local Capacity.’ Relevance in increasing demand for sanitation products and services The Activity’s first intermediate result aims to increase demand for sanitation products and services. However, given that the vast majority of households (around 96%9 ) had latrines (improved and unimproved) prior to the Isuku Iwacu Activity – and the Rwandan government and other actors are actively working towards closing the sanitation gap (latrine access and coverage) – there was very little need to generate demand of basic access to sanitation. As a result, the Activity developed its action by generating increased willingness and ability to access to improved sanitation and to 7 maintain satisfactory standards (proper quality of construction, usage and excreta management). In that sense, the selection of the Designing for Behavior Change (DBC) Framework tool – that is, identifying the priority and effective determinants for behavior change – is very appropriate for the Activity BCC strategy development and implementation. Improved sanitation coverage as reported by the JMP was at about 75% in 2015 (rural: 71%, urban: 83%), but including shared toilets.10 Per the NSP: “Open defecation has practically been eradicated and most of Rwandan households have already financed and built their on-site private sanitation premises, albeit only about two thirds comply with the international standard definitions of an improved sanitation facility. Very few Rwandan households have installed flush toilettes. The prevailing practice remains that water is used for cooking and washing (grey water, discharged mostly on surface) while excreta are disposed with waterless latrines, which is a rational solution considering the scarcity of the average water supply and financial constraints.”11 Rather, based on the context – and as either uncovered or verified by this evaluation – there is need for Isuku Iwacu to promote (1) latrine cleaning with very little water, (2) improved hygiene habits, namely handwashing with soap, and (3) overall safe latrine usage (leading to proper excreta management). These topics fall under BCC and directly work towards Isuku Iwacu’s Strategic Objective of supporting “correct, consistent use of household sanitation and hygiene facilities” – that is, the safe, equitable and proper use of latrines for improved health and nutrition outcomes. Below, each of the three aforementioned BCC needs and focus areas are analyzed: 1. Latrine cleaning Many FGD and KII respondents raised concerns about the best methods to clean their improved latrines with very little or even no water (i.e. using tree leaves or branches). There were several reasons for raising such concerns: (a) little or no water supply in the community; (b) fear of filling the pits too fast, given the cost of building a pit latrine and in some areas, the difficulty in digging; and (c) little or no access to cleaning products in the village. While the Activity tried to engage communities to have clean latrines, the fact that respondents requested additional support from Isuku Iwacu may suggest that the Activity has not successfully supported the communities to solve such challenges, especially regarding nearby access to cleaning products and desludging services. While Isuku Iwacu-supported District Sanitation Center (DSC) have not helped much so far to create access to cleaning and other sanitation products for households, this might change after DSCs become fully operational (extent of change to be measured by the end of Activity accordingly). 2. Improved hygiene habits Handwashing with soap at the key handwashing times is a major indicator and contributing factor when it comes to measuring sanitation and hygiene and their associated health and nutrition outcomes. The Activity’s BCC strategy focused on handwashing, even promoting the installation and use of Tippy-taps, which was successful to a limited extent. During site visits, the evaluation noted few handwashing stations (with or without soap). Field visit pictures of hygiene promotion materials and obvious outcomes are provided in Annex J: Supporting photographs from site visits, under the header ‘Tippy-taps.’ 10 As cited in the National Sanitation Policy, p.9 11 Ibid, p.9 8 Being that handwashing with soap is as important as having access to an improved, clean latrine, any issues with water supply would therefore jeopardize sanitation efforts. Two villages visited during this evaluation had no nearby water source (Rugarama in Nyanza and Batikoti in Nyabihu). Where there is no water for handwashing and cleaning the latrines, the sanitation approach cannot be considered complete and little health impact should be expected. It is worthwhile to remember the well-informed and acknowledged links between Water, Sanitation and Hygiene (all together) and nutritional status, especially that the Activity’s Strategic Objective aims at decreasing childhood stunting. While the scope of USAID funding for this Activity did not include water supply, the absence of permanent access to water in some sites can be considered as a critical barrier for improved sanitation, since handwashing needs a stable water supply. In those villages, the relevance of Activity is reduced, as it should have taken water access into consideration or intervened elsewhere for an increased effect. USAID should consider this for future programming and program design. 3. Safe latrine usage Overall, this means that households in villages where Isuku Iwacu works should understand how to secure and ensure the proper use of improved latrines for all household members. Properly using an improved latrine means keeping it clean and handwashing with soap afterwards. These steps contribute to improved health, as overall, the sanitation facility and the environment remains free from contaminants (i.e. safe). All in all, during site visits, this evaluation noted many unimproved latrines and, as mentioned, few handwashing stations (with or without soap) near the vicinity of these (both improved and unimproved) latrines. USAID’s Multi-Sectoral Nutrition Strategy (2014-2025), as cited in the Isuku Iwacu PD, notes that “WASH addresses both the direct and underlying causes of stunting” and that “essential WASH interventions include handwashing with soap, treatment and safe storage of drinking water, and sanitary disposal of human feces.”12 Hence, gaps in hygiene behaviors and overall safe latrine usage will hinder the Activity’s outcomes. In addition to promoting safe latrine usage, Isuku Iwacu’s BCC messaging should continue to encourage the population to install or improve (as relevant) sanitation facilities. This messaging is in line with the Activity’s goals, and it means promoting different methods for sanitation financing, considering that the 2018 Formative Research found that lack of financial capacity was the biggest barrier to having an improved household latrine. Relevance in creating access to sanitation products and services The Activity’s second intermediate result aims to improve the supply and availability of sanitation products and services (refer to Glossary) in the private sector. It stresses on a critical shift from a traditional, heavily subsidized latrine construction approach to the enhanced ability of low-income families to build and maintain proper sanitation standards. This IR relevantly considers the business side (private sector knowledge of sanitation, existing supply, geographic location of shops) and consumer side (accessibility to the sites, affordability, interest in products and services). To do so, “Isuku Iwacu’s activities will include commercial approaches to understanding local demand throughout the value chain, financial flows in the sector, new product development, and supply chain analysis. Isuku Iwacu 12 Isuku Iwacu Program Description, p.30 of 95 9 will identify and bring in relevant examples from the region focusing on aspects that can be adapted for Rwanda’s primarily rural, low-income population.”13 Through a series of local studies, research and workshops, and trial-and-error with the private sector (both new and existing sanitation service providers as well as financial institutions), the Activity realized the original PD was not relevant to the context. Isuku Iwacu needed to study more, learn and adjust its strategy and approach. The following pages break down the different components of this approach, their purpose, progress to date, and how and why they have changed overtime. This learning curve explains many of the delays in working towards IR1 and IR2, and the Mid-Term Evaluation recognizes valuable efforts by Isuku Iwacu to improve sanitation by engaging the private sector. The Activity furthermore coordinated many national and regional learning workshops to discuss ideas and issues and set workplans, which helped in getting stakeholders interested and promoted knowledge sharing. 1. Voucher approach A primary component of Isuku Iwacu’s intervention was the development and application of a latrine voucher, given to households selected by local authorities, and ‘redeemed’ for the construction of an improved household latrine. Voucher beneficiaries needed to provide certain in-kind contributions, both helping to keep the cost of the vouchers down and promoting ownership of the works. In most cases, in-kind contributions included making mud bricks, collecting water for construction, and some other labor such as digging pits. This approach works towards sanitation access – that is, the availability of a private latrine at the household level, which is a primary goal of the NSP. Thus, the voucher approach supported national goals and district-level performance indicators, so district-level buy-in was smooth. The cost￾effectiveness and relevance of the voucher strategy to scale up rural sanitation, developed by the Activity, can nevertheless be questioned (refer to EQ3), as well as some reported issues in the in￾kind contributions and potential risk of exclusion of the most vulnerable households (refer to EQ5). Finally, the Activity’s development of a low-cost technologies manual and latrine prototypes addressed a relevant need and is appreciated by key stakeholders. Yet their installation and use (access and coverage) depend on the following critical factors: 2. Engaging the private sector To create access, Isuku Iwacu relevantly aimed to create a direct link between consumers (rural households) and suppliers (new and existing sanitation suppliers and construction workers). The Activity spent significant dedicated efforts to identify, train and engage with business owners, construction companies, entrepreneurs/masons, and the district Private Sector Federations (PSFs). It became clear over time that “private sector parties (unless they are consultants) are not interested in doing work for the project (e.g. design of loans and guarantee fund, market assessment studies, prototype design, draft Memorandum of Understandings, etc.), but rather are much more intrigued by potential lucrative business opportunities.”14 This means that the private sector – DSC operators, entrepreneurs/masons and construction companies (and even financial institutions) – will not go out of their way to design better Activity approaches with Isuku Iwacu. The majority have a profit-focused thinking that needs to be leveraged 13 Ibid, p.42 of 95 14 Isuku Iwacu New Private Sector Implementation Strategy, p.4 10 and applied to sanitation marketing. This lack of engagement in the planning and design phase put a great burden on Isuku Iwacu – as did the task of recruiting quality construction companies. For long-term durability of service, identification and enhancement of local private actors is a key factor. Construction companies that could be engaged and trained to build latrines were not found in every district; in some cases, the company was based in Kigali but agreed to do the works in the selected districts, using trained masons from that district. While the Activity trained the companies and masons, and outlined the terms of their agreement with Isuku Iwacu, some companies needed to be replaced after the voucher pilot phase due to substandard performance. On one hand, these actions ensured that the most motivated and qualified private sector actors were engaged in the Activity; yet on the other hand, finding and engaging them placed a burden on the Activity and caused some of its delays. In evaluating supply chain linkages, and in keeping in line with NSP goals, Isuku Iwacu aims to construct, equip and train/support District Sanitation Centers (some still ongoing). This valuable approach should both increase demand and access for sanitation products and services, as follows: - Demand would increase by generating community interest in the latrine prototypes on display, but for this, the DSCs need to be in centralized and easily accessible locations as well as supported by suitably promoted/advertised communication activities (including posters and display boards). - Demand would also increase through the trainings provided at the DSCs, which aim to further educate and engage the population. BCC outreach such as radio programs must keep informing the population about the DSCs to create more traffic. Moreover, the DSCs can further transform into greater learning centers by posting more flyers and sanitation information for visitors (such as updated sanitation ladder maps, information about open defecation risks and handwashing benefits, recent monitoring data, etc.). - On the supply side, the District Sanitation Centers serve to increase the community’s access to products and services by being a central point (or one-stop shop) for all sanitation-related needs. Given that the majority of households in Rwanda have latrines already, the DSCs main focus should – and will – be on promoting improved latrines, seat modifications for the elderly and persons with disabilities, and cleaning products – as well as linking consumers with entrepreneurs who can carry out such construction/latrine modification works. This purpose necessitates a business-minded DSC operator who would be motivated to promote improved latrines and other products if he/she (or his/her company) can derive a direct benefit or profit from the transaction. The Mid-Term Evaluation acknowledges that engaging the (local) private sector in the new business field of rural sanitation should be a long-lasting process, taking time and requiring repetitive efforts and patience. However, for the sake of increased effectiveness and sustainability of service delivery, there are some identified areas for improvement the Activity should take into account for the rest of the implementation phase. This is a particular area of attention for DSC success and long-term existence. A DSC opening late, still having no operator, or having an operator not invested in creating more demand and promoting profitable supply may not have the same kind of impact in another district. From the Mid-Term Evaluation timing and from the inspection of 50% of districts, below is a quick Best Practices description on how a DSC should become functional and effectively managed. At this time, some DSCs are not yet operational (refer to Annex J: Supporting photographs from site visits, under the header ‘District Sanitation Centers’). Such delays will minimize the current and expected impact of those DSCs, yet the following description provides an ideal scenario for the Activity to work towards: 11 While the extent of viable business perspectives remain to be closely monitored in the upcoming months, it demonstrates valuable quality standards of a good business model with increased chance of being sustained after the Activity ends. Isuku Iwacu has relevantly utilized Nyanza’s success model to train and promote others in the success factors and potential for business opportunities. 3. Enabling market access DSCs aim to be a tangible central point for outreach and marketing. However, the district approach is too wide to reach the most vulnerable rural households (living in remote villages, with no regular occasion to visit the districts). It is not apparent to what extent a village’s proximity to the DSC can impact their demand or access to sanitation products, but other factors play a stronger role, like affordability of the products/services and awareness of the DSC. The Activity (perhaps through local Civil Society Organizations) may consider reassessing the supply chain from the district to village levels to see where the gap comes; for instance, if cell-level traders should be engaged.15 Nonetheless, the Activity applied a relevant district-level focus to market-based sanitation approaches by engaging the private sector, “one of the key methods of promoting sustainable development and ensuring a lasting program impact.”16 Though, along with the fact that some construction companies were not based in their district of intervention, the Activity faced another challenge. 4. Enabling financial capabilities Critical to complementing BCC outreach and market access is the consumer’s ability to afford sanitation products and services. Isuku Iwacu was designed to help address “lack of financing as a factor limiting demand for sanitation and hygiene.”17 In this regard, the Isuku Iwacu Activity is undertaking two financial approaches: vouchers and loans. As already introduced, the voucher approach relies on subsidies in association with in-kind contributions from beneficiaries to construct the improved latrines. At this time, Isuku Iwacu is preparing to introduce sanitation loans to the communities, offered by MFIs. The Activity initially meant to engage with institutional banks via partnerships, per the PD; however, the Activity later learned that there was very little interest from institutional banks to venture into the sanitation market. This caused some delays for the Activity, again delaying progress towards targets. Isuku Iwacu then strengthened its focus on MFIs, and has recently signed two Memorandums of Understanding (MOUs) with ACB and AMIR – though no sanitation loans have been coordinated yet. Sanitation loans require that the population is aware and interested in them. To explore the views of potential consumers and begin loan marketing, the Activity led meetings with ACB and AMIR,as 15 As this has been tested in other areas of Rwanda, Isuku Iwacu should inquire with UNICEF about related lessons learned from that experience. 16 Isuku Iwacu Program Description, p.34 of 95 17 Ibid, p.37 of 95 The DSC operator in Nyanza (Neco) reported that Isuku Iwacu motivated him to enter the sanitation business, and provided him support to do so, like building the DSC he runs and providing him trainings on how to build the latrines. Neco is a construction company, and its owner (the DSC operator) also owns another shop that sells cement and iron sheets – two materials needed to build improved latrines. The biggest factor of his motivation is the fact that he is a business man; it is to his direct advantage to create more demand and support supply and construction. 12 well as outreach meetings in Busasamana sector in Nyanza.18 Another key step in loan marketing would be to target VSLAs, yet the Activity has not yet engaged those groups (planned for FY2020). To encourage potential consumers to take out sanitation loans (and even to promote self-financing of improved latrines), outreach must be adapted to this objective. For instance, BCC messaging needs to surpass reminding the population about the importance of sanitation and hygiene, and enter into a discussion about leveraging, for sanitation purposes, both personal finances and community-based VSLAs and Amasibo. This was intended from the beginning of the Activity, yet there is much more outreach needed to set up a sustainable framework for this financing approach. This evaluation noted little awareness by stakeholders and communities of such self-financing methods. 5. Building public latrines In addition, Isuku Iwacu planned to construct public latrines in each of the districts, the locations of which were determined by the district authorities. On the one hand, it is possible to view this activity as a way to support the public’s sanitation access; and with the local authorities deciding the location, Isuku Iwacu was positively applying a community-based approach. On the other hand, while looking at how to focus on most priority issues related to rural sanitation (i.e. public latrine construction as a higher environmental health need), it is relevant to consider how much money was invested into these public latrines that could have been instead channeled towards more voucher latrines or latrine improvements to help close the household (improved) sanitation gap. In this view, building and handing over public latrines may not totally fit into Isuku Iwacu’s goals, though it is part of the Activity’s scope. A specific issue observed during field investigations is the fact that “public” latrines visited in Nyabihu are in fact facilities for Jaba cell office staff and their visitors. The facilities were closed at the time of the evaluation visit. Much different than the Evaluation Team’s analysis, Isuku Iwacu’s opinion is presented here: some districts requested that these public latrines be constructed close to public institutions, such as markets, health centers, truck parking, bus stations, and local authority offices. The close proximity helps facilitate and ensure their proper maintenance and cleanliness. Without a certain level of ownership, the public latrines would not be sustainable. The fact that they were constructed at the cell office (as requested by the district) doesn’t take them away from being “public latrines”. The cell offices serve multiple uses for use by the public. To ensure the public latrines are maintained properly and kept clean, when staff in charge of maintenance/cleaning is away, the doors are locked. This is a standard practice and it does not mean the latrines are reserved only for Jaba cell office staff and visitors. The Mid-Term Evaluation cannot confirm this based on field observations and interviews. Therefore, while Isuku Iwacu’s construction of these latrines can be considered a way of keeping good relations with the local authorities, the Mid-Term Evaluation considers that the ones in Nyabihu should be redefined as ‘latrines for cell offices’ and are not considered as relevant for the scope of this Activity. This should remain a critical point of attention for future programming. Other public latrines visited were not operational yet (in Kabugeri cell in Ngoma district). In all cases, though, these facilities will require consistent management and regular cleaning. Looking at former experiences in the country, the durable and safe use of public toilets is very closely linked with the critical importance of being a sustainable business opportunity for operation and maintenance operators; hence the concern of suitability and sustainability to be further followed-up during the course of the Activity. 18 As reported in the FY19Q2 report; though this evaluation was unable to identify people (to interview) in Busasamana who had participated in such financial outreach. 13 Major delays or deviations from initial Activity design Discussions with all Isuku Iwacu Senior Managers highlighted the issue of some important knowledge gaps between Activity design (Program Description) and implementation. The Isuku Iwacu Activity has a variety of relevant innovative approaches, and KIIs highlight a common consensus about the challenges to adapt to the nature and purpose of some of the innovative approaches at the project start. Hence, this evaluation acknowledges Isuku Iwacu’s efforts to readjust Activity capacity and mitigate delays, in part by undertaking many studies to better tailor this market-based sanitation Activity to the realities on the ground. A New Private Sector Implementation Strategy was developed in March 2018, which readjusted the Activity’s direction as an ‘anticipated sanitation market facilitator,’ though aside from “technical gaps and persistent delays in the implementation of private sector activities,”19 it is not clear why this Strategy only came out in FY18Q2. The Activity would have benefited from having this defined Strategy earlier on or testing assumptions sooner in order to have more progress towards the Activity goals. Nonetheless, the New Strategy builds on the assumptions and Activity goals outlined in the Isuku Iwacu PD, and provides additional data and key steps for outlining the way forward with the private sector in Rwanda. Voucher subsidies Regarding the cost of the voucher latrines, the PD originally estimated that each voucher would be about 20,000 Rwandan Francs (RWF). The New Strategy realized that the voucher cost would need to be much higher, even with the in-kind contributions from the beneficiaries. The Activity needed to rethink its voucher approach, and ultimately settled on an average cost of 85,000 RWF per latrine voucher construction. There were then some deviations in this cost. In Nyabihu, due to the specific volcanic area, the price of a voucher latrine construction, requested by the construction companies, would be at minimum 100,000 RWF. This readjustment caused additional delays in the Activity implementation, though the voucher constructions seemed to progress fairly quickly after that, with minimal, manageable challenges and finally, quality outputs. To focus on quality and a private sector-driven approach, Isuku Iwacu took on many factors in the voucher latrine construction process that other sanitation approaches did not. Nevertheless, it is worth noting that through discussions with other WASH actors and local authorities in Rwanda, this evaluation identified other sanitation approaches (by Gikuriro, UNICEF and the District offices) that offered latrine ‘vouchers’ or assistance amounts from 15,000 to 35,000 RWF per household, with some reported success. These other agents provide the materials, and the beneficiaries (or Amasibo groups) mobilize to do the construction and installation (i.e. pit digging, making mud bricks, assembly of superstructure with roof and door). According to community members, doors and iron sheets are the most expensive items to secure; they usually cost around 22,000 RWF. Those other sanitation approaches leveraged community interest and Amasibo support to promote 100% latrine coverage in specific villages. This information suggests that Isuku Iwacu could have proceeded with a lower voucher cost (closer to 20,000 RWF as originally foreseen, at least not four times higher) that – if coupled with strong community mobilization and BCC outreach – may have been successful. From Isuku Iwacu’s opinion, it may have not contributed to sparking private sector engagement in sanitation and may have resulted in lower quality construction. 19 FY18Q1 Report, p.4 14 Alternatively, the Activity could have at least piloted such a set-up in different areas. This lower subsidy would have helped increase access to sanitation nationwide by reaching a higher number of households (this is further discussed in EQ2 and EQ3). Key to both types of approaches (high and low voucher costs) are the materials by which to build the latrines, and that is where Isuku Iwacu worked, trying to create a link between community members and suppliers of sanitation products and services. Engaging and supporting Civil Society Organizations (CSOs) Isuku Iwacu planned to engage local CSOs (LCSOs) from the beginning of the Activity, as explained in the PD: “With LCSOs, Isuku Iwacu will support districts and MFIs in co-designing loanextension plans that outline marketing strategies, loan support mechanisms, and follow-up activities to support group and individual loans in each district to increase sanitation and hygiene loan demand.”20 However, efforts to identify local CSOs have only been launched at the time of this evaluation, in FY19Q3. It is expected that CSOs will indeed fill the role as outlined in the PD, yet with just about 15 months of implementation left, Isuku Iwacu needs to put forward a strong capacity building and monitoring plan that builds CSO interest and capacity and also engages local authorities to work with them in order for this approach to be sustainable. Engaging and supporting MFIs The Activity recently realigned its engagement of financial institutions to focus almost exclusively on MFIs, as previously mentioned, although Phase 2: Loan testing of the New Private Sector Implementation Strategy has not started yet. This work with financial institutions like ACB and AMIR21 is still at a very early stage. Initially through institutional banks, Isuku Iwacu also meant to “establish a Sanitation Guarantee Fund (SGF) managed by a financial institution in each project district.”22 As of now, SGF actions are cancelled considering this shift from institutional banks to MFIs and lack of sufficient financing to push the SGF model. Even with the expected engagement of LCSOs in following-up sanitation marketing after Isuku Iwacu ends, this evaluation finds that there isn’t enough time left in the Activity to effectively test and sustainably carry out the Strategy’s remaining phases 2-3 (Loan testing and scale up). Scaling up requires, per the Strategy, “(i) an affordable product and (ii) an accepted loan scheme with clearly defined procedures, conditions and interest rates.”23 Isuku Iwacu “will need to carefully negotiate terms and conditions of future partnership agreements to ensure that their leveraged resources increase and [the Activity’s] guarantee percentage decreases, in order to gradually decline the guarantee to 0% by the end.”24 This objective requires thorough testing of the loan approach to avoid issues during the scale up, and may demand an extension of the Activity. Fecal Sludge Management (FSM) Sanitation in Rwanda is not complete without consideration of healthy, sustainable options for fecal sludge management, both at the household and commercial levels. This consideration was documented early on, in the PD, yet the Mid-Term Evaluation finds few actions to date that have been taken in this regard. Initially, “with private sector and GOR partners, the [Activity] will create environmental solutions, including safe sludge removal and risk mitigation steps to avoid contamination of the water table.”25 20 Isuku Iwacu Program Description, p.41 of 95 21 AMIR oversees MFIs and SACCOs in Rwanda 22 Isuku Iwacu Program Description, p.43 of 95 23 New Private Sector Implementation Strategy, p.5 24 Ibid, p.5 25 Isuku Iwacu Program Description, p.44 of 95 15 - Commercial Fecal Sludge management (FSM) Isuku Iwacu recently began exploring options for sustainable FSM in Nyanza district. The Activity reached out to Umuja Ni Nguvu Rwanda Cooperative, an agricultural cooperative in Nyanza, and trained them on desludging of latrines. However, this intervention does not assist households for the reasons of logistical constraints (no wide enough road/path for the trucks to access the property in villages) and lack of service affordability So far, Umuja Ni Nguvu’s clientele for this new task have been (mainly) commercial and public buildings: local institutions, schools, hotels, restaurants, and health centers and hospitals. The Mid-Term Evaluation observed that the financial and business development models are unclear to an interviewed focal point from this cooperative. This might be explained by the low level of undertaken work by the Activity on that topic (still at a very early stage). For quality and sustainability matters, Isuku Iwacu will need to allocate significant efforts in developing small-scale enterprise capacity for the service provider. There are also some critical concerns regarding the business model feasibility – and therefore viability – if there is little increase of domestic sanitation desludging services (doubts about development of household demand, except a possible adaptation to serve households down the road). The evaluation visited Umoja Ni Nguvu’s Sanitation Landfill in Ngorongari village, Busasamana sector (Nyanza district). The site has been operated by WASAC for two years. It treats solid and liquid waste, ultimately producing fertilizer which is sold. Isuku Iwacu recently provided some support to WASAC regarding this and other sites, helping to develop operational and business plans. At the time of the evaluation visit, the liquid waste processing was off. Key stakeholders on-site shared that due to low supply (inputs), the liquid waste processing part functions, but not daily, just when the operator comes. These efforts demonstrate long-term thinking for Rwanda’s sanitation, and are a good starting point for Isuku Iwacu to expand on, if the Activity wishes to prioritize commercial FSM. - Household Fecal Sludge management (FSM) Considering that the aforementioned ‘commercial FSM’ pilot option cannot reach villages (mostly logistics issues with no wide path/road for trucks, but also for affordability reasons), Isuku Iwacu should focus on ‘household FSM.’ The rural household-level FSM situation presently is alarmingly basic, with two simple yet risky options for when pit latrines are full: 1. Dig a new pit, make a connection to the first full pit to allow it to drain – and then use waste from the full pit as agricultural compost or discharge in the yard - Well-founded / locally appropriate considering that most rural households work in agriculture – but not a regular, accepted practice in all parts of Rwanda - Environmental health concerns observed during the Mid-Term Evaluation: • To what extent human waste are suitably treated through this composting process, how, and for how long? • On what types of crops are they adding the compost (fruits, vegetables, root crops)? • Do userstake precautions to protect themselves while in contact with the new compost? To protect their livestock and children? An interview with a Health Center indicated a high prevalence of intestinal worms within the rural population. The exact pathogen transmission path through agriculture has not been identified during the Mid-Term Evaluation, though evidence-based findings should be a great area for further sectoral learning, from which Isuku Iwacu can and should contribute. - If issues can be addressed, this option effectively merges sanitation, livelihoods and self-reliance and can offer greater interest from the population in proper sludge management. 16 2. Seal the full pit and construct a new latrine - Cheaper than hiring a private service to come empty the pit - Most rural villages are hard or even impossible to reach by desludging truck, when operating. - Not enough land to sustain digging new pits every 5-10 years - Not a sustainable option Considering the above information, the more sustainable option is to help set up a system where human waste from the pit latrines can be used for compost in a safe way. The activity could use this opportunity to collaborate with SNV’s agricultural projects in country. Although time is short, The Mid-Term Evaluation identifies this as a possible activity to implement in FY20. Mostly this effort from Isuku Iwacu would involve outreach/BCC on the risks associated with using waste for compost and subsequently, how to safely do so. 2. Effectiveness of the Activity Behavior Change & Communication The selection and use of the Designing for Behavior Change Framework makes a lot of sense26 considering the existing situation in rural Rwanda (almost 100% access to existing latrines; hence, need to focus on improved access and coverage). The DBC Framework calls for ‘1-Assessment, 2- Strategy Development, and 3-Communication efforts’ in order to ensure that BCC messaging is tailored appropriately to the context. Yet the Activity’s BCC implementation faced some consequential timeline sequencing and delays, notably a delayed Barrier Analysis process (with potential concerns on accurate identification of determinants for change): - Delays to the 2017 baseline survey pushed back the entire Activity, including delaying the Formative Research (carried out in FY18Q1) that helped shape Isuku Iwacu’s BCC agenda. This agenda for the first half of the Activity was derived from the 2017 baseline and 2018 Formative Research results, which were used to define priority groups (heads of households) and behavior statements. These documents, and first communication efforts, were applied in lieu of a standalone Barrier Analysis. - Eventually, a second (formal) Barrier Analysis has been implemented on the 3rd year of Activity, in FY19. It “will be completed in Q3 with sample data being collected in Nyanza, Nyabihu, Ngoma and Nyarugenge districts. The findings of the survey will be used to develop the final design of the behavior change framework and key BCC messages.”27 At the time of the Mid-term Evaluation, the Barrier Analysis is still under progress: raw data have not been analyzed yet. Implementation delays and inconsistent sequencing of the DBC Framework might deteriorate behavior change effects of the Activity, leading to lower outcomes. 26 World Vision, in charge of BCC tasks within the Isuku Iwacu consortium, developed the DBC Framework for WASH practitioners, 2010 27 FY19Q2 Report, p.7 EQ2: How effective are Isuku Iwacu’s current implementation strategies to successfully expand access to sanitation services? What changes could be made to increase effectiveness in the remaining two years? 17 On paper, as mentioned in EQ1, the two USAID Activities (Gikuriro and Isuku Iwacu) were relevantly intended to collaborate closely, (especially in the field of BCC). In reality, the different tasks (Who does What, Where and When?) are rather unclear. In that regard, strengthening collaboration between Gikuriro and Isuku Iwacu is an identified key room for improvement: - As mentioned in EQ1, despite a good design approach to integrate Isuku Iwacu’s objectives with Gikuriro’s, community members, village leaders and local authorities are unclear about who is doing BCC outreach. This makes the effects of the work harder to review, except by assessing WASH outcomes, i.e. looking at the BCC outreach goals, such as latrine usage and handwashing, without paying attention to who distributed the messages. In doing so, the evaluation noted some gaps in hygiene behaviors and related infrastructure (refer to EQ1) which are expected to hinder the Activity’s outcomes unless soon rectified. - Given that the Activity should work in close collaboration with Gikuriro, Isuku Iwacu’s BCC approach was meant to complement Gikuriro’s radio awareness and other communications. However, BCC is inherently difficult to quantify and measure as such. Additionally, The Mid￾Term Evaluation could not clarify in detail the exact division of labor between the two Activities and if (and to what extent) the BCC messaging took any difference in content or form. For instance, during village visits, many residents reported hearing some Gikuriro radio messaging; on top of that, some key informants and Isuku Iwacu reports mention mass-media BCC messaging transmitted through the Urunana soap opera. It appears that bothGikuriro and Isuku Iwacu are using mass media to improve existing sanitation and hygiene practices, noting again that open defecation is low in Rwanda. Therefore – as reported by the communities themselves during site visits – Isuku Iwacu messaging focused on informing the population that sanitation is a basis for health (thus, aiming to promote safe latrine usage). The Mid-Term Evaluation can acknowledge that BCC is inherently difficult to quantify and measure as such. However, the main associated risk of low clarity for each Activity’s contribution for each village/cell/sector/district is on effective monitoring of Isuku Iwacu, with an associated confusion that might cause overlapping efforts by both Isuku Iwacu and Gikuriro, or even worse, gaps in coverage, making the Isuku Iwacu Activity less effective. As a consequence, the actual contribution of Isuku Iwacu on promoting rural sanitation in the eight districts is hard to capture in this evaluation. Moreover, it appears not possible to clearly measure to what extent Isuku Iwacu’s current implementation BCC strategy is effective in successfully expanding access to sanitation services, especially through Isuku Iwacu on its own or through benefit of similar Gikuriro BCC efforts. Nonetheless, according to FY18 reports, Isuku Iwacu ultimately did mobilize communities through BCC and by working through Amasibo in the communities to construct or improve (as relevant) latrines. Site visits during this evaluation verified that some households in the selected villages were mobilized to construct or improve their own latrines, with their own funds, after Isuku Iwacu began intervening in their community (see Annex J: Supporting photographs from site visits, under the header ‘Enabling financial capabilities, coupled with BCC/outreach: Evidence of self-financed latrines’). Furthermore, Isuku Iwacu needs to continue focusing on ramping up its BCC approach as part of the new LCSO strategy. By engaging multiple LCSOs, BCC activities will have further reach and greater impact. The Mid-Term Evaluation recommends providing support to the LCSOs in this process. World Vision, in charge of BCC within Isuku Iwacu consortium, will complete completing its involvement on Isuku Iwacu by March 2020, i.e. six months before Activity completion. This brings some concerns on Isuku Iwacu capacity to keep on BCC efforts. Findings from field visits strongly suggest that efforts towards behavior change should be pursued and even reinforced for the rest of the Activity. BCC needs repetition and must be contextual. Otherwise, as seen in a ZOD 18 pre- declared village during a site visit (in Kicukiro), there are voucher latrines that are dirty, with many flies and no handwashing station or soap. On-going contributions and monitoring towards Zero Open Defecation (ZOD) The Mid-Term Evaluation finds some consequential discrepancies when comparing the revised JMP Sanitation Ladder (Annex I), that the Isuku Iwacu Activity is reported to follow, and the NSP’s definitions of ‘improved’ and ‘unimproved’ latrines. This has had implications on Isuku Iwacu￾supported village level mapping and related ZOD pre-declarations, likely linked to the fact that since the finalization of Isuku Iwacu’s Program Description, the Millennium Development Goals (MDGs) transitioned to the Sustainable Development Goals (SDGs). This affects the sanitation ladder and definitions. The changes are summarized in the figure below. 19 Information from Isuku Iwacu: The project follows the set guidelines and procedures of the Government of Rwanda’s National Sanitation Policy which align with the new SDGs. Additionally, the adoption of a ZOD process at the village level is a unique and new concept being piloted only by USAID/Rwanda. Villages move from a ZOD pre-declaration to a ZOD declaration. During periodic village monitoring visits, conducted by the cell SEDO, the head of the village and CHC’s facilitators, and heads of Amasibo at selected Amasibo zones, the team monitors changes on villages sanitation status. The team uses a ZOD Verification Checklist for Household monitoring, allowing them to track and record the number of newly constructed or rehabilitated latrines and comment on next steps required to move households up the sanitation later. Following the visit, the team summarizes the status of the entire village using the ZOD Verification Summary Report, showing the global sanitation picture for the entire village. This form is used to pre-declare a village as ZOD. Villages are pre-declared ZOD when all houses in the village have at least an on-site toilet which separate human excreta from human contact, regardless of the standard/ quality. Isuku Iwacu then conducts a final transect walk to verify if the village can be pre-declared as ZOD. Then the period of post pre declaration starts to ensure households upgrade the standards of their latrines (e.g.: Hole that’s deep enough and reinforced, Platform that is washable and stable, Superstructure for privacy and protection from rain, Door or curtain that provides good privacy and safety (especially for women and girls), Wiping materials available, Ventilation if necessary, Safe for children under five, Elderly, Disabled) and culture of cleanness is maintained. The actual ZOD declaration is the responsibility of each district and involves all the local authorities, including MININFRA, MINALOC, and MOH. After deciding on and setting the criteria for declaration, the mentioned authorities conduct a thorough verification and follow-up with the village(s) regularly for close to a year. Once all the residents of the village have adopted the proper culture and are using latrines, then the village is fully declared ZOD. Actual ZOD declaration is at the discretion of the local authorities, MINIFRA, MINALOC, and the MOH. Although not yet known, it is likely that they will use the SDG definition to focus on getting all households to have “basic sanitation services”. To date, no villages have been fully declared ZOD under Isuku Iwacu, only pre-declarations have occurred. While this evaluation believes that ZOD declarations in Isuku Iwacu villages should imply 100% improved latrine coverage (per JMP definitions), it in fact seems to mean 100% latrine coverage, which is in-line with the Government of Rwanda’s requirements. However, this appears to go against the goal of the Isuku Iwacu Activity to help people gain access to improved household sanitation. Evaluation findings and discussion: The table below highlights the discrepancies in the ZOD process identified and summarized by this evaluation. JMP Sanitation Ladder applied to Isuku Iwacu, per the PD GOR NSP Village mapping of Sanitation Ladder under Isuku Iwacu guidance ZOD process under Isuku Iwacu 1. Safely managed (Improved) Improved Improved28 (green) Pre-declaration of 2. Basic (Improved) ZOD 3. Limited (shared – Improved) Unimproved 4. Unimproved including Unimproved (blue) 5. Open defecation Open defecation Open defecation (red) n/a 28 There was a case of visiting a woman in Bisambu village in Kicukiro district (Kigali) who had received a voucher latrine from the Activity, and was sharing it with her neighbor who did not have a private latrine. In this case, if Isuku Iwacu is considering that they constructed for this woman a private improved latrine, then her neighbor would be considered as openly defecating, and thus, the village level sanitation ladder would place limited (shared) latrines as ‘improved.’ 20 These discrepancies in the definitions of improved latrines and ZOD villages make it challenging to monitor and report against the objectives stated in the Program Description, leading to one seemingly non-rigorous ZOD pre-declaration process nor a uniform/standard one (it might be a national issue). According to Isuku Iwacu’s ZOD Physical Verification minimum standards for latrines and hand washing facilities constructed,29 there should be washable slabs and handwashing stations next to latrines, for instance, neither of which is always the case in latrines categorized as improved under Isuku Iwacu. There should furthermore be ‘cleaning every day’ of the latrines, or at least, they should have been clean the day of the (unannounced) ZOD monitoring inspection visit. During the site visits, the Evaluation team visited villages pre-declared, but noted some unclean latrines, depicted in Annex J: Supporting photographs from site visits, under the header ‘Evidence of unclean, unsafe latrines.’ In principle, the ZOD pre-declaration and declaration process includes using the Isuku Iwacu Verification Checklist and conducting a transect walk with local leaders’ support. Also, there should be a ceremony by the district office to declare the village ZOD, but some villages were pre-declared without any reported district support. Hence, this evaluation finds that ZOD pre-declaration and follow-up are irregular and/or not rigorous (i.e. lacking district support; and not going deep enough into latrine use and cleanliness). At the village level, the key factor of sustainability to the village leaders, they said, is whether or not the latrines are clean, though site visits revealed that this is often an issue (and as mentioned earlier, latrine cleanliness and proper use need to be greater focus areas of BCC).This is likely due to the fact that villages are still in the “post pre-declaration” phase. The goal of the Isuku Iwacu Activity is to create access to improved household sanitation, as defined by the JMP. ZOD is applied to give villages a sanitation goal and the Activity encourages and trains community members and leaders, and then tracks and reports on it. Isuku Iwacu uses ZOD pre￾declaration as a ‘sign of success’ as those villages are on the way to becoming ZOD villages. Some of these latrines are ‘unimproved’ for few reasons, and per the revised JMP definitions, these reasons are discussed here: 1. Pit latrines being categorized as unimproved because the brick walls are not plastered and there is no door and/or lock, only a curtain for privacy, yet those latrines offer environmental protection because they prevent the feces from seeping out; and 2. Pit latrines that are not effectively protecting humans and the environment from feces due to poor pit design or slab structure. According to the JMP-defined ‘improved sanitation,’ the first type of pit latrines would be considered improved – while the second type listed above would be considered, indeed, unimproved. Yet latrines not effectively protecting humans and the environment against feces were identified in villages pre-declared ZOD by the Isuku Iwacu Activity (see further Annex J: Supporting photographs from site visits, under the header ‘Unimproved latrines and implications for ZOD’). This may be an issue related to monitoring of the Isuku Iwacu Activity. Field visits revealed that village latrine maps were not updated regularly (e.g. monthly), and discussions with village leaders pointed to inconsistencies in SNV Monitoring, Evaluation and Learning (MEL) data (see Annex C: Site visit selection). 29 Part of the ‘ZOD Verification Checklist for HouseholdMonitoring’ 21 While this evaluation team believes that ZOD pre-declarations in Isuku Iwacu villages should imply 100% improved latrine coverage (per JMP definitions), it in fact seems to mean 100% latrine coverage, which is in-line with the Government of Rwanda’s requirements. However, this goes against the goal of the Isuku Iwacu Activity to help people gain access to improved household sanitation. The adoption of a ZOD process at the village level (i.e. training, village mapping, action plans, verification through transect walk, etc.) is a unique and new concept being piloted only by USAID/Rwanda. However, such discrepancies in the ZOD definition, technical criteria, and monitoring process must be addressed to strengthen this process. Critically, the Activity must consider where to include JMP-defined unimproved latrines versus the GOR NSP guidelines. This clarification process must involve all local stakeholders, as USAID/Rwanda will need to engage local ministries and authorities to reaffirm the ZOD process (from the national and district levels all the way down to the cell and village levels). The Mid-term Evaluation recommends that when declaring villages ZOD, the GOR and local authorities only declare if the village has reached an improved level on the sanitation ladder. The Isuku Iwacu Activity should continue to facilitate this process and ensure villages are declared ZOD defined by “improved sanitation.” Market-Based Approaches / Sanitation Marketing The Activity would have been more effective if there had been early installation of the DSCs, links with LCSOs, and engagement with MFIs, SACCOs and VSLAs to promote sanitation loans from the outset of the Activity implementation. Regarding improved access to financial products (loan, saving groups, etc.), it is not apparent that VSLAs were targeted as intended – or they have not yet been targeted. The work with MFIs / AMIR / ACB / SACCOs is still at a very early stage (with various potential challenges). The New Private Sector Implementation Strategy (2018) points out that local savings groups should be engaged, starting with Phase 2: Loan testing (this has not been significantly launched yet). The Activity engagement with these groups in FY2020 is therefore expected. Notably, the sooner this outreach can start, the better chance the Activity will have in successfully promoting sanitation loans. Meanwhile, Isuku Iwacu targeted Community Health Clubs (CHCs) to engage them in community mobilization to stir up community interest and awareness in sanitation facilities and hygiene behaviors – but they have not yet been engaged to help increase community-based lending. FGD participants gave no mention of this, even participants who are members of local savings groups. Moreover, if Isuku Iwacu tried to target CHCs or other groups to stir up community-driven sanitation and hygiene, the objectives were not clear enough. As an example, one focus group of CHC members reported that they were waiting for a distribution of materials to make liquid soap; this indicating that they might hardly grasp the concept of community-driven approaches. Beyond the issue of planning work on DSC coming on a late notice during the course of Activity, openings of DSC come with large delays, eventually limiting any effective access to sanitation products and services. This might also jeopardize actual ownership of District for now, i.e. a low level of business activity leading to a low conviction about its relevance from local authorities, leading to low ownership at the end). In general, it is too early for the Mid-Term Evaluation to measure the impact of DSCs (if they are being visited, by whom and how frequently; if community access to products and services has increased). This is critical to consider those points moving forward in order to make the Centers as effective as possible. So far, this approach does not appear to have substantially increased the community’s access to sanitation products and services and a project extension may allow sufficient time for the Activity to address the gaps. When FGD participants were asked if the availability and choice of sanitation 22 products and/or services has changed since Isuku Iwacu intervention, all groups replied that products are not available in the local market at village level. Data show that there are little to no sanitation supplies in the villages (often just soap; no cleaning supplies), and no village-level link to shops elsewhere, such as the DSCs. The available materials found at the district and even sector levels do not manage to reach the villages, despite this being a shared objective of both Isuku Iwacu and Gikuriro. All of the aforementioned findings indicate measuring success in this area at this stage of the Activity implementation remains limited. 3. Performance, Impact and Sustainability Prospects Progress and Performance to date The Mid-term Evaluation focuses on a qualitative approach; assessing the Activity progress is therefore based on review of reports and data, complimented by KIIs, FGD findings and field observations. Such data indicate an estimated overall progress of 20-25% of the outcome indicators by FY19 Q2, i.e. after 31 months of project corresponding to 60% of completed project timeline. The large delays faced by the Activity, especially during the first part (Inception and Implementation Phase 1), are well acknowledged by Isuku Iwacu and can be pondered by the challenges of working on innovative approaches for a sector (sanitation) that deals with (personal and cultural) sensitive issues. Sanitation programs are often further challenged with reluctant engagements from all types of stakeholders (raising female hygiene issues or shifting from subsidized latrine to self-financing approaches are very typical of sectorial challenges in that regard). Isuku Iwacu efforts to bring the proposed innovative ways to foresee 100% sanitation coverage is fully relevant. In return, the important shift from heavy subsidy to more and more self-financing approaches comes with many obstacles from institutional, community and private stakeholders. As the first project of its kind in Rwanda, Isuku Iwacu was designed to place SNV (and the consortium members) as facilitators for promoting market-based sanitation – rather than exclusively as implementers. Hence, this Activity was designed, implemented and adapted using innovative approaches. Considering this, some may view Isuku Iwacu as an experimental Activity – a program not intended to meet the quantitative targets outlined, but rather a program that is testing market￾based theories in Rwanda, trying to find the most appropriate, effective, efficient, and inclusive approach to ensuring Sanitation for All. As an innovative program, key informants highlighted that, internally, SNV has taken many steps towards learning, such as by commissioning several consultants to lead research in order to adapt Activity implementation. There are furthermore plans to document capitalization in 2020 during the Activity’s last year of implementation – a beneficial step. Nevertheless, considering the delayed implementation plan, that should include ambitious efforts to meet an acceptable level of target, the Activity may consider the opportunity of an extension to have more time to work towards achieving its targets with high quality. With an extension, the Activity can also meet the need for further learning throughout implementation, readjustment of financial approaches as needed, and document capitalization for internal processing and external sharing. EQ3: To what extent have the different implementation approaches been successful? Which of them appear to be sustainable and should be prioritized going forward? 23 Targeting the lowest poverty quintile Isuku Iwacu’s latrine vouchers are intended for the poorest and most vulnerable 10% of households. The PD states outright that this voucher approach would target households from the poorest quintile, Ubudehe 1 (U-1). However, site visits confirm that Isuku Iwacu provided voucher latrines to households from Ubudehe 1 and 2, and in some villages, U-3 (Isuku Iwacu indicated that corrective measures were taken and all U-3 households were replaced with U-1 and U-2). The process by which this happened is unconfirmed, yet this information sheds light onto beneficiary selection by local authorities (refer to EQ5). Sanitation loans, as planned, will target households from all Ubudehe categories, as reported by key informants: ACB will work with U-3 and U-4 households, and AMIR, for the construction of new latrines, will target their own clients in U-3 and U-4 categories and for improvements to existing latrines, AMIR will target U-1 and U-2 customers. Since it is more difficult for U-1 households to get loans from institutional banks, Isuku Iwacu’s partnership with AMIR (and SACCOs) will promote access to (some) financing for the lowest quintile for latrine improvements. Alternatively, U-1 and U-2 households can self-finance latrine construction/improvements alone or by going through community-based VSLAs and Amasibo for sanitation loans; hence, the Activity should target those savings groups in FY20. Prospects of Activity approach on 100% improved sanitation Risk of counter-productive or inefficient ‘Subsidized Voucher vs. Self-financing’ approaches Considering that the loan approach encourages community members to self-finance their own improved latrines – without Isuku Iwacu subsidies – the Activity’s preceding voucher approach risked creating an aura of assistance. Households not targeted by Isuku Iwacu vouchers are reportedly waiting for the next round of voucher assistance to get latrines or improve their own. This diverging approach takes a different form in each district, even in each cell and village, due to factors related to socio-economic class, distance from the DSC, amount of BCC outreach received, and mobilization generated. As mentioned in EQ1, the voucher amount was significantly readjusted during the first phase of the Activity (almost four times higher than initial design). The Activity opted for a clear focus on quality rather than coverage for access to sanitation facilities. In fact, the latrine vouchers were found to be of good quality during site visits. While the voucher cost is much higher than initial design, this certainly affects progress towards target achievement. Having a lower voucher cost would allow for increased coverage, promoting ZOD while focusing more on community-driven approach (this being a successful model reported by other WASH actors). A lower voucher cost could also result in lowered construction quality and less interest from the private sector. Nonetheless, planning on half of the actual voucher cost, i.e. about40,000 RWF (which is still higher than other WASH programs in the country), would lead to double the amount of targeted beneficiaries with the same allocated Activity budget. However, the Activity timeline seems now too tight to realistically shift to another voucher amount; yet this cost reduction for greater coverage should be strongly considered for future sanitation programs. The voucher pilot occurred roughly as intended but was then quickly scaled up – reportedly because the Activity had already faced many delays and it was time to really launch the intervention. Meanwhile, piloting of sanitation loans has been pending since the PD was written. As a preface to loan testing, the Activity has signed two MOUs and began exploring the views of potential consumers through outreach activities. 24 Risk of counter-productive ‘Nationwide vs. Adapted’ financial approaches With two distinct financial approaches, the Activity cannot expect the results of these diverging approaches to be uniform nationwide. The Activity did not pause long enough to learn from the voucher pilot phase and readapt the approach, and it could have benefited from doing so. The challenges and successes vary incredibly by district – even by cell and village. Considering this, the Activity may prefer to adjust slightly the sanitation loan approach to the specific differences and needs of the sector and cell levels – as opposed to carrying out a nationwide approach. Thus, the Activity should pause to learn from the pending sanitation loan pilot. Reflecting on that analysis, a case study of how different approaches may work better in different areas – as opposed to the voucher approach applied in a standard way in all eight districts – is presented in the box below. Batikoti village in Nyabihu has recently been included in the Activity. There are currently 24 voucher latrine constructions ongoing, and even after they are done, there will still be a gap in coverage of improved latrines. Here, a lower voucher cost would have allowed Isuku Iwacu to increase latrine coverage in order to close this gap. With more coverage, the village would see better vector control, and thus, improved health and nutrition outcomes. However, should Isuku Iwacu construct more voucher latrines in Batikoti to close this gap or should the Activity promote self-financing mechanisms? Many of the households in Batikoti reported having accounts at SACCO already. Of course, the terrain in Nyabihu is volcanic, so latrines need to be adapted to the soil. This is where Isuku Iwacu’s far- reaching work researching and producing a variety of low-cost latrines comes into play. The Activity has identified the improved models best suited to this village and other villages in the area. However, this village is far from the DSC, and likely, village residents won’t travel there to see the latrine prototypes. Furthermore, sand for making bricks is not available in this area. So the village residents – without Isuku Iwacu – remain uninformed of new prototypes and without access to materials. This is where more sanitation marketing is critical. Business-oriented construction companies and local masons, supported by Isuku Iwacu, would have the incentive of profit to market new, relevant improved latrines to these households, who could be mobilized to self-finance them. In this context, the loan approach coupled with heavy BCC and supported by private sector-driven sanitation marketing might have been the better model, from the start (this is planned for the FY2020 workplan as part of the LCSO approach). BCC should, and will, focus on educating the population about the importance of improved latrines for better health and should promote handwashing and proper latrine use and cleaning, for sustained outcomes. 25 Long-lasting local capacity In terms of Capacity Building, EQ4 describes the significant efforts of Isuku Iwacu as well as the related challenges encountered. At this stage of implementation, reviewing the Activity’s contribution for enhanced national and local capacity highlight the issue of local authorities showing little knowledge of how to scale up innovative sanitation approaches – not suggesting that the intervention has been inappropriate in its design, but rather creating concern for the Activity’s overall sustainability. Strengthened national and local capacity building actions on those innovative directions should be therefore prioritized for the rest of the Activity, and more time may be needed to focus on effective sanitation marketing, provision of loans, and awareness of and access to sanitation products and services (down to the village level). 4. Building Local Capacity National and district governments As part of IR3, the Activity has provided hands on support to the government. Isuku Iwacu has found a very good engagement with national stakeholders, being that the Activity aligns with the NSP and thus, also district-level goals. Local officials have incentive to encourage Sanitation for All given the targets in their performance contracts, and Isuku Iwacu is seen as helping work towards them. Improving capacity Defined in the Activity’s original design, “capacity building is about building skills, knowledge, institutions, and incentives that can make development processes more durable and adaptable.”30 Yet the key objectives for Isuku Iwacu’s work under IR3 differ from capacity building in the traditional sense, which often focuses on knowledge sharing and incentive and skills building. Rather, the Activity has focused on what is better suited for the Rwandan context, which is facilitating sustained access to sanitation products and services by partnering with local stakeholders. These stakeholders include local governments, LCSOs, construction companies, and financial institutions. In summary: “Adequate institutional arrangements and capacities need to be in place for the sustained delivery of high-quality sanitation services. Isuku Iwacu will work to facilitate local ownership and engagement at all stages of planning and implementation, allocation of resources, and ongoing management responsibility for service delivery. … Taking a multi-stakeholder approach, including public, private, education, and civil sector actors, Isuku Iwacu will create systems driven by Development Plans and Sector Strategies, integrated into the Joint Action Development Forum and district-level equivalents.”31 As mentioned, fortunately, ownership and engagement by the national and district governments were already present; thus, the Activity needed to focus its work on engaging other local actors (i.e. LCSOs, construction companies and financial institutions). These key players, with Isuku Iwacu support, should work together to complete the supply chain for sanitation market access, and capacity building should thus focus on encouraging a business-focused (or entrepreneurship) mentality. 30 Isuku Iwacu Program Description, p.35 of 95 31 Ibid, p.45-46 of 95 EQ4: To what extent has Isuku Iwacu been successful in building capacity to improve sustainable access to sanitation? 26 However, aside from the gaps in the supply chain already noted, this evaluation found little local understanding of how to support ‘durable and adaptable’ sanitation. Few key informants understood the importance of various self-financing approaches and how to scale up (or expand) sanitation in development. This information highlights a gap in local “official’s capabilities to support private sector involvement in sanitation and hygiene supply chain” (Output 3.1.3). Likely, this is due to the delays in rolling-out loans and identifying financial institutions to engage with. To close the sanitation gap, district staff mentioned that they also have their own latrine assistance programs whereby they provide materials to households for latrine construction. These efforts are no doubt fruitful; however, if the GOR and USAID/Rwanda want to both shift from assistance (i.e. vouchers) to development, this requires a stronger focus on (1) testing loan and other self-financing approaches (i.e. through VSLAs) and (2) ensuring completion of the sanitation supply chain all the way down to the cell level. This change in mentality is still ongoing. The evaluation finds that deeper sanitation concepts like adjusting/scaling up financing approaches, ensuring coverage (vs. access), inclusion (EQ5), and verifiable monitoring haven’t fully reached all community members. Government officials conducted community outreach and explained the content of WATSAN polities and strategies. However, these concepts are currently lost at the sector, cell and village levels. There, just the goal of building latrines remains – which is partly effective. Although the Activity supported local officials to target households from the poorest quintile (U-1) and the study conducted on low-cost WASH technologies emphasized affordability and safety, the Evaluation Team found these concepts still lost at lower levels. As such, this evaluation recommends this be a component of messaging and training going into FY20. Therefore, as Isuku Iwacu adapts and tests new approaches – such as the forthcoming loan approach – it must continue to engage with national and district-level officials to build their capacity in durable approaches. This is in fact incorporated in the Activity’s FY19 workplan: “Local government officials will play an influential and persuasive role in pushing Ubudehe 3 and 4 to adopt and use improved sanitation facilities.”32 Improving data and information management The other side of improving local stakeholders’ capacities in market-based sanitation is the “use of mechanisms for data analysis and harmonized standards for sanitation improvement” (Outcome 3.2.2). Having harmonized standards is where this evaluation finds some of its most concerning issues, namely in the consistency of the ZOD pre-declaration process (as elaborated in EQ2). In the context of Isuku Iwacu, there are discrepancies between the ZOD approach taught and the realities on the ground. These inconsistencies are simply explained in the following table, adapted from the FY19 workplan of the ZOD training agenda/goals:33 32 FY19 Workplan, text box p.4 33 Ibid, p.13 27 Training topic Evaluation results Village mapping, and reporting the change on sanitation ladder situation in the village Village mapping follows three sanitation categories: improved latrines, unimproved latrines, and no latrine (open defecation), which is a simplified version of the JMP Sanitation Ladder. In site visits, though, the evaluation found that many village mappings had not been updated for several months, despite the fact that they should be updated monthly. The use of improved latrines to attain ZOD Local leaders did not understand the ‘ZOD’ terminology, but understood that they should achieve 100% ‘green’ (improved) latrine coverage. It is good that they recognize the importance of improved latrines, given that ZOD itself (as defined in the Activity) implies any latrine access (improved and unimproved). Train village committees on construction of hand washing stations (through BCC) This evaluation found diverse evidence showing that Isuku Iwacu promoted the creation of Tippy-taps using locally sourced materials for handwashing. However, actual ‘coverage’ of Tippy-taps is substandard. During site visits, many visited latrines did not have a Tippy-tap nearby, and if they did, many had no water nor soap. There were few examples of success. Develop an action plan for sanitation improvements in our village Local leaders were aware of the work needing to be done in their communities to close the sanitation gap, yet formal action plans were not presented. This is due to the fact that, although action plans were completed and provided to Isuku Iwacu, no copies were kept at the cell level. Going forward, the Mid-term Evaluation recommends a copy be kept with local leaders. Additionally, timeliness seems to be an issue, if district and cell-level officials have competing priorities when it comes to supporting the villages in working towards ZOD. Considering the above discrepancies, this evaluation questions the process of ZOD monitoring, highlighting the need for standard definitions and approaches to ZOD pre-declarations, declarations, and verifications, supported by further capacity building with local authorities on this topic. Understanding that the ZOD approach is new in Rwanda, its definition and technical criteria should be revisited for improved monitoring and data management. Additionally, as the concept is new, it needs to be incorporated in the health and environmental policy and harmonized with the revised Community Based Environmental Promotion Program (CBEHPP). Village leaders and local groups Key village-level stakeholders are the CHCs, local leaders, savings groups, and Amasibo. Contextually, these are the people who will help Isuku Iwacu achieve its goals all the way down to the households. Much like there is room for improvement in terms of the knowledge and capacity of district-level stakeholders, these groups need more capacity building and support to push improved Sanitation for All (thus, ZOD). Much of their engagement thus far has been based on the results of the Formative Research report, and they now need to be further mobilized to encourage self-financing approaches, proper latrine use, and handwashing with soap – all of which are key elements for sustained sanitation and health outcomes. 28 Ultimately, the CHCs are responsible for community health, and their role has been drawn out from the beginning. “Although sanitation hasn’t been the main focus over the past few years, the CHCs provide an excellent platform to promote sanitation improvements.”34 The CHC president is responsible for updating the village sanitation mapping and working with the village leader and cell-level officials. Notably, a recent study concludes that CHCs had no effect on diarrhea reduction rates in Rwanda; thus, no effect on anticipated health outcomes.35 The study also moderates their findings by saying that results should be different if improved sanitation and handwashing stations are available. With CHCs being used as primary focal points for community health – and with improved sanitation relying on a multitude of factors like water supply, hygiene, financial capacities, and access to sanitation products and services – this evaluation highlights the importance of strengthening CHC capacity. On the same note, Isuku Iwacu should strengthen the capacity of local leaders, savings groups (i.e. VSLAs) and Amasibo to encourage sanitation coverage. This finding builds on an existing point already in the Formative Research report, stating that: “A referral mechanism comprising local leaders should be initiated at scale, to provide information about availability of low-cost and affordable sanitation products/services, and access to saving plans and small loans (with the information about materials credit/installment payment terms).”36 Next steps Isuku Iwacu was able to build on the GOR’s interest in and ownership of Sanitation for All, which trickled down to the district and even cell level officials, even if terminologies and deeper sanitation goals did not. The Activity needs to continue capitalizing on this ownership by giving more technical support to local officials to promote latrine coverage and verifiable monitoring according to the Government’s and therefore the Program’s ZOD standards. At the village level, Isuku Iwacu must further leverage CHCs, local leaders,savings groups, and Amasibo to educate communities about sanitation, health, nutrition, and market access points, and mobilize them to self-finance improved latrines for all. This includes leveraging LCSOs in FY20. Moreover, Isuku Iwacu has demonstrated methods for engaging and training existing construction companies and local masons/entrepreneurs. While procurement processes may have been tedious, the Activity kept strict requirements to ensure that contractors delivered high quality latrines. Companies and individual masons were mobilized in sanitation works, trained and used in the construction of voucher latrines. Looking ahead, just like with CSOs and MFIs, they must take a role in sanitation marketing to produce a well-rounded sanitation market. Given that these companies and entrepreneurs have a business profit-focused mentality, they would be encouraged (like the DSC operator in Nyanza) to promote sanitation works all the way down to the village level. 34 National Sanitation Policy (2016), p.9 35 Sinharoy, Sheela S., et al. “Effect of community health clubs on child diarrhea in western Rwanda: cluster￾randomized controlled trial.” Lancet Glob Health 2017; 5. 36 Formative Research Report (2018), p.20 29 5. Gender and Equity EQ5: How has Isuku Iwacu’s approach taken into consideration issues of gender and equity? Gender Isuku Iwacu needs to conform to USAID’s Policy on Gender Equality and Women’s Empowerment, which requires USAID-funded programs to integrate gender equity in their work through ‘an inclusive approach’ that considers the needs of men, women, boys, and girls equally. This means thoroughly researching and taking into consideration contextual differences in gender roles and responsibilities, associated barriers, and special needs of these different groups – and creating a program that addresses those differences. This approach is also known as gender equity, gender equality and gender integration (refer to Glossary). This approach requires that gender-specific considerations should be mainstreamed from the beginning (at the program design phase), helping to give programs greater focus to the specific needs of population groups earlier on, so the activities can be better tailored. Isuku Iwacu, by targeting all population groups, managed to include women, men, boys, and girls; still, the Activity did not take additional steps to design programming around their specific needs. The full extent of gender inclusion in the Activity is explored below. Relevance to gender policies and local understanding Alongside USAID’s Policy, the Government of Rwanda has furthermore recognized the importance of gender equity. The National Gender Policy sets a contextual framework for understanding of the importance of including women and girls. As a direct result, national and district-level authorities are already aware of the importance of streamlining consideration for women and girls in programs. When interviewed, many local key informants could express that the GOR requires attention to females. Yet to them, the extent of which this attention can be translated into Isuku Iwacu’s work just means giving access to sanitation to everyone. Thus, USAID concepts of gender-sensitive approaches are not apparent in local stakeholders’ definitions of gender equity. To local authorities, if you target everyone for sanitation, you have included gender. Gender integration To address the gender-related barriers outlined in many key research reports that have influenced the development of the Isuku Iwacu Activity, one action stands out: “BCC activities should reach men and women (heads of households and their spouses) to increase the likelihood that men will approve of and take part in saving, acquire loans, and constructing improved latrines in their households. This is important, because research showed that community outreach campaigns are attended predominantly by women.”37 Addressing BCC outreach and sanitation marketing to both men and women helps educate the entire community, rather than keeping one sex removed from activities, which can be detrimental to family dynamics. This inclusive approach would reach all heads of households and encourage joint decision-making – including men so they can encourage saving for sanitation purposes (including acquiring/paying back loans) and including women, as historically, they are responsible for child￾rearing, and having them involved in decision-making (not just outreach) has a direct correlation on improved health outcomes in the family.38 37 Ibid, p.20 38 USAID CHAIN Gender Analysis (2014) 30 It appears that Isuku Iwacu has taken steps to target both men and women in outreach, considering that a lot of its BCC messaging go through mass-media campaigns that target entire communities. In instances where village members are unaware of DSCs, saving for sanitation, loan regulations, or anything else related to sanitation and hygiene, it is the entire village that is unaware – not just men or women. From the site visits, this information reflects the delays in Isuku Iwacu’s activities or the smaller-than-expected DSC reach, rather than lack of gender inclusion. Looking deeper, helping address female-specific needs falls under ‘gender’ considerations. Though not defined in the scope of the Activity, the Mid-Term Evaluation recommends that Isuku Iwacu add steps to target female- specific sanitation needs as gender consideration, specifically attention to menstrual hygiene management (MHM) in this context. MHM and sanitation go hand-in-hand, as women’s sanitary needs are unique and demand methods for their management. These needs include water access, disposal facilities and handwashing – in or directly near the vicinity of the latrine. A group of female FGD respondents cited that, since the latrine pits are hard to dig, they do not want to dispose of their sanitary pads in them; they prefer to bury them elsewhere. This concern, plus a noted lack of handwashing facilities during site visits, keeps safe and convenient MHM as a need for women. Regarding access to products, this Activity places a large focus on improving access to sanitation products, which include MHM materials, i.e. sanitary pads. Isuku Iwacu may consider in future supply chain mapping looking at if MHM materials are available and affordable, where, and what local options are for disposing of them. Equity The Isuku Iwacu Activity offers no distinct definition for equity. This evaluation argues that equity is when everyone gets what they need for sanitation – as opposed to equal (or blanket) approaches where everyone gets the same thing (refer to Glossary). A program that considers equity hence considers the needs of people with disabilities (PWD), the elderly, children under five, youth, orphans, the poorest households, and any other vulnerable population group. Efforts to address the specific sanitation needs of PWD, the elderly and children under five have recently been launched. For instance, Isuku Iwacu is preparing to distribute wooden seats that should be placed on top of a pit (latrine) that will aid these groups to sit down to use the latrine. Though relevant, these efforts could have been started earlier on. Other efforts to target vulnerable groups have focused on targeting the poorest households, those from the Ubudehe 1 category, in the latrine voucher selection and in facilitating access for U-1 households to sanitation loans. The Evaluation team could not confirm that all U1 households in the targeted villages have been assisted with an improved latrine Concerns about the beneficiary selection process and social class categorization highlight a risk for Isuku Iwacu in achieving sanitation access for the most vulnerable, as outlined below – and lack of updated village mappings provide substandard clarity for monitoring changes in coverage. Beneficiary selection for the voucher subsidies Isuku Iwacu’s vouchers are intended for the poorest and most vulnerable households, that is, households from U-1. During site visits, though, it came up that Isuku Iwacu provided (or was providing currently) voucher latrines to households from Ubudehe 1 and 2, and in some villages, U￾3. FGD participants (community members) and some key informants highlighted that the selection of beneficiaries for the voucher distributions should be improved to take into account the views of community members and leaders. FGD respondents didn’t trust the social class categorization because there could be “many mistakes” in the classification (mentioned in four of the six FGDs). 31 While this approach builds on local ownership of the Activity – with a draft of beneficiaryselection coming from the community leaders, and the final selection list coming from each district’s Mayor – the data show that there is a risk of selection bias and favoritism. This risk may be inherent to the Activity, but it is worth noting. The latrine construction workers do verify that the person in the household is the same as the person on the selection list provided by the Mayor, which is good, but this check does not verify the household’s Ubudehe classification and need. Concluding remarks In all, Isuku Iwacu needs to modify or expand its approach to a more inclusive one in order to achieve the above gender and equity objectives. This may include updating and endorsing the 2018 draft Gender Plan. While women and girls have been targeted through the activities realized by Isuku Iwacu, it is because the Activity targets all households (thus, females were not excluded). Still, more efforts could have been done to ensure the Activity’s consideration of the menstrual needs of women and adolescent girls in an attempt to fully incorporate ‘gender’ needs. Similarly, other vulnerable groups were not excluded from the Activity, but specific steps to include them only came later. Distributions of the pit latrine seat modifications for persons with disabilities, the elderly and children under five were planned at the time of the evaluation, and should occur in the next quarter (FY19Q4). Distributions will not meet the needs of all vulnerable groups (there are not enough seat modifications for all people in need), and that is where Isuku Iwacu’s sanitation marketing and access to sanitation products will work to fill this gap. 32 6. Learning and Capitalization The Isuku Iwacu Activity encountered many challenges and delays, but it also took informed steps to mitigate them, learn from them and adapt. To date, the Activity has many achievements and is recognized by local stakeholders for its work in helping close the sanitation gap in Rwanda by 2024. This Mid-term Evaluation was able to identify several key areas from which the Activity can further learn, as presented below, organized according to lessons learned to date as well as areas for further learning and advocacy. This analysis should help feed into Activity capitalization. 1. Lessons learned to date 1. Piloting financial approaches Per the PD, Isuku Iwacu was meant to pilot in three districts both the voucher and loan approaches. The latrine voucher pilot did occur, but was then quickly scaled up. In doing so, the Activity did not have time to learn from the pilot phase challenges; and it could have benefited from doing so. On the other hand, the sanitation loan pilot did not happen and is currently in the early stages rollout phase at the time of this evaluation. As the remaining project time is limited, Isuku should quickly gather lessons learned at the end of AMIR’s pilot phase and incorporate into the second phase of the sanitation loan roll-out. This is further justification for a time extension. Given what Isuku Iwacu has learned thus far – that Rwanda is incredibly diverse in its terrain, affecting sanitation product types, and that sanitation market access and affordability are complex undertakings – the Activity should test sanitation loans in a smaller scale first. Then, Isuku Iwacu can readjust, as needed, loan marketing and packages before expanding to new areas. Based on learning from the voucher approach and the pending sanitation loan pilot, the Activity may prefer to adjust slightly the sanitation loan approach to the specific differences and needs of the sector and cell levels – as opposed to carrying out a nationwide approach. 2. Ambitious targets This Activity is undertaking an innovative approach to sanitation marketing. Consequently, it needed more time than expected to carry out quality research/studies, recruit staff, solidify government engagement, and test sanitation approaches. Hence, in recognizing this, the targets could have been reduced. Simply evaluating this Activity by its Monitoring, Evaluation and Learning (MEL) indicators does not depict the whole story; rather, it gives hints of an unsuccessful Activity and does not reflect Isuku Iwacu’s learning curve. 3. Learning within the Rwandan WASH sector This evaluation finds various evidence that highlights the low level of consistency within theWASH sector in Rwanda, as some WASH actors work in sanitation on much cheaper approaches (refer to EQ1). It is therefore strongly suggested that the WASH sector in Rwanda should agree on a uniform approach for voucher support on latrine construction. Likewise, the sanitation sector will benefit from similar approaches on how to support U-1 and other vulnerable groups (refer to EQ5), with obvious values on advocacy to the national and local authorities on innovative scaling-up approaches, market-based approaches and sanitation marketing. There is overall a need for improved collaboration within the WASH sector to share learning and best practices. This would be an opportunity for Isuku Iwacu to share its expertise, lessons learned and best practices. 33 2. Topics to assess for further learning and advocacy 4. Health-related impact The overall goal of the Activity is to contribute to improved nutrition status through increased sanitation coverage. This goal will be effective and confirmed when WASH-borne disease reduction is stated. However, the Activity does not require, nor has collected data on health outcomes. For broader monitoring, the Activity relies on the robust Demographic and Health Surveys. While communities and local leaders indicate the decrease of diseases, there is no formal evidence. Five of the six FGDs (except for the one with beneficiaries whose voucher latrines are currently under construction) mentioned that [sanitation-related] illnesses/diseases and deaths have reduced, though it is certainly a possibility that participants are explaining the expected response, and there is no data to verify their perceptions. For these reasons, an evidence-based survey (showing the direct impact of Isuku Iwacu on nutrition) can be planned for the last months of the Activity. In addition to being an essential process for accountability and quality learning, the related findings will clearly inform about the actual link and effects with the expected nutrition goal. This end-line must look beyond the presence of latrines, instead investigating handwashing rates (with soap), water access, health and nutrition outcomes (including health data collection from health centers), proper latrine use (coverage), latrine cleanliness, barriers to achievement, beneficiary satisfaction, inclusion of the most vulnerable groups, rates of self-financing and factors of success, etc. The findings would be very beneficial and will support USAID/Rwanda in assessing actual effectiveness of coordination and matching with CHAIN and Gikuriro initiatives. As an innovative sanitation Activity, it also makes sense to inform with evidence-based outcomes related to health. Eventually, this will be a great opportunity for USAID/Rwanda and Isuku Iwacu partners to promote best practices and lessons learned. This will also be a decision-making tool about relevance of further advocacy for scaling up opportunities. 5. Impact evaluation USAID/Rwanda can explore the possibility of an Ex-Post Impact Evaluation to strengthen learning for future programming and advocacy. This impact evaluation would look at all Isuku Iwacu Activity aspects: access to sanitation products and services, self-financing and loan approaches, the role of local governments, leaders and groups, including CHCs and VSLAs, latrine coverage, urban and rural sludge management models, public latrines, health and nutrition outcomes, gender and equity inclusion and needs, etc. Such an evaluation would examine the durability and adaptability of Isuku Iwacu 1-3 years after the Activity ends, assessing the longevity of the Activity aspects, how they’ve changed over time, and what outcomes are sustained. Given that the Isuku Iwacu Activity is the first sanitation project of its kind in Rwanda, an Ex-Post Impact Evaluation would be a tremendous learning opportunity. 34 7. Recommendations Based on the key findings and conclusions, and upon further discussion with SNV and USAID/Rwanda, this Mid-term Evaluation offers several verified recommendations to help improve the final year and a half of the Isuku Iwacu Activity. Some of these recommendations and points for learning (as explained above) would benefit from a time extension of the Activity implementation. In addition, the evaluation outlines some recommendations for future WASH programming. 1. Quality completion of the Isuku Iwacu Activity 1. Continue BCC/outreach activities in FY20, including further training and encouraging CHCs, local leaders and Amasibo to mobilize their communities and follow up. Isuku Iwacu’s BCC can continue to be tailored to the Activity’s unique market- based approach, promoting sanitation marketing and touching more on sanitation financing methods (saving for sanitation, loan access, etc.), while also promoting certain sanitation cleaning products and emphasizing some hygiene-related topics of sanitation technologies. In further detail, this recommendation includes: • First, readjusting communication messages from 2019 Barrier Analysis findings (not yet available at the time of the Mid-Term Evaluation), through appropriate use of the DBC Framework (identified determinants for behavior change). • Secondly, priority BCC activities should concern the following (not comprehensive list): - Scaling up handwashing station (with soap) near the latrine, with high priority to villages with low handwashing (tippy-tap) rate - Safe and environmentally sound MHM - Access to DSC for sanitation services and products - Options on how to empty latrine pits, e.g. safe composting - Satisfactory ZOD status community-wide 2. Explore the relevance of other business model options available in the country (such as cell-level traders) to expand local access to sanitation products and services at sector/cell/village levels via deeper supply chain mapping. Taking into account feedback from beneficiaries, this can include strongly suggesting to the DSC’s that they include preferred MHM materials (although beyond the scope of the project) and contextually relevant latrine cleaning products. 3. In collaboration with the local authorities and the GOR, reaffirm and strengthen the process to reach and declare ZOD status (defined as “improved sanitation”) in the targeted villages, and continue to strengthen local/district monitoring and verification capacity and support the Ministry of Health through ongoing revisions of the CBEHPP – including measuring if all Ubudehe 1 households have been assisted with improved latrines. Strengthening the ZOD process would also improve the accuracy of information flow to Isuku Iwacu, enabling better reporting. 35 4. Given the few but serious discrepancies and issues reported, the Activity should consider strengthening its monitoring and conducting more monitoring activities to assess/follow-up the quality of built latrines (and their ongoing proper use), handwashing, the number of voucher latrines in certain villages, and beneficiary satisfaction (for issues or inequities). Clearer reporting and monitoring would help clarify the exact contributions of the complementary USAID Activities (e.g. in BCC), the extent of ZOD, community mobilization efforts, and the final number of voucher latrines constructed (and later, loans supported). 5. Considering the amount of time left in the Activity, act quickly to pilot the sanitation loan approach via AMIR and SACCOs. If piloted alongside more outreach to raise local awareness, the loan approach may find an early success (and incidentally, more outreach can spark further self-financing by households, thus working to close the sanitation gap). Isuku Iwacu must also incorporate learning into the loan approach after the pilot phase of AMIR’s implementation is complete, taking into consideration differences in village populations, and re￾evaluating the budget to discuss whether or not reintroducing the voucher approach in certain villages might be more effective. More time would allow the Activity to better understand and implement its financial approaches. 6. More capacity building efforts need to be done with LCSOs, MFIs, local leaders and community groups, service providers, and local authorities in order to leave a solid framework in place after the Activity ends. These efforts need to leverage local interests and focus on information flows (top-down), consistent monitoring, and sanitation marketing (including by the service providers themselves, e.g. masons from the villages and DSC operators). Such capacity building efforts should include stronger understanding and monitoring of sustainable market-based sanitation approaches and ZOD at the sector, cell and village levels. For instance, as Isuku Iwacu adapts and tests new approaches, such as the loan approach, it must continue to engage with local stakeholders to build their capacity in durable approaches. 7. Encourage districts to fully operationalize District Sanitation Centers as soon as possible to increase access to sanitation products and overall Activity effectiveness. The districts should further support the DSCs to increase sanitation marketing and visibility through a detailed and timely follow-up of business development plans. 8. Further review the effectiveness and impact of the Isuku Iwacu innovative approaches and develop best practices, including of the various financial and sanitation marketing approaches, the ZOD/100% Sanitation process (i.e. facilitation of community mapping, declaration, and follow-up), business models for fecal sludge management and public toilets, etc. This should be a part of the capitalization component of the FY20 workplan, also supported by strengthened monitoring. 9. Consider requesting a project extension from USAID, based on conditions to: (1) improve Activity delivery (reaching targets and more capacity building of the various stakeholders), and (2) document learning outcomes and best practices coming from this innovative approach and share them within the WASH Sector and through advocacy. 36 2. Future WASH programming 10. Address water supply if not secured to gain health and nutrition-related outcomes. This will effectively work towards securing proper handwashing with soap practices and increase the WASH-related nutrition and health status within the communities. 11. For SNV, as a nexus of its programming, promote safe practices for emptying latrine pits in villages, including the safe reuse of human waste as compost in agriculture when locally feasible, and explore the potential for integrated Sanitation and Livelihoods programs to enhance affordable access to sanitation services and products for the vulnerable/poorest populations. 37 8. Utilization Plan This mid-term evaluation is intended primarily for SNV, USAID/Rwanda and the Isuku Iwacu NGO consortium members, Water for People and World Vision. It is intended as a learning and guidance tool, serving to document challenges, successes and lessons learned in the Activity thus far as well as highlight areas for improvement relevant to the quality completion of the Isuku Iwacu Activity. In order to maximize quality results in the remaining year and few months of Isuku Iwacu implementation, SNV should immediately discuss in greater detail these findings and recommendations with the following actors: • Isuku Iwacu consortium members – to map out the best way forward; • Gikuriro focal points – to map out the best way forward; • Rwandan ministries and local government focal points at the district, sector and cell levels – to reinforce gaps in local capacity that may hinder the promotion of quality sanitation works, including reaffirming ZOD monitoring methods at the national level and quality monitoring methods (vs. exclusively quantitative monitoring) at the district level and below; ‘triggering’ behavior change for sanitation behavior change and CHC follow-up; and pushing for business-oriented DSC ownership and guidance; • USAID – to consider how the Activity can meet quantitative targets in the time left, and if an extension is both relevant and necessary; and • Other WASH actors in Rwanda, such as UNICEF and the Japan International Cooperation Agency – to share learning from this Activity and discuss the potential for uniform national approaches in market-based sanitation and monitoring. Ultimately, the results of this mid-term evaluation should be used not only to help SNV (as the lead NGO) achieve Isuku Iwacu’s Strategic Objective of improved access to and support of correct, consistent use of household sanitation and hygiene facilities – but also to promote learning in the East Africa region and globally. List of Annexes The following annexes are attached separately: A. Evaluation Design Matrix B. Methodology C. Site visit selection D. Detailed schedule of field phase E. Interview log F. Data collection Interview Protocol G. Key Informant Semi-Structured Interview Questionnaire H. Community FGD tool I. JMP Sanitation Ladder J. Supporting photographs from site visits