March 2022 This report is made possible by the support of the American People through United States Agency for International Development (USAID). The contents of this report are the sole responsibility of Navanti Group, LLC and Social Impact, Inc. and do not necessarily reflect the views of USAID or the United States Government. EFFECTIVE WATER, SANITATION & HYGIENE (E-WASH) ACTIVITY FINAL EVALUATION FINAL REPORT Nigeria Monitoring, Evaluation, and Learning (MEL) Support Activity i | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY EFFECTIVE WATER, SANITATION, & HYGIENE ACTIVITY FINAL EVALUATION FINAL REPORT Nigeria Monitoring, Evaluation, and Learning Support Activity (MEL Support Activity) Contracted under 72062021C00002 Submitted to: Dr. Chidimma Anyanwu, Contracting Officer’s Representative (COR) United States Agency for International Development/Nigeria Plot 1075 Diplomatic Drive, Central District Area Abuja, Nigeria Copied to: Dr. Joachim Ezeji, Program Manager Water Resources Management and WASH United States Agency for International Development/Nigeria Office of Economic Growth and Environment Mr. Oladele Kolade, Agriculture and Environment Specialist United States Agency for International Development/Nigeria Office of Economic Growth and Environment Submitted by: Joy Garba, Deputy Chief of Party, Nigeria MEL Support Activity The Evaluation Team Titilola Bright-Oridami Boluwaji Onabolu, PhD Victor Malaolu ii | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY ABSTRACT The United States Agency for International Development awarded the Research Triangle Institute (RTI) International the five-year Nigeria Effective Water, Sanitation, and Hygiene (E-WASH) Activity in 2018. The primary purpose of this final performance evaluation was to determine whether the E￾WASH Activity achieved its stated development objectives and to understand lessons learned. The evaluation team (ET) developed questions that assessed the Activity’s role in supporting state utilities’ commercial orientation, financial and operational viability, professional management, city￾wide sanitation mapping, and use of environmental and construction guidelines. The ET collected qualitative data from November 29 – December 18, 2021, across five Activity states (Abia, Delta, Imo, Niger, and Taraba), and the Federal Capital Territory and triangulated it with secondary qualitative and quantitative data. The Activity’s results were evaluated based on how they met the following desired outcomes: 1. Creating professionally managed and commercially oriented State Water Boards (SWBs), now State Water Corporations (SWCs); 2. Improving the financial and operational viability of SWBs (now SWCs); 3. Strengthening policy, institutional, and regulatory frameworks for improved water, sanitation, and hygiene (WASH) services; and 4. Building national and state WASH advocacy, coordination, and communications for reform. The results suggest that the Activity was fairly successful in supporting utilities’ strides toward commercial orientation and professional management through the implementation of a suite of activities. The sequencing of implementation, insufficient infrastructure, capacity challenges, as well as utilities’ high non-revenue water, low-cost recovery, and poor water quality inhibited the realization of operational and financial viability. As such, service delivery improvements were minimal, and water was still not being produced in Abia. Professional management practices and transparency improved, and accountability structures were in place across the five utilities, but sustainability is uncertain due to funding and capacity challenges. Despite foundational progress with WASH policies and city-wide sanitation mapping commencement, the Activity’s sanitation interventions did not achieve their objectives due to insufficient prioritization by utilities, utilities’ human resources sanitation capacity challenges, and sequencing and timing issues. The Activity’s implementation of environmental and construction guidelines showed mixed results, with good community understanding but a clear need for dedicated in-house health and safety and environmental oversight for the implementing partner to facilitate adaptive management. iii | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY CONTENTS Executive Summary .................................................................................................................................................. vii Background and Purpose.................................................................................................................................... vii Methods.................................................................................................................................................................. vii Findings and Conclusions.................................................................................................................................. viii Key Recommendations...................................................................................................................................... xiv Introduction ................................................................................................................................................................. 1 Background.............................................................................................................................................................. 1 Context .................................................................................................................................................................... 1 Activity Objectives................................................................................................................................................. 2 Evaluation Purpose and Audience ...................................................................................................................... 3 Evaluation Questions (EQs)................................................................................................................................. 3 Methodology................................................................................................................................................................ 4 Overview.................................................................................................................................................................. 4 Research Design..................................................................................................................................................... 4 Data Analysis and Synthesis................................................................................................................................. 6 Quality Control Mechanism for Data ............................................................................................................... 6 Guiding Principles and Values.............................................................................................................................. 6 Limitations................................................................................................................................................................ 7 Findings, Conclusions, and Recommendations.................................................................................................... 7 Evaluation Question 1........................................................................................................................................... 7 Evaluation Question 1A......................................................................................................................................14 Evaluation Question 1B ......................................................................................................................................16 Evaluation Question 2.........................................................................................................................................17 Evaluation Question 3.........................................................................................................................................22 Evaluation Question 3A......................................................................................................................................24 Evaluation Question 4.........................................................................................................................................28 Evaluation Question 5.........................................................................................................................................33 Implementation of the MTE Recommendations and Descoping ..............................................................36 Annex I: Full Listing of References and Reports Utilized................................................................................40 Annex II: Full Listing of Persons Interviewed.....................................................................................................41 Annex III: Data Collection Tools..........................................................................................................................42 Annex IV: Scope of Work ......................................................................................................................................69 Annex V: E-WASH Activity Final Performance Evaluation Workplan.........................................................84 Annex VI: E-WASH Performance Indicators.................................................................................................. 100 Annex VII: Table of Excerpts From CWIS Exercise Report....................................................................... 102 Annex VIII: Examples of Priority Actions From Sanitation Mapping Reports......................................... 104 Annex IX: RTI International Team E-WASH Approach Flow Chart........................................................ 105 Annex X: Disclosure of Any Conflicts of Interest......................................................................................... 106 iv | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY LIST OF TABLES AND FIGURES Table 1: KII and GI Participants by Gender.......................................................................................................... 4 Table 2: FGD Participants by Gender.................................................................................................................... 5 Table 3: Infrastructure Improvements by State................................................................................................... 9 Table 4: E-WASH State Collection Efficiency by State (Indicator 1.2.1).....................................................17 Table 5: Capacity Utilization Efficiency................................................................................................................18 Table 6: CWIS Implementation by State.............................................................................................................30 Table 7: Progress by State* ....................................................................................................................................37 Figure 1: Operating Cost Coverage Ratio........................................................................................................... ix Figure 2: Map of States with E-WASH Operations and RTI’s Head Office in Abuja ................................. 1 Figure 3. Capacity Utilization Efficiency by State ................................................................................................ 9 Figure 4: Drinking Water Quality Compliance Rate from Effective Utility Management (EUM) Primer Performance Measures............................................................................................................................................11 Figure 5: NRW Rates by State ..............................................................................................................................11 Figure 6: Operating Cost Coverage Ratio..........................................................................................................12 Figure 7: Number of Utility Staff per One Thousand Connections in E-WASH States..........................19 Figure 8: OCAT Strength Index by State............................................................................................................19 Figure 9: City Wide Inclusive Sanitation Process..............................................................................................29 v | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY ACRONYMS BOD Board of Directors COVID-19 Coronavirus Disease 2019 CSDA City Services Delivery Assessment CSO Civil Society Organization CSTF City Sanitation Task Forces CWIS City-Wide Inclusive Sanitation DESUWACO Delta State Urban Water Corporation DMA District Metered Area DO Development Objective E-WASH Effective Water, Sanitation, and Hygiene EMMP Environmental Mitigation and Monitoring Plan ERP Enterprise Resource Planning System ESMP Environmental Social Mitigation Plans ET Evaluation Team EQ Evaluation Question EUM Effective Utility Management FGD Focus Group Discussion FCT Federal Capital Territory FMWR Federal Ministry of Water Resources FSM Fecal Sludge Management FY Fiscal Year GI Group Interview GIS Geographic Information System GM General Manager GoN Federal Government of Nigeria HR Human Resources ISWSC Imo State Water and Sewerage Corporation IP Implementing Partner IPSAS International Public Sector Accounting Standards ISWASC Imo State Water and Sewerage Corporation JMP United Nations Children’s Fund KII Key Informant Interview LAN Local Area Networks LGA Local Government Area LoP Life of Project MD Managing Director M&E Monitoring and Evaluation MEL Monitoring, Evaluation and Learning MOU Memorandum of Understanding MTE Mid-Term Evaluation NGN Nigerian Naira NORM National Outcome Routine Mapping NRW Non-Revenue Water OCAT Organizational Capacity Assessment Tool OCCR Operating Cost Coverage Ratio PRC Performance Regulatory Committee PBC Performance-Based Contract vi | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY Q Quarter RTI Research Triangle Institute SDG Sustainable Development Goal SFD Shift Flow Diagram SOW Scope of Work SURWASH Sustainable Urban and Rural Water Supply, Sanitation and Hygiene Program SWB State Water Board SWC State Water Corporation TAWASCO Taraba State Water and Sewerage Corporation TBD To Be Determined TSA Treasury Single Account TWG Technical Working Group USAID United States Agency for International Development USD United State Dollar USF Utility Support Fund USG United States Government WASH Water, Sanitation, and Hygiene vii | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY EXECUTIVE SUMMARY BACKGROUND AND PURPOSE The United States Agency for International Development (USAID) awarded RTI International the five-year Nigeria Effective Water, Sanitation, and Hygiene (E-WASH) Activity in 2018. The E-WASH Activity is part of USAID’s wider efforts to build closer coordination with the Government of Nigeria (GoN), including the Federal Ministry of Water Resources, select state water boards (SWBs), and allied stakeholders. The Activity aims to advance broad-based economic growth and resilience through improved water, sanitation, and hygiene (WASH) services in urban areas. The primary purpose of this final performance evaluation is to determine whether the E-WASH Activity achieved its stated development objectives, to understand lessons learned, and to reflect on the findings and recommendations from the mid-term performance evaluation (MTE) and the subsequent descoping of the E-WASH Activity. The evaluation team (ET) in coordination with USAID developed the following evaluation questions (EQs) for this evaluation: EQ # Evaluation Question 1 To what extent did SWBs become commercially oriented and improve their financial viability? 1a To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? 1b What impacts, if any, has COVID-19 had on SWB’s financial viability 2 To what extent did the Activity support operational viability and professional management of SWBs 2a What accountability mechanism related to the implementation of policy and frameworks did the Activity put in place, how successful have they been, how could they be improved, and how might they impact sustainability 3 To what extent did the Activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping 4 What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management METHODS The ET conducted a desk review of the E-WASH Activity and sectoral documents to inform the evaluation design and triangulate it with findings. Subsequently, in-person and remote qualitative data collection occurred from November 29 – December 18, 2021, spanning Abia, Delta, Imo, Niger, Taraba, and the Federal Capital Territory (FCT).1 Qualitative data collection consisted of 69 key informant interviews (KIIs), 12 group interviews (GIs) of two to four, six focus group discussions (five or more people), and nine direct observations. Participants included GoN officials, SWB officials, civil society organizations (CSOs), community members, utility customers, contractors, and other stakeholders. Qualitative and quantitative secondary data were collected where possible. The 1 Small number interviews were conducted after the end of data collection due to COVID-19. viii | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY ET conducted a thematic analysis of the qualitative data and a descriptive analysis of secondary quantitative data. FINDINGS AND CONCLUSIONS EQ 1: To what extent did SWBs become commercially oriented and improve their financial viability? The Activity supported all five utilities in the development and passing of the State WASH Laws, which changed the utility status from SWBs to State Water Corporations (SWCs). SWCs are ostensibly responsible for legal, human resources, and operational processes rather than the GoN. The utilities’ mandate as aligned with the WASH Law is to provide safe, adequate, and affordable water supply and services, including sewerage and wastewater management services, to residents of the urban service areas. The Activity also supported SWCs’ commercial orientation, as described below. Enterprise Resource Planning System (ERP): According to respondents in four out of the five states (Delta, Imo, Niger and Taraba), the ERP supported reliable data and automation for billing and reporting. Respondents reported that the Activity’s website support and customer enumeration improved their commercial orientation. Nonetheless, the utilities have not been able to fully optimize some of the modules within the ERP. Monitoring and Evaluation (M&E) and Data Management: The Activity sought to improve data collection, data reliability, and the promote the use of data for decision-making to improve utilities’ commercial orientation. Prior to the Activity, there was no standardized way of capturing data, according to respondents. All the utilities reported that the Activity helped strengthen their M&E units through the development of performance indicators; data collection tools and processes; monitoring, evaluation, and learning (MEL) plans; and the establishment of a Technical Working Group (TWG) to ensure data integrity and timely data collection. Despite these interventions, challenges such as cumbersome bureaucratic processes in Abia, slow updating of documentation in Delta, and reluctance to provide quality data in Imo continue to persist. Workforce Capacity and Attitude: Respondents noted that the capacity-building exercises helped to an extent to shift staff mindsets towards a more commercial-centric orientation. Respondents believed that the embedded E-WASH staff within the utilities were integral in pushing the utilities towards a commercial orientation. However, respondents from the E-WASH team also noted that a resistance to change persists, which slows progress towards commercial orientation. Respondents also noted challenges related to an aging workforce and rapid capacity loss due to retirements. Customer Orientation and Satisfaction: The Activity supported customer care centres across all utilities and customer charters in Delta and Imo. In Delta and Imo, customers positively regarded the utilities’ ability to address complaints. The utilities also had access to the ERP’s customer complaint module, though some were unable to take full advantage due to capacity limitations. Water Production Capacity Utilization:2 The Activity supported the utilities in the rehabilitation of selected water production facilities to improve capacity utilization, thereby improving financial viability. All utilities remained below their capacity utilization targets; however, with E-WASH 2 Capacity utilization efficiency is defined as the percentage of total capacity of water production that is actually being achieved in a given period. It is the relationship between the output that is produced with the installed equipment and the potential output which could be produced with it, if capacity was fully used. ix | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY support, Taraba and Niger improved their capacity utilization by 30 percent and 40 percent total respectively. Key challenges identified by respondents include high operational expenses, a lack of power supply, network limitations and heavy leakages, improper scoping of rehabilitation works, and dependence on the GoN. Financial Viability: The implementation status of financial viability activities varies across utilities. In the case of tariff studies, none of the findings have been implemented in any of the utilities. Numerous challenges exist to the utilities’ financial viability. Non-payment by both institutions and consumers is quite high. Among consumers, resistance to paying for water is impacted by irregular water supply, a lack of consumer confidence, reliance on boreholes, unclear utility rates, difficulties recharging pre-paid meters, and poor water quality. Additionally, non-revenue water (NRW) across all the utilities was very high, ranging from 61 percent in Taraba to 98 percent in Imo. Moreover, fee collection rates were also low, which further hurts cost recovery and overall Operating Cost Coverage Ratios (OCCR), as seen in Figure 1.3 Figure 1: Operating Cost Coverage Ratio Government Intervention: To garner political will and government commitment for the Activity, the Activity signed memorandums of understanding (MOUs) with each implementing state. Milestones were also introduced to improve government spending on the utility and were well-appreciated by respondents. However, implementation challenges persisted, such as insufficient coordination between the Federal Ministry of Water Resources and utilities, as well as a general lack of funding from state governments. E-WASH Needs Mismatch: Some respondents reported a mismatch between what the Activity delivered and what was needed by the utilities. Respondents noted that, due to Activity centralization, the advice of state teams and the utilities were not always taken into consideration. Respondents also reported poor intervention sequencing. For example, laboratory training with the utilities was conducted before the equipment arrived. 3 OCCR identifies the level to which operating costs are covered by revenues. OCCR=1.40 is regarded by the International Benchmarking Network for Water and Sanitation Utilities as a good practice benchmark. OCCR is a ratio of total amount of internally generated revenue of the utility to the amount of the operational expenditure. x | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY EQ1 Conclusions The Activity’s support facilitated SWCs strides towards commercial orientation through the strengthening and establishment of the ERP, data management, capacity-building, and other systems. However, existing capacity challenges due to insufficient infrastructure, supplies, and expertise to utilize the ERP and other data processes continue to hinder progress. While the utilities’ water production capacity, which is fundamental to operational and financial viability, improved over the course of the Activity, except in Abia, capacity utilization targets were not met. All functioning utilities grappled with high NRW, poor water quality, and low-cost recovery, which inhibited operational and financial viability. EQ 1a: To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? The Utility Support Fund (USF) was used for key interventions aimed at improving utility operations and financial viability, ranging from capacity building to the supply of equipment, rehabilitation of waterworks, reconnections and extension of transmission networks, and accountability software. USF Implementation Status Mixed Results: There is wide variation in the completion and effective utilization of the USF between states, with many interventions not yet complete. While the USF was widely appreciated across respondent groups, respondents cited six primary reasons for USF implementation challenges: implementing partner (IP) delays in fund mobilization, poorly sequenced IP training processes, COVID-19 delays, the descoping, political interference, and poor targeting of utility-specific needs. USF ERP Implementation: Despite significant variations in implementation status between the states in the status of ERP implementation, respondents felt the ERP provided a major step forward in ensuring accountability and strengthening SWCs. While respondents lauded the ERP as an effective tool for accountability, other respondents noted concerns about its sustainability and continued utilization after the exit of E-WASH staff and doubt as to whether the GoN will pay the necessary costs to keep it operational and overcome lingering staff resistance to change. EQ1a Conclusions While the USF was appreciated across all states and benefits were noted, even the strongest performing utilities faced challenges. Improvements to USF fund distribution, sequencing, and design of interventions based on utility-specific needs will be critical to future success. In its current state, the USF may not be sufficient to put SWCs on track to overall financial and operational viability. EQ 1b: What impacts if any has COVID-19 had on SWB’s financial viability? The primary challenges posed by COVID-19 were delayed procurement and construction activities and a decline in revenue in two of the five utilities (Niger and Delta). Respondents attributed the revenue decline to challenges with paying at banks, the inability to recharge meters, and the staff’s inability to distribute bills and follow up on revenue collection. Some utilities reported positive impacts in that COVID-19 offered them an opportunity to educate people on the importance of safe water and to leverage technology to drive efficiency. xi | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY EQ1b Conclusions Implementation delays were the primary negative impact of COVID-19. The Niger and Delta utilities were also affected financially. On the positive side, COVID-19 helped create awareness of the importance of water supply among utility customers and pushed utilities to adopt new technologies. EQ 2: To what extent did the Activity support operational viability and professional management of SWBs? Operational Viability: The Activity achieved mixed results in improving operational viability. Water production improvement efforts, a component of operational viability, were successful in Niger, Taraba, and Imo states and were ongoing in Abia and Delta. Abia has yet to produce any water after three years of E-WASH implementation. Professional Management: E-WASH helped SWBs become SWCs and improve their autonomy. Four out of the five utilities now have functioning boards of directors (BODs). The Activity also introduced performance-based contracts and developed strategic business plans. Niger, Delta, and Taraba implemented organizational development recommendations, while Imo and Abia have not because they felt their comments were not incorporated into the final recommendations. The Organizational Capacity Assessment Tool (OCAT) strength index indicator shows an average improvement from 29 to 50. Despite this progress, several challenges hamper success: a lack of incentives for change, such as updated salaries, benefits, and performance contracts; a lack of transparency and teamwork; rivalry amongst staff; shortages of staff; inadequately qualified staff; and struggles over management roles. EQ2 Conclusions The ERP has significant potential for the sector. However, the long-term impact is contingent on sufficient political will, ownership, and local government funding after the end of E-WASH. The WASH laws enabled SWB corporatization, and data show that the utilities were making progress toward professional management practices. However, more work is needed. EQ 3: To what extent did the Activity strengthen policy, institutional and regulatory frameworks for improved WASH SERVICES at the state and national level? E-WASH contributed to strengthened policy, laws, and institutional and regulatory frameworks in all five states. E-WASH supported the SWBs through sensitization and advocacy, capacity-building, mentoring, linkages, and relationship brokerages. These efforts pushed the utilities towards a more commercial orientation. The WASH laws helped all five utilities to transform into SWCs, which offer them some level of autonomy. Respondents said the WASH laws helped provide clarity on roles and responsibilities; strengthened institutional and regulatory frameworks; created various support agencies, including regulatory, rural water supply, small town, etc.; and provided utilities some level of autonomy. Despite this progress, challenges remain as financial constraints, political challenges, and a lack of water production have stalled or hampered effective implementation in most states. xii | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY EQ3 Conclusions The Activity successfully supported the development, implementation, and partial operationalization of WASH policies and laws in all five states. However, these improvements have not yet translated into improved water service delivery due to a lack of water production and adequate funding. EQ 3 a: What accountability mechanism related to the implementation of policy, and frameworks did the Activity put in place, how successful have they been, how could they be improved, and how might they impact sustainability? E-WASH supported a wide range of accountability mechanisms, the most important of which includes the establishment of BODs, the ERP, and improved auditing and procurement processes. Other accountability efforts are discussed in the main body of the report. BODs: E-WASH successfully supported BOD creation in four (Abia, Delta, Niger, and Taraba) of the five states. Imo was unable to establish a functional BOD due to political instability and a lack of political will to fund the BOD. The ET also noted that BODs in Abia and Niger hired well-qualified members and staff, while BOD in Delta and Taraba did not. The BODs are likely sustainable due to their establishment by law. ERP: The ERP automates financial management, customer billing, revenue collection, human resources, customer relations, and operations processes. Several of the utilities where the ET was able to conduct a direct observation could not fully benefit from the ERP, like Abia, Delta, and Imo, as usage varied across the states and the ERP is only useful when a utility is producing water. In addition, capacity issues persisted after E-WASH trainings. Some respondents also voiced concerns about sustainability after E-WASH ends. Auditing and Procurement Processes: E-WASH effectively supported improved auditing processes in all five states and procurement systems in Delta and Abia. These audit processes and procurement systems may be sustainable because they do not require outside funding and occur usually only once per fiscal year. EQ3a Conclusions The Activity put in place several accountability improvements in all five utilities. Internal accountability structures, such as the BODs and procurement and audit processes, moved the needle to promote transparency and accountability. However, many challenges remain, including a lack of funding, water supply limitations, management commitment, ERP-related capacity gaps, insufficient local ownership, and incomplete implementation due to the descoping. For these reasons, some structures may not be sustainable. EQ4: To what extent did the Activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping? Key Challenges Identified 1. Challenging Sanitation Context 2. Incomplete City-Wide Inclusive Sanitation (CWIS) Implementation 3. Utility Staff Capacity 4. Insufficient Water Supply 5. Timing and Descoping Issues xiii | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY Institutional Framework: E-WASH helped establish sanitation units and City Sanitation Task Forces (CSTFs) in cities in all five states. However, the ET found that all CSTFs and sanitation units were weak and struggled with capacity, staffing, and funding challenges. Enabling Environment: The CWIS process was implemented in all five states. However, none of the SWBs have translated the CWIS process into the recommended sanitation actions, including the planned fecal sludge treatment plants. EQ4 Conclusions E-WASH’s sanitation interventions did not achieve their objectives, though they helped to create awareness of and campaigns against open defecation. While some sanitation progress was made and groundwork laid through laws, policies, and frameworks, the utilities are still weak and struggling with capacity challenges and competing priorities. Insufficient sanitation expertise on the IP team and within the utilities contributed to the challenges and hampered progress. EQ 5: What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? E-WASH facilitated the implementation of USAID’s environmental and construction guidelines. Through various plans, audits, and impact assessments, environmental and social risks were mitigated to an extent. Assessing risks and mitigation measures is essential for adaptive management. Capacity and Oversight: The IP lacked in-house health and safety or environmental experts. Respondents noted this as a substantial challenge for effective oversight and adaptive management. This role was subcontracted out, which led to an inefficient flow of information. Compliance: Respondents noted that E-WASH used the guidelines to ensure minimal disruption of livelihoods, reduce environmental impact, protect workers, and avoid paying compensation while ensuring rehabilitation. Mitigation measures included working outside of work hours, routing trenches so existing housing and other properties were not affected, using safety signs and gear, and engaging with community leaders before construction began. Adaptive Management: Challenges to adaptive management included a lack of USAID field visits, a lack of relevant experts on the IP team, and inadequate utilization of the plans to prevent incidences. Nonetheless, adaptive management principles were used to address some challenges, albeit belatedly (e.g., the Activity adapted by diverting water in waterlogged construction areas). EQ5 Conclusions E-WASH’s implementation of the environmental and construction guidelines showed mixed results with noted areas for improvement. The primary IP’s lack of dedicated in-house health and safety or environment experts on their team impacted overall adaptive management and oversight of the environmental and construction guidelines. Nonetheless, the guidelines helped to identify and mitigate negative impacts from construction works by addressing them proactively. xiv | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY ANALYSIS ON THE UTILIZATION OF THE MTE REPORT AND DESCOPING Effect of Descoping: Descoping resulted in the prioritization of water production interventions, which was an expressed need of government respondents. Some respondents noted that the rapid disengagement of E-WASH staff with short notice led to a loss of stakeholder confidence. Communication with Stakeholders: Stakeholders across the states noted that communication around the MTE findings and the descoping process was poor. Most state E-WASH teams did not fully understand the MTE’s recommendations and state stakeholders had no knowledge of the MTE’s findings or descoping decisions. MTE Conclusions The IP largely implemented the MTE’s recommendations, which prioritized shifting the Activity’s focus to water supply delivery and strengthening M&E, which had positive effects on utility performance. However, the descoping resulted in some negative effects for utilities. The MTE recommendations and the resulting descoping were not adequately disseminated to stakeholders. KEY RECOMMENDATIONS The ET provides detailed recommendations by EQ in the report. Key recommendations are provided below: EQ# Recommendation EQ1 For future activity design, USAID should: 1. Ensure that utility-based interventions directly target individual utility needs and be based on independent assessments or studies that take into consideration the maturity level of the utility to accept such interventions, including the availability of water and infrastructure. 2. Ensure logical sequencing and prioritization of implementation and inputs. For example, key interventions such as ERP should be prioritized early in activity implementation. 3. Ensure that advocacy to state governments includes clear project deliverables and MOUs include verified commitments to implement identified activities (e.g., funding, human resources). EQ1a 1. For future activity design, USAID should ensure that the scope of interventions is designed in collaboration with the utility and captures all components required to make the system fully functional and to secure ownership. 2. If future activity design includes a mechanism similar to the USF, the projects should be implemented earlier in the activity to maximize benefit and impact. Descoping Impacts by State: Abia – several components (e.g., gender policy and customer charter) ended Delta – NRW and CSO activities Niger – challenges with provision of equipment impacting water supply Taraba – concerns with staff capacity to address water supply Imo – cessation of oversight processes, capacity building, and removal of support staff xv | E-WASH FINAL EVALUATION USAID.GOV NIGERIA MEL SUPPORT ACTIVITY EQ1B 1. For future activity design, USAID should continue to support utilities’ improvements for billing and collection processes (e.g., prepaid meters and other e-method payment options). EQ2 1. Future USAID projects should consider conducting and implementing organizational development early in the activity so that functional structures and human resources are in place. This should include proper staff recruitment and placement as well as policies for staff remuneration and incentives. 2. For future activity design, USAID via its IPs should develop a roadmap to assist the utilities in planning the full implementation of anticipated activities, such as organizational development, strategic plans, and performance-based contracts. This roadmap should include information on the timing, funding sources, and other pertinent details to ensure effective implementation. EQ3 1. Through other development partners and IPs, USAID should continue advocacy for the development of roadmaps and strategies that identify the sources of funding to ensure that the government fully implements current WASH policies and laws, as well as complete the ongoing development of WASH policies and laws. EQ3a 1. For future activities, USAID should identify targeted accountability mechanisms within utilities and support future planning for capacity-building, knowledge transfers, and sustainable funding streams. 2. The utilities through the customer units should maintain engagement with the customers and WASH media forums to ensure their sustainability. EQ4 1. USAID should ensure RTI develops an exit plan as part of the activity closeout for each intervention state. The exit plan should incorporate the sanitation investment plans from the CWIS exercise to identify the key actions, opportunities, partnerships, and funding required to operationalize and strengthen sanitation units and CSTFs. 2. For future programming, USAID should consider the challenging sanitation environment clearly, as itemized in the sanitation mapping reports. This should be done prior to project design. USAID should focus interventions on preparing the mandated agencies to address these complex challenges. 3. USAID should provide oversight to ensure that the institutional frameworks established by the IPs comply with best practices in Nigeria. For example, the composition of the CSTFs hampered, rather than aided, implementation. 4. USAID should ensure that IPs’ sanitation strategies have sufficiently skilled staff and assess implementation context early and often. EQ5 1. Future USAID WASH activities should advocate for and provide technical assistance to states for the inclusion of environmental and construction guidelines in state procurement processes, as well as stakeholder training on the role of adaptive management and facilitating ownership. USAID should ensure that IPs hire qualified health and environmental safety officers as a condition for funding. MTE 1. USAID should ensure that learning and adaptations, including descoping, based on evaluation findings are disseminated internally and externally in a timely manner. 2. USAID should ensure that any future activity descoping processes include an exploration of the possibility of other donors or partners supporting descoped activities. 1 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION INTRODUCTION BACKGROUND The United States Agency for International Development (USAID) awarded the five-year Nigeria Effective Water, Sanitation, and Hygiene (E-WASH) Activity in 2018. Research Triangle Institute (RTI) International led a consortium of three implementing partners (IPs): Plan International (Plan), Segura Consulting LLC, and CDM Smith. E-WASH is part of USAID’s wider efforts to build closer coordination with the Government of Nigeria (GoN), including the Federal Ministry of Water Resources, select state water boards (SWBs), and allied stakeholders. The Activity aims to advance broad-based economic growth and resilience through improved water, sanitation, and hygiene (WASH) services in urban areas. Interventions are targeted in states where state governments have shown a willingness to embrace enduring reforms to improve water service delivery. CONTEXT The GoN’s 2002 WASH Policy delegated responsibility for providing WASH services to state governments, who in turn have mandated SWBs provide urban WASH services. In most cases, SWBs have limited financial and operational autonomy and are not commercially viable. The systemic weaknesses observed with SWBs are the consequence of a combination of an uncoordinated and fragmented operational framework, tariff structures that achieve neither cost recovery nor financial sustainability, insufficient funding, mismanagement, a decades-long lack of regulation, and an acute governance challenge. E-WASH was designed to reform six SWBs in Nigeria. The objective was to improve the financial and operational viability of the utilities and create an enabling environment for the utilities to grow. Following an assessment of fourteen pre-selected states, the criteria for selecting six states were to be based on willingness to reform, existing functionality of infrastructure, and potential for positive impact. Figure 2: Map of States with E-WASH Operations and RTI’s Head Office in Abuja Note: Sokoto pulled out of E-WASH 2 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION The E-WASH Activity commenced in May 2018 with the SWBs from Abia, Delta, Sokoto, Imo, Taraba, and Niger. Sokoto subsequently dropped out of the Activity in early 2020, leaving the Activity with five states. In August and September 2020, the USAID Mission in Nigeria commissioned a mid-term evaluation (MTE) of the Activity’s performance in the five participating states. The USAID Mission issued the MTE report in October 2020. The outcome and recommendations from the MTE led to the descoping, reorientation, and streamlining of activities and a revised implementation work plan to end the Activity in 2022. The descoping and COVID-19 adjustments resulted in Activity reorientation. By the end of Q2 2021, many Activity interventions has stopped, except for water production and distribution monitoring; water quality, assets management, and non-revenue water (NRW) reduction strategies; performance monitoring; and limited support to Abia. The descoping prioritized: (i) provision of infrastructure and equipment under the Utility Support Fund (USF) by completing ongoing procurement activities and initiating nine new construction activities to improve water supply, production, and distribution across all five states; (ii) targeted technical assistance the ensure the sustainability and adoption of tools, processes, and systems that the Activity had introduced since its inception; and (iii) support for monitoring, evaluation, and learning (MEL) interventions, reporting, and Activity closeout. ACTIVITY OBJECTIVES The objective of the USAID-funded E-WASH Activity was to assist the GoN in expanding and improving urban water service delivery by strengthening the governance and financial and technical viability of select SWBs. E-WASH concentrated on solidifying state and utility-level governance gains, which manifested in improved water supply services and an improved ability to regulate, manage, and expanding sanitation services in line with activity targets. To address the enormous need for urban WASH services in Nigeria, E-WASH focused on reforming SWB governance, which would then lead to cost recovery, increased investment in infrastructure, and expanded access to water and sanitation to large unserved and underserved populations. As a result, E-WASH aimed to help SWBs demonstrate that better-performing water boards would raise the quality of services for their customers, facilitate service-driven economic performance, improve state finances by increasing cost recovery, reduce subsidies, and increase the state’s capacity to sustainably serve all customers, including the poor, in selected urban areas. E-WASH worked toward achieving the following mutually reinforcing objectives: • Create professionally managed and commercially oriented SWBs. • Improve the financial and operational viability of SWBs. • Strengthen policy, institutional, and regulatory frameworks for improved WASH services. • Build national and state WASH advocacy, coordination, and communications for reform. After the MTE and descoping, the revised implementation work plan included two focus areas: • Implementing city-wide inclusive sanitation (CWIS) processes resulting in city-wide sanitation mapping and shift flow diagrams (SFDs) in the five major cities of the E-WASH implementing states. The Activity sought to improve sanitation services through CWIS plans, the establishment of sanitation task forces, and the organization of sanitation workers. 3 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION • Training and implementation of USAID’s Construction Sector Environmental Guidelines, including training on environmental mitigation and monitoring plans (EMMPs) and use of the guidelines. Upon completion, E-WASH expected to achieve the following high-level results: • Five SWBs professionally managed that: (i) achieve greater commercial viability; (ii) are overseen by a board of directors (BOD) appointed by the State Governor based on existing laws; (iii) demonstrate greater managerial autonomy; (iv) can directly hire personnel and set salaries; and (v) achieve greater cost recovery by reducing inefficiencies, NRW, and corruption. • Up to United States Dollar (USD) ten million of additional public and/or private sector financing mobilized. • Five SWBs that have implemented revised policies, institutional frameworks, or regulations that promote access to improved WASH services. • At least 54,809 households (426,325 people) gained access to basic drinking water or improved supplies piped onto their premises across selected states. EVALUATION PURPOSE AND AUDIENCE The primary purpose of this final performance evaluation is to determine whether E-WASH achieved its stated development objectives, to understand the lessons learned, and to reflect on the findings and recommendations from the MTE and the subsequent descoping. The performance evaluation is intended to provide an independent examination of the Activity’s overall progress, accomplishments, weaknesses, and potential for sustainability. This final performance evaluation will provide USAID/Nigeria (Economic Growth and Environment Office and others), its IPs, the GoN, and WASH sector stakeholders with findings, conclusions, and recommendations to advance future WASH service expansion through commercialization in Nigeria. EVALUATION QUESTIONS (EQS) This evaluation design assesses the performance of the E-WASH Activity against its stated development objectives by focusing on relevance, effectiveness, sustainability, and how it considers gender and social inclusion in its implementation. In order to determine the achievement of these objectives, the evaluation will answer the following key questions: 1. To what extent did SWBs become commercially oriented and improve their financial viability? a. To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? b. What impacts, if any, has COVID-19 had on SWB’s financial viability? 2. To what extent did the Activity support operational viability and professional management of SWBs? 3. To what extent did the Activity strengthen policy, institutional and regulatory frameworks for improved WASH services at the state and national levels? a. What accountability mechanism related to the implementation of policy and frameworks did the Activity put in place, how successful have they been, how could they be improved, and how might they impact sustainability? 4. To what extent did the Activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping? 4 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 5. What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? METHODOLOGY OVERVIEW The evaluation team (ET) used mixed methods approaches including both primary and secondary data. The methods included a desk review, key informant interviews (KIIs), group interviews (GIs), focus group discussions (FGDs), and direct observation. In-person and remote data collection occurred from November 29 – December 18, 2021, spanning Abia, Delta, Imo, Niger, Taraba, and the FCT.4 Due to COVID-19, data collection in Taraba was conducted remotely. The ET collected data from a range of stakeholders, described below, and processes to ensure triangulation of results. RESEARCH DESIGN DOCUMENT REVIEW The ET conducted a desk review of existing documents and reports on the USAID/Nigeria E-WASH Activity and the broader WASH sector. Documents reviewed included 32 E-WASH quarterly and annual reports, WASH policies by state, work plans, and other related documents (see Annex I). The ET undertook the desk review to understand the context and the strategic pathway of the E￾WASH design, implementation variation, and implementation successes and challenges. The ET extracted key results from the review and triangulated them with data from other sources. KEY INFORMANT INTERVIEWS AND GROUP INTERVIEWS The ET conducted KIIs and GIs with two to four participants using semi-structured guides designed for each stakeholder group (Table 1). The KIIs/GIs aimed to elicit information on whether the assistance provided by the Activity achieved the stated development objectives. The ET conducted a total of 69 KIIs and 12 GIs. Across the KIIs and GIs, 89 men and 32 women participated. Table 1: KII and GI Participants by Gender Stakeholder Abia Imo Delta Niger Taraba Abuja Total Gender M F M F M F M F M F M F M F SWB Staff 2 0 1 0 4 0 3 0 3 1 0 0 13 1 IP Staff 2 0 3 0 1 1 5 0 4 0 5 4 20 5 Sanitation Task Force 0 0 0 0 0 0 2 1 0 0 0 0 2 1 State Government Secretary 1 0 0 1 1 0 1 0 1 0 0 0 4 1 Water Commissioner 1 0 0 0 0 0 1 0 1 0 0 0 3 0 Federal/State Ministries 2 3 0 0 0 0 0 0 0 0 0 0 2 3 4 Small number interviews were conducted after the end of data collection due to COVID-19. 5 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Stakeholder Abia Imo Delta Niger Taraba Abuja Total Gender M F M F M F M F M F M F M F Utility Staff 2 2 5 8 4 1 0 0 0 0 0 0 11 11 Utility Customers 0 0 2 0 2 1 6 1 1 3 0 0 11 5 Local Government Area (LGA) Sanitation Unit 4 0 1 0 0 0 2 0 0 0 0 0 7 0 State Commissioner 0 0 0 0 0 0 1 0 0 0 0 0 1 0 Contractors/ Construction workers 0 0 3 0 0 0 5 0 1 0 0 0 9 0 USAID 0 0 0 0 0 0 0 0 0 0 5 0 5 0 Community5 0 0 0 3 0 2 1 0 0 0 0 0 1 5 Total 14 5 10 12 10 5 12 0 3 0 8 0 89 32 FOCUS GROUP DISCUSSIONS The ET conducted FGDs with five to eight participants to obtain a utility customer perspective (rom functional utilities to answer EQs 1-3. The ET conducted six FGDs with 32 males and seven female participants (Table 2). The ET asked questions on E-WASH implementation to understand utility customers' thoughts and perceptions regarding their experiences with E-WASH and E-WASH suppor Stakeholder ted entities. Table 2: FGD Participants by Gender6 Abia Delta Taraba Total Gender M F M F M F M F Utility Staff/Civil Society Organizations (CSOs) 3 4 0 0 8 0 11 4 Utility Customers 0 0 0 0 6 3 6 3 Contractors/ Construction workers 10 0 5 0 0 0 15 0 Total 13 4 5 0 14 3 32 7 DIRECT OBSERVATION The ET conducted limited site visits to eight Activity locations across states to observe and confirm the successful execution of a sample of interventions based on information from document reviews and with the guidance of SWBs. Due to COVID-19, the team was not able to conduct observations in Taraba. Site observations provided the opportunity to engage with laboratory facilities, water treatment sites, and infrastructure improvement sites that facilitate access to quality water and sanitation services as part of the Activity. Site observations helped the ET to confirm actions taken; 5 Community refers to people impacted by construction work. 6 GIs and KIIs were conducted in Imo and Niger rather than FGDs due to limited access to customers. 6 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION clarify the sustainability of processes to achieve commercial and financial viability; and confirm the findings from other data sources. SECONDARY DATA The ET collected secondary data from SWBs’ documents and records on production, distribution, finances, and their customer database. Quantitative secondary data was formulated into an indicator performance tracking table (IPPT) Matrix table from the E-WASH Fiscal Year (FY) 2021 Report. Secondary data analysis and triangulation were dependent on the ability to receive access to data within the evaluation timeframe. DATA ANALYSIS AND SYNTHESIS KIIs, GIs, FGDs, and observation data were coded and analyzed in stages using Dedoose® software. Initially, interview guide questions were mapped to EQs to aggregate and synthesize results for each EQ. A codebook with themes and code application definitions was then developed. Data were coded and a thematic analysis was conducted, which included a review of data and themes across states, stakeholders, and where appropriate, by gender. Secondary quantitative data were analyzed using Microsoft Excel software. The quantitative analysis was guided by the EQs and focused on implementation performance and Activity objectives. The ET used descriptive analysis techniques for secondary data and created charts to visualize these data. Performance indicator calculations were conducted using standard formulas.7 QUALITY CONTROL MECHANISM FOR DATA For quality control, with respondent consent, the ET recorded all KIIs, GIs, and FGDs. Team members also took handwritten notes on emerging themes relating to the EQs. The notes and recordings facilitated fact-checking during analysis and report writing. During data analysis, only the ET, Data Analyst, and MEL Support staff had access to qualitative data and the analysis tool (Dedoose®). All analyses were handed over to MEL Support to ensure confidentiality. GUIDING PRINCIPLES AND VALUES The ET ensured sound ethical consideration during data collection. Respondents were informed about the purpose of the evaluation as part of the informed consent process. The ET chose interview locations where the respondents were free to talk without interruption and to maintain their privacy. In accordance with policy and best practices, respondents are referred to with descriptors that will not reveal their identity (e.g., role, job title, or stakeholder type and location). The ET ensured that everyone, including the participants, adhered to COVID-19 safety protocols, particularly the use of masks and social distancing. The ET also ensured that security and travel plans were approved and followed. 7 NRW = (total water supplied into the distribution system – total water sold) / (total water supplied into the distribution system) ×100 OCCR= (internally generated revenue) / (operational expenditure) *100 Collection efficiency = (internally generated revenue from billing / total amount billed) * 100 Capacity utilization efficiency = (actual production / designed capacity) * 100 7 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION LIMITATIONS As with any evaluation design, there are limitations and risks to consider. The ET identified the following challenges and devised mitigation strategies during the evaluation: Biases: Biases such as recall bias and positive response bias may have occurred. For example, respondents may have wanted to provide the “correct or expected” answer because of social norms. To guard against these potential biases, the ET triangulated findings among several sources and data types. The team also conducted independent observations. Implementation Complexities: E-WASH had several objectives and spanned several different interventions and approaches, and descoping led to significant changes. Thus, it was not always possible to fully assess all of E-WASH’s objectives. This limits the ET’s ability to draw conclusions about how implementation affected the ability to reach the objectives. However, the ET made inferences based on the triangulation of data sources. COVID-19 and Remote Data Collection: Data collection occurred during COVID-19 and a surge of the Omicron variant. As a result, the ET had to complete data collection remotely in Taraba; thus, direct observations in Taraba were not possible. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The findings, conclusions, and recommendations are presented by EQ, with findings related to the Activity’s response to the MTE presented last. EVALUATION QUESTION 1 To what extent did SWBs become commercially oriented and improve their financial viability? E-WASH implemented several interventions to push the SWBs towards a commercial orientation. First, the Activity supported the conversion of all five utilities from SWBs to State Water Corporations (SWCs). It also put in place systems to support the commercial orientation of the utilities, including provision of computer to support operations automation, capacity building, and implementation of the Enterprise Resource Planning (ERP) system, the status and outcomes of which are discussed in more detail under EQs 1a and 3a. Changes in the utilities’ commercial orientation occurred in six primary areas: monitoring and evaluation (M&E) and data management, workforce capacity and attitude, customer orientation, water production efficiency, financial viability, and autonomy from government support. Each of these areas is discussed below. Lastly, this section discusses E-WASH implementation challenges affecting commercial orientation. “[Before the Activity, there was] no standardized way of capturing and reporting of information. E￾WASH has helped to develop and implement key performance indicators” (Excerpts from Abia KII) M&E and Data Management: The Activity sought to improve data collection, data reliability, and promote the use of data for decision-making to improve utilities’ commercial orientation. Respondents noted that prior to the Activity, there was no standardized way of capturing data. All the utilities reported that the Activity helped strengthen their M&E units. In Delta, the Activity even supported the creation of an M&E unit under the Department of Planning and Statistics. The utilities also reported the development of performance indicators, such as collection efficiency, capacity 8 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION utilization, water quality parameters, etc.; data collection tools; and MEL plans. The Activity also supported staff development on data collection processes, identification of indicators, data quality assessments, and the importance of using data for decision-making. A Technical Working Group (TWG) was created to ensure data integrity and the timely collection of data within all five utilities, with members drawn from key departments acting as data collection focal persons Despite these interventions, challenges persist in most states. For example, respondents in Imo reported that the timely data collection is still a challenge. In Delta, data are used to track performance but are not actively used for management decisions. In Abia, the utility cannot operationalize the M&E system since they are not yet producing water and there are no data to track. However, feedback was quite positive in Taraba, where respondents indicated that data are used to drive performance, particularly in the areas of revenue generation, production efficiency, number of leakages reported, and number of repairs completed. In Taraba, respondents felt that the TWG had helped to significantly improve the quality of data collected. Workforce Capacity and Attitude: Respondents noted that the capacity-building exercises helped to an extent to change the staff mindset from civil service to a more commercial-centric orientation. Respondents appreciated the embedded E-WASH staff within the utilities and found the approach novel and integral to the Activity’s accomplishments towards commercial orientation. A KII respondent from Taraba shared, “Key lesson is that data is key in planning and management of any organization and capacity building is also very important in management of utilities.’’ In addition, respondents noted that training has changed the attitudes of staff, who are adapting to the corporate philosophy. Changing from SWBs to SWCs has also changed staff’s perspective. In Delta, a respondent shared, “Staff attitude to work has been lukewarm hitherto, but now people are enthusiastic to work. Before, the utility staff cannot operate a system, but now they can do so. Training and workshops led to the improvement.” Respondents from the E-WASH team noted that, despite the progress made, many staff still maintain a civil service attitude and approach to their work. This attitude has slowed progress in the transition to a more commercial orientation within the utilities. Before the advent of E-WASH in Imo state, the utility had no functional office; utility activities were conducted under a tree. As a result of the Activity, the utility has a more conducive working environment. However, challenges to improvement remain in many of the states. In Imo, the upper echelon of the utility staff has not been exposed to best practices, which limits the extent to which change can be realized throughout the utility. There is also a general sense of entitlement by the utility staff, which leads to a belief that “whether they work or not, they should get paid,’’ a resistance to change, and poor attendance. Thus, more work remains for the Imo utility to be performance-driven. In Abia, according to respondents, the challenges include overbearing attitudes from utility management and a nonchalant attitude of the staff to their work. Additionally, respondents noted that staff sent to E-WASH trainings in some instances were not always the ones who would utilize the training. Respondents from Niger and Abia noted additional difficulties in the uptake of E-WASH capacity-building efforts. They noted that the trainings where they were too short and too rapid to create substantial impact. In another case, the Coca-Cola training was conducted via Zoom, which was challenging for many participants due to poor internet connectivity. Customer Orientation and Consumer Satisfaction. The Activity supported the creation of customer care centers in all the utilities. In addition, CSOs reported the development of customer charters in Delta and Imo. In these two states, customers positively regarded the utility’s availability to address complaints and complimented their prompt response, often within two to three hours, to customer complaints such as pipe leakages. The customers can contact the utility via the customer care office or telephone calls. The utilities also reported the availability of the Customer Relationship 9 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Management module within the ERP, which is used to record customer complaints. Abia confirmed the availability of a customer care center; however, it was not functioning due to a lack of water supply. Water Production Capacity Utilization. The Activity supported utilities in the rehabilitation of selected water production facilities to improve capacity utilization and financial viability. Four out of the five states improved in their capacity utilization. Taraba had the greatest improvement from baseline, followed by Niger. Delta and Imo, which were not producing water, have started to produce. Abia is still not producing water (Figure 3). Abia, Delta, and Imo continue to grapple with low-capacity utilization. In light of the improvements in Niger and Taraba, all of the utilities remain below their anticipated targets for efficiency (Figure 3). Figure 3. Capacity Utilization Efficiency by State8 In each of the states, E-WASH supported infrastructure improvements to assist in increasing capacity utilization. In Taraba, E-WASH provided an automated voltage regulator, which allowed the utility to increase their use of energy-intensive equipment. In Niger, E-WASH supported the rehabilitation of Chanchaga Waterworks in Minna and the Suleja water treatment plant. E-WASH also installed dozing pumps in Bida. At the Otanmiri Waterworks in Imo, the E-WASH procured three high and low lift pumps at the headworks and intakes, as well as filters and nozzles. E-WASH additionally supported the optimization of the Otanmiri treatment plant; however, this effort is limited by a faulty backwash pump. The rehabilitation of treatment plants in Abia and Delta are still underway. In Abia, the treatment plant will not be able to be utilized once completed due to the lack of a water transmission line, which was overlooked in the scoping of rehabilitation work. Table 3: Infrastructure Improvements by State States Infrastructure improvement and network extension from E￾WASH activity Percent improvement in capacity Taraba • New automated voltage regulator • 16 km piped water network rehabilitated 30% Niger • Rehabilitation of the Chanchaga Waterworks in Minna • Rehabilitation of the Suleja water treatment plant • 9 km District Metered Area (DMA) extended 40% 8 USAID E-WASH Indicator Progress Tracking Table 10 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION States Infrastructure improvement and network extension from E￾WASH activity Percent improvement in capacity Imo • Otanmiri Waterworks: three high and low lift pumps at the headworks and intakes, respectively, as well as filters and nozzles • 2 km piped water network extended <10% Abia • Rehabilitation of Ariaria water treatment plant <5% Delta • Ongoing rehabilitation • 10km piped water network extended <5% The Activity is also supporting the extension of the piped water network in many states: ten km in Delta, 16 km in Taraba, and two km in Imo. In Niger, E-WASH is supporting a nine km DMA, the disconnection of 2,200 customers from the transmission line to improve water pressure, and the installation of a 2.2 km parallel network. Despite these notable improvements, respondents in Abia noted that while the NRW strategy was developed and the M&E TWG is in place, they cannot be implemented due to a lack of water supply. Across the states, respondents noted other consistent challenges to improving their capacity utilization, including high operational expenses, insufficient power supply, network limitations, heavy leakage, improper scoping of rehabilitation works, and government dependence. Financial Viability: The activities designed to improve financial viability are at varying stages of implementation within the utilities. The findings of the E-WASH tariff studies have not been implemented in any states, except for Taraba which did not need a tariff review. Niger has reviewed its tariff study and its strategic plan is awaiting government approval to implement. Also in Niger, many government institutions were not paying their water bills. Customer satisfaction, which can have a carryover effect on financial viability, varied across the states. In Imo, customers complained that the water was not healthy due to bad odor and inconsistent supply. Additionally, several customers had filed complaints with the utility and have not received responses. As a result of these issues, many customers depend heavily on boreholes rather than the utility. Some customers also reported instances of people using water without payment, which the utility has not addressed. As stated by an Imo KII, “There are some large families in my compound that are not metered but use free water without payment to ISWASC [Imo State Water and Sewerage Corporation]. Several complaints have been made to the utility but have not been addressed.” In Delta, customers noted that their primary water supply source is a borehole. The utility water is primarily used as backup source due to the intermittency of the water supply from the utility. The poor level of service has led to a loss of customer confidence. Customers also reported difficulties finding information on the utility’s water rates, as well as on recharging their prepaid meters. In Niger, respondents reported that water had not been supplied for the last three to twelve months. Utility and USAID respondents also noted that in Delta, Niger, and Imo, many customers are not willing to reconnect to the utilities. Due to the long-term unavailability of water through the utilities, customers have resorted to their own water sources. In some cases, when customers do reconnect, they refuse to pay or claim to not have a connection, which contributes to high NRW. 11 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Figure 4: Drinking Water Quality Compliance Rate from Effective Utility Management (EUM) Primer Performance Measures9 As shown in Figure 4, none of the utilities are meeting their water quality compliance targets. These data corroborate the customers’ complaints. The irregular supply and the poor quality of water supplied has continued to impact the financial viability of the utility. All the utilities except Abia have reduced their NRW from the baseline. Niger and Taraba have reduced NRW from 93 percent and 89 percent to 61 percent and 73 percent, respectively. Niger met the activity target. Notwithstanding the improvements, NRW across all the utilities was very high, ranging from 61 percent in Taraba to 98 percent in Imo (Figure 5). In Imo, this means that the utility is generating income from only two percent of the water it produces. Additionally, of the two percent, Imo was only able to collect 18 percent of this income. In Niger, the NRW rate is 73 percent, of which they are only collecting 30 percent. These high NRW rates and low recovery rates are a significant impediment to financial viability. Figure 5: NRW Rates by State10 All the states except Abia have started to recover costs for operations, as measured by their Operating Cost Coverage Ratio (OCCR). Taraba and Niger had the most improvement from the baseline. Taraba met and exceeded its target, while Niger’s OCCR declined in 2021 as a result of a decline in collection efficiency from COVID-19 and the Treasury Single Account (TSA). A collection efficiency of 80 percent is considered low, as it means there is a negative inflow of revenue to cover costs. This is particularly the case when the OCCR is below one, as it is in all the utilities (Figure 6). The global benchmark for well-performing utilities is 1.4. As a point of reference, the 25 worst￾performing utilities in Africa report an OCCR of 0.83, and the 25 top-performing report an OCCR of 1.36. Taraba has the highest OCCR, 0.29, which is four times less than the worst performing utility in middle-income Africa.11 When the utilities cannot cover their operating expenses, they 9 USAID E-WASH Indicator Progress Tracking Table 10 USAID E-WASH Indicator Progress Tracking Table 11 https://www.ib-net.org/. 12 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION must delay maintenance and reduce service levels. Ultimately, they must continue to depend on the government to subsidize their operations, which undermines their autonomy. Figure 6: Operating Cost Coverage Ratio12 Government Intervention: To garner political will and government commitment for E-WASH, the Activity signed memorandums of understanding (MOUs) with all five implementing states. Milestones were also introduced to improve government spending on the utilities. Respondents believed that the milestones and MOUs were useful innovations that helped hold the state government accountable. One milestone that was implemented in each state was the provision of laboratory infrastructure by the state, combined with funding for equipment and reagents by USAID. In Delta, E-WASH supported the utility in accessing an additional Nigerian Naira (NGN) 50 million in funding from the government, but there were no systems put in place to track how the funds were utilized. The collaboration also faced coordination issues between the NMWR and the utility. Respondents across the states reported that a lack of government commitment and funding was a primary challenge for the utilities. A respondent in Niger noted that the “government [is] not meeting its own side of the bargain timeously e.g., chemicals not being ordered timeously.” This has posed a significant challenge. They also noted that the E-WASH did not plan to get the government’s commitment for cost-sharing or resource provision. “Timing, planning resources from both sides [are] grossly inadequate and that six months after the exit, the gains will decline in Niger.” In Delta, the lack of funding and government commitment was also cited as a challenge. As one respondent said, “The state government has to be more committed beyond the mere signing of a baseless MOU.” “E-WASH helped the corporation to get NGN 50 million from the government, but they did not follow up with the usage of the funds for accountability.” (Excerpt from Delta KII) Respondents in Imo said that political instability was also a barrier. During the implementation period, the state had three State Chief Executives, nine Permanent Secretaries, and three Leaders of the State House of Assembly. Respondents in Taraba and Abia also felt that implementing reform in Nigeria is very challenging and that little can be achieved in the current environment, where there is an emphasis on “business as usual” and a corresponding lack of urgency. E-WASH Implementation Challenges: Respondents noted that one challenge during implementation was inconsistencies between what was delivered by the Activity versus what the utilities needed most. This was in part due to the separation between the Activity’s central 12 USAID E-WASH Indicator Progress Tracking Table 13 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION headquarters staff and those working directly in the states. Some respondents said that the advice of the state teams and utilities was not always taken into consideration. Respondents also noted challenges with poor intervention sequencing. For instance, laboratory training was conducted before the arrival of all the equipment and the customer enumeration in Abia was completed even though there was no water production. In a similar vein, respondents in Abia noted that “the major lesson is that USAID will lose the massive investment because of non-production. Most training is almost forgotten due to a time lapse between training and implementation. NRW strategy developed, M&E working group in place, but all cannot be implemented due to no water supply.” Improper intervention scoping was also identified as a challenge. In Imo, booster pumps were provided, but the network was not adequate for using them. There was also a general misunderstanding of how the USAID funds should be utilized; many stakeholders thought that the funding would be used to provide physical infrastructure. On the customer side, many customers rely on other water sources due to long water outages and loss of confidence over time. In many cases, they are not willing to return and rely on water from the utility. As noted above, even when they do connect, there are challenges in collecting payments and some even claim not to be connected at all, all of which contribute to the utility’s high NRW. Conclusion: The SWCs have made some shifts towards a commercial orientation. This has led to increased awareness and attitudinal changes, as well as a shift towards improved revenue and overall capacity. However, many challenges remain, particularly regarding the utilities’ financial viability and overall service delivery. In Taraba and Niger, improvements have been strong, but they have not met the anticipated targets. If states can improve capacity utilization and improve collection efficiency, then they are likely to be able to reduce NRW and improve OCCR, as demonstrated in Niger and Taraba. All utilities except Abia saw improvements in water production. However, all five continue to grapple with high NRW, poor water quality, and low-cost recovery. This undermines the gains of E￾WASH. A lack of water production is a significant hurdle, particularly in Abia. This is a specific challenge for the uptake of capacity-building efforts, as staff do not have the opportunity to implement what they have learned, and many trainings risk being forgotten. The Activity was heavily involved with the implementation of the ERP, which will improve the utilities’ commercial orientation. However, the utilities continue to struggle with the implementation of several modules due to a lack of capacity. Recommendations: For future activity design, USAID should: 1. Ensure that utility-based interventions directly target individual utility needs and be based on independent assessments or studies that take into consideration the basic requirements of the utility to accept such interventions, including the availability of water and infrastructure. 2. Ensure logical sequencing and prioritization of implementation and inputs. For example, key interventions such as the ERP should be prioritized early in activity implementation. 3. Ensure that advocacy to state government includes clear project deliverables and MOUs include verified commitments to implement identified activities (e.g., funding, human resources,). 14 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 4. The state government milestones within E-WASH were novel and well appreciated. However, future milestone efforts should consider including critical inputs, such as chemicals and electricity cost, to ensure smooth implementation and sustainable operations. EVALUATION QUESTION 1A To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? The Utility Support Fund (USF) was used for key interventions ranging from capacity-building to the supply of equipment, rehabilitation of waterworks, customer reconnections, extension of transmission networks, and accountability software. To enhance operations, the USF was utilized for power supply and waterworks improvements for some utilities. The Activity improved the information technology environment of the utility headquarters by providing desktops, laptops, tablets, and printers, as well as the installation of wireless Local Area Networks (LANs). The USF was also utilized to reduce NRW with the provision of acoustic electrical water leak detection equipment and pipe locators, and the supply and installation of bulk production facility meters and prepaid domestic meters. A respondent in Taraba noted appreciation for these efforts, stating, “a level of impact can be felt on the water board and consumer, the metering was commended because sharp practices [were] curbed.” The Activity also aimed to address drinking water quality and water production challenges by providing production equipment (high and low lift pumps), improving automation with the supply and installation of automated control panels and circuit breakers in selected waterworks, and the provision of water treatment (chemical dosing pumps, etc.) and water quality testing equipment. Sections of the water work treatment processes, such as the clarifiers and sedimentation tanks, were also rehabilitated. Mixed USF Implementation Status: ET site visits confirmed the availability of procured equipment at the sites visited. However, the ET observed that some of the USF construction activities were not yet completed, which hindered their intended impacts. For example, during the visit to the Chachanga waterworks in Minna, many improvements were noted, but some challenges remained. The ET observed improved automation and confirmed that the dosing pumps had been supplied and were in use. Furthermore, the ET observed that the rehabilitation of the filter and clarifier had improved the turbidity of the water. However, significant amounts of algae were seen in one of the sedimentation chambers. Staff explained that they were still working on this and were scheduling the cleaning of the tanks to avoid a total shutdown, which would affect water production. In Niger, the drinking water quality report from July to September 2021 indicates challenges with turbidity and related parameters. Other E-WASH successes reported by respondents include the improvement of data management processes and the generation of revenue from the USF-supported equipment purchases. In Imo, one respondent noted “the Utility is generating revenue from the use of laboratories by the public and organizations like Shoprite patronize the utility to conduct water quality test.” Additional areas of success include bulk metering, implementation of the ERP, and shifts toward automation. Despite these successes, there is wide variation in the completion and effective utilization of the USF between states. Two examples include the ongoing, innovative DMA intervention in Minna, Niger. This intervention is laying nine km of uPVC distribution pipe and installing 1,670 prepaid household meters. As of the ET’s visit, the intervention was 90 percent complete and has the potential to 15 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION reduce NRW and improve collection efficiency. This intervention is highly advanced in comparison with the intervention in Owerri, Imo, where meters have been purchased but not yet deployed due to the absence of transmission mains, which creates a major gap in DMA isolation. Some respondents attributed the challenges experienced with the USF to the late uptake of USF￾supported activities by the IP. As one stakeholder framed it, “[The USF] has been quite successful, the success would have been higher if it had been utilized earlier in year two as was planned.” Poor scheduling of training and delivery was also a challenge, as previously noted in EQ 1. Other respondents noted procurement delays due to COVID-19. One respondent in Delta indicated that “…COVID-19 has a serious negative impact on SWB in the sense that it slowed the Utility operation and affected the USF implementation.” They highlighted that “it took over a month to get the supply of filter media to the treatment plant from nearby Delta State due to the lockdown.” Respondents also reported poor deployment of the USF to address site-specific needs. For example, according to a utility respondent in Imo, despite USF support for the water production facility, it cannot be utilized due to a defective backwash system. The USF intervention did not take this into account when planning the other upgrades. A respondent explained that “the permanent solution is to rehabilitate the blower machine. When the blower machine is working, the backwashing can be done. If the filter belt works, that means there will be a water supply. And when there is water supply, consumers will pay for service, and revenue is generated to the utility. USAID has done well but backwashing without the blower will be an issue.” A respondent in Imo also noted that rehabilitating the backwash pump had been removed from the USF, which reduced the intended effect on water production. Reports from the Design and Supervisory Consultant in September and November 2021 discussed USF delays, indicating that they were due to, “(1) there are many effects of political, social, financial encumbrance on the individual projects, for example, sit-at-home order in the South, exchange rate, banditry in the North, etc. (2) the subcontractors did not follow terms of agreement with respect to channel of communication; some still take instructions from non-appropriate sources, (3) inadequate equipment on-site, general low capability, and competence of few subcontractors.” Feedback from a variety of stakeholders indicates that these delays were a significant limiting factor to the effectiveness of the USF. Stakeholders also linked the delays to a failure to realize the aims of the USF interventions, which were designed as pilots to catalyze other utility interventions. One respondent noted that “USF was to provide support where it is needed most, mobilize the private sector, and the piloting, identify key areas and provide a boost from the USF.” They noted, “Though a good plan, there were substantial delays [which impacted the activity].” Respondents also said that the implementation of the project towards the end of E-WASH will limit the realization of the outcomes during the project’s lifetime. As one respondent noted, “we will not get to see the full potential of the fund.” USF ERP Implementation: Despite significant variation between the states in the completion and effectiveness of the ERP, its supply and installation is a major step forward in ensuring accountability and strengthening the governance, financial, and operational viability of SWCs. The ERP uses the SWC’s LAN and has been customized to include asset management and M&E modules in addition to the standard modules. Currently, the ERP can only be used in the utility headquarters due to its dependence on the LAN of the utility, which is a limitation. While respondents lauded the ERP as an effective tool for accountability, some noted concerns about its sustainability and continued utilization after the exit of E-WASH staff. Other respondents 16 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION doubted that the government will pay the ERP’s recurring costs and fear resistance by utility staff who may have been benefiting financially from weak systems of accountability. Conclusion: The USF is expected to have some positive effect on utility operations, including on key indicators related to water production, customer confidence, revenue generation, and accountability. The limitations described above may inhibit progress, even for the more strongly performing utilities in Niger and Taraba, and the improvements may not be sufficient to place the SWBs on track towards overall financial and operational viability. The ERP has the potential to be a major contribution to the sector. However, its long-term impact is contingent on sufficient political will and ownership to ensure that SWCs continue to use the ERP systems after embedded E-WASH staff leave. Success is also contingent on local governments providing funds for retraining, hosting the platform, and manning the help desk. Recommendations: 1. For future activity design, USAID should ensure that the scope of interventions is designed in collaboration with the utility, capturing all components required to make the system fully functional and to secure ownership. 2. If future activity design includes a mechanism similar to the USF, the projects should be implemented early in the activity to maximize benefit and impact. EVALUATION QUESTION 1B What impacts if any has COVID-19 had on SWB’s financial viability? COVID-19 affected the utilities’ financial viability negatively and positively. IPs continued to support the utilities through virtual trainings and meetings during lockdowns. Procurement and Construction: COVID-19 delayed procurement and construction activities which could have benefited the delivery of water supply. For example, in Imo, the delivery of filter media for the treatment plant was delayed. The E-WASH procurement and construction activities were delayed in all states by between nine and twelve months. Revenue: The negative impacts of COVID-19 include a decline in revenue reported in two out of the five utilities (Niger and Delta). In Niger, revenue dropped from NGN nineteen million to just NGN three million, even though production was not impacted. Respondents indicated that the primary factors affecting the decline in revenue were the inability of customers to pay at the bank, customers’ inability to recharge meters, and the inability of staff to distribute bills and follow up on revenue collection. COVID-19 also negatively affected the ability to physically support the M&E process and the periodic verification of data in Taraba. To mitigate the effect of decline in revenue, respondents in Niger noted that the utility introduced e-payment platforms, bulk text messages were sent to customers with the USSD codes, and additional sensitization was carried out by the utility. These were widely accepted by the customers and have assisted towards improving the revenue. On the other hand, the Imo state SWC took advantage of the period to undertake maintenance. COVID-19 also provided the utilities the opportunity to leverage technology to drive efficiency. In Niger, Interswitch payments were introduced to help customers pay from home. In Abia, the use of virtual communications and meetings became the norm. 17 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION A look at the collection efficiency indicator (Table 4) shows steady improvement from 2019 to 2021 across all the utilities except Abia, where there was no water production, and Imo, where the treatment plant was being rehabilitated. Between 2020 and 2021, the collection efficiency of Imo and Niger declined from 38 percent to18 percent and 44 percent to 30 percent, respectively. Imo’s decline was as a result of the rehabilitation of the treatment plant while Niger’s was attributed to COVID-19 and the introduction of the TSA government policy. Table 4: E-WASH State Collection Efficiency by State (Indicator 1.2.1)13 Baseline LoP Targets FY19 Reach FY20 Reach FY21 Reach Abia 0% 40% 0% 0% 0% Delta 0% 75% 26% 44% 72% Imo 0% 40% 0% 38% 18% Niger 14% 75% 42% 44% 30% Taraba 18% 60% 0% 61% 68% Average 6% 58% 14% 37% 38% Increased Awareness: Despite the challenges posed by COVID-19, the utilities noted some positive impact in that it provided them the opportunity to educate people on the importance of clean water supply to combat COVID-19. Conclusion: Overall, COVID-19 had negative impacts on the utilities’ financial viability. Niger and Delta were affected financially, and for the other utilities, the primary impact was in the form of delays in the delivery of the USF activities. COVID-19 helped create positive awareness of the importance of water supply among utility customers and the opportunity for the utilities to begin to leverage technology and innovative payment platforms. Recommendations: 1. For future activity design, USAID should continue to support utilities improvements to billing and collection processes (e.g., prepaid meters and other e-method payment options). 2. Future activities should leverage available technology platforms for capacity development and meetings. EVALUATION QUESTION 2 To what extent did the Activity support operational viability and professional management of SWBs? E-WASH supported both the operational viability and professional management of the SWBs through the deployment of the USF (see EQ1a) for capacity-building, rehabilitation of treatment plants and network expansions, and organizational development. Operational Viability: The Activity supported water production improvements in some targeted treatment plants, including Niger, Taraba, and Imo. The Activity also supported improvements to water treatment plants in Abia and Delta, which were still ongoing at the time of evaluation. In Taraba, the automatic voltage regulator system was provided to improve power supply to the treatment plant. In Niger, the clarifiers, sedimentation tank, and other systems were rehabilitated to 13 USAID E-WASH Indicator Progress Tracking Table. 18 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION improve the operational efficiency of the water treatment plant. In Imo, the rehabilitation of the Otanmiri water treatment plant has improved water supply to the city. The ongoing rehabilitation of the Ariaria water treatment plant seeks to provide water to the residents of Abia. The installation of bulk meters in some treatment plants was also directed towards improving operational viability. Table 5 demonstrates an overall improvement in capacity utilization from 2020 to 2021. Table 5: Capacity Utilization Efficiency14 State Baseline LoP Target FY19 Reach FY20 Reach FY21 Reach FY22 Reach % Reach in LoP Abia 0% 60% Unavailable 0.0% N/A TBD 0% Delta 0% 60% Unavailable 0.3% 0.3% TBD 0.4% Imo 0% 65% Unavailable 1.3% 2.8% TBD 4% Niger 33% 83% Unavailable 33.6% 33.5% TBD 40% Taraba 12% 72% Unavailable 13.8% 21.7% TBD 30% Average 9% 68% 0% 9.8% 11.6% TBD 17% Challenges to Operational Viability: Before the commencement of the Activity, there was no water production for over a decade in three out of the five states (Abia, Delta, and Imo) according to baseline reports. It took the introduction of the Emergency Action Plan to restore production to Imo and Delta, while Abia had still not produced water more than three years into implementation. This limited operational efficiency improvements in comparison to Niger and Taraba, which had water production before the commencement of the Activity. Another key component of operational viability is efficient use of staff resources. A common measure of this efficiency is the ratio of staff per thousand connections (Figure 7). The industry standard is three staff per 1,000 connections. The staff per 1,000 connection rates for all states showed some improvements. Niger and Taraba reduced their ratio from 55 to 34 and 184 to 133, respectively, but the rates are still high in comparison to industry standards. The ratio was also improved due to institutional reorganization and the creation of the Small-Town Water Supply Agency in Taraba and Niger, which was the impetus for some staff to be moved to the new Agency. 14USAID E-WASH Indicator Progress Tracking 19 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Figure 7: Number of Utility Staff per One Thousand Connections in E-WASH States In addition, the OCAT index score measures the ability of a water utility to provide effective service to urban populations. Attainment of a score of 75 percent indicates the utility is performing against good practices of well-performing utilities. All five utilities made progress towards improving their OCAT index from baselines but did not achieve the LoP target and did not meet the industry target of 75 percent (Figure 8). Both Niger and Abia’s scores declined in 2021. Niger’s decline was attributed to TSA and Abia’s was linked to the inability of the utility to provide water services. Figure 8: OCAT Strength Index by State15 *The OCAT Index tracks the utilities’ overall progress towards achieving improved capacity. The average OCAT index score rose from 29 percent at baseline to 40 percent, reflecting some improvement by the utilities. Attainment of a score of 75 percent indicates the utility is performing against good practices of well￾performing utilities. All five utilities made progress towards improving their OCAT index from baselines but did not achieve the LoP target and did not meet the industry target of 75 percent. Both Niger and Abia’s scores declined in 2021. Niger’s decline was attributed to TSA and COVID-19and 15 The OCAT index measures the ability of a water utility to provide effective service to the urban population. Attainment of a score of 75 percent indicates the utility is performing against good practices of well-performing utilities. USAID E￾WASH Indicator Progress Tracking Table 20 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Abia’s was linked to the inability of the utility to provide water services. Overall, there was improved OCAT strength index scores across all the utilities. Professional Management: To support the professionalism of the SWBs, the WASH law helped give legal backing to utility corporatization (see EQ 3). Four out of the five utilities have established BODs, who now hold regular board meetings. The Activity supported the utilities in Niger, Delta, and Taraba to develop their vision, mission, and value statements. An organizational development study and Performance-Based Contracts (PBCs) were developed for all five states. Three states (Niger, Delta, and Taraba) started to make progress towards the implementation of the recommendations from the organizational development documents, while Abia and Imo indicated that their comments were not incorporated. In Imo, utility respondents noted that there would have been a clash in the work schedule of the Water Services and Commercial Departments if the recommendations were implemented. A respondent from Delta noted that, “[Human resources] is the biggest beneficiary of the organizations with [organizational development] undertaken, performance management plan, development plan, staff recruitment plan, [Human resources] Manual, [Human resources] staff retention, succession, staff training plan, modules in the ERP. Some of the documents are consulted when occasion demand. In terms of staff development, there is also the [Human resources] Module within the ERP.” In addition, strategic business plans were developed for all the water utilities to improve their operational viability. The utilities of Abia and Taraba introduced time and attendance monitoring to track staff punctuality. A respondent in Abia noted that time and attendance were taken seriously and were strictly enforced by the Managing Director (MD). There were several reports of positive changes in staff attitudes towards work; more conducive working environments; and improvements in punctuality, operational viability, and professionalism. Furthermore, a respondent in Imo noted, “Workers’ attitudes and dressing codes have changed, there is an improvement. The staff shows professionalism now.” Another responded noted, “[General Managers] are more enthusiastic, I noticed a change in resumption times, early start, late closing when I visited.” Capacity-building was also noted across all states as integral to attitudinal change and work culture. This sentiment was shared by a respondent in Delta who noted, “The staff cannot operate a system, but now they can. The utility has changed from water board to water corporation, which has changed staff orientation. Staff attitude to work has been lukewarm hitherto, but now people are enthusiastic to work. Training and workshop led to the improvement.” Improvements in data monitoring and collection systems were noted in all states. The use of data to drive management decisions was noted in Taraba to have helped in improving operational productivity, described in EQ1. The introduction of automation through the supply of information technology infrastructure and capacity-building was also viewed as improving operational efficiency and reducing manual processes in finance and human resources. Similarly, the introduction of the ERP for automated billing in Taraba, Imo, and Niger and the MCASH integration in the Niger ERP to support web payments were viewed by the utilities as important tools for operational viability. Challenges to Professionally Managed Utilities: Despite Activity interventions, several challenges continued to hamper success. In Niger and Taraba, the key conditions and incentives for a push away from a civil service approach have not been implemented, such as salaries, benefits, and performance contracts, due to a lack of approval by the state governments. In Delta, a lack of transparency and teamwork, staff rivalries, staff shortages, inadequately qualified staff, and struggles over departments among management staff have hampered progress. Imo and Abia have not accepted the recommendations of the organizational development documents; they both noted that 21 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION the comments from the utility were not incorporated into the assignment. As one respondent from Imo state mentioned, “There is Organizational Development (OD). However, the OD document developed has been very difficult to implement for certain reasons. Another shortcoming of the E-WASH project is the utilities suggestions were not taken into consideration while designing the organizational structure. The consultant consulted the utility workers but never took on board their suggestions and recommendation. The merging of the water services department with the commercial department cannot work. This is because of their differences in skills possessed. For example, the question of who leads the department remains unanswered.” Additionally, job descriptions and other policies (e.g., human resources policy, staff retention plans,) have not been implemented, even where additional funds are not required. Lack of autonomy across the states, and the inability of MD’s to hire and fire staff are affecting the operations of the utilities. For example, 80 percent of the staff in Taraba are contract staff. While the government of Delta has signed the PBC with the MD and zonal managers, it has not been cascaded down to other staff. There were no monitoring mechanisms to track the achievement of the PBC. In Niger, PBC implementation was stalled by the introduction of the TSA. Abia, Imo, and Taraba have not signed the PBCs. Respondents also questioned the impact of capacity-building efforts, noting that a critical proportion of those trained was close to retirement at the time of the training and have since retired. Respondents in Niger noted that ‘’70 percent of the management staff has barely a year left in service.” They also noted inadequate on-the-job training. Respondents mentioned that the duration of some of the trainings was too short to make a meaningful impact. The Coca-Cola training was conducted online, which led to issues for many attendees due to poor internet connections. Conclusion: There were significant enhancements through capacity-building and training, which have led to a reported improvement in work ethic and culture. Many performance weaknesses will not be remedied without a functional structure, appropriate incentives, remuneration packages, staff retention plans, and autonomy. Recommendations: 1. Future USAID projects should consider conducting and implementing organizational development early in the activity so that functional structures and human resources are in place. This should include proper staff recruitment and placement as well as policies for staff remuneration and incentives. 2. For future activity design, USAID via its Ips should develop a roadmap to assist the utilities in planning full implementation of anticipated activities (organizational development, strategic plans, PBC, strategic plan, etc.). This roadmap should include details regarding the timing, funding sources, and other pertinent details to ensure effective implementation. 3. USAID should consider future activities to support the states’ implementation of the PBCs and ensure they are cascaded across the utility staff. This should include realistic targets, proper incentives and penalties, and ensure that the regulatory agencies are empowered to monitor and track progress. 4. The Delta, Taraba, and Niger SWCs should intensify the implementation of the organizational development study and the states should support the utility with funding to realize full implementation. 5. Future projects in Abia and Imo should review the existing organizational development study jointly with the utility and support its implementation. 22 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 6. The utilities should continue to prioritize capacity development of its staff, to sustain the various systems established by the activity. EVALUATION QUESTION 3 To what extent did the Activity strengthen policy, institutional and regulatory frameworks for improved WASH SERVICES at the state and national level? State Level: WASH laws and policies provide effective administrative and regulatory mechanisms for sector management. E-WASH contributed to strengthened policy, laws, and institutional and regulatory frameworks at the state level across the five SWBs. The passage of WASH laws occurred in Delta 2019 and the remaining states in 2020.16 E-WASH supported the SWBs through technical support, sensitization, advocacy, capacity-building, mentoring, linkages, and relationship brokerages. These efforts pushed the utilities towards a more commercial orientation. An E-WASH respondent said, “The [WASH] policy was able to achieve its objective to a large extent. It helps to strengthen their role in water policy institutions in the states and the policy law in which the citizens were able to know their rights and that of government obligation in place.” The WASH law also provided for the creation of a small￾town water agency, a Rural Water Supply and Sanitation Agency, and an urban water agency.17 However, only a few of these agencies were operational due to a lack of water production (e.g., in Abia), indicating important enabling factors are inhibiting service delivery. The WASH laws helped all five utilities transform their SWBs into SWCs, which supported commercial orientation. Utility and government respondents in Abia and Delta noted that the laws enabled some of the corporations like Abia State Water and Sewage Corporation and Delta State Urban Water Corporation (DESUWACO) some level of autonomy. However, they also said that full autonomy will only be realized when the utilities achieve financial independence. A respondent noted that in Delta, despite the utility having its own budget and ability to decide on capital projects, the NMWR still handles procurement. Rather than the NMWR playing an oversight role as the policy indicates, the NMWR is duplicating the utilities’ procurement process and engaging in construction activities, which indicates that there is not yet full compliance with the WASH law. In Niger, respondents shared that the law provides autonomy for hiring and firing staff. However, this component has not yet been implemented as permission is still needed from the government to hire or fire staff. Notably, in Imo, the SWC was autonomous from the State Ministry of Water Resources and it was removed it from the civil service structure. While strides were made, there remains significant work for utilities realize fully autonomy. The law also established regulatory mechanisms such as BODs, the ERP, PBCs, and other processes and systems to improve WASH service provided by utilities (please see EQ3 A for a description E￾WASH supported regulatory activities). However, a lack of funding was cited across states as a primary barrier to the implementation of a full suite of regulatory mechanisms. In Delta, a Steering Committee on Sanitation and a Sanitation Agency were established, and the water sector regulatory commission was trained on how to monitor performance. However, respondents noted that the regulatory commission has been limited by the lack of a proper working environment (e.g., no office space) as well as inadequate staffing and funding. The ET observed in Taraba that the regulatory commission has yet to deliver on its mandate. This was further corroborated by a stakeholder that 16 The WASH Policy was established and then was introduced as a bill to the House of Assembly for passage into the WASH Law. 17 The small-town water agency served communities with 5,000 to 20,000 people, the Rural Water Supply and Sanitation Agency served communities with less than 5,000 people, and an urban water agency served communities with more than 20,000 people. 23 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION reported this was due to a licensing issue and a lack of permanent office. In Imo, the regulatory commission and BOD had not yet been implemented due to lack of funding. In Niger, the law provided for a regulatory unit in the Ministry of Water Resources, but it is not yet functioning, as the initial two staff employed had to stop work due to budget constraints. In Taraba, regulatory implementation was impacted by issues of autonomy; the current Permanent Secretary was the former MD of the utility, which created questions of regulatory independence. A respondent stated that the executive secretary of the regulatory commission reports to the Permanent Secretary and the Commissioner, so the arrangement does not provide independence to the regulatory commission. National Level: E-WASH did not work on policy, law, regulatory frameworks, and others at the national level. However, the E-WASH national office supported the sector through their facilitation, engagement, and deployment of consultants and experts who work with the state offices to develop the policies, laws, and regulatory frameworks. Other: E-WASH also supported other policies and guidelines including gender policies, organizational development documents, and MEL plans. However, some of them have not been fully implemented. The paragraphs below summarize the status of these policies. Gender Policy: There was evidence that E-WASH supported and influenced the participation of women in decision-making through the gender policy developed for the utilities. As one IP respondent noted, “…four female out of eleven management members in [DESUWACO] have been empowered to amplify their voices in the management decision-making process.” Similarly, the CSO gender policy put in place in Delta has continued to advocate for the rights of women, children, and marginalized youths as well as carrying out some programs centered on girls’ hygiene in secondary schools. This is because the gender policy specifies the need to carry out gender￾specific activities. In Abia, the utility has been utilizing the gender policy E-WASH supported to develop a balanced representation of females and males in training and other utility activities. A KII respondent in Abia said “Previously, there was no gender unit because the utility did not know that gender can be mainstream into its activities. This has helped the utility to know how to plan for the ratio of females to males in any training to be conducted.” Gender and pro-poor units were also set up in Niger, which a respondent said increased women’s decision-making ability. Organizational Development Documents: The organizational development documents are currently being implemented in Delta and Taraba and it awaits Governor approval in Niger. A respondent from Taraba said, “the organizational development Activity has not only led to change with staff but also in data to monitor performance.” Meanwhile, the organizational development documents were not implemented in Imo and Abia because the utilities felt their views were not reflected in the document. A stakeholder from the Abia State Water and Sewage Corporation noted that the organizational development documents may not work because the feedback staff provided to the consultant was not reflected and the consultant could not have created the organizational development alone without taking into consideration the perspectives of the utility. The organizational development documents have been difficult to implement at ISWASC. One challenge was that ISWASC could not merge the water services department with the commercial department because of the differences in personnel skills. The question of who leads the department remains unanswered. The different structures and nomenclatures that exist in the states may explain why some of the states could implement the organizational development documents and some could not. MEL Plan: The MEL Plan is currently implemented in Delta, albeit slowly. According to a utility respondent, it was not implemented in Imo as planned because the key stakeholders were 24 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION unable to appreciate the importance of data because there was little or no water production. In Delta, MEL plans were stalled by lack of capacity. MEL plans were not implemented in other states as part of the Activity. Conclusion: E-WASH successfully supported the development, implementation, and partial operationalization of WASH policies and laws across all five states. It also supported the strengthening of the institutional and regulatory framework in Imo and Taraba. The policies and laws steered the utilities towards corporatization and a more commercial orientation, which granted some of the utilities a level of autonomy. However, there is no evidence to indicate that the policies, laws, and institutional and regulatory frameworks translated into improved service delivery in terms of provision of water for the citizens. Of note, a lack of adequate funding also hindered the operationalization of some of the regulatory frameworks. There were other policies and guidelines (e.g., gender, sanitation) that are still currently undergoing development in some states and other aspects of the policies have not yet been fully implemented due to constrains in capacity and funding. The implementation of the MEL Plan in Delta was stalled by a lack of capacity. Recommendations: In an effort to ultimately improve service provision: 1. Through its implementing partners, USAID should advocate for the inclusion of a roadmap that identifies the sources of funding for future WASH policies and laws to ensure effective implementation. 2. Through its implementing partners, USAID should continue advocacy to the utilities to ensure that the government fully implements current WASH policies and laws, as well as complete the ongoing development of WASH policies and laws. EVALUATION QUESTION 3A What accountability mechanisms related to implementation of policy and frameworks did the Activity put in place, how successful have they been, how could they be improved, and how might they impact sustainability? Evidence suggests that E-WASH supported the utilities to put in place several accountability mechanisms and frameworks related to policy implementation across the five states. The accountability mechanisms include the establishment of BODs, the ERP, PBCs, WASH media forums, auditing processes, procurement systems, International Public Sector Accounting Standards (IPSAS)/cashless payments, a performance monitoring and revenue assessment, a tripartite performance agreement, Performance Regulatory Committees (PRCs), and Urban WASH Forums. BODs: E-WASH supported the establishment of functional BODs in four states, Abia, Delta, Niger, and Taraba. Imo was unable to establish a functional BOD due to political instability, which distorted the utility’s activities and challenges with political will due to the financial commitment it required. According to E-WASH and utility staff, the BODs performed oversight through approval of yearly expenditures, reviews of audited accounts, and approval during the procurement processes where necessary. This has helped to improve transparency and accountability in the utilities. The BODs in Abia, Delta, Niger, and Taraba have been successful in performing their mandates. As a respondent in Abia framed it, “[The] BOD is in place, and they meet regularly. They provide an annual report, they take decisions on certain issues. The audited account of the utility is looked into by the board. The utility gets approval from the board.” This was also corroborated by a stakeholder in Taraba that said, “We have a charter for the board, we have meetings, we are given various areas to supervise, places where we visit, 25 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION there are committees that supervise the technical operations of utility by extending lines and do oversight functions. The utility gets approval from the board.” However, the ET observed that the composition of the BODs varied across the states. In Abia and Niger, professionalism was taken into account in the selection of BODs members. The BODs in both states were comprised of seasoned professionals representing a variety of stakeholder groups. A respondent in Abia stated, “The selection of the BOD is well represented. The members cut across all spectrum of human endeavors like Engr, Doctors, Barrister, and other professionals.” The Abia BOD assigned functions to different committees, such as legal, performance monitoring, and other specialized committees, and stated their commitment to advocating funding. In Delta and Taraba, there was no consideration for professionalism in the selection of the BODs members. According to respondents in Delta, the selection of BOD members was based on political patronage and their meetings were usually informed by the benefit that will accompany them. In terms of the sustainability of the BODs, they appeared to be sustainable beyond E-WASH due to the law that created them. One stakeholder stated that “the BOD is sustainable because by the virtue of the law it is existing, and it will continue to exist unless the law is repealed.” Another respondent from the Niger utility said, “the BOD is sustainable because of the professionals leading the board based on the clear-cut selection criteria in the law, and the representation by professional bodies, Nigerian bar association, Nigerian Engineering, Chambers of commerce.” However, as noted above, in Niger the BOD had not been implemented due to a lack of funding. ERP: E-WASH supported the development of an ERP in each of the five states. The ERPs were designed to help the utilities migrate broad financial management practices, customer billing, revenue collection, human resources, customer relations and operations process from manual to automated processes. As a respondent from the utility in Imo said, “The ERP put in place brought about transparency and checkmate leakages in the system.” Usage of the ERP varied across the states. The direct observations done by the ET in Abia, Delta, and Imo found that all the ERP modules are functional. However, they are not fully operationalized due to the non-production of water in Abia, and capacity issues of the staff in Delta and Imo. Several states faced staffing-related challenges in maximizing the utility of the ERP. In Imo, the ERP was used by the utility to automate and improve data quality. However, ERP usage was limited by the staff’s inability to effectively use computers, a reality confirmed during the ET’s visit. The ET found that the finance department’s system in Imo had been idle for several months. Imo also had staffing and succession challenges for managing the ERP. The staff that were professionally trained retired, and the new staff lack adequate skills and knowledge to perform the tasks. A utility respondent in Imo said, “Another issue is that most of the workers trained are leaving soon as they have completed their required years of service. Some have even stopped coming to the office.” Staff preferred their previous manual processes and record-keeping methods. The ET could not physically ascertain the extent of the ERP usage in Niger and Taraba, as site visits were not possible due to COVID-19. However, it appears that the assigned staff are struggling to effectively manage the ERP. Because of the current challenges, some respondents were concerned about the sustainability of the ERP after E-WASH ends. A stakeholder stated that “The issue is if the utility will continue using the ERP after the embedded E-WASH team leaves. Will they feel that it is a barrier to the financial benefits some individuals enjoy before or without the ERP installation?” Performance-Based Contracts (PBCs): E-WASH advocacy and mentoring led to the formation of PBCs in Delta and Imo. However, they have not been operationalized in either state. In Delta, the DESUWACO General Manager has signed the PBC with the Delta government. However, it is not 26 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION yet enforced because the regulatory commission mandated to enforce it is not functioning because it does not have an office or staff. In Imo, the ISWASC General Manager has not signed the PBC with the Imo government. The PBC was not operationalized in Abia because of a lack of water. A government respondent said, “Performance Management Contracts can’t be signed now until the Ariaria and Umuahia water scheme starts operation; then the government can be releasing funds.” In Niger, there is a draft PBC developed, but it was delayed due to issues around the TSA. However, a special performance monitoring group was set up with prizes and staff rewards. The lack of success with PBCs is due to the inability of the utilities to produce water. If the government can support the utility to generate water, then the utility managers can be held accountable for the sales and revenue generation. WASH Media Forums: E-WASH also supported the utilities to develop WASH media forums, WASH customer forums, and customer charters to strengthen the relationship between the utilities and the public in each of the five states. External accountability structures like the WASH media forum promote transparency in utility operations by sharing key data and information about performance. The utility in Imo started to change the general public’s perception of the utility and has boosted the utility’s image. The WASH media forum was successful in delivering its mandates. However, the descoping halted these activities and they no longer function due to a lack of funds. A respondent from the IP recommended, “The challenge is resources for them to convey because most of their activities are funded by E-WASH. Government should allocate resources for CSO activities.” Auditing Processes: The advocacy and mentoring effort of E-WASH led to an auditing process in each of the five states. The internal audit processes are reportedly helping to improve transparency and accountability. These audit processes may be sustainable because they do not require any financial commitment or further support from E-WASH to function, and because most of the utilities have functional BODs. Procurement Systems: E-WASH supported the creation of a procurement system in Delta and Abia. The new procurement processes aimed to improve transparency and accountability. As a part of these processes, the BODs provide oversight to the utilities’ procurements. Similar to the sustainability of the auditing processes, the procurement systems may be sustainable due to functional utility BODs and lack of additional required E-WASH support or financial commitment. IPSAS/Cashless Payments: E-WASH supported the establishment of IPSAS/cashless payments in Abia and Delta. Both states have been using the IPSAS/cashless payment system to improve transparency in the utilities. A respondent from the Abia utility said, “The application of a cashless policy is now automated. No middleman sees it; it is from the utility to the end-users. There is no more use of driver to go for a withdrawer as the old system was. Also, the docket system has been introduced to the purchase of diesel in the filling station no more handling of cash. All this started under the corporation.” In terms of the sustainability of the IPSAS/cashless payments, a respondent from the Abia utility said, “The structures are sustainable. E-WASH is doing nothing or little now to the utility, but they are carrying on their activities. The board is also functioning, they are meeting regularly.” Performance Monitoring & Revenue Assessment: E-WASH supported the establishment of a performance monitoring and revenue assessment in Niger. The ET found that this mechanism was not yet functioning at the time of evaluation. A respondent in Niger said, “Five-year strategic plan was done by the state after E-WASH gave them a plan which they could not implement. These include performance monitoring, revenue assessment, and holding people accountable for their designation.” 27 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Tripartite Performance Agreement: Due to the advocacy effort of E-WASH, a performance agreement was signed between the Niger SWC MD, the board, and the state government in Niger. A utility respondent in Niger said, “The tripartite arrangement is a success.” This structure seems to be sustainable in Niger because the utility sees its relevance. Performance Regulatory Committee (PRC): The E-WASH Activity supported the Taraba State Water and Sewerage Corporation’s (TAWASCO) establishment of a PRC in Taraba. The PRC is domiciled in the Ministry of Water Resources and the utility is accountable for it. The PRC is used to assess the performance of the utility. In terms of its sustainability, a stakeholder said, “The PRC has the issue of funding and has no accommodation as of now.” Thus, the PRC’s sustainability rests on the ability of the state government to fund it. Urban WASH Forum: There were E-WASH-supported urban WASH forums in three states (Imo, Delta, and Niger). These forums are not sustainable because the CSOs who championed them stopped receiving funding from E-WASH after the descoping. Others: In Delta, E-WASH also supported the utilities set up other accountability mechanisms, such as gender forums, men’s clubs, open governance processes, budget committees, reform champions forums, and customer charters. However, only some of these mechanisms continue to function. Respondents report that those that do not require funding, such as the customer forums and the complaints systems, are continuing, while those that require outside funding are not. Activities run by CSOs were affected by the descoping and thus they could not continue the activities. An IP respondent indicated that out of the ten zones where E-WASH set up accountability structures, only the customer charter in the Asaba zone, where water is being produced in small quantities, is still operational. In Imo, E-WASH supported a web-based complaint delivery system and a performance improvement program where the MD and the Head of Departments present their operational plans and targets. As noted with Delta, funding is a key factor in sustainability. A respondent from the Niger utility said, “The accountability mechanisms are sustainable if regular water services are provided which is dependent if the government gives money for operational fundaments for chemicals and pumping water. Though the government provided a generator, they are unwilling to provide diesel. Minna substation has never produced beyond 30 percent but can produce at 80 percent of its capacity if operational funds are supplied.” Conclusion: The Activity has put in place several accountability structures across the five utilities to improve the quality of relationships (i.e., clear roles and responsibilities, transparency, and accountability) between the different stakeholders. Internal accountability structures like the BODs, procurement systems, and audit processes moved the needle to promote transparency and accountability within utilities. External accountability structures like the WASH media forums have increased the transparency of utility operations. In some cases, the efforts have also contributed to changes in public perceptions of the utilities and boosted their image. However, many challenges remain, including a lack of funding, water supply, and commitment by the management of the utilities; capacity gaps in the use of the ERP; insufficient ownership of some of the frameworks; and incomplete activity implementation due to the descoping. Because of these challenges, some of these structures may not be sustainable beyond E-WASH. Recommendations: 1. For future activities, USAID should identify targeted accountability mechanisms within utilities and support future planning for capacity, knowledge transfers, and sustainable funding streams. 28 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 2. The utilities, through the customer units, should sustain engagement with the customers and WASH media forums to ensure their sustainability. 3. The utilities should strengthen ERP execution through further training and retraining of staff. EVALUATION QUESTION 4 To what extent did the Activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping? The Activity attempted to address sanitation in the five states through the SWBs. Sanitation interventions included policy and legal frameworks, institutional frameworks, CWIS processes, and service delivery. Each component is discussed below, followed by a discussion of key challenges and future opportunities. Policy and Legal Framework: E-WASH assisted all five states in developing and passing WASH policies and laws (see EQ3). As a part of this, sanitation management was added to the SWCs’ statutory responsibilities. Though a sanitation mandate was added, funding mechanisms to support the mandate were not established. This limits the SWCs’ ability to address sanitation. Institutional Framework: The E-WASH Activity supported the establishment of sanitation units or water conveyance and sewerage units within each SWC. It also supported coordination platforms such as the intersectoral City Sanitation Task Forces (CSTFs) in the urban cities of Aba, Asaba, Owerri, Jalingo, and Minna. The CSTFs, by virtue of their mandate and intersectoral nature, had a high potential for preparing the utilities to address sanitation challenges. A quarterly E-WASH report noted, “The tasks of the CSTF are to raise awareness among citizens on improved sanitation management policies and planned services; approve and endorse the city sanitation mapping reports and other deliverables (i.e., [SFDs]); participate in the CWIS plan development; coordinate city-level sanitation activities implementation; coordinate, build consensus and agree on how to define institutional responsibilities involved in sanitation management…); and inform the public and state government about CWIS plan progress.”18 There was a uniform mandate across all states. However, the ET found that all the CSTFs and sanitation units were weak and struggled with capacity, staffing, and funding challenges. Staffing of the sanitation units ranged from one to five people. The desk review found that these capacity challenges were often linked to insufficient guidance regarding funding mechanisms to build the capacity of the CSTFs. A government official in Niger state shared, “Sanitation awareness, capacity and infrastructure is very weak,” while a government official in Delta state shared similar sentiments, “Before now, sanitation was limited to the rural, but when E-WASH came, Urban sanitation was introduced. Which is still growing.” City-Wide Inclusive Sanitation Processes: The Activity supported the SWCs to manage sanitation in one city each (Aba, Asaba, Owerri, Jalingo, and Minna). To do so, E-WASH supported a CWIS process between June and December 2020 (Figure 9). The CWIS process aims to ensure that the statutory duty bearers have the necessary knowledge to promote a range of safely managed sanitation solutions so that everyone has access to safely managed sanitation. 18 Quarterly Progress Report: Quarter 2 FY 2021. 29 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Figure 9: City Wide Inclusive Sanitation Process19 The CWIS process included the identification of stakeholders, the establishment of committees, the generation of location-specific information around the sanitation management chain (containment, emptying, transportation, treatment, and reuse), and the identification of formal and informal stakeholders, infrastructure, and opportunities. The CWIS process also included the creation of SFDs, which give an indication of the flow of fecal matter through each city and the degree to which excreta is safely managed. It also included a City Service Delivery Assessment (CSDA). The CSDA identifies key aspects of a CWIS program including policy, planning, budgeting, expenditure, equity, and service outcomes. A CSDA process scorecard was also used for benchmarking. The analysis of the Information should ideally lead to further engagement with stakeholders, development of investment plans, and proffer costed, actionable solutions. However, the development of investment plans and timebound implementation plans did not take place in any of the five states. The CWIS led to increased awareness and campaigns against open defecation in most of the states (Abia, Delta, Imo). It also created additional income for an innovative activity in Aba who recycled the feces into manure. Service Delivery: The Activity intended to extend sanitation work into service delivery. However, none of the SWBs has been able to translate the CWIS process into action to address the identified challenges. For example, no state constructed the planned fecal sludge treatment plants (Table 6), despite this being a key expected deliverable of the project. 19 State Sanitation Mapping Reports, March 2021. 30 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Table 6: CWIS Implementation by State20 City STF SFD SMs Consult Investment and Strategic Plans Solution Implementation Abia Y Y Developed N N N Asaba Y Y Developed Y–- debrief with stakeholders N Public toilet construction ongoing in a market Owerri Y Y Developed N N App reportedly developed but not operational Minna Y Y Developed N N N Jalingo Y Y Developed N N N–- Forms developed to start formalizing de￾sludgers *STF – Sanitation Task Force; SM – Sanitation Maps Utility and government respondents provided several reasons why the fecal sludge treatment plants were not constructed as planned. These reasons include unavailable resources due to the descoping, the limited expertise of the E-WASH team to deliver this requirement, a governor not providing land for the “dump site,” water supply challenges, and challenges with Activity scheduling by the IP. However, a review of the actions recommended in the sanitation mapping reports suggests that some of the reasons given may not be root causes and are likely incomplete. The reports recommended numerous actions that did not require government or USAID funding and some of these activities were not implemented. For example, a respondent noted, “They have not established mandates to tell residents about sanitation disposal sites.” Annex VIII provides additional examples of actions proposed in the sanitation mapping reports. The CSDA provided several examples of sanitation activities that could have been completed, even without supplemental funding. Respondents reported that a campaign was conducted around open defecation by CSOs in Abia. Other examples of potential activities include engagement with stakeholders, exploration of additional sources of funding; development of investment plans; the use of community health workers that are already active in Aba, Owerri, and Jalingo to raise awareness about open defecation and fecal sludge management; coordination among statutory institutions that have sanitation-related activities; the revival of sanitary inspectors; and the formalization of manual de-sludgers. This was recommended because the manual de-sludgers work under very poor public health and statutory production conditions, face stigma and, in some cases, have to work at night as the activity is considered illegal by the state (Annex VIII). Sanitation Challenges: Deeper analysis of the CWIS reports indicates that there are six main reasons why the states failed to implement proffered solutions from the sanitation mapping: Challenging Sanitation Context: An extremely challenging sanitation context was highlighted in all the state sanitation mapping reports. This includes the lack of safely managed sanitation in all states, as noted in the SFDs; the absence of centralized sewerage infrastructure; weak decentralized sewerage infrastructure; a weak enabling environment; and the absence of functional 20 For all states listed, the planned start date was between Q3 of FY20 and Q4 of FY21. Actual start dates were between June 2020 and December 2020. 31 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION sanitation bylaws. A respondent shared, “Most of the utilities have not been mandated previously to manage waste,” and added, “Most of them don’t have sewers.” Incomplete CWIS Implementation Status: At the time of this evaluation, the CWIS exercise was not concluded in any of the states. CWIS follow-on activities were not yet implemented, including engaging with key stakeholders, presenting the outputs of the CWIS process, and translating findings into investment plans. No investment plans have been developed by any of the states. Timing was a significant challenge, as noted by a government official in Niger who shared, “The program came in very late, late 2020s. the city-wide sanitation mapping was not completed or concluded. The maps could not be used for the design of the wastewater treatment plants. The descoping also closed this down.” Poor Staff Capacity: Some of the key informants thought that staff capacity for sanitation was weak, both within the utilities and on the E-WASH team. They saw this as a significant limiting factor in the implementation of the sanitation component. According to a range of respondents in all states, the utility staff hired to lead sanitation in the newly established units were not sanitation experts and had little experience working in well-performing utilities or in wastewater treatment. A respondent indicated that “[the] plan was to develop fecal sludge treatment plants. RTI did not have the capacity or the expertise.” A government official in Niger state noted "sanitation awareness, capacity and infrastructure are very weak.” An exception is Taraba, which selected a candidate with a master’s degree in Environmental Management to head its newly established water and sewerage unit. Niger selected a female candidate who is an engineer and undergoing training in water and wastewater treatment engineering as the head of its sanitation unit. The Niger case is also an example of a utility advancing gender equity; five of the seven-unit heads in the state are female. Insufficient Resources: All the sanitation units were under-resourced, with only one to five staff in each. According to the Owerri State Sanitation Mapping report, “While the professional staff working at the city are dedicated, their departments are handicapped by understaffing and lack of budget.”21 Formal training in wastewater management and sanitation could have mitigated some of these capacity challenges. However, E-WASH respondents indicated that there was very little formal training in sanitation and no training in wastewater management in any of the states. Respondents indicated that a training plan was compiled and developed by the National Water Resources Training Institute after a capacity needs assessment conducted by E-WASH, but the respondents said the training could not be implemented after the activities were descoped. Other respondents queried the timing of the capacity needs assessment, with one noting “Staff capacity needs assessment should be done at the beginning.” While training on wastewater treatment was not conducted, some limited trainings were conducted in sanitation, but respondents felt that it was ineffective because some of it was conducted remotely. Limited/no water supply: As previously noted, many states faced water supply challenges. The lack of consistent water supply limited the states’ ability to manage sanitation. Varied respondents spanning the utilities, the sanitation task forces, and government officials identified water supply issues as a major challenge to the utility’s ability to improve sanitation, noting there can be no sanitation without water. Some of the respondents also linked poor water supply to poor customer confidence. They noted that the failure to provide water limited their ability to interest 21 Owerri State Sanitation Mapping Report, March 2021. 32 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION the community in engaging in sanitation improvement efforts through the utilities. As one senior stakeholder put it, “It was not completed in Niger state. Unfortunately, sanitation has always remained an orphan largely because you can’t do much about sanitation until you have water.” Timing and Sequencing of Activities: By the time the sanitation mapping reports and sanitation service delivery assessments were available in March 2021, two years into the Activity, the WASH policies had been developed and efforts to pass the laws had gained significant traction. This limited E-WASH’s ability to plan activities that could prepare the utilities for their sanitation duties. In addition to these scheduling challenges, respondents pointed to challenges with the utilities’ unfamiliarity with sanitation and its implications for the sustainability of interventions. One utility respondent indicated, “It should be emphasized that some of the E-WASH Activity is like putting the cart before the horse; despite the completion of the sanitation mapping in some cities and the development of an app to monitor sanitation-related activities, the app is not operational as there is no fecal sludge treatment plant.” Review of the RTI technical proposal suggests that the strategy of engagement before generating and analyzing information might be a contributory factor to the inability of E-WASH to use evidence from the CWIS or a pre-intervention situation analysis. Other stakeholders from the government believed that other Activity components should have been fully addressed before additional components were added. As one government official noted, “choose a specific component, focus on it and let it be sustainable rather than doing so much without sustainability.” Respondents noted similar timing challenges regarding the capacity needs assessment, with one noting, “Staff capacity needs assessment should be done at the beginning.” Opportunities to Build on Sanitation Preparation: During the interviews, some utility and other respondents noted that there may be opportunities to implement the sanitation mapping solutions in the future. For example, they could be implemented through the World Bank Sustainable Urban and Rural Water Supply, Sanitation and Hygiene Program (SURWASH) in Imo and Delta states. A respondent in Abuja noted, “Delta and Imo are expected to receive funds from the SURWASH project to construct fecal sludge treatment plants," and a utility staff member in Niger shared that “Road map and guidelines are available in Niger state for the utility to move on to implementation.” Conclusions: The E-WASH Activity’s sanitation-related interventions did not achieve their objectives. While some sanitation progress was made and groundwork laid with laws, policies, and frameworks, the policies and laws did not address funding mechanisms, which have hampered implementation. The utilities are still weak and struggling with capacity challenges, underfunding, and competing priorities. Though cited as best practice, the inclusion of sanitation in the mandate of the utilities was in its very early stages and the enabling environment for CWIS was not sufficient. Utilities lacked familiarity with sanitation and the severity of the challenges in the sanitation sector in Nigeria. These challenges include the absence of required infrastructure, skills, and resources; inadequate water supply; and the uncertain policy and institutional environment. These barriers proved difficult to overcome. Better timing relative to decision-making could have helped. For example, by the time the sanitation reports were available, the advocacy for the inclusion of sanitation into the mandate of the utilities had gained traction. The weak capacity and expertise in sanitation within the IP and the utilities, in conjunction with the challenges faced in the provision of water supply, contributed to an inability of the IP to appreciate the enormity of the situation, maximize opportunities, and implement the proffered solutions from the CWIS exercise. “Easy wins” could have included working with the utilities to use the opportunities created by the intersectoral CSTF to involve the already existing community health workers and sanitary inspections in sanitation promotion. 33 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Recommendations: 1. USAID should ensure RTI develops an exit plan as part of Activity closeout for each intervention state. The exit plan should incorporate the sanitation investment plans from the CWIS exercise to identify the key actions, opportunities, partnerships, and funding required to operationalize and strengthen sanitation units and task forces. 2. For future programming, USAID should consider the challenging sanitation environment clearly, as itemized in the sanitation mapping reports. This should be done prior to project design. USAID should focus interventions in preparing the mandated agencies to address these complex challenges. 3. USAID should provide oversight to ensure that institutional frameworks established by its IPs comply with best practices in Nigeria. For example, the composition of the CSTFs hampered instead of aided implementation. 4. USAID should make it a priority to support the State Water and Sewerage Corporations with high-level advocacy to state governors and external support agencies and identify donors who can continue to support this work. The objective would be to maximize the opportunities that the mapping identified. 5. USAID should ensure that IPs’ sanitation strategies assess the context early and often. This should begin before implementation and continue throughout. Additionally, USAID should require that IPs maintain staff with sufficient skills and abilities in sanitation. Penalties should be enabled if the requirements are not met. EVALUATION QUESTION 5 What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? The E-WASH Activity put in place numerous processes and plans to facilitate the implementation of the USAID environmental and construction guidelines. The Activity: 1. Developed an EMMP as part of the contracting process. 2. Implemented a Best Practice Report audit by USAID. 3. Conducted a two-day training for RTI staff in intervention states by the Design Engineer in June 2021. 4. Required contractors to contribute to the EMMP as part of progress reporting. 5. Required the design and supervising contractors to carry out Environmental Impact Assessments and develop Environmental Social Mitigation Plans (ESMPs) for all six construction sites. To assess the Activity’s implementation of USAID’s environmental and construction guidelines and how these efforts could be used for adaptive management, the ET first discusses the Activity’s capacity and oversight of the guidelines, followed by a discussion of compliance. Then, the ET discusses how these guidelines have or could in the future be used for adaptive management. Capacity and Oversight: Respondents indicated that the Activity did not have in-house staff at any point in time with expertise in USAID’s environmental and construction guidelines. They said that this posed a challenge and could have affected the Activity’s ability to respond to concerns in a timely and appropriate manner. As one respondent framed it, “Challenges included the fact that they do not have staff in place that understand what they are supposed to be doing, engineers do not understand the requirements, proper laying of pipes, to limit damage to the environment, proper signage, etc.” Instead of 34 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION being managed in-house, the function of training, monitoring, and overseeing compliance with the guidelines was outsourced to Abfort, a design and construction contractor. The contractor then worked with six sub-contracted health and safety officers to facilitate compliance. The absence of in￾house capacity and multiple subcontracts had implications for adaptative management, which is very dependent on timely responses and expertise (See Using Environmental and Construction Guidelines for Adaptive Management for more details). Compliance: Respondents noted that the Activity took measures to comply with the guidelines. These measures helped ensure minimal disruption of livelihoods, reduce environmental impacts, protect workers, and avoid paying compensation while ensuring rehabilitation. These measures included working outside work hours, finding ways to route trenches so existing housing and other properties were not affected, and using safety signs and safety gear. Despite the efforts to implement all guidelines and mitigation measures, the ET did not observe workers using safety gear during their visits to some sites. A contractor shared, “The aspect of workers protection in EMMP should also be integrated into state contracts.” In addition, a contractor noted, “There were challenges with communication due to conflicting communication by state and state consultants.” In addition to the above measures, some respondents indicated that the guidelines were used to provide important and proper notices to community members. For example, a contractor in Imo shared that in accordance with guidelines, “Prior information notice was given to shop owners. To ensure proper compaction, the spoil (Laterite) from the dug pit were carted away on a daily basis while sharp sand was imported as lining. Where trenches must be left open, temporary walkways were created with timber to allow for people to cross with ease. There are caution signs and caution tapes at every one meter where work is going on. Where there is a road closure, flagmen were on the ground to help in controlling and diverting traffic to prevent traffic holds up and accidents.” Further, the contractor noted, “People’s farmland was impacted, to mitigate the effect, the school had informed the farmers before the project commencement and the farmers were to remove the cassava stumps, which were replanted immediately after the pipes were laid and backfilled. There is no impact on the marginalized people.” Reports also indicated that the activities were implemented in a manner that minimized disruptions to beneficiaries’ livelihoods and their environment. Additionally, the provision of appropriate amenities such as toilets for workers created goodwill and enhanced ownership. For example, utility staff in Delta shared, “When you are doing a government project, don’t put the beneficiary at a disadvantage, don’t leave him or her poorer than you met him or her. Make sure there is a contingency plan on the ground to mitigate the likely impact of your construction activities.” Using Environmental and Construction Guidelines for Adaptive Management: These environmental processes and plans helped identify potential risks and anticipate potential mitigation measures to avoid, minimize, and reduce negative environmental and social impacts at the project level. This process of assessing risks and identifying mitigation measures is essential for adaptive management. According to USAID best practices, adaptive management is “an intentional approach to making decisions and adjustments in response to new information and changes in context.”22 It is an integral aspect of ensuring compliance with USAID’s environmental and construction guidelines. The role that the guidelines play in adaptive management was identified by a contractor in Niger who shared, “In terms of adaptive management, the guidelines can be used to be proactive before the problem arises rather than being reactive.” Similarly, a contractor in Niger shared that, “States now know the benefits of the guidelines, due to USAID’s insistence on the use of it and the sensitization on it.” 22 Program Cycle, Discussion Note: Adaptive Management, Discussion Note: Adaptive Management (usaidlearninglab.org). 35 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION To enhance adaptive management, the monitoring and validation of compliance is critical through regular monitoring visits. USAID was unable to conduct field visits due to COVID-19 during the Best Practice Report, thereby reducing oversight. Respondents also noted that the IP’s lack of in-house environmental expert or health and safety officer limited activity-wide adaptive management. Adaptive management is very dependent on timely response to emerging issues. Without an in￾house expert, the IP had to depend on the information that flowed from the sub-contractors to the design contractors, which bypassed the state utilities and state E-WASH staff. This is because the monthly reports which include reporting on environmental incidents for the Environmental Mitigation and Monitoring Report are sent directly to RTI, Abuja by the design and supervising contractor. The ET found that the Activity used adaptive management to address challenges in several cases. One site experienced water logging before the start of construction. The Activity adapted by creating diversion channels. Other examples of the beneficial use of adaptive management principles during construction related to women and youth are described below: According to respondents in Taraba, youth were initially resistant to the laying of a pipe network. The youth complained that they had not been involved in the project. The contractors adapted to the challenge by consulting with community leaders and employing the youth in the project. This helped obtain their buy-in and goodwill. In another case involving women, the construction activities facilitated buy-in by ensuring there was a quota for female employment in short-term activities. The Activity also targeted increased income generation by workers purchasing wares from community members and engaged in petty trade of consumables, who are mostly female. Despite the positive cases, the ET found that the ESMP was not used to its fullest potential in preventing incidents. For example, the project in Delta was delayed by one to three months. Some customers in Niger reported a lack of water for almost a year. In Taraba, electricity was disrupted due to falling poles during excavation. Other examples noted by community stakeholders include the obstruction of access roads to markets, the collapse of a wall, illness of little children, discomfort due to absence of water, and the siting of toilets close to the building, despite the ESMP itemizing mitigation measures for this. A community member in Abia state noted that “Adequate notice was not given.” Relatedly, a contractor in Delta state noted that 70 shops were affected by the trenches, but they were compensated. Conclusions: The Activity’s implementation of the environmental and construction guidelines showed mixed results, with noted areas for improvement. The IP’s lack of a dedicated in-house health and safety officer or environmental expert impacted overall adaptive management and oversight of the environmental and construction guidelines. Overall, communities were well-informed and the use of the environmental and construction guidelines helped to identify and address the potential impact of the construction on the environment and livelihoods of recipient communities before they occurred, reducing negative impact by addressing them proactively. To enhance the implementation of the guidelines and adaptive management, the monitoring and validation of compliance is critical through regular monitoring visits. Recommendations: 36 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 1. Future USAID WASH activities should advocate for and provide technical assistance to states for the inclusion of environmental and construction guidelines in state procurement processes, as well as stakeholder training on the role of adaptive management and facilitating ownership. 2. USAID should ensure that Ips hire qualified health and environmental safety officers as a condition for USAID funding. IMPLEMENTATION OF THE MTE RECOMMENDATIONS AND DESCOPING At the request of the USAID Nigeria Mission, the ET also explored questions related to the use of the MTE report and subsequent descoping. Specifically, the ET aimed to understand if the recommendations were taken into account, how the descoping was carried out, and its potential impact. The MTE was completed in October 2020. In response to the MTE and to several contextual shift, the Activity assessed its work in early 2021. This assessment responded to (1) the impact of COVID-19 and (2) Activity adaptations to the changing implementation context. This internal assessment built on the findings and recommendations from the MTE and discussions with USAID/Nigeria. The assessment resulted in descoping several activities, a change in the M&E approach, and a renewed focus on water supply. Descoping: In line with the MTE recommendations, descoping significantly reduced most technical assistance activities across components one to four, except for water supply production and distribution monitoring; the management of water quality assets; NRW interventions, such as the creation of DMAs in Niger; some customer reconnection and disconnections; and limited support to Abia. The changes also resulted in the termination of all contracts via the Small Grants Fund and of any new procurements planned under the USF. This shift increased the program’s attention on (1) delivering committed infrastructure and equipment under the USF, including the start of nine construction activities; (2) providing targeted technical assistance for sustainability (e.g., transferring tools, processes, and improved practices as activities wind down); and (3) continuing MEL interventions, reporting, and overall program closeout. “The descoping helped to focus on the delivery of water supply which is the positive effect. The negative effect is that ongoing activities were disrupted amongst which was the CSO activities and the local structures put in place within the community that the E-WASH had been engaging with” (KII, Abia) Monitoring and Evaluation (M&E): E-WASH respondents in all states noted that the Activity strengthened M&E at the utility level. They noted that data collection tools were developed and disseminated, M&E TWGs were formed in all states, and capacity-building was sustained. See EQ1 for additional details. Improvement of Water Supply: The Activity also prioritized efforts to improve water supply in all the states, increase customer connections, and improve commercial viability. Table 7 provides an overview of the status of each of the remaining water supply construction activities. 37 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Table 7: Progress by State* State Project Lot Expected Duration (Days) Days Spent Days Spent (%) Progress (%) Status Previous Progress Total Abia Rehabilitation of Ariaria water scheme Umuahia 101 79 78.2 37.3 11.5 48.7 Delay Delta Rehabilitation of Iselu-ukwu water scheme, extension of 10 kms pipe network and power cables 100 75 75.0 21.2 7.9 29.1 Delay Imo 3.8 kms pipe network replacement 91 77 84.6 26.9 7.6 32.5 Delay Niger 2.2 kms pipe network replacement in Minna 91 65 71.4 70.6 3.5 74 Steady Pilot DMA establishment including 9kms piped network in Minna 91 65 71.4 63.4 10.6 74 Steady Taraba Rehabilitation of 16k m piped network in Jalingo city 92 63 68.5 30.3 11.0 41.4 Delay Overall 74.9 41.6 8.7 50.3 Overall Delay *By November 2021, the six construction projects were 82 percent completed. Effect of Descoping: The direct impacts of descoping varied based on the Activity’s implementation status and other factors. Individual state-level impacts are highlighted below. Across the states, the refocus on water production-related activities was in line with stakeholder expectations. Though some respondents highlighted the discontinuation of work with CSOs and disengagement of some E-WASH staff in non-water production roles such as the Utility Operations Specialist and Institutional and Gender Expert, others noted the increase in M&E staff strength with insufficient notice. A stakeholder noted “Midterm recommendation of strengthening M&E was acted on. Staff strength from 1 to an average of 3 per utility. Abia: Respondents asserted that the descoping resulted in the incompletion of several activities in Abia, such as the gender policy and customer charter. Delta: As a result of the descoping, NRW activities were not implemented in Delta. The focus was placed on construction activities related to water production, water quality improvement, and strengthened M&E. Some of the external accountability structures the CSOs previously 38 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION supported in establishing ceased to exist, such as the reform champion team. CSOs have positioned themselves to leverage the incoming SURWASH for continuity of the structures. Niger: In Niger, there were challenges with the provision of equipment because of the descoping, which impacted the water supply. A respondent from the utility noted, “Reduction in the scope of work, some of the hardware – provision of some equipment in new bussa resulting in the inability to improve the new bussa station. Affected pumping dose equipment supply to the four of the five urban centers etc.” Taraba: The descoping raised concerns about staff capacity and the ability to address water supply challenges, including network expansion and its supporting equipment and software. A respondent from Taraba mentioned, “The descoping has affected us, for example on the canal separation in some of the low line areas by enumeration of water reservoir and strategic locations has been identified so that water supply can be boost in the lowlines areas, we have not been able to do that, we have to seek other intervention for that. We are also enjoying the capacity of staff and then the sudden descoping stopped them.” Further, it was elaborated that “We proposed a 3,500 cubic meter water tanks, five million cubic meter is another project funded by Taraba state gov. it cannot reach the lowline due to hydraulic study, but the 3,500 cubic meter if installed will do that purpose. Another negative effect is that we have a proposal of 55 km of network extension which is scoped down to 16km ongoing, also lead selection equipment and training on GIS expert on GIS mapping and leak detection and management, we did not get single equipment, we could not get any software for GIS to practice, we have to outsource.” Imo: In Imo, the impacts were immediately felt and included the cessation of oversight processes and capacity-building and the removal of support staff. This negatively impacted government perceptions. An E-WASH respondent noted, “The descoping had a serious negative impact. After the descoping, the usual E-WASH steering committee meeting stopped immediately. All workshops stopped. The Advocacy and Communication Specialist and Institutional and Policy Development Specialist were disengaged which affect the advocacy aspect of E-WASH as the STL [State Team Lead] has to be the only one doing it. The perception of the Government was that E-WASH has pulled out of the state, it however took some time before they were convinced that E-WASH is still around.” Findings further revealed that, even though the IP carried out an internal “North Star” workshop to decide which activities to prioritize in line with the MTE recommendations and key expectations of stakeholders, most of the state E-WASH teams did not have full knowledge of the recommendations from the MTE. Stakeholders within the states, meanwhile, reported that they had no knowledge of the findings and recommendations nor did they know why the activities were descoped. Construction activities focused on improving water supply in Abia and Delta, the DMA project in Minna, and the network distribution interventions were still underway at the time of the ET’s visit (Table 7). At the time of the ET’s visit, construction in Abia and Delta had reached 60 and 36 percent, respectively, while Niger, Imo, and Taraba were nearly complete. Thus, the effects on overall project deliverables could not be completely ascertained. Some MTE recommendations were not implemented and some were not carried out as expected. The MTE recommended that E-WASH focus on reforms in Abia and Imo, capacity-building through WASH media forums, and prioritize implementation of organizational development. However, these were not executed. 39 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Conclusion: To a large extent, the IP implemented the recommendations from the MTE, which has assisted the Activity to focus on delivery of water supply. However, the implementation strategy adopted by the IP in descoping led to both positive and negative effects, as reported by most of the utilities. Similarly, the MTE reports were not adequately disseminated among stakeholders. Recommendations: 1. USAID should ensure that learnings and adaptions, inclusive of descoping, based on evaluation findings are disseminated both internally and externally in a timely manner. 2. USAID should ensure that any future activity descoping processes include the assessment of the impact of non-completion on activity objectives and assess if other donors or partners could support de-scoped activities to avoid loss of investment. 40 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION ANNEX I: FULL LISTING OF REFERENCES AND REPORTS UTILIZED 1. Request for Task Order Proposals 2. RTI Technical Proposal for E-WASH (1) 3. Original Contract 4. Contract Amendments 5. All Approved E-WASH MEL PLANs from Y1 to Date 6. E-WASH Baseline Assessment reports for the states 7. E-WASH Minimum Results and Deliverables per Component 8. E-WASH Yr1, Yr2, Yr3, and Yr4 Annual Work Plans 9. USAID E-WASH Political Economic Analysis for the states 10. E-WASH Quarterly Report – October 1, 2018, to December 31, 2018 11. Quarterly Report – January 1, 2019, to March 31, 2019 12. Quarterly Performance Report – 3rd Quarter FY 2019 (April 1 – June 30, 2019) 13. FY 2019 Annual Progress Report 04_E-WASH 14. Quarterly Progress Report: Quarter 1 FY 2020 (Oct-Dec 2019) 15. Quarterly Progress Report: Quarter 2 FY 2020 – (January – March 2020) 16. Quarterly Progress Report: Quarter 3 FY 2020 – (April – June 2020) 17. FY 2020 Annual Report – October 2020 18. Quarterly Progress Report: Quarter 1 FY 2021 – (October – December 2020) 19. Quarterly Progress Report: Quarter 2 FY 2021 – (January – March 2021) 20. Quarterly Progress Report: Quarter 3 FY 2021 – (April – July 2021) 21. FY 2021 Annual Report – October 2021 22. Service Improvement Plan for the states 23. Data Quality Assessment Reports 24. List of Partners of the Activity 25. List of Financial Institution Partners 26. ERP Utilization Assessment Report Abia state 27. Abia State Water and Sewerage Corporation Monitoring and Evaluation Plan 28. USAID E-WASH Indicator SOPs 29. WASH Policy for Taraba 30. WASH Policy for Imo 31. Save the Children Quarterly Report, FY16 Q1 32. Save the Children Annual Report, FY 16 41 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION ANNEX II: FULL LISTING OF PERSONS INTERVIEWED The full listing of persons interviewed was submitted separately in line with data de-identification policies. Please contact Olufemi Gisanrin, ogisanrin@melsa.ng to request the data. 42 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION ANNEX III: DATA COLLECTION TOOLS Interview Guide 2 - E-WASH STAFF KII TOOL DATA COLLECTION PROTOCOLS – E-WASH Nigeria Hello. My name is __________, and I am here on behalf of Social Impact, an independent U.S. based monitoring and evaluation firm. We are conducting an independent Final Performance Evaluation of the Effective Water, Sanitation and Hygiene (WASH) program. This Activity is jointly implemented by the Research Triangle Institute (RTI) in partnership with Plan International (Plan), and Segura Consulting LLC and CDM Smith (CDM) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. The information will be used by USAID to improve future WASH programming. I’d like to invite you to participate in an interview with a series of questions related to E-WASH program being implemented in your state. You were identified because of your role in supporting the implementation of the E-WASH activity. The purpose of this questionnaire is to assess the progress of the Activity and how it is being implemented. As a stakeholder, we want to hear your views and opinions concerning this important Activity. If you chose to participate you would be one of ~89 interviews with ~123 interviewees. With your permission, we would like to record this interview to help us ensure that we have a full account of what you share for our notes. Only the evaluation team will have access to the recording. We are able to continue the interview even if you do not want to be recorded and will take notes to capture what you say. We will ask if you consent to this recording in a moment. We are not aware of any risks from participating in this study other than the loss of your time. There are no direct benefits from participating. [Only read this paragraph for in-person interviews and focus groups] Since we are meeting in person today, there is a potential health risk due to the on-going COVID-19 pandemic. To mitigate this risk, we are practicing social distancing and mask wearing and ask you to do the same. In addition, if you have felt ill with fever, cough, or other COVID-19 symptoms or if you have been in direct contact with anyone who has been diagnosed with COVID-19 in the last two weeks, we would ask that we transition this interview to a virtual interview for everyone’s health. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria and used to draft reports and provide information to USAID and other stakeholders. Only the evaluation team will have access to data that has identifying information. For any questions about the study, contact: Fiona Rowand – Acting Chief of Party, The Monitoring, Evaluation and Learning Support Activity, No. 22 Kumasi Street, Wuse II, Abuja, FCT, Nigeria – frowand@melsa.ng. Do you have any questions? Do you agree to participate in this evaluation? Yes/No (interviewer must document response) Are you willing to be recorded? Yes/No (interviewer must document response) Names of Respondents Position 43 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Name of Organization Office Address Email address Phone numbers Gender State of residence Date EVALUATION QUESTIONS S/N Key Evaluation Questions Follow up Questions Introduction What is your role with the E￾WASH Activity? 1. How long have you been on the Activity? 2. What are your functions? EQ1 To what extent did SWBs become commercially oriented and improve their financial viability? Questions for: State Team Lead, Chief of Party (COP), M&E 3. Will you say that E-WASH supported SWBs are commercially oriented? Why or why not? 4. What was the utilities billing and collection efficiency before the commencement of the E-WASH and what is it currently? (Ask for Indicators Performance Tracking Table) 5. What activities has been put in place by E-WASH to improve commercial and financial viability? Have these been achieved? 6. What were the major challenges to achieving better results on the E-WASH Activity? 7. What can be done better/differently? 8. How can we replicate the lessons learnt from the activity in other states/utilities? a. To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? Question for: USF Specialist / Procurement / Utility Operations Specialist (UOS) 9. Are all the proposed USF projects completed and fully operational? Yes or No. a. If yes, why? b. If no, why not? 10. Have these projects in anyway contributed to improving the utilities financial and operational viability? 11. If yes how? 12. If no, what are the inhibiting factors? b What impacts if any has COVD-19 had on SWB’s financial viability? (The State Team Lead) 13. What type of impact has it had? E.g., Is it positively or negatively? 14. What did you/the activity do to minimize the negative impacts if any? 15. What are the lessons learnt and how will this help to manage sudden changes in future? 44 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION EQ2 To what extent did the Activity support operational viability and professional management of SWBs? (M&E/ UOS/Team Lead, COP) 16. What specific things have changed in the SWB to show that the work ethics and culture within the organization has changed? How do these changes indicate the SWB is on track to achieve operational viability? a. What are the activities that led to this? 17. Are there specific SWB data and E-WASH results reflecting the attainment of these goals? a. For example, what is the capacity utilization of the production facilities before E-WASH intervention and now? 18. What are the challenges encountered and what are the lessons for future programming? EQ3 To what extent did the Activity strengthen policy, institutional and regulatory frameworks for improved WASH services at the state and national levels? (State Team Lead, COP, M&E) 19. What policy issues did the E-WASH activity work on, if any, and what were the contributions of the policy issues in transforming the water sector in Nigeria and your State? 20. What are the regulatory and institutional frameworks put in place by the activity in your state? 21. What has changed as a result of E-WASH policy work? E.g., are policies being implemented a. What accountability mechanism related to implementation of policy, and frameworks did the activity put in place, how successful have they been, how could they be improved and how might they impact sustainability? (State Team Lead, COP, M&E) 22. What are the accountability structures/process put in place as a result of the Activity? 23. Are they functional? Yes / No a. If yes, what has led to the success and how can we improve? b. If no, what are the challenges and how can we guide against these? 24. Are these structures sustainable beyond the E-WASH Activity? 25. How appropriate/effective is the E-WASH program capacity building of policy actors and other stakeholders? EQ4 To what extent did the activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping? (UOS/ State Team Lead) 26. How was sanitation mainstreamed in E-WASH program? a. What were the key activities undertaken? 27. Please discuss where a city-wide sanitation mapping exercise was conducted? 28. What were the key success and challenges? a. How can we overcome these challenges and scale the successes? 45 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION EQ5 What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? (USF Specialist/Procurement/UOS 29. How was USAID’s environmental and construction guidelines implemented in E-WASH program? Was an ESIA required for any intervention? 30. How was the environment and construction guidelines utilized to minimize the possible environmental impacts? 31. What was the impact of the Activity on the people, environment and the ecosystem? a. And how was the impact remedied or minimized 32. What are the lessons learnt in the implementation of USAID’s environmental and construction guidelines? 33. How would you recommend using the environmental and construction guidelines for adaptive management in the future? MTE 34. Please discuss how E-WASH has or has not adapted the MTE Recommendations? a. If only some were adapted, reasoning for why some were adapted/implemented better than others? b. Differences by implementation state, etc? c. Why? GESI 35. How was gender and social inclusion accounted for in this activity? a. During design, implementation? b. Is there anything that could have been done differently in the implementation GESI? 36. Is there anything else you would like to add 46 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Interview Guide 6 - KII/GROUP INTERVIEW GUIDE FOR CONTRACTORS & CONSULTANTS DATA COLLECTION PROTOCOLS – E-WASH Nigeria Hello. My name is __________, and I am here on behalf of Social Impact, an independent U.S. based monitoring and evaluation firm. We are conducting an independent Final Performance Evaluation of the Effective Water, Sanitation and Hygiene (E-WASH) program. This Activity is jointly implemented by the Research Triangle Institute (RTI) in partnership with Plan International (Plan), and Segura Consulting LLC and CDM Smith (CDM) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. The information will be used by USAID to improve future WASH programming. I’d like to invite you to participate in an interview with a series of questions related to E-WASH program being implemented in your state. You were identified because of your role in supporting the implementation of the E-WASH activity. The purpose of this questionnaire is to assess the progress of the Activity and how it is being implemented. As a stakeholder, we want to hear your views and opinions concerning this important Activity. If you chose to participate you would be one of ~89 interviews with ~123 interviewees. With your permission, we would like to record this interview to help us ensure that we have a full account of what you share for our notes. Only the evaluation team will have access to the recording. We are able to continue the interview even if you do not want to be recorded and will take notes to capture what you say. We will ask if you consent to this recording in a moment. We are not aware of any risks from participating in this study other than the loss of your time. There are no direct benefits from participating. [Only read this paragraph for in-person interviews and focus groups] Since we are meeting in person today, there is a potential health risk due to the on-going COVID-19 pandemic. To mitigate this risk, we are practicing social distancing and mask wearing and ask you to do the same. In addition, if you have felt ill with fever, cough, or other COVID-19 symptoms or if you have been in direct contact with anyone who has been diagnosed with COVID-19 in the last two weeks, we would ask that we transition this interview to a virtual interview for everyone’s health You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria and used to draft reports and provide information to USAID and other stakeholders. Only the evaluation team will have access to data that has identifying information. For any questions about the study, contact: Fiona Rowand – Acting Chief of Party, The Monitoring, Evaluation and Learning Support Activity, No. 22 Kumasi Street, Wuse II, Abuja, FCT, Nigeria – frowand@melsa.ng. Do you have any questions? Do you agree to participate in this evaluation? Yes/No (interviewer must document response) Are you willing to be recorded? Yes/No (interviewer must document response) Names of Respondents Position Name of Organization Office Address 47 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Email address Phone numbers Gender State of residence Date EVALUATION QUESTIONS S/N Key Evaluation Questions Follow up Questions Introduction Please share how you have interacted with the E-WASH activity? 1. How long have you been involved with the E-WASH Activity and in what capacity? EQ1a. To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? 2. Describe the Activity’s effort in using the USF to support the SWBs 3. What are the benefits? And what are the challenges? 4. Are all the E-WASH activity’s components you’re involved with completed and fully operational? a. If not, why? 5. Do you know how these projects have impacted the utility? If yes, tell us about it? EQ5 & GESI What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? 6. Was an EIA conducted before the construction activities related to your E-WASH work? 7. What were the likely effects on the environment, the people, and the ecosystem of the construction activity from the EIA report? a. Was there any planning on how to limit the impacts on marginalized groups? 8. What was the actual impact of the project on the environment, people and the ecosystem following the construction? a. How was the impact remedied, mitigated or minimized? 9. Were there any specific impacts on marginalized groups? Why and how many people were affected by the construction activities across the E-WASH states or in the state you were involved in? a. How was the environment and construction guidelines utilized to minimize these effects? 10. What are the lessons learnt in the implementation of USAID’s environmental and construction guidelines? 11. How would you recommend using the USAID environmental and construction guidelines for adaptive management in the future? 48 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION MTE 12. Have there been any difference in E-WASH guidance/implementation since xxx (date of MTE)? a. If changes were made, are you aware of why those changes were made and what impact have they had? 13. Is there anything else you would like to add? Interview Guide 5: KII/GROUP INTERVIEW GUIDE FOR SANITATION (Taskforce, Workers, LGAs) DATA COLLECTION PROTOCOLS – E-WASH Nigeria Hello. My name is __________, and I am here on behalf of Social Impact an independent U.S. based monitoring and evaluation firm. We are conducting an independent Final Performance Evaluation of the Effective Water, Sanitation and Hygiene (WASH) program. This Activity is jointly implemented by the Research Triangle Institute (RTI) in partnership with Plan International (Plan), and Segura Consulting LLC and CDM Smith (CDM) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. The information will be used by USAID to improve future WASH programming. I’d like to invite you to participate in an interview with a series of questions related to E-WASH program being implemented in your state. The purpose of this questionnaire is to assess the progress of the activity and how it is being implemented. As a stakeholder, we want to hear your views and opinions concerning this important Activity. If you chose to participate you would be one of ~89 interviews with ~123 interviewees. With your permission, we would like to record this interview to help us ensure that we have a full account of what you share for our notes. Only the evaluation team will have access to the recording. We are able to continue the interview even if you do not want to be recorded and will take notes to capture what you say. We will ask if you consent to this recording in a moment. We are not aware of any risks from participating in this study other than the loss of your time. There are no direct benefits from participating. [Only read this paragraph for in-person interviews and focus groups] Since we are meeting in person today, there is a potential health risk due to the on-going COVID-19 pandemic. To mitigate this risk, we are practicing social distancing and mask wearing and ask you to do the same. In addition, if you have felt ill with fever, cough, or other COVID-19 symptoms or if you have been in direct contact with anyone who has been diagnosed with COVID-19 in the last two weeks, we would ask that we transition this interview to a virtual interview for everyone’s health You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria and used to draft reports and provide information to USAID and other stakeholders. Only the evaluation team will have access to data that has identifying information. For any questions about the study, contact: Fiona Rowand – Acting Chief of Party, The Monitoring, Evaluation and Learning Support Activity, No. 22 Kumasi Street, Wuse II, Abuja, FCT, Nigeria – frowand@melsa.ng. Do you have any questions? 49 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Do you agree to participate in this evaluation? Yes/No (interviewer must document response) Are you willing to be recorded? Yes/No (interviewer must document response) Names of Respondents Position Name of Organization Office Address Email address Phone numbers Gender State of residence Date EVALUATION QUESTIONS S/N Key Evaluation Questions Follow up Questions Introduction Please share how you have interacted with the E-WASH activity? 1. How long have you been involved with the E-WASH activity and in what capacity? 2. How has the E WASH Activity engaged with your office? Probe for Activity implementation and strategy (Taskforce and LGAs) EQ4 To what extent did the activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping? 3. What was your involvement in the City Sanitation Mapping? 4. How was sanitation mainstreamed in the E-WASH program? (Modify sanitation workers) 5. Which areas of the city sanitation environment challenges has the Activity been most successful in improving /strengthening? Why? 6. How prepared is the utility as a result of the CWIS to manage sanitation activities in the state? (Modify for sanitation workers) 7. What has changed as a result of E-WASH city-wide sanitation mapping exercise? 8. What are the major challenges in the sanitation environment that the Activity is addressing? a. How can we overcome these challenges? 50 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION MTE 9. Have there been any difference in E-WASH guidance/implementation since xxx (date of MTE)? a. If changes were made, are you aware of why those changes were made and what impact have they had? GESI 10. How if at all did the city-wide sanitation mapping activity include gender and social inclusion considerations? 11. Is there anything else you would like to add? 51 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Interview Guide 1: USAID STAFF KII TOOL DATA COLLECTION PROTOCOLS – USAID Nigeria Hello. My name is __________, and I am here on behalf of Social Impact an independent U.S. based monitoring and evaluation firm. We are conducting an independent Final Performance Evaluation of the Effective Water, Sanitation and Hygiene (E-WASH) program. This Activity is jointly implemented by the Research Triangle Institute (RTI) in partnership with Plan International (Plan), and Segura Consulting LLC and CDM Smith (CDM) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. The information will be used by USAID to improve future WASH programming. I’d like to invite you to participate in an interview with a series of questions related to E-WASH program being implemented in your state. You were identified because of your role in supporting the implementation of the E-WASH activity. The purpose of this questionnaire is to assess the progress of the Activity and how it is being implemented. As a stakeholder, we want to hear your views and opinions concerning this important Activity. With your permission, we would like to record this interview to help us ensure that we have a full account of what you share for our notes. Only the evaluation team will have access to the recording. We are able to continue the interview even if you do not want to be recorded and will take notes to capture what you say. We will ask if you consent to this recording in a moment. We are not aware of any risks from participating in this study other than the loss of your time. There are no direct benefits from participating. [Only read this paragraph for in-person interviews and focus groups] Since we are meeting in person today, there is a potential health risk due to the on-going COVID-19 pandemic. To mitigate this risk, we are practicing social distancing and mask wearing and ask you to do the same. In addition, if you have felt ill with fever, cough, or other COVID-19 symptoms or if you have been in direct contact with anyone who has been diagnosed with COVID-19 in the last two weeks, we would ask that we transition this interview to a virtual interview for everyone’s health. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria and used to draft reports and provide information to USAID and other stakeholders. Only the evaluation team will have access to data that has identifying information For any questions about the study, contact: Fiona Rowand – Acting Chief of Party, The Monitoring, Evaluation and Learning Support Activity, No. 22 Kumasi Street, Wuse II, Abuja, FCT, Nigeria – frowand@melsa.ng. Do you have any questions? Do you agree to participate in this evaluation? Yes/No (interviewer must document response) Are you willing to be recorded? Yes/No (interviewer must document response) Names of Respondents Position Name of Organization 52 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Office Address Email address Phone numbers Gender State of residence Date EVALUATION QUESTIONS S/N Key Evaluation Questions Follow up Questions Introduction What is your role within USAID related to the E-WASH Activity? 1. How long have you been in this role supporting the E-WASH Activity? 2. What are your functions? 3. What are your thoughts on the overall implementation of the E-WASH Activity? 4. Describe the high-level results of EWASH? How were they crafted? Why were they revised? Will the revised results, if achieved, be considered success of EWASH intervention? 5. How has the descoping of the activities helped to streamline the achievements of the project activity? How has this affected the project generally? 6. What impacts if any have you seen on the activity related to COVID-19? EQ1 To what extent did SWBs become commercially oriented and improve their financial viability? 7. One of the main objectives of the E-WASH activity was for utilities to become more commercially oriented and improve commercial viability, in your opinion are the SWBs on the way to achieve this? a. To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? 8. Please discuss in your opinion how the USF fund has contributed towards the achievement of the project objectives? EQ2 To what extent did the Activity support operational viability and professional management of SWBs? 9. What specific things have changed in the SWBs to show that the work ethics and culture within the organizations has changed? How do these changes show that the SWBs are on track to become operational viableility? a. What are the activities that led to this? 10. What are the challenges encountered and how can we improve? EQ3 To what extent did the Activity strengthen policy, institutional and regulatory frameworks for improved WASH services at the state and national level? 11. What has changed as a result of E-WASH policy work? 12. Has USAID engaged in policy work as a result of E-WASH? If so, how? 53 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION a. What accountability mechanism related to implementation of policy, and frameworks did the activity put in place, how successful have they been, how could they be improved and how might they impact sustainability? 13. From USAID’s perspective, how successful has E-WASH been in supporting accountability mechanisms related to policy and frameworks? a. What are some examples of the accountability structures/process, put in place as a result of the activities, that best address the governance goals of the E-WASH Activity? 14. What are your thoughts on if these structures sustainable beyond the E-WASH Activity? EQ4 To what extent did the activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping? 15. What are some examples of sanitation processes, put in place as a result of the activities, that best address the sanitation goals of the E-WASH Activity? a. Has USAID observed any of these activities and what are your thoughts on possible improvement? EQ5 What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? 16. What is USAID’s visibility on how the environmental and construction guideline were implemented in E-WASH program? a. Where they used for adaptive management and how does USAID envision their use in the future for adaptive management? MTE 17. Please discuss what you know about how E-WASH has or has not adapted the MTE Recommendations? a. If only some were adapted, reasoning for why some were adapted/implemented better than others? b. Differences by implementation state, etc? c. Why? GESI 18. What are your thoughts on how gender and social inclusion was incorporated and implemented in the E-WASH activity? Conclusion: 19. What were the major challenges to achieving better results on the E-WASH Activity? 20. Overall, what are the key lessons learned from the E-WASH activity? 21. How can they be used/scaled within USAID or WASH sector? 22. Is there anything else you would like to add? 54 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Interview Guide 4: KII/GROUP INTERVIEW QUESTIONNAIRE FOR UTILITY STAFF DATA COLLECTION PROTOCOLS – E-WASH Nigeria Hello. My name is __________, and I am here on behalf of Social Impact, an independent U.S. based monitoring and evaluation firm. We are conducting an independent Final Performance Evaluation of the Effective Water, Sanitation and Hygiene (E-WASH) program. This Activity is jointly implemented by the Research Triangle Institute (RTI) in partnership with Plan International (Plan), and Segura Consulting LLC and CDM Smith (CDM) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. The information will be used by USAID to improve future WASH programming. I’d like to invite you to participate in an interview with to a series of questions related to E-WASH program being implemented in your state. . You were identified as because of your role in supporting the implementation of the E-WASH activity. The purpose of this questionnaire is to assess the progress of the Activity and how it is being implemented. As a stakeholder, we want to hear your views and opinions concerning this important Activity. With your permission, we would like to record this interview to help us ensure that we have a full account of what you share for our notes. Only the evaluation team will have access to the recording. We are able to continue the interview even if you do not want to be recorded and will take notes to capture what you say. We will ask if you consent to this recording in a moment. We are not aware of any risks from participating in this study other than the loss of your time. There are not direct benefits from participating. [Only read this paragraph for in-person interviews and focus groups] Since we are meeting in person today, there is a potential health risk due to the on-going COVID-19 pandemic. To mitigate this risk, we are practicing social distancing and mask wearing and ask you to do the same. In addition, if you have felt ill with fever, cough, or other COVID-19 symptoms or if you have been in direct contact with anyone who has been diagnosed with COVID-19 in the last two weeks, we would ask that we transition this interview to a virtual interview for everyone’s health You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria and used to draft reports and provide information to USAID and other stakeholders. Only the evaluation team will have access to data that has identifying information. For any questions about the study, contact: Fiona Rowand – Acting Chief of Party, The Monitoring, Evaluation and Learning Support Activity, No. 22 Kumasi Street, Wuse II, Abuja, FCT, Nigeria – frowand@melsa.ng. Do you have any questions? Do you agree to participate in this evaluation? Yes/No (interviewer must document response) Are you willing to be recorded? Yes/No (interviewer must document response) . Names of Respondents Position Name of Organization 55 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Office Address Email address Phone numbers Gender State of residence Date EVALUATION QUESTIONS S/N Key Evaluation Questions Follow up Questions Introduction What is your role with the e-wash project? 1. How long have you been on the staff of the SWB? 2. When did you beginning working on the E-WASH activity and what are your functions? 3. Are you aware of the objectives of the EWASH Activity? Please, describe. 4. In your own opinion, would you say that the Activity has been implemented in line with its objective? 5. In what ways would you say the Activity has supported your SWB towards contributing to commercial orientation and financial viability? 6. Have you had any improvement in your organizational performance since EWASH? Please explain. EQ1 To what extent did SWBs become commercially oriented and improve their financial viability? (The MD will respond to only sub question7,10, and 10) Commercial, finance, M&E) 7. What are your thoughts on the SWBs commercial orientation? a. Before E-WASH and now? Any changes/shifts in commercial orientation? Why? Why not? 8. What was the utilities billing and collection efficiency before the commencement of the E-WASH Activity? 9. What is the current billing collection efficiency? a. Why do you think this is the case? 10. Why do you think there are/ are not differences in billing and collection efficiency? 11. What activities have been put in place by E-WASH to improve commercial and financial viability? 12. What can be done better/differently? 13. How can we replicate the lessons learned from the Activity in other activities? a. To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? (Commercial/production and water quality, NRW) 14. What are the projects implemented under the USF and which of these are operational? 56 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 15. Have these projects in anyway contributed to improving the utilities financial and operational viability? a. If yes how? b. If no, what are the inhibiting factors? b What impacts if any has COVD-19 had on SWB’s financial viability? (MD/Commercial) 16. How did COVID-19 impact the SWB? Is it positively or negatively? Why do you say so? 17. What did you do to minimize the negative impacts if any? 18. What are the lessons learned? 19. What should we put in place in the future to ensure that the utility is not impacted negatively by sudden changes? EQ2 To what extent did the Activity support operational viability and professional management of SWBs? (MD/HR) 20. What specific things have changed in the SWB to show that the work ethics and culture within the organization has changed and put the SWB is on track to become a commercially oriented and financial viability? 21. Are there specific SWB data and E-WASH results reflecting the attainment of these goals? a. For example, what is the capacity utilization of the production facilities before E-WASH intervention and now? 22. What are the challenges encountered and how can we improve? EQ3 To what extent did the Activity strengthen policy, institutional and regulatory frameworks for improved WASH services at the state and national level? (MD) 23. Did the E-WASH program work on any policy issues? a. Which issues were these and how critical and necessary are the policy issues chosen in transforming the water sector in Nigeria and/or your State? b. Are the policies being implemented? If yes, where and in what ways? If not, why not? 24. What are the regulatory and institutional frameworks put in place by the activity to ensure the sustainability of WASH in your state? 25. What has changed as a result of E-WASH policy work? a. What accountability mechanism related to implementation of policy, and frameworks did the activity put in place, how successful have they been, how could they be improved and how might they impact sustainability? (MD) 26. What are the accountability structures/process put in place as a result of the activities? 27. Are they functional? Yes / No a. If yes, what has led to the success and how can we improve? b. If no, what are the failure factors and how can we guide against it? 28. Are these structures sustainable beyond the E-WASH project? 57 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 29. How appropriate is the E-WASH program capacity building of policy actors and other stakeholders? EQ4 To what extent did the activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping? (MD/HEAD Sanitation) 30. How was sanitation mainstreamed in E-WASH program? a. What were the key activities undertaken? 31. Was a city-wide sanitation mapping exercise conducted? a. If, so what has changed as a result of E-WASH city￾wide sanitation mapping exercise? 32. What were the key success and challenges? a. How can we overcome these challenges EQ5 What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? 33. How were USAID’s environmental and construction guidelines implemented in E-WASH program? a. How do you think the process could be used in the future for adaptive management? MTE 34. Have there been any difference in E-WASH guidance/implementation since xxx (date of MTE)? a. If changes were made, are you aware of why those changes were made and what impact have they had? GESI 35. How if at all did E-WASH impact gender and social inclusions as part of their work with your utility? e.g. GESI framework, etc. Conclusion: 36. What effect did the descoping of activities have on the achievements of the E-WASH activity? How has this affected the project generally? 37. Overall, what are the key lessons learned from the E-WASH activity? 38. How can they be used/scaled within USAID or WASH sector? 39. Is there anything else you would like to add 58 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Interview Guide 3: KII/GROUP INTERVIEW QUESTIONNAIRE FOR FEDERAL/STATE OFFICIALS – COMMISSIONERS, REGULATORY AGENCY & SECRETARY OF STATE GOVT DATA COLLECTION PROTOCOLS – E-WASH Nigeria Hello. My name is __________, and I am here on behalf of Social Impact, an independent U.S. based monitoring and evaluation firm. We are conducting an independent Final Performance Evaluation of the Effective Water, Sanitation and Hygiene(E-WASH) program. This Activity is jointly implemented by the Research Triangle Institute (RTI) in partnership with Plan International (Plan), and Segura Consulting LLC and CDM Smith (CDM) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. The information will be used by USAID to improve future WASH programming. I’d like to invite you to participate in an interview with a series of questions related to E-WASH program being implemented in your state. You were identified because of your role in supporting the implementation of the E-WASH activity. The purpose of this questionnaire is to assess the progress of the Activity and how it is being implemented. As a stakeholder, we want to hear your views and opinions concerning this important Activity. If you chose to participate you would be one of ~89 interviews with ~123 interviewees. With your permission, we would like to record this interview to help us ensure that we have a full account of what you share for our notes. Only the evaluation team will have access to the recording. We are able to continue the interview even if you do not want to be recorded and will take notes to capture what you say. We will ask if you consent to this recording in a moment. We are not aware of any risks from participating in this study other than the loss of your time. There are no direct benefits from participating. [Only read this paragraph for in-person interviews and focus groups] Since we are meeting in person today, there is a potential health risk due to the on-going COVID-19 pandemic. To mitigate this risk, we are practicing social distancing and mask wearing and ask you to do the same. In addition, if you have felt ill with fever, cough, or other COVID-19 symptoms or if you have been in direct contact with anyone who has been diagnosed with COVID-19 in the last two weeks, we would ask that we transition this interview to a virtual interview for everyone’s health You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria and used to draft reports and provide information to USAID and other stakeholders. Only the evaluation team will have access to data that has identifying information. For any questions about the study, contact: Fiona Rowand – Acting Chief of Party, The Monitoring, Evaluation and Learning Support Activity, No. 22 Kumasi Street, Wuse II, Abuja, FCT, Nigeria – frowand@melsa.ng. Do you have any questions? Do you agree to participate in this evaluation? Yes/No (interviewer must document response) Are you willing to be recorded? Yes/No (interviewer must document response) Names of Respondents Position 59 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Name of Organization Office Address Email address Phone numbers Gender State of residence Date EVALUATION QUESTIONS S/N Key Evaluation Questions Follow up Questions Introduction 1. Are you aware of the objectives of the EWASH Activity? Please, describe. 2. In your own opinion, would you say that the Activity has been implemented in line with its objective? 3. How does your role relate to the work that the E-WASH Activity is conducting? 4. What is your general opinion of the E-WASH activity? EQ1 To what extent did SWBs become commercially oriented and improve their financial viability? 5. What are your thoughts on the implementation of the E-WASH with SWBs in particular? 6. Do you think the Activity is on track to achieving its objectives of commercial orientation and financial viability of SWBs? If yes, why do you think so? If no, why not? EEQ2 To what extent is the activity on track to achieving operational viability and attaining a professionally managed SWB? 7. What specific things have changed in the SWB to show that the work ethics and culture within the organization has changed? a. Why do you think these changes occurred and what impact if any are they having? 8. Discuss what the activity has put in place to achieve the operational viability of the SWB? a. What are the challenges encountered in this area and how can we improve? EQ3 To what extent did the Activity strengthen policy, institutional and regulatory frameworks for improved WASH services at the state and national level? 9. What policy issues did the E-WASH activity work on, if any, in transforming the water sector in Nigeria and your State? a. Are the policies being implemented? If yes, where and in what ways? If not, why not? b. What has changed as a result of E-WASH policy work? 10. What are the regulatory and institutional frameworks put in place by the activity? 60 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION a. What accountability mechanism related to implementation of policy, and frameworks did the activity put in place, how successful have they been, how could they be improved and how might they impact sustainability? 11. What are the accountability structures/process put in place as a result of the Activity? a. Are they functional? Yes / No b. If yes, what has led to the success and how can we improve? c. If no, what are the challenges and how can we guide against these? 12. Are these structures sustainable beyond the E-WASH Activity? 13. How appropriate/effective is the E-WASH program capacity building of policy actors and other stakeholders? EQ4 To what extent did the activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping? 14. What is your awareness and understanding of how sanitation was mainstreamed in the E-WASH Activity? a. What were the key activities undertaken? 15. Please speak to how the institutions that E-WASH worked with for city-wide sanitation mapping were or were not prepared to manage sanitation challenges both before and after the activity? a. What were the key success and challenges? b. How can we overcome these challenges MTE 16. Have there been any difference in E-WASH guidance/implementation since xxx (date of MTE)? a. If changes were made, are you aware of why those changes were made and what impact have they had? GESI 17. What if any awareness do you have, of how the E-WASH activity accounted for/implemented gender and social inclusion practices? Conclusion 18. Overall, what were the major challenges to achieving better results on the E-WASH activity? 19. What could have been done better/differently? 20. How can the state/nation replicate the best practices/lessons learnt from the activity in other projects? 21. Is there anything else you would like to add? 61 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Interview Guide 7: COMMUNITY/UTILITY CUSTOMERS FGD GUIDE DATA COLLECTION PROTOCOLS – E-WASH Nigeria Hello. My name is __________, and I am here on behalf of Social Impact, an independent U.S. based monitoring and evaluation firm. We are conducting an independent Final Performance Evaluation of the Effective Water, Sanitation and Hygiene (E-WASH) program. This Activity is jointly implemented by the Research Triangle Institute (RTI) in partnership with Plan International (Plan), and Segura Consulting LLC and CDM Smith (CDM) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. The information will be used by USAID to improve future WASH programming. I’d like to invite you to participate in a focus group discussion (FGD) with a series of questions related to your experiences with utility provides and water access (that was a part of the E-WASH program being implemented in your state). You were identified because you were or are a customer of {NAME of] utility. The purpose of this discussion is to understand your experiences with [Name of] utility and accessing water services in your community. As a stakeholder, we want to hear your views and opinions concerning this important topic. With your permission, we would like to record this interview to help us ensure that we have a full account of what you share for our notes. Only the evaluation team will have access to the recording. We are able to continue the interview even if you do not want to be recorded and will take notes to capture what you say. We will ask if you consent to this recording in a moment. We are not aware of any risks from participating in this study other than the loss of your time. There are no direct benefits from participating. [Only read this paragraph for in-person interviews and focus groups] Since we are meeting in person today, there is a potential health risk due to the on-going COVID-19 pandemic. To mitigate this risk, we are practicing social distancing and mask wearing and ask you to do the same. In addition, if you have felt ill with fever, cough, or other COVID-19 symptoms or if you have been in direct contact with anyone who has been diagnosed with COVID-19 in the last two weeks, we would ask that you not participate in the discussion today. You are free to voluntarily choose to participate in this discussion, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria and used to draft reports and provide information to USAID and other stakeholders. Only the evaluation team will have access to data that has identifying information. Due to the private nature of this research, we ask that all focus group participants agree not to share anything that is discussed with anyone outside of this group once the conversation ends. Nonetheless there is a risk that other discussion participants will repeat what is shared here today. For any questions about the study, contact: Fiona Rowand – Acting Chief of Party, The Monitoring, Evaluation and Learning Support Activity, No. 22 Kumasi Street, Wuse II, Abuja, FCT, Nigeria – frowand@melsa.ng. Do you have any questions? Do you agree to participate in this evaluation? Yes/No (interviewer must document response) Are you willing to be recorded? Yes/No (interviewer must document response, note if one person does not consent to recording then only notes can be taken) 62 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Names of Respondents Position Name of Organization Office Address Email address Phone numbers Gender State of residence Date EVALUATION QUESTIONS S/N Key Evaluation Questions Follow up Questions 1 To what extent did SWBs become commercially oriented and improve their financial viability? 1. Please share your experiences accessing water in your community? 2. Please share your experience accessing water supplied by the SWB? 3. Do you pay for your water supply from the SWB? If no, why not? a. If yes, how long have you had your connection/supply from the utility? b. When did you start to pay for the service? a. To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? 4. Are you aware of any equipment installed by the SWB with the support of the E-WASH activity to improve access to water in your community? 5. If yes, has this equipment installed contributed to improved access to water in your community? b What impacts if any has COVD-19 had on SWB’s financial viability? 6. How has COVID affected your use of water? 7. How has COVID affected your interaction with the utility? 8. Are there any actions introduced by the SWB/E-WASH to mitigate the impact of COVID on customer relations? 3 To what extent did the Activity strengthen policy, institutional and regulatory frameworks for improved WASH services at the state and national level? 9. Are you aware of any policy issues the E-WASH program work on to improve customer experiences? 10. If yes, which issues were these and how critical and necessary are the policy issues chosen in improving customer-SWB relations? a. What accountability mechanism related to implementation of policy, and frameworks did the activity put in place, how successful have they been, how could they be improved and how might they impact sustainability? 11. To what extent is the E-WASH program engaged in capacity building to improve accountability and responsiveness in customer-utility relations? 12. Do you have ways of communicating your complaints on services to the utility? 13. Does the utility respond and how prompt do they respond to customer complaints? 14. Have there been any changes in the utilities communication and response to complaints [since E-WASH started implementation]? Please share examples. 63 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 5 What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? 15. Were you or your environment affected in anyway by the implementation of E-WASH construction activities? a. If yes, what are the measures taken to mitigate or compensate you for this impact? b. What could have been done differently? GESI 16. How, if at all, are/were marginalized people impacted in your community by SWB water access, and/or construction activities? 17. What could be done to better address the water and sanitation needs of marginalized people in your community? 18. Is there anything else you would like to add? 64 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Interview Guide 8: COMMUNITY MEMBERS IMPACTED BY E-WASH INTERVENTIONS FGD GUIDE DATA COLLECTION PROTOCOLS – E-WASH Nigeria Hello. My name is __________, and I am here on behalf of Social Impact, an independent U.S. based monitoring and evaluation firm. We are conducting an independent Final Performance Evaluation of the Effective Water, Sanitation and Hygiene (E-WASH) program. This Activity is jointly implemented by the Research Triangle Institute (RTI) in partnership with Plan International (Plan), and Segura Consulting LLC and CDM Smith (CDM) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. The information will be used by USAID to improve future WASH programming. I’d like to invite you to participate in a focus group discussion (FGD) with a series of questions related to your experience of construction and infrastructure activities that were conducted by [NAME of E-WASH partners] as part of the E-WASH program being implemented in your state. The purpose of this discussion is to assess your experience and understand any impacts based on the construction/infrastructure activities implemented. As a stakeholder, we want to hear your views and opinions concerning this important Activity. With your permission, we would like to record this interview to help us ensure that we have a full account of what you share for our notes. Only the evaluation team will have access to the recording. We are able to continue the interview even if you do not want to be recorded and will take notes to capture what you say. We will ask if you consent to this recording in a moment. We are not aware of any risks from participating in this study other than the loss of your time. There are no direct benefits from participating. [Only read this paragraph for in-person interviews and focus groups] Since we are meeting in person today, there is a potential health risk due to the on-going COVID-19 pandemic. To mitigate this risk, we are practicing social distancing and mask wearing and we ask you to do the same. In addition, if you have felt ill with fever, cough, or other COVID-19 symptoms or if you have been in direct contact with anyone who has been diagnosed with COVID-19 in the last two weeks, you not to participate in the discussion today. You are free to voluntarily choose to participate in this interview, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria and used to draft reports and provide information to USAID and other stakeholders. Only the evaluation team will have access to data that has identifying information. Due to the private nature of this research, we ask that all focus group participants agree not to share anything that is discussed with anyone outside of this group once the conversation ends. None, the less there is a risk that other discussion participants will repeat what is shared here today. For any questions about the study, contact: Fiona Rowand – Acting Chief of Party, The Monitoring, Evaluation and Learning Support Activity, No. 22 Kumasi Street, Wuse II, Abuja, FCT, Nigeria – frowand@melsa.ng. Do you have any questions? Do you agree to participate in this evaluation? Yes/No (interviewer must document response) Are you willing to be recorded? Yes/No (interviewer must document response note if one person does not consent to recording then only notes can be taken) 65 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Names of Respondents Position Name of Organization Office Address Email address Phone numbers Gender State of residence Date EVALUATION QUESTIONS S/N Key Evaluation Questions Follow up Questions Introduction 1. How long have you lived in the location? 2. Type of business/or residence/activity. 3. Was there a construction by the E-WASH activity recently in your area? 4. What type of construction was it? EQ5 & GESI What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? 5. Did you have a prior knowledge of the construction activity? 6. Were you or your environment affected in anyway by the implementation of E-WASH construction activities? a. If yes, how were you affected b. If you are aware, can you share how the broader community felt about the activity? c. How if at all were marginalized people impacted? 7. How did the contractor/consultant manage the situation? a. Where there any special considerations for marginalized people? 8. What are the measures taken to mitigate or compensate you for the impact of the construction? 9. Were you satisfied with the way the situation was handled? Yes/No? a. Why or why not / b. If no, what are your concerns? 10. What could have been done differently? 11. Is there anything else you would like to add? 66 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Interview Guide 9: UTILITY ACTIVITY SITE VISIT/DIRECT OBSERVATION GUIDE DATA COLLECTION PROTOCOLS – E-WASH Nigeria Hello. My name is __________, and I am here on behalf of Social Impact, an independent U.S. based monitoring and evaluation firm. We are conducting an independent Final Performance Evaluation of the Effective Water, Sanitation and Hygiene (E-WASH) program. This Activity is jointly implemented by the Research Triangle Institute (RTI) in partnership with Plan International (Plan), and Segura Consulting LLC and CDM Smith (CDM) with funding from the United States Agency for International Development (USAID)’s Nigeria Mission. The information will be used by USAID to improve future WASH programming. We would like to observe your facility as part of the E-WASH evaluation. We may also ask you questions related to our observations. The purpose of this observation is to assess the progress of the Activity and how it is being implemented. As a stakeholder, we want to hear your views and opinions concerning this important Activity. We are not aware of any risks from participating in this study other than the loss of your time. There are no direct benefits from participating. [Only read this paragraph for in-person interviews, observations and focus groups] Since we are meeting in person today, there is a potential health risk due to the on-going COVID-19 pandemic. To mitigate this risk, we are practicing social distancing and mask wearing and ask you to do the same. In addition, if you have felt ill with fever, cough, or other COVID-19 symptoms or if you have been in direct contact with anyone who has been diagnosed with COVID-19 in the last two weeks, we would ask that we transition this observation to a virtual observation for everyone’s health You are free to voluntarily choose to have your facility participate in this observation, you can refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your help in answering these questions is greatly appreciated. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be combined with those from other stakeholders in Nigeria and used to draft reports and provide information to USAID and other stakeholders. Only the evaluation team will have access to data that has identifying information. For any questions about the study, contact: Fiona Rowand – Acting Chief of Party, The Monitoring, Evaluation and Learning Support Activity, No. 22 Kumasi Street, Wuse II, Abuja, FCT, Nigeria – frowand@melsa.ng. Do you have any questions? Do you agree to participate in this evaluation? Yes/No (interviewer must document response) Names of Respondents Position Name of Organization Office Address Email address Phone numbers Gender State of residence 67 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Date 1. Description of the aspect of the Activity being observed 2. How is the E-WASH activity being implemented related to this site – lab, infrastructure, etc.? 3. Who is involved in the implementation? 4. Who are the beneficiaries? 5. Evaluation Team Key Observations in relations to relevant EQs 6. General Comments E-WASH MTE: FINDINGS AND RECOMMENDATIONS FOR REFERENCE Findings • While enthusiasm for the program was high at all levels across states, and capacity-building and stakeholder-engagement initiatives were effective to improve water supply, the program’s scope was ambitious to achieve within its 39-month timeframe. • The program has succeeded in advancing state sectoral reform (through the passing of state laws and policies and the setup of regulatory frameworks) and in progressing organizational development (OD) and business processes (such as customer enumeration, BoD development, Enterprise Resource Planning [ERP] system rollout, geographic information systems (GIS) application, etc.). However, the SWC-level reforms remained centralized at the headquarters level, and zonal offices did not fully absorb the institutional changes. • At the technical level, effective water quality management activities in all SWCs are lacking. SWCs in Niger and Taraba that have water production struggle because their piped networks are limited. In addition, all SWCs suffer from inadequate sanitation management; minimal revenue collection; high NRW levels; and deficiencies in effective monitoring and evaluation, data analysis, and data￾informed decision-making. 68 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION • Some activities were fragmented and out of sequence. Customer enumeration and billing system improvements should be linked and combined as well as financial modelling and tariff studies. OD activities started two years into the program. • USF deployment was slow, and COVID-19 pandemic restrictions contributed to delays. • The program’s engagement through CSOs and the media were effective, as were the program’s efforts to build CSOs’ capacity to support advocacy and reform interventions. Recommendations • “In order to enhance activity implementation, [USAID E-WASH] should review the first two years of implementation, fill the gaps, and come up with achievable targets for the remaining life of the activity.” • The program should continue to prioritize OD implementation and intensify corporatization, especially with SWCs in Abia and Imo. The program should also facilitate better information-sharing between SWC management and staff to improve staff performance and help achieve the overall corporatization objectives. • Increased leveraging with other donors for infrastructure development support should be elevated such as with the World Bank in Abia and Imo and with the African Development Bank in Taraba. • “USF should support the improvement of water quality in all states [and focus support on] Niger, Taraba, and Delta to maximize the gains made till date.” • To secure long-term financial viability, billing and collection improvements should continue, especially with SWCs in Niger and Taraba. • Sectoral reform efforts in Abia should remain and link with local CSOs. 69 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION ANNEX IV: SCOPE OF WORK EFFECTIVE WATER SANITATION AND HYGIENE SERVICE FINAL PERFORMANCE EVALUATION SCOPE OF WORK (SOW) 1. BACKGROUND INFORMATION 1.1 Identifying Information Development Objective Activity Title Broadened and Inclusive Economic Growth in targeted states Effective Water Sanitation and Hygiene Services Award Number Project Dates AID-OAA-I-15-00033/72062018F00003 May 24, 2018, to February 22, 2022 Type of Contract Project/Activity Funding Task Order USD 41,202,973 Contracting Officer’s Representative (COR) Implementing Partner (IP) Jean Jolicoeur Research Triangle Institute (RTI) B. Development Context 1. Problem or Opportunity Addressed by the Project/Activity being evaluated When the E-WASH Activity was designed, Nigeria had substantial needs for improved water, sanitation, and hygiene services. Per the 2020 World Health Organization/United Nations Children’s Fund Joint Monitoring Programme (JMP) for Water Sanitation and Hygiene, only 78 percent of Nigeria’s population has access to improved drinking water sources, while as few as 43 percent have access to improved sanitation facilities out of an estimated population of 210 million people. What this means is that over 46 million people in Nigeria are unable to access safe water and over 120 million people lack access to adequate sanitation. The GoN has made some investment in these essential areas; as a result, Nigeria made only modest progress against the water and sanitation Millennium Development Goal (MDG) between 2015 and 2020. The percentage of the population gaining access to an improved water supply rose from 69 percent in 2015 to 78 percent in 2020, whereas the percentage of the population having access to improved sanitation over the same period has increased slightly from 38 to 43 percent. WASH service delivery is not keeping pace with rapid urbanization. Services in the urban areas are not just inadequate, but in decline. According to a recent World Bank analysis, the urban water sector in Nigeria is losing the battle to keep pace with population growth and changing demographics. In 2015, 48 percent of the Nigerian population was urban. While the percentage of the urban population having access to improved water has increased, these numbers do not guarantee that all those with access are connected to the city pipe network. World Health 70 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Organization and the United Nation Children’s Fund Joint Monitoring Programme (JMP) data also show a significant decline in the proportion of households with access to piped water to premises in urban areas, which dropped from 13 percent in 2015 to 10 percent in 2020. Water is provided only by shared service, mainly water kiosks, standpipes, and multi-family connections. The current rate of investment in the sector is inadequate to maintain performance, with much fewer improvements in coverage or service quality. The World Bank estimates that USD 6 billion needs to be invested over the next ten years to reach universal coverage. The financing gap dwarfs current sector investments. Between 2006 and 2010, Nigeria spent around USD 1 billion annually in the water sector, which amounts to approximately 0.47 percent of its GDP, on average.3 Although Nigeria has experienced strong economic growth for a decade, its economy entered into a recession in 2016, and state budgets have been hit particularly hard by the downturn. Many economists predict that Nigeria will rebound to positive growth in 2017, but not at the same level as in years past. As a result, many state governments cannot currently make the necessary investments to strengthen the urban WASH delivery sector. While GDP has increased significantly over the years, the share of the budget channeled to the water sector has substantially decreased. This shortfall in funding exacerbates deterioration due to the absence of systematic operations, maintenance, and investment over many years. It is clear that increased sector financing is just part of the solution. In Nigeria, urban water services are provided by 37 publicly owned and operated water boards that have limited capacity to serve their urban populations. In Rivers State, for example, Port Harcourt Water Corporation (PHWC) can only meet 5 percent of the demand for water services in the state capital of Port Harcourt, despite being a hub for the cash-rich oil industry. Overall, service quality is poor. The Cross River State Water Board (SWB), which used to deliver 24/7 service 365 days a year, does no longer operate as previously. It achieved its goal by signing a management contract with a private Nigerian company (Ortech) 11 years ago to manage and maintain the water treatment plant and distribution network, however, the contract was not renewed. Other SWBs, which are all publicly managed, deliver water between four and 16 hours a day according to the International Benchmarking Network for Water and Sanitation (IBNET). SWBs have high operating costs (particularly due to the extreme unreliability of electricity supply, which means that many need to use diesel generators to meet their pumping needs) and very low, sometimes nonexistent, tariffs. Moreover, the systemic deficiencies evident within the system include excessively high leakage rates, aging water pipelines, and poor water quality. For example, on average, over five pipe breaks per kilometer occur annually as a result of poor maintenance and operations, with the exception of two providers in Abuja and Lagos. State water ministries have delegated the responsibility for providing water and sanitation services in urban areas to SWBs. In most cases, SWBs have limited financial and operational autonomy and are not commercially viable. The system weaknesses described previously are the consequence of a combination of an uncoordinated, fragmented operational framework, tariff structures that achieve neither cost recovery nor financial sustainability, insufficient funding, mismanagement, and a decades￾long lack of regulation and effective governance. Addressing governance issues is essential to improve performance, particularly given the correlation between inadequate water services and suboptimal governance. Greater financial viability of service provision and capacity development must be underpinned by governance improvements. Additionally, increasing water tariffs from their very low rates and enhancing payment collections have historically been viewed as politically unpalatable, and there has been a general lack of both public and political will for their administration. In the absence of effective utilities, the clear majority of the urban population relies 71 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION on boreholes (seldom safe to drink due to fecal contamination); standpipes; traditional sources, such as dug wells and rivers; and alternative providers. Nigerian citizens spend an estimated USD 700 million annually on obtaining water from alternative water providers. The money that they spend on alternative water providers could otherwise be used to pay monthly water bills to SWBs to finance operation and maintenance, the construction of additional water treatment plants and new pipe networks to provide water to new customers. As mentioned above, this investment is estimated at USD 600 million annually. Even more worrying is the total absence of sewerage services in Nigeria and the lack of any regulation or allocation of responsibility for sanitation services. Many of these WASH-related challenges were discussed in the 2011 performance assessment of water utilities carried out by the World Bank and the Government of Nigeria, which identified three main urban water supply challenges: growing demand for water due to accelerated urbanization, poor cost recovery, and institutional and governance constraints. On the basis of this evidence and analysis, the USAID/Nigeria E-WASH Activity seeks to improve the performance of SWBs in providing improved water supply services for the urban population in select Nigerian states. By doing so, E-WASH will help the SWBs demonstrate that better performing water boards will raise the quality of services for their customers; facilitate services-driven economic performance; improve state finances by reducing better targeting subsidies; and increase the chance of serving all customers in a particular area, including the poor. 2. Target Objectives and Stakeholders The Activity focused on four integrated objectives as follows: Objective 1: To professionalize management and improve the commercial orientation of SWBs. Objective 2: To improve the financial and operational viability of SWBs. Objective 3: To strengthen policy, institutional, and regulatory frameworks for improved WASH services (including establishing accountability mechanisms; and Objective 4: Build coordination, advocacy, and strategic communications to promote reforms. To address the enormous need for urban WASH services in Nigeria, the Activity focused on reforming SWB governance, which will lead to cost-recovery, increase investment in infrastructure, and expand access to water and sanitation to large unserved and underserved populations. By doing so, E-WASH intended to help the SWBs demonstrate that better performing water boards will raise the quality of services for their customers; facilitate services-driven economic performance; improve state finances by increasing cost recovery, reducing subsidies, and increase the state’s capacity to sustainably serve all customers in selected urban areas, including the poor. C. Intended Results of the Project/Activity being Evaluated At the end of this agreement, USAID expected the Activity to achieve the following high-level results: • Five SWBs professionally managed that: (i) achieve greater commercial viability; (ii) are overseen by a board of directors appointed by the State Governor based on existing laws; (iii) demonstrate greater managerial autonomy; (iv) can directly hire personnel and set the 72 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION salary structure; and (v) achieve greater cost recovery by reducing inefficiencies, nonrevenue water losses, and corruption. • At least USD 10 million of additional sector financing mobilized from public and private sources. • Five SWBs that have implemented revised policies, institutional frameworks, or regulations that promote access to improved WASH services; and • At least 54,809 households (426,325 that have gained access to basic people) drinking water or improved supplies piped onto their premises across selected states by the close of the four-year period. E-WASH will strengthen governance structures to address gaps in urban WASH service delivery in select urban and peri-urban settings across Nigeria. The Activity’s Results Framework (RF) hypothesizes that WASH service delivery will be improved if SWBs are professionally managed and commercially oriented; the financial and operational viability of SWBs is improved; policy, institutional, and regulatory frameworks for improved WASH services are strengthened; and WASH sector reforms at the state and national levels are promoted through targeted advocacy, coordination, and strategic communications. Further, a focus on states that perform well in the WASH sector will provide successful role models for lower-performing states. The identification of state tiers will allow for staged interventions in states that have already embraced reform, states that are prepared for reform, and states that have not yet planned for reform. Effective implementation of this Activity will require substantial coordination among numerous donors and stakeholders that are deeply invested in the long-term reform of Nigeria’s urban WASH sector. It will also require coordination with other United States Government (USG) investments in the governance, health, and education sectors. In designing E-WASH, USAID notes that interventions that have focused on physical infrastructure development alone has not been able to address service delivery needs in a sustainable manner unless they are combined with technical assistance and institutional reforms. In designing E-WASH, USAID notes that interventions that have focused on physical infrastructure development alone has not been able to address service delivery needs in a sustainable manner unless they are combined with technical assistance and institutional reforms. D. Approach and Implementation The contractor was requested to develop a program of interventions that includes the following four components: Create professionally managed, commercially oriented SWBs; Improve financial and operational viability of SWBs; Strengthen policy, institutional and regulatory frameworks for improved WASH services (includes establishing accountability mechanisms); and Build coordination, advocacy, and strategic communications to promote reforms. Create professionally managed, commercially oriented SWBs: The Activity proposed an agenda to create professionally managed, commercially oriented SWBs in targeted urban areas. Utility strengthening will focus on improvements in corporate governance and increased political and managerial autonomy of utilities. The SWBs will adopt a corporate structure that includes the management and a board of directors which represents a diverse cross-section of stakeholders and possible shareholders (including consumer interests) with clearly defined roles and responsibilities. Activities must include developing corporate plans, establishing and training plans for board members, creating human resource strategies including gender inclusion, and developing and implementing performance improvement plans. The contractor was required to provide technical assistance in the development and implementation of performance agreements with appropriate 73 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION incentive structures between the state governments and SWBs. RTI has proposed a suite of tasks to achieve the required high-level results and tailored to the needs of each utility. The following minimum results and deliverables were expected under this component: Increased management autonomy of all selected SWBs as measured by key indicators to be developed in five selected states by Year 3; Improved institutional performance of all selected water boards using a set of indicators, some of which have been defined by the Mission and others to be defined by the contractor, in close collaboration with relevant stakeholders (Year 4); 100 percent existing state water board management and key operational staff transitioned from civil service to corporate status of the new water company (Year 3); Staff development, retention and transition plan developed and operationalized in all five elected states by Year 3; Increased number of women in the staff, management and corporate board (Year 3); All management and key operational staff and 50 percent of all other staff receiving. Continuous training (Year 4); and Improved customer orientation of utility by enumerating customers, establishing a platform for customer engagement, and customer center; and a database created in five states by Year 4. Improve financial and operational viability of SWBs: The Activity was intended to provide tailored capacity development in financial, commercial, managerial, and operational aspects of the SWBs to improve their financial and operational viability. Financial viability can be achieved through reforms such as tariff adjustments that better reflect the cost of water service delivery, strengthening internal processes and controls (including financial accounting systems and procurement systems), improvements in commercial and operational practices (such as improvements in revenue collection), reducing water losses, and increasing cost recovery. Cost recovery remains one of the biggest challenges in the Nigerian water sector, given that the average cost recovery ratio of a state water board is around 63 percent, with the state authorities subsidizing a large portion of a utility’s operational costs such as labor, electricity, and chemicals. Such subsidies are then passed on to consumers via free water, nonpayment, or underpayment of water bills. The contractor was expected to engage with key stakeholders in the urban water supply sub-sector, especially the State Government and other existing or potential financiers (including donors and development banks), to prepare the water boards for both small and large-scale investments, paying special attention to financial sustainability. The contractor may also explore options for SWBs to transition to a private sector management contract to improve service delivery at the close of E￾WASH. Capacity development activities targeted to utility staff should be included in the package of activities. E-WASH tasks under Component 2 will broadly fall within the following sub-components: Explore options for private sector participation, Improve business processes, financial management practices, billing and collection and customer service, Promote and support the implementation of progressive tariff increases, Develop Performance Improvement Plans; Develop asset maintenance management systems; Design and operationalize a USF for emergency repairs and small-scale improvements to increase operational efficiency; Carry out activities to Reduce Non-Revenue Water (NRW). The following Minimum results and deliverables were expected under this component: Tested options for private sector participation in the five selected states by Year 3; A USF designed, established, and utilized to support quick service delivery improvements in the five states by Year 3; Developed and implemented tariff studies with pro-poor policies in the five states by Year 3; Developed and implemented PIPs, including scheduled, planned, preventive maintenance in the five states by Year 3; Operationalized asset maintenance management system, including a manual to formalize guidelines for developing such a system within the water boards in the five states by Year 74 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 3; Reduced non-revenue water (especially the billing and commercial side) by at least 50 percentage points in all targeted states (Year 4); Share of cost that is unrecovered decreases by at least 50 percentage points from the baseline in all selected states (Year 4); and Percentage of water utility customers not paying according to consumption decreases by at least 75 percent from the baseline in all selected states (Year 4). Strengthen policy, institutional and regulatory frameworks for improved WASH services (includes establishing accountability mechanisms): This Activity intended to create a favorable enabling environment for urban WASH sector reform in select states, focusing on policy, institutional and regulatory frameworks. Having the selected SWBs ready for investments (“finance￾ready”) requires not only utility level reform but also the transformation of the environment in which they exist. Sustained sector reform requires strengthened policy, institutional, and regulatory frameworks; and a shift in the mindset of both politicians and the people that they serve, the consumers. E-WASH will work to establish or deepen accountability mechanisms between the State government and the service providers through both supply and demand-side accountability measures. The Activity intended to Support the design and implementation of specific policy reforms necessary to improve WASH service delivery; Establish or strengthen regulatory unit or commission in five states; Design and implement a strategy to improve the financing of WASH service delivery expansion and improvements; Establish public accountability tools, including expanded benchmarking of sector performance in coordination with the World Bank and the Federal Ministry of Water Resources; Enhance social accountability and institutional transparency within the WASH sector in the select states; Map and develop plans for public sanitation service improvement in five cities; Design and operationalize a Small Grants Fund program to support and stimulate innovations and community-level interventions for improved WASH service delivery. The following minimum results and deliverables were expected: Implemented city-wide sanitation mapping and analysis with recommendations piloted in five cities by Year 3; Designed and utilized Civil Society Engagement Fund to support community-level WASH activities in the five states by Year 3; Designed, established and strengthened regulatory unit/commission in five states by establishing a sector working group consisting of the state sector ministries and the SWBs in all the five states by Year 3; Established and operational sector performance monitoring and benchmarking system in the five states by Year 3; Positive net change in organizational capacity of core partner CSOs, in terms of their engagement and their advocacy, using an organizational assessment tool developed by USAID or proposed by the contractor in the five states by Year 3; Increased percentage of SWB customers expressing trust in state government officials or institutions in selected states in relation to WASH service delivery (Year 3); and Increased amount of SWB expenditure on urban WASH infrastructure (Year 4). Build coordination, advocacy, and strategic communications to promote reforms. This Activity was intended to improve WASH service delivery through heightened and more effective coordination, advocacy, and strategic communication efforts that will spur transformational change in the sector. The contractor must build upon and promote good practices and lessons learned in the sector, not just under E-WASH, but also in other ongoing initiatives at the national and state levels. The contractor must support USAID in leading partner and organization coordination under an “architecture” which includes the Federal Ministry of Water Resources Steering Committee, donor group consultations, CSO round tables, and local level utility and consumer meetings necessary to build coalitions and advocacy for sector reform. Advocacy entails a set of coordinated strategic activities that aim to bring about lasting political change and is most effective when adapted to the 75 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION specific context in focus. In doing so, USAID intends to highlight the need to prioritize urban WASH sector reform among existing national priorities. The contractor will carry out strategic communications to help build commitment to reform both in the selected states and at the national level; strategic communication efforts are also required in pilot areas identified in Component 2. In particular, the Activity was intended to build the capacity of national and state-level WASH coordination agencies and facilitate donor group coordination; Develop and implement communications for reform strategy; Develop and execute a knowledge management strategy and action plan; Carry out or support research activities to help build political support for reform. The following minimum results and deliverables were expected under this component: Completed political economy, conflict, and gender analyses, in the three selected states. For which good-quality and/or WASH-relevant such analyses do not already exist in the five states by Year 3; Developed and implemented an advocacy and communication strategy, including regular coordination group meetings and quarterly outreach to media representatives (Year 2); Increased and continuous participation of CSOs and other relevant stakeholders in key utility reform processes such as tariff adjustments and service delivery monitoring in order to ensure sustainability in the five states by Year 3; and Developed and implemented a stakeholder knowledge management strategy and action plan for the platform, including completing a minimum of four knowledge products and/or services per year for each of the four years of project implementation (e.g. training modules, detailed best practices, others TBD in consultation between Implementing Partner and the Task Order Contracting Officer’s Representative [TOCOR]) in the five states by Year 3. Following the recommendations from the August 2020 mid-term Activity evaluation, USAID/Nigeria and RTI have identified the way forward for the Activity. This way forward points to a streamlined set of interventions primarily aimed at delivering infrastructure and equipment under E-WASH’s USF and providing targeted technical assistance. Consequently, RTI has requested to modify E-WASH’s SOW to address the recommendations of the evaluation. RTI has revised the remaining budget and technical interventions until the end of the Activity. RTI anticipates a reduction of the Total Estimated Cost (TEC) of the Task Order award from USD 60,424,459 to USD 41,202,973. Therefore, RTI has also submitted a modified budget reflecting the new TEC. It should be noted that the modification of E-WASH’s SOW will result in fewer beneficiaries and a potential decrease in previously projected results across all indicators (see revised MEL Plan). Assumptions There were a few critical assumptions for the Effective Water Sanitation and Hygiene Activity to achieve its targets and objectives. • The state governments that own the water boards will be committed to reform efforts. • Governance improvements in the water boards will be strengthened by state governments’ motivation and willingness to entrench corporate structure in the water boards. • The state governments are committed to adjusting tariffs in terms of both cost recovery and affordability. • Broad-based stakeholder engagement, effective civil society engagement, and accountability mechanisms are necessary to monitor and advocate for reform. • USAID can continue to work effectively with other development partners, particularly the World Bank and African Development Bank, to achieve meaningful scale and impact; and • USAID’s efforts will catalyze governance improvements within the SWBs. 76 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Existing Data The Evaluation Team will have access to the vital documents relevant to conducting this evaluation. These documents will include the technical proposal, original contract, monitoring and evaluation plan, annual work plans, quarterly and annual reports, data quality reports, contract amendments, memorandums of understanding, and any other relevant materials documenting the management, implementation process, results for the E-WASH Activity, modified SOW, as permitted by the Office of Acquisition and Assistance. 2. EVALUATION RATIONALE 2.1 Evaluation Purpose The primary purpose of this final performance evaluation is to determine whether the assistance provided by USAID/Nigeria through the E-WASH Activity achieved the stated development objectives and to understand the lessons learned from this particular Activity in Nigeria, taking into account the findings, recommendations of the mid-term performance evaluation and the subsequent descoping of the E-WASH Activity (see attached revised E-WASH SOW). It is intended to provide an independent examination of the overall progress and accomplishments of the Activity. The evaluation should provide a detailed picture of the major accomplishments and weaknesses of the Activity since its inception and determine how its successes can be sustained. This final performance evaluation will provide USAID/Nigeria, its implementing partners, and WASH sector stakeholders with data on outcomes and achievements. The evaluation should also elucidate lessons learned and include specific recommendations to USAID/Nigeria on how the Activity interventions can be sustained and scaled by other WASH activities, partners, private sector firms, or by the Government of Nigeria (GoN) both at the federal and state levels. B. Audience and Intended Use The primary users of the evaluation findings will be the USAID Economic Growth and Environment (EGE) Office, other USG officials, non-USG donor organizations, GoN, and private sector firms, all of whom can use the findings to improve and build knowledge (Table 1 below). Table 1: Audience and Intended Uses # Intended Use Target Audience USAID/ Nigeria WASH/ USAID/ Other USG IPs GoN / Other Dev. Partners Research Communities 1 Inform policy ✔ ✔ ✔ 2 Inform project ✔ ✔ ✔ ✔ ✔ design 3 Improve project ✔ ✔ ✔ ✔ ✔ monitoring 4 Improve ✔ ✔ ✔ ✔ operational policy and planning 5 Improve resource ✔ ✔ ✔ ✔ management 77 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION # Intended Use Target Audience USAID/ Nigeria WASH/ USAID/ Other USG IPs GoN / Other Dev. Partners Research Communities 6 Enhance professional growth ✔ ✔ ✔ ✔ ✔ 7 Add scientific Knowledge ✔ ✔ ✔ The result of this evaluation will be used by the Mission, WASH implementing partners, and other stakeholders to enhance policy formulation, analysis, and implementation in the country. USAID will develop a dissemination plan in accordance with the Evaluation Policy as specified in Automated Directive System (ADS) Chapter 201. C. Evaluation Questions The E-WASH Activity had four main objectives: 1. Professionalize management and improve the commercial orientation of SWBs. 2. Improve the financial and operational viability of SWBs. 3. Strengthen policy, institutional, and regulatory frameworks for improved WASH services (including establishing accountability mechanisms; and 4. Build coordination, advocacy, and strategic communications to promote reforms. In determining the achievement of these objectives, the evaluation will focus on relevance, effectiveness, sustainability and consider gender and social inclusion impacts; and the following key evaluation questions should be answered: 1. To what extent did SWBs become commercially oriented and improve their financial viability? a. To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? b. What impacts if any has COVID-19 had on SWB’s financial viability? 2. To what extent did the Activity support operational viability and professional management of SWBs? 3. To what extent did the Activity strengthen policy, institutional and regulatory frameworks for improved WASH services at the state and national levels? a. What accountability mechanism related to the implementation of policy, and frameworks did the Activity put in place, how successful have they been, how could they be improved and how might they impact sustainability? 4. To what extent did the Activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping? 5. What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? D. Evaluation Method and Methodology Evaluation Design The Evaluation Team will use a mixed-methods approach to complete a thorough and effective final performance evaluation within the timeline and budget parameters set in the contract. The 78 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION evaluation team should consider a range of possible methods and approaches for collecting and analyzing the information required to answer the evaluation questions. The methodology should include, but not be limited to, the following techniques (listed below) to conduct the evaluation. Prior to arriving in the country and conducting fieldwork, the team should submit to USAID/Nigeria three key deliverables: 1) Document Review Summary, 2) Methodology/Detailed Work Plan, 3) Site Visit Schedule and Plan. USAID/Nigeria will review these three deliverables, turning them around within one workweek to the evaluation team. • Document Review/Data Analysis. Prior to arriving in the country and conducting fieldwork, team members will review various documents and reports, including but not limited to the WASH original agreement and amendments; USAID/Nigeria strategy document; DO 3 Project Appraisal Document; Activity quarterly and annual reports; Activity technical studies; and other relevant documents such as the specification of the Activity deliverables. USAID/Nigeria and RTI will provide the relevant documents. • Key Informant Interviews. The team will conduct interviews and focus groups with a variety of stakeholders including USAID staff, project beneficiaries, government staff, implementing partner staff, and other key donor partners. A list of stakeholders and contacts will be provided by USAID/Nigeria and RTI, and additional individuals may be identified by the evaluation team at any point during the evaluation prior to the drafting of the final report. • Site Visits. In addition to the many key informant interviews that will take place in Abuja, the evaluation team will visit Activity sites to interview beneficiaries, service providers, private sector entities engaging with beneficiaries, and federal, state, and local government authorities as indicated. The evaluation team will apply robust data analysis techniques including triangulation to draw conclusions and recommendations linked to key findings that are based on strong evidence. The evaluation team will compare data collected from key informant interviews with results from the Activity documents and reports. USAID staff from the EGE Office and other Mission offices may accompany the Evaluation Team as needed. 3. EVALUATION PRODUCTS A. Deliverables The Evaluation Team will be responsible for producing the following deliverables: 1. Detailed Work Plan: That details evaluation methodological approach and draft schedule/timeline of field activities, draft of interview/focus group discussion guide(s) to be used during interviews, and a list of proposed sites to visit. This will also include a summary of the document review. 2. In-briefing/Presentation: While developing the detailed evaluation design, the evaluation team is expected to hold an In-briefing presentation with key USAID/Nigeria staff to discuss having a shared understanding of the design approach and proposed methodology, etc. 3. Out-briefing/Presentation: After the fieldwork concludes, the evaluation team is expected to hold an out-briefing presentation using PowerPoint to discuss the preliminary evaluation findings, conclusions, and recommendations. 79 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 4. Draft Evaluation Report: The draft report should be consistent with the guidance provided in Section: B Reporting Guidelines. The report will address each of the questions identified in the SOW and any other issues the team considers bearing on the evaluation objectives. Any such issues can be included in the report only after consultation with USAID. 5. Final Evaluation Presentation: To share the findings more broadly within the mission. 6. Final Evaluation Reportwithin two weeks of receiving Mission comments on the draft report). In addition to being compliant with the criteria of ensuring the quality of the evaluation report, the final report will address the comments provided by USAID/Nigeria and other stakeholders on the draft report. 7. Two-page brief: Summarizing the key findings of the evaluation (timing TBD) 8. One-page infographic: Document sharing key findings of the evaluation (timing TBD) All reports are to be submitted in English electronically. Hard copies of materials for in-person activities/presentations/meetings will be provided. The Final Evaluation Report should not exceed 30 pages in length in its body, not including title page; Table of Contents; List of Acronyms; usage of space for tables, graphs, charts, or pictures; and/ or any material deemed important and included as Annexes. The Final Evaluation Report and PowerPoint addressing the Mission’s comments should be submitted in both Word and PDF formats. Once the PDF format has been approved by the Mission, the Team will submit the Final evaluation report to the Development Experience Clearinghouse for archiving. Reports should be submitted consistent with the Automated Directives System (ADS) 579. B. Reporting Guidelines USAID’s evaluation policy requires that all evaluation SOWs include USAID’s Criteria to Ensure the Quality of the Evaluation Report (USAID Evaluation Policy, 2011). The policy also indicates that the report will outline in detail, any additional expectations USAID has regarding a report’s structure, format, and length. The format for the evaluation report is as follows (number of pages is illustrative): • Executive Summary (5 pp.) • Table of Contents (1 pp.) • Introduction (1 pp.) • Background (1-2 pp.) • Methodology (1-2 pp.) • Findings/Conclusions (15- 20 pp.) • Issues and Challenges (1-2 pp.) • Recommendations/Future Directions (3-5 pp.) • References • Annexes The draft and final reports will be submitted electronically. All reports will be in English. The report must: • Distinguish clearly between findings, conclusions (based strictly on findings), and recommendations (based clearly on the evaluation findings and conclusions). • Comply with USAID’s Evaluation Policy • Ensure submission to the Development Experience Clearinghouse after finalization. 80 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION C. Dissemination Plan It is expected that USAID/Nigeria will plan a dissemination session with other technical units within the mission, implementing partners, donor community as well other stakeholders. Dissemination activities will include a preliminary findings presentation with the USAID technical team, a final presentation with the wider mission, a 2-page brief, and an infographic document. These varied deliverables will help ensure that the learning from the evaluation can be shared as broadly as possible and will help encourage uptake and adaptation in response. For any in-person dissemination activities (such as the final presentation), hard copies of materials will be provided to participants. 4. TEAM COMPOSITION & SUGGESTED LEVEL OF EFFORT The Evaluation Team will consist of four key members, including at least one (1) international consultant (who will lead the team) and local consultants. The team members should represent a balance of several types of knowledge and expertise related to WASH policy reform programming and urban WASH-led development. USAID/Nigeria recommends the following staffing structure for the evaluation: Team Leader/WASH Specialist: The Team Leader will be responsible for the overall management of the evaluation. S/he will be the primary person responsible for ensuring a rigorous and high-quality evaluation design The Team Leader will develop the outline for the draft report, present the report, and after incorporating USAID/Nigeria staff comments, submit the final report to USAID/Nigeria through the MEL Support Activity within the prescribed timeline. Qualifications for the Team Leader: • Minimum of ten years of professional experience in the WASH sector, with specific expertise in urban water supply and utilities, preferably in Nigeria. • Demonstrated experience in, water and sanitation promotion in urban contexts in Africa. • Demonstrated experience with WASH capacity building and policy reform, with preference to those with considerable working experience in Nigeria with institutions and civil society organizations in the urban water and WASH sectors. • Experience in preparing persuasive technical reports and making verbal presentations, explaining issues clearly and concisely. • Required oral and written communication and presentations skills in English. • Demonstrated experience in collaborating across projects stakeholder groups and donor projects; and • A minimum of a master’s degree in a relevant field: engineering, public/global health, water resources, sanitation, or allied fields. Other team members will include: 1. A Senior WASH Governance Specialist: S/he will have at least a masters’ degree in Engineering, Urban Planning, Business Management, or any relevant social science; Over six years of experience working on water and sanitation sector reforms in Nigeria. Five of these years should be in utility management, some at a senior level; Clear understanding of the key performance indicators of a utility will be a necessity. Experience in working on urban water supply systems; community-based water supply and sanitation program implementation and operation essential; A good understanding of operational challenges of water service providers in Nigeria; Strong communication skills, both interpersonal and written, to fulfill the technical and managerial responsibilities proposed. English language written and verbal fluency. 81 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 2. A WASH Institutional Development Specialist: S/he will have a minimum of 5 years of professional experience in the implementation of large and complex water and sanitation development projects including WASH-related environmental assessments and expansion of service coverage. Advanced graduate degree in a relevant field: engineering, public/global health, water resources, sanitation, or allied fields. International work experience in developing countries required, work in Nigeria desired. Excellent ability to communicate in English, both verbally and in writing 3. An Evaluation Specialist: with at least five (5) years relevant experience and strong logistics and planning skills. S/he must demonstrate experience in organizational capacity building/assessment required. S/he must demonstrate strong experience with data collection procedures, surveys, interview guides, and analysis of data. USAID leaves to Navanti’s discretion other necessary team members/staff for the evaluation (e.g., logistics, scheduling, and translation, data analysis). Aside from the above-mentioned key personnel, the offeror must decide how the evaluation team should be structured in order to successfully address the evaluation questions. All attempts should be made for the team to be gender-balanced and to include local (Nigerian) experts. Intended Participation of Other Parties: USAID may propose internal staff from USAID/Nigeria or headquarters, Implementation Partners, National Counterparts, and/or beneficiaries to accompany the team in this evaluation as observers. As observers, their role will be to provide guidance and background information and to rely on the external evaluators’ questions. They will review and comment on the report for accuracy, but evaluators may accept or reject comments. The final report should reflect the opinions of the external evaluators and is the sole responsibility of the selected evaluation team. Conflicts of Interest: All evaluation team members will provide a signed statement attesting to a lack of conflicts of interest or describing an existing conflict of interest relative to the Activity being evaluated. USAID/Nigeria will provide the conflict-of-interest forms. 5. SCHEDULING AND LOGISTICS Responsibilities: The funding source will be through the budget of the USAID/Nigeria Monitoring, Evaluation, and Learning (MEL) Activity. The EGE Office of USAID/Nigeria contributes to the budget of the MEL Activity. The MEL Activity /Navanti will be responsible for all offshore and in-country administrative and logistical support, including identification and fielding appropriate consultants. Navanti will arrange and schedule meetings, international and local travel, hotel bookings, working/office spaces, computers, printing, and photocopying. Navanti will make all logistical arrangements, including the vehicles for travel throughout Nigeria, and should not expect any logistical support from the Mission. Navanti will also make their own arrangements about space for team meetings and equipment support for producing the report. USAID/Nigeria will provide: • Background documents to be provided to the evaluation team as early as possible prior to working on the evaluation, but at least two weeks before the start of the evaluation. • A list of key informants, institutions, organizations, and other stakeholders. • Ensure constant availability of Mission Point of Contact to provide technical leadership and direction for the evaluation team’s work. • Assistance with arrangements/letters of introduction for formal and official meetings, and where necessary for high‐level meetings, will accompany teams on introductory interviews. 82 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION • If necessary and deemed appropriate, assist in identifying and helping set up meetings with local development partners relevant to the assignment. 6. PERFORMANCE PERIOD & TRAVEL The period of performance will start on or about October 2021 depending on the availability of consultants and available funding. USAID/Nigeria will designate a technical representative to work in coordination with the Evaluation Team; however, the Team Leader will have the primary responsibility for ensuring the final deliverables are completed in a timely manner and are responsive to the SOW and Mission comments. 6.1 Timeframe Months Oct 2021 Nov 2021 Dec 2021 Jan 2022 Feb 2022 March 2022 Activities Contract Award Submission of Work Plan Inclusive of Preparation of Guidelines for Data Collection Review of Work Plan Finalize and Submit Final Work Plan In-brief Field Visits/ Data Collection Data Analysis Out-brief Preparation of Report Submission of Draft Report USAID Review Draft Report MELSA Review/ Accept Comments Mission Wide Presentation of Findings Submission of Final Report to USAID Final Report by February 15, 2022. Submission of Brief to USAID Submission of Infographic to USAID B. Travel Field sites suggestions for data collection and duration of travel: S/N Destinations State Government (SG), State Water Boards (SWB), Media (M), Private Sector (P), CSOs/Associations © Duration (days) 1 Abia SG, SWB, M, P, C 3 2 Delta SG, SWB, M, P, C 3 3 Imo SG, SWB, M, P, C 3 4 Niger SG, SWB, M, P, C 3 83 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION S/N Destinations State Government (SG), State Water Boards (SWB), Media (M), Private Sector (P), CSOs/Associations © Duration (days) 5 Taraba SG, SWB, M, P, C 3 6.2 Level of Effort (LOE) Provide information about the suggested LOE (in hours) of each team member. Title LOE/Days LOE/Hours Team Leader/WASH Specialist 66 528 A Senior WASH Governance Specialist 51 408 WASH institutional Development Specialist 39 312 Evaluation Specialist 38 304 84 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION ANNEX V: E-WASH ACTIVITY FINAL PERFORMANCE EVALUATION WORKPLAN 1. INTRODUCTION AND STUDY PURPOSE 1.1. EVALUATION OVERVIEW AND PURPOSE The primary purpose of this final performance evaluation is to determine whether the United States Agency for International Development (USAID)/Nigeria Effective Water, Sanitation, and Hygiene (E￾WASH) Activity achieved its stated development objectives, to understand the lessons learned, consider the findings and recommendations from the mid-term performance evaluation and the subsequent de-scoping of the E-WASH Activity. USAID awarded the Research Triangle Institute (RTI) the five-year E-WASH Activity in 2018. RTI led a consortium of implementing partners (IPs): Plan International (Plan), Segura Consulting LLC, and CDM Smith to implement the Activity and in an effort to achieve its goal of E-WASH is to assist the Government of Nigeria (GON) to expand and improve urban water service delivery by strengthening the governance, financial, and technical viability of select State Water Boards (SWBs) in Nigeria. The evaluation is intended to provide an independent examination of the Activity’s overall progress, accomplishments, weaknesses, and potential for sustainability. This final performance evaluation will provide USAID/Nigeria, its implementing partners (IPs); and water, sanitation, and hygiene (WASH) sector stakeholders with data on outcomes and achievements as well as elucidate lessons learned. The evaluation findings and recommendations will be helpful to the USAID Economic Growth and Environment Office, other United States Government (USG) officials, the Ips, broader WASH sector, GON, and private sector firms, all of whom can use the results to improve and build knowledge of WASH service expansion though commercialization approaches. 1.2. BACKGROUND: URBAN WASH CONTEXT IN NIGERIA Nigeria’s rapidly rising population of over 200 million is grappling with a lack of adequate water supply and sanitation access. According to the WASH-National Outcome Routine Mapping (NORM) 20191 reports, up to 171 million people in Nigeria are off the Sustainable Development Goal (SDG) target for access to safely managed drinking water supply services, and 157 million are without safely managed sanitation services. Only 28 out of 36 states plus the Federal Capital Territory (FCT) have urban water utilities actively producing water. Out of these 28 states, only 16 have functional water utilities.2 Access to piped water to homes is in decline, dropping from over 30 percent access in the 1990s to less than 10 percent by 2016. In response to this dire situation and the need to realign the sector to its goal of achieving SDG 6.1 and 6.2 by 2030, the GoN declared a state of emergency in the WASH sector in November 2018. It launched a National Action Plan for the revitalization of the water, sanitation, and hygiene sector. When the E-WASH Activity was designed, Nigeria had substantial needs for improved water, sanitation, and hygiene services. Per the 2019 WASH-NORM, an average of 70 percent of the population had access to basic drinking water supply, reflecting a 10 percent decrease from the 2010 figure. Nonetheless, WASH-NORM estimated that 92 percent of urban and 61 percent of rural 1 Federal Ministry of Water Resources 2019 National Outcome Routine Mapping of Water, Sanitation and Hygiene. https://www.unicef.org/nigeria/media/3576/file/WASH%20NORM%20Report%202019.pdf. 2 Federal Ministry of Water Resources 2019 National Outcome Routine Mapping of Water, Sanitation and Hygiene. https://www.unicef.org/nigeria/media/3576/file/WASH%20NORM%20Report%202019.pdf. 85 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION populations have access to basic drinking water supply. About 44 percent of the total population has access to basic sanitation services, with 53 percent of the urban population, compared to 40 percent of the rural population having access. The higher 2019 data for water supply coverage in urban areas mask the disparity in service levels and quality across urban residents and the sources of drinking water supply. A key challenge is access to the city water network. With a 4.2 percent annual urban population growth rate, WASH service delivery is not keeping pace with rapid urbanization. Services in the urban areas are not just inadequate, but in decline. According to a recent World Bank analysis, as well as World Health Organization and the United Nations Children’s Fund Joint Monitoring Programme data, the urban water sector in Nigeria is losing the battle to keep pace with population growth and changing demographics with access to piped water to premises in urban areas, dropping from 13 percent in 2015 to 10 percent in 2020. Nigerian citizens spend an estimated USD 700 million annually on obtaining water from alternative water providers. The challenge is how to harness this substantial household expenditure towards payment of monthly water bills to SWBs to finance operation and maintenance, the construction of additional water treatment plants, and new pipe networks to provide water to new customers. It is estimated that SWBs need approximately a USD 600 million investment annually, over the next ten years, to maintain performance and improve coverage or service quality.3 The statistics for urban sanitation are even more concerning. According to WASH-NORM 2019, 10 percent of urban residents defecate in the open. No Nigerian city, besides Abuja and a few housing and industrial estates and university campuses in Cross River (Tinapa), Ibadan, Lagos, and Zaria, has a central sewerage system or organized fecal sludge management. 5.3 percent of fecal matter in towns and cities goes through sewers, while for over 90 percent of households who use on-site toilets, their collection pits are emptied and transported by trucks to improvised fecal sludge dump sites.4,5 The total available market size and huge potential for fecal sludge emptying services are estimated at USD 15.7 million for Ibadan, USD 8.6 million for Abuja, and USD 2.3 million for Yenogoa.6 State and local government departments with weak regulation or enforcement powers or sanitation services and infrastructure share sanitation responsibilities. 2. ACTIVITY BACKGROUND 2.1 CONTEXT The GoN’s 2002 WASH Policy delegated responsibility for providing WASH services to state governments, who in turn have mandated SWBs to provide urban WASH services. In most cases, SWBs have limited financial and operational autonomy and are not commercially viable. The systemic weaknesses observed with SWBs are the consequence of a combination of an uncoordinated and fragmented operational framework, tariff structures that achieve neither cost recovery nor financial sustainability, insufficient funding, mismanagement, and a decades-long lack of regulation and an acute governance challenge. The E-WASH Activity was designed to reform six (6) SWBs in Nigeria. The Activity’s objective was to improve the financial and operational viability of the utilities and create an enabling environment for the utilities to grow. Following an assessment of 14 pre-selected states, the 3 Berta Macheve, Alexander Danilenko, Roohi Abdullah, Abel Bove, and L. Joe Moffitt, State water agencies in Nigeria; a performance assessment. World Bank Group Washington 2015. 4 2013 Demographic and Health Survey (DHS). 5 https://www.wateraid.org/ng/a-context-analysis-of-urban-sanitation-in-three-nigerian-cities. 6 https://www.susana.org/_resources/documents/default/2-1662-chowdhury-2012-business.pdf. 86 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION E-WASH Activity commenced in May 2018 with the SWBs from Abia, Delta, Sokoto, Imo, Taraba, and Niger. Sokoto subsequently dropped out of the Activity in quarter (Q) 2 of fiscal year (FY) 2020, leaving the Activity with five states. In August and September 2020, the USAID Mission in Nigeria commissioned a mid-term evaluation of the Activity’s performance in the five participating states through a third party. The USAID Mission issued the mid-term evaluation report in October 2020. The outcome and recommendations from the evaluation led to the descoping, reorientation, and streamlining of activities and, hence, a revised implementation workplan to end the Activity life in 2022. On sanitation, the Activity completed city-wide sanitation mapping studies and shift flow diagrams (SFDs) for five cities, one in each of the five States. Similarly, E-WASH organized environmental mitigation and monitoring plan (EMMP) training for its prospective contractors in Q3 FY2021. 2.2 OBJECTIVES The overall goal of E-WASH is to assist the GoN to expand and improve urban water service delivery by strengthening the governance, financial, and technical viability of select SWBs in Nigeria. The E-WASH Activity worked towards achieving this goal through the implementation of tasks and activities designed to achieve four objectives: (i) create professionally managed and commercially oriented SWBs; (ii) improve the financial and operational viability of SWBs; (iii) strengthen policy, institutional, and regulatory frameworks for improved WASH services; and (iv) build national and Figure 1: State with E-WASH operations (including Sokoto which pulled out) and RTI Head Office in Abuja 87 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION state WASH advocacy, coordination, and communications for reform.7 Upon completion, the E￾WASH Activity expects to achieve the following high-level results: a) Five SWBs professionally managed that: (i) achieve greater commercial viability; (ii) are overseen by a board of directors appointed by the State Governor based on existing laws; (iii) demonstrate greater managerial autonomy; (iv) can directly hire personnel and set the salary structure; and (v) achieve greater cost recovery by reducing inefficiencies and non￾revenue water losses and corruption. b) Up to USD 10 million of additional sector financing mobilized from public and/or private sources. c) Five SWBs that have implemented revised policies, institutional frameworks, or regulations that promote access to improved WASH services; and d) At least 54,809 households (426,325 people) that have gained access to basic drinking water or improved supplies piped onto their premises across selected states by the close of the four-year period. 2.3 ACTIVITY HYPOTHESIS For the implementing states to improve and sustain WASH services and to achieve its objective, the Activity proposes the following theory of change: • If SWBs can demonstrate a robust governance structure that embodies autonomy, accountability, and transparent financial and operational management for inclusive service provision; and • If policy, institutional, and regulatory frameworks for improved WASH services are strengthened; and • If WASH sector reforms at the state and national levels are promoted through targeted advocacy, coordination, and strategic communications; • Then, increased access to sustainable WASH services will be achieved in urban areas within Nigeria and will raise the quality of life of its customers, facilitate economic performance in their service areas, improve finances by reducing or better targeting of subsidies, and increase the chance of serving all customers in their area by improving staff morale and increasing their competence. 2.4 ACTIVITY ASSUMPTIONS The E-WASH Activity made six vital assumptions to successfully reform the utilities and to turn them around to viable commercial and operational utilities. Firstly, it assumes that the state governments that own the water boards are committed to reform efforts. The second is that the governance improvements in the water boards will be strengthened by state governments’ motivation and willingness to entrench corporate structure in the water boards. Thirdly the state governments are committed to adjusting tariffs in terms of both cost recovery and affordability. The fourth assumption is that broad-based stakeholder engagement, effective civil society engagement, and accountability mechanisms are necessary to monitor and advocate for reform. The fifth assumption is that USAID can continue to work effectively with other development partners, particularly the World Bank and African Development Bank, to achieve meaningful scale and impact. 7 This component of the E-WASH Activity implementation was descoped after the mid-term evaluation. 88 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Lastly, the sixth assumption is that USAID’s efforts would catalyze governance improvements within the SWBs. 2.5 RESULTS FRAMEWORK To achieve this goal and the principal objectives, the overall results framework (see Appendix 1) for the E-WASH Activity provides the model for the results for the Activity. The results framework adapted for this Activity shows the pathways by which the Activity will increase access to sustainable WASH services in selected urban areas in Nigeria. This framework is central to the management, monitoring, and evaluation of this Activity, and to this end, it will guide this Activity performance evaluation. Within the USAID Country Development Cooperation Strategy, the E-WASH Activity is located under Development Objective (DO) 3: Strengthened Good Governance; Intermediate Result (IR) 3.2: Improved responsiveness of targeted government institutions and 3.3: Increased civic advocacy, monitoring, and engagement, with Business Environment Improved, Sub-IRs 3.2.3: Capacity and accountability of key WASH stakeholders strengthened and 3.3.2: Civil Society Organization (CSO) oversight and technical advocacy capacity strengthened. 2.5.1 E-WASH ACTIVITY’S IMPLEMENTATION STRATEGY The general approach adopted by E-WASH to achieve the expected results for the Activity was to focus on strengthening the financial and operational viability and managerial capability of SWBs as commercial water utilities. To do this, the E-WASH Activity worked directly with five SWBs of Abia, Delta, Imo, Taraba, and Niger while concurrently addressing state and national-level governance issues around policy, institutional, and regulatory reform. In addition, E-WASH adopted the use of advocacy, coordination, and strategic communications. E-WASH implemented these activities across the four key components to enable and promote the following: (i) Creating professionally managed and commercially oriented SWBs. The main objective of this component was to create professionally managed and commercially oriented SWBs in urban areas of five states. The Activity focused on improving and strengthening the corporate governance and increasing political and managerial autonomy of utilities through the development and implementation of tools such as Board charters to define roles and responsibilities; developing incentives and performance-based contracts between SWBs and state governments; strengthening a corporate culture that motivates and inspires staff to excel and achieve the SWB mission; enhancing organizational structure with staff development, retention, and transition plans; and developing appropriate human resources policies and transition plans towards a fully corporatized SWB. (i) Creating professionally managed and commercially oriented SWBs. The objective of this component was to provide tailored capacity development in financial, commercial, managerial, and operational aspects of the SWBs to improve their financial and operational viability. The activities targeted to achieve operational and financial viability included: development of Enterprise Results Planning systems for the utility; tariff studies that included tariff adjustment to reflect the actual cost of water and support the implementation of tariff increase; non￾revenue water reduction strategy, development of standard operating procedures (SOPs) for commercial and operational activities, the introduction of E-Collection platforms in some state utilities; operationalizing a USF for emergency repairs and small-scale improvements to increase operational efficiency; deployment of a customer enumeration exercise; design and implementation 89 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION of customer-centric processes, such as the establishment of dedicated customer service centers; and exploring options for private sector participation. (ii) Strengthening policy, institutional, and regulatory frameworks for improved WASH services. The objective of the third component was to create an enabling environment for urban WASH sector reform in select states to thrive, focusing on policy, institutional, and regulatory frameworks. To achieve this, RTI implemented activities such as: • Review and/or develop WASH policy for all states; • Develop a new law (except for Delta state); • Develop the institutional framework for WASH services; • Support the establishment of a regulatory unit in transition to creating a regulatory commission; • Promote improved sector financing; • Establish public accountability tools and processes; • Enhance social accountability and institutional transparency; • Plan and implement sanitation service improvement plans; • Operationalize a small grants fund to support advocacy and innovations in service delivery; and • In Niger: assist with the tracking, evaluation, and promotion of increased sector financing from the Niger State Government and development partners. (iii) Building national and state WASH advocacy, coordination, and communications for reform.8 The objective of the fourth component is to improve WASH service delivery through heightened and more effective coordination, advocacy, and strategic communication efforts that will spur transformational change in the sector. To achieve this, the IP implemented activities such as: • build the capacity of national and state-level WASH coordination agencies; • develop and implement a communications strategy to support reform; • develop and execute a knowledge management strategy and action plan; and • carry out and support research to build political support for reform. City Wide Sanitation Mapping As part of the revised FY2020-2022 work plan, the Activity implemented the city-wide sanitation mapping in the five major cities of the E-WASH implementing states. The Activity seeks to improve the cities’ sanitation services through the initiation of Citywide Inclusive Sanitation (CWIS) plans and the development of sanitation mapping for the following cities: Aba in Abia, Asaba in Delta, Owerri in Imo, Minna in Niger, and Jalingo in Taraba, and the setting up of the sanitation taskforce in each implementing state and activities to organize sanitation workers. USAID’s Environmental and Construction Guideline The Activity organized training on EMMP for its prospective contractors in Q3 FY2021, using the USAID’s Construction Sector Environmental Guidelines (SEG). The guidelines provide guidance on key requirements for mandatory environmental compliance procedures. These procedures are 8 This component of the E-WASH Activity implementation was descoped after the mid-term evaluation. 90 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION USAID’s principal mechanisms for assuring environmental impact assessment of USAID-funded activities, and thus, for protecting environmental resources, ecosystems, and the health and livelihoods of beneficiaries and other groups. Training topics covered include an introduction to EMMP concepts, the implementation approach, how to mitigate hazards, and health safety. The USAID Mission in Nigeria approved a descoping of the E-WASH Activity in Q1 of FY2021. The descoping resulted in a program reorientation with a streamlined set of activities for FY2021-2022 based on the mid-term evaluation findings, contextual shifts, adjusted priorities, and the impact of the Coronavirus Disease 2019 (COVID-19) pandemic on overall Activity implementation. This involved the cessation of most technical assistance activities across all Activity components by the end of Q2 FY2021, except for water supply production and distribution monitoring; water quality management, assets, and Non-Revenue Water (NRW); performance monitoring; and limited support to Abia. The descoping prioritized: (i) provision of infrastructure and equipment under the USF by completing ongoing procurement activities and initiating nine new construction activities to improve water supply production and distribution across all five states; (ii) targeted technical assistance for ensuring sustainability and the adoption of tools, processes, and systems that the Activity has introduced since its inception; and (iii) support for monitoring, evaluation, and learning (MEL) interventions, reporting, and overall Activity closeout activities. 2.5.2 PERFORMANCE INDICATORS Following the recommendations from the mid-term evaluation of the E-WASH Activity and the subsequent descoping of the Activity, RTI revised its work plan to the end of the Activity, which USAID subsequently approved. RTI identified a total of 29 indicators in the FY 2020-FY2022 work plan to help track the progress across the Activity’s goals and the four Activity result areas (see Appendix 2: E-WASH Activity Performance Indicators). 2.6 IMPLEMENTATION CHALLENGES Activity implementation was not without its attendant challenges. The top challenge faced by the Activity was the advent of the novel COVID-19 pandemic in December 2019, which subsequently led to a lockdown in the month of March 2020 for about six months and the restrictions in gatherings. This impacted the overall delivery of the Activity, especially the procurement activities under the USF. This also affected the delivery of equipment due to factory lockdown. It slowed down some activities and resulted in the cancellation of others, especially with contractors/short￾term technical assistance (STTA) (both local and international). E-WASH staff working from home/remotely balancing personal and program activities led to a loss of momentum and slow delivery of planned activities. However, some have argued that the COVID-19 pandemic was a golden opportunity for the WASH sector globally, and for water utilities. In particular, to raise their performance and impact, considering the global attention to the essential role of handwashing in abating the spread of the virus and consequent demand for water supply. 3. EVALUATION OVERVIEW 3.1 PURPOSE AND OBJECTIVES OF THE EVALUATION The primary purpose of this final performance evaluation is to determine whether the assistance provided by USAID/Nigeria through the E-WASH Activity achieved the stated development objectives and to understand the lessons learned from this Activity. It also considers the findings and recommendations of the mid-term performance evaluation and the subsequent descoping of the E- 91 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION WASH Activity. It intends to provide an independent examination of the overall progress and accomplishments of the Activity. The final performance evaluation will provide a detailed picture of the major accomplishments and weaknesses of the Activity since its inception and determine how the Activity’s success can be sustained. This final performance evaluation will provide USAID/Nigeria, its IPs, and WASH sector stakeholders with data on outcomes and achievements. The result of this evaluation will be useful to the Mission, including the USAID Economic Growth and Environment Office and other USG officials, GoN, WASH IPs, private sector firms, and other stakeholders and to enhance urban WASH policy formulation, analysis, and implementation in the country. 3.2 EVALUATION QUESTIONS This evaluation design assesses the performance of the E-WASH Activity against its stated development objectives by focusing on relevance, effectiveness, sustainability, and how it considers gender and social inclusion in its implementation. In order to determine the achievement of these objectives, the evaluation will answer the following key questions: 1. To what extent did SWBs become commercially oriented and improve their financial viability? a. To what extent has the Utility Support Fund been effectively utilized to achieve its objectives? b. What impacts, if any, has COVID-19 had on SWB’s financial viability? 2. To what extent did the Activity support operational viability and professional management of SWBs? 3. To what extent did the Activity strengthen policy, institutional and regulatory frameworks for improved WASH services at the state and national levels? a. What accountability mechanism related to the implementation of policy and frameworks did the Activity put in place, how successful have they been, how could they be improved, and how might they impact sustainability? 4. To what extent did the Activity prepare target institutions to manage sanitation challenges through city-wide sanitation mapping? 5. What lessons can be learned from the Activity’s implementation of USAID’s environmental and construction guidelines? How can they be used for adaptive management? 4. EVALUATION DESIGN AND METHODOLOGY The evaluation team (ET) will use mixed-methods approaches drawing on primary and secondary data to conduct the final performance evaluation of the USAID E-WASH Activity. The ET will conduct primary and secondary data collection through a desk review, key informant interviews (KIIs)/group discussions, focus group discussions (FGDs), and direct observation during site visits in five states (Abia, Delta, Imo, Niger, and Taraba), and the FCT. The ET will collect data from a range of stakeholders (described below) and processes to ensure triangulation of results. 4.1 DATA COLLECTION PROCESS The ET’s four members will be divided into two groups to conduct field data collection visits. • The Northern Team will collect data in Niger, Taraba, and Abuja, led by the Team Leader/WASH Specialist Timeyin Uwejamomere and Institutional Development Specialist Boluwaji Onabolu. 92 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION • The Southern Team will collect data in Abia, Delta, and Imo and be led by Senior WASH Governance Specialist Titilola Bright-Oridami and Evaluation Specialist Victor Malaolu. 4.2 DOCUMENT/REVIEW The ET will conduct a desk review of existing documents and reports on the USAID/Nigeria E￾WASH Activity and the broader WASH sector. The review will help the ET understand the context and the strategic pathway of the E-WASH design, variations, and implementation, and the E-WASH implementation successes and gaps. The review will also provide the necessary background information/knowledge required for the effective conduct of the final performance evaluation of the E-WASH Activity. The ET will extract key results from the review and triangulate them with the data from the field visits. Desk review findings. The ET conducted a preliminary desk review to inform the work plan and data collection planning. The ET assessed key E-WASH documents to target the evaluation’s scope and indicators required to address each evaluation question. In addition, the ET used the review to identify areas for follow-up and gaps to address during data collection. Key findings. A significant milestone in the lifeline of the E-WASH Activity was the conduct of a baseline study, an assessment of the water supply and production, and a political economy analysis for the six initial states. These studies set the stage for critical interventions and a strategic approach for the Activity. For instance, the teams of technical experts assessed water production and supply systems and determined the most appropriate areas of support to improve SWB water-supply delivery, especially through the Activity’s USF. An additional key finding is that surface water is the main source of the water supply in Abia, Imo, Niger, and Sokoto, while groundwater is the main source in Delta and Taraba. Imo and Abia SWBs were non-operational for over five years, prior to the study, in October 2019. Key gaps/areas for further investigation. Subsequent reports reviewed revealed that the E￾WASH Activity, in some instances, led to significant achievements and improvements in SWBs. However, the reports presented several of these interventions as a one-time achievement, and in later reports, did not provide an update on the progress or the specific facilitation and support implemented. For example, E-WASH supported financial management improvements in each SWB, resulting in raised collection efficiency ranging from three percent to over 40 percent in support of the performance improvement plans. There is a need for the team to unpack the specific support to understand E-WASH’s contribution better. In 2019, the E-WASH Activity achieved some foundational results; however, there are no updates in subsequent years on the status of the result. For example, to improve urban sanitation, the Commissioners in Delta state set up a dedicated Sanitation and Quality Control Department within the SWB (DSUWC) on September 18, 2019, with a Sanitation Unit, a Laboratory and Quality Control Unit, and a Liquid Waste Treatment Plant Unit. There is no mention in the subsequent years of the progress of this department. The ET will review key documents as they become available during preparation and data collection to refine interview guides and inform evaluation findings. 4.3 KEY INFORMANT INTERVIEWS/GROUP INTERVIEWS The ET will conduct KIIs and group interviews (two to five participants) using the semi-structured guides designed for each stakeholder group. There, the ET will hold KIIs/group interviews with the stakeholders listed below and in Table. 93 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 1. General managers involved in the Activity in USAID/Nigeria’s focus states 2. Board of Directors of SWBs 3. E-WASH state team 4. Secretary to the State Government 5. Honourable Water Commissioners 6. Federal and state ministries 7. E-WASH contractors/consultants 8. Utility staff 9. LGA Sanitation Unit 10. Sanitation Task Force The ET will also conduct KIIs with the E-WASH central office staff (Chief of Party, MEL Advisor, USF/Procurement Specialist, and Utility Operations Advisor). The ET also aims to conduct KIIs with USAID/Nigeria staff involved in the E-WASH Activity (e.g., Contracting Officer’s Representative [COR], the Alternate COR, and the MEL point of contact). The KIIs will elicit information on whether the assistance provided by the Activity achieved the stated development objectives. The ET will conduct small group interviews with two to four similar stakeholders as needed. Group interviews might include members of a senior management team of an SWB. The KII’s advantage is the derived knowledge from respondents; it usually provides data and insight that the ET cannot obtain with other methods. 4.4 FOCUS GROUP DISCUSSIONS The ET will carry out focus group discussions (FGDs) to obtain a utility customer perspective (from functional utilities) related to answering EQs 1-3. There will be 10 FGDs at the rate of two FGDs per state. The ET will ask a series of predetermined thematic questions on E-WASH Activity implementation to understand their thoughts and perceptions regarding their experiences with E￾WASH Activity or entities supported by the E-WASH Activity. The ET will ensure approximately five to eight participants per FGD. Participants will be a mixture of relevant stakeholders to allow for a robust discussion and allow each participant to share insights. The ET will ensure that the composition of the focus group participants is characterized by homogeneity but with enough variation to allow for contrasting opinions and voices. The FGD facilitator will ensure the process is interactive and participatory. 4.5 DIRECT OBSERVATION The ET will make limited site visits to Activity locations to observe and confirm the successful execution of a sample of Activity interventions from document reviews, and with the guidance of SWBs. Site observations will allow the opportunity to engage with laboratory facilities, water treatment sites, and infrastructure improvement sites that facilitate access to quality water and sanitation services as part of the Activity’s implementation in the states. The ET will potentially carry out observation of sanitation sites depending on the implementation phase in the states. Site observations will: (i) confirm that the ET has reported actions taken; (ii) get clarity on the sustainability of process implemented, or otherwise, to achieve commercial and financial viability; and (iii) confirm the findings from the Activity documents reviewed, and gaps observed. In addition, site observations will address the EQ on the effectiveness of the USF (EQ1a) and the lessons to learn from the Activity’s implementation of the USAID’s environmental and construction guidelines (EQ5). The tools for the field visits are field notes and pictures while observing standard research consent protocols. The ET will record field observations in a narrative or descriptive style. 94 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 4.6 SECONDARY DATA COLLECTION The ET will aim to collect secondary data to answer the EQs. The ET will source from SWBs’ documents/records on production, distribution, financial, and their customer database. Secondary data analysis and triangulation will be dependent on the ability to receive access to data within the evaluation timeframe. 4.7 SAMPLE SIZE AND SELECTION OF RESPONDENTS KII/Group Interviews. The ET will purposively sample stakeholders for KIIs and group interviews. The desk review will inform the purposive sample and initial KIIs with USAID and RTI stakeholders based on their ability to answer specific EQs. The ET will employ snowball sampling as necessary when additional key informants are identified. The ET intends to hold approximately 84 KIIs/group interviews with around 149 respondents, as stated in Table . FGDs. The ET will conduct a total of 10 FGDs: one FGD per state with utility customers and one with community members impacted by the construction activities. The ET will derive FGD participants for the utility customers from the SWB utility customer database. The ET will identify Sanitation FGD participants for community members impacted by the construction activities through a snowball sampling or from USF site visits. Direct Observation. Based on the review of the USF in the state, the ET will identify no more than five sites per state for direct observation. These sites, as stated above, are the laboratory facilities, water treatment sites, and infrastructure improvement sites that facilitate access to quality water and sanitation services as part of the Activity’s intervention in the states. The annexes of this report contain the draft KII and FGD guides. See Table for the anticipated 13 main stakeholder groups and number of respondents: 95 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Tables 1 and 2 below represent that the ET will conduct approximately 84 KIIs or group interviews with approximately 149 respondents across five states and the FCT. Table 1: Approximate Number of Key Informant/Group Interviews and Respondents TARGET RESPONDENTS ABIA IMO DELTA NIGER TARABA FCT TOTAL KEY INFORMANTS USAID - - - - - 3 3 E-WASH Staff 1 1 1 1 1 4 9 Federal and State Ministry (Sanitation, Water & Environment) 1 1 1 1 1 1 6 State Commissioner 1 1 1 1 1 - 5 Secretary to the State Govt. 1 1 1 1 1 - 5 Regulatory Agency 1 1 1 1 1 - 5 General Managers/MDs 1 1 1 1 1 - 5 Board of Directors of SWBs 1 1 1 1 1 - 5 SWBs (Water Supply & Sanitation) 2 2 2 2 2 - 10 Sanitation Task Force 1 1 1 1 1 - 5 Sanitation Workers 1 1 1 1 1 - 5 LGA Sanitation Unit 1 1 1 1 1 - 5 E-WASH consultants and contractors 2 2 2 2 2 - 10 TOTAL 15 15 15 15 15 9 ~78 Please note that some respondents may not be available during data collection, or an agency may not be active. In this case, the team will attempt to identify substitutes. In addition, activities in some states may not have fully implemented activities, and therefore, some interviews may not be necessary. 96 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION Table 2: Approximate Number of Group Interview Participants TARGET PARTICIPANTS ABIA IMO DELTA NIGER TARABA FCT TOTAL GROUP INTERVIEWS USAID 0 0 0 0 0 3 3 E-WASH Staff 3 3 3 3 3 4 19 Federal and State Ministry (Sanitation, Water & Environment) 2 2 2 2 2 2 12 State Commissioner 1 1 1 1 1 0 5 Secretary to the State Govt. 1 1 1 1 1 0 5 Regulatory Agency 1 1 1 1 1 0 5 General Managers/MDs 1 1 1 1 1 0 5 Board of Directors of SWBs 3 3 3 3 3 0 15 SWBs (Water Supply & Sanitation) 8 8 8 8 8 0 40 Sanitation Task Force 2 2 2 2 2 0 10 Sanitation Workers 2 2 2 2 2 0 10 LGA Sanitation Unit 2 2 2 2 2 0 10 E-WASH consultants and contractors 2 2 2 2 2 0 10 TOTAL ~28 ~28 ~28 ~28 ~28 ~9 ~149 97 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 4.8 QUALITY CONTROL AND OVERSIGHT To ensure quality control, the ET will implement two levels of quality assurance. The first is at the level of the ET conducting the data collection. The second is an oversight role of the ET conducted by the MEL Support Activity team. During fieldwork, the ET will combine audio recording with notetaking while the interviews are ongoing. This will be necessary for a better interpretation of the data. Additionally, the ET will conduct all KIIs and FGDs. The ET will adopt multiple data sources and a triangulation method to ensure the validity of the data throughout the evaluation process. The ET will compare the data it collects from KIIs with the Activity’s progress reports and desk review results to validate some of the key findings. The ET will apply robust data analysis techniques, including qualitative coding software, to analyze the qualitative data to draw evidence-based conclusions and recommendations. 4.9 DATA ANALYSIS The MEL Support Activity technical team will backstop the consultants and provide the ET with technical and logistical support to assist with respective beneficiaries identified for the evaluation. To ensure quality control and oversight during the data collection for the evaluation, Olufemi Gisanrin (Senior Assessment and Evaluation Advisor) and Holly Dentz (Project Director) will provide oversight of the ET. The Senior Assessment and Evaluation Advisor and the Project Director will frequently check in with the team regarding progress in the field, ensure that interview notes are uploaded daily and help the ET troubleshoot any issues encountered. They will also ensure ongoing monitoring of the security situation on the ground, providing security clearance for each location ahead of the field visits. The MEL Support Activity team will also engage the services of a data analyst to assist the ET as needed in the coding and analysis. 4.9.1 QUANTITATIVE DATA The ET will analyze quantitative data using SPSS, STATA, or similar software using a descriptive analysis approach such as mean, percentages, variance, etc., of key Activity indicators to assess implementation as well as assess SWB operational and financial data. The quantitative analysis will involve the descriptive analysis of some of the Activity indicators to assess the implementation performance and the extent to which Activity objectives have been achieved. The ET will assess and clean, as necessary, all data it receives for analysis. The preceding analysis plan is dependent on timely access to secondary data. The ET may complement this approach with a graphical presentation of findings, while also presenting the interpretation of the data in terms of meaning and significance. The ET will disengage, by gender, any relevant secondary data or include gender equality and social inclusion analysis. 4.9.2 QUALITATIVE DATA The ET will conduct a qualitative analysis of KII, group interview, FGD, and observation data using standard analytic approaches. The team will identify a software package (e.g., Dedoose) or use a findings matrix to synthesize and analyze data. The initial data preparation and analysis steps will include ensuring accurate data transcription, assessing data to identify meaningful segments (likely by EQ), codebook development, and application of codes. The code application process will facilitate the ET’s ability to identify themes and categories, similar data pieces, look for outliers, and individually organize thinking around the data. The EQ will guide the thematic analysis, and the ET will record appropriate tallies of thematic concepts’ frequencies. Where appropriate, data will be disengaged by gender or include gender equality and social inclusion analysis. 98 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 4.10 ETHICAL & COVID-19 CONSIDERATIONS The ET will ensure sound ethical consideration during the data collection process. Respondents will be informed about the purpose of the evaluation, and the ET will seek their consent prior to the commencement of the data collections exercise. The ET will choose locations where the respondents will be free to talk without interruption and maintain their privacy during the interview. In addition, the ET will ensure that everyone, including the participants, adheres to the COVID-19 safety protocols, particularly the use of masks, social distancing, etc. The ET will ask questions with consideration for the respondents’ personalities and interests. Also, the team will comply with Social Impact, Inc.’s standards by ensuring that respondents are referred to by role/job title and location. The team will ensure that data collection protocols do not reflect any sensitive information like the organizational mode of operations, which could later pose a threat to their performance. The ET will ensure that security clearances are always granted by the MEL Support Activity security operatives at each point of the field for data collection before embarking on any journey. In addition, while on the field, the team will ensure strict compliance with all security advice issued by the MEL Support Activity and/or its partners where applicable. 4.11 GENDER AND SOCIAL ANALYSIS PLAN The ET will collect quality data that accurately reflects the perspectives and opinions of participants by paying attention to gender, social equity, and marginalization issues at every stage of the data collection and as defined in USAID’s Gender Equality and Female Empowerment Policy. The ET will operationalize these considerations by ensuring that data collection tools include questions focused on important gender and social inclusion issues, directed at a broad range of stakeholders to collect various perspectives and cross-validate findings from different segments of activity implementation of beneficiary populations. As relevant to this evaluation, this may include, for example, the explicit or implicit ways that women, youth, or other potentially marginalized or disadvantaged groups are considered in Activity design, how they may be included or excluded from project activities including policies, associations, community initiatives, grant programs, and other efforts, and the ways that such project activities may impact them, whether intentional, unintentional, positive, or negative. The ET will be sensitive to the power relations between genders during interviews and ensure social inclusion by creating an atmosphere that is friendly enough to allow the female, the poor, and/or those with a disability to express their opinions. Also, the ET will consider the location and when to hold the interviews so that both women and men can attend. Where relevant, sex- and age￾disaggregated individual-level data will be analyzed and presented in the evaluation report. 4.12 RISKS Predicting all the risks that might arise in the course of this evaluation is not possible. However, some risks considered likely to arise are listed below: 1. There may be a disruption of the ET’s data collection travel schedule if there is an outburst of violence and heightened insecurity cases in any Activity location. To mitigate this risk, the team will seek travel advice from the Activity implementer and maintain close communication while in the field. Where threats exist, the team will revert to remote data collection, e.g., using telephone or vestural methods. 2. The ET will be meeting and interacting with different stakeholders while considering that COVID-19 continues to pose a risk. To mitigate this risk, the ET will insist on adherence to COVID-19 safety protocols as advised by the Nigeria Centre for Disease Control. The team 99 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION may resort to virtual means, such as telephone calls and Zoom meetings, where it is impossible to adhere to the COVID 19 protocols. 3. Sometimes, the respondents may not answer questions with 100 percent accuracy or without bias. This will be mitigated by giving and explaining the consent forms to the respondents, which explains that we are independent evaluators to help minimize response bias. An additional way to ensure data validity and mitigate bias is to triangulate findings among respondents. 4. There might be a perception that the evaluation threatens the financial and/or technical assistance provided by USAID/Nigeria. To mitigate this, the evaluation team will provide the objectives and purpose of the evaluation before commencing the interviews. The team will also administer the informed consent form to mitigate this possibility. 5. DISSEMINATION AND UTILIZATION The USAID Economic Growth and Environment Office, other USG officials, WASH IPs, and other stakeholders will use the result of this evaluation to enhance policy formulation, analysis, and implementation in the country. USAID will develop a dissemination plan in accordance with the Evaluation Policy as specified in the Automated Directive System (ADS). A dissemination session will be conducted with other technical units within the Mission, IPs, the donor community, and other stakeholders. Dissemination activities will include a preliminary findings presentation with the USAID technical team, followed by a final presentation to a wider Mission team. The dissemination package also will include a two-page brief and an infographic document. These varied deliverables will help ensure that the learning from the evaluation can be shared as broadly as possible and will help encourage uptake and adaptation. The USAID EGE team will provide hard copies of materials to participants. 6. STUDY MANAGEMENT AND TIMELINE It is estimated that this evaluation will commence in October 2021 and end by February 2021 (see Table 3). The ET will commence field visits from the end of November 2021 to December 18, 2021. During this period each team will collate data and interact with stakeholders as detailed below. Table 3: Evaluation Team Data Collection Itinerary Deployment Dates Locations Deployment Team Nov 29- Dec 3 Abuja Northern Team Imo Southern Team Nov 29- Dec 3 Niger Northern Team Abia Southern Team Nov 29- Dec 3 Taraba Northern Team Delta Southern Team 100 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION ANNEX VI: E-WASH PERFORMANCE INDICATORS S/N INDICATOR DEFINITION LEVEL ID IN E- WASH TYPE UNIT OF MEASURE DATA SOURCE FREQ. 1 HL 8.1.2: Number of people gaining access to safely managed water services as a result of USG assistance (standard from DO Performance Management Plan [PMP] indicators). (Impact) DO Standard Impact Number Activity records 2 HL.8.1-1: Number of people gaining access to basic drinking water services as a result of USG assistance DO Standard Outcome Number Activity records Annual 3 HL 8.4.1: Value of new funding mobilized to the water and sanitation sector as a result of USG assistance. (Standard from DO PMP indicators). (Impact) DO Standard Impact USD Activity records Annual 4 DO3-16: Percentage of OCAT Strength Index achieved (custom) DO Custom Outcome Percentage Activity records Annual 5 Number of Active Connections Output Number Activity records Annual 6 EWA2.1.1.3.1: Number of Staff per thousand Connections Output Number Activity records 7 EWA2.1.1.5.2: Percentage of staff who are women (custom) Custom Output Percentage Activity records 8 EWA2.1.1.5.3: Number of internal or external leadership development/training/skills development opportunities provided to employees (indicator from EUM primer performance measures) Output Number Activity records Annual 9 Indicator 1.5.4: Percentage of positive customer satisfaction survey responses based on a statistically valid survey or an immediately after-service survey Output Percentage Activity records 10 EWA2.1.1.1.1: Percentage of income from tariff (custom) Custom Outcome Percentage Activity records Annual 11 DO3-17: Percentage of water utility consumers who pay according to consumption (standard from DO PMP indicators list) DO Standard Outcome Percentage Activity records 12 EWA2.1.1.1.3: Operating Cost Coverage Ratio (custom) Custom Outcome Activity records 13 EWA2.1.1.2.1: Collection efficiency (indicator from EUM primer performance measures) Outcome Percentage Activity records 14 Capacity utilization efficiency (custom)- NEW INDICATOR Custom Outcome Percentage Activity records 15 EWA2.1.1.2.2: Production efficiency (standard from EUM primer performance measures) Standard Outcome Percentage Activity records 101 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION S/N INDICATOR DEFINITION LEVEL ID IN E- WASH TYPE UNIT OF MEASURE DATA SOURCE FREQ. 16 EWA2.1.1.2.3: Percentage of NRW (custom) Custom Outcome Percentage Activity records 17 EWA2.1.1.4.1: Drinking water quality compliance rate (standard from EUM primer performance measures) Standard Outcome Percentage Activity records 18 DO3-18: Number of reforms introduced, adopted, repealed, changed, or implemented consistent with citizen input (custom) DO Custom Outcome Number Activity records 19 EWA2.1.1.2.2: Number of WASH sector reforms introduced, adopted, repealed, changed, or implemented consistent with knowledge, best practices, lessons learned, and evidence from collaborating, learning, and adapting events and monitoring, evaluation, research, learning and adapting metadata (custom) Custom Outcome Number Activity records Annual 20 Indicator 3.1: Number of policies, laws, agreements, regulations, or investments (public or private) that promote access to improved water supply and sanitation (standard from Master list of indicators) Standard Outcome Number Activity records Annual 21 Percentage of population using safely managed sanitation services. USAID Standard Standard Output Percentage 22 Number of operators trained to provide urban sanitation services as a result of E-WASH intervention” custom Custom Output Number Activity records Annual 23 EWA2.1.1.3.5: Number of States with a specific institutional mandate for urban sanitation management (custom) Custom Outcome Number Activity records 24 EWA3.1.1.1.1: Number of oversight or internal accountability mechanisms strengthened, adopted, and implemented by host nation through USG assistance Outcome Number Activity records Annual 25 DO3-20: Net change in advocacy technical capacity of core partner civil society organizations (custom) Custom Outcome Percentage Activity records Annual 26 DO3-21: Number of CSOs receiving USG assistance engaged in advocacy interventions (standard from DO PMP indicators, Master List2019) DO Standard Output Number Activity records Annual 27 EWA3.1.1.2.1: Number of collaborating, learning, and adapting events organized to promote WASH-enhancing knowledge, best practices, lessons learned, and research data from monitoring, evaluation, research, learning and adapting processes (custom) Custom Output Meetings Personal logs Annual 28 EWA3.1.1.2.4: (Number of community-based WASH engagements targeting vulnerable groups (Women/Men/Boys/Girls/People Living with Disabilities - PLWDs). (custom) Custom Output Meetings Personal logs Annual ANNEX VII: TABLE OF EXCERPTS FROM CWIS EXERCISE REPORT 102 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION SFW - Safely Managed State City State Sanitation Mapping Report March 2021 fecal sludge and CSDA Scorecard wastewater Abia Aba The enabling environment for sanitation improvement 0 “In all areas of the service chain, there is little to is lacking in Aba. build on.” There is no functional, modern, and enforced local bylaw on sanitation. “All work on sanitation improvement will need to state at a basic level.” Access to capital for citizens to afford on-site system installation or desludging is very limited. There is no fecal sludge treatment plant in existence in Aba, although one is planned. Delta Asaba There are also manual de-sludgers operating in 0 Asaba has many areas for improvement. Asabam, but their numbers are unknown. The practice is considered illegitimate. The enabling environment for sanitation improvement is weak in Asaba. Also, there is no modern, functional, or enforced local bylaw. Access to capital for citizens to afford on-site system installation or desludging is limited. Asaba does not safely manage any fecal sludge or wastewater generated in the city. Imo Owerri The enabling environment for sanitation improvement 0 Work on developing sanitation interventions is lacking in Owerri but improving rapidly, mostly must start at the basic level. through the new water law and policy. Niger Minna The enabling environment for sanitation improvement 0 The work achieved in developing the state water is weak in Minna. law and policy and existing sanitation policies is a While the city’s professional staff are dedicated, their significant starting point for a CWIS program. departments are understaffed and lack budgets to send inspectors to the field. Also, there is no functional, modern, or enforced local bylaw on sanitation, and access to capital for citizens to afford on-site system or desludging is limited. State City State Sanitation Mapping Report March 2021 SFW - Safely Managed fecal sludge and wastewater CSDA Scorecard Taraba Jalingo There are no centralized sewers in Jalingo. Desludging occurs in Jalingo. Although there is no recognized and approved location for fecal waste disposal. 0 All work on sanitation improvement will need to start at a basic level. 103 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 104 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION ANNEX VIII: EXAMPLES OF PRIORITY ACTIONS FROM SANITATION MAPPING REPORTS State Examples of Priority Actions from Sanitation Mapping Reports (March 2021) Aba Use of community health environmental workers that are already active in Aba for door-to-door sanitation messaging; unlined pits should be eliminated throughout the city in favor of lined pits or proper septic tanks. Owerri This suggests immediate areas for capacity building. Community health environmental workers are active in Owerri, going door-to-door could be used for sanitation messaging and promotions campaigns around fecal sludge management programs; immediate and sustained capacity-building and greater coordination with the building department, surveyors, and environmental health and planning departments at the city level and state and national ministries. Jalingo Formalize manual de-sludgers; construction of some form of fecal sludge treatment facilities; use of public health workers for sanitation messaging; restore the use of sanitary inspectors; standardizing the informal sector. Minna, Asaba The state should explore decentralized sanitation systems for densely populated regions. Helping to formalize the manual emptiers would be an important component of any reform efforts. 105 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION ANNEX IX: RTI INTERNATIONAL TEAM E-WASH APPROACH FLOW CHART 106 | E-WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION ANNEX X: DISCLOSURE OF ANY CONFLICTS OF INTEREST 107 | E -WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 108 | E -WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 109 | E -WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION 110 | E -WASH ACTIVITY USAID.GOV FINAL PERFORMANCE EVALUATION United States Agency for International Development Abuja, Nigeria