Final Report: An Endline Evalua of the PRO-WASH Associate A (2018-2023) tion ward SEPTEMBER 2022 This report is made possible by the generous support of the American people through USAID. As an independent evaluation, the contents are solely the responsibility of Causal Design and do not necessarily reflect the views of USAID, the United States Government, Save the Children, or the PRO-WASH Associate Award. The PRO-WASH Associate Award being evaluated was also made possible by the generous support of the American people through USAID. Recommended Citation: Causal Design. (2022). Final Report: An Endline Evaluation of the PRO-WASH Associate Award (2018-2023). Washington, DC: The PRO-WASH Associate Award. Causal Design partners with international development clients to provide rigorous independent program evaluation, expand cultures of evidence within organizations, and join them in efforts to relieve human suffering and end poverty. 525 Santa Fe Dr. Denver, CO 80204 USA Tel: +1 (720) 260 4837 Email: Info@CausalDesign.com This page is intentionally left blank. Endline Evaluation of the PRO-WASH Associate Award (2018-2023) iv TABLE OF CONTENTS ACRONYMS__________________________________________________________________________ VI ACKNOWLEDGEMENTS ______________________________________________________________ VII EXECUTIVE SUMMARY _______________________________________________________________ VIII 1.0 INTRODUCTION___________________________________________________________________1 1.1 Evaluation Framework ___________________________________________________________1 1.2 Evaluation Questions ____________________________________________________________1 2.0 PRO-WASH BACKGROUND_________________________________________________________3 2.1 PRO-WASH’s Target Audience ____________________________________________________3 2.2 Key Learning Deliverables ________________________________________________________3 2.3 Engagement Intensity ____________________________________________________________5 2.4 Steering Committee______________________________________________________________5 2.5 Adaptive Program Delivery Processes _______________________________________________5 3.0 EVALUATION SCOPE AND METHODOLOGY___________________________________________6 3.1 Primary Data Collection __________________________________________________________6 3.2 Sampling ______________________________________________________________________8 3.3 Analysis _______________________________________________________________________9 3.4 Evaluation Ethics________________________________________________________________9 3.5 Data Storage, Privacy, and Security _________________________________________________9 3.6 Limitations _____________________________________________________________________9 4.0 FINDINGS _______________________________________________________________________11 4.1 Award Design and Objectives _____________________________________________________11 4.2 Implementing Partner Engagement and Impacts of Technical Assistance and Influence _______21 4.3 Outreach to Implementing Partners ________________________________________________32 4.4 Thought Leadership and Influence on BHA Policies, Standards, Program Quality ____________33 5.0 CONCLUSIONS __________________________________________________________________35 6.0 RECOMMENDATIONS______________________________________________________________1 ANNEXES ____________________________________________________________________________7 Annex 1 | PRO-WASH Final Evaluation Scope of Work_______________________________________7 Annex 2 | PRO-WASH Implementation Timeline Summary ____________________________________7 Annex 3 | Key Informant Interview and Focus Group Discussion Semi-Structured Interview Guides ____7 Annex 4 | Quantitative Survey Instrument _________________________________________________7 Annex 5 | Quantitative Survey Respondents Categorized by Position and Country or Region _________7 Annex 6 | Identification of Qualitative Target Respondents ____________________________________7 Annex 7 | Qualitative Code Book ________________________________________________________7 Annex 8 | Evaluation Ethics ____________________________________________________________8 Annex 9 | Case Studies________________________________________________________________8 Endline Evaluation of the PRO-WASH Associate Award (2018-2023) v LIST OF FIGURES AND TABLES Table 1 | Overview of Qualitative Respondents ________________________________________________6 Table 2 | Geographic Coverage represented by IP respondents in the qualitative sample_______________7 Table 3 | Quantitative Survey Respondents___________________________________________________8 Figure 1 | 'Please describe PRO-WASH in three words or phrases' _______________________________11 Table 4 | ‘What Three (3) Factors Most Influenced the Extent to Which PRO-WASH Responded to Your Changed Needs?’______________________________________________________________________12 Figure 2 | PRO-WASH Support to BHA-funded Emergency Partners and RFSA Partners______________21 Table 5 | Remaining or Emerging Gaps Requiring Further Technical Assistance_____________________24 Table 6 | Average Quality Rating by Average Number of Resources or Tools Used in the Respondent's Program _____________________________________________________________________________24 Table 7 | Average Utility Scores for Focal Topics _____________________________________________25 Table 8 | Average Effectiveness Rating of PRO-WASH Approaches to Support _____________________27 Figure 3 | Proportion of Respondents Who Have Used Tools and Resources in Their Program, of Those Who Engaged with the Tool or Resource________________________________________________________29 Table 9 | Factors Influencing the Extent to Which Respondents Use Tools and Resources ____________30 Table 10 | Likelihood of Continued Use of Learning Type _______________________________________31 Figure 4 | Perceived Effectiveness of PRO-WASH's Outreach to IPs in Relation to the Timing of Initial Outreach within the RFSA Program Cycle (N=74)_____________________________________________33 Table 11 | Findings-Conclusions-Recommendations (FCR) Matrix_________________________________1 Endline Evaluation of the PRO-WASH Associate Award (2018-2023) vi ACRONYMS BHA Bureau for Humanitarian Assistance CAQDA Computer-Assisted Qualitative Data Analysis FCR Findings-Conclusions-Recommendations FFP Food for Peace FGD Focus Group Discussion iDE International Development Enterprises IDEAL Implementer-Led Design, Evidence, Analysis and Learning IWRM Integrated Water Resource Management KII Key Informant Interview LoE Level of Effort MMCA Make Me a Change Agent MBS Market-Based Sanitation OFDA PCS Office of U.S. Foreign Disaster Assistance Program Cycle Support PII Personally Identifiable Information PRO-WASH Practices, Research, and Operations in Water, Sanitation, and Hygiene REAL Resilience, Evaluation, Analysis, and Learning RFA Request For Applications RFSA Resilience Food Security Activity R&I Refine and Implement SC Steering Committee SCALE Strengthening Capacity in Agriculture Livelihoods and Environment SEAH Sexual Exploitation, Abuse, and Harassment TOPS Technical and Operational Performance Support Program ToR Terms of Reference ToT Training of Trainers USAID United States Agency for International Development WASH Water, Sanitation, and Hygiene Endline Evaluation of the PRO-WASH Associate Award (2018-2023) vii ACKNOWLEDGEMENTS Causal Design would like to acknowledge and thank the dedicated and professional support the PRO-WASH team has provided to this evaluation team. The report was also made possible by the cordial working relationship offered by Bodhi Global Analytics, who led the administration of the quantitative survey. Causal Design thanks all respondents who provided their time and critical reflection to the findings and recommendations presented within this report. Endline Evaluation of the PRO-WASH Associate Award (2018-2023) viii EXECUTIVE SUMMARY “Interfacing with PRO-WASH who would actually assure you that WASH is something; WASH is a subject; WASH means something; WASH will add value and change lives. You refine and renew your commitment to your subject area, your program area, and say no, I think I know I'm in the right sector and I will be able to drive with the strength and confidence that it requires because it matters.” – Implementing Partner (IP) Respondent Introduction The Practices, Research, and Operations in Water, Sanitation, and Hygiene (PRO-WASH) Associate Award was an intervention devised by the United States Agency for International Development (USAID) and delivered by Save the Children US to enhance the water, sanitation, and hygiene (WASH)-related technical capacity of implementing partners (IPs) who received funding through USAID’s Office of Food for Peace (FFP). In 2020, FFP merged with the Office of U.S. Foreign Disaster Assistance (OFDA) to become the Bureau for Humanitarian Assistance (BHA). This endline evaluation was conducted over a period of 5 months in the final year of PRO-WASH. It examines the outcomes of the PRO-WASH Associate Award with particular focus on the internal delivery dynamics, service delivery mechanisms, and management practices established through the Award. The evaluation focuses primarily on learning, by examining the range of factors influencing participation, utilization, training uptake, and relevance among target audiences. Methodology The evaluation employed both qualitative and quantitative methodologies to elicit feedback from a wide range of stakeholders, including implementing partners (IPs), current and former PRO-WASH team members, colleagues from other Associate Awards,1 including Strengthening Capacity in Agriculture Livelihoods and Environment (SCALE), Program Cycle Support (PCS), PRO-WASH Steering Committee members, USAID/BHA representatives, and USAID Mission staff who engaged with the activity. A total of 38 respondents participated in 27 key informant interviews (KIIs) and focus group discussion (FGDs), whilst 178 individuals responded to the quantitative survey. The evaluation also reviewed key literature provided by the PRO-WASH team, such as learning products produced and disseminated, training reports, and internal monitoring data generated throughout delivery. Limitations include: • Respondents self-selected into the quantitative survey, which may have resulted in responses that are more strongly felt than the norm • Potential for recall bias and limitations in respondents’ knowledge • Suggestive rather than statistically representative findings may have occurred due to the small sample size and sampling methodology • Translation from French to English, which may have led to nuances being overlooked Findings Findings are presented in line with the learning questions provided in the evaluation Terms of Reference (ToR), available in Annex 1. 1 SCALE is a Technical and Operational Performance Support Program (TOPS) Associate Award and PCS is an Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) Associate Award. Both are USAID/BHA funded. Endline Evaluation of the PRO-WASH Associate Award (2018-2023) ix Overall, the data clearly and consistently present a remarkably positive image of the PRO-WASH Associate Award. Award Design and Objectives Data demonstrate that PRO-WASH was generally considered to be responsive and flexible to the needs of IPs. This was influenced by both the design of the Award by USAID, and the management and delivery of the technical advisory support by PRO-WASH. Issues relating to flexibility include the challenges created in designing a team and structures for the effective and efficient delivery of PRO-WASH, and challenges associated with the approval process for programmatic procurements. PRO-WASH was a demand-driven function by design and delivery. While data suggest that this was recognized as a positive factor within PRO-WASH’s value-add to IPs, the design did create challenges such as ambiguity over PRO-WASH’s mandate to make sustainable impacts on IP activities, particularly in cases where IPs did not request the support. Data also suggest that PRO-WASH has been instrumental in informing as well as responding to IP demand. Numerous factors emerged from the data as having influenced the uptake of technical assistance by IPs, including their knowledge of how to access support, digital connectivity and language barriers, IPs’ perception of the relevance and quality of the support, time availability of the IPs, personalities and team dynamics within the IPs, and the perception of PRO-WASH as ‘Save the Children,’ making it a potential competitor. Data suggest that USAID has had a definitive impact on IPs’ uptake of technical assistance. This report provides an overview of the role that PRO-WASH team members had in the Award’s success, as well as the challenges faced by an under-resourced staffing structure. The report goes on to discuss the processes PRO-WASH undertook to work with consultants and the risks this structure posed to the quality. Issues relating to the aid localization agenda are discussed with reference to a debate that emerged in the data relating to international and national consultants. Finally, the findings relating to the Steering Committee (SC) are presented, which include strengths such as the SC’s role in providing a sounding board for PRO-WASH and expanding the Award’s network of technical experts. Challenges include a lack of clarity over the purpose of the SC, limited knowledge of SC members regarding PRO￾WASH management and delivery, limited interaction between the SC and USAID, a lack of clarity and consistency regarding the time/level of effort expectations of SC members, and the composition of the SC membership, which neither included representatives from countries in which Resilience Food Security Activities (RFSAs) are implemented, nor members of often marginalized groups. Implementing Partner Engagement and Impacts of Technical Assistance and Influence Data suggest that PRO-WASH offered significant added value beyond that which an IP can independently offer to a RFSA. This added value manifests in PRO-WASH’s ability to convene RFSAs, profile RFSAs’ work, draw on networks of specialists, and attract researchers due to their strong reputation in the field. Another example of added value is the experiential learning generated for PRO-WASH team members themselves in technical advisory support, which data suggest is being passed onto IPs. PRO-WASH also provided added value in the role of a partner staying abreast of emerging learning and through its positional power to influence RFSA decision￾makers, who may otherwise deprioritize WASH within the spectrum of issues covered through a RFSA. Findings suggest that IPs generally felt that PRO-WASH support was relevant, of high quality, and utilizable. Exceptions to these general findings were incidents where a lack of contextual adaptation of tools and resources diminished an IP’s ability to apply the tools and resources in the field. Language barriers and COVID-19 also emerged as issues inhibiting the utilization of tools and resources. In terms of the approaches PRO-WASH adopted to provide technical advisory support, respondents were highly satisfied with all approaches. Coaching emerged as a catalytic approach that acted as a cornerstone to other methodologies. Respondents shared multiple examples of how tools and resources have been used in RFSAs, including adaptations to behavior change strategies, development of new strategies, and development of new monitoring processes to track delivery. Endline Evaluation of the PRO-WASH Associate Award (2018-2023) x Respondents demonstrated a high intention to continue using resources, though data suggests this would be primarily dependent on need. Data suggest that tools and resources generally have not been institutionalized beyond use in specific activities. Many respondents noted that they have not yet discussed plans for the continued use of tools and resources following PRO-WASH’s closure, which is of particular concern to IPs who are engaging in ongoing support with the Award. Outreach to Implementing Partners PRO-WASH’s approach to outreach was typically appreciated by respondents. Data suggest that initial contact in the early stages of the RFSA cycle is related to higher perceived relevance and the use of tools and resources. Thought Leadership and Influence on BHA Policies, Standards, Program Quality Respondents with positional insight into PRO-WASH’s thought leadership cited some examples of PRO-WASH’s influence on BHA policies, standards, and program quality, though challenges that limited PRO-WASH’s ability to influence this included the complexity of USAID/BHA structures and processes and the limited level of effort (LoE) available to the PRO-WASH team. Conclusions Causal Design presents the overall assessment of PRO-WASH’s key achievements against its objectives, as derived from the findings. Lessons presented have been generated on the basis of the evaluation of evidence, with a particular focus on future phases of technical assistance support. Key conclusions relate to the overall added value that PRO-WASH is perceived to have had for RFSAs, beyond that which an IP could independently offer to RFSAs. Additionally, the conclusions discuss the factors that both supported and inhibited the value add, including resourcing, governance structures, approaches taken by PRO-WASH, and factors outside of PRO￾WASH’s control. Recommendations Finally, the report closes with a set of recommendations for USAID/BHA, PRO-WASH, and Save the Children US that seek to respond to the provided conclusions. Key recommendations include the continued and increased investment in PRO-WASH, suggested changes in the approach to engaging IPs, which are designed to motivate uptake, a revised governance structure, and approaches to further the aid localization agenda. 1 1.0 INTRODUCTION The PRO-WASH Associate Award was an intervention devised by USAID and delivered by Save the Children US to enhance the WASH-related technical capacity of implementing partners (IPs) receiving funding through USAID’s Office of Food for Peace (FFP). In 2020, FFP merged with the U.S. Office of Foreign Disaster Assistance (OFDA) to become the Bureau for Humanitarian Assistance (BHA). This endline evaluation examines the outcomes of the PRO-WASH Associate Award, with a particular focus on the internal delivery dynamics, service delivery mechanisms, and management practices the Award established. The evaluation primarily focuses on learning by examining the range of factors influencing participation, utilization, training uptake, and relevance among target audiences. 1.1 Evaluation Framework In accordance with the learning questions provided in the ToR (Annex 1), Causal Design approached the evaluation as a systematic and objective process review. Seeking to understand how the mechanisms designed and delivered under the PRO-WASH Associate Award contributed toward the achievement, or otherwise, of the Award’s objectives, the evaluation provides an assessment of the Award’s effectiveness, efficiency, and sustainability, with an emphasis on what facilitated or hindered engagement among key stakeholders. 1.2 Evaluation Questions The specific learning questions that the evaluation investigated are presented below under their respective category. Award Design and Objectives • How did the technical design of the Award and the level of flexibility affect the ability to respond to IP and donor needs, collaborate with partners, and adapt to external circumstances (e.g., COVID-19 pandemic, Food for Peace/Office of Foreign Disaster Assistance merger)? o What were the favorable or impeding factors related to the design and management of the Award that affected the Award’s ability to meet its objectives? • How appropriate and effective is PRO-WASH’s organizational and staffing structure for the implementation of our work? o How effective was the design, composition, and management of the PRO-WASH Steering Committee? How could this be improved? • How did the design, staffing, and management of the Award affect its ability to engage with and meet the needs of BHA-funded emergency partners? Implementing Partner Engagement and Impacts of Technical Assistance and Influence • How did IPs perceive the relevance, utility, and quality of the different types of support offered by PRO￾WASH (e.g., applied research, technical guides, learning briefs, coaching, remote and in-person training, technical knowledge-sharing events, webinars, and communities of practice)? o To what extent did IPs perceive that PRO-WASH tailored priorities to their requests/needs? o Which technical focus areas were the most useful? o What remaining gaps/existing challenges have not been met through PRO-WASH learning and research? o What were the barriers/enablers in terms of working with PRO-WASH on this research? • How did the combination and intensity of different approaches to engagement (e.g., training, applied research, coaching, etc.) influence the WASH/Integrated Water Resource Management (IWRM) technical quality improvements? o Which approaches most effectively and efficiently influenced WASH/IWRM technical quality improvements within RFSAs? 2 o To what extent did layering engagements (e.g., coaching, mentoring, in-person, or virtual training, applied research) contribute to IPs being able to use the resources/tools? o How did PRO-WASH contribute to WASH/IWRM knowledge-sharing and an understanding of best practices and promising/innovative models for WASH service delivery among RFSAs? • To what degree did IPs use resources/tools (technical tools, training packages, learning briefs, webinars and applied research findings) developed by PRO-WASH within the specific projects that the Award supported? o How easy was it for partners to find and access resources/tools? o Which PRO-WASH resources/tools did partners use and how often? o What are the barriers/enablers to IPs using the resources/tools? o What are the barriers/enablers to IPs replicating training? • To what degree do partners believe they will continue to use and share the resources/tools developed by PRO-WASH after the end of the Award? o Which resources? o How will they be used or shared, and with whom? o What are the barriers to this continued use and sharing? o To what extent will the knowledge management platforms PRO-WASH used ensure the sustainable access to and uptake of their respective resources? • To what degree did IPs institutionalize the resources/tools developed by PRO-WASH into their organizations beyond the specific project that the Award supported? o What are the barriers/enablers to IPs institutionalizing the resources/tools? Outreach to Implementing Partners • At what points during the RFSA cycle was IP outreach the most and least relevant and effective? Why? • How effective was PRO-WASH’s outreach as part of the PCS Inception and Culmination workshops? o How can this be improved? Thought leadership and Influence on BHA Policies, Standards, and Program Quality • How did the Awards inform advancements or changes to USAID/BHA policies, standards, and guidance? The learning questions purposefully did not seek to determine the extent to which the quality of WASH interventions delivered by IPs were improved, or WASH behaviors changed, as a result of PRO-WASH technical assistance. This decision reflects an agreement between the PRO-WASH team and the USAID/BHA due to the impracticability of assessing impact level change in this regard. PRO-WASH was not designed to include the monitoring and evaluation that would be needed to collect and analyze data around the adoption/implementation of new practices and the quality and impact of these practices within RFSAs. 3 2.0 PRO-WASH BACKGROUND PRO-WASH aimed to enhance the water, sanitation, and hygiene (WASH)-related technical capacity of implementing partners (IPs) receiving funding from USAID/BHA (former FFP and OFDA).2. Recognizing that: a) poor WASH facilities and practices are drivers of malnutrition and food insecurity, b) partners need to implement technically sound WASH interventions to address these drivers, and c) there are challenges and knowledge gaps in this sector, USAID created PRO-WASH as an Associate Award under the Technical and Operational Performance Support (TOPS) Program.3 PRO-WASH was “intended to strengthen the impact and sustainability of WASH activities within the context of FFP’s (latterly BHA’s) food security programs, in both emergency and development contexts.”4 Though TOPS did not initially include WASH as a technical area, PRO-WASH was designed with reference to the same overarching focal themes of TOPS, which were Knowledge Capture, Knowledge Generation, and Knowledge Sharing. Save the Children was awarded the PRO-WASH Associate Award in January 2018 with an initial budget of US$3.75 million over 3 years. In the third year of the initial Award, PRO-WASH was expanded and awarded an additional US$1.25 million to continue for 2 years, and with an additional focus on IWRM. During the extended 5- year award period (January 2018 – January 20235), PRO-WASH has been consulting with key partners and stakeholders to devise a quality technical assistance activity that is relevant and useful to BHA IPs and the wider WASH and food security sector. The figure provided in Annex 2 presents an implementation timeline summary of the key events that occurred up to March 2022. 2.1 PRO-WASH’s Target Audience PRO-WASH’s primary clients are Title II-funded RFSAs and from Year 4 Quarter 3, RFSAs funded through Community Development Funds (CDF). IPs of these activities, specifically the WASH elements of IPs that implement multi-sectoral food security awards, were targeted for outreach and feedback for the technical focus areas that PRO-WASH should prioritize and the capacity strengthening strategies. These outlined topical focus areas as well as modalities for the provision of support, which were developed to respond to their needs. Whilst open resources such online knowledge-sharing webinars and resources available on the Food Security and Nutrition (FSN) website have been used by a wider audience, including International Disaster Assistance funded partners, these were not a target audience mandated under the type of funding that PRO-WASH received. 2.2 Key Learning Deliverables While the PRO-WASH team had a broad focus on a range of learning and technical focus areas, particular emphasis was placed on the key deliverables described below, which were prioritized through a partner desk mapping and consultation process carried out early in the Award and through ongoing partner consultations/prioritization. Make Me a Change Agent Training Package | The Make Me a Change Agent (MMCA) training of trainers (ToT) package is a social and behavior change (SBC) approach that was co-created by PRO-WASH and the Strengthening Capacity in Agriculture, Livelihoods, and Environment (SCALE) Associate Award, and adapted from the Make Me a Change Agent: A Multisector SBC Resource for Community Workers and Field Staff Guide (produced in 2015 under the TOPS Program). Work on MMCA began early in the delivery of PRO-WASH and 2 In 2020, the Bureau for Humanitarian Assistance was established to streamline USAID humanitarian responses, bringing together the vast expertise and resources of the former USAID Offices of U.S. Foreign Disaster Assistance (OFDA) and Food for Peace (FFP) 3 The TOPS Program (2010-2018) was conceived to: facilitate knowledge exchange among practitioners in the food security community for two primary purposes: to improve the efficiency and effectiveness of U.S. Government-funded development food security activities, and to strengthen the knowledge, skills, and practices of the implementing partners (IPs) in delivering those activities to improve the lives of vulnerable families around the world. The summative TOPS evaluation report is available here: https://pdf.usaid.gov/pdf_docs/PA00W468.pdf 4 Program Description for WASH Associate Award Under TOPS, pp 1 5 This period includes start-up and close-out. Implementation will stop around November 30th 2022 for a 60 day close-out period, with many of PRO-WASH’s activities starting to wrap-up in October 2022 to allow enough time for this close-out. 4 was developed through a series of piloting and feedback processes from years 1-3. Learning briefs have been produced to share lessons from the development process. Market Based Sanitation Training Package | PRO-WASH contracted technical experts to develop a training module including a facilitator guide on market-based sanitation (MBS). The development process included stakeholder consultations with IPs, USAID/BHA staff, International Development Enterprises (iDE), FHI360 Wateraid, USAID WASHPALS, the Office of Water (E3), as well as webinars to discuss the materials in development. The materials were peer reviewed by a range of global experts. The development process concluded just as the COVID-19 outbreak commenced (see more below), resulting in field testing being delivered at a smaller scale in the Democratic Republic of Congo (DRC), Burkina Faso, and Niger. A special MBS working group was developed in Quarter 3 of Year 4 to support RFSA IPs in Niger. Human-Centered Design (HCD) | In addition to MMCA, human-centered design was an important component of supporting behavior change as part of PRO-WASH’s activities. HCD was included as part of the applied research activities in Bangladesh and Niger, and PRO-WASH and iDE co-led several online activities including a HCD learning journey in Year 3. In Year 4, PRO-WASH contracted iDE to undertake an intensive HCD process to develop an online WASH resource center to provide tailored tools, templates, and self-directed learning for IPs to use HCD techniques in their WASH work. The full resource center is anticipated to be available in Year 5. PRO￾WASH also began providing intensive one-on-one support to the Takunda RFSA in Zimbabwe on HCD for sanitation in Year 5. Water Quality Monitoring | Water quality monitoring was identified as a priority topic for PRO-WASH support during the desk mapping carried out in Years 1 and 2. PRO-WASH began work on this package but, due to contractor issues, this work was delayed until the end of Year 4. The water quality technical guide was disseminated early in Year 5 along with a draft training package. The final training package will be pilot tested in Zimbabwe in Year 5 and disseminated afterwards. WASH Systems Strengthening | Under the IWRM expansion of the PRO-WASH Award, PRO-WASH began in Year 4 to provide further support to partners for systems strengthening to sustain WASH services in rural areas. This included contracting Water for People along with its partner IRC to carry out four activity groups: 1) stakeholder engagement; 2) online introductory training on WASH systems strengthening for IPs in English and French, including live webinars and office hours; 3) selection of three IPs for support on lifecycle costing; and 4) implementation support that will respond to additional technical assistance needs during the implementation of tailored tools and approaches for IPs. Following this work, Water for People and IRC will share a debrief with participating USAID Missions to share opportunities and constraints related to utilizing tools and approaches for ongoing Awards and to inform future USAID/BHA efforts. Applied Research Deliverables | During the period under review, PRO-WASH supported applied research activities to support field-viable solutions to improve WASH activities across a broad geographic range. Applied Research Requests for Applications (RFAs) were devised based on IP consultation processes. Topics included: • Finding Solutions to Sanitation Challenges Facing Flood-Prone Populations in Bangladesh (completed) • Comprehensive Market Development Strategies Focusing on Availability and Provision of Improved Latrines and Hand-Washing Related Products in the Maradi and Zinder Regions of Niger (completed) • Pathogens Pathway Study in Kenya and Madagascar (ongoing) • Multi-Use Water Systems in Zimbabwe (ongoing) • Strengthening Sanitation and Handwashing within the Amhara and Oromia Regions of Ethiopia (ongoing) Webinars and Other Online Knowledge-Sharing Platforms | Whilst webinars and online sessions were included in the PRO-WASH design from its start, these knowledge-sharing mechanisms became fundamental following the COVID-19 outbreak and were consequently scaled up as a delivery approach. PRO-WASH conducted 33 webinars from 2018 to 2021, covering a range of topics such as gender and WASH, barrier analysis, market-based sanitation, a series on operation and maintenance, WASH systems strengthening, and more. 5 2.3 Engagement Intensity PRO-WASH clients, including BHA IPs, interacted with PRO-WASH to varying degrees of depth and intensity, as is discussed in Findings. Whilst some partners attended in-person training sessions, others may have followed the newsletter and accessed materials online, whilst others may have interacted with a range of PRO-WASH approaches to knowledge sharing and capacity strengthening, including one-on-one support (coaching) or applied research activities. This is considered and also utilized for sampling, as discussed below. 2.4 Steering Committee The PRO-WASH Steering Committee (SC) was established at the commencement of the Award, with the purpose of assisting the process of setting capacity-strengthening and research priorities. The SC had six U.S.-based members representing IPs and academic institutions. 2.5 Adaptive Program Delivery Processes PRO-WASH established monitoring processes based on pre- and post-tests involving training participants, end￾of-training evaluation forms, and post-training follow-up, to gauge measures of relevance, quality, and utility. PRO￾WASH reported several adaptations derived from knowledge generated through these processes. 6 3.0 EVALUATION SCOPE AND METHODOLOGY The evaluation employed both qualitative and quantitative methodologies to elicit feedback from a wide range of stakeholders, including IPs, PRO-WASH team members, colleagues from IDEAL and TOPS Associate Awards (including SCALE and Program Cycle Support), PRO-WASH Steering Committee members, USAID/BHA representatives, and USAID Mission Staff who engaged with the Award. The evaluation also reviewed key literature provided by the PRO-WASH team, such as learning products produced and disseminated, training reports, and internal monitoring data generated throughout delivery. 3.1 Primary Data Collection 3.1.1 Qualitative Data A total of 26 key informant interviews (KIIs) and focus group discussions (FGDs) were conducted by Causal Design with a total of 38 respondents. One additional interview was conducted by Bodhi Global Analytics, the consultancy firm undertaking the SCALE endline evaluation, for which interview notes were provided to Causal Design for inclusion in the analysis for this evaluation. Table 1 provides an overview of the qualitative respondents. Table 1 | Overview of Qualitative Respondents Respondent Group Total potential respondent groups Total KIIs/FGDs conducted with this group Proportion of potential respondents as per total engaged Total respondents engaged Notes IPs – High Engagement 15 11 73% 12 IPs – Medium Engagement 10 3 30% 3 One targeted respondent did not respond to requests to participate IPs – Low Engagement 11 3 27% 3 One targeted respondent declined to participate and was replaced with an alternative, identified using the same sampling methodology noted in Section 3.2 USAID BHA Staff engaged with PRO-WASH 1 1 100% 3 USAID Mission Staff 2 2 100% 4 One targeted respondent did not attend the scheduled Focus Group Discussion PRO-WASH Team members 1 1 100% 3 Scale Team Members 1 1 100% 3 PCS Team Members 1 1 100% 1 Steering Committee Members 3 Groups of 2 members (Total 6 respondents) 3 100% 5 One respondent was unable to participate in an interview but shared brief thoughts over email, which were included in the qualitative data set IP WASH HQ Technical Advisor 1 1 100% 1 Total 27 38 Of the low engagement IPs, one program was a BHA-funded emergency program. This respondent declined to participate, resulting in the presence of no emergency partners in the qualitative data set. 7 Table 2 presents the geographic coverage represented by the IP respondents included in the qualitative sample. Table 2 | Geographic Coverage represented by IP respondents in the qualitative sample Country Number of IPs included in the Qualitative Sample Malawi 1 Zimbabwe 1 Madagascar 2 Burkina Faso 1 Niger 3 Democratic Republic of Congo 2 Ethiopia 2 Kenya 1 Uganda 1 Haiti 1 Bangladesh 2 Total 17 Respondents were identified through a purposive sampling methodology outlined in section Sampling. Respondents were introduced to Causal Design by PRO-WASH and SCALE representatives. Causal Design then led the coordination of meeting dates and times, as per the availability of the respondents, which were undertaken remotely via Zoom or Google Meets by two interviewers, one male and one female who are based in USA and Malawi respectively and are nationals of USA and UK respectively. Both interviewers were English speakers. For French-speaking respondents, Causal Design employed the services of AIR Communication to provide simultaneous interpretation via Zoom. All interviews were recorded and transcribed using Otter.ai software. The KIIs and FGDs were undertaken with reference to the semi-structured interview guides provided in Annex 3, which were designed by Causal Design in response to the learning questions outlined in the ToR, before being reviewed and approved by the PRO-WASH team. KIIs and FGDs took place over a period of 60 – 90 minutes. The tools were purposefully sequenced to facilitate a natural flow to the conversations with neutral questions that allowed for open-ended responses and indicated key probes to elicit nuanced, thoughtful responses. The qualitative data form the foundation of the evaluation process, providing the opportunity to interrogate key issues relating to the internal and external factors that influence the achievement of PRO-WASH’s objectives. 3.1.2 Quantitative Data Quantitative data was collected to reach respondents who could not participate in the qualitative process due to time limitations. A 22-question quantitative survey instrument (Annex 4) was developed in collaboration with PRO￾WASH, SCALE and Bodhi Global Analysis, and respondents who were primarily drawn from a pool of BHA-funded partners supported by both Associate Awards were asked to engage in one coherent survey. This coordinated approach was designed to elicit a higher response rate and to avoid respondent fatigue. Skip logic guided respondents toward providing feedback only on the Award that they wished to comment on. The survey was launched through existing PRO-WASH networks comprised of newsletter subscribers, participants who had registered for online PRO-WASH events, training workshop participants, and key stakeholders including USAID Mission staff who engaged with the Award. Respondents were provided with a two-week window within which to submit their feedback. Quantitative survey questions were primarily close-ended and include Likert scales and ranking processes. The survey was designed to generate meaningful and helpful information. Every effort was made to keep the survey 8 concise and avoid asking for information, which would not be used in the final analysis. Bodhi Global Analysis translated the survey into French. A total of 178 respondents completed the PRO-WASH survey and were made up of the categories presented in Table 3. Table 3 | Quantitative Survey Respondents Organization type Frequency Percent BHA-funded RFSA 46 26% BHA-funded emergency program 13 7% USAID-funded Associate Award/Food Security Support Mechanism 11 6% USAID non-BHA staff 2 1% USAID BHA staff 4 2% Research/academic institution 9 5% National/local government 11 6% Local organization 18 10% International organization 54 30% Other 10 6% Total 178 Quantitative survey respondents categorized by position, country, or region are provided in Annex 5. Of the 178 survey respondents, 159 reported to have either received a PRO-WASH-authored learning document or technical tool, participated in a PRO-WASH-facilitated knowledge-sharing event, or participated in training or capacity-strengthening support. However, due to an error in the skip logic of the survey, only 65 of these respondents were directed to provide feedback on the quality, relevance, and utility of this support and PRO￾WASH’s outreach processes. Of these 65 respondents, 10 were BHA-funded activity representatives whose responses, given the small sample size, were generally analyzed together with the other 55 respondents. The exception to this is in the Support to BHA-funded Emergency Partners section below. The remaining 113 respondents were directed to only provide feedback on PRO-WASH’s outreach processes. 3.2 Sampling Qualitative Data | Respondents were purposely targeted across a range of partners, including those who engaged with high, medium, and low intensity,6 to elicit information relating to enablers and barriers to participation 6 Intensity of engagement was defined by PRO-WASH as below: None No memorable (from the perspective of the PRO-WASH team) engagements, or PRO-WASH had a one-off call with the individual/institution where PRO-WASH offered support or asked them to provide input into the desk mapping, but there was no further engagement (for example, the programme was closing out) Low Actor or institution attended a webinar, small group discussion or similar Medium Actor or institution participated in one-off training and received little to no follow-up support, or the actor or institution worked closely with PRO-WASH to co-write a learning brief/conference abstract. High Actor or institution participated in multiple training sessions, mentorship, or refreshers, applied research support, etc. For example, a partner had MBS training + coaching (ViMPlus), or training + applied research (Niger RFSAs or SHOUHARDO III 9 in PRO-WASH activities. Respondents were also selected to represent a broad geographic coverage, allowing for effective representation across participants and stakeholders. The evaluation conducted 27 KIIs or FGDs. The process through which target respondents were identified is provided in Annex 6. Quantitative Data | Respondents to the quantitative survey were self-selecting, on the basis of outreach through PRO-WASH networks. Response rates were routinely monitored while the survey was live, and follow-up reminders were sent to respondents from where response rates are especially low. 3.3 Analysis Qualitative Data | Causal Design utilized ATLAS.ti—its preferred Computer Assisted Qualitative Data Analysis (CAQDA) software—to code all interview transcripts, as the evaluation team wanted to connect responses with outcomes through a case study methodology. Coding was undertaken by two coders who had undertaken the interviews as described above. An a priori code book was devised with reference to the learning questions provided in the ToR. Four transcripts (15% of all transcripts) were coded by the two coders initially, through which additional codes were added to the data set according to the emerging key themes. All transcripts were then re￾coded with the final code book, which is provided in Annex 7. Quantitative Data | Causal Design triangulated the qualitative analysis with a descriptive analysis of online survey results using STATA statistical analysis software. 3.4 Evaluation Ethics Though risks of harm and perpetration of Sexual Exploitation, Abuse and Harassment (SEAH) are minimized by the nature of the online data collection process, Causal Design nevertheless applied its rigorous approach to evaluation ethics for this assignment. This approach is further outlined in Annex 8. 3.5 Data Storage, Privacy, and Security All qualitative data, including transcripts and audio backups, have been stored in a secure, password-protected, cloud-based server (Causal Design’s Google Drive). Only members of the evaluation team had access to this data set, and all data that is reported to stakeholders, funders, and implementers was de-identified to ensure Automated Directives Systems (ADS) Chapter 508 compliance with regard to personal identifiable information (PII). 3.6 Limitations Limitations to the methodology include, but are not limited to: Self-selection for the quantitative survey | Given that respondents self-selected to respond to the quantitative survey, data gathered through this process is likely to be skewed toward those who had a higher level of engagement, or who had strongly held views (both positive and negative) regarding the technical support PRO￾WASH provided. Since the survey utilized a “convenience sampling” approach, the data, while salient with respect to the learning questions, will not be representative in any statistical sense. Potential for recall bias and limitations in respondents’ knowledge | Given the nature of the independent evaluation as end-line only and taking place in the final year of the Award, as opposed to an evaluation methodology that gathers independent baseline values that endline values may be compared to, staff turnover and recall bias are likely to influence the information gathered, the findings, and the conclusions derived. In some cases, those interviewed were unaware of the full history of the RFSA’s relationship with PRO-WASH and were unable to speak definitively about how initial contact was made between the RFSA and the Award. Suggestive, rather than statistically representative, findings | Given the small sample size and emphasis on qualitative data, the findings presented in this report are not statistically representative. As quantitative findings reflect a small number of respondents, these findings have been included given the illustrative value of the data. 10 Translation | Causal Design used professional translation services and complete transcripts for analysis, but nevertheless, there remains some risk that some nuance or subtleties were overlooked. Causal Design considers this to have been a very minimal risk to data quality. Concurrence with the SCALE evaluation | It is possible that respondents, both quantitative and qualitative, may have been influenced by the SCALE evaluation, which was taking place concurrently with the PRO-WASH evaluation, and provided information that would be more relevant to SCALE than PRO-WASH. This risk was mitigated through qualitative interviews through probing and clarifying questions, and in the quantitative survey through repeated reference to PRO-WASH. However, the risk could not be entirely removed given the joint nature of support some respondents received from both Awards. 11 4.0 FINDINGS This section outlines the key findings of the endline evaluation presented in accordance with the learning questions provided in the ToR. Overall, the data clearly and consistently present a remarkably positive image of the PRO-WASH Associate Award. Qualitative respondents were asked to describe PRO-WASH in ‘three words or phrases.’ Eighty words were gathered and presented as a word cloud in Figure 1, which highlights the key perceptions of PRO-WASH as collaborative, supportive, and technically sound. Implementing partners, Associate Award representatives, Steering Committee members, and USAID representatives invariably provided positive descriptors of PRO￾WASH, while only PRO-WASH team representatives noted more constructive terms such as ‘lack of clarity’ and ‘busy.’ Figure 1 | 'Please describe PRO-WASH in three words or phrases' 4.1 Award Design and Objectives This section explores the technical design of the Award by USAID, and the delivery and management of the Award by Save the Children US. While the relevance of PRO-WASH’s technical focus is discussed in detail below (see Implementing Partner Engagement and Impacts of Technical Assistance and Influence), this section focuses on the Award’s responsiveness and flexibility to changing needs. Also explored are issues relating to organizational and staffing structure, the Steering Committee, and other factors that influenced the extent to which the Award met its objectives. Learning Question | How did the technical design of the Award and the level of flexibility affect the ability to respond to IP and donor needs, collaborate with partners, and adapt to external circumstances (e.g., COVID-19 pandemic, Food for Peace/Office of Foreign Disaster Assistance merger)? 4.1.1 IPs experience with PRO-WASH’s responsiveness and flexibility PRO-WASH was widely described as responsive and flexible by IPs and Stakeholders. Forty-five survey respondents (almost 70%) noted that their technical assistance needs changed over the period of PRO-WASH’s implementation; a finding generally echoed in the qualitative data. 12 Of the survey respondents who reported that their WASH technical assistance needs did change, 64% (N=29) reported that PRO-WASH responded to their change in needs ‘a lot’ or ‘very much,’ while 20% (N=9) reported that PRO-WASH responded to their change in needs ‘somewhat.’ Survey respondents were then asked to identify three factors that they felt most influenced the extent to which PRO-WASH responded to their changed needs. These responses are presented in Table 4. There is no extensive difference in the factors that were selected by respondents who rated PRO-WASH as ‘somewhat’ responding to their change in needs compared to those who rated PRO-WASH as responding ‘a lot’ or ‘very much’ to their needs. Therefore, key factors influencing this rating are unclear. Table 4 | ‘What Three (3) Factors Most Influenced the Extent to Which PRO-WASH Responded to Your Changed Needs?’ Extent to which PRO-WASH responded to the change in needs PRO-WASH staffing PRO-WASH feedback systems PRO-WASH technical focus areas PRO-WASH approaches PRO-WASH duration PRO-WASH timing PRO-WASH geographic scope Title II/CDF funding focus Somewhat (N = 9) 11% 33% 56% 56% 11% A lot (N = 25) Very Much (N = 4) 21% 36% 75% 79% 18% 4% 4% 18% Qualitative data further suggest an overall positive experience, as all IP respondents noted that PRO-WASH adapted their delivery approach as required. It was generally noted that PRO-WASH became aware of IPs’ changing needs through their continuous engagement, referred to as ‘coaching’ by some respondents. Examples of PRO-WASH’s flexibility cited as by respondents included a pivot to online training sessions due to COVID-19 travel restrictions, adaptations to Request for Proposal (RFP) processes for applied research due to a lack of submissions, and pivoting session plans to respond more directly to the experiences and challenges the participants faced. “We have worked with, like collaborated with, many other USAID funded projects, but they were most responsive and there was not a single time when we didn’t get a response from them. They were very proactive and responsive.” – IP respondent Learning Question | What have been the favorable or impeding factors related to the design and management of the Award that have affected its ability to meet its objectives? 4.1.2 Flexibility From the Perspective of the PRO-WASH Team The level of flexibility built into the Award by virtue of USAID’s design was widely noted as a key enabling factor in PRO-WASH’s receptiveness to IP needs. USAID did not specify the WASH technical focus areas to be addressed, giving the PRO-WASH team the opportunity to listen to and respond to IPs developing the PRO￾WASH annual plans and budgets. While this level of flexibility was generally appreciated, it did create the challenges outlined below. Difficult to Design For When submitting the bid for the Award, Save the Children US had little technical direction upon which to draw from in designing and budgeting the PRO-WASH team and mechanisms. Consequently, the data suggest that PRO-WASH was under-staffed, creating excessive burden on the team to deliver. Save the Children US does not 13 seem to have raised this issue with USAID during delivery, though the reason for this was not fully explored. From the perspective of IP respondents, this issue manifested as an over-reliance on one or two staff members as focal people for all IPs, though it does not appear to have had an impact on IPs’ perceptions of PRO-WASH’s responsiveness, as noted above. This issue is discussed further below (see Staffing Structure). “For me, PRO-WASH is Nicole and Nicole is PRO-WASH” – IP Respondent Risk of Mission Drift Given the expansive nature of WASH as a technical field, which can range from social and behavior change to water quality and water resource management, the lack of technical direction outlined in the design of PRO-WASH reportedly created the opportunity for mission drift dependent on the interests of USAID/BHA at any given time. This is an idiosyncratic finding, referenced specifically by respondents internal to PRO-WASH. No references to mission drift were made by IP respondents. Whilst there is no evidence to suggest that PRO-WASH deviated from the agreed-upon work plan, one respondent noted that PRO-WASH was willing to go beyond their mandate if budget and demand existed, as demonstrated in the quotation below. “I’m thinking back to conversations with Nicole in years two and three of PRO-WASH but then even things that weren’t in the work plan, if there was budget for it and a demand from a partner, they’re willing to meet that need or request… Right now, we’re in discussions with… one of the Zimbabwe programs about… helping them use the MMCA learning course and Nicola said, ‘You know, we don’t have this in our work plan. We don’t need to do it to meet our indicators or like what we’re obligated to do from BHA but like, let’s make it work for this partner because we have the budget and they’re interested, and they want this’” – Associate Award Respondent Approvals Process One respondent internal to PRO-WASH noted that PRO-WASH’s requirement to get AOR approval for ‘day to day programmatic things’ hindered its ability to respond to IP requests. The data suggest a divergence on this issue between IP perspectives and those coming from respondents internal to PRO-WASH. While IPs reported high satisfaction with PRO-WASH’s flexibility, which, in some cases, was ascribed to PRO-WASH’s relationship with USAID, respondents internal to PRO-WASH noted that their contractual arrangement with USAID slowed their ability to respond to changing needs and circumstances. An example is provided in the quotation below. “If you want to hire a trainer for a five-day training… the amount of time it would take us to get BHA approval to have someone for less than $5,000 do this training…, what are we going to do? It’s just tough because of every little thing like that.” – Respondent internal to PRO-WASH 4.1.3 Demand-Driven Support It is widely recognized that PRO-WASH was designed to be, and was delivered as, a demand-driven technical assistance facility. For IPs, this design function appears to have lent itself to a sense of ownership and buy-in, which the data suggest is related to the willingness to apply learning. Further points related to the demand-driven design are shared below. 14 Ambiguity Over PRO-WASH’s Mandate While there is consensus that the demand-driven aspect of the design was appropriate, one respondent did note the limitations that this design function placed on PRO-WASH’s ability to influence improvements in IP WASH programming, as articulated in the quotation below. “There is a lot of push for [PRO-WASH] to have a sustainable impact on partners’ activities. But there’s also a limitation because [PRO-WASH] is not managing a RFSA… so [PRO￾WASH] can provide the most accurate technical advice and training… whether or not [the RFSAs] actually apply the skills and have the resources to do it, or the desire is up to [the RFSA]. That’s out of [PRO-WASH’s] control. If [PRO-WASH’s] support is not required by the partners, partners don’t have to follow [PRO-WASH’s] guidance.” – Respondent internal to PRO-WASH References such as the one noted above suggest that at times, there has been a disconnect between USAID’s expectations of PRO-WASH’s mandate to improve RFSA performance in WASH, and the limitations in PRO￾WASH’s ability to affect such improvement. Conversely, USAID/BHA respondents noted a recognition of the limitations placed on PRO-WASH by virtue of the demand-driven design. In reflecting that PRO-WASH is only able to invest its time and support where it is requested, one respondent questioned, “are the resources going to the RFSA partners that need the most help or just the ones that are aware and able to access that help?” In discussing this issue, respondents internal to PRO-WASH noted that the Award should not become a technical approval or accountability mechanism, as this will likely diminish the open relationship PRO-WASH has experienced with IPs in the first phase of the Award. Push and Pull While respondents consistently referred to PRO-WASH as being led by partner demand, an alternative approach was also noted—that of PRO-WASH as informing partner demand by highlighting the global evidence base on what works and sharing knowledge amongst RFSAs as discussed further below (see The Added Value of a WASH-focused Associate Award). Examples of the PRO-WASH guiding partner demand include integrating behavioral science into WASH and moving beyond community-managed WASH systems to more strategic WASH Systems Strengthening. Key terms used to describe PRO-WASH are “leading edge” and “transformative.” 4.1.4 Factors Influencing the Uptake of Technical Assistance Being a demand-driven technical assistance facility, the factors influencing the uptake of IP technical support are fundamental to understanding the Award’s achievements. These factors are discussed below. Lack of Knowledge on How to Access Support Of the eighty (80) survey respondents who reported to have engaged with PRO-WASH support relatively infrequently,7 44 respondents (55%) attributed their infrequent engagement to their lack of knowledge of how to access support from PRO-WASH. Of these 44 respondents, only 9% (N=4) were attached to BHA-funded RFSAs, suggesting that a lack of knowledge on how to access support was not a significant barrier for respondents who were attached to RFSAs. One anecdote in the qualitative data provides an example of lacking knowledge of the support PRO-WASH can provide. When discussing an emergency within the country where a respondent’s RFSA is operating, the respondent noted that they had not discussed the WASH infrastructure issues emerging from the humanitarian situation with PRO-WASH, as they were “not sure if we can bring that aspect of wash infrastructure challenges to that platform.” IP Perception of Relevance and Quality 7 Relatively infrequent engagement was classified as any respondent to who reported to have engaged with PRO-WASH only once, approximately once per year, or approximately once every 6 months. 15 Where IPs reported a proactive uptake of PRO-WASH’s support, the respondent often associated this by characterizing the support as high quality and relevant to their work, as demonstrated by this quotation from an IP respondent. “If you are asking for technical support, it has to come from the best and those that have given the confidence in terms of we know what do we want to deliver and why we want to do it… because I’m also learning from that and building my own confidence… without that perhaps you will see it wouldn’t be easy to move the agenda and engage as and when you will not have the best from somebody... you will perhaps be the last person to get into a conversation [regarding technical assistance needs] that you know it will not yield results” – IP respondent This notion is supported by survey data. Of the 80 survey respondents who reported to have engaged with PRO￾WASH support infrequently, only 8% (N=6) and 6% (N=5) attributed their infrequent engagement to the technical focal areas holding little value to them, and poor quality of support. Time Availability Within the IP There was a consistent recognition of the time demands for IPs to derive the maximum value from PRO-WASH support. Some qualitative respondents noted a lack of time to attend webinars, with two noting that this was particularly problematic when times were set without consulting IPs’ availability. Some qualitative respondents noted that webinar recordings were shared, and that they did occasionally watch the recordings, though not all respondents were asked this specific question. Other qualitative respondents noted the challenges of scheduling training sessions when a program had entered its implementation phase. Amongst quantitative respondents, 12 respondents who were attached to RFSAs reported ‘relatively infrequent’8 engagement with PRO-WASH. For these 12 respondents, ‘lack of time to engage or access support’ was the most frequently selected reason to explain their infrequent engagement, with 7 respondents (58%) selecting this option. Amongst qualitative respondents, this was generally related to the fast-paced and often unpredictable nature of their work in delivering RFSAs, and the timing of the commencement of their partnership with PRO-WASH as discussed further below (Impact of timing within the RFSA cycle). Personalities Within the IP A key thread emerging from the qualitative data was the integral influence of individual IP team members in determining whether the RFSA would engage in PRO-WASH support. Numerous IP respondents discussed their own values, working styles, and ambitions as key factors in determining engagement, as exemplified in these quotations. This is likely related to the design of PRO-WASH to be a demand-driven facility, as noted above. “I think it’s coming from me because I really forced my team to continue learning because learning is part of the attitude that each development worker should have, otherwise we just [implement] the same thing over and over again… it’s not only about following webinars, exchanging and then don’t think about how do I apply it” – IP respondent “I’m a bit ambitious and always want to test new things, new ideas, new challenges. That is why I’m using all the knowledge and skills that I’ve learned from PRO-WASH” – IP respondent 8 As per footnote 7 16 Team Dynamics Relatedly, team dynamics within the RFSAs emerged as an important factor influencing uptake. One respondent discussed the challenges of needing to seek Chief of Party (CoP) approval for all decision-making related to PRO￾WASH support, which caused a bottle-neck and ultimately contributed toward low engagement on behalf of that RFSA. Another respondent discussed the need for a culture of learning within the team, noting that fast-paced working can create an environment where team members do not take the time to reflect on learning. Perception of PRO-WASH as ‘Save the Children’ Though somewhat of an outlier, one issue apparent in the data set relates to the political economy of International Aid Agencies. In discussing an inherent challenge faced by IPs delivering technical assistance awards, one IP respondent noted that in the delivery of PRO-WASH, Save the Children is required to “be convening, collaborating with all the other implementers who are also their kind of competition and their peers.” Another IP respondent attributed their low engagement to the fact that PRO-WASH was delivered by an agency that they did not belong to, as noted in the quotation below. IP Respondent “It is hard to be as free as possible to somebody in a different organization so that alone has somehow stopped us in the smooth flow of communication or request, as and when it should have been.” Interviewer “So do you feel that if PRO-WASH was being delivered by [respondent’s organization] would you have engaged with it differently? IP Respondent “I think yes because I had many engagements with our colleagues in the [IP HQ country] and we look at some of the technical areas that PRO-WASH would have supported.” Of the 80 survey respondents who reported to have engaged with PRO-WASH support infrequently, 4 respondents (6%) ascribed their low engagement to a feeling of discomfort with the PRO-WASH staff. This issue could not be further explored due to the nature of the survey, so the underlying cause of this discomfort is unclear. Given the wider feedback noted below (see Staffing), it is considered unlikely that this discomfort relates to individual staff members. It is plausible to suggest that the discomfort may relate to the political economy issue of International Aid Agencies, outlined herein. Digital Connectivity Another notable issue limiting engagement related to digital connectivity challenges, particularly facing field staff. Where training sessions were delivered online primarily due to COVID-19 travel restrictions, there were reports from qualitative respondents that field staff were distracted by their delivery responsibilities, which may have been avoided through in-person training. Lack of internet and lack of computers was also cited as a barrier to team members attending webinars beyond those in Head Offices. Language Barriers Some respondents cited language barriers and the perceived over-reliance on English over French as a cause of low engagement, particularly for consultant engagement and webinars. This finding was variable amongst qualitative respondents, with some noting that translation had been helpfully provided, whilst others noted that it had not been. Of the 80 survey respondents who reported to have engaged with PRO-WASH support infrequently, 11% (N=7) noted that language barriers were a factor preventing them from engaging more frequently. Language is discussed further below (see IP Perception of Utility), as data relate to the impact of language barriers on the perceived utility of tools. Role of USAID in Influencing Uptake 17 One IP respondent noted the strong influence USAID/BHA had in generating demand for technical assistance from IPs, stating “there was the interest in [the IP] to have a study that was supported by PRO-WASH because it’s very encouraged by USAID. Honestly. USAID is very, very happy when we’re taking advantage of these associate awards’ technical support”. Conversely, a USAID/BHA respondent shared a reflection of their limited scope of influencing IPs, stating “it’s almost easier to have it be directly from PRO-WASH to the partner and sidestep [USAID/BHA] because if the BHA overlords get in the way, it sometimes can make it worse… we can’t browbeat them into like wanting to sign up for a workshop”. The role of USAID Missions in this dynamic appears to be variable, given the varying relationships Missions have with RFSAs. Anecdotes in the data provide examples of one Mission participating in the development of Scopes of Work (SoWs) for PRO-WASH’s engagement with a RFSA, while another Mission was unclear on both the genesis and outcome of PRO-WASH’s support for RFSAs in their country. PRO-WASH’s direct engagement with Missions also appears to be changeable. Learning Question | How appropriate and effective is PRO-WASH’s organizational and staffing structure for the implementation of our work? 4.1.5 Staffing The findings suggest that while the staffing structure placed limitations on the extent to which PRO-WASH met its expectations, the individual staff members themselves were huge contributors to PRO-WASH’s perceived success. Staff Competencies The role that individuals involved in PRO-WASH had in its success cannot be understated. Data suggest that not only were team members technically expert in adult education, well-networked, and experienced in providing technical expertise as needed, they were also able to navigate a tricky arena, creating space for IPs to take ownership of their RFSA WASH capacity building plans, while also meeting the expectations of IP HQ WASH Advisors and USAID/BHA. Data demonstrate that this was achieved due to the team’s diplomacy, organizational, time-management and problem-solving skills, as well as deeply held personal beliefs about the value of partnership and collaboration. Data demonstrate that PRO-WASH would not have been successful without the people who delivered it. “They are true technical assistance. The mission, the vision, the willingness to assist you to help you grow and improve your program.” – IP respondent Staffing Structure As noted above, data suggest that the team was under-resourced to deliver against its objectives, as well as the changing circumstances facing RFSAs and key stakeholders’ changing expectations. This reportedly resulted in unsustainably heavy workloads experienced by PRO-WASH team members. Key roles that were not included in the PRO-WASH team design, but that are reported to be important, are: • Knowledge management • Adult education • Communications, including print and digital media • Technical specialists as determined by the priority focal areas going forward In line with the above finding relating to staff competencies, the data suggest that these skills were available within the PRO-WASH team, though the absence of these specific technical roles resulted in burdening other team members, including the Director, to deliver in these areas, which resulted in unsustainably heavy workloads and possible detractions from other deliverables, such as influencing BHA (as discussed further under section Thought Leadership). 18 4.1.6 Working with Consultants Given the expanse of the WASH technical field and the consequent diverse range of technical focal areas prioritized by PRO-WASH, the Award worked with a range of consultants to deliver key pieces of technical work. Anecdotes within the data, though not necessarily typical, are illustrative of the potential issues this working style raises, as outlined below. Risks of Poor-quality Deliverables In one reported case, PRO-WASH was required to remove and replace a consultant due to the poor quality of their work. Changes in Consultants Causing Delays In other reported cases, consultants became unavailable to undertake PRO-WASH assignments, again leading to the need to find replacement consultants. One set of respondents noted that this situation led to delays and the need for RFSAs to adapt to new working styles. The Aid Localization Agenda Data suggest that PRO-WASH typically engaged ‘international’ consultants whose technical specialties were prioritized over their contextual understanding and experience. Some respondents noted an appreciation of the ‘new’ ideas that ‘international’ consultants may bring, particularly in a circumstance where a specific specialism is not widely available within a country. Conversely, other respondents discussed cases in which the lack of contextual understanding led to broad recommendations, which affected the utility of the deliverable (see IP Perception of Utility for further discussion). The discussion emerging from the data is relevant to USAID’s Commitment to Advancing Localization,9 as discussed below (see Conclusions). 4.1.7 Inconsistency of Feedback Loops Though the general trend in the data portrays PRO-WASH as open and receptive to continuous dialogue with IPs, and proactive in seeking feedback on training workshops, one anecdote suggests that the absence of a specific platform for providing more critical feedback relating to consultant support may have prevented IPs from sharing challenges. One respondent noted their dissatisfaction with the work delivered by a consultant, which was considered unfit for their context. When asked if they had shared their concern with PRO-WASH directly, the respondent noted that they had informed the consultant but not PRO-WASH, as “there was no platform to give feedback.” Learning Question | How effective was the design, composition, and management of the PRO-WASH Steering Committee? How could this be improved? 4.1.8 Steering Committee The PRO-WASH Steering Committee (SC) appears to have been a helpful and supportive mechanism for the technical direction and review of deliverables produced by the Award. SC members were generally satisfied with their engagement in the committee and appreciative of PRO-WASH’s approach to collaboration. Respondents noted key strengths of the SC such as: • Highlighting existing resources • Providing a ‘soundboard’ for PRO-WASH, sharing ideas and feedback in specific technical areas • Expanding PRO-WASH’s network of partners or experts • Providing technical review and input into specific deliverables Other issues relating to the SC that emerge from the data are detailed below. Lack of Clarity on the Purpose of the SC 9 See https://www.workwithusaid.org/blog/usaid-s-commitment-to-advancing-localization, accessed on 11th August 2022. 19 The data suggest a lack of clarity among respondents regarding the purpose of the SC. Steering Committee members themselves reported a shift in SC focus following the initial stages of PRO-WASH set up, which raised unanswered questions about the purpose of the group, as illustrated in the quotation below. Other stakeholders noted a divergence in expectations of the SC amongst key stakeholders. “The nature of the Steering Committee and how it interacted with the PRO-WASH staff changed as their focus changed. And there was one point I think, Nicole Weber had reached out to the committee and said ‘Hey, is it really useful for us to have these regularly scheduled Steering Committee meetings? Or would it be better if we’re just interacting directly individually, or only scheduling meetings when there’s really a need for them?’ That was a little awkward, as far as you know, trying to figure out what’s the real purpose of the Steering Committee if we’re not really meeting on these things, but I did definitely see an increase in the direct interaction individually with PRO-WASH…It did signal a change in how the committee would interact with PRO-WASH staff and what the focus would be going forward. So again, you know, as I mentioned earlier, early on the Steering Committee was much more engaged in trying to lay out exactly how the PRO-WASH Award was going to be structured and what it would specifically be responding to and around that time [meaning the time the question was raised regarding interaction between PRO-WASH and the Steering Committee] would be where it became less and less relevant. Probably a better word than less relevant, but it was not as critical to PRO-WASH moving forward” – Steering Committee member respondent Limited Knowledge of SC Members Regarding PRO-WASH Management and Delivery Steering Committee members consistently referenced limitations in their knowledge relating to PRO-WASH management and delivery, including the PRO-WASH staffing structure, deliverables achieved, feedback from support recipients, and use of tools. Whilst Steering Committee member respondents did not attribute their limited knowledge to any specific cause, it is plausible to suggest that this may relate to the lack of clarity on the goal of the SC, which may have influenced the type of information shared with the SC. “It’s hard because, you know, as someone on the Steering Committee, I’m not that involved with what PRO-WASH has done and actually I’m not sure how helpful I’m going to be [for the purpose of the evaluation].” “I’m not that familiar with the staffing structure.” Respondent 1 “I’m not sure the extent to which they did much, or enough, on water quality and water quality testing. Respondent 2 “Yeah, good point…I’m not super aware of that dimension either.” - Extracts of comments made by Steering Committee Members Limited Interaction Between the SC and USAID Data also suggest limited engagement between the SC and USAID/BHA. While SC members typically note that they had anticipated greater engagement with USAID/BHA, one USAID/BHA respondent described an intention to step back from the SC to create space for more honest discussion. Again, it is plausible to suggest that these diverging expectations may be related to the lack of clarity on the purpose of the SC. 20 Lack of Clarity and Consistency on Time Expectations Respondents generally noted that PRO-WASH drew on some SC members more than others, which was typically attributed to individual members’ availability and interest. Some respondents noted that expectations on time requirements had been unclear when they joined the SC. Composition of SC Membership Data show a divergence in views related to the suitability of the SC composition. Where some respondents reflected that the membership may not have been wholly fit for purpose, this was linked to the lack of clarity over the goal of the SC. Some members, for example, were felt to have limited experience and understanding of the constructs in which PRO-WASH operates. Moreover, and relevant for USAID’s Commitment to Advancing Localization referenced above, it was noted that the SC membership includes no representatives of organizations which are local to the countries in which RFSAs are delivered. Overlaps Between SC Membership and PRO-WASH Contractors One respondent raised a concern over the ‘murky ground’ created by an overlap between some members’ roles in the SC and, later on, their roles as bidders competing for work funded by PRO-WASH. Noted were challenges in navigating interests when SC members had bid for RFPs released by PRO-WASH, as well as concerns that SC members may have been advantaged by their prior knowledge of an RFP before it was released publicly. Ideas Shared for Strengthening SC Mechanisms Respondents shared the below suggestions to improve the functioning of the SC: • PRO-WASH should clarify and build consensus on the purpose of the SC. SC meetings could begin with a review and re-affirmation of the purpose of the committee, and the purpose of the discussion • PRO-WASH could create a framework for the SC, which could outline expectations and enable a mechanism for SC reflection concerning which of those expectations are being met • PRO-WASH could create more opportunities for SC members to connect with each other and share experiences. A greater understanding of the background of SC colleagues may generate more discussion and collaboration to PRO-WASH’s benefit Learning Question | How did the design, staffing, and management of the Award affect its ability to engage with and meet the needs of BHA-funded emergency partners? 4.1.9 Support to BHA-funded Emergency Partners As noted above (see Methodology), the qualitative sample did not include any BHA-funded emergency partners due to limitations in the ability to reach out to such respondents. Of the 65 survey respondents who provided feedback on PRO-WASH support, 10 identified as BHA-funded emergency respondents. Given the small respondent number, any findings specific to this group should be interpreted as suggestive and not conclusive. A comparison of responses from BHA-funded Emergency Partners and RFSA Partners was undertaken for key indicators, the findings of which are presented in Figure 2. Given the limited sample size, however, clear conclusions cannot be drawn in this regard. 21 Figure 2 | PRO-WASH Support to BHA-funded Emergency Partners and RFSA Partners Limitations in Scope to Support Emergency Partners due to Funding Streams PRO-WASH was limited in its ability to support certain types of programs, particularly emergency partners, given restrictions associated with the funding streams used to invest in PRO-WASH. Data suggest that these funding restrictions were not immediately understood by key stakeholders, particularly following the BHA merger, which initially created expectations that PRO-WASH should support emergency partners, and later on, the realization that direct support would not be possible. 4.2 Implementing Partner Engagement and Impacts of Technical Assistance and Influence This section explores IPs’ perceptions relating to the relevance, quality, and utility of the technical support provided by PRO-WASH. Issues explored include PRO-WASH’s response to technical needs, reflections on the approaches and methodologies used to provide technical support, IPs’ reports of using the learning, resources, and tools gained from PRO-WASH, their plans for continued use and the extent to which they institutionalized their learning beyond the project they were implementing. As noted above (see Methodology), survey responses from BHA-funded Emergency partner respondents have been combined with those from RFSA respondents to provide the survey findings presented in this section, given the small sample size. Learning Question | How do IPs perceive the relevance, utility, and quality of the different types of support offered by PRO-WASH (e.g., applied research, technical guides, learning briefs, coaching, remote and in-person training, technical knowledge-sharing events, webinars, and communities of practice)? 4.2.1 The Added Value of PRO-WASH Respondents consistently noted the added value that PRO-WASH provided to RFSA programming, above and beyond that which IPs could do independently. Reflections on added value were particularly noted by IPs who were categorized as ‘high engagement’ (see above Sampling). Learning Question | How has PRO-WASH contributed to WASH/IWRM knowledge-sharing and understanding of best practices and promising/innovative models for WASH service delivery among Resilience Food Security Activities (RFSAs)? Networking Among RFSAs 30% 30% 40% 25% 50% 25% 20% 80% 2% 7% 9% 59% 23% 3% 10% 13% 60% 13% 14% 59% 27% 0% 20% 40% 60% 80% 100% 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 Rating of the extent to which PRO-WASH responded to needs Rating of extent to which PRO-WASH responded to a change in needs Rating of overall quality of support Emergency Partner respondents RFSA respondents 22 A key thread emerging from the data is the recognition of PRO-WASH as a convener of RSFAs within and across countries - a function typically noted to be beyond the faculty of individual IPs. Related to the political economy of International Aid Agencies noted above (see Perception of PRO-WASH as ‘Save the Children’), respondents discussed the challenges of IPs and RFSAs often operating in silos. Data suggest that PRO-WASH has had some success as an emissary between agencies, sometimes directly facilitated joint work between RFSAs, such as the example of Case Study 2 in Niger (Annex 9). Profiling RFSAs’ Work Relatedly, numerous examples were cited where PRO-WASH provided a catalytic platform for profiling RFSAs’ work. Of the 65 survey respondents who engaged with PRO-WASH support, 52% (N=34) reported that PRO￾WASH provided them with an opportunity to share best or promising practices or innovations. Examples of this profiling include helping RFSAs present within PRO-WASH-hosted webinars and knowledge-sharing events, inviting RFSA WASH Leads to co-facilitate online training sessions, supporting Technical Brief writing, and submitting abstracts to international conferences. PRO-WASH’s Networks and Reputation The breadth of expertise available through PRO-WASH, possibly due to the Award’s network of prominent researchers and practitioners as well as PRO-WASH’s notable reputation, was also consistently cited by respondents as a quality unique to the Award, which IPs alone often cannot provide. Given the expanse of WASH as a technical field, respondents noted that IPs are often unable to maintain WASH technical teams comprised of specialists in all relevant subjects, as this would be too burdensome on overhead costs. As one respondent stated, “there’s some exceptional expertise that I should acknowledge… really getting some level of specialty that in my course of working with [the IP] I have not engaged with.” Technical Advisory Service Delivery Expertise Data suggest that PRO-WASH, as a unit commissioning and delivering technical advisory services as its core mandate, may have benefited from practice-based learning relating to designing SoWs and tendering RFPs, which was then passed onto IPs. Some respondents reported to have been fully supported throughout these challenging processes, particularly where, as practitioners, they had no previous experience. Though the culmination and benefit of this experiential learning was assumed by one respondent, it is plausible to suggest that PRO-WASH would generate substantive practice-based learning related to the technical advisory service delivery, which could be beneficial for RFSAs. “Definitely the experience [PRO-WASH has] had with other similar Pathogens Studies and other contexts [were factors that enabled them to support the IP beyond what the IP could do independently]. And how those findings have been used as well. I think [PRO-WASH] having seen the whole process through, like scoping evaluations, working with research partners, to implement the evaluations and then actually having the findings of the evaluations be utilized by implementing partners, having seen that whole process through gave [PRO-WASH] a much clearer sense of what the feasibility to impact balance would be to help us go. And then [PRO-WASH’s] relationships with these research institutions, but also specific researchers who do this type of work to be able to reach out to [PRO-WASH] directly, either [PRO-WASH has] engaged with [the researchers] or they kind of are within the same circles, so they already had those relationships. [PRO-WASH was] kind of a known entity also to those researchers, which probably made the researchers more interested themselves.” – IP respondent Staying Abreast of Emerging Learning One insightful reflection highlighted the value of having a trusted partner who is staying abreast of emerging learning and innovations in the field and encouraging RFSAs to create space for critical reflection. The respondent stated, “things are evolving... When you are implementing you are deeply inside, you are like inside a water, you swim inside it. You will not see what's going on in the beach. So, you need someone to keep looking at the beach. 23 Are there new boats coming or the new fish coming in?... Would you like to get out of the water let's sit about talk about it and think about it.” Continuity of the Relationship Some respondents noted the value of having a continuous relationship with PRO-WASH, compared to alternative capacity-building opportunities that tend to be one-time training workshops or seminars. One respondent noted, “If you want to prioritize all those webinars that are organized by World Bank or WSP, it’s just a one-time but if you go with PRO-WASH it becomes a long-term relationship”. Positional Power to Influence As an associate Award being promoted by USAID, PRO-WASH was in a unique position to draw attention to WASH, which is reportedly often deprioritized within the wide spectrum of issues addressed by RFSAs. WASH Lead respondents noted the value of having an influential ally to raise the profile of WASH in the consideration of RFSA decision-makers, and the impact this had on their own confidence and motivation. “[PRO-WASH] has also helped me quite a lot in terms of having the confidence to profile WASH in contexts and settings where WASH is not popular. Believe you me, in some other context WASH is the final thing that people talk about… Your WASH is more of an underdog… But interfacing with PRO-WASH who would actually assure you that WASH is something; WASH is a subject; WASH means something; WASH will add value and change lives. You refine and renew your commitment to your subject area, your program area, and say no, I think I know I'm in the right sector and I will be able to drive with the strength and confidence that it requires because it matters.” – IP Respondent “[PRO-WASH] was incredibly helpful for me. I believe that it has been very helpful for WASH Advisors within the RFSAs because they have kind of an advocate. They have someone else who is saying ‘hey, look, this is somebody who is kind of being brought out here by USAID to coach us, to guide us and they’re kind of pushing us in this direction. We should listen to them”. – IP Respondent Learning Question | To what extent do implementing partners perceive that PRO-WASH tailored priorities to their requests/needs? 4.2.2 IP Perception of Relevance Respondents largely noted that PRO-WASH responded to their needs for technical assistance and provided relevant support. Of the 65 survey respondents who had engaged with PRO-WASH support, 72.3% (N=47) reported that PRO-WASH responded to their needs ‘a lot’ or ‘very much,’ though 15.4% (N=10) noted that PRO￾WASH had responded to their needs ‘a little’, or ‘not at all. As noted above (see Factors Influencing Uptake of Technical Assistance) IP perception of relevance appears to be a key driver of uptake of support. Tailoring Focal Area Priorities to IP Needs Of the 74 survey respondents who reported that PRO-WASH had reached out to them to ask for their views on technical area priorities, 88% (N=65) reported that this outreach process was ‘effective’ or ‘very effective’ in the extent to which they felt their input would contribute toward appropriate technical support for them and their program. Where IP respondents noted that they had developed a WASH capacity-building plan or SoW, they typically consider this process to have ensured that PRO-WASH tailored the focal areas to their needs. Learning Question | What remaining gaps/existing challenges have not been met through PRO-WASH learning and research? Unmet Need 24 Where IP respondents noted that some of their needs remained unmet, two primary causes emerged: the application of learning has not yet been undertaken because support is ongoing, and the utility of learning is limited due to a lack of contextual adaptation (see IP Perception of Utility for further discussion in this regard). Table 5 presents specific topics that were noted as remaining or emerging gaps for technical assistance. Table 5 | Remaining or Emerging Gaps Requiring Further Technical Assistance Topic Referenced in Further guidance on contextualizing approaches, particularly to Market Based Sanitation Quantitative Survey – three respondents Qualitative respondents – three IP respondents Baby WASH Qualitative Survey – one IP respondent Competition between irrigation and WASH Quantitative Survey – one respondent Climate adaptive WASH programming Quantitative Survey – two respondents Qualitative respondents – one IP respondent Other areas noted as ‘unmet needs’ within the quantitative survey findings either relate to topics that PRO-WASH already provides, or to approaches or methodologies for the provision of capacity-building support. For example, individual respondents noted that in-person training, regional experience exchanges, how to draft ToRs, and qualitative enquiry for WASH interventions would be helpful. It is assumed that the respondents who noted needs that PRO-WASH already provides had not accessed the support available. IP Perception of Quality Qualitative respondents consistently described PRO-WASH’s work as high quality. Overall, 85% of survey respondents (55 respondents) rated PRO-WASH support as good or excellent, whilst the remaining 15% (10 respondents) rated the support as average quality. Respondents’ rating of overall quality was compared to the overall number of resources or tools they reportedly used, as shown in Table 6. Surprisingly, those who rated overall quality as average reported that they used a higher number of resources or tools in their program than those who rated the quality as good, though this is slightly less than those who rated the quality as excellent. Given the small number of respondents being discussed, this finding is not rigorous. Table 6 | Average Quality Rating by Average Number of Resources or Tools Used in the Respondent's Program Quality rating Number of respondents Average number of resources or tools used in program Average 10 3.9 Good 40 2.6 Excellent 15 4.1 Notable references to quality in the qualitative data focus on: • The content, presentation, and delivery of training curricular, which were generally described as participatory, practical, and tailored to participant needs • The ability to respond to wide-ranging technical questions, quickly and comprehensively • The specialists that were engaged through Applied Research grants • The learning products including research reports and abstracts Where respondents did note exceptions to the standard experience of high-quality support from PRO-WASH, these were discussed as particular incidents rather than trends, including: 25 • The quality of some consultants, particularly where local context was not considered in designing deliverables. In one reported instance, PRO-WASH removed and replaced a consultant due to the poor quality of their work. • The quality of online training workshops necessary due to COVID-19 restrictions, which were associated with higher levels of fatigue and lower engagement by participants. PRO-WASH noted the lower achievement in knowledge gain in the first online training workshop the Award delivered, compared to previous deliveries of the same curriculum, and adapted the methodology for future online sessions. These adaptations appeared to have had a marked impact on improved knowledge gain. Nevertheless, the general sense from IP respondents is that in-person sessions are of higher quality than those delivered online. 4.2.3 IP Perception of Utility Again, IP respondents generally rated the utility of PRO-WASH support highly, giving numerous examples of the application of learning, as discussed below (see Use of PRO-WASH facilitated learning, resources, and tools). Learning Question | Which technical focus areas were most useful? Technical Focus Areas of Most Use Survey respondents were asked to rate their experience with various focal topics in terms of their utility, where a score of 1 denotes a poor experience, and a score of 5 denotes an excellent experience. As shown in Table 7, all focal areas scored higher than 3.5 on average. Market Based Sanitation scored most highly, while COVID-19 scored lowest. Table 7 | Average Utility Scores for Focal Topics Topic Mean Score Market Based Sanitation 4.04 WASH Behavior Change (Make Me A Change Agent) 3.96 WASH Systems Strengthening 3.96 WASH and Nutrition 3.88 Water Quality 3.80 Human Centered Design approaches to WASH 3.76 Market Based Programming for Humanitarian WASH 3.75 Gender and WASH 3.61 WASH Operation and Maintenance 3.60 Integrated Water Resource Management 3.58 COVID-19 3.54 Examples of factors that reportedly contributed toward the high utility of the support include: • The modular approach to training curriculum, which enabled IPs to select key topics to cascade as needed by the RFSA • Support provided to translate materials into different languages as needed Examples of factors that reportedly diminished the utility of the support include: Lack of Contextual Adaptation 26 As noted above, a key theme emerging from the data is the relationship between contextual adaptation and utility, with some examples provided of tools and approaches being ill-suited to the RFSA’s needs. In the qualitative data, this tends to be attributed to consultants lacking contextual understanding, as discussed above (see the Aid Localization Agenda). Contrary to the findings presented in Table 7, which show that quantitative respondents scored MBS as the most utilizable resource, qualitative respondents most commonly cited Market Based Sanitation as an area where consultants either took time or were unable to adapt the tools as needed. This discrepancy between responses from the quantitative survey and qualitative data is likely to be caused by the small sample size. Given that the data are not representative, there is a higher likelihood of contradictions between the two sets of data. “the consultants were good, [they] took a while to get up to speed. They didn't know the [country] context and the recommendations they gave for someone, they were just very broad and a lot of times what you need for MBS is very specific, very sort of ground level recommendations that fit within the existing scope and budget of the Award.” – IP Respondent “The tools for the entrepreneurs [were] on Excel basis. [The entrepreneurs] needed a computer, but the entrepreneurs working with us has no laptop, no computer, never seen a computer. And we were asking if there's a possibility to make it on a phone basis like an Android. And that's where we [did] not get that. So we get the training, we got the concept but… this could not be used in field.” – IP Respondent Language Barriers Counter to the finding noted above regarding the translation into appropriate languages as a factor contributing to utility, language barriers both in terms of staff or consultants’ language skills, and the non-translation of tools were also commonly cited by qualitative respondents as an inhibitor of utility, presumably where support for translation was not provided. COVID-19 One provided anecdote noted that COVID-19 caused a delay in the commencement of an Applied Research grant, such that findings were not available for consideration during the RFSA’s Refine and Implement year. Therefore, the RFSA moved into implementation without reference to the research findings, diminishing the utility of the Applied Research grant for that particular RFSA. Learning Question | How did the combination of the different approaches to engagement (e.g., training, applied research, coaching, etc.) and the intensity of engagement influence the WASH/IWRM technical quality improvements? Which approaches most effectively and efficiently influence WASH/IWRM technical quality improvements within RFSAs? 4.2.4 IP Reflections on Approaches to Support Trends emerging from the data suggest that all approaches to support adopted by PRO-WASH were appreciated by respondents, with no clear outlier in terms of favored mechanism, other than ‘coaching’ as discussed further below (see Coaching). Respondents typically noted that the various approaches employed had diverse objectives, and could not be compared in terms of effectiveness and efficiency. This sentiment is echoed in the quantitative data, through which survey respondents who had engaged with PRO-WASH support rated all approaches as 3.9 or above in terms of the approaches’ contribution toward improvements in their work (a score of 1 denoted low effectiveness; a score of 5 denoted high effectiveness), again with no outliers suggesting relatively high or low effectiveness. Table 8 presents the average scores provided for each approach. 27 Table 8 | Average Effectiveness Rating of PRO-WASH Approaches to Support Learning Question | To what extent does layering engagements (e.g., coaching, mentoring, in-person, or virtual training, applied research) contribute to implementing partners being able to use the resources/tools? Coaching Though respondents did not frequently use the term ‘coaching,’ the data does suggest that continued, consistent dialogue has been instrumental in creating an open channel for sharing challenges, adapting approaches, and highlighting emerging needs. This process appears to be so engrained within PRO-WASH’s approach, particularly for high-engagement IPs, that it seems not to be thought of as a capacity-building methodology, but rather, as key facet of how PRO-WASH operates across all its support mechanisms. “I work more with field actors… during the 2-week meetings, then we do a summary of what happened in the field, the difficulties that we faced… So this way of doing so it is really, I think, interesting because we do not wait for long and I found it very efficient. We do detect problems earlier and then talk to the person in charge have an exchange and discussion with them. And we better analyze certain things also. There are also things that we could anticipate, and we could get pieces of advice during the meeting.” – IP respondent Combining Various Approaches The combination of types of support appears to have been greatly appreciated by respondents who engaged in multiple approaches, particularly with one-to-one coaching or mentoring as a key cornerstone to other approaches, as discussed above. Of the 65 survey respondents who reported to have engaged in PRO-WASH support, 65% (N=42) reported to have combined various types of support, citing examples of attendance at webinars, use of learning briefs, technical guides and newsletters, attendance training courses, and engagement in one-on-one dialogue (referred to within this evaluation report as coaching). Survey respondents were asked to describe ‘how the combination resulted in greater improvements in the technical quality of your work’ though only Outcome Mean Score WASH related resources in the FSN Network library 4.45 Video tutorials / YouTube channel 4.32 In-person training workshops 4.29 Webinars 4.28 Newsletters 4.23 Applied research findings 4.18 PRO-WASH website 4.15 Coaching 4.09 Technical guides 4.08 Remote training workshops 4.08 Learning briefs 4.00 Technical knowledge-sharing events 3.94 28 4 respondents provided detail in this regard. These comments typically described combining technical guides and learning briefs, as well as webinars, in-person learning exchanges, training workshops and one-on-one technical calls to gain technical capacity as individuals, cascade this amongst teams, build into program strategy, and monitor activity delivery. This trend is echoed in the qualitative data from those who discussed the impact of combining approaches. The below quotations illustrate these points. Data suggest that the provision of multiple approaches creates greater opportunities not only to engage, but to retain the topic as a priority within the RFSA strategy and work plan, which is likely to be an enabling factor given the fast-paced and unpredictable workloads many respondents discussed. Technical guides and learning briefs this helped me adapt some key things that worked in other areas of implementation and used them with my teams and got better results. The virtual webinars, in-person technical field visits/learning exchanges, one-on-one technical calls, and group discussions were all useful types of support that helped our WASH team, especially during the R&I year, reflect on their interventions and strategies for the upcoming years of the RFSA. Remote coaching, exchanges through calls, training workshops, webinars and technical knowledge-sharing events as well as the knowledge available through applied research results, newsletters, technical guides, learning sheets, have allowed me to better organize and monitor my activities, to acquire new knowledge and to exchange ideas with other WASH specialists from various backgrounds. – Survey IP respondents Learning Question | What are barriers / enablers in terms of working with PRO-WASH on this research? Applied Research As discussed above (The Added Value of PRO-WASH), key factors that enabled IPs to engage in the applied research grants were PRO-WASH’s network of experts to whom they could publicize RFPs, and the good reputation held by the Award that attracted quality consultants to bid for research opportunities. One IP respondent also referenced the timing of their initial engagement with PRO-WASH as a strong enabling factor, noting that their initial contact was made during the RFSA’s Refine and Implement year. This is focused on research, which created the opportunity for them to work with PRO-WASH in this regard. In terms of barriers to working with PRO￾WASH on applied research, of the qualitative respondents who had not engaged with research, one noted that they were not aware that Applied Research grants were available. Quantitative survey respondents were not asked to provide reasons regarding whether or not they worked with PRO-WASH on applied research. One concern noted by some respondents who had accessed Applied Research grants were the challenges related to the perceived ownership and authorship of the research. Respondents shared experiences of research partners failing to publicly acknowledge the role of IPs as co-Principal Investigators in research, and noted their observation that PRO-WASH could have done more during the planning phase of the research to advocate for their recognition during publication. Learning Question | To what degree did implementing partners use resources/tools (technical tools; training packages; learning briefs; webinars and applied research findings) developed by PRO-WASH within the specific projects that the Award supported? Which PRO-WASH resources/tools partners have been used and how often? 29 4.2.5 Use of PRO-WASH-Facilitated Learning, Resources, and Tools Whilst this evaluation was not designed to generate rigorous evidence of improvements in RFSA implementation attributable to PRO-WASH (see Evaluation Questions), the research did investigate IP reported use of the learning briefs, resources, and tools provided by PRO-WASH. Qualitative IP respondents typically noted that they applied the learning they had gained, in many cases provided compelling examples that are shared in the case studies in Annex 9. Typically, qualitative IP respondents noted that they had: • Adapted their implementation strategies to incorporate specific approaches they learned from PRO￾WASH. For example, the introduction of ‘Negotiating for Behavior Change’ and ‘Storytelling for Behavior Change’ approaches by the Sustainable Agriculture and Production Linked to Improved Nutrition Status, Resilience and Gender Equity RFSA (SAPLING) in Bangladesh. • Developed new implementation strategies on the basis of shared knowledge with PRO-WASH. For example, the development of the Scaling up Wash Interventions in Remote Locations (SWIRL) approach by the Strengthening Household Ability to Respond to Development Opportunities RFSA (SHOHARDO) in Bangladesh. • Developed new monitoring processes to track the delivery of approaches influenced by PRO-WASH. A number of IP respondents referenced the development of monitoring tools to track intervention fidelity and change brought about by approaches such as MMCA. Survey respondents who reported to have engaged with various types of support were also asked about their use of resources to deliver their program, the findings of which are presented in Figure 2. A high number of respondents did not respond to this question regarding the various tools and resources that they reported to have engaged with, resulting in a high level of ‘missing data’ and a relatively low number of respondents. The findings presented in Figure 2, therefore, should be interpreted as suggestive, rather than conclusive. These findings suggest that knowledge gained through webinars was most commonly used by respondents, followed by learning gained through support related to Water Quality, though the low number of respondents for the latter should be noted. The findings suggest that the most infrequently used forms of support were knowledge gained through a Learning Brief, knowledge gained through a Technical Guidance Note, and knowledge gained through coaching. As noted above (see Coaching), qualitative respondents typically did not use the term ‘coaching’ to refer to the consistent dialogue with PRO-WASH that they reported to have engaged with. It is plausible to assume that this observation is also applicable to quantitative respondents, which further suggests that findings relating to ‘coaching’ shown in Figure 3 should be interpreted with caution. Figure 3 | Proportion of Respondents Who Have Used Tools and Resources in Their Program, of Those Who Engaged with the Tool or Resource Learning Question | What are the barriers/enablers to implementing partners using the resources/tools? 30 Survey respondents who reported to have engaged with PRO-WASH support (N=65) were also asked to identify the factors that influence the extent to which they have used resources or tools in their everyday work. These findings are presented in Table 9. The most influential factors are the respondents’ understanding of the guidance, followed by the time available to them to apply their learning. Table 9 | Factors Influencing the Extent to Which Respondents Use Tools and Resources Factor influencing use of tool or resource Percent of respondents N My understanding of the guidance 58% 38 My time availability to apply new learning 52% 34 The degree to which my team was open to adapting our work 45% 29 External context factors (e.g., COVID-19 outbreak) 35% 23 Donor flexibility to adapt my work based on new learning 34% 22 My manager's flexibility to adapt my work based on new learning 32% 21 Budget availability 29% 19 Other 6% 4 Learning Question | How easy was it for partners to find and access resources/tools? Accessing Tools and Resources Respondents typically noted the ease with which they were able to access tools and resources when needed, with references to ‘links shared via email’ and the ‘FSN Network website’ consistently cited as sources of PRO￾WASH resources. Of the 65 survey respondents who reported to have engaged with PRO-WASH support, 58% (N=38) stated that accessing PRO-WASH resources is ‘easy’ or ‘very easy,’ whilst 25% (N=16) rated the access as ‘neither difficult nor easy.’ In some instances, respondents acknowledged the existence of the website, but noted that they did not have time to visit it. Learning Question | What are the barriers/enablers to implementing partners replicating training 4.2.6 Replication of Training within RFSAs Where qualitative respondents noted that they had cascaded or replicated training, the factors enabling them to do so were typically related to their time availability and the resources PRO-WASH provided. Regarding the latter, credit was ascribed to the modular nature of the tools, enabling trainers to select which topics were most relevant to their teams. One qualitative respondent also noted that they had been provided with support to translate the materials into a local language, which helped them cascade. Typically, barriers were noted as a lack of time to plan for cascaded training sessions. Questions related to the cascading of training were not included in the quantitative survey. Learning Question | To what degree do partners believe they will continue to use and share the resources/tools developed by PRO-WASH after the end of the Award? Which resources? How will they be used or shared, and with whom? What are the barriers to continued use and sharing? 4.2.7 Planned Continued Use of PRO-WASH Learning, Resources, and Tools Qualitative respondents generally noted an intention to continue using PRO-WASH learning, resources, and tools. The specific tools and resources cited by qualitative respondents tended to focus on those that the respondents had engaged with most closely through their partnership with PRO-WASH, though this would depend on future need. Likewise, when discussing how resources might be shared in future, and with whom, qualitative respondents 31 typically noted that this would depend on future need, and did not provide specific examples, which may have been difficult to predict. Beyond specific tools and resources, respondents shared reflections of experiential learning that will support their future work, such as ‘how to engage a research partner.’ Of the 65 survey respondents who had engaged with PRO-WASH support, 75% (N=49) reported that they would ‘likely’ or ‘definitely’ continue to use resources, tools, or knowledge gained from PRO-WASH after the Award has ended, whilst 23% (N=15) reported that this would be possible. These respondents were then asked to rate various types of learning insofar as which they are likely to continue using. A score of 1 denotes a low likelihood of continued use, and a score of 5 denotes a high likelihood of continued use. As demonstrated in Table 10, all types of learning were rated 4.3 or above on average. Table 10 | Likelihood of Continued Use of Learning Type Type of Learning Mean Score Learning I gained through Make Me A Change Agent training 4.95 Learning I gained through Market-based sanitation training 4.59 Learning gained from WASH Systems Strengthening training 4.58 Learning gained from Applied Research Reports 4.48 Learning gained from a Learning Brief 4.47 Learning gained from a Technical Guidance note 4.42 Learning I gained through another knowledge-sharing platform 4.41 Learning I gained through mentoring or coaching 4.39 Learning I gained through a webinar 4.33 Respondents were also asked to select all the factors that would influence the extent to which they continue using tools, resources, and learning gained from PRO-WASH. The most commonly selected factor was ‘my understanding of the guidance’ (72%; N=47), followed by ‘my time availability to apply new learning’ (63%; N=41), ‘budget availability’ (52%; N=34), ‘the degree to which my team is open to incorporating my ideas’ (46%; N=30), ‘my manager’s openness to incorporating my ideas’ (42%; N=27) and finally, ‘my donor’s openness to incorporating my ideas’ (32%; N=21). Learning Question | To what extent will the knowledge management platforms PRO-WASH used ensure sustainable access to and uptake of their respective resources? Lack of Knowledge of PRO-WASH’s Close-Out When qualitative respondents stated which knowledge management platforms they would use to access PRO￾WASH resources in the future, respondents consistently noted that they were unaware of whether knowledge management platforms such as the FSN website would still be available, and that they had not considered PRO￾WASH’s closure as part of their plans for their partnership with the Award. No respondents reported to have discussed their on-going support needs with PRO-WASH, and how these might be resourced following the end of the PRO-WASH Associate Award. In some cases, where an Applied Research grant would be generating findings, respondents noted their intention for PRO-WASH to accompany them in adapting their strategy in response to learning, and had not considered alternatives to such support in the absence of PRO-WASH. Learning Question | To what degree did implementing partners institutionalize the resources/tools developed by PRO-WASH into their organizations and beyond the specific project that the Award supported? What are the barriers/enablers to implementing partners institutionalizing the resources/tools? 32 4.2.8 Institutionalization of Learning, Resources, and Tools IP respondents were asked to share examples of new policies, guidelines, or approaches that had been adopted beyond their RFSA into their wider organization. Examples of such institutionalization are not commonly cited in the data, as respondents noted that they tended to be focused on specific RFSAs and had not widely shared their experiences within their organizations. It is plausible to suggest that this may be another example of siloed working within IPs, which acted as a barrier to institutionalizing learning, resources, and tools. Some exceptions were noted, particularly where PRO-WASH had engaged with IP Head Quarter WASH Advisors, which could be understood to be an enabler of institutionalization, though this tended to be in the form of sharing PRO-WASH materials with other units within the organization, rather than by creating new policies and approaches on the basis of PRO-WASH learning. This trend is somewhat echoed in the quantitative data. Of the 65 survey respondents who had engaged with PRO-WASH support, 32% (N=21) reported that new policies, processes, or systems had been developed within their organization on the basis of learning gained from PRO-WASH. Of these respondents, however, most examples provided were unclear, including responses such as ‘staff training on learning and development needs’, or ‘the development of the sanitation market for example’. 4.3 Outreach to Implementing Partners This section explores the processes undertaken by PRO-WASH to engage RFSA IPs and generate demand for the WASH technical assistance being offered. Issues explored include the outreach processes, the timing of the outreach in relation to RFSA program cycles, the value of the outreach process from the perspective of IPs, and the role PRO-WASH played within the Inception and Culmination processes facilitated by the Program Cycle Support (PCS) Award. Learning Question | At what points in the RFSA cycle was the outreach to implementing partners most and least relevant and effective? Why? How effective was PRO-WASH’s outreach as part of the Program Cycle Support (PCS) Inception and Culmination workshops? How can this be improved? 4.3.1 Initial Outreach As noted in Limitations, qualitative IP respondents tended to lack the historical knowledge of initial engagements with PRO-WASH. Other qualitative respondents noted that the PRO-WASH team initially ‘cold contacted’ IPs, sometimes finding their contact details through internet searches, to introduce the Award and the support it could offer. This is supported by survey data in which 53% (N=91) of 172 respondents noted that they first heard about PRO-WASH through an email from the Award team, while 37% (N=63) of respondents reported to have been referred to PRO-WASH by a manager or colleague. As the Program Cycle Support (PCS) Award became established, USAID/BHA and PCS began to introduce PRO-WASH to IPs during their initial Refine and Implement (RnI) activities, which appears to have created greater buy-in at the early stage of RFSA commencement, which data suggests is linked to greater perceived relevance, utility, and reported use of knowledge and resources. Overall, qualitative respondents who had engaged with PRO-WASH through the PCS Inception and Culmination workshops noted that this support was effective and influential in setting their strategic objectives for the Refine and Implement periods. Qualitative respondents typically did not give feedback on how this could be improved. Impact of Timing with the RFSA Cycle A clear thread emerging from the data is the relationship between the point in the RFSA program cycle at which PRO-WASH reached out to, and the ultimate value derived from the support provided. IPs who were already engaged in implementation when they began working with PRO-WASH noted challenges with scheduling time for PRO-WASH training sessions and adapting delivery strategies due to a lack of time and budget. Data suggest that as RFSAs have been launched throughout PRO-WASH’s program cycle, enabling PRO-WASH to engage during the refine and implement years, PRO-WASH has had more opportunity to influence logic model development, and IPs have had more of an ability to adapt and implement the learning gained. As demonstrated in Figure 3, this notion is supported by the quantitative data, which finds a decline in the perceived effectiveness of PRO-WASH’s initial outreach in relation to the implementation stage of the RFSA at the point of initial outreach. 33 Figure 4 | Perceived Effectiveness of PRO-WASH's Outreach to IPs in Relation to the Timing of Initial Outreach within the RFSA Program Cycle (N=74) 4.4 Thought Leadership and Influence on BHA Policies, Standards, Program Quality Finally, this section presents findings related to thought-leadership on behalf of PRO-WASH, with particular reference to any influence PRO-WASH had on BHA Policies, Standards, and Program Quality mechanisms. Learning Question | How did the Awards inform advancements or changes to USAID/BHA policies, standards, and guidance? 4.4.1 Examples of Influence IP respondents typically failed to recall any guidelines, standards, or policies provided to them by USAID/BHA that they were aware PRO-WASH had influenced. This finding is echoed in the quantitative data, as 92% (N=163) of 178 respondents reported that they were unaware of any changes to USAID policies, standard guidelines, or practices that were influenced by PRO-WASH. Other respondents with perhaps more positional insight into PRO-WASH’s thought leadership cited examples such as: • PRO-WASH driving interest in innovations such as Human-Centered Design • PRO-WASH promoting the need for the Make Me a Change Agent curriculum, where it met some resistance • PRO-WASH technical guides being included in USAID literature • PRO-WASH promoting adaptations of technical assistance processes, such as the inclusion of PRO￾WASH at the commencement of the Refine and Implement phase for new RFSAs 4.4.2 Factors Affecting PRO-WASH’s Influence on USAID/BHA Diverse views on PRO-WASH’s ability to influence USAID/BHA are present within the data. PRO-WASH’s Positional Power One reflection some respondents shared suggests that PRO-WASH is well-placed to influence USAID/BHA thinking and delivery, given their proximity to decision-makers and their trusted position as a known entity with WASH expertise. Complexity of USAID Systems Alternative views suggest that the complexity of USAID bureaus and processes, as well as the multiple WASH￾focused units within the US Government overall, limit the extent to which PRO-WASH can capitalize on its position to affect change. 0% 20% 40% 60% 80% 100%    Inception phase   Commencement of delivery    During delivery   Wrap-up stage   Following the end of the program Not effective or A little effective Effective or Very effective 34 “one limitation though is… it's so confusing, WASH is like across so many different bureaus, not just at USAID, but also with the current US government. Like there's also WASH team at CDC. There's also the USAID Innovation lab, the Global WASH Cluster and Emergency WASH…” “the other thing that also limited it is the reach within BHA and the reach within [PRO￾WASH], like who takes into account BHA changes depends on who hears about [PRO￾WASH]. And so that's dependent… where are people in BHA able to widely share what [PRO-WASH has] done… and then [was PRO-WASH] able to share sufficiently as well” - Respondents with positional insight into PRO-WASH thought leadership PRO-WASH Level of Effort As stated above, other reflections noted that PRO-WASH may not have been sufficiently resourced to enable the team to strategically engage in thought leadership with USAID/BHA. 35 5.0 CONCLUSIONS This section presents Causal Design’s assessment of PRO-WASH’s key achievements against its objectives as derived from the findings. Lessons presented have been generated on the basis of the evaluation evidence, with particular focus on those that have implications for future phases of technical assistance support. Distinct added value that PRO-WASH is perceived to have had for RFSAs | The findings present a clear and consistent portrayal of the significant added value that PRO-WASH has provided for RFSAs, above the technical support IPs could provide independently. Numerous examples of applied learning were presented. Though it was beyond the scope of this evaluation to determine the extent to which PRO-WASH contributed toward improved delivery of WASH interventions, or improvements in the outcomes of RFSAs’ WASH work, it is plausible to suggest that PRO-WASH’s work likely created positive impact in this regard on the basis of the evidence presented herein. The earlier the engagement, the greater value of the support | Findings clearly demonstrate that PRO￾WASH’s support has greater value for RFSAs if it is provided early on in their program cycle, and ideally, at the beginning of the R&I period. PRO-WASH not only responds to RFSAs’ known needs, but it also introduces new thinking and interrogations of RFSA logic models as functions that are most instrumental when RFSAs are determining their program strategy. PRO-WASH funding did not align with expectations of BHA from PRO-WASH | Findings demonstrate that PRO-WASH was limited in its ability to directly support certain programs, notably emergency programs, due to restrictions associated with the Award’s funding. This appears to have been in contradiction to expectations by USAID/BHA particularly following the BHA merger, though greater clarity was provided to PRO-WASH on the Award’s mandate. Demand-driven support is appropriate, but this design creates opportunities to miss RFSAs most in need | Though the findings present a clear case for retaining the design of PRO-WASH as a demand-driven service, given the benefits this strategy creates for IP buy-in, ownership, and application of the technical support received, it does mean that PRO-WASH has been limited in its ability to create meaningful impact where the services were not requested. Multiple factors influenced the uptake of technical support, the majority of which are outside of PRO-WASH’s control. Findings suggest that USAID/BHA has played a definitive role in generating demand, whilst factors related to RFSA leadership, team dynamics, and WASH technical team members are also key drivers of IP engagement with PRO-WASH. Reflections on the aid localization agenda | As USAID advances its commitment to localizing aid, the findings suggest that this agenda can be gainfully applied to PRO-WASH, and possibly other technical support functions that provide specialist advisory services for program delivery. It is clear that contextual knowledge, or the ability to quickly understand and adapt to various contexts, is imperative in the provision of technical support, which could be obtained through proactive and intentional measures designed in-line with localization efforts, such as investing in the technical expertise of specialists who are national to, and/or reside in, countries where RFSAs will likely be implemented over the medium to long-term, as detailed below (Recommendations). Risks to the program associated with staffing | The competencies and values ascribed to individual staff members on the PRO-WASH team, including personal beliefs in the value of collaboration and partnership, skills in diplomacy, time management and organization, as well as technical expertise in adult education and leadership, are exceptional qualities that may be difficult to recruit for. While this demonstrates Save the Children U.S.’s success in recruiting for PRO-WASH, it does highlight the risks for future phases of support, should individual team members decide to move on. This risk is heightened when the team is particularly lean and lacks specific roles that focus on key deliverables PRO-WASH needs such as knowledge management, adult learning, and communications. Given the clear relationship between individual staff members and the success PRO-WASH has realized in this first phase, as demonstrated in the findings, risks associated with staffing are a key concern. Political economy of aid agencies | Given the reported tendency of IPs and possibly RFSAs to operate in silos and as competitors, findings suggest that PRO-WASH being seen as a ‘Save the Children’ program may have negatively impacted the uptake of support. 36 PRO-WASH governance | The findings suggest a lack of clarity over the Steering Committees purpose and process has diminished the full benefit the SC could have provided to PRO-WASH. SC members appeared to have limited knowledge of strategic issues that influenced the achievement of PRO-WASH’s objectives, including its staffing structure and technical deliverables. Restructuring the Award’s governance structure may provide greater benefit to PRO-WASH insofar as the achievement of its objectives. Processes adopted by PRO-WASH resulted in relevant, quality, and useable technical deliverables | The findings suggest clear linkages between many of the processes adopted by PRO-WASH and the resulting value of the support they provided. These processes include early outreach to IPs, the creation of dialogue and feedback loops, creating access to tools and resources through the FSN Network website and more. There may be areas to improve procedurally, such as through the creation of a formal, confidential feedback and complaints mechanism, through which IPs could comfortably share concerns over support they have received. Relatedly, the contractual approval’s process agreed on between USAID and PRO-WASH appears to have been an inhibitor to PRO-WASH’s ability to be responsive and flexible to IPs’ needs. Findings related to the use of tools suggest that all forms of support are valuable | It is understood that PRO-WASH is eager to learn which types of support and which types of combinations of support are most effective and efficient in improving WASH and IWMR interventions delivered by RFSAs. The data suggest that RFSA's use of tools and resources and combinations of support are varying and likely to be influenced by a range of other factors, including those noted under ‘Factors Influencing Uptake of Technical Assistance,’ and those noted in Table 9 above, as well as the methodology of support itself. Given this wide range of confounding variables and small sample sizes, it is not possible to draw strong conclusions regarding these learning questions. The findings suggest, however, that PRO-WASH’s approach to tailoring technical support provision on the basis of agreements with the IPs has been instrumental in ensuring the value of the methodology of support. Coaching appears to be particularly instrumental | Where IPs have received one-on-one advisory support, either through the development and follow-up of a capacity-building workplan in the delivery of Applied Research or other means, this mentoring appears to have been instrumental in supporting IPs to utilize the technical support provided. This approach to coaching has facilitated problem identification and problem solving, has given confidence to WASH Leads within RFSA teams, and has motivated leaders to strive for continued improvement. The findings suggest that the cordial relationship developed between PRO-WASH and RFSAs has been integral to this process. The evaluation was limited in its ability to generate solid conclusions related to planned use of tools and resources | It is understood that PRO-WASH is eager to learn about IPs’ planned continued use of tools and resources following the end of this phase of PRO-WASH support. Data gathered in this regard is shared in the findings section, though respondents typically found it difficult to predict their planned use of tools in detail, often noting that this would depend on future need. Quantitative respondents rated all focal areas similarly and highly in terms of their likely continued use of tools and resources and reported that the factors most likely to influence this are their understanding of the guidance, their availability to apply the knowledge, and budget availability. The limited reflections on these learning questions may be related to the timing within which the evaluation took place in the final year of the Award, meaning that respondents were required to predict their future use, rather than provide clear examples of actual future use. COVID-19 had a negative impact on the perceived quality and utility of support, though PRO-WASH made an effort to minimize this impact | COVID-19 is referenced throughout the findings as a challenge that inhibited the delivery and achievement of PRO-WASH’s support. Numerous respondents noted PRO-WASH’s proactive attempts to mitigate this detriment, which likely did reduce the negative impacts on PRO-WASH’s objectives, though it is plausible to suggest that the outcome of PRO-WASH’s work would likely have been greater if the global pandemic had not occurred. IPs are unaware of planned next steps for PRO-WASH, which could impact their work | IPs currently drawing on PRO-WASH technical support are unaware of the plans for the next phase of PRO-WASH and lack clarity on what on-going support they can expect following the end of the first phase. PRO-WASH has had less impact beyond the objectives of supporting RFSAs | The findings suggest that, though PRO-WASH resources, tools, and knowledge are being widely used at a programmatic level, the extent to which these have been institutionalized within IPs and within USAID/BHA appears to be limited. This 37 may be a function of the limited LoE the small team could dedicate to engaging IP regional and headquarter Advisors and Program Managers who would have the positional power to institutionalize learning, and is likely also a function of the size and complexity of USAID/BHA as an institution. Endline Evaluation of the PRO-WASH Associate Award (2018-2023) 1 6.0 RECOMMENDATIONS This section presents proposed approaches to build on, or improve, the approach to WASH technical assistance for future phases of support, presented through a Findings-Conclusions-Recommendations (FCR) Matrix. Table 11 | Findings-Conclusions-Recommendations (FCR) Matrix Findings Conclusions Recommendations Award Design and Objectives • The added value of PRO-WASH • IP perception of relevance • IP perception of quality • IP perception of utility • IP reflections on approaches to support • Use of PRO-WASH facilitated learning, resources, and tools Distinct added value PRO￾WASH is perceived to have had for RFSAs 1. USAID/BHA should continue investing in PRO-WASH to retain this highly valued WASH technical support for RFSAs. • Limitations in scope due to funding streams PRO-WASH funding did not align with expectations the Award 2. USAID/BHA should ensure that the funding streams of future iterations of PRO-WASH are aligned with the intended recipients of support. • Demand-driven support • Factors influencing the uptake of technical assistance Demand-driven support is appropriate, but this design creates opportunities to miss RFSAs most in need, and limits PRO-WASH’s ability to ensure technical advisory support is followed, and WASH interventions are improved as a result of this support 3. PRO-WASH should position themselves as a ‘technical ally’ to RFSAs, such that their technical guidance is drawn from determining strategic direction during the Refine and Implement Year. PRO-WASH, therefore, would not be required to review or approve strategic workplans, as they would have been instrumental in developing these strategies. This approach would reinforce the collegial relationship between RFSAs and PRO-WASH. 4. Building on recommendation 26, where PRO-WASH and a RFSA disagree on strategic direction or on a response to an implementation challenge, the redress mechanism outlined in the Partnership Agreement should be enacted. This could include an agreement from both parties to defer to USAID as an ultimate decision-maker, such that the RFSA’s trust in PRO-WASH is not undermined. 5. USAID/BHA should consider agreeing on Key Performance Indicators (KPIs) with RFSAs that outline the expectation that IPs will draw on PRO-WASH, thereby Endline Evaluation of the PRO-WASH Associate Award (2018-2023) 2 Findings Conclusions Recommendations motivating RFSA leadership to create space for, and encourage, team engagement with the Award. 6. USAID/BHA should consider providing technical assistance for Leadership, Management and Coordination to RFSAs, to ensure that CoPs, DCoPs and their support structures are drawing on best leadership and managerial practice in overseeing large and diverse teams in challenging contexts. It is hypothesized that improved leadership, management, and coordination within the RFSAs will reduce the extent to which IP team members’ workloads are unpredictable and over-burdened, therefore creating more opportunity for IPs to engage with PRO-WASH. 7. Building on recommendation 16, if PRO-WASH is provided with further investment to recruit RFSA Relationship Managers, PRO-WASH should be more intentional about reaching out to RFSAs from which demand is low, and tailoring their communication and delivery approaches to the working styles of the individuals involved. • Staffing Risks to the program associated with staffing 8. USAID/BHA should ensure that any tenders for future phases of PRO-WASH include the explicit requirement that PRO-WASH team members are highly competent in relationship management, adult education, and coordination. Team members should be further required to demonstrate the lived values of partnership and collaboration, which were so fundamental to the success of PRO-WASH in its first phase. 9. PRO-WASH should ensure that any new team members to be recruited are assessed for these skills, competencies, and values. 10.USAID/BHA should consider increasing its investment into PRO-WASH, such that the team can be restructured to bring in roles with specialist focusing on knowledge management, adult learning, and communications. • Factors affecting the uptake of technical assistance • The added value of PRO￾WASH Political Economy of aid agencies 11.Save the Children U.S. should consider subtle adjustments, such as setting up email addresses using a PRO-WASH specific domain, such as name@pro-wash.org, which may minimize the risks outlined in the findings. • Steering Committee • Lack of clarity on the purpose of the SC • Limited knowledge of SC members regarding PRO￾WASH management and delivery PRO-WASH Governance 12.USAID/BHA and PRO-WASH should consider reviewing the governance structure of the Award. USAID and PRO-WASH may consider creating a larger ‘Technical Advisory Group’ comprised of technical experts in the field who would be expected to volunteer their time to review deliverables and provide technical guidance on strategy. From the Technical Advisory Group, a ‘Steering Committee’ could be elected or selected, which would have a more deliberate focus on governing the Award, which could include, amongst other responsibilities: • Guiding the strategic direction • Reviewing PRO-WASH’s capacity to deliver the strategic direction Endline Evaluation of the PRO-WASH Associate Award (2018-2023) 3 Findings Conclusions Recommendations • Limited interaction between the SC and USAID • Lack of clarity and consistency on time expectations • Composition of SC membership and a lack of representation from nationals of countries in which RFSAs are implemented • Acting as an advocate to USAID relating to issues that inhibit PRO-WASH’s capacity to deliver its strategic direction, including resourcing It is hypothesized that such an arrangement could generate greater buy-in from the WASH sector in general, provide greater clarity on roles and expectations, and provide a more independent voice to raise concerns to USAID as needed. 13. USAID/BHA and PRO-WASH should consider undertaking a process to ensure the meaningful engagement of often marginalized groups within the Technical Advisory Group and Steering Committee. This could include, but not be limited to, nationals of countries in which RFSAs are implemented, particularly women, young people, and people with disabilities. Steps to meaningfully engage these groups within the PRO￾WASH governance structure could include: a) Undertaking a consultation process of targeted groups, asking for their inputs on their interest in engaging, what barriers they would face when engaging, and how they could be supported to meaningfully engage. b) Accepting that it would not be appropriate to assume that members of these groups would be able to ‘volunteer’ their time in the same way that current members of the SC (who are primarily full-time employed and USA-based) are able to. Having an open conversation about appropriate mechanisms for compensating people’s time and expertise for their involvement in the governance structures. c) Considering how to intentionally address power imbalances through re-designing meeting agendas and governance processes, and ensuring that spaces are comfortable and open for members of marginalized groups. This re-design process should be undertaken with all members of the governance structures who should be consulted either individually or in peer-groups. d) Considering setting quotas for membership from various groups. Again, such quotas should be devised through a consultative process using an intersectional lens. Whilst quotas can create incentives for tokenistic engagement of vulnerable groups, this risk should be mitigated if all other steps outlined above are followed and documented, resulting in quotas that set minimum expectations for inclusion. Implementing Partner Engagement and Impacts of Technical Assistance and Influence • Flexibility from the perspective of the PRO￾WASH team • IP perception of utility • IP reflections on approaches to support Processes adopted by PRO￾WASH resulted in relevant, quality, and useable technical deliverables Findings related to the use of tools suggest that all forms of support are valuable 14. PRO-WASH should continue adopting similar methodologies, including coaching, applied research, technical guides, and in-person and online self-paced training workshops, to support future phases of programing, with particular emphasis on coaching and mentoring. Building on the fact that the best methodologies to adopt are likely to be specific to the RFSA and influenced by a range of confounding variables, the methodologies to be utilized for support provision should be discussed with IPs and built into an agreed Scope of Work, which may be annexed to the Partnership Endline Evaluation of the PRO-WASH Associate Award (2018-2023) 4 Findings Conclusions Recommendations Coaching appears to be particularly instrumental Agreement. The Scope of Work should be regularly reviewed and adapted based on feedback from the IP. 15.PRO-WASH should revert to in-person training workshops as opposed to remote delivery, given the feedback that online sessions can be difficult for some team members to access and remain engaged in. 16.USAID/BHA should consider increasing its investment in PRO-WASH, such that the team can be restructured to bring in RFSA Relationship Managers whose responsibility would be to act as the RFSAs’ focal coach or mentor within PRO-WASH to support their capacity-strengthening journey. The profile of a RFSA Relationship Manager could include a strong background in partnership management, organizational capacity strengthening, and adult education. 17.As far as is possible within procurement policies, USAID/BHA should consider reducing the administrative burden on PRO-WASH to seek approval for delivery-related procurements or decisions. 18.Drawing on their experience and engagement with RFSAs, PRO-WASH should consider developing a Short-Term Technical Advisory (STTA) pool of consultants with relevant expertise based on projections of focal areas likely for the coming year or years. The STTA pool should include multiple consultants with relevant technical expertise and should be submitted to USAID for pre-approval. USAID/BHA and PRO￾WASH should agree that if a consultant in the STTA pool has been pre-approved, PRO￾WASH may contract them for short-term deliverables, without the need for further approval from USAID. • Factors influencing the uptake of technical support • IP perception of utility Reflections on the aid localization agenda 19.PRO-WASH should consider investing in national or regional technical expertise to build cadres of technical specialists in countries that are likely to receive continued RFSA support. Drawing on the substantive learning of WASH priority areas gained through this first phase of PRO-WASH, PRO-WASH could identify and train technical specialists in focal areas considered to be key for the coming years. This would create greater and more localized competition for RFPs published by RFSAs, would provide a cadre of potential consultants who have both specialized technical expertise and national contextual knowledge, and would be a more accessible resource to wider programming within countries. • Inconsistency of feedback loops Processes adopted by PRO￾WASH resulted in relevant, quality, and useable technical deliverables. There may be areas to improve procedurally, such as through the creation of a formal, confidential feedback 20. PRO-WASH should consider the creation of a formal feedback and complaints mechanism for IPs to provide anonymous feedback or complaints regarding the support they have received, should the need arise. The key steps involved in a Formal Feedback Loop are: a) Collect feedback. This requires the creation of an anonymous, accessible and ‘known’ mechanisms through which IPs can share feedback or complaints b) Acknowledge and record the feedback Endline Evaluation of the PRO-WASH Associate Award (2018-2023) 5 Findings Conclusions Recommendations and complaints mechanism, through which IPs could comfortably share concerns over support they have received. c) Take action to investigate, refer, and follow-up on the feedback provided d) Communicate the actions and outcomes to IPs, whilst always protecting the anonymity of the individual who submitted the initial report. Clear policies and procedures guiding each step are required, and should be developed in consultation with IPs. The aid sector has progressed significantly in the development of Accountability to Affected People (AAP), and much of the literature available would be applicable to PRO￾WASH in this regard, though this would need to be adapted from a focus on communities to a focus on IPs. For example, the following resources may be helpful: • Community Feedback Mechanisms: Guidance and Toolkit by Danish Refugee Council • CHS Alliance website (The CHS Alliance is a global alliance of humanitarian and development organizations committed to making aid work better for people) • Lack of knowledge of PRO-WASH’s closeout IPs are unaware of planned next steps for PRO-WASH, which could impact their work 21. PRO-WASH should urgently liaise with all IPs receiving on-going support to provide as much information as is currently available regarding future plans following the end of the first phase of support. While PRO-WASH may not be in a position to provide certainty of future plans, the Award should open dialogue in this regard and support IPs to plan for various likely scenarios, based on the information available at this time. • Planned continued use of PRO-WASH learning, resources, and tools The evaluation was limited in its ability to generate solid conclusions related to the planned use of tools and resources 22.USAID/BHA and PRO-WASH may wish to consider conducting a further review into the sustained use of PRO-WASH resources and tools after a defined period of time following the closure of the first phase of support. This will provide more solid examples and learning regarding the sustained use of resources and tools. Outreach to Implementing Partners • Impact of timing of outreach with the RFSA cycle The earlier the engagement, the more valuable the support 23.USAID/BHA should consider including explicit expectations of RFSA engagement with PRO-WASH into tenders for future RFSAs. It may be beneficial to the RFSA design and tendering process for information relating to PRO-WASH to be publicly available and referenced within tenders, such that agencies bidding for RFSAs are able to build the LoE for this engagement into their initial bid workplans and budgets. If this recommendation is enacted, LoE should be included for PRO-WASH to respond to enquiries from agencies seeking to bid for RFSA tenders. 24.PRO-WASH should continue to reach out to RFSAs at the earliest possible point in their Refine and Implement year, so that LoE for technical assistance can be built into detailed implementation plans. Continuing to engage through the PCS workshops appears to be the most coherent and efficient way to facilitate early outreach to new RFSAs. Endline Evaluation of the PRO-WASH Associate Award (2018-2023) 6 Findings Conclusions Recommendations 25.USAID/BHA should continue to provide an entry point for PRO-WASH to reach out to newly commencing RFSAs both to ease communications, and to generate buy-in to technical support on behalf of the RFSA. 26.PRO-WASH should consider being intentional about developing Partnership Agreements with RFSAs outlining expectations of the partnership, communication channels, and an agreed redress mechanism, should disagreements arise. The redress mechanism should be devised based on what is acceptable to both parties – if an informal agreement is more acceptable, then this can be used rather than a ’legal’ mechanism, which may be intimidating and overly formal. The MoU could include an annexed Scope of Work or Capacity Building Plan, such as those developed during the first phase of PRO-WASH, given the finding that these plans were demonstrably valuable in ensure technical support was relevant to IPs’ needs. Thought Leadership and Influence on BHA Policies, Standards, Program Quality • Institutionalization of learning, resources, and tools • Factors influencing PRO￾WASH’s influence on USAID/BHA PRO-WASH has had less impact beyond the objectives of supporting RFSAs 27.USAID/BHA and PRO-WASH should determine whether the institutionalization of resources and thought-leadership with USAID/BHA are clear objectives for PRO￾WASH. If this is the case, the design of the Award should be reviewed to ensure sufficient funding is allocated to increase the LoE for PRO-WASH to strategically meet this objectives. a. Regarding IP institutionalization of resources and tools, PRO-WASH should be more intentional about working directly with IP HQ/Regional WASH Advisors, who hold the mandate of institutionalizing policies and approaches across IPs, as well as WASH Leads within RFSAs, whose focus is directed at the RFSA delivery, rather than the wider organization. b. Regarding thought-leadership with USAID/BHA, with additional LoE, PRO-WASH should both think through the complex political economy within USAID and strategically plan how to engage in order to promote PRO-WASH's thought leadership. If funds were available, PRO-WASH could commission a Political Economy Analysis (PEA) to be undertaken to support this process. Endline Evaluation of the PRO-WASH Associate Award (2018-2023) 7 ANNEXES Annex 1 | PRO-WASH Final Evaluation Scope of Work Annex 2 | PRO-WASH Implementation Timeline Summary Annex 2 PRO-WASH Implementation Timel Annex 3 | Key Informant Interview and Focus Group Discussion Semi￾Structured Interview Guides Annex 3 KII and FGD Semi-Structured Inter Annex 4 | Quantitative Survey Instrument Annex 4 Quantitative Survey Instrument.doc Annex 5 | Quantitative Survey Respondents Categorized by Position and Country or Region Annex 5 Quantitative survey respondents ca Annex 6 | Identification of Qualitative Target Respondents Annex 6 Identification of target respondents. Annex 7 | Qualitative Code Book Annex 7 Qualitative Code Book.docx Endline Evaluation of the PRO-WASH Associate Award (2018-2023) 8 Annex 8 | Evaluation Ethics Annex 8 - Evaluation Ethics.docx Annex 9 | Case Studies Annex 9 - Case Studies