EVALUATION OF WASHPLUS: SUPPORTIVE ENVIRONMENTS FOR HEALTHY COMMUNITIES PROJECT FINAL REPORT This report was produced for review by the United States Agency for International Development (USAID). It was prepared by Optimal Solutions Group, LLC, for USAID’s Learning, Evaluation, and Analysis Project (LEAP), Contract Number: AID-OAA-C-11-00169. UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT LEARNING, EVALUATION, AND ANALYSIS PROJECT (AID-OAA-C-11-00169) EVALUATION OF THE WASHPLUS: SUPPORTIVE ENVIRONMENTS FOR HEALTHY COMMUNITIES PROJECT FINAL REPORT Prepared for the United States Agency for International Development Prepared by Optimal Solutions Group, LLC University of Maryland Research Park, M Square 5825 University Research Court, Suite 2800 College Park, MD 20740 www.optimalsolutionsgroup.com Prepared by Mark Turner, LEAP Chief of Party Clarissa Brocklehurst, WASH Expert/ Team Leader Simon Mead, WASH Expert Christabel Dadzie, LEAP Program Manager/Evaluation Specialist Joanne Pérodin, Project Coordinator/Analyst November 25, 2014 Disclaimer The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States government. CONTENTS LIST OF FIGURES .................................................................................................................... iii LIST OF TABLES ..................................................................................................................... iv ACRONYMS .............................................................................................................................. v EXECUTIVE SUMMARY ....................................................................................................... vii Background ............................................................................................................................... vii Evaluation Methodology ........................................................................................................... vii Findings ..................................................................................................................................... vii Conclusions ................................................................................................................................ ix Lessons Learned for the Future .................................................................................................. xi Recommendations ..................................................................................................................... xii 1. INTRODUCTION .............................................................................................................. 1 1.1. Background and Context................................................................................................. 1 1.2. Purpose and Scope of the Evaluation.............................................................................. 2 1.3. Key Evaluation Questions............................................................................................... 3 2. METHODOLOGY ............................................................................................................. 4 2.1. Data Collection Approaches ........................................................................................... 4 2.2. Study Limitations ............................................................................................................ 7 3. FINDINGS AND CONCLUSIONS ................................................................................... 8 3.1. USAID Context ............................................................................................................... 8 3.2. Benefits of WASHplus as a Core Mechanism for Missions ......................................... 11 Review of Financial Data ...................................................................................................... 13 WASHplus in the context of USAID investments in WASH ............................................... 14 3.3. Impact at Country Level ............................................................................................... 19 Quality of service delivery and impact on national policy and capacity ............................... 19 Engagement with the private sector ....................................................................................... 21 3.4. Integrated Programming ............................................................................................... 22 3.5. Project Management ..................................................................................................... 25 3.6. Global Leadership and Knowledge Management ......................................................... 27 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report ii Global leadership ................................................................................................................... 27 Knowledge management ....................................................................................................... 28 4. LESSONS LEARNED FOR THE FUTURE ................................................................... 33 5. RECOMMENDATIONS .................................................................................................. 34 Annex 1: Evaluation of the WASHplus: Supportive Environments for Healthy Communities Project Cooperative Agreement No. AID-OAA-A10-00040 - Statement of Work .................. 35 Annex 2. Country Profiles ......................................................................................................... 45 Bangladesh ................................................................................................................................ 45 Bangladesh ................................................................................................................................ 47 Benin ......................................................................................................................................... 49 Kenya ........................................................................................................................................ 50 Madagascar................................................................................................................................ 51 Mali ........................................................................................................................................... 52 Uganda ...................................................................................................................................... 53 Zambia ....................................................................................................................................... 54 Annex 3: Summary of Accomplishments by Country ............................................................. 55 Annex 4. List of Documents Reviewed .................................................................................... 57 Annex 5. Interview Protocols .................................................................................................... 62 Annex 6. WASHplus Budget by Country (obligated funds vs. actual expenses to date) ......... 75 Annex 7. WASHplus Engagement in Global Activities ........................................................... 76 Annex 8. WASHplus Global Leadership contributions as identified in Annual Reports ......... 81 Addendum – WASHplus Response to Evaluation Report ........................................................ 82 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report iii LIST OF FIGURES Figure 1: Obligated funds spent as of May 2014 .......................................................................... 15 Figure 2. Expenditures by country as percent of annual total....................................................... 16 Figure 3. Expenditures by partner ................................................................................................. 16 Figure 4. Expenditures by country (to May 31 2014) ................................................................... 17 Figure 5. Webinars and estimated participants ............................................................................. 31 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report iv LIST OF TABLES Table 1: WASHplus evaluation interviews by country and respondent category .......................... 5 Table 2: Summary of WASHplus activities by country ............................................................... 13 Table 3: USAID budget allocations for the water sector by thematic area (USAID Water and Development Strategy, 2013–2018) ............................................................................................. 14 Table 4. WASHplus webinars ....................................................................................................... 29 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report v ACRONYMS AOR Agreement Officer's Representative APHIA AIDS, Population and Health Integrated Assistance AVAS Association of Voluntary Actions for Society CCT Controlled Cooking Test CHEWs Community Health Extension Workers CHWs Community Health Workers CLTS Community-Led Total Sanitation COP Chief of Party CQI Continuous Quality Improvement DAM Dhaka Ahsania Mission DANIDA Danish International Development Agency DHS Demographic and Health Surveys DPHE Department of Health and Engineering DORP Development Organization of the Rural Poor FABRI Further Advancing the Blue Revolution Initiative FHI360 Family Health International360; WASHplus prime partner FJKM Fiangonan'i Jesoa Kristy eto Madagascar (faith-based NGO in Madagascar) FTF Feed the Future BMGF Bill and Melinda Gates Foundation GACC Global Alliance for Clean Cookstoves GH/HIDN Global Health, Office of Health, Infectious Diseases and Nutrition GIZ German Agency for International Cooperation GoB Government of Bangladesh HIP Hygiene Improvement Project HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome IAP Indoor Air Pollution ICS Improved Cook Stoves IDIQ Indefinite Delivery/Indefinite Quantity (contract) ICDDR,B International Centre for Diarrheal Disease Research, Bangladesh IRC International Research Centre JICA Japan International Cooperation Agency JMP Joint Monitoring Program KM Knowledge Management KR Knowledge Resources LEAP Learning, Evaluation, and Analysis Project M&E Monitoring and Evaluation MCH Maternal and Child Health Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report vi MDGs Millennium Development Goals MICS Multiple Indicator Cluster Surveys MOEd Ministry of Education MOH Ministry of Health MOPHS Ministry of Public Health and Sanitation MPI Multidimensional Poverty Index NGO Non-Governmental Organization NTD Neglected Tropical Diseases PLAN Plan International USA PPP Public and Private Partnerships PPPHW Public Private Partnership for Handwashing SAF Sampan'Asa momba Fampandrosoana (faith-based NGO in Madagascar) SAP South Asia Partnership SNV Stichting Nederlandse Vrijwilligers (Foundation of Netherlands Volunteers) SN2 Santénet2 SOW Statement of Work SPLASH School Promoting Learning Achievement through Sanitation and Hygiene SPRING Strengthening Partnerships, Results, and Innovations in Nutrition Globally STAR SW Strengthening TB and HIV/AIDS Response in the South-Western Region of Uganda SUWASA Sustainable Water and Sanitation in Africa SWA Sanitation and Water for All UNC University of North Carolina UNICEF United Nations Children’s Fund UNDP United Nations Development Programme USAID United States Agency for International Development VERC Village Education Resource Center WADI Water and Development IDIQ WASH Water, Sanitation and Hygiene WEDC Water, Engineering and Development Centre WHO World Health Organization WSP Water and Sanitation Program WSUP Water and Sanitation for the Urban Poor Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report vii EXECUTIVE SUMMARY Background The WASHplus Cooperative Agreement is housed in the United States Agency for International Development (USAID) Bureau for Global Health’s Office of Health, Infectious Diseases, and Nutrition (GH/HIDN), Maternal and Child Health Division. It is a 5-year activity focused on the implementation and scale-up of environmental health interventions to prevent morbidity and mortality from infectious diseases, primarily in young children, as well as among vulnerable adult populations. The WASHplus Cooperative Agreement began in October 2010 and ends in September 2015. The strategic objectives of the WASHplus activity are: 1. Increase the availability and use of water supply, sanitation, hygiene (WASH), and indoor air pollution (IAP) interventions. 2. Develop and implement WASH/IAP integration strategies. 3. Support USAID’s participation in strategic partnerships. 4. Develop and test new and innovative approaches and tools. Evaluation Methodology The evaluation methodology included a document review, budget analysis, and stakeholder interviews. A total of 55 out of 81 interviews (67 percent response rate) were completed with WASHplus staff in Washington, DC, WASHplus staff in-country, global partners, government representatives, USAID Missions staff, USAID staff in Washington, DC, and local implementing partners. Stakeholders were interviewed in all seven of the WASHplus country programs (i.e., Bangladesh, Benin, Kenya, Madagascar, Mali, Uganda, and Zambia). The team also evaluated the content and functionality of the WASHplus website. As per the statement of work (SOW), visits to field programs were not part of the evaluation methodology. Findings USAID Context • WASHplus has operated within a dynamic and evolving institutional context at USAID that puts emphasis on increased engagement with local partners, particularly in building the capacity of and working through national and local governments. Benefits of WASHplus as a Core Mechanism for Missions • WASHplus is designed to offer a variety of benefits, including that it is easier and faster than using the other mechanisms available to USAID Missions to implement WASH programming, and it provides Missions with access to trusted partners. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report viii • Obligated funds are currently just over a third of the $100 million WASHplus ceiling, which, though high, constitutes a small fraction of USAID funding to WASH. Some WASHplus country programs are not close to using their obligated budget. Countries with large service delivery account for a great portion of expenditures. • There was limited demand for the WASHplus mechanism from USAID Missions. Impact at Country Level • WASHplus has undertaken service delivery in its country programs with programming designed to positively impact the lives of the poor. • The WASHplus mechanism did not have a strong emphasis on government capacity and leadership. • WASHplus was not designed, and has not been used, to help Missions to build skills to conduct a government-to-government approach of implementing WASH activities in the future. • Despite some attempts, public-private partnerships (PPP) and other forms of private￾sector involvement have not evolved into a prominent feature of the WASHplus program to date. Integrated Programming • WASHplus has incorporated integrated programming, has made related resources available, and has undertaken innovative country-level integrated programming. • The IAP component of WASHplus has had a reach in IAP globally, and in applying lessons from behavior change programming in the WASH sector. However, WASHplus was unable to achieve notable country reach with IAP. Project Management • Respondents suggested that WASHplus is well managed and the staff is highly responsive. • WASHplus has faced constraints due to its complicated contractual structure involving multiple partners and multiple bureaucracies. • WASHplus is perceived to have not demonstrated flexibility with regard to staff assignments and its use of experts. Global Leadership and Knowledge Management • Interview results suggest that WASHplus has not developed an evident global leadership role and is largely absent from the current global policy debates. Opportunities exist to bring the learning from WASHplus country programming to global attention. • Certain aspects of WASHplus knowledge management are highly valued by users, in particular the “WASHplus Weekly.” • The reach of WASHplus knowledge management has been uneven. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report ix • Webinars for which attendance data are available show that they attracted more than 500 participants and that between 14 and 28 percent of attendees have been from developing countries. • There is a considerable volume of English content on the WASHplus website and Twitter, but limited content in French, which is needed considering that three of the country programs are francophone. Conclusions The following conclusions were documented as a result of the evaluation’s findings and analysis: USAID Context: WASHplus has operated within a dynamic and evolving institutional context at USAID that puts great emphasis on increased engagement with local partners, particularly in building the capacity of and working through national and local governments. Benefits of WASHplus as a Core Mechanism for Missions: The benefits and challenges of the WASHplus central mechanism, combined with the evolving strategy within USAID and the ongoing commitment of the agency to WASH, suggest that as Missions are looking at larger WASH projects, they will be less likely to use a centrally managed mechanism for projects that include service delivery, and will work directly with local NGOs and national and local governments. Moreover, it is also evident that the need for advice, training, and technical assistance will be greater, particularly for projects that aim to work through national governments, local governments, and local implementing agencies. Financial: Most WASHplus country programs had insufficient demand for their programs which led to an under-utilization of their budgets. Countries with significant service delivery represent a large portion of WASHplus expenditures compared with other small-scale programs. WASHplus accounts for a small portion of USAID expenditures on WASH, but its strategic objectives were ambitious. However, it appears that the demand-driven model used has not been consistent with these objectives, particularly with the objective of “moving beyond basic service delivery through non-governmental organization (NGO) projects. Impact at Country Level: WASHplus is seen as a program that has had a significant impact and has implemented effective service delivery in the countries where it was in operation. Although the WASHplus country activities in Zambia and Bangladesh aimed to build government capacity at the national and the district levels, WASHplus has not been used to build government capacity and leadership Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report x systematically across all countries in which WASHplus operated. Interviewees suggested that more could be achieved in terms of building the capacity of the local private sector for provision of WASH products and services and to develop new PPPs, as called for in the strategic objectives. Integrated Programming: Based on interview findings and documents reviewed, WASHplus has energetically embraced integrated programming, has made many related resources available, and has undertaken innovative country-level integrated programming. Integrated programming is becoming an accepted approach, both within USAID and other agencies. Nonetheless, understanding the institutional obstacles to making integrated programs work is not yet well understood. Project Management: According to interview respondents (approximately 90 percent), WASHplus staff have been highly responsive to country offices, with good communication quality and promptness as indicated in the interview responses. Because of its contractual structure, WASHplus has also faced some constraints. These constraints have led to a delay in the start-up of some activities. Interview responses also suggest that the structure and responsibilities of this centrally funded mechanism were often not clear. Global Leadership and Knowledge Management: Based on a review of WASHplus activities and interview responses, the program has been somewhat active at the global level, it could have a higher profile. Opportunities exist to bring the learning from WASHplus country programming to global attention. USAID seeks to take a leadership role in the sectors in which it engages. There is considerable opportunity to be the voice of the United States on specific policy issues relating to WASH, in particular those around sanitation and hygiene and the role of governments (especially the leadership of Ministries of Health). The attendance rate at webinars suggests there is a strong interest in presentations organized by WASHplus; some respondents recommended that webinars could be both more frequent and more consistently scheduled. The availability of recorded webinars online seems to be an effective way to attract a greater audience from developing countries. The WASHplus website and presence on Twitter have been used effectively to communicate information. However, considering that three of the seven countries participating in WASHplus are Francophone, the evaluation team found that limited material was available in French on their website. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report xi Lessons Learned for the Future The following lessons learned for future programming were gathered: • In order to be relevant, a centrally funded mechanism needs to have some influence on other USAID WASH investments, not just the funding that flows through the mechanism itself. This could be achieved through knowledge management, capacity building, and the development of innovations. • The demand-driven model used has not been consistent with the ambitious strategic objectives set for this centrally managed mechanism. In the future, if a demand-driven approach is used, USAID should ensure it benefits from more widespread promotion to Missions, and for country-level project requests to be in line with the intended scale, goals, and strategy of the program. It is important to develop approaches that allow the learning and innovations from country-level activities to be widely adopted among a large number of USAID Missions. • Future investments and programming should reflect that USAID support to WASH will continue in the future. USAID will expect these interventions to be consistent with the strategies that it has recently developed, and closely coordinated with the other WASH activities of USAID. Given USAID’s new strategies and the evolution of development thinking (e.g. leveraging the whole of government approach), there is a need to develop approaches that foster government leadership, build government capacity, work with the private sector at scale, and increase the ability of U.S. Missions to work directly with the host government. • In the future, it will be important to provide Missions with advice and support necessary to work more effectively with national governments and local partners to respond to the imperative articulated in the USAID Forward strategy. • There is considerable opportunity to be the voice of the United States on specific policy issues relating to WASH, in particular those around sanitation and hygiene and the role of government, especially the leadership of Ministries of Health. • Future knowledge management could focus on resources relating to working with and through governments on water, sanitation, and hygiene, which would address the specific needs of the Ministries of Health to deliver on their mandate with respect to sanitation and hygiene. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project — Final Report xii Recommendations The evaluation team recommends the following activities, which can be implemented during the final year of the program: • The last year should have a strong focus on building country-level capacity and mechanisms to ensure sustainability of the interventions in the program countries. When possible, emphasis should be given to supporting existing entities—especially national and local governments—to build upon WASHplus’ effective activities once WASHplus support comes to an end. • USAID should consider extending WASHplus for a period of 6 to 12 months, which would provide additional time for countries where there were start-up delays or under-utilization of budgets to meet their goals. This additional time period may also allow for further country￾level capacity development and support to government agencies where necessary. • Knowledge management activities, especially webinars, should be ramped up to disseminate the learning from WASHplus, and to make a wider group of stakeholders aware of recent evidence and best practices in WASH. Knowledge management products and webinars should be available in both English and French, and should be targeted to a variety of audiences, including a larger proportion of in-country stakeholders. • The final knowledge management activities and products should focus not only on perceived successes, but on the constraints and challenges faced by the WASH sector and experienced through WASHplus activities. A thorough examination of the challenging aspects of activities such as achieving integration within development agencies, communicating the need for behavior change, and engaging with the private sector would and identifying mitigation strategies to these challenges, would be an excellent way of identifying additional available resources to other programs both within USAID and externally. • WASHplus should develop a proposed future research agenda for USAID based on the needs identified and lessons learned through its activities. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 1 1. INTRODUCTION 1.1.Background and Context The WASHplus Cooperative Agreement is housed in the United States Agency for International Development (USAID) Bureau for Global Health’s Office of Health, Infectious Diseases, and Nutrition (GH/HIDN), Maternal and Child Health Division. This 5-year activity is focused on the implementation and scale-up of proven environmental health interventions to prevent morbidity and mortality from infectious diseases, primarily in young children, as well as among vulnerable adult populations. The WASHplus Cooperative Agreement commenced in October 2010 and ends in September 2015. The strategic objectives of the WASHplus activity are: 1. Increase the availability and use of water supply, sanitation, and hygiene (WASH), and indoor air pollution (IAP) interventions. 2. Develop and implement WASH/IAP integration strategies. 3. Support USAID’s participation in strategic partnerships. 4. Develop and test new and innovative approaches and tools. Strategic Objective 1: Increase the availability and use of proven, high-impact WASH and IAP interventions. The project aims to achieve measurable improvement in the prevention of diarrhea through increased access to a water supply and sanitation, and improved hygiene practices, including hand washing with soap, the treatment and safe storage of water, the safe disposal of feces, and food hygiene. The project also aims to prevent the health impacts from IAP by improving indoor air quality in rural and urban households still reliant on biomass fuels and basic stoves to meet their energy needs for cooking and heating. Moving beyond basic service delivery through non-governmental organization (NGO) projects, WASHplus uses innovative approaches to expand access and use of WASH services, such as the development of innovative approaches for WASH financing; building the capacity of the local private sector for provision of WASH products and services; and institutional strengthening, especially at lowest levels of governance. Strategic Objective 2: Develop and implement WASH integration strategies. The project aims to achieve improved health outcomes by integrating WASH programs with other health and non￾health activities. Primary focal areas are the integration of WASH in education, HIV/AIDS, nutrition, food security, and neglected tropical diseases. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 2 Strategic Objective 3: Support USAID’s participation in strategic partnerships. WASHplus supports USAID’s participation in strategic partnerships with other donors and cooperating agencies as a vehicle for increasing both the quality and scale of program activities. At the global level, a key factor in enhancing USAID’s leadership role and maximizing its contribution to the sector is identifying and pursuing the most effective platforms and partnerships (e.g., for scaling up sanitation). WASHplus will support USAID’s existing partnerships and explore the development of new public-private partnerships (PPPs). Strategic Objective 4: Develop and test new and innovative approaches and tools. Problems that remain in the WASH/IAP sectors include technical efficacy, affordability and financing, behavior change strategies for continued use, and sustainable models to reach scale. These challenges call for innovations in technology, tools, and approaches. WASHplus will focus on developing or adapting promising technologies and approaches developed elsewhere for WASH and IAP implementation. Cross-cutting initiatives that are developed across all of the strategic objectives include knowledge management and monitoring and evaluation (M&E). The knowledge management program will build knowledge platforms for information exchange, capacity building, and networking. These issues should be considered in all aspects of the strategic objectives. The activities described above were designed to ultimately achieve a large-scale public health impact. Approximately 80 percent of WASHplus efforts were to be focused on increased availability and the use of WASH interventions, while 20 percent were to be dedicated to IAP. 1.2.Purpose and Scope of the Evaluation The primary objectives of this evaluation are to: 1. Assess whether WASHplus is achieving its objectives and planned outputs as stated in the WASHplus agreement, project description, and in approved implementation plans. The performance review should evaluate successes, shortcomings, impact, and lessons learned from project interventions to date. 2. Make recommendations to improve the implementation of the current project and to inform future environmental health investments. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 3 1.3.Key Evaluation Questions The statement of work provided five principal questions to guide the evaluation, using the broader categories as a framework for inquiry. Country impact: 1) To what extent, in what ways, and with what challenges has WASHplus acted to increase the coverage of high-impact WASH interventions at local, national, and global levels? 2) What is the efficacy of the methods used to increase coverage? 3) Specifically, what approaches to achieving increased coverage were chosen by the project to guide improved programming, contribute to post-Millennium Development Goals (MDG) process, and facilitate access to financing and technical assistance, etc.? Integrated programming: 1) Has the WASHplus activity been effective at integrating WASH with other activities, including IAP, health, and non-health initiatives such as nutrition? 2) What new approaches have been tested, documented, and disseminated? 3) Does this integrated programming provide added benefits that are not observed for independent programming? 4) If WASHplus integrated programs, is there any carryover effect for other organizations to integrate programming? Benefits of the WASHplus core mechanism for Missions: 1) What is the impression and efficiency of a Washington-based core mechanism within USAID Missions? 2) Did the project meet the missions’ needs? If so, which needs? 3) In what areas were needs unmet? 4) How is a Washington-based core mechanism for WASH effective? Project management: 1) Why were many of the WASHplus projects delayed? Global leadership (including public private partnerships and knowledge management): 1) Have the WASHplus activities demonstrated “thought leadership” in the sector and how? 2) Specifically, what forums have WASHplus led or participated in, what written and oral (e.g., recordings of webinars) resources have they developed, and what is the impact of the knowledge management activities? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 4 2. METHODOLOGY The evaluation team used a mixed-method approach for data collection and analysis to gather comprehensive feedback on improving the implementation of the current WASHplus and for future environmental health investments. The primary sources of information for this evaluation were stakeholder interviews and document review. As per the statement of work, visits to field programs were not part of the evaluation methodology. 2.1.Data Collection Approaches The following activities were conducted as part of data collection: Document review: USAID provided the evaluation team with various WASHplus program￾related documents, including annual work plans, annual and quarterly reports, and other relevant documents for the seven program countries participating in WASHplus (Annex 4). The team conducted a thorough review of the quarterly and annual reports and work plans. The WASH sector specialists on the evaluation team read key documents in detail to gain an understanding of the program and to be informed during the interview process. The team also created profiles for each country program, which included the deliverable status, duration of program, funding levels, and other general information based on the program documentation received (refer to Annex 2). The WASHplus team provided financial information (Annex 6) about the project to the evaluation team. This information provided an overview on obligated funds and expenditures of WASHplus. Stakeholder interviews: WASHplus DC-based staff and a representative from the USAID Environmental Health team provided the evaluation team with a list of 81 potential stakeholders for interviews. With their assistance, the stakeholders were assigned to one of seven categories: • WASHplus staff in Washington, DC • WASHplus staff in-country • Global partners • Government representatives • USAID staff in Missions • USAID in Washington, DC • Local implementing partners Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 5 The evaluation team developed interview protocols (Annex 5) for the aforementioned stakeholder categories. These interview protocols provided a short list of questions to guide a structured interview. The list of interviewees was prioritized to ensure that a suitable number of respondents were interviewed in each country and category, and to reflect the extent to which each respondent could contribute to an understanding of the WASHplus project. As interviews progressed, respondents proposed additional people for the evaluation team to contact; these were added where possible. The team also requested that the WASHplus team suggest additional respondents to fill gaps. A total of 55 interviews were completed (67 percent response rate). Each interview was led by one (or in some cases, both) of the WASH sector experts, with another member of the team taking notes. Five of the interviews were conducted in French by two members of the evaluation team, using French translations of the relevant interview protocols, and three of the interviews were with individuals working on IAP and clean-cook stoves. The interview respondents were evenly distributed between the seven categories and across the seven country programs (see table 1 below). With the exception of government representatives, interviewees from almost all respondent categories were contacted in each country. It was difficult to identify government representatives with suitable knowledge of the WASHplus project in three of the country programs (i.e., Bangladesh, Madagascar, and Uganda). The evaluation team requested, when possible, to interview government representatives at both district and central level, a request that brought in three respondents in Kenya. Table 1: WASHplus Evaluation Interviews by Country and Respondent Category CATEGORY/ COUNTRY WASHplus staff in DCi WASHplus staff in country Global partnersii Govt reps.iii USAID staff in missions USAID staff in DC Local imp. partners TOTAL Bangladesh 1 0 1 2 4 Benin 1 1 1 1 4 Kenya 1 3 1 1 6 Madagascar 1 0 1 1 3 Mali 1 1 1 0iv 3 Uganda 1 0 1 2 4 Zambia 1 1 1 1 4 Global/Other Countries 8 1 9 DC 10 8 18 TOTAL 10 7 8 6 8 8 8 55 i Included interview with one individual involved in indoor air pollution and clean-cook stoves. ii Included interviews with two individuals involved in indoor air pollution and clean-cook stoves. iii Suitable government interviewees could not be identified for Bangladesh, Madagascar, and Uganda. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 6 iv Interviewees from local implementing partners were not provided for Mali. Each interviewee signed a formal consent form agreeing to the interview and, if respondents agreed, the interview was recorded. Interview transcripts were typed and collated for data analysis. The evaluation statement of work (SOW) required focus group discussions, but after consultation with the USAID, it was agreed that these were not practical given the dispersed nature of the stakeholders. Information from the interviews was analysed quantitatively in which respondents were asked a “yes” or “no” question. When appropriate, the results of these questions are reported. Upon completion of the interviews, WASH sector specialists reviewed the transcripts to determine trends and themes arising from the responses. Illustrative quotes were extracted to provide firsthand insight into the opinions of the respondents, some of which are provided in this report, as applicable. Review of WASHplus Website and Webinars Website: Both the functionality (such as functional links) and the content of the WASHplus website were reviewed to assess whether or not the site was appropriate, useful, and contained current material that would be beneficial to a wide range of potential users. The evaluation team also reviewed all the characteristics above from the perspective of francophone users, given that of the seven WASHplus country programs, three of them are francophone: Benin, Mali, and Madagascar (where Malagasy is the second official language). The evaluation team evaluated the availability of French materials throughout the website through two levels of searches. The first￾level search included browsing through the landing pages of the main sections of the website. The second-level search included opening 50 percent of the hyperlinks provided in the text of each section’s landing page. Webinars: The WASHplus team provided data on the subject, timing, and attendance of webinars. The evaluation team analyzed these data, which are graphically presented later in this report. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 7 2.2. Study Limitations Limitations of interviews: Given the global distribution of the interviewees, the SOW required that interviews were conducted via teleconferencing (refer to Annex 1 for data gathering activities). This accommodation sometimes led to difficulties in sound quality that could limit understanding. To mitigate this challenge, two members of the evaluation team participated in each call. Team members cross-referenced interview transcriptions after the calls. In cases in which the interviewees gave permission, the evaluation team recorded the calls. It was anticipated that there would be reluctance among some respondents to be frank. To mitigate this reluctance, the evaluation team clearly stated to each interviewee that his or her responses would remain anonymous and identifying comments would not be used. In addition, each interviewee signed a consent form that also explained the confidentiality of the interviews. Given the time differences and the time commitments of stakeholders, it was difficult to schedule interviews with some key interviewees. This difficulty was mitigated by having two lead interviewers on the team, so that, for example, two early-morning interviews could be conducted in parallel. Additionally, a small minority of people had less formal, shorter interviews, and a small number were asked specific questions via e-mail. Limitations of document review: • Gaps in reporting: Some of the countries did not provide reports for every quarter. For example, there were only two quarterly reports for Uganda over a 1-year period. • Number of beneficiaries: Quantitative data on beneficiaries were not systematically available for all countries. The exception was Zambia, which had thorough documentation of the targeted number of beneficiaries and the actual numbers achieved. • Quality of descriptions of project outcomes: Descriptions of some of the project objectives and results were vague, making it difficult to set coherent criteria to evaluate project outcomes. Because of the limited beneficiary data available, the evaluation team obtained approval from the USAID to not conduct an analysis of beneficiaries. The scarcity of information on project results led the team to probe for more details from the interviewees (refer to Annex 5 for interview questions). Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 8 3. FINDINGS AND CONCLUSIONS 3.1.USAID Context In the last few years, USAID has embarked upon specific organization-wide initiatives that determine how mechanisms such as WASHplus are to be implemented. These strategies and initiatives essentially define how the agency intends to carry out its activities both in general, and specifically in the WASH sector. The evaluation team reviewed six initiatives of USAID; many stress an emphasis on increased engagement with local partners and building the capacity of and working through national and local governments USAID Forward Introduced in 2010, a ground-breaking new strategy titled “USAID Forward” proposes a major shift in the way USAID works based on principles that “reflect the global discourse on development that has evolved over the last decade.”1 The strategy aims to create “a critical shift in the way we administer assistance, placing a greater emphasis on public-private partnerships and working with local governments and civil society organizations,”2 directing USAID staff to work more closely with partners in-country to ensure sustainable results. The stated objective of the strategy is far reaching: “working ourselves out of business and replacing our efforts with those of responsible institutions, vibrant private sectors, and thriving civil societies . . . our goal is to use assistance and development to help nations realize their own potential, [develop] their own ability to govern, and become our economic partners.”3 1 USAID Forward Progress Report, 2013. 2 ibid. 3 ibid. Eradicating Poverty by 2030 USAID has also committed to support the eradication of extreme poverty by 2030. In early 2013, President Obama stated, “The United States will join with our allies to eradicate extreme poverty in the next two decades by connecting more people to the global economy; by empowering women; by giving our young and brightest minds new opportunities to serve and helping communities to feed, and power, and educate themselves; by saving the world’s children from preventable deaths; and by realizing the promise of an AIDS-free generation, which is within our reach.” USAID responded that “. . . building on a wealth of expertise, strategic thinking, more targeted resource allocation, and global partnership platforms, we are well￾positioned to seek smart, sustainable solutions to extreme poverty.” The discussion paper “Getting to Zero” points out that poverty has many dimensions, which income alone does not Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 9 capture, and refers to the Multidimensional Poverty Index (MPI) developed by UNDP. The MPI includes access to water and sanitation as two of the 10 indicators used to calculate the index. Water and Development Strategy, and 2014–2025 Multi-Sectorial Nutrition Strategy At the same time, the agency has developed two new strategies that directly relate to water sanitation and hygiene. The Water and Development Strategy covers the period 2013 to 2018 and outlines the approach of the agency in both water for health and water for food, and states that it draws on USAID Forward “by supporting host country systems and emphasizing an integrated approach to development.”4 The strategy includes a target of reaching 10 million people with an improved water supply and six million with improved sanitation. USAID’s 2014– 2025 Multi-Sectorial Nutrition Strategy is described as the first of its kind at USAID. The strategy’s multi-sectorial approach addresses both direct and underlying causes of malnutrition, and focuses on high-impact actions across health, nutrition, agriculture, water and sanitation, and humanitarian assistance programs. It includes WASH as a “nutrition-sensitive intervention” and contains a well-referenced subsection on WASH outlining “essential WASH actions.” These include hand washing with soap, the treatment and safe storage of drinking water, and the sanitary disposal of human feces. The strategy singles out sanitation in particular and cites its strong correlation with stunting: “even in the absence of diarrhea, a fecal-contaminated environment is linked to environmental enteropathy.”5 Sanitation and Water for All (SWA) USAID has also been an active participant in the Sanitation and Water for All (SWA) partnership. This alliance of 90 partners, including governments, donors, civil society organizations, and other development partners, started in 2010 and aims to create a cycle of robust planning, institutional strengthening, better use of resources, and higher investment to achieve universal access to water, sanitation, and hygiene. Partners agree to adhere to the SWA Guiding Principles, which are largely based on the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action. The principles include country ownership, alignment, harmonization, managing for results, mutual accountability, and predictability. The principles aim to facilitate improved domestic investment, increased donor flows through country systems, and more effective use of all resources in the sector to ultimately achieve sustainable access to services for everyone. USAID has been a member of the partnership since its first biennial High￾Level Meeting in 2010. 4 USAID Water and Development Strategy, 2013–2018. 5 A subclinical condition of the small intestine caused by constant fecal-oral contamination and resulting in blunting of intestinal villi and intestinal inflammation, and consequent decreased nutrient absorption and infiltration of microbes. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 10 Further, at the most recent High-Level Meeting of the SWA partnership in April 2014, USAID Administrator Rajiv Shah announced numerous new commitments on behalf of USAID. Many of commitments reflected the direction described in the USAID Forward strategy. For example, the administrator stated that the agency wanted to “identify a partnership framework that recognized that the majority of resources going towards [WASH] efforts come from countries themselves, and we play a modest but sometimes catalytic role.” Water and Development IDIQ (WADI) More recently, the Water Office within USAID issued a water and development indefinite delivery indefinite quantity (WADI) contract to “carry out task orders that reflect the [agency’s] strategic direction and focus in water programming.” Under the contract, USAID wishes to “acquire innovative and effective practices in the water sector supporting USAID’s work in sustainable water, sanitation, and hygiene, and in enhancing food security through productive use of water in agriculture . . . [and] provide technical services and support to USAID Missions, Bureaus and Offices worldwide in implementing priority water sector activities.” The total ceiling under the IDIQ is $1 billion dollars. WASH is a significant part of the IDIQ, and awardees will provide services that include various WASH related activities, technical assistance services, capacity building and institutional strengthening (specifically capacity building for USAID staff), and training of technical personnel within host-country institutions on topics including WASH. The indefinite delivery indefinite quantity (IDIQ) contract is designed to support the vision and strategic objectives as presented in the USAID Forward policy framework and the water and development strategy, which includes “building local capacity, nurturing lasting institution systems, and capacities in developing countries that enable them to confront development challenges effectively.” The IDIQ states that the proposed contract “will enable USAID Missions to implement and manage water and sanitation programs that embrace a broad range of policy, legal, regulatory, and institutional reform efforts. These reforms will anchor the long-term transformation of the water and sanitation sector, leading to sustainable service provision on an equitable basis.” The services to be provided by awardees also include supporting innovation, organizing seminars and workshops, and conducting sustainability analyses of USAID water and sanitation sector programs. This IDIQ and the other strategic plans demonstrate that USAID plans to continue its support of the WASH sector. Conclusion: The WASHplus mechanism has operated within a context of evolution at USAID. USAID will expect these interventions to be consistent with the strategies that it has recently developed, and closely coordinated with the other WASH activities of USAID. The evaluation team concludes that future investments and programming should reflect this evolution. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 11 3.2.Benefits of WASHplus as a Core Mechanism for Missions USAID Missions decide whether or not to use the services of a central mechanism such as WASHplus, and the mechanism is expected to respond to requests from Missions without the mandate to approach them directly. Interviewees have voiced their opinions on the benefits and challenges of using a centrally managed mechanism in general, and WASHplus specifically. Findings from the interviews and financial data provided by WASHplus helped to determine the involvement of USAID Missions and their financial contributions (refer to Annex 6 for obligated fund and expenditures). Finding 1: The WASHplus mechanism offers some significant benefits • The WASHplus mechanism is perceived to be easy and fast: Many respondents believed that the WASHplus mechanism was both easier and faster than using other mechanisms available to USAID Missions undertaking WASH programming. Missions variously referred to WASHplus as “a quick way to access program funds,” a “learning project while designing something larger,” and a way to put “less burden on the mission.” It was also used by at least one Mission (in Kenya) to make very strategic use of a small amount of funding. • WASHplus offers other opportunities to Missions: In some cases, Missions were attracted to the WASHplus mechanism because of the particular partners involved. Many respondents acknowledged that the WASHplus mechanism is expensive, but as one respondent said, it was “worth it to work with partners we felt were qualified.” Some respondents referred to the fact that WASHplus could help Missions that do not have WASH experience. One interviewee said, “For people who [do not] have a WASH background who need to do a WASH activity, WASHplus is great.” Finding 2: There are numerous challenges related to the current WASHplus mechanism • Despite the benefits that are available to Mission staff through advice and technical assistance provided by WASHplus, it was evident from numerous interviews that there is a growing interest within Missions in undertaking WASH programming directly. One respondent described the WASHplus centrally managed mechanism as a “middleman.” Several USAID respondents referred to the new USAID Forward strategy, which they felt meant that, in the future, they would “work directly with NGOs and [the] government” and not with WASHplus. As stated by another respondent, the introduction of new strategies such as USAID Forward could mean that “as Missions give funds directly to NGOs, the role of WASHplus will be smaller.” • Mission staff members need access to WASH expertise, particularly around working with governments (Ministries of Health, Water, and Education). This expertise is important as Missions will be implementing larger WASH projects in the future. The capacity of Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 12 mission-level staff to design and support WASH programming is uneven. There are very few Missions with dedicated WASH staff, and Missions were acknowledged by a respondent within the USAID itself to be “weak” when it comes to WASH. Capacity in Africa, in particular, is lacking, which is a systematic problem. As one respondent said, “One week of WASH training won’t fix this.” Cross-country learning was also acknowledged to be difficult because being absent from their posts is difficult for people. USAID staff expressed frustration that they would “never be a global player without stronger WASH people.” • Short time frame: Several respondents commented on the short time frame (5 years), and the problems it posed for Missions. This short time frame is particularly problematic as the current WASHplus project lost an entire year at the beginning due to the suspension of the original lead contractor just after start-up. Several respondents pointed out that to start something with WASHplus when it only had 1 or 2 years left to run was not practical, or that it had been hard to have an impact in the short time frame. The evaluation team was also told that, in some cases, valuable time was lost in extended negotiation with the Missions about approaches and work plans. • Loss of control using WASHplus mechanism: Several Missions commented that WASHplus was “too far away,” and that they did not like the indirect communication with implementing partners. Several respondents commented on the fact that they “lost control” when they used the WASHplus mechanism. • Limited sharing of WASHplus materials with colleagues: As noted in the knowledge management section (see Section 3.6), USAID mission staff often appreciated the material made available to them by WASHplus, but they seem to rarely pass it on to local colleagues. This observation was noted in particular in francophone countries as limited WASHplus material is available in French. • Misunderstanding of the role of WASHplus mechanism: Among some respondents in Missions, there seemed to be a misunderstanding of the role of a Washington-based mechanism. Some appeared to be under the impression that working with WASHplus would keep them “on the radar of USAID in Washington,” with one respondent saying that it “allows them to have interaction with Washington.” Another respondent said they wanted a mechanism that could “lobby for funding for [their country] within the USAID system.” • Lack of demand from USAID Missions: An important indicator of the perceived benefits to Missions of the WASHplus project is the small number of Missions that have actually used the mechanism (eight to date) Out of a total of 109 Missions globally, 30 were targeted as MCH priority countries by the Global Health Bureau and as the focus for WASHplus. While all of these Missions could not have been expected to buy into the mechanism, the original ambition of the program suggests that more than eight were expected to use it. WASHplus is expected to reach about 44 percent of its (admittedly Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 13 high) budget ceiling due to low Mission buy-in. Of concern is the fact that there seems to be reluctance on the part of some USAID staff in Washington to recommend WASHplus as a mechanism to Missions. Lack of confidence in WASHplus seems to have resulted in a type of vicious cycle, whereby WASHplus is not recommended to Missions, so few Missions use WASHplus, resulting in a narrow scope of experience that further undermines the credibility of the program. There was an acknowledgment of this in some of the interviews; one USAID interviewee said that “we never really bought in ourselves” to WASHplus. Review of Financial Data WASHplus has worked at the global level and in eight country programs to date: Bangladesh, Benin, Kenya, Liberia, Madagascar, Mali, Uganda, and Zambia (refer to Annex 2 for country profiles). These country programs have included activities in a range of geographic areas (e.g., rural, urban and peri-urban) and subsectors (see Annex 3 for detailed profiles of each country program). Table 2 summarizes the main WASHplus activities by country. Table 2: Summary of WASHplus activities by country Country and start and end dates Main partners Expenditures to May 31, 2014 Obligated budget Main activities Zambia 9/2011– 9/2015 FHI360 CARE $10,001,968 $13,690,448 WASH in schools Hygiene education Behavior change Menstrual hygiene management Liberia 9/2012– 10/2012 FHI360 $110,739 $127,260 Assessment of IWASH program Bangladesh 9/2012– 5/2014 FHI360 Winrock Water Aid $1,277,589 $4,863,058 Water supply, including techno innovations Improved sanitation in challenging terrain Improved local and community governance WASH-Nutrition Integration IAP-including consumer segmentation and WTP study Mali 4/2013– 4/2015 CARE FHI360 $515,810 $2,500,000 WASH-Nutrition Kenya 9/2011– 9/2014 FHI360 $1,822,032 $2,160,522 WASH-HIV Community-led total sanitation IAP Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 14 Country and start and end dates Main partners Expenditures to May 31, 2014 Obligated budget Main activities Madagascar 8/2011– 11/2013 FHI360 CARE Winrock Water Aid $1,148,665 $1,158,000 Sanitation Fecal sludge management Benin 10/2012– 10/2015 FHI360 $286,905 $750,000 Urban Hygiene Improvement Program Uganda 12/2012– 11/2014 FHI360 $269,364 $500,000 Strengthening district competencies to budget, manage, and implement WASH (CLTS, school/ MHM, rainwater harvesting) WASH-HIV WASH-Nutrition WASHplus in the context of USAID investments in WASH Finding 3: Obligated funds are currently just over a third of the $100 million WASHplus ceiling, which, though high, constitutes a small fraction of USAID funding to WASH. Some WASHplus country programs are not close to using their obligated budget. USAID is investing in WASH with an average budget allocation of $318 million a year between 2003 and 2011 (see table 3). WASHplus has channeled expenditures of approximately $23 million in the first 3 and a half years of its operation, an average of $6.5 million a year, which is equivalent to 2 percent of average annual USAID investment in the WASH sector over the last 9 years. Even at its highest expenditure year so far, with slightly over $10.7 million channeled through WASHplus in Year 3 of the project, the program only accounted for 3.4 percent of the average annual USAID expenditures on WASH (see Annex 6 for WASHplus obligated funds and expenditure). Table 3: USAID budget allocations for the water sector by thematic area (USAID Water and Development Strategy, 2013–2018) The WASHplus program has a budget ceiling of $100 million over the 5 years of its duration. The total funds obligated to date have been $36,547,543, just over one third of the $100 million Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 15 ceiling for the program. As of May 31, 2014, 44 months through the 60-month duration, $23,848,704, (about 65 percent) of these obligated funds had been spent (see Annex 6 for detailed budget figures). Partners with the largest expenditures are FHI360 (64 percent), CARE (24 percent), and WaterAid (5 percent). CARE primarily operates in Zambia and Mali, and WaterAid in Bangladesh. The expenditure figure for FHI360 includes field-based operations and core program costs (figure 3). While many countries are on track or close to completion, several are behind in their budget utilization, including Bangladesh, Benin, and Mali, (figure 1). This is most likely due to later start-up in many countries, with funding and activities not starting until the second or third year. This delay was in part due to the suspension of the original lead contractor just after start-up, and the transfer of the assets of this contractor to FHI360, who took over leadership of the program. As can be seen in figure 2, in Year 1, other than WASHplus core spending, only Madagascar utilized any of its budget, and in Year 2, only slightly more than half of the expenditures were in the field. Figure 1: Obligated funds spent as of May 2014 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 16 Figure 2. Expenditures by country as percent of annual total Figure 3. Expenditures by partner Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 17 Finding 4: Countries with a significant service delivery component account for a large portion of expenditures. Zambia and Bangladesh, the two countries with large service delivery components, account for 49 percent of total expenditures to date. Other countries and global expenditures account for 19 percent of the total, and core costs account for 32 percent of the total (figure 4). The relatively low expenditure at country level has resulted in the core program budget (which includes global leadership, integration, and innovation activities, as well as partnership contributions) costs constituting a significant proportion of project costs; 32 percent to date and a projected 30 percent once all obligated funds are spent. Figure 4. Expenditures by country (to May 31 2014) USAID Missions decide whether or not to use the services of a central mechanism such as WASHplus, and the mechanism is expected to respond to requests from Missions without the mandate to approach them directly. The evaluation team’s analysis reveals that as a result of the requests it received, WASHplus became a combination of two different approaches: service delivery at scale, largely by NGOs in two countries (i.e., Zambia, Bangladesh), and a variety of smaller-scale interventions in others. This analysis raises questions as to whether or not such a combination is consistent with the original ambition of the program as expressed in the strategic objectives: “moving beyond basic service delivery through NGO projects, WASHplus uses innovative approaches to expand the access and use of WASH services, such as the development of innovative approaches for WASH financing; building the capacity of the local private sector for the provision of WASH products and services; and institutional strengthening, especially at the lowest levels of governance.” Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 18 Conclusion: Some WASHplus country programs are behind in utilizing their budget. Countries with large service delivery use a great portion of the WASHplus expenditures compared with other small-scale programs. WASHplus accounts for a small portion of USAID expenditures on WASH, but its strategic objectives, set by USAID, were ambitious. It appears that the demand￾driven model used has not been consistent with these objectives, particularly with the objective of “moving beyond basic service delivery through non-governmental organization (NGO) projects.” The benefits and challenges of the WASHplus central mechanism, combined with the evolving strategies within USAID and the ongoing commitment of the agency to WASH, suggest that as Missions are looking at larger WASH projects, they will be less likely to use a centrally managed mechanism for projects that include service delivery, and will work directly with local NGOs and national and local governments. Moreover, it is also evident that the need for advice, training, and technical assistance will be greater, particularly for projects that aim to work through national governments, local governments, and local implementing agencies. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 19 3.3.Impact at Country Level To have an understanding of the approaches used by WASHplus to achieve increased coverage of high-impact WASH interventions, including leveraging resources, engaging other partners such as the private sector, and developing evidence to guide improved programming, the evaluation team investigated: • Quality of service delivery and impact on national policy and capacity, and • Engagement with the private sector. Quality of service delivery and impact on national policy and capacity Finding 5: WASHplus has successfully undertaken service delivery in its country programs with programming designed to positively impact the lives of the poor. The evaluation team focused primarily on two types of impact6 at the country level: quality of service delivery and impact on policy and capacity. From the interviews, the evaluation team gathered many positive responses about the quality and impact of WASHplus in the countries where the mechanism was utilized. For example, the Community Connector program in Uganda to integrate WASH and nutrition was positively reviewed by all interviewees in the country and described by all as “very successful.” In Kenya, one respondent said, “Small, doable actions have been well taken by the sector” and that the work on HIV/AIDs has been “critical.” Another respondent stated that the Kenya “HIV/AIDS handbook and guidelines developed by this project belong to the government of Kenya and are integrated in its policies.” In another country, respondents shared with the evaluation team that the WASHplus project had made a state governor “change his way of thinking.” A respondent in Madagascar spoke of having the “ability to innovate” through WASHplus. Another spoke of Madagascar as being the “ideal scenario,” saying that “the sector was dynamic, the Mission was dynamic, and there was strong support.” Respondents also told the team that the cook stove program has been effective in Bangladesh. One respondent said, “The consumer segmentation study has impacted manufacturers and distributors to offer options that are more desired by consumers.” Many respondents singled out specific members of the WASHplus team and spoke enthusiastically of the excellent support and advice they had received. In terms of the effectiveness of service delivery provided through WASHplus, interviewees generally stated that programming approaches are of high quality. Most of the service delivery 6 Despite the use of the team “impact” in the SOW, the evaluation team clarified with USAID that the assignment was not a conventional impact evaluation. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 20 under WASHplus was provided in the two countries with major implementation components: Zambia and Bangladesh. Competent implementing partners were used in both countries. Finding 6: The WASHplus mechanism could have been used to place a stronger emphasis on government capacity and leadership. The influence of WASHplus on government policy and capacity does not appear to have been significant in all program countries. In some countries, interventions were small or at pilot scale, and respondents commented that they felt it would take some time before the policy implications became apparent. In the case of Zambia, the evaluation team learned that the Schools Promoting Learning Achievement through Sanitation and Hygiene (SPLASH) program had an influence on government, particularly in terms of including WASH in schools as an indicator in government monitoring. However, there were still calls for government leadership in the sector to be further fostered, and for capacity for the future to be built. As one respondent explained, “existing government structures should be used” and “the ministry should give direction.” The USAID Mission in Zambia ending the project 1 year early certainly limited the potential influence of WASHplus in institutionalizing WASH in schools within the Ministry of Education, as the final year was designed to be a year of consolidation and building of district government capacity. The feedback from respondents indicated that, in general, the quality of the implementation in Zambia was good, however, there was an apparent difference of opinion over approach between the implementing agencies and the Mission in Zambia. The initial design of the project in Zambia called for strengthening and sub-granting of funds to the District Education Boards to allow them to lead on school WASH facility construction. However, the priority of the Mission was to achieve progress in terms of beneficiary numbers. In response to the Mission’s request, FHI360 and CARE contracted directly with local artisans in the initial stages and only later developed a “hybrid” approach that included direct service delivery and the sub-granting of funds to the District Education Boards to facilitate the planned capacity building, leadership role, service delivery, and strengthening of operation and maintenance systems. The evaluation team interviewed district-level government staff in several countries, and the findings suggested that WASHplus had effectively provided implementation support, but had not necessarily been able to significantly increase government capacity. One respondent within a district government stated that “there has been no planning for long-term sustainability—too short and no real exit strategy.” The emphasis on beneficiary numbers, which emanated from the USAID Mission, seems to have undermined the building of government capacity. One respondent explained that “beneficiary Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 21 numbers are the primary numbers that USAID count and report rather than, for example, the number of district school WASH systems functioning in a sustainable way.” Finding 7: WASHplus was not designed, and has not been used, to help Missions to build skills for taking a government-to-government approach in the future. WASHplus has not been used to help Missions acquire skills to build government capacity. Engagement with the private sector Finding 8: Despite some attempts, PPPs and other forms of private-sector involvement have not evolved into a significant feature of the WASHplus program to date. Changes in program implementation or misalignment between services provided by selected companies from the private sector and the WASHplus project requirements did not lead to major accomplishments in terms of establishing PPPs. For example, an initiative with Unilever on WASH and newborn health in Kenya was initially to be implemented through WASHplus, but USAID decided instead to make it part of its flagship child health project. Likewise, potential opportunities to partner with Procter & Gamble and Medentech were initially promising, but fell through mainly because of a lack of compatibility between what technical inputs the company had to offer and what WASHplus projects required. In Zambia, Medentech’s Flogenic technology was designed to work with piped water systems rather than the borehole systems used by WASHplus. In Benin, it was found that chlorine levels in water were not the issue as they were originally thought to be, so technical assistance from the private sector was no longer required. There has been some success with small-scale private sector engagement. WASHplus has had successful engagement with the private sector in the IAP program in Bangladesh, particularly with regard to using the private sector to build latrine slabs in the country. Other successes include the development of public-private ventures to develop community WASH blocks in Madagascar, the training of local artisans such as masons in some countries, and working with a local company in Zambia to produce menstrual hygiene management kits for distribution to schools. WASHplus continues its programmatic partnership with the global Public Private Partnership for Handwashing (PPPHW) by participating on the steering committee and providing technical assistance. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 22 Conclusion: WASHplus is seen as a program that has had a significant impact and implemented effective service delivery in the countries where it works. However, the WASHplus mechanism has not been used to place significant emphasis on building government capacity and leadership. More could be achieved in terms of building the capacity of the local private sector for provision of WASH products and services and to develop new PPPs, as called for in the strategic objectives. 3.4.Integrated Programming The evaluation team looked into the effectiveness of WASHplus activity integration in WASH with other activities, including nutrition, HIV/AIDS and education. The team also evaluated the effectiveness of WASHplus activities in IAP. Finding 9: WASHplus has successfully incorporated integrated programming, has made many related resources available, and has undertaken innovative country-level integrated programming. The evaluation team found that WASHplus has successfully been a leader in developing integrated programming, for example, WASH in Schools, WASH and nutrition, and WASH and HIV-AIDS. WASHplus and its predecessor, the Hygiene Improvement Project (HIP), took an early lead in the area of integration, while other entities such as the World Bank, UNICEF, and groups within USAID are also engaged in such initiatives. As of the time of this evaluation, integration initiatives have become of interest sector-wide. This observation is particularly relevant in the case of WASH and nutrition. During the 2 years that it has taken to develop a joint document with UNICEF, USAID, and WHO on WASH and nutrition, other actors, including within USAID, have embraced and developed many of the ideas. As one respondent said, “The problem is that now [the WASHplus supported document] is no longer as new and interesting as when they started . . . the challenge is to make sure they contribute something new.” The WASHplus website provides a range of resources related to the integration of WASH with sectors such as HIV/AIDs and nutrition. These include reports, toolkits, recorded webinars, and training manuals developed over the past 4 years. These resources offer the opportunity to learn about activities in WASHplus country programs, and to use materials for learning and/or teaching, etc. The work of WASHplus in WASH and HIV/AIDS integration was particularly appreciated. One respondent said, “WASHplus has been a leader and has influenced the country at national level.” However, the same respondent also said that WASHplus “could have done better at knowledge management and dissemination” on the subject of HIV/AIDS and WASH integration. Another Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 23 respondent commented on the fact that a focus on HIV/AIDS and WASH integration is particularly important, as it “targets people who have been marginalized,” but the respondent also commented that scaling up is needed. In this interviewee’s opinion, much more training of local-level staff is required, particularly government staff. Respondents shared that structural problems (outside of the control of WASHplus) within development agencies (including USAID) have also contributed to difficulties in developing integrated programming at the country level. For example, respondents told the evaluation team that integration was sometimes seen as a “new cost” often added after programming had already started and work plans were developed. As one respondent said of other USAID staff, “They are convinced, but their annual plan has been approved.” Another mentioned, “It is hard to integrate with other sectors unless it was initially planned and budgeted.” Clearly, the existence of silos within development agencies is still an obstacle to integrated programming. As mentioned in the SOW, the evaluation team had to determine where integrated programming provided added benefits that are not observed in independent programming. Because the team was not performing an impact evaluation, they and USAID agreed that this question could not be addressed empirically. However, several of the interview protocols posed the question to interviewees, all of whom confirmed that, in their opinion, integrated programming yields greater benefits for beneficiaries, but these responses did not seem to be directly related to the specific experience of WASHplus. Conclusion: WASHplus has energetically embraced integrated programming, has made many related resources available, and has undertaken innovative country-level integrated programming. Integrated programming is becoming an accepted approach, both within USAID and within other agencies. Nonetheless, understanding the institutional obstacles to making integrated programs work are not yet well understood. Finding 10: The IAP component of WASHplus has had a significant reach in IAP globally, and success in applying lessons from behavior change programming in the WASH sector. However, WASHplus was unable to achieve significant country reach with IAP. The IAP project in Bangladesh has had some significant successes. Globally, the IAP component has had significant reach in IAP promotion and thought leadership activities, including the facilitation of working groups, technical committees, and workshops. However, because of a number of challenges related to IAP being linked to USAID’s energy rather than health programming, the IAP component has not been able to achieve the country reach foreseen at the outset of WASHplus. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 24 Although it was envisioned that IAP would be less than 20 percent of WASHplus efforts, the integration of both WASH and IAP with other sectors was originally a significant component of the WASHplus program, and was reflected in two of the four goals of the program: • Reduce diarrheal diseases and acute respiratory infections. • Integrate WASH and IAP interventions into education, HIV/AIDS, maternal and child health, and nutrition programs. However, there were a number of challenges in realizing these goals. WASHplus staff indicated that: • USAID informed WASHplus that they could not ask Missions to provide health funds for IAP programming. “IAP funds had to come from the energy/climate offices; accordingly, the project in collaboration with USAID readjusted expectations.” • WASHplus has to rely on USAID to make overtures to Missions and the staff stated, “Our USAID Environmental Health partners did not often have the necessary contacts with or access to mission energy offices.” Consequently, in the Year 3 Workplan, WASHplus informed USAID that activities related to IAP integration would no longer be a significant focus of the program. The Workplan states: “In Year Three, IAP integration will not be a major WASHplus effort; rather we will be open to opportunities and respond as feasible. WASHplus remains attentive to opportunities to promote IAP improvements alongside WASH program activities when appropriate, including identifying practices that outreach workers (agriculture, community health, WASH) can promote alongside WASH practices during household visits to limit exposure to IAP. We will look for additional opportunities to leverage resources in countries, such as the development and training of small￾scale entrepreneurs who will market products (e.g., point-of-use water treatment, soap, stoves, and sanitation products) and the development of credit and financing strategies that would be relevant for both areas.” Despite these issues, some important activities were undertaken. WASHplus staff informed the evaluation team that “The activities that have been funded have been primarily research. This was not the way the project was originally envisioned, but the way it has played out based on available funding and accompanying scopes.” Out of the $1.5 million target for IAP core funds, the total expenditure, as of June 2014, was $1,161,500. WASHplus staff, however, shared that the full $1.5 million will be utilized by the end of the program. IAP programming also includes $386,200 from the Asia Regional Bureau and $476,858 from the Bangladesh Mission for IAP activities in that country (refer to Annex 2 for Bangladesh IAP activity profile). The work in Bangladesh was the primary IAP country activity (aside from a study of cook stoves in Kenya). FHI360 and Winrock, funded by WASHplus, worked in partnership with the Global Alliance for Clean Cookstoves (GACC) on IAP in Bangladesh. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 25 The main activity in the Bangladesh IAP program was a consumer study of needs, preferences, and willingness to pay, along with a controlled cooking test stove testing study resulting in a detailed report and a technical brief. A market strategy and consumer segmentation study are currently ongoing. Interviewees spoke positively of the program in Bangladesh, noting that the study along with other interactions and advocacy work has led to the engagement of the Government of Bangladesh (GoB) in promotion of improved cook stoves (ICS) and developing a country action plan (CAP) for ICS. WASHplus provided in-line edits to and comments on the CAP draft, at the GoB’s request. The mechanism has also resulted in businesses offering a wider range of ICS to consumers. One respondent said, “Now there [is] a wider variety of improved stoves available, including designs from other countries. . . . Some manufacturers have modified their stoves and the government is promoting better stoves.” Interviewees shared that the World Bank has also been influenced by the study and now promotes a wider choice and range of cook stoves. While country-level activities have been limited, WASHplus IAP activities on a global scale have been successful. For example, WASHplus was engaged in a number of global activities to support the IAP sector, including support to the GACC in advocacy; creating a working group and community of practice; and supporting numerous other knowledge sharing activities such as working groups, technical committees, workshops, and webinars. Although the interviewees did not consider the links with WASH essential, they did note that there were some significant benefits, particularly related to the behavioral change communication experience in the WASH sector. A presentation by a WASHplus specialist on behavioral change was noted by one respondent as being particularly useful: “[The specialist] gave an excellent presentation that helped to integrate [behavioral] change lessons from the WASH sector [into the IAP context].” WASHplus co-presented a webinar with TRAction in May 2014 titled “Behavior Change Approaches to Enable Uptake and Use of Clean Stoves and Fuels: Lessons From the Field on What Works, What Doesn’t and What’s Next?” This webinar focussed particularly on lessons from TRAction projects, but there are plans to have an IAP webinar focussed on WASHplus closer to the end of the program. 3.5.Project Management As stated in their annual work plans, the WASHplus management strategy cuts across all strategic objectives. This strategy aims to strengthen in-country activities and increase sustainability over the long term as activity plans are agreed upon with the USAID and funding Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 26 becomes available. The evaluation team assessed the management successes and challenges in implementing WASHplus. Finding 11: WASHplus is well managed and the staff is highly responsive. Of the respondents in country offices who were asked about the timeliness of fund transfer, 80 percent said that it had been acceptable. Many respondents commented on the quality of communication with the WASHplus team and their prompt responses to e-mails at all times. Despite this, some respondents commented on the challenges with communication given that Missions are not expected to communicate with implementing partners directly, but must do it through WASHplus, which seemed to be particularly challenging for countries with a significant time zone difference. Of the 14 respondents who answered the question about whether technical assistance has been available when needed, 13 replied that it had, as one respondent confirmed, “There has never been a time when I needed support that was not there.” Finding 12: WASHplus has faced constraints due to the complicated contractual structure, involving multiple partners and multiple bureaucracies. Several respondents commented on the problem of multiple layers of bureaucracy. The combination of bureaucracy within the USAID, FHI360, and the implementing partners has slowed down processes and led to complications in some cases. For example, in one project country, procurement of motorcycles took 6 months. Also, delays in approval of the work plan by the USAID Mission in Uganda led to a delay in start-up of 1 year. Even though 75 percent of respondents stated that aspects of the partnership structure had supported success, more than 85 percent of respondents commented that the contractual structure had contributed to the challenges. While 88 percent of respondents (15 out of 17) stated that the reporting requirements were appropriate, there was some confusion about reporting structures given the complex structure and multiple actors in some countries. One respondent in a Mission simply said, “I do not really know what the reporting structure is.” The responsibilities of different actors were not always well defined, and at least one respondent referred to WASHplus engagement as “micro￾managing” and said that “it was confusing to help colleagues in the field navigate.” Finding 13: WASHplus is perceived to have not demonstrated flexibility with regard to staff assignments and the use of experts, and to lack the innovation required to optimize usefulness to the USAID. Some respondents commented on what one termed a lack of “nimbleness” on the part of the WASHplus team. The evaluation team heard similar quotes, such as “They were never agile.” There were also comments about their “conservative response” to requests to take on tasks, and another respondents said they do not have the “fluidity of staff that is needed,” and “good staff Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 27 are over committed, and they don’t bring in experts when they need them.” With respect to requests to take on tasks, there were comments on FHI360’s preference to work using its existing staff and its reluctance to take on consultants when specific skills or more capacity was needed. Respondents commented on the need for a better relationship, much more tightly integrated with USAID, in the future. Conclusion: WASHplus staff has been highly responsive to country offices, with good communication quality and promptness as indicated in the interview responses. Because of its contractual structure, WASHplus has also faced some constraints. These constraints have led the delay in start-ups of some activities. Interview responses also suggest that the structure and responsibilities through this centrally funded mechanism are often not clear. 3.6.Global Leadership and Knowledge Management The evaluation team was requested to assess whether WASHplus activities to date have demonstrated or contributed to thought leadership in the sector. Global leadership Finding 14: Interview responses suggest that WASHplus has not had a high enough profile to have significant influence at the global level. There are indications that WASHplus is leading in terms of innovation in certain areas, such as HIV/AIDS integration and the development of new latrine designs for difficult terrain. The accomplishments listed in the Years 1–3 Annual Reports are indicative of WASHplus’s efforts to have an influence at the global level. For example, at the end of its first year, WASHplus had engaged in meetings, planned and led discussions, and conducted webinars, including presenting at the Water, Engineering and Development Centre (WEDC) conference in the United Kingdom on “Combining Sanitation and Hand Washing Promotion: An Example from Amhara, Ethiopia” (refer to Annex 7 for more details of WASHplus engagement in global activities). Global partners interviewed during this evaluation were aware of what WASHplus was doing and were appreciative of some of the global work. The PPP for Hand Washing (PPPHW) was particularly well regarded. A summary of achievements in the global leadership arena are listed in Annex 8. However, findings from interviews suggest that WASHplus has neither a high enough profile, nor an adequately strategic approach to have a significant influence at the global level. One interviewee stated: “[WASHplus] missed opportunities to engage, for instance, with Africa Water Week or AfricaSan. They could have had a higher profile.” WASHplus seems to have struggled to get the right balance between implementation and global policy advocacy. One respondent said they have had little policy influence and were “mostly disappointingly absent” as Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 28 they worked in so few countries. In general, there were calls for USAID to do more in the WASH sector. One respondent said, “In terms of the ability to influence, there is a huge missed opportunity in terms of how the U.S. could use its political capital and convening power.” WASHplus helped support the inclusion of a hand-washing indicator in both the Multi-Indicator Cluster Surveys (MICS) and the Demographic and Health Surveys (DHS). WASHplus contributed at the global level when the USAID asked WASHplus to assist them in engaging in the global discussions around Post-2015 targets for hygiene. One respondent stated that “[WASHplus] pulled together top experts into the Hygiene Working Group and commissioned a background report. . . . They were pragmatic but also visionary, for instance, getting menstrual hygiene management onto the agenda.” Another respondent suggested that there are indications that the WASHplus inputs to the Post-2015 debate were not entirely strategic, saying that they “have a good understanding of the practical challenges associated with improving hygiene on the ground,” but that they were “less effective in translating this into concrete proposals for monitoring progress at national and global levels.” Knowledge management Finding 15: Certain aspects of WASHplus knowledge management are highly valued by users, in particular the “WASHplus Weekly.” In general, anglophone respondents, in particular those based in the offices of global agencies or expatriate staff in U.S. Missions, were highly appreciative of the work WASHplus does to disseminate the latest research on WASH. In particular, the “WASHplus Weekly” briefings were found to be valuable and were mentioned by many of the respondents who answered the question about use of knowledge management resources. However, respondents who were program￾country government staff, local staff in U.S. Missions in-country, or the staff of local implementing agencies, were less likely to say that they had used WASHplus knowledge management resources. Only half of respondents in these categories said they had. Francophone respondents almost universally said they did not use them. The reason given by one respondent was that there are very few French resource materials available through washplus.org, an issue that is discussed in the next section. Finding 16: The reach of WASHplus knowledge management has been uneven. Although some resources are highly regarded, such as the “WASHplus Weekly,” the utility of others, especially at the field-level, has been limited. Several respondents commented on the mismatch between the knowledge management resources WASHplus offered and the needs of the broad WASH sector. One respondent stated there is an “uptake gap” in the sector that is due to a “westernized understanding of how knowledge is transferred.” Other respondents observed that although materials made available by WASHplus on recent research are useful and highly Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 29 accessible to a certain audience, they do not serve the needs of others. For example, district-level government staff are unlikely to want to read full journal articles, and also unlikely to have internet access that would allow them to download the resources. Respondents also mentioned not transferring WASHplus knowledge resources to their local partners because they were not suitable for this audience. The evaluation team found a lack of evidence of systematic cross-country learning. WASHplus has arranged for some country-level project stakeholders to attend regional or global conferences, including AfricaSan, WEDC, and the UNC Water and Health Conference. Of the 13 respondents who were questioned about this aspect, only four said that they had been able to share experiences and information with other countries through WASHplus (seven said no and two gave ambivalent answers). Webinars Finding 17: Webinars for which attendance data are available show they attracted more than 500 participants, and that between 14 and 28 percent of attendees have been from developing countries. WASHplus developed seven webinars with an average of two webinars per year. These seven webinars attracted more than 500 attendees and more than 2,500 later online viewers. All but one of these webinars was hosted in partnership with other programs. The number of participants per webinar ranged from 43 to 132, and the number viewing after the event has been significant (table 4). Table 4. WASHplus webinars Webinar name Date Presenters Number of participants Number of developing country participants Number of YouTube or Slideshare views as of 7/7/2014 Lessons Learned from Working at Scale; WASHplus only event June 23, 2011 Julia Rosenbaum, WASHplus; Sarah Fry, WASHplus 43 12 (28%) 1,165 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 30 Webinar name Date Presenters Number of participants Number of developing country participants Number of YouTube or Slideshare views as of 7/7/2014 Leading with Sustainability—Laying the Groundwork for Sustainable Services; joint WASHplus/WASH Advocates/Global Water Challenge event Dec. 15, 2011 Braimah Apambire, Conrad N. Hilton Foundation; Andy Narracott, WSUP; Harold Lockwood, Aguaconsult 92 N/A 1,073 Inclusive WASH: WASH and HIV Integration; hosted by WaterAid Australia, with participation by WASHplus January 16– 17, 2012 Presenters and facilitators: Julia Rosenbaum, WASHplus; Lucina Schmich, Burnet Institute N/A N/A N/A Environmental Enteropathy and WASH; joint WASHplus/TOPS/USAID WASH and Nutrition community event Sept. 11, 2013 Laura Smith, Cornell University; Helen Petach, USAID 102 N/A N/A WASH Sustainability Index Tool; joint USAID/Rotary International H20 Alliance/ WASHplus event Nov. 21, 2013 Harold Lockwood, Ryan Schweitzer, Aquaconsult; Helen Petach, USAID 86 N/A 157 Behavior Change Approaches to Enable Uptake and Use of Clean Stoves and Fuels: Lessons From the Field on What Works, What Doesn’t and What’s Next?; joint WASHplus/ TRAction event May 7, 2014 Marc Jeuland, Duke University; Theresa Beltramo, Impact Carbon; Allen Namagembe, PATH 51 7 (14%) 111 WASH, Nutrition and Early Childhood Development: New Evidence in ECD and Findings from the Field; joint WASHplus/USAID WASH and Nutrition Community event June 25, 2014 Helen Petach, USAID; Jennifer Orgle, CARE; Maureen Black, University of Maryland 132 34 (26%) 109 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 31 WASHplus team members commented that they were “quite thrilled” with the levels of participation from developing countries given the challenges that connectivity and time differences pose. The webinars started relatively early in the program, during the third quarter (figure 5). No webinars were conducted for a full year from the second quarter of 2012 to the first quarter of 2013. Since then, they have become more frequent. WASHplus staff informed the evaluation team that this gap was due to the complexity of finding and working with partners. The WASHplus team expects an increased number of webinars as projects come to an end and more lessons learned become available for sharing. Figure 5. Webinars and estimated participants Social media Finding 18: There is a considerable volume of English content on the WASHplus website and Twitter, but limited content in French even though three of the seven country programs are francophone. The WASHplus project website–washplus.org–is a key part of the knowledge management function of the project. The website is well populated with current content and appropriate and useful resources, related directly to the program and external information. The “What’s New” and “Latest Updates” sections are current and give information about the latest research, reports, and conferences in a number of appropriate areas, including IAP, sanitation, WASH, and nutrition. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 32 WASHplus has a strong presence on Twitter; the account has 1,504 followers and achieved 76,700 retweets in 1 week in January 2014. This suggests that WASHplus is using Twitter as an excellent way to promote and disseminate information. Some of the WASHplus materials disseminated that month via Twitter included a review of hand washing literature, policy implications for improved cook stove programs, and a focus on WASH and nutrition. It is unclear how much of the Twitter activity is by users in developing countries. WASHplus also has a Facebook account, but it does not seem to be used well, having just 46 “likes.” Data on total hits on the website are unavailable. The visits to the WASHplus blogs or news sites for the program in Year 3 were: Sanitation Updates, 180,000; IAP Updates, 87,700; Urban Health Updates, 67,100; Household Drinking Water Quality, 81,000.7 None of the WASHplus website pages are offered in French by WASHplus. However, the website provides users with the option to have web content translated into 70 languages by Google Translate, a free online language translation service that instantly translates text and web pages. Aside from country pages for Madagascar and Benin in which reports originally drafted in French are available, the quality of the French translation through Google Translate is questionable due to the limitations of a machine translation. Other language materials are produced by WASHplus, but are not currently posted on the website. Conclusion: The number of attendees at the webinars suggests there is a strong interest in presentations organized by WASHplus, and webinars could be both more frequent and scheduled more systematically, without gaps. The availability of recorded webinars online seems to be an effective way to attract a greater audience from developing countries. The WASHplus website and Twitter are used effectively to communicate information. However, although three of the seven countries participating in WASHplus are francophone, the evaluation team found limited material available through the website in French. USAID seeks to take a leadership role in the sectors in which it engages. There is considerable opportunity to be the voice of the United States on specific policy issues relating to WASH, in particular those around sanitation and hygiene and the role of governments (especially the leadership of Ministries of Health). Opportunities exist to bring the learning from WASHplus country programming to global attention. Although WASHplus has been somewhat active at the global level, it could have higher profile. 7 “WASHplus Year 3 Annual Report,” October 2013 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 33 4. LESSONS LEARNED FOR THE FUTURE As defined in the SOW, the evaluation gathered successes, shortcomings, and lessons learned from project interventions to date to inform future WASH investments and activities of USAID. The lessons learned are as follows: • To be relevant, a centrally funded mechanism needs to have some influence on other USAID WASH investments—not just the funding that flows through the mechanism itself. This could be achieved through knowledge management, capacity building, and the development of innovations. • In the future, if a demand-driven approach is used, it would benefit from more widespread promotion to Missions, and for country-level project requests to be in line with the intended scale, goal, and strategy of the program. It is important to develop approaches that allow the learning and innovations from country-level activities to be widely adopted among a large number of USAID Missions. • Future investments and programming should reflect that USAID support to WASH will continue in the future. USAID will expect these interventions to be consistent with the strategies that USAID has recently developed, and closely coordinated with the other WASH activities of USAID. Given USAID’s new strategies and the evolution of development thinking, there is a need to develop approaches that foster government leadership, build government capacity, work with the private sector at scale, and increase the ability of U.S. Missions to work directly with the host government. • In the future, it will be important to provide Missions with advice and support necessary to work more effectively with national governments and local partners to respond to the imperative articulated in the USAID Forward strategy. • There is considerable opportunity to be the voice of the United States on specific policy issues relating to WASH, in particular those around sanitation and hygiene and the role of government, especially the leadership of Ministries of Health. • Future knowledge management could focus on resources relating to working with and through governments on water, sanitation, and hygiene, which would address the specific needs of the Ministries of Health to deliver on their mandate with respect to sanitation and hygiene. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 34 5. RECOMMENDATIONS The WASHplus program is scheduled to come to an end by September 2015, approximately 1 year from the time of this evaluation. This short time remaining on the current project makes it unrealistic to recommend dramatic changes, and the team’s understanding has been that the main purpose of this evaluation is to make recommendations to inform future environmental health investments by USAID (contained in a separate document). However, there are some changes and interventions that the evaluation team believes would be practical and achievable during the final year of the program: • The last year should have a strong focus on building country-level capacity and mechanisms to ensure sustainability of the interventions in the country programs. When possible, emphasis should be given to supporting existing institutions, especially national and local governments, to build upon WASHplus success once WASHplus support comes to an end. • Should USAID allow for an extension of six to 12 months (which the evaluation team deems advisable), the additional time period should be used to allow for further country￾level capacity development and support to government agencies where necessary. This extension would also give some additional time for countries where there were start-up delays or low budget utilization to meet their goals. • Knowledge management activities, especially webinars, should be ramped-up to disseminate the learning from WASHplus, and make a wider group of stakeholders aware of recent evidence and best practices in WASH. Knowledge management products and webinars should be available in both English and French, and should be targeted to a variety of audiences, including a larger proportion of in-country stakeholders. • The final knowledge management activities and products should focus not only on the successes, but on the constraints and challenges faced by the WASH sector and experienced through WASHplus activities. A thorough examination of the challenging aspects of activities such as achieving integration within development agencies, communicating the need for behavior change, and engaging with the private sector would and identifying mitigation strategies to these challenges, would be an excellent way of identifying additional available resources to other programs both within USAID and externally. • WASHplus should develop a proposed future research agenda for USAID, based on the needs identified and lessons learned through its activities. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 35 Annex 1: Evaluation of the WASHplus: Supportive Environments for Healthy Communities Project Cooperative Agreement No. AID￾OAA-A-10-00040 - Statement of Work I. USAID WASHplus Evaluation Activity: Conduct an evaluation of the WASHplus project to determine the effectiveness of the project, recommend revisions or adjustments to the work plan and program design, as necessary, and inform the design of future Environmental Health activities. II. Performance Period: May 1–August 31, 2014 III. Funding Source: USAID/GH/MCH IV. Objectives and Purpose of the Assignment The USAID Global Health Bureau’s Office of Health, Infectious Disease, and Nutrition (HIDN), Maternal and Child Health Division, requests technical assistance to carry out an evaluation of the WASHplus Project, HIDN’s flagship environmental health activity. The evaluation findings will be used to inform implementation of the project in Year 5, to inform design of future environmental health activities, and to ensure alignment with the new USAID Water and Development Strategy. The primary objectives of this evaluation are to: 1. Assess whether WASHplus is achieving its objectives and planned outputs as stated in the WASHplus agreement and project description, and in approved implementation plans. The performance review should evaluate successes, shortcomings, impact, and lessons learned from project interventions to date. 2. Make recommendations to improve implementation of the current project and to inform future environmental health investments. Include recommendations for strategic interventions post-2015 to both enhance the Agency’s Water and Development Strategy and to work toward ending preventable child and maternal deaths. Project Title: WASHplus: Supportive Environments for Healthy Communities Cooperative Agreement No. OAA-A-10-00040 Start and End Dates: October 1, 2010 to September 30, 2015 This evaluation will assess the portfolio of projects funded through the WASHplus activity. The primary audiences for this evaluation include the USAID AOR, the Environmental Health team, USAID MCH and Water Office staff, and WASHplus. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 36 V. Background The WASHplus Cooperative Agreement is housed in the Bureau for Global Health’s Office of Health, Infectious Diseases and Nutrition (GH/HIDN), Maternal and Child Health Division. This is an evaluation of a 5-year activity focused on the implementation and scale-up of proven environmental health interventions to prevent morbidity and mortality from infectious diseases, primarily in young children, but also among vulnerable adult populations. The WASHplus Cooperative Agreement commenced in October 2010 and ends in September 2015. The strategic objectives of the WASHplus activity are: • SO1: Increase the availability and use of water supply, sanitation and hygiene (WASH) and indoor air pollution (IAP) interventions • SO2: Develop and implement WASH/IAP integration strategies • SO3: Support USAID’s participation in strategic partnerships • SO4: Develop and test new and innovative approaches and tools 1. Increase the availability and use of proven, high impact WASH and IAP interventions The project aims to achieve measurable improvement in the prevention of diarrhea through increased access to a water supply and sanitation, and improved hygiene practices, including hand-washing with soap, treatment and safe storage of water, safe disposal of feces, and food hygiene. The project also aims to prevent negative health impacts from IAP by improving indoor air quality in rural and urban households still reliant on biomass fuels and basic stoves to meet their energy needs for cooking and heating. Moving beyond basic service delivery through NGO projects, WASHplus uses innovative approaches to expand access and use of WASH services, such as development of innovative approaches for WASH financing; building the capacity of the local private sector for provision of WASH products and services; and institutional strengthening, especially at lowest levels of governance. 2. Develop and implement WASH integration strategies The project aims to achieve improved health outcomes by integrating WASH programs with other health and non-health activities. Primary focal areas are integration of WASH in education, HIV/AIDS, nutrition, food security, and neglected tropical diseases. 3. Support USAID’s participation in strategic partnerships WASHplus supports USAID’s participation in strategic partnerships with other donors and cooperating agencies as a vehicle for increasing the quality and the scale of program activities. At the global level, a key factor in enhancing USAID’s leadership role and maximizing its contribution to the sector is identifying and pursuing the most effective platforms and partnerships, e.g., for scaling up sanitation. WASHplus will support USAID’s existing partnerships and explore the development of new public private partnerships (PPPs). 4. Develop and test new and innovative approaches and tools Problems that remain in the WASH/IAP sectors include technical efficacy, affordability and financing, behavior change strategies for continued use, and sustainable models to reach scale. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 37 These challenges call for innovations in technology, tools, and approaches. WASHplus will focus on developing or adapting promising technologies and approaches developed elsewhere for WASH and IAP implementation. Cross-cutting initiatives that are developed across all the strategic objectives include knowledge management (KM) and monitoring and evaluation (M&E). The KM program will build knowledge platforms for information exchange, capacity building, and networking. These issues should be considered in all aspects of the strategic objectives. The activities described above are designed to ultimately achieve significant public health impact. By design, approximately 80 percent of WASHplus efforts have focused on increased availability and use of WASH interventions, with no more than 20 percent dedicated to IAP. As a secondary focal area, IAP interventions have a limit of $1.5 million of core investment. VI. Evaluation Questions The evaluation team is expected to perform an evaluation of the USAID WASHplus A activity reviewing performance from October 1, 2010 to May 1, 2014. The evaluation should pose and answer the following questions using the broader categories as a framework for inquiry. A subset of countries may be specified for some categories of evaluation. Country impact: 1. To what extent, in what ways, and with what challenges has WASHplus acted to provide increased WASH access using high-impact interventions at local, national, and global levels? What do we know about how well the methods used to increase coverage worked? What can we expect to know by end of the project? Are there additional or alternative actions that can be undertaken to better inform this question during the time remaining? Specifically, what approaches to achieving increased coverage were used by the project, such as leveraging resources, engaging other partners—including the private sector— targeting the underserved, collaborating with USAID bilateral programs, developing evidence to guide improved programming, contributing to post-MDG process, facilitating access to financing and technical assistance, etc.? Integrated programming: 2. Has the WASHplus activity been effective at integrating WASH with other activities, including health and non-health initiatives such as nutrition and IAP? What new approaches have been developed, tested, documented, or disseminated? Does this integrated programming provide added benefits that are not observed for independent programming? Use knowledge from stand-alone WASH programs as a comparison for the integrated WASH programs. Consider the specific ways that these programs have been accomplished that could inform future USAID programming. Consider the extent to Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 38 which WASHplus is effectively documenting successes and challenges in these approaches. Is there any carryover effect to other organizations to integrate programming that could be a result of WASHplus integrated programming? Benefits of the WASHplus as a core mechanism for Missions: 3. What is the usefulness of a Washington-based core mechanism for USAID Missions? Did the project meet their needs and provide useful technical support through the core mechanism? If so, which needs? In what areas were needs not met? How has WASHplus been used to support USAID/W’s role as a technical advisor to the Missions? Determine the involvement and financial contribution (data provided by WASHplus) of USAID field Missions to the core mechanism, and the reasons for presence or absence of field support and participation in core-funded projects. Include interviews for two Missions that did not decide to use the core WASHplus mechanism. Project management: 4. What have been management successes and challenges in implementing WASHplus? What factors [within USAID, within AED/FHI360 and partners, and within the Agreement] have supported or inhibited success? What changes should be implemented for the remainder of WASHplus and what lessons are there for the future? Global leadership (including public private partnerships and knowledge management): 5. Have the WASHplus activities demonstrated or contributed to “thought leadership” in the sector, and how? Specifically, what forums has WASHplus led or participated in (conferences, workshops, meetings, processes), what written and oral (e.g. webinars, website, social media) resources have they developed or planned, or are in progress? What is the influence of the knowledge management activities? Use quantitative measures as appropriate, such as the number of visits to the Web pages, reach of social media efforts, and qualitative measures (including the results from the annual WASHplus KM surveys) to determine the quality and uptake of these resources. VII. Methodology The evaluators should consider a range of possible methods and approaches for collecting and analyzing the information that is required to assess the evaluation objectives. Data collection methodologies will be discussed with, and approved by, the USAID Environmental Health Team prior to the start of the assessment. Document review • USAID/W will provide the team with background documents such as work plans, strategies, reports, and technical products. The evaluation team will review these documents in preparation for the initial team planning meeting. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 39 Initial planning meeting • A 1-day team planning meeting will be held in Washington, DC. before the evaluation begins. This meeting will allow USAID to present the team with the background, purpose, expectations, and agenda of the assignment. In addition, the team will: o review and finalize the timeline and work plan, o develop data collection methods, instruments, tools and guidelines; consolidate into an evaluation framework o review and clarify any logistical and administrative procedures for the assignment, and o develop a preliminary draft outline of the team’s report. Data gathering activities: interviews and focus groups • The evaluation team will conduct a thorough review of the WASHplus activity projects, which will include conducting in-person and phone interviews. From this information, the team will analyze their findings. • The evaluation team will make appointments and conduct phone interviews of persons named in Section XIII. Data analysis and report preparation The evaluation team will: • Use thorough data analysis techniques, such as thematic qualitative analysis. • Use additional tools, as necessary, to determine how the WASHplus activity has contributed to thought and other leadership in the sector. • Analyze the raw qualitative and quantitative data collected. Ensure the data validity and reliability. Determine whether the WASHplus project was useful to the Missions. • Discuss key findings and recommendations with WASHplus and USAID staff. • Prepare a draft report, obtain comments from USAID, and create a final report and presentation. The final report will include an executive summary, purpose, background, main evaluation questions, methodology, findings, conclusions, recommendations, and lessons learned. VIII. Team Composition, Skills and Level of Effort The team should be comprised of two members with the following characteristics: • Expertise in program evaluation • Master’s degree or higher level of education in a relevant technical area (e.g., environmental health, engineering, public health) • Knowledge, skills, and experience with USAID contracting and reporting requirements; policies and initiatives; tools; and results frameworks • Advanced written and oral communication skills • Experience working in developing countries • Experience working in the international donor environment, especially with other development agencies (e.g. UN agencies, WHO, or other USG agency) Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 40 • Strong quantitative and qualitative analytical skills Team organization A team leader will be designated with responsibility for the overall organization of the work as well as the overarching management and administration aspects of the SOW. The team leader will: (i) facilitate preparation of the executive summary and the full report; (ii) ensure that the draft and final products are prepared in accordance with the scope of work; and (iii) ensure that the required revisions for the final report are incorporated. Both members of the team will work together on all aspects of the evaluation, including preparation, interviews, and drafting the final report. Level of Effort for each team member: Task Team members Estimated LOE (days) Background reading Team Leader Specialist 3 3 Planning meeting Team Leader Specialist 1 1 Interviews/meetings (D.C.) Team Leader Specialist 11 11 Analysis/drafting report Team Leader Specialist 11 11 Debriefings with USAID and presentation Team Leader Specialist 1 1 Report finalization Team Leader 4 TOTAL LOEs Team Leader Specialist 31 days est. 27 days est. IX. Logistics No work should begin on the Evaluation until the work plan is approved by the WASHplus AOR. The USAID/GH/MCH Team point of contact will be responsible for the following technical and logistical support: • Provide background documents • Provide contact information for list of interviewees in Section XII The Evaluation Team will be responsible for the following technical and logistical support: • Arrange the schedule of interviews with grantees and partners • Provide support and editing services for the preparation of the final versions of the deliverables Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 41 X. Deliverables and Products 1. Planning Meeting and Evaluation Framework: The consultant team will prepare an evaluation framework that includes methodology for evaluation activities, timeframe of those activities, roles and responsibilities of team members, and data analysis plan. The initial framework will be delivered and discussed during the planning meeting. Evaluators will participate in a 1-day team planning meeting in Washington at the beginning of the project as stated in Section VII. 2. Data Collection Tools (including interviews): Team will prepare and deliver questionnaires for key interviews and templates for data collection. These data collection tools will be reviewed and approved by the USAID AOR for WASHplus prior to the team’s interviews. Team members will conduct interviews in accordance with the guidelines set forth in Section VII, and the tools and methods established during the planning meeting. 3. USAID/W Debriefing and Presentation The team will present the major findings to two different groups through a presentation at the conclusion of the interviews and analysis: (i) to USAID/W; and (ii) to both USAID/W and WASHplus audiences. This debriefing will include a discussion of past achievements and issues, as well as any recommendations the team has for future programming. 4. Draft Report: A complete draft report, not to exceed 30 pages (not including annexes) that will include a clear executive summary, will be submitted to USAID/W no later than 2 weeks after the end of interviews. This report will include a summary of findings, including feedback on performance and implementation, recommendations, analysis of all work plans and sub-awards, and recommendations for improved implementation. USAID/W will have 2 weeks to provide comments and suggestions to the evaluator that will be addressed in the final report. 5. Final Report: After comments have been provided to the team, the final executive summary and full report will be prepared by incorporating the comments received from the review of the draft. The team will submit the final but unedited report for USAID approval no later than 1 week after USAID has provided comments on the draft. The contractors will submit the final edited report to the Development Evaluation Clearinghouse (DEC) within 90 days after USAID has approved the evaluation. The final report will be no more than 30 pages, not including annexes. A suggested outline would include: • Executive Summary (3–4 pages) • Evaluation Purpose and Questions Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 42 • Project Background • Methods and Limitations • Findings, Conclusions, Recommendations (main body of the report; findings are empirical facts and should not rely only on opinion; conclusions synthesize and interpret findings; recommendations are based on findings and conclusions) – present graphical formats to demonstrate the evidence that supports conclusions and recommendations. • Annexes (including Statement of Work, Evaluation Instruments and data, “Statements of differences” regarding significant unresolved difference of opinion, disclosure of conflicts of interest forms, etc.) All data sets collected by USAID or one of the Agency’s contractors or grantees for the purposes of an evaluation must be uploaded and stored in a central database. The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. Until this database is established, data can be submitted to DevelopmentData@usaid.gov XI. Relationships and Responsibilities USAID Point of Contact: Merri Weinger, Environmental Health Team Leader USAID/GH/HIDN/MCH Ronald Reagan Building 3.7.26 1300 Pennsylvania Ave, N.W. Washington, DC 20523-3700 Telephone: 202-712-5102 Email: mweinger@usaid.gov Other technical contact: Helen Petach, Ph.D. USAID/GH/HIDN/MCH Ronald Reagan Building 3.7.63 Telephone: 202-712-1414 Email: hpetach@usaid.gov XII. Cost and Time Estimate Our current estimate is $100,000 and 2 months of effort. These costs include the full LOE as described earlier in the document. XIII. USAID and Grant or Contract Contact Persons for Interviews The expectation is that an appropriate subset of these interviews will take place. To be approved prior to the completion of interviews. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 43 Organization WASHplus WASHplus Country-level staff Zambia partners Kenya partners Mali partners Uganda partners Winrock PLAN CARE USAID/Madagascar USAID/Bangladesh USAID/Zambia USAID/Uganda USAID/Kenya USAID/W MCH USAID/W Water Office WSP UNICEF WHO UNC Columbia University Emory University GACC WASH Advocates JMP Hygiene Task Force XIV. List of Documents and References WASHplus: Years 1, 2, 3, and 4 Work Plans and Budget WASHplus: Years 1, 2, and 3 Annual Reports WASHplus: Project Quarterly Reports for Year 4 (1st quarter) Country-specific quarterly reports WASHplus communication products (e.g., Quarterly Partner Updates, WASHplus News, WASHplus Weeklies, WASHplus customer survey results) www.washplus.org WASHplus products: MHM Toolkit Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 44 WASH Friendly Schools Training (revised) WASH and Nutrition Trainings Bangladesh Sanitation Options job aids IAP/HAP article in PLOS (not found) Inclusive San Job Aids Country NACS input USAID Water and Development Strategy Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 45 Annex 2. Country Profiles Bangladesh Program Name(s) WASHplus Bangladesh WASH activity Core Partner FHI360 Start Date: February, 2013 End Date: May 31, 2015 Budget (Obligated funds): Approved work plan $6,000,000 Obligated $4,000,000 Collaborating Partners: WaterAid, DPHE (Department of Health and Engineering), USAID FTF partners SPRING and Shikha, ICDDR,B, iDE Description of Key Activities: In work plan • WASHplus will implement or rehabilitate appropriate and reliable water technologies. • Implement the CLTS approach; explore the feasibility of launching a hand-washing device. • Design a comprehensive behavior change strategy. • Strengthen the capacity of community members (leaders, youth club members, WASH volunteers). • Training of local government institutions, strengthen programs within FTF, stove trial study. • Promote innovation and engage local institutions. Delivered in reports • Deep tube wells installed w/ some additional rehabilitation • One new water supply technology adapted (this language follows indicators) • Community ignition in process (1,206 communities thru May 2014). • Comprehensive BC strategy designed, translated in Bangla and disseminated. • Local implementing NGOs and government trained in BC approaches for WASH, refresher trainings provided • Schoolteachers trained in WASH including MHM to accompany hardware inputs (348). • Local government capacity strengthened, focus on O&M, planning, budgeting, representation • New, upgraded, and renovated household latrines. (new = 3939, renovated/upgraded = 2775, provided to poorest of the poor - 283) Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 46 • WASHplus will explore the feasibility of launching a hand￾washing device. (reported in May 2014). • Technical support and training to USAID implementing partners to integrate WASH into nutrition (SPRING Project assessment showed statistically significant differences in presence of at least 2 HW stations and observed HW practices) Integrated Programming: In work plan WASH-Sanitation (including inclusive sanitation), WASH￾Schools (including MHM), WASH-Nutrition, WASH￾Hygiene. Delivered in reports CLTSplus (plus focuses on high coverage, low quality latrines) SDAs for upgrading leaky latrines, HW) Sanitation Marketing linkages w CLTS WASH-School (including MHM) Other: Total Beneficiary Target: 272,530 (Work plan) Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 47 Bangladesh Program Name(s) WASHplus Bangladesh IAP activity Core Partner FHI360 and Winrock Start Date: September 2012 End Date: Officially extended through May 2014, again extended through Dec 14 Budget (Obligated funds): $476,823 * Originally budgeted at $863,182 through a combination of field support ($476,823) and Asia Bureau ($386,359) funds. *The IN COUNTRY work was completed using just field support funds plus a supplemental $72,604 grant from the United States Office of the Secretary of State, Global Partnership Initiative (S/GPI) for local capacity building and testing of the performance of improved stoves in the field. *Regional funds were then approved in Nov 2013 for reprogramming to disseminate the Bangladesh Consumer Preference methodology and findings (originally budgeted in the $863,182) and carry out a revised/ improved version of the Consumer Preference Study in Nepal, with development and extensive dissemination of a Consumer Preference Toolkit. Collaborating Partners: International Development Enterprises, Berkeley Air, Aprovecho, in coordination with key stakeholders including USAID Implementing partner CCEB; Global Alliance for Clean Cookstoves (funding iDE contract), SNV Description of Key Activities: In work plan • Consumer Preference and Willingness to Pay study • Kitchen Performance Tests (KPTs) • Marketing and behavior change strategy developed and disseminated • Local institutions trained in research methods and stove testing Delivered in reports • Consumer Preference and Willingness to Pay study • Kitchen Performance Tests (KPTs) • Local implementing NGOs and other institutions trained in qualitative and quantitative consumer research and stove testing. • Marketing and Behavior Change Strategy Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 48 • Dissemination of findings through Asia region • Dissemination at US and global fora Integrated Programming: In work plan WASH-Sanitation (including inclusive sanitation), WASH￾Schools (including MHM), WASH￾Nutrition, WASH-Hygiene. Delivered in reports CLTSplus (plus focuses on high coverage, low quality latrines) SDAs (for upgrading leaky latrines, HW) Sanitation Marketing linkages with CLTS WASH-School (including MHM). Other: Total Beneficiary Target: 272,530 (Work plan) Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 49 Benin Program Name(s) WASHplus Benin Core Partner FHI360 Start Date: October 2012 End Date: October 2015 Budget: 750,000 Collaborating Partners: UNICEF, Government of Benin, MOH, ABMS Description of Key Activities In work plan • Design, carryout, and analyze baseline survey. • Design and conduct innovative multi-media BC program in target neighborhood. • Design and carry out social marketing of WASH products and services • Partner with UNICEF and others for advocacy and increased investment in urban WASH • Support MOH in developing Urban WASH Strategy Delivered in reports • Conducted baseline survey. • Contracted ABMS/PSI to carry out neighborhood pilot program: • Carried out a situational analysis of 2 pilot neighborhoods • Hired 1 advocacy and 1 social mobilization specialist, hired and trained community health workers • Conducted 4 day inclusive Strategic Workshop to develop details of pilot BC/SocMar program • Supported MOH is development of TORs for urban strategy development consultant • Elaborated MOU with the municipality of Cotonou in support of program activities • Recruited expert facilitator to design and conduct multi-stakeholder urban WASH planning workshop (FY15) • Established partnership with UNICEF to conduct joint Call to Action event once national urban WASH strategy is done (FY15) Integrated Programming In work plan N/A Delivered in reports Other: Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 50 Kenya Program Name(s) WASHplus Kenya Core Partner FHI 360 Start Date: January, 2010 (WASHplus efforts started July 2012) End Date: September, 2014 Budget: $2,160,522 Collaborating Partners: MOH, APHIAplus projects and their partners, CDC partners Other collaborating partners include: WSP, World Vision, PSI Description of Key Activities: In work plan • Conduct baseline survey. • Train community health extension workers in districts where the baseline was conducted. • Provide technical assistance to partner organizations to implement WASH-HIV in other parts of the country • Assist MOH with CLTS activities in 1 district Delivered in reports • Completed WASH baseline survey. • Built capacity of CHEWs and CHWs on WASH and HIV integration to support outreach and WASH improvement. • Trained over 650 NGO and government trainers. • Supported USAID and CDC implementing partners to train CHEWs and CHWs across the country. • Finalized and disseminated WASH-HIV Integration materials. Provided technical support to MOH to develop a National WASH training module for community health workers. • Innovated CLTSplus to include inclusive sanitation; now part of government policy; trained over 600 government CHWs and natural leaders; developed supporting job aids. • Assisted MOH with CLTS+ activities in Naivasha subcounty • Trained third-party verifiers • Supported government to identify most effective sanitation marketing options/promising practices. Integrated Programming In work plan WASH-HIV Delivered in reports WASH-HIV Other: Introduced inclusive sanitation that was adopted by the GOK CLTS program. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 51 Madagascar Program Name(s) WASHplus Madagascar Core Partner FHI 360 Start Date: 2010 End Date: November 2013 Budget: $1,158,000 Collaborating Partners: Practica, WSUP Description of Key Activities: In work plans • Conduct assessment implementation, governance, and basic financial management training in WASH sector for future selected NGOs. • Conduct capacity building in project implementation of the WASH￾Everywhere community approach to increase the effectiveness and efficiency of the selected NGOs in the project’s SOW. • Urban project in collaboration with WSUP Delivered in reports • Monitoring of documentation activities. • Capturing lessons learned, best practices, and success stories from other USAID funded WASH programs. Other: PRACTICA fecal sludge management pilot in Ambositra was funded with innovation funds from SO4. Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 52 Mali Program Name(s) WASHplus Mali Core Partner CARE, FHI360 Start Date: April 2013 (six month delay due to civil unrest in the north of the country) End Date: April 2015 (25 months) Budget: 3,000,000 (Promised by USAID/Mali) 2,500,000 (Actually obligated by USAID/Mali) Implementing Partners: CARE Mali Description of Key Activities: In work plans • Understand private sector engagement and identify priority opportunities to increase their engagement. • Train CLTS animators. • Implement and follow up CLTS in 18 communes. • Implement the behavior change component with CLTS mobilization. • Promote Improved WASH and nutrition practices. • Use multi-media and national awareness raising opportunities. • Lead coordination in WASH/nutrition at district levels Delivered in reports • WASHplus-developed intervention strategies and project management tools. • Drafted baseline protocol. • Established or revitalized WASH committees in villages. • Continue CLTS triggering in remaining villages. • Nutrition demonstrations • Support community volunteers to screen infants for malnutrition • Developed Small doable actins for WASH/nutrition. Other: Beneficiary (Work plan): 187,000 women and 60,000 children Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 53 Uganda Program Name(s) WASHplus Uganda Core Partner FHI360 Start Date: December 1, 2012 End Date: Approx. August 31, 2014 (extended to Oct/Nov 2012 Budget: $500,000 Implementing Partners: USAID Implementing Partners STAR SW, SPRING, Community Connector and District Governments of Kabale, Kisoro and Kanungu (SW Uganda) Description of Key Activities: In work plan • Support USAID Implementing Partners to integrate WASH into HIV and nutrition activities • Determine available small doable actions and feasible local solutions. • Support implementation of USAID WASH Grants (up to $1.5 million for districts) by providing technical support to SDS Project (tasked with disbursing grants) and 3 districts to plan, budget and implement WASH ‘menu’ of options. Delivered in reports • Supported implementation of USAID WASH Grants (up to $1.5 million for districts) by providing technical support to SDS Project (tasked with disbursing grants) and 3 districts to plan, budget and implement WASH ‘menu’ of options. • Developed budgeting tool for district planning of WASH. • District and IP capacity building on WASH integration. Workshops on budgeting, monitoring and indicators, O&M, local governance, WASH into nutrition, WASH into HIV, WASH Friendly Schools including MHM. • Led collaborative process to plan integration of HIV, Nutrition • Small doable actions and job aids including innovations for water supply (DIY rainwater catchment), nutrition/food hygiene. Integrated Programming In work plan WASH-HIV, WASH-Nutrition Delivered in reports WASH-HIV, WASH-Nutrition Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 54 Zambia Program Name(s) SPLASH Core Partner FHI360, CARE Start Date: 2011 End Date: 2015 Budget: 13,690,448 Collaborating Partners: (Listed in order of importance) Ministry of Education Science, Vocational Training and Early Education, UNICEF, Ministry of Local Government and Housing, Emory University. WaterAid made a contribution early in the project. Description of Key Activities: In work plan • Improve drinking water and sanitation facilities rehabilitated and installed in schools using a service-delivery framework (Boreholes, latrines, trainings). • Rehabilitated or new latrines with washrooms for senior girls and urinals for boys. • Rehabilitated or new water supplies. • Permanent hand-washing stations. • Develop or adapt a national school hygiene education program that includes: o School-program guidance documents for school leaders o Educational materials for classrooms, offices (e.g. posters) o Lesson inserts for teachers on folding WASH themes into curriculum o Student materials (booklets, etc.) that reinforce the USAID’s reading outcomes Delivered in reports Corrected Achievements as of March 2014 • 334/443 water points rehabilitated or constructed • 1,318/3,995 new latrines constructed • 173,122/250,000 people gaining access to improved drinking water • 62,089/144,000 people gaining access to improved sanitation facilities • 44/270 schools with MHM facilities • 61/370 schools with handwashing facilities • 379/1,014 teachers trained in WASH • 172/370 schools with established WASH clubs and trained peer educators • 45/30 school administrators/officials trained in WASH Integrated Programming: In initial 5-year proposal WASH-Schools Delivered in reports WASH in Schools Other: Beneficiary (Work Plan): 741 schools Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 55 Annex 3: Summary of Accomplishments by Country (Information provided by WASHplus) Countries Characteristics of geographic area where project operates Accomplishments to date (depending on the status of program activities and planning as of August 2014) Bangladesh Program operates in an area of 266,000 households and targets some sub-districts and villages in that geography • # of people gaining access to improved drinking water source = 13,900 • # of people gaining access to improved sanitation facilities = 5,400 (number of people getting off open defecation still to be accounted for) • # of communities certified as ODF as a result of USG assistance = 282 • # of households with installed hand washing device/station = 13,900 Benin Targeting population of two peri-urban districts in Cotonou with about 25000 households # of households heads/main child caretakers reached = 10,000 Kenya Mainly a TA and training program, with some specific pilot activities in two community units Kenya was funded incrementally and therefore did not have targets • 650 public health officials/NGO workers in 26/47 counties trained in WASH-HIV integration • 8,029 volunteer community health workers trained in WASH-HIV integration and inclusive sanitation • Around 1.6 million Kenyans reached with inclusive sanitation messages • WASHplus staff triggered 100 villages in pilot sites • 25 villages ODF certified (# of people stopping open defecation still to be accounted for) Madagascar Program operated in secondary cities • # of people gaining access to improved drinking water source =19,420 • # of people gaining access to improved sanitation facilities = 11,225 • Two WASH blocks improved/rehabilitated in high￾transient areas with an average of 12,000 users per month Zambia (to date as of Sept. 2015) Operates in Eastern Province targeting 60% of elementary schools in the area, with water points • # of people gaining access to improved drinking water source = 173,122 • # of people gaining access to improved sanitation facilities = 62,089 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 56 used by surrounding villages • Interventions in 375 schools in four districts • Constructed 110 boreholes • Rehabilitated 265 water points • Constructed 1,243 latrine compartments for male and females students and school staff • Constructed 88 wash rooms for senior girls • Constructed 285 permanent handwashing facilities • Installed/constructed 203 drinking water facilities • trained 622 teachers in hygiene, 100 teachers in borehole drilling supervision, 78 artisans and 46 pump menders • formed 291 WASH committees (at the community level) and 106 (school) WASH clubs • supported 291 existing PTAs and • ignited 250 schools in SLTS- School led total sanitation Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 57 Annex 4. List of Documents Reviewed Document type Title of documentation Status Date received USAID Forward Report – 2013 Received 6/3/2014 USAID Water Strategy Report Received 6/3/2014 WASHplus M&E Plan Received 6/13/2014 Annual work plans Core Work Plans Revised WASHplus Year 1 Work Plan (2/11/2011) Received 5/30/14 Revised WASHplus Year 2 Work Plan – Activity Table (11/14/2011) Received 5/30/14 Revised WASHplus Year 2 Work Plan – Narrative (11/14/2011) Received 5/30/14 Revised WASHplus Year 3 Work Plan – Activity Table (December 2012) Received 5/30/14 Revised WASHplus Year 3 Work Plan – Narrative (December 2012) Received 5/30/14 Revised WASHplus Year 4 Work Plan – Narrative (January 2014) Received 5/30/14 Revised WASHplus Year 4 Work Plan – Activity Table (01/14/2014) Received 5/30/14 List of Products for WASHplus Year 4 Work Plan Draft (01/14/2014) Received 5/30/14 Country Work Plans Kenya: WASHplus Final Work Plan FY14 Received 5/30/14 Kenya: WASHplus Final Work Plan FY13 Received 6/13/14 Bangladesh: WASHplus Work Plan (12/07/2012) Received 5/30/14 Bangladesh: WASHplus IAP Final Work Plan (10/24/2012) Received 5/30/14 Bangladesh: Year 1 Implementation Plan (7/29/2013) Received 5/30/14 Bangladesh: Year 2, 3 Implementation Plan (9/16/2013) Received 5/30/14 Benin: WASHplus Final Work Plan (11/20/2012) Received 5/30/14 Benin: WASHplus Year 2 Final Work Plan FY14 Received 5/30/14 Madagascar: WASHplus Work Plan 2013 Received 5/30/14 Madagascar: WASHplus Revised Work Plan (11/18/2011) Received 5/30/14 Mali: WASHplus Work Plan – Final (9/27/2012) Received 6/13/14 Uganda: WASHplus District Work Plan – Kabale Received 5/30/14 Uganda: WASHplus District Work Plan – Kanungu Received 5/30/14 Uganda: WASHplus District Work Plan – Kisoro Received 5/30/14 Uganda: Final District Menu of Options Received 5/30/14 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 58 Uganda: Bullet Points for Discussion with USAID (8/14/2013) Received 5/30/14 Uganda: Procedures and TA Required for WASH Grant Items (2/12/2014) Received 5/30/14 Uganda: WASHplus Revised Work Plan (2/15/2013) Received 5/30/14 Uganda: Summary of WASHplus Trainings and Meetings (June 2013 – May 2014) Received 5/30/14 Uganda: WASHplus – Brief Summary Received 5/30/14 Uganda: Revised Work Plan Received 5/30/14 Zambia: SPLASH Revised Program Plan (4/10/2012) Received 5/30/14 Zambia: SPLASH Year 2 Final Work Plan (1/4/2013) Received 5/30/14 Zambia: SPLASH Year 3 Work Plan Received 5/30/14 Kenya: WASHplus Final Work Plan (September 2013) Received 5/30/14 Quarterly reports Core WASHplus Quarterly WASHplus Project Y1Q1 Quarterly Report Received 5/30/14 WASHplus Project Y1Q2 Quarterly Report – Final Received 5/30/14 WASHplus Project Y1Q3 Quarterly Report Received 5/30/14 WASHplus Project Y2Q1 Quarterly Report Received 5/30/14 WASHplus Project Y2Q2 Quarterly Report – Final Received 5/30/14 WASHplus Project Y2Q3 Quarterly Report Received 5/30/14 WASHplus Project Y3Q1 Report Received 5/30/14 WASHplus Project Y3Q2 Received 5/30/14 WASHplus Project Y3Q3 Quarterly Report Received 5/30/14 WASHplus Y3Q4 Report – Supplement Received 5/30/14 WASHplus Project Y4Q1 Report Received 5/30/14 WASHplus Project Y4Q2 Received 5/30/14 Country Quarterly Reports Bangladesh: WASHplus IAP Q1 Jan–Mar 2013 Final Received 5/30/14 Bangladesh: WASHplus Quarterly Report April–June 8/22/2013 Received 5/30/14 Bangladesh: WASHplus Quarterly Report Y1 July– Sept 2013 Received 5/30/14 Bangladesh: WASHplus Y2 Q2 Narrative Report – Finale Received 5/30/14 Bangladesh WASHplus Quarterly Report – Oct–Dec 2013 Q1 Received 6/13/2014 Benin: Q1 FY14 Report 1/28/2014 Received 5/30/14 Benin: Q1 Y1 Final Report Received 5/30/14 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 59 Benin: Q2 FY14 Final Report Received 5/30/14 Benin: Q2 Yr1 Final Report Received 5/30/14 Benin: Q3 Y1 Final Report 7/19/2013 Received 5/30/14 Benin: Q4 FY13 Final Report 10/30/2013 Received 5/30/14 Kenya: FS 1.4 WASH FY12 Q1 Final Received 5/30/14 Kenya: HIP Milestones April 2010 – Final Received 5/30/14 Kenya: WASH 2nd Quarterly Report FY2011 Received 5/30/14 Kenya:WASH-HIV Integration Update Nov–Dec 2012 Received 5/30/14 Kenya: WASH-HIV Integration Update Sept–Oct 2012 Received 5/30/14 Kenya: WASH-HIV Integration Update Jan–Feb 2012 Received 5/30/14 Kenya: WASH-HIV Integration Update July–August 2012 Received 5/30/14 Kenya: WASH-HIV Integration Update March–April 2012 Received 5/30/14 Kenya: WASH-HIV Integration Update Nov–Dec 2011 Received 5/30/14 Kenya: WASH-HIV Integration Update Sept–Oct 2011 Received 5/30/14 Kenya: WASHplus 3rd Quarterly Final 2013 Received 5/30/14 Kenya: Program Update July–August 2013 – Final Received 5/30/14 Kenya: WASHplus Update March–April 2013 – Final Received 5/30/14 Kenya: WASHplus Update May–June 2013 – Final Received 5/30/14 Kenya: WASHplus Update Nov–Dec 2013 – Final Received 5/30/14 Madagascar: Quarter 1 Report FY2012 Received 5/30/14 Madagascar: Quarter 2 FY2012 Final Received 5/30/14 Madagascar: Quarter 3 FY2012 Received 5/30/14 Madagascar: WASHplus FY2013 Q1 Report Received 5/30/14 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 60 Madagascar: WASHplus Quarter 4 Progress Report FY12 Received 5/30/14 Madagascar: WASHplus FY13 Q3 Report – Edited Received 5/30/14 Madagascar: WASHplus FY14 Q1 Report Received 5/30/14 Madagascar: WASHplus Q2 Report – Final Received 5/30/14 Madagascar: WASHplus Q4 Report – Final SC Approved Received 5/30/14 Mali: ENG Quarterly Activity Report (April–June 2013) Received 5/30/14 Mali: Quarterly Report Jan–March 2014 – 4.30.2014 Received 5/30/14 Mali: Quarterly Report Oct–Dec 2013 Received 5/30/14 Mali: WASHplus Annual Activity Report Oct 2013 Received 5/30/14 Uganda: Quarterly Report – Jul–Sept 2013 Received 6/13/14 Uganda: WASHplus Y3 Q4 – Oct–Dec 2013 Received 6/13/14 Zambia: SPLASH FY12 Report July–Sep 2012 – Final Received 5/30/14 Zambia: SPLASH FY13 Q3 Report 7/29/13 – Final Received 5/30/14 Zambia: SPLASH FY13 Q4 Report – Final Received 5/30/14 Zambia: SPLASH FY13 Q2 Report Jan–Mar 2013 – Final Received 5/30/14 Zambia: SPLASH FY14 Q2 Report – Final Received 5/30/14 Zambia: SPLASH FY14 Q1 Report Received 5/30/14 Zambia: SPLASH Q1 Report Oct–Dec 2012 – Final Received 5/30/14 USAID annual work plan budgets Core Budgets WASHplus Year 2 Budget 11/16/11 Received 5/30/14 WASHplus Year 3 Core Budget Proposal 10/25/12 Received 5/30/14 Year 1 Work Plan Budget Version 3 Revised 05/12/2011 Received 5/30/14 Year 4 Work Plan Budget Revised Final 01/15/2014 Received 5/30/14 Country Budgets Bangladesh: WASH Year 4 Work Plan Budget 9/17/2013 Received 5/30/14 Bangladesh: WASHplus Budget 12/07/2012 Received 5/30/14 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 61 Benin: Budget Proposal (revised) 11/10/2012 Received 5/30/14 Benin: Y2 FY14 Budget – Final Received 5/30/14 Uganda: Budgeting for District WASH – Tool Received 5/30/14 Uganda: Final FY14 Uganda Integration Budget 02/19/2014 Received 5/30/14 Uganda: Kabale WASH Budget (revised) Received 5/30/14 Uganda: Kanungu WASH Budget (revised) Received 5/30/14 Uganda: Kisoro WASH Work Plan 2013 Received 5/30/14 Zambia: SPLASH 5 Year Budget – External Received 5/30/14 Zambia: SPLASH Budget Year 2 – External Received 5/30/14 Zambia: SPLASH Year3 Budget 2/24/2014 Received 5/30/14 Madagascar: WASHplus Year 2 Final Budget 11/18/2011 Received 5/30/14 USAID WASHplus annual reports USAID: WASHplus Project Annual Report Yr 3 Received 5/30/14 USAID: WASHplus Project Annual Report Yr 2 Received 5/30/14 USAID: WASHplus Project Annual Report Yr 1 Received 5/30/14 Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 62 Annex 5. Interview Protocols Category WASHplus staff in DC Interviewer(s) Form Number Interview Date Time begins Time ends Name of Interviewee Title/Position Country 1. Can you tell us about your role within the WASHplus mechanism? 2. From your perspective, who is WASHplus designed to serve? 3. In your opinion, has WASHplus met the expectations of the people it was designed to serve? Can you explain your responses? 4. In your opinion, does a Washington-based mechanism yield benefits that would not be realized if such a mechanism did not exist? 5. In your opinion, do you think the benefits provided by the WASHplus mechanism have been in line with the costs? 6. In your opinion, what factors have contributed to management success and management challenges in terms of: a. Partnership structure? b. Contractual structure? c. USAID context? 7. In your opinion, to support WASH programming, should USAID continue using a centralized Washington-based mechanism? 8. Are there any improvements you would propose for future mechanisms to support USAID WASH programming? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 63 Category WASHplus staff in country Form Number Interview Date Time begins Time ends Name Phone number Email Country Interviewed by 1. Can you tell us a little about your role in the WASHplus mechanism? 2. In what ways has WASHplus supported the WASH projects you have worked on? a. Which of these has had the greatest impact? b. Are there other components of WASHplus support you would like to have? c. Are there components of WASHplus support you think were not worthwhile? d. If WASHplus were not there, how would you access the kind of support you have mentioned? 3. Please tell us about your interaction with other local implementing partners and resource partners. 4. Have you had experience with the work of WASHplus to develop programming that is integrated with or coordinated with programming in other sectors, such as HIV AIDS, nutrition, or education? If yes, please tell us about your experience. a. In your experience on the WASHplus supported project you worked on, did this kind of programming provide benefits? Drawbacks? Please explain. b. In your experience, has this WASHplus-supported programming influenced other organizations (not associated with WASHplus) to carry out similar integrated or coordinated programming? c. If yes, can you give examples? 5. What has your experience been of the WASHplus: a. Partnership structure? b. Contractual structure? c. USAID context? 6. In your opinion, have you had adequate management support from the WASHplus in Washington? a. Have funds been transferred in a timely manner? b. Have the reporting requirements been appropriate? c. Has technical assistance been available when needed? d. Have communications been smooth and effective? e. Have you been able to share experiences and information with other countries through WASHplus? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 64 f. What other issues related to management have you encountered? g. Would you recommend any improvements for the future? 7. Have you used WASHplus knowledge resources? (documents, the website, newsletters, weekly briefings, presentations at conferences etc.)? If yes, please provide specific examples. a. How did you use these knowledge resources? b. In your opinion, do the WASHplus knowledge resources meet your needs? c. Do they meet the needs of colleagues outside WASHplus in the country where you work? 8. In your opinion, has WASHplus in general had an influence on thinking in the WASH sector in the country where you work? a. Please explain your responses:____________ b. Please give an example of something WASHplus has done that has, in your opinion, been influential in sector thinking in the country where you work. 9. In your opinion, what would be the best way for USAID to support WASH programming in the future? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 65 Category Global Partners Form Number Interview Date Time begins Time ends Name Phone number Email Country Interviewed by 1. Please tell us a little about your interactions with the WASHplus mechanism. 2. Have you been exposed to WASHplus knowledge resources? (documents, the website, newsletters, weekly briefings, presentations at conferences etc.)? If yes, please provide specific examples. a. How did you use these knowledge resources? b. Do the WASHplus knowledge resources meet your needs? c. In your opinion, have WASHplus knowledge resources had an influence on the WASH sector in the countries where you work or globally? d. If yes, please give an example 3. Do you have any experience with the work of WASHplus to develop programming that is integrated with or coordinated with programming in other sectors, such as HIV/AIDS, nutrition, or education? If yes, please tell us about your experience. a. In your opinion, does this kind of programming contribute to improved outcomes for beneficiaries? b. Please explain:____________ c. In your experience, has this kind of WASHplus-supported programming influenced other organizations (not associated with WASHplus) to carry out similar integrated or coordinated programming? d. If yes, can you give examples? 4. In your opinion, has WASHplus had an influence on thinking in the WASH sector globally or in the country where you work? Please give an example of something WASHplus has done that has, in your opinion, been influential in sector thinking in the country where you work. 5. In your opinion, what benefits has USAID provided to the sector via WASHplus? a. What things would you like to have seen USAID do to support the WASH sector? b. What other interventions do you think WASHplus could have undertaken? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 66 Category Government Representatives Form Number Interview Date Time begins Time ends Name Phone number Email Country Interviewed by 1. Please tell us a little about your interactions with the WASHplus mechanism. (if necessary, prompt with reminders about the nature of the WASHplus programming in the country) 2. Thinking about the WASHplus project(s) in your country, in what ways has WASHplus supported the projects? a. Which of these has had the greatest impact? b. Are there other components of WASHplus support you would like to have? c. Are there components of WASHplus support you think were not worthwhile? 3. In terms of the support you received through WASHplus in your country: a. Have funds been transferred in a timely manner? b. Have the reporting requirements been appropriate? c. Has technical assistance been available when needed? d. Have communications been smooth and effective? e. Have you been able to share experiences and information with other countries through WASHplus? 4. Have you been exposed to WASHplus knowledge resources? (documents, the website, newsletters, weekly briefings, presentations at conferences etc.)? If yes, please provide specific examples. a. How did you use these knowledge resources? b. In your opinion, do the WASHplus knowledge resources meet your needs? 5. In your opinion, has WASHplus had an influence on thinking in the WASH sector in the country where you work? a. Please explain. b. Please give an example of something WASHplus has done that has, in your opinion, been influential in sector thinking in the country where you work. 6. Do you have any experience with the work of WASHplus to develop programming that is integrated with or coordinated with programming in other sectors, such as HIV/AIDS, nutrition, or education? If yes, please tell us about your experience. a. In your opinion, does this kind of programming contribute to improved outcomes for beneficiaries? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 67 b. Please explain:____________ c. In your experience, has this kind of programming supported by the WASHplus mechanism influenced other organizations to carry out similar integrated or coordinated programming? d. If yes, can you give examples? 7. What, in your opinion, what are the benefits of having WASHplus support? a. What could be improved? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 68 Category USAID in Missions (where WASHplus mechanism was not used) Form Number Interview Date Time begins Time ends Name Phone number Email Country Interviewed by 1. Please tell us a little about your experience with the WASHplus mechanism. 2. How did you hear about WASHplus? a. What made you decide not to use this mechanism? 3. I’d like to ask you about costs. (clarify: approximate, relative costs, exact costs) a. How do you think the costs of support interventions (such as technical advice, training) provided through the WASHplus mechanism would compare to other projects carried out by your Mission? b. How would the cost per beneficiary for WASH projects undertaken through the WASHplus mechanism compare to other WASH projects carried out by your Mission? 4. Have you used WASHplus knowledge resources? (documents, the website, newsletters, weekly briefings, presentations at conferences etc.)? If yes, please provide specific examples. a. How did you use these knowledge resources? b. Do the WASHplus knowledge resources meet your needs? c. In your opinion, have WASHplus knowledge resources had an influence on the WASH sector in the countries where USAID works? d. If yes, please give an example 5. In your opinion, has WASHplus had an influence on thinking in the WASH sector? a. Please explain your responses: b. Please give an example of something WASHplus has done that has, in your opinion, been influential in sector thinking. 6. In your opinion, what are the advantages and disadvantages of a Washington-based mechanism to support USAID’s work in WASH? a. Advantages:____________ b. Disadvantages:__________ 7. Do you think USAID should continue to use a Washington-based mechanism? a. Why? b. Are there any improvements you would propose for a future mechanism? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 69 Category USAID in Missions (where WASHplus mechanism has been used) Form Number Interview Date Time begins Time ends Name Phone number Email Country Interviewed by 1. Please tell us a little about your experience of the projects you have worked on which were carried out through the WASHplus mechanism. 2. How did you hear about WASHplus? a. What made you decide to use this mechanism? 3. In your opinion, did the organizations working with WASHplus in the country where you work have adequate access to resources such as: a. Timely finance? b. Knowledge? c. Exchange of information with other WASHplus countries? d. In your opinion, did they receive adequate management support? 4. In your experience, has this kind of programming supported by the WASHplus mechanism influenced other organizations (not associated with WASHplus) to carry out similar integrated or coordinated programming? a. If yes, can you give examples? 5. In your opinion, are there management changes that should be made for future mechanisms used by USAID to support WASH programming? 6. I’d like to ask you about costs. (clarify: approximate, relative costs, exact costs) a. How do the costs of support interventions (such as technical advice, training) provided through the WASHplus mechanism compare to other projects carried out by your Mission? b. How does the cost per beneficiary for WASH projects undertaken through the WASHplus mechanism compare to other WASH projects carried out by your Mission? 7. Have you used WASHplus knowledge resources? (documents, the website, newsletters, weekly briefings, presentations at conferences etc.)? If yes, please provide specific examples. a. How did you use these knowledge resources? b. Do the WASHplus knowledge resources meet your needs? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 70 c. In your opinion, have WASHplus knowledge resources had an influence on the WASH sector in the country where you work? d. If yes, please give an example 8. In your opinion, has WASHplus had an influence on thinking in the WASH sector in the country where you work? a. Please explain your responses: ____________ b. Please give an example of something WASHplus has done that has, in your opinion, been influential in sector thinking in the country where you work. 9. In your opinion, what have been the advantages and disadvantages of a Washington-based mechanism to support USAID’s work in WASH? a. Advantages:_________________ b. Disadvantages:_______________ c. Are there other components of WASHplus support you would like to have? d. Are there components of WASHplus support you think were not worthwhile? 10. Did WASHplus meet your expectations? 11. Do you think USAID should continue to use a Washington-based mechanism? a. Why? b. Are there any improvements you would propose for a future mechanism? 12. Do you have any experience with the work of WASHplus to develop programming that is integrated with or coordinated with programming in other sectors, such as HIV/AIDS, nutrition, or education? If yes, please tell us about your experience. 13. In your opinion, does this kind of programming contribute to improved outcomes for beneficiaries? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 71 Category USAID in DC Form Number Interview Date Time begins Time ends Name Phone number Email Country Interviewed by 1. Please tell us a little about your interactions with the WASHplus mechanism. 2. How does WASHplus fit into the context of the other work you do at USAID? 3. In your opinion, how well aligned is WASHplus with USAID’s other policies and strategies (Water Strategy, Nutrition Strategy etc.)? 4. Have you been exposed to WASHplus knowledge resources? (documents, the website, newsletters, weekly briefings, presentations at conferences etc.)? If yes, please provide specific examples. a. How did you use these knowledge resources? b. Do the WASHplus knowledge resources meet your needs? c. In your opinion, have WASHplus knowledge resources had an influence on the WASH sector in the countries where you work or globally? d. If yes, please give an example 5. Do you have any experience with the work of WASHplus to develop programming that is integrated with or coordinated with programming in other sectors, such as HIV/AIDS, nutrition, or education? If yes, please tell us about your experience. a. In your opinion, does this kind of programming contribute to improved outcomes for beneficiaries? b. Please explain:_________________ c. In your experience, has this kind of WASHplus-supported programming influenced other organizations (not associated with WASHplus) to carry out similar integrated or coordinated programming? d. If yes, can you give examples? 6. In your opinion, has WASHplus had an influence on thinking in the WASH sector globally or in the countries where you work? a. Please give an example of something WASHplus has done that has, in your opinion, been influential in sector thinking in the country where you work. 7. In your opinion, what have been the advantages and disadvantages of a Washington-based mechanism to support USAID’s work in WASH? a. Advantages: ____________ b. Disadvantages: __________ Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 72 c. Are there other components of WASHplus support you think should be included? d. Are there components of WASHplus support you think were not worthwhile? 8. Has WASHplus met your expectations? 9. Do you think USAID should continue to use a Washington-based mechanism to support WASH programming? a. Why? b. Are there any improvements you would propose for a future mechanism? Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 73 Category Local implementing partners Form Number Interview Date Time begins Time ends Name Phone number Email Country Interviewed by 1. Please tell us a little about your interactions with the WASHplus mechanism. 2. In what ways has WASHplus supported the WASH projects you have worked on? a. Which of these has had the greatest impact? b. Are there other components of WASHplus support you would like to have? c. Are there components of WASHplus support you think were not worthwhile? d. If WASHplus were not there, how would you access the kind of support you have mentioned? 3. Has being an implementing partner in WASHplus met your expectations? a. Why? If the local implementing partner has a financial relationship with WASHplus ask the following questions. 4. In your opinion, have you had adequate management support from the WASHplus staff in￾country? a. Have funds been transferred in a timely manner? b. Have the reporting requirements been appropriate? c. Has technical assistance been available when needed? d. Have communications been smooth and effective? e. Have you been able to share experiences and information with other countries through WASHplus? f. What other issues related to management have you encountered? g. Would you recommend any improvements for the future? 5. Have you been exposed to WASHplus knowledge resources? (documents, the website, newsletters, weekly briefings, presentations at conferences etc)? If yes, please provide specific examples. a. If yes, how did you use these knowledge resources? b. Do the WASHplus knowledge resources meet your needs? c. In your opinion, have WASHplus knowledge resources had an influence on the WASH sector in the country where you work? d. If yes, please give an example Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 74 6. Have you had any experience with the work of WASHplus to develop programming that is integrated with or coordinated with programming in other sectors, such as HIV/AIDS, nutrition, or education? If yes, please tell us about your experience. a. In your opinion, does this kind of programming contribute to improved outcomes for beneficiaries? Core Obligated Funds Global Health NTD / Core Obligated Funds Madagascar / Field Obligated Funds Zambia / Field Obligated Funds Kenya / Field Obligated Funds Bangladesh / Regional Obligated Funds Bangladesh / Field IAP Obligated Funds Bangladesh / Wash Obligated Funds Benin/ Field Obligated Funds Mali / Field Obligated Funds Uganda / Field Obligated Funds Liberia / Field Obligated Funds Totals Description 9,797,955.00 1,000,000.00 1,158,000.00 13,690,448.00 2,160,522.00 386,200.00 476,858.00 4,000,000.00 750,000.00 2,500,000.00 500,000.00 127,260.00 36,547,243.00 WASHplus FHI360 1,578,274 63,202 1,641,476 CARE 175,464 175,464 Winrock International 250,039 250,039 Rotary International 22,077 22,077 Other Partners 87,393 87,393 Subtotal Year 1 - 10/01/10 - 09/30/11 2,113,248 0 63,202 0 0 0 0 0 0 0 0 0 2,176,449 WASHplus FHI360 1,593,373 284,102 1,270,898 115,648 693 41,333 15,757 95,644 3,417,449 CARE 157,767 620,183 777,950 Winrock International 162,793 67,394 49 19,172 249,408 Rotary International 201,624 201,624 Other Partners 118,261 144,179 26,586 289,026 Subtotal Year 2 - 10/01/11 - 09/30/12 2,233,819 0 495,676 1,917,667 115,648 742 60,505 0 15,757 0 0 95,644 4,935,457 WASHplus FHI360 1,634,290 109,268 359,460 2,543,010 1,152,507 95,940 131,078 115,698 152,900 5,966 103,700 8,249 6,412,066 CARE 217,266 8,593 2,951,795 584 116,634 3,294,872 Winrock International 91,901 137,575 131,607 361,082 Rotary International 76,651 76,651 WaterAid 46,445 408,954 68,681 524,080 Other Partners 54,055 (35,972) 49,026 67,109 Subtotal Year 3 - 10/01/12 - 09/30/13 2,074,163 117,860 461,063 5,590,277 1,152,507 95,940 671,639 184,964 152,900 122,600 103,700 8,249 10,735,861 WASHplus FHI360 1,193,946 23,935 90,396 1,365,997 553,877 1,017 29,564 114,739 110,704 54,132 165,664 6,846 3,710,818 CARE 72,327 10,403 1,099,965 6,948 339,078 1,528,721 Winrock International 72,770 38,328 90,228 7,544 208,870 Rotary International (40,506) (40,506) WaterAid 33,068 (419,373) 970,937 584,633 Other Partners 13,407 (5,006) 8,401 Subtotal Year 4 - 10/01/13 - 09/30/14 1,311,944 34,338 128,725 2,494,025 553,877 1,017 (299,581) 1,092,625 118,248 393,210 165,664 6,846 6,000,937 Total Cummulative Costs Thru 05.31.14 7,733,173 152,199 1,148,665 10,001,968 1,822,032 97,699 432,562 1,277,589 286,905 515,810 269,364 110,739 23,848,704 Total Obligations Remaining Balances 2,064,782 847,801 9,335 3,688,480 338,490 288,501 44,296 2,722,411 463,095 1,984,190 230,636 16,521 12,698,539 % of obligated budget spent as of 05/31/14 79% 15% 99% 73% 84% 25% 91% 32% 38% 21% 54% 87% 65% Total Obligations Vs Total Expenditures by Source Funds Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 75 Annex 6. WASHplus Budget by Country (obligated funds vs. actual expenses to date) Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 76 Annex 7. WASHplus Engagement in Global Activities Document searched Collaboration summary Collaboration type Year One Annual report WASHplus team, USAID, resource partners, and other organizations gathered to develop a vision for the implementation of the project. Meeting WASHplus held its first webinar on June 23 on “Lessons Learned from Working at Scale for Better Hygiene and Sanitation Practices” (presented by Julia Rosenbaum) Webinar KM activities for year one included developing a communication strategy with input from the WASHplus team, USAID, and resource partner SecondMuse and IRC Communication strategy input WASHplus participated in the January meeting of the Sanitation Working Group and presented lessons learned on HIP’s Ethiopia experience of jointly promoting sanitation and hand washing. Meeting Meetings and discussions were held (at various times) with staff from USAID/Washington, USAID/Ethiopia, and the World Bank’s Water and Sanitation Program (WSP) headquarters, as well as with regional and Ethiopia staff to explore the potential for collaborative activities within the new Total Sanitation/Sanitation Marketing (TSSM) program in Ethiopia—a follow-up to HIP/WSP collaboration in Amhara, Ethiopia. Meeting Other outreach efforts included preliminary planning discussions with USAID/Global Health Bureau staff regarding the possibility of technical assistance or training for USAID Missions in WASH programming (possibilities might include non-Global Water for Sustainability [GLOWS] countries Planning discussion WASHplus held a series of discussions with USAID staff on incorporating IAP within Healthy Households programming, including how to best promote IAP in an integrated context and simultaneous promotion of IAP along with point-of￾use treatment, sanitation, and hand washing Discussions Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 77 Document searched Collaboration summary Collaboration type Additional activities included meeting with USAID’s Office of HIV/AIDS to explore opportunities to integrate WASH into various fora Meeting Over the course of the year WASHplus collaborated with the USAID-funded FANTA 2 (Food and Nutrition Technical Assistance) project on a number of activities to integrate WASH and nutrition Collaboration WASHplus continued its active participation in conferences and workshops that provided opportunities for involvement in integration discussions and planning, such as World Bank’s Water Week, World Water Week held in Stockholm, and the Handwashing for Newborn Survival Workshop, held in June as part of a USAID/Maternal and Child Health Integrated Program and Unilever-Lifebuoy public-private alliance to reduce neonatal mortality through improved hand washing practices of birth attendants and mothers. Conferences and workshops WASHplus also participates in the WASH and health integration meetings led by Defeating Diarrhea. Finally, PATH used a WASH-HIV case study from Uganda (conducted under HIP) in the recently released publication, “Join Up. Scale Up,” http://www.defeatdd.org/join-up-scale-up. Meeting WASH in Schools Call to Action. Ms. Fry attended the second annual WASH in Schools Call to Action partners meeting in New York in December. The purpose of the meeting was for the working groups to provide updates on their 2010 work and to develop a work plan for 2011 Meeting Public-Private Partnership for Handwashing. WASHplus continued to represent USAID within the PPPHW and provided technical support on behalf of USAID, specifically reactivating the Behavior Change and M&E working groups, helping to plan and participating in Steering Committee and Behavior Change working group meetings held in June, and planning the PPPHW-led session at AfricaSan Meeting Participated in the Engineers in Technical and Humanitarian Opportunities of Service conference held in Kirkland, Washington in January Conference Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 78 Document searched Collaboration summary Collaboration type Co-presented household water treatment and safe storage indicators for a webinar with the International Network on Household Water Treatment and Safe Storage in April Webinar Represented WASHplus at the National Institutes of Health Indoor Air Pollution conference in Washington, DC, in May. During the conference, Ms. Derby participated in the Women’s Empowerment Working Group, and provided inputs to NIH on research questions central to women’s empowerment along with suggestions on how other health research or implementation activities can impact women’s empowerment. WASHplus also provided support to NIH in the form of direct conference costs. Conference Attended the annual Global Health Council Conference in June and presented a paper, “Community-Led WASH Innovations Improve Behaviors, Reduce Diarrheal Disease, and Strengthen Health Systems in Amhara, Ethiopia.” Conference Presented a peer-reviewed paper at the 35th WEDC conference in the United Kingdom in July on “Combining Sanitation and Hand Washing Promotion: An Example from Amhara, Ethiopia.” Conference WASHplus collaborated with resource partner Practica to perform a feasibility study of various technical options for hygienic sludge removal at two sites in Madagascar. Collaboration Year Two Annual Report This year WASHplus participated in several webinars to disseminate research/lessons learned and collaborate with its partners. Senior Behavior Change Specialist Julia Rosenbaum worked with WaterAid/Australia to jointly facilitate a webinar on “HIV, AIDS and AIDS” in January 2012. The webinar was part of a broader series on inclusive WASH. Resources and recordings are available at: http://www.inclusivewash.org.au/hiv-aids-chronic￾illness. Webinars Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 79 Document searched Collaboration summary Collaboration type In addition, the project cosponsored a webinar with WASH Advocacy Initiative and Global Water Challenge on December 15, 2011, on “Leading with Sustainability—Laying the Groundwork for Sustainable Services,” Webinars Finally, on July 17, Global Public-Private Partnership for Handwashing (PPPHW) with support from WASHplus, hosted a webinar onto share highlights from the Handwashing Think Tank held with UNICEF in June in New York City Webinars In addition to providing M&E support to the country program in Zambia, WASHplus coordinated activities and meetings of a the Hygiene Working Group for Post-2105 Global Monitoring working under the Joint Monitoring Programme to propose goals, targets, and indicators for the post 2015 Sustainable Development Goals agenda. Meeting WASHplus also participated in the WASH Monitoring Exchange meetings for the WASH sector and in the M&E Technical Workshop of the Global Alliance for Clean Cookstoves (GACC) Meeting WASHplus engaged in numerous meetings throughout the year with current and potential partners such as: microfinance and VSLAs, the Support Fund for Sanitation/FAA as well as with RAN’Eau, JICA, UNICEF, NGO Brother Saint Gabriel, USAID, and the World Wildlife Fund. Meetings Ms. Rosenbaum prepared and delivered a webinar presentation and 2-week online dialogue on WASH￾HIV integration for WaterAid/Australia’s Inclusive Sanitation web series. Webinar WASHplus shared its technical expertise and programming experience in a wide variety of global venues focusing on WASH and IAP topics. Ms. Derby and Ms. Rosenbaum represented WASHplus at the Global Alliance for Clean Cookstoves partners’ meeting in Washington, DC Meeting Providing behavior change input to the National Institutes of Health’s Indoor Air Quality Writing Workshop in October 2011 Workshop Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 80 Document searched Collaboration summary Collaboration type Participating in a TRAction Technical Advisory Group Meeting to review research of three grantees on the acquisition and correct use of improved cookstoves to reduce indoor air pollution, and also a TRAction Project Launch Meeting Year Three Annual Report Organizing a panel session on environmental enteropathy at the CORE annual meeting, which included a presentation on WASH. WASHplus also presented on WASH, nutrition, and ECD integration in a different session at the same meeting. Meeting WASHplus continued planning activities for a Household Air Pollution workshop/meeting exploring data and evidence gaps on ICS adoption and correct and consistent use. Workshop/Meeting WASHplus staff participated in panel discussions and made presentations at the following fora this year: University of North Carolina’s Policy, Science and Innovation conference, Water and Sanitation Rotarian Action Group World Water Summit V, Latinosan 2013, IRC Symposium on Monitoring Sustainability of WASH Services, Colorado WASH symposium, WASH Sustainability Forum, 36th WEDC International Conference, and Integrating WASH and Freshwater Conservation. Conference/Symposiums Evaluation of WASHplus: Supportive Environments for Healthy Communities Project – Final Report 81 Annex 8. WASHplus Global Leadership contributions as identified in Annual Reports Leadership arena Verifiable activities Integration WASH-nutrition WASH & NACS Clean Fed & Nurtured Initiative WASH in schools WASH-HIV Inclusive sanitation MHM Joint Monitoring Program (JMP) Support of the inclusion of a hand washing indicator in both the MICS and demographic and health surveys (DHS) (http://wsp.org/sites/wsp.org/files/publications/Post￾2015-WASH-Targets-Factsheet.pdf Sanitation Inclusive sanitation (http://www.inclusivewash.org.au/hiv-aids-chronic￾illness) Role at international fora Addendum: WASHplus Response to Evaluation Report November 24, 2014 WASHplus appreciates the evaluation team’s effort. However, the WASHplus team finds that the evaluation report does not accurately reflect its work in several key domains. We submit this document as an official response of record and have limited comments only to the most serious concerns. Methodological limitations  The lack of time and resources available to travel to any country where WASHplus works limited understanding and accurate conclusions as to how the project functions within countries. For example, the comment that WASHplus’ approach is “service delivery at scale, largely by NGOs,” does not capture the project’s systematic focus on working within existing systems to build capacity and enhance sustainability, especially within district and local government entities.  The report makes generalizations based on a single informant’s view without further substantiation. For example, the comment “There has been no real planning for long term sustainability—too short and no real exit strategy” is left unanalyzed and does not reference existing sustainability plans and exit strategies captured within country program plans or recognize that funding may have changed. Comments were taken at face value, without the confirmation necessary to avoid subjective bias and to assure sound findings. Inaccurate characterizations  The stated limitation “descriptions of some of the project objectives and results were vague, making it difficult to set coherent criteria to evaluate project outcomes,” does not recognize that country programs in Bangladesh, Benin, Mali and Zambia have Mission approved M & E plans/logical frameworks or results frameworks, all of which specify indicators and targets for approved indicators. WASHplus reports accomplishments annually against these targets.  Certain findings and conclusions lacked appropriate contextualization as to how global USAID projects function, notably, field support and Missions’ role in establishing specific scopes of work. For example, the evaluation team selectively focused on “impact on national policy” and “engaging the private sector” as criteria to assess country level work, without referencing whether Missions requested, prioritized or specified these outcomes within approved work scopes and plans.  Global leadership findings confound the roles of USAID and a global project. WASHplus only engages in global policy work with and on behalf of USAID. For example, USAID requested WASHplus to play a global policy advocacy role to promote including hygiene within the post 2015 agenda. The report cites a single respondent saying inputs “were not entirely strategic…[and were] less effective at translating [their understanding of hygiene improvement] into concrete proposals for monitoring progress at the national and global level.” The evaluators failed to note that WASHplus recommendations for indicators are included in the post-2015 WASH document prepared and circulated globally by the Water Supply and Sanitation Collaborative Council.  The review of Knowledge Management was limited to selected global level products and activities. Critiques did not benchmark WASHplus against similar global USAID projects and did not encompass the breadth of KM work being done within WASHplus countries at multiple stakeholder levels.