i | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV USAID/Indonesia Urban Water, Sanitation and Hygiene Penyehatan Lingkungan Untuk Semua (IUWASH PLUS) Final Performance Evaluation Report Contracted under GS-00F-210DA / 72049720M00001 Indonesia Monitoring, Evaluation, and Learning Platform Prepared by: Popy Indrawati Risyana Sukarma Hartono Kurniawan Keri Culver Kwan Men Yon Siti Nur Aini JANUARY 2022 USAID.GOV DRAFT REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | ii ABSTRACT This final performance evaluation report of the USAID Indonesia Urban Water, Sanitation and Hygiene Penyehatan Lingkungan Untuk Semua (IUWASH PLUS) answers evaluation questions relating to performance indices and drivers, non-revenue water and energy efficiency interventions, the human resources roadmap, spring vulnerability assessments, and alternative financing. The evaluation team carried out a mixed-methods design remotely because of COVID-19 restrictions, using interviews and an online mini-survey. The report concludes that IUWASH PLUS made important gains in many areas and will leave important tools available to the Government of Indonesia and local partners. The evaluation team also offers a set of recommendations for the remaining time IUWASH PLUS is active, and for future USAID interventions in this sector. iii | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV TABLE OF CONTENTS ABSTRACT..............................................................................................................................................................................II LIST OF FIGURES................................................................................................................................................................. IV LIST OF TABLES...................................................................................................................................................................IV ACRONYMS........................................................................................................................................................................... V EXECUTIVE SUMMARY...................................................................................................................................................VIII 1. BACKGROUND .......................................................................................................................................................... 1 DESCRIPTION OF THE USAID ACTIVITY........................................................................................................... 2 2. EVALUATION PURPOSE, AUDIENCES, AND USES ......................................................................................... 3 3. EVALUATION QUESTIONS..................................................................................................................................... 3 4. METHODS AND LIMITATIONS.............................................................................................................................. 5 LIMITATIONS............................................................................................................................................................... 6 5. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ........................................................................... 7 EQ1: PERFORMANCE INDICES AND DRIVERS................................................................................................. 7 CONCLUSIONS ..............................................................................................................................15 RECOMMENDATIONS .................................................................................................................16 EQ2 NON-REVENUE WATER AND ENERGY EFFICIENCY INTERVENTIONS .................................... 16 CONCLUSIONS ..............................................................................................................................25 RECOMMENDATIONS .................................................................................................................26 EQ3: HUMAN RESOURCES ROAD MAP .......................................................................................................... 26 CONCLUSIONS ..............................................................................................................................34 RECOMMENDATIONS .................................................................................................................35 EQ4: SPRING VULNERABILITY ASSESSMENTS ............................................................................................... 36 CONCLUSIONS ..............................................................................................................................38 RECOMMENDATIONS .................................................................................................................38 EQ5: ALTERNATIVE FINANCING ...................................................................................................................... 39 CONCLUSIONS ..............................................................................................................................44 RECOMMENDATIONS .................................................................................................................45 6. CROSS-CUTTING THEMES: GENDER AND MEASUREMENT ................................................................... 45 ANNEX A. EVALUATION MATRIX: GETTING TO ANSWERS............................................................................ 50 ANNEX B. DATA COLLECTION TOOLS................................................................................................................... 52 ANNEX C. SITE AND SAMPLE SELECTION............................................................................................................... 79 ANNEX D. LIST OF KEY INFORMANTS INTERVIEW............................................................................................. 82 ANNEX E. LIST OF DOCUMENTS AND DATA SOURCES................................................................................... 85 ANNEX F. EVALUATION STATEMENT OF WORK................................................................................................ 88 ANNEX G. EVALUATION TEAM AND LOGISTICS................................................................................................. 99 ANNEX H. RESULT FRAMEWORK FOR IUWASH PLUS .....................................................................................103 ANNEX I. SURVEY DATA ANALYSIS.........................................................................................................................104 ANNEX J. DISCLOSURE OF CONFLICT OF INTEREST .......................................................................................118 ANNEX K. COMPARISON OF BPPSPAM AND IUWASH PLUS PI.....................................................................124 USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | iv LIST OF FIGURES Figure 1: Water access targets and achievements (left); sanitation access (right)...................................... 1 Figure 2. PI score changes, 2017–2020, by sub-index scores .......................................................................... 9 Figure 3. Perceptions of PDAM performance, IUWASH PLUS-SECO vs. IUWASH PLUS ...................10 Figure 4. Perceived improved aspects, IUWASH PLUS-SECO vs. IUWASH PLUS ................................. 11 Figure 5. Perceived drivers of improvement, IUWASH PLUS-SECO vs. IUWASH PLUS......................11 Figure 6Average overall PI scores by PDAM size .............................................................................................13 Figure 7. Process steps for sanitation services to merge into PDAMs........................................................14 Figure 8. Iterative process of NRW reduction in PDAM ...............................................................................17 Figure 9. Operational indicator by site and IUWASH PLUS-SECO vs. IUWASH PLUS.........................20 Figure 10. Experience of COVID-19 impact, IUWASH PLUS-SECO vs. IUWASH PLUS......................23 Figure 11. Training topics, by IUWASH PLUS-SECO vs. IUWASH PLUS..................................................28 Figure 12. Offline capacity-building activities were more intensive than online........................................30 Figure 13. Personal changes as a result of training, IUWASH PLUS-SECO vs. IUWASH PLUS...........32 Figure 14. Organizational changes as result of training, IUWASH PLUS-SECO vs. IUWASH PLUS ...33 Figure 15. GOI projection for extreme dry climate events, 2020–2025, RPJMN.....................................37 Figure 16. Financial performance on the PI (base to PY4)..............................................................................41 Figure 17. Training topic participation, by gender............................................................................................47 Figure 18. IUWASH PLUS Results Framework/TOC................................................................................... 103 Figure 19. Respondents by PDAM job type .................................................................................................... 104 Figure 20. Respondents by duration as PDAM staff...................................................................................... 104 Figure 21. Respondents by employment status and educational attainment........................................... 105 Figure 22. Training topic participation, by gender......................................................................................... 105 Figure 23. Training topics, by IUWASH PLUS-SECO vs. IUWASH PLUS trainees............................... 106 Figure 24. Offline capacity-building activities were more intensive than online..................................... 107 Figure 25. Most recent IUWASH PLUS training, by year............................................................................ 107 Figure 26. Relevance of training to their work, by sex................................................................................. 108 Figure 27. Relevance of training to their work, by IUWASH PLUS-SECO vs. IUWASH PLUS......... 108 Figure 28. Satisfaction with training content, by sex..................................................................................... 109 Figure 29. Satisfaction with the training content, by IUWASH PLUS-SECO vs. IUWASH PLUS...... 109 Figure 30. Satisfaction with training method, by IUWASH PLUS-SECO vs. IUWASH PLUS ............. 110 Figure 31. Satisfaction with real-life examples, by IUWASH PLUS-SECO vs. IUWASH PLUS .......... 110 Figure 32. Satisfaction with practice equipment, IUWASH PLUS-SECO vs. IUWASH PLUS............. 111 Figure 33. Satisfaction with trainer(s), IUWASH PLUS-SECO vs. IUWASH PLUS............................... 111 Figure 34. Action plan development, IUWASH PLUS-SECO vs. IUWASH PLUS ................................. 112 Figure 35. Personal changes as a result of training, IUWASH PLUS-SECO vs. IUWASH PLUS........ 113 Figure 36. Organizational changes as result of training, IUWASH PLUS-SECO vs. IUWASH PLUS 113 Figure 37. Perceptions of PDAM performance, IUWASH PLUS-SECO vs. IUWASH PLUS .............. 114 Figure 38. Perceptions of improved performance, IUWASH PLUS-SECO vs. IUWASH PLUS......... 115 Figure 39. Perceptions, factors affecting performance, IUWASH PLUS-SECO vs. IUWASH PLUS . 116 Figure 40. Experience of COVID-19 impact, IUWASH PLUS-SECO vs. IUWASH PLUS................... 116 Figure 41. Perceived gains from interventions, IUWASH PLUS-SECO vs. IUWASH PLUS ............... 117 LIST OF TABLES Table 1. Crosswalk between evaluation purposes and evaluation questions............................................... 5 Table 2. Sites consulted for the final performance evaluation......................................................................... 6 Table 3. High relevance and high satisfaction, sex and IUWASH PLUS-SECO vs. IUWASH PLUS.....29 Table 4. Moderate response rates for the PDAM trainee online survey....................................................81 Table 5. Evaluation Team Members and Skill Sets............................................................................................99 Table 6. Estimated Level of Effort per Team Member.................................................................................. 101 v | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV ACRONYMS APBD regional government budget APBN central government budget BAPPEDA Provincial Development Planning Agency BAPPENAS National Development Planning Agency BLUD Badan Layanan Umum Daerah, Regional Public Service Agency to manage services BOT build-operate-transfer BP business plan BPR Badan Perkreditan Rakyat (Smallholder Credit/Financing Bank), a type of MFI BPPSPAM agency for improving the implementation of water supply system BUMD Badan Usaha Milik Daerah, a form of potentially for-profit Regional owned enterprise (PDAM is a BUMD) B2B business-to-business BTAM Balai Teknik Air Minum dan Sanitasi/ Technical Center for Water Supply and Sanitation CapEx capital expenditure CB capacity-building CSR corporate social responsibility DEC Development Exchange Clearinghouse DMA district metering area EE energy efficiency EQ evaluation question ESP Environmental Service Program FCR full-cost recovery FGD focus group discussion FS feasibility study GDP gross domestic product GIS geographic information system GOI Government of Indonesia HH household HR human resources HRD human resources department IP Implementing partner USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | vi IUWASH PLUS Indonesia Urban Water, Sanitation and Hygiene Penyehatan Lingkungan Untuk Semua IUWASH PLUS-SECO That part of programming that was funded by SECO KKMA Kajian Kerentanan Mata Air/ Spring Vulnerability Program KMP Kualifikasi Mutu Pelatihan/ Indonesia’s training qualifications standards KSPPS Koperasi Simpan Pinjam dan Pembiayaan Syariah/ Saving and Loan, and Sharia Microfinance Cooperative L2T2 Layanan Lumpur Tinja Terjadwal/ scheduled desludging services LG local government LPS liters per second MEL-P Monitoring, Evaluation, and Learning Platform MFI microfinance institutions MOEF Ministry of Environment and Forestry MOF Ministry of Finance MOH Ministry of Health MOHA Ministry of Home Affairs MOU memorandum of understanding MPWH Ministry of Public Works and Housing MTE midterm evaluation NGO non-governmental organization NPL non-performing loan NRW non-revenue water NUWSP National Urban Water Supply Project OD open defecation ODF open defecation free OpEx operational expenditure OSH open-source hardware PAMSIMAS water and sanitation for low-income communities PBG performance-based grants PDAM water utilities PERPAMSI Persatuan Perusahaan Air Minum Seluruh Indonesia/ Association of Indonesian Water Supply Companies PI performance index PIAP Performance Improvement Action Plan PMPD Penyertaan Modal Pemerintah Daerah/ local government equity funding PPP public-private partnerships vii | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV PT SMI PT Sarana Multi Infrastruktur PY Project Year RKAP Rencana Kerja dan Anggaran Perusahaan/ company’s annual budget plan RPAM Rencana Pengamanan Air Minum/ water safety plan RPJMN National Medium Term Development Plans SDG sustainable development goal SECO Swiss State Secretariat for Economic Affairs SOP standard operating procedure SOW statement of work SPAM drinking water supply TA technical assistance TOC theory of change UPTD Unit Pelaksana Teknis Daerah, Regional Technical Implementing Unit USAID United States Agency for International Development WASH water, sanitation, and hygiene WRM water resources management YPTD Yayasan Pendidikan Tirta Dharma USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | viii EXECUTIVE SUMMARY The United States Agency for International Development (USAID) in Indonesia commissioned a final performance evaluation of its Indonesia Urban Water, Sanitation and Hygiene Penyehatan Lingkungan Untuk Semua (IUWASH PLUS) activity. USAID supports the Government of Indonesia (GOI) in increasing access to better water and sanitation services and improving key hygiene (WASH) behaviors for the urban poor and vulnerable, contributing to the GOI’s universal coverage target and the sustainable development goals. The activity targets these high-level results by February 2022: (i) Increased access to improved water service quality to at least 1.1 million urban residents, of which 500,000 are from the poorest 40 percent of the population (denoted the “B40”) (ii) Increased access to safely managed sanitation services for at least 500,000 urban residents In 2019, USAID created a partnership with the Swiss State Secretariat for Economic Affairs (SECO). SECO added US$4.5 million to support addressing the technical challenges of municipal water utilities (PDAMs). IUWASH PLUS-SECO activities included technical assistance (TA) and training with related equipment for seven PDAMs in West and Central Java to reduce non-revenue water (NRW) and improve energy efficiency (EE). The Mission commissioned this final evaluation to assess and analyze achievements, impacts, and progress toward IUWASH PLUS and USAID-SECO goals, and to examine sustainability and lessons learned. The evaluation conducted key informant interviews and a mini-survey remotely, focusing on a purposive sample of 11 PDAMs out of the activity’s 35. Four of these PDAMs were funded by IUWASH PLUS-SECO funds, and the remaining 7 by IUWASH PLUS funds alone. Below, we reprise the key conclusions and recommendations from evaluation questions and cross-cutting issues. EQ1: What is driving PDAM performance improvements and what is the role of the BPPSPAM and IUWASH PLUS indices in performance improvement? IUWASH PLUS developed a performance index (PI) designed to support individual PDAMs to identify and address specific areas for planning and growth. From 2017 to 2020, across 33 supported PDAMs, average ratings on this PI increased by a little over 15 percent. We also examined performance improvement in a survey of PDAM training participants. Most respondents rated their performance as better than at baseline. A set of promising, replicable characteristics drove this improved PDAM performance during the IUWASH PLUS activity: • Improved communication between local governments (LGs) and PDAMs • LG commitment—including financial commitment—to PDAM successes • New, effective leadership in the PDAMs, which used standard operating procedures (SOPs), data, and prioritization to good effect • TA and capacity-building (CB) using the PI as a guide Improvements on the administrative sub-index have been an important part of average PI score gains, along with HR reforms and reforms in operations to a lesser extent. Finance and coverage (“service”) sub-indices have not improved much on average, if at all, while raw water scores improved but for only a small group of PDAMs. ix | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Key operational, human resource, and financial characteristics appeared to differ between smaller and larger, or weaker and stronger, PDAMs, in line with findings in other sections of this report. PI scores for smaller, weaker PDAMs were lower on average and improved less. IUWASH PLUS’ efforts toward merging drinking water and wastewater under the PDAM’s management have been in line with PDAM needs; most PDAMs are not yet ready for this, but IUWASH PLUS support has brought them closer to this goal with technical assistance. The GOI suggests this integration in their master planning but it is not mandatory. RECOMMENDATIONS • Make PDAM and LG commitment a criterion for participation in future sector activities • Build on the success of the PI and collaborate with GOI on the development of their own index, including possible inclusion of parts of the PI. Support the interagency process and offer solid analysis and input on 2020 BPPSPAM data and the PI. • In the same process with GOI, share lessons learned on mentoring: types of experts needed, cost estimates, TA guidelines, and applying TA with PDAMs that have different needs. • Develop the lessons learned in merging wastewater management under the PDAM into a document that can be shared with GOI prior to the activity’s end. • USAID should support GOI to develop regulations, technical guidelines, and IUWASH PLUS￾type pilots to merge water supply and wastewater services, where this is a priority for the Mission and the GOI. • Hold a roundtable with IUWASH PLUS TA teams on the topic of specifying the PI for PDAMs that are smaller (<20,000 connections) versus larger (>100,000 connections), given that the smaller and more remote PDAMs had sometimes quite different needs. • Indicators and targets in the PI could be scaled for smaller PDAMs, to allow for more indication of process advancement. EQ2 Are IUWASH PLUS activities resulting in sufficient reduction of NRW and EE improvements to put PDAMs on a pathway to business viability? IUWASH PLUS and IUWASH PLUS-SECO supported reduction of NRW in 12 PDAMs (5 IUWASH PLUS, 7 IUWASH PLUS-SECO) and energy efficiency improvements in eight PDAMs (four of each type). IUWASH PLUS-SECO helped PDAMs prepare business plans (BPs), advised on tariff adjustments, and supported customer reclassification, customer satisfaction surveys, and financial reporting. IUWASH PLUS-SECO also provided equipment and tools that helped staff accurately measure NRW and energy usage. The activity helped PDAMs establish a dedicated NRW reduction task force in most PDAMs. IUWASH PLUS also developed guidelines for feasibility studies on NRW and EE management, as well as EE audit and a tutorial on how to operate the tools for NRW and EE. To date, the reduction of NRW and improvements in EE are promising but insufficient for business viability. The GOI has prioritized reaching 20 percent NRW, but even in project PDAMs, water loss remains consistently higher than 30 percent. Reducing water loss should result in lower water production, and lower operational cost. This would improve cash flow, and if connections increase as a result, revenue will increase, helping the PDAM toward business viability. Impact on revenue has as yet been minimal, however, with NRW/EE activities still in early stages. Early data do point to USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | x increased household connections1 as a result of the NRW interventions – which over time should bring increased revenue. The project has yet to report on improved energy efficiency figures. • Feasibility studies played an important role in determining technical feasibility, financial viability, and institutional applicability of the NRW/EE program. • The activity encouraged LGs to provide financial support to PDAMs. • The higher cost of implementing with the IUWASH PLUS-SECO model – in both equipment and TA costs – make it unrealistic that this could be replicated nationally. RECOMMENDATIONS • For future interventions, activities should identify and incentivize a champion (a person or unit/division) within a PDAM who will consistently prioritize the NRW/EE program. • Also, for future programming, all supported PDAMs should have an intensive NRW/EE program, preferably using the IUWASH PLUS-SECO approach with technical equipment and hands-on practice. However, IUWASH PLUS, USAID and SECO must examine scale-up options for equipment sharing, cascading training and TA, peer-to-peer learning, streamlining capacity building, and other cost savings that would make this recommendation more achievable. Invest in cost-benefit research taking into account the projected savings, revenue gains and economies of scale, to justify greater GOI investment in the sector. • In order to bring the PDAMs closer to business viability, count new additional connections as an intermediate target, and examine network expansion. PDAMs will also have to be supported to reduce physical losses as well as commercial losses. • Most LGs do not comply with the current requirement to allocate equity fund in advance of Water Hibah/PBG program implementation. To avoid delays, compliance for the equity fund allocation should be deferred to the time of grant release. Consider, too, bringing together MPWH and MOF on this issue to seek a solution that works for both ministries. EQ3: In what ways has USAID IUWASH PLUS’s training and human resources development roadmap contributed to the improved PDAM performance? What accounts for successes? What accounts for failures? IUWASH PLUS developed fit-for-purpose CB modules for PDAMs, and offered significant mentoring and support to achieve gains, notably using the PI as a guide. IUWASH PLUS-SECO interventions built on the learning from earlier IUWASH PLUS training and increased the level of technical assistance, added equipment on which they also trained PDAM staff, and added action planning as a condition for receiving training certificates – all to good effect. Starting in 2020, IUWASH PLUS worked with the GOI to develop the human resources development roadmap, which was launched in 2021 and adapted further, and awaits official GOI adoption. • Tailored, demand-driven training was valued and resulted in positive outcomes. But, this customized model meant that the training modules created by the activity were not aligned 1 Increase in net connections occurred not only in the 12 NRW/EE PDAMs, but across almost all supported PDAMs, indicating that any intervention impact is due not only to NRW/EE work. The increase varied across PDAMs from fewer than 100 units to over 15,000 units. xi | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV with competency standards. As a result, IUWASH PLUS has worked in recent months to develop modules that meet the competency standards. Neither fully customized nor fully standardized models will meet PDAM needs nationwide; standard modules and customized TA will be required to meet the wide variation in needs across PDAMs. • The HR roadmap is a work in progress, and must be formalized with a ministerial decree, a long and uncertain process. IUWASH PLUS has been working on this since 2020 but it is a long process to embed the model with the GOI. And, at the same time, the responsible agencies are themselves in transition—creating concerns for sustainability. • IUWASH PLUS-SECO’s addition of providing tools and equipment, the use of an action plan, and the standardization of some models produced some convincing technical results. • Resources, TA, and best practices differ for larger and smaller PDAMs, whose training should fit their resource and technical levels. • There was little, if any, follow-up on with IUWASH PLUS trainees, and the follow-up for IUWASH PLUS-SECO trainings was by group, not individual. There was no flexible database of trainees. • If IUWASH PLUS has a model behind the training design for how adult learners will learn, master, and use the training, that might be tracked and tested, it is not apparent from the way they undertook the trainings or monitored gains and gaps. RECOMMENDATIONS • IUWASH PLUS should capture staff knowledge on differences between smaller and larger, or weaker and stronger, PDAMs to support the GOI in its use of the roadmap and for follow-on USAID activities. • IUWASH PLUS should advocate for a ministerial regulation on the roadmap. Seek Ministry of Public Works and Housing (MPWH) buy-in on the PDAM roadmap and the completion of key components before the close of IUWASH PLUS. • For future activities, require better planning for and monitoring of training gains. This means maintaining a database of trainees (including turnover), tracking individual and organizational change, and identifying ways to embed tracking in the operationalized roadmap. • For future activities, require that implementers base training modules on the occupational map, Indonesian qualification standards, and staff competencies required for PDAM roles. EQ4: What evidence is there that spring vulnerability assessments have resulted in improved sustainable management of raw water sources and use by drinking water providers? IUWASH PLUS supported a subset of PDAMs to reduce the vulnerability of their spring-fed raw water sources in developing a framework titled “Kajian Kerentanan Mata Air (KKMA)” – the Spring Vulnerability Program. The activity helped PDAMs identify weaknesses and build infiltration ponds which appear to have strengthened the water sources with additional supply. Unfortunately, this was not measured accurately before and after the activity, so it cannot be fully confirmed, and PDAMs report they could not build all the recommended ponds, so it is unclear how much greater the effect might have been. A GOI regulation is in place to replicate source vulnerability assessments and actions but lacking definitive assignment of responsibility among GOI agencies. • The PDAMs participating in KKMA recognize it as successful, as do their government partners. The Ministry of Environment and Forestry (MOEF) has begun to adjust the spring recovery strategy following the IUWASH PLUS method to delineate recharge areas. This is not yet official policy, however. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | xii • Operationalized GOI regulations with defined responsibilities to replicate KKMA nationwide do not yet exist. • The lack of before-and-after data from the effort is a critical gap that makes definitive decision-making on the topic much riskier. RECOMMENDATIONS • Align IUWASH PLUS raw water source programming to Rencana Pengamanan Air Minum (RPAM)—Water Safety Plan— to support GOI’s goals. With evidence, this may encourage regulations for nationwide use of these tools. • IUWASH PLUS should support a KKMA action plan with the GOI and help them identify and test reliable tools to measure before and after implementation for the long term. • IUWASH PLUS and future implementers should help GOI operationalize their KKMA program among the relevant agencies and other actors, including interagency communication and budget. EQ5: In what ways have IUWASH PLUS’s activities contributed to PDAMs’ ability to secure alternative financing? PDAMs were highly variable in the degree to which they were ready for alternative financing such as public-private partnerships (PPPs), business-to-business arrangements (B2B), partnerships with microfinance institutions (MFIs), grants, and other sources outside their traditional LG funding. As a result, IUWASH PLUS worked with them on their readiness in different ways. For PDAMs that were weaker financially and still fully dependent on their LG – a large majority of PDAMs – IUWASH PLUS offered capacity building around financial performance. These PDAMs and their LGs tend not to have alternative financing as a goal. Where a few PDAMs were more ready to secure alternative financing, IUWASH PLUS assisted these with feasibility studies and the tendering process. • Most PDAM teams are not aware about partnerships or are not eager to pursue them, and consider “alternative financing” to include increased capital injection from their LGs. • Capacity building support to increase financing performance was robust, but the PI scores on the finance sub-index do not show strong growth across the PDAMs. • B2B options were not explored widely, but there are opportunities and GOI appetite for this to close the large WASH infrastructure financing gap. Most LGs and PDAMs seem well beneath the interest and capacity levels necessary to carry these to fruition. • The GOI has put in place the requisite PPP regulations, toolkit and guidelines. But greater use of these and other alternative financing methods will depend on PDAM leadership seeking them out and being able to implement them. • Since activity-supported regulations and guidelines are only recently published but not yet launched, the impact of IUWASH PLUS support for those guidelines cannot be evaluated. RECOMMENDATIONS • LGs themselves, as well as future USAID activities, should continue to support financial health and full-cost recovery to attract partnerships and financing, with creditworthiness an important steppingstone to readiness. Microfinance collaborations also offer entry points for to accelerate WASH access goals. USAID activities should also support a related business￾enabling environment reform with the GOI. • GOI and donors should enhance awareness and build capacity on alternative financing for PDAMs and LGs by combining capacity building, technical assistance, toolkits, and guidelines. xiii | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV To reach the ambitious GOI goals for private sector investment in WASH infrastructure, make B2B and PPPs a top priority. • Support GOI efforts to improve policy and regulations toward the business enabling environment in the WASH sector, to facilitate PPP and B2B processes. Greater detail on this is offered in the main body of the report. • Develop alternative financing guidelines covering detailed B2B processes, new regulations, partnership mechanisms, risks and rewards, case studies on crucial issues, and the like. CROSS-CUTTING THEMES: GENDER AND MEASUREMENT We offer the following considerations—outside the evaluation team’s immediate purview—to spark stakeholder discussion. We do not provide conclusions or recommendations, but rather offer interpretations we hope are of service. The evaluation questions did not include attention to gender, though the SOW did ask that evaluators examine gender differences among respondents. We reviewed IUWASH PLUS’s research for their Gender Strategy to inform training content, such as with PDAMs and community water groups. We also interviewed with the IUWASH PLUS team on how they trained institutions on gender-responsive planning and budgeting and built an activity-wide gender working group. However, it is noteworthy that the water sector is traditionally led by men in Indonesia, as elsewhere, and women hold few leadership or technical positions – so the sector has a built-in bias in operations and leadership. IUWASH PLUS reported training numbers with broadly equitable inclusion of women and men in training sessions (though our survey’s randomized sampling captured a much lower percentage of women). It is unclear how IUWASH PLUS got such high participation from women in training, when the sector has built-in bias toward men, nor why only 11% of survey respondents were women. Similarly, only 22 of the evaluation interviews were with women, out of 70 total; only five PDAM respondents were women, out of 22 interviewees, and 1 of 12 LG respondents. These figures indicate that those who speak for the PDAMs (and, presumably, make more PDAM decisions) remain overwhelmingly male. Findings in the mini-survey appear to confirm differences between men and women IUWASH PLUS trainees as well. The evaluators note IUWASH PLUS’ exemplary effort to calculate attributable access, so USAID can show how the activity effects change for households, including poor ones. Respectfully, we offer another perspective. The logic chain from IUWASH PLUS’ TA, CB, and some equipment inputs to new connections is complex and risks over or understating contribution. This is a reputational risk for USAID: readers may question the link from inputs to results. An additional concern with this indicator is that in estimating connections as a percentage of a global figure, the indicator does not allow for reliable gender breakdowns that might aid programming, particularly programming directed at B40 households. This distinction can incentivize selecting larger and stronger PDAMs that are more likely to add connections. The focus on counting new connections also turns attention from B40 and female-headed households—a concern the evaluation team feels is justified, since the activity will meet just half of its B40 access targets. The activity’s only gender indicators are disaggregated trainee data but with no outcome target. The evaluation team noted additional areas that would have benefited from measurement, with the goal of having evidence to inform IUWASH PLUS’ learning and adaptation as well as the GOI’s own goals and data needs. These are not formal conclusions or recommendations per the evaluation questions, but interpretations offered by the evaluation team based on our experience and field work: USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | xiv • There is a lack of before-and-after measures in the spring vulnerability assessments. • There is a need to understand how to support PDAMs that vary by size and strength, and other variations across PDAMs, including how best to sequence and prioritize interventions. • There is a need to assess differentiated needs of smaller and larger PDAMs. Larger and smaller are relative terms; the evaluation team is referring to the different clusters of needs identified among those with more than 100,000 connections (“larger”) and those with fewer than 20,000 connections (“smaller”). • There is a lack of a training database and data on turnover to support follow-up. 1 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 1. BACKGROUND Clean water and safe sanitation are basic human needs that contribute to a healthy community and affect the human development index. The sustainable development goals (SDGs) include ensuring that the community achieves universal access to clean WASH resources (SDG 6) by 2030. The GOI is fully committed to the SDGs and operationalizes this through its national development agenda (National Mid-Term Development Plan or RPJMN). For water, the 2030 SDG target is 100 percent access to clean water and 43.5 percent access to safe water2. For sanitation, the target is 100 percent access to safe sanitation facilities. GOI plans to achieve these targets in stages, according to its RPJMN.3 The GOI cites that some 90 percent of households have access to improved drinking water, per the SDG definition, with access to safe drinking water and sanitation services improving. Figure 1: Water access targets and achievements (left); sanitation access (right) Indonesia’s rate of urbanization and decentralization is the fastest of any major economy in East and South Asia. These two dynamics challenge the provision of sustainable WASH services, with local institutions doing their best to ramp up quickly. LGs, which own the PDAMs, mandate that the PDAMs supply water in urban and semi-urban areas. Of the 380 PDAMs, national agencies consider just under 70 percent to be financially healthy. They face human resource limitations and raw water availability and sustainability challenges, which vary in time and across geographies. Technically, they face high NRW, low EE, low billing efficiency, and tariffs that do not cover their costs. For sanitation, 2 “Clean” water is shallow well, spring, river or rainwater used by communities for daily needs but not treated. “Safe” water is water that has been treated and piped to communities according to Ministry of Health drinking water standards. 3 Sources: Draft SDGs National Action Plan Goal 6, SDGs National Secretariat, Bappenas, 2021; Voluntary National Reviews (VNR), Bappenas, 2019. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 2 the challenges include a lack of community awareness and little funding, regulation, or stable institutions. DESCRIPTION OF THE USAID ACTIVITY In Indonesia, USAID supports the GOI to increase access to improved water and sanitation services and improve key hygiene behaviors for the urban poor and vulnerable, contributing to the GOI’s universal WASH coverage target. The key outcomes of USAID’s investments in Indonesia are 1) increased household level hygiene behaviors and use of water and sanitation services; 2) improved technical, operational, and financial performance of PDAMs and LGs that regulate, oversee, and deliver water and sanitation services; and 3) strengthened governance and finance functions of WASH service providers, targeting beneficiaries in the bottom 40 percent of the population by wealth, known as the “B40” group. The USAID Indonesia Urban Water, Sanitation and Hygiene Penyehatan Lingkungan Untuk Semua (IUWASH PLUS) activity under USAID contract number AID-497-TO-16-00003 with funding of $48.36 million, implemented by DAI Global LLC, is in its fifth year of a contract spanning five years and eight months (June 22, 2016–February 21, 2022). The activity supports and works with GOI and other donors, the private sector, non-governmental organizations (NGOs), and communities to increase access to water supply and sanitation services and improve hygiene behaviors among urban poor and vulnerable populations. The activity builds on two decades of related work. USAID designed IUWASH PLUS to strengthen the urban WASH ecosystem to reach the most vulnerable households and communities effectively and inclusively. To ensure sustainability of improvements in access to WASH services, the IUWASH PLUS development hypothesis focuses on strengthening systems to deliver these services as a whole, which includes reducing access barriers for the poorest and most vulnerable and strengthening the underlying enabling environment. This contract therefore involves a network of partners: national and LG agencies, utilities, the private sector (including investors), financial institutions (both micro and more standard lenders), and the communities and households themselves. We include the activity’s theory of change (TOC) diagram in Annex H. Its four components are below: 1) Improving household WASH services 2) Strengthening city WASH institutional performance 3) Strengthening the WASH financing environment 4) Advancing national WASH advocacy, coordination, and communication4 The activity targets the following high-level results by February 2022: (i) Increased access to improved water service quality to at least 1.1 million urban residents, of which 500,000 are from the poorest 40 percent of the population (ii) Increased access to safely managed sanitation services for at least 500,000 urban residents 4 Information comes from IUWASH PLUS Annual Reports, the evaluation statement of work (SOW), and documents the implementer, USAID, or online sources have provided. 3 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV In 2019, USAID created a partnership with the Swiss State Secretariat for Economic Affairs (SECO). SECO added US$4.5 million to support the technical challenges of PDAMs: high rates of NRW and poor EE. SECO-funded activities provide in-depth mentoring, TA, and related equipment to help seven PDAMs in West and Central Java achieve NRW reduction and EE improvement. At the end of Project Year (PY) 4, in September 2020, USAID IUWASH PLUS was on target to achieve increased access to improved water service quality but not its target of 500,000 people from among the bottom 40 percent, or B40. The activity achieved and surpassed the high-level result on safely managed sanitation services, with 604,670 residents. In a further indicator, IUWASH PLUS reached 121,365 people in the B40 with access to improved sanitation services, equal to 48.6 percent of its target of 250,000. The Mission commissioned this final evaluation to assess and analyze activity achievements, impacts, and progress toward IUWASH PLUS activity water goals, especially the USAID-SECO scope, and to examine the sustainability and replicability of the activity approaches. The evaluation also discusses lessons learned to inform USAID’s global learning agenda on urban water interventions. The evaluation focuses on Components 2 and 3, though also reports results from Components 1 and 4. 2. EVALUATION PURPOSE, AUDIENCES, AND USES Per the USAID Statement of Work (SOW), the purposes of this final evaluation are to: 1) Assess the performance of USAID-SECO partnership program 2) Measure the project contribution to improving PDAM performance 3) Assess IUWASH PLUS’ contribution to improving groundwater recharge and ensuring access to safe drinking water for the community 4) Assess IUWASH PLUS’ contribution to improving WASH financing policies and regulation and increasing private sector investment in the sector Per the SOW and discussions with the Mission, the evaluation will inform the Mission and their SECO partners on challenges, opportunities, and lessons learned during implementation, which will support future program development. A follow-on activity is expected, and for this reason the evaluation results will be particularly timely, and the Mission will share the results widely with stakeholders, such as national and LG agencies, NGOs, USAID implementing partners (IPs), community partners, and donors. The evaluation report will build the evidence base and learning priorities of USAID’s global learning agenda on urban WASH. This final evaluation will be a performance evaluation as defined in the USAID Evaluation Policy. USAID will post all evaluation materials to the USAID Development Exchange Clearinghouse (DEC). 3. EVALUATION QUESTIONS The evaluation addresses the following evaluation questions (EQs) and subsidiary lines of inquiry. (1) What is driving PDAM performance improvements and what is the role of the BPPSPAM and IUWASH PLUS indices in performance improvement? a. What is needed to replicate the use of USAID IUWASH PLUS’ performance index to improve performance of other PDAMs? b. To what extent has USAID IUWASH PLUS assistance prepared PDAMs to manage both water supply and wastewater services as mandated by RPJMN 2020-2024? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 4 (2) Are IUWASH PLUS activities resulting in sufficient reduction of NRW and EE improvements to put PDAMs on a pathway to business viability?5 a. In what ways have USAID IUWASH PLUS interventions contributed to the improved capacity of PDAMs to reduce NRW and increase EE? b. In what ways have USAID IUWASH PLUS interventions influenced local government investment for water utilities to reduce NRW and increase EE? c. To what extent have IUWASH PLUS activities established the foundations on which PDAMs can continue to make improvements in NRW and EE beyond IUWASH PLUS? d. Did NRW and EE remain prioritized areas of investments for PDAMs, even in times of fiscal constraints due to COVID-19? e. Have the measurements and baseline studies supported the commitment and ownership of the PDAMs to work on NRW and EE? f. What is the likelihood that measurement equipment purchased by the project will be used to undertake NRW and EE improvements independent from the project in the future (in the supported PDAMs, or in PDAMs outside the project)? g. To what extent was the IUWASH PLUS-SECO component successfully integrated into the overall IUWASH PLUS program and made efficient use of existing structures, networks, and processes? (3) In what ways has USAID IUWASH PLUS’s training and human resources development roadmap contributed to the improved PDAM performance? What accounts for successes? What accounts for failures? a. Are there plans to continue the completion of the PDAM human resources development roadmap? b. In what ways has USAID IUWASH PLUS helped with identification of PDAM training centers? c. To what extent does the roadmap present a feasible and systematic capacity strengthening model for utilities? Will IUWASH PLUS-SECO training materials and trainers support this? Are the NRW/EE field examples helpful? (4) What evidence is there that spring vulnerability assessments have resulted in improved sustainable management of raw water sources and use by drinking water providers? a. To what extent are there plans to replicate USAID IUWASH PLUS spring vulnerability assessments in other spring catchment areas? b. To what extent has the addition of infiltration ponds increased drinking water source yield or reduced flooding, if at all? c. What project-supported water resources management (WRM) models, if any, are the most promising for the sustainability and resilience of drinking water sources? (5) In what ways have IUWASH PLUS’s activities contributed to PDAMs’ ability to secure alternative financing? Given that government budget allocation is insufficient to reach GOI’s own goals for WASH services nationwide, this EQ will provide insights on the 5 Business viability in this case refers to the potential for PDAM to recover cost for their NRW/EE investments and to increase revenue from service expansion as a result of the amount of water saved and redistributed to new customers. 5 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 6 Lines of inquiry are: a. In what ways have USAID IUWASH PLUS interventions contributed to the development of the alternative financing guidelines? b. In what ways have USAID IUWASH PLUS interventions helped PDAM with business-to￾business preparation? The evaluation team disaggregated perceptions data by gender and considered gender as applicable to each evaluation question in our instruments and protocols, per the requirements of the evaluation Statement of Work (SOW). We analyzed the data for evidence of differential results or other gender-related issues. More on this is included in the findings chapters. Given the long list of purposes for the evaluation (above) and that of EQs here, the evaluation team offers this “crosswalk” to show how the EQs relate to the purposes (Ps) of the evaluation: Table 1. Crosswalk between evaluation purposes and evaluation questions EQ1 EQ2 EQ3 EQ4 EQ5 P1: Assess the performance of USAID-SECO partnership program P2: Measure project contribution to improving PDAM performance P3: Assess IUWASH PLUS contribution to improving groundwater recharge and ensuring access to safe drinking water for the community P4: Assess IUWASH PLUS contribution to improving WASH financing policies/regulation and increasing private sector investment 4. METHODS AND LIMITATIONS The evaluation team designed this evaluation7 to collect data and information from a broad range of stakeholders and beneficiaries. The team ensured independence of the evaluation process and accuracy and completeness of the subsequent findings, conclusions, and recommendations. Because of the COVID-19 crisis, all methods were virtual. The team began collecting data on August 1, 2021. Evaluation methods included individual and small group interviews (open-ended instruments for which are included at Annex B) and analysis of existing documents and data. We selected, in consultation with USAID, SECO, and IUWASH PLUS, 11 sites from among the 35 implementation areas: 6 Examining impact on LG fiscal capacity and solvency became unnecessary during the evaluation. The team assumed IUWASH PLUS had explored alternative financing at the LG level, but this turned out not to be the case. 7 Further details on the evaluation methods can be seen in: MEL-P, 2021. Evaluation Design and Work Plan and Annex C. degree of success in IUWASH PLUS efforts, as well as areas for improvement. We will examine the impact on LG fiscal capacity and solvency. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 6 Table 2. Sites consulted for the final performance evaluation PDAM SITE PROVINCE Kota Surakata Central Java Kabupaten Magelang Central Java Kota Depok West Java Kabupaten Karawang West Java Kota/Kabupaten Jayapura Papua Tirtanadi (Kota Medan) North Sumatra Kota Pematang Siantar North Sumatra Kabupaten Probolinggo East Java Kabupaten Sidoarjo East Java Kabupaten Gresik East Java This sample is not representative and cannot be construed to reflect all the IUWASH PLUS sites because of the variation among PDAMs. Site selection criteria can be found in Annex C. The team concluded interviews with these sites (senior and mid-level PDAM management, technical teams, finance members, and others) and with central government and other interviewees in Jakarta in late September 2021. We carried out an online mini-survey of sampled CB participants, chosen to represent the larger group of trainees from across the country. We also examined these responses by whether the respondent came from a small or large PDAM. More information on the sample and how we selected the sample and analyzed the data is in Annex C. For this survey, we closed data collection on October 5, to allow respondents to complete it. We collected and analyzed quantitative data from key program documents and data (e.g., work plans and monitoring, evaluation, and learning and performance reports, as well as results from before￾and-after PDAM data and PI data). Please see Annex E for the list of documents and data consulted. The Team Leader and a Monitoring Evaluation and Learning Platform (MEL-P) evaluation specialist reviewed incoming notes, cross-checked interviews, and identified edits. MEL-P regularly monitored data collection to ensure team members properly completed, named, and archived their notes, with standard metadata values and EQ coverage. During fieldwork, the team identified a lack of information on alternative financing and attempted to rectify it; when that was unsuccessful, they reached out to USAID and the IP. The team worked across specialties to analyze the suite of EQs. We shared formally at mid-point and again at the close of interviews, and identified gaps that we subsequently filled with additional data collection—interviews, document/data review, or both. We triangulated findings on most aspects of each EQ from multiple sources. Kabupaten Barru South Sulawesi LIMITATIONS The following limitations affect data quality and thus the findings, conclusions, and recommendations. They are common to qualitative evaluations, though fully remote evaluation methods, like those used in this evaluation, do exacerbate the possibility of biases. It is impossible to evaluate with no 7 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV limitations, but our strong, local team has mitigated most negative effects per the strategies described below. (1) Recall bias: IUWASH PLUS has been in operation for over four years; some respondents have difficulty recalling details of past activities. This is particularly true with the presence of the COVID-19 crisis in recent memory, which may dwarf other recall. Our team worked with respondents as necessary to set their recall in context to mitigate this bias. (2) Social desirability bias: Respondents tend to under-report socially undesirable answers and reply with what they suspect the interviewer wants to hear. To mitigate, the evaluation team used informed consent and confidentiality as possible and established strong rapport. (3) Translation: Most interviews were conducted in Bahasa; notes were prepared in English. To avoid inaccurate recording or understanding, the team quickly and repeatedly reviewed their notes after interviews and other exchanges, often with their colleagues for sense checking, to mitigate this limitation. (4) COVID-19-related effects: Changes in conditions due to COVID-19 restrictions and budget changes no doubt affected IUWASH PLUS interventions, and likely affected the evaluation as well. Findings reflect these effects and make explicit where and when COVID￾19 affected both. While we cannot mitigate these effects per se, by making them explicit, the reader can better set them in context. The reader should note that the evaluation is not tasked with verifying quantitative project-related data as reported in project annual and other reports. It is assumed that (historic) figures recorded in these reports are accurate per the Mission’s standard Data Quality Assessment protocols, and the evaluation team uses those percentages as a shorthand for activity successes and challenges. 5. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS This chapter explains our answers to the EQs and cross-cutting issues. EQ1: PERFORMANCE INDICES AND DRIVERS EQ1: What is driving PDAM performance improvements, and what is the role of the BPPSPAM and IUWASH PLUS indices in performance improvement? The IUWASH PLUS PI and the BPPSPAM index measure PDAM performance across a series of sub￾indices and indicators. The IUWASH PLUS team custom-built the PI for this activity, and noted in interviews that the PI was not intended for comparative purposes between PDAMs, but rather to be used within (and where desired by) an individual PDAM to identify and address specific areas for planning and growth. The PI also measures progress from IUWASH PLUS interventions. Within the PI are six sub-indices, covering administration, service delivery, raw water, financial performance, operations, and HR development. IUWASH PLUS collected data annually from 2017 to 2020 on its intervention PDAMs using this tool. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 8 At the same time, the GOI had its own index,8 which BPPSPAM developed as a benchmarking tool by which to grade and compare all PDAMs. The approach is similar approach but there are differences, particularly around HR and financial indicators. Respondents told the evaluation team that the biggest problem with the BPPSPAM index was the annual data collection using self￾completed questionnaires that are not independently verified. This index is easily and inexpensively scaled nationwide. The PI procedure, by comparison, involved face-to-face meetings twice a year to verify the data, with a mentoring modality that would be very expensive to replicate nationally. The Drinking Water Directorate is likely to share the final 2020 report near the end of 2021. Going forward, GOI will have to create a new index. When asked about the drivers or factors behind PDAM performance improvements, interviewees cited the use of the IUWASH PLUS PI coupled with mentorship and CB or TA as key drivers. This mentorship was customized for the PDAMs and applied intensively and iteratively. Across the IUWASH PLUS-SECO and IUWASH PLUS sites, average PI scores improved, which interview and survey respondents attributed in part to the mentorship. With IUWASH PLUS-SECO, the TA was more intensive and it is notable that the PI scores from these sites averaged almost double the average gain in IUWASH PLUS sites. This result is not likely entirely due to the differences between mentoring intensity, but may in fact reflect SECO’s selection of strong PDAMs to begin with. However, combined with providing equipment and hands-on practical training in its use in the IUWASH PLUS-SECO sites, it does provide some evidence that the intensity is a driver. Other drivers included new PDAM leadership, and good relationships with LGs. Across our sample of 11 sites, seven also had new leadership in place during the project lifetime, and respondents said new management techniques like task prioritization, using data for decision-making, and using SOPs were important drivers. Respondents cited a third driver as well: the commitment and support some PDAMs felt from the LGs to which they report. The commitment and support included investment of capital in infrastructure, passing tariff adjustment regulation, increased and improved communications, and refraining from political intervention. Interviewees attributed these improvements to the IUWASH PLUS interventions and mentoring, in whole or in part. This engagement from the LG complemented PDAMs’ own commitment to work closely with IUWASH PLUS and to take best advantage of the support. A respondent described the PDAM selection process like this: “We always assessed locations by their potential to improve services to reach USAID targets, but only in locations with high enough commitment. We measured this by their (active) participation in our meetings, their documents, their historical performance, one-on￾one talk with key staff, their ongoing business plan showing their priorities, etc.” Participating PDAMs often exceeded targets, and while credit is clearly due to USAID IUWASH PLUS for the support, this strong level of commitment has also been vital for success. When we examine the PI by sub-index themes, the average improvements come mostly from the administration category, as the following table shows. This category includes the presence and implementation of SOPs, a signed BP, and a suitable activity plan and budget. 8 A comparison of the two indices is offered in Annex K. 9 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 2. PI score changes, 2017–2020, by sub-index scores 62% 77% 46% 43% 68% 14% 61% 73% 49% 48% 70% 18% 67% 78% 50% 50% 80% 24% 62% 80% 53% 54% 88% 34% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Finance Service Operations HRD Admin Raw water 2017 2018 2019 2020 Source: IUWASH PLUS PI dataset; evaluation team calculations It is important to note here that the improvements in PI scores may not be entirely the result of IUWASH PLUS programming, given that PDAM contexts and conditions are dynamic and affected by a variety of factors, including GOI interventions. That said, average scores in operations and HRD categories also improved, though modestly. Financials did not improve, and the services category, which reflects increases in household connections, increased slightly. The raw water category showed strong relative improvement but with low absolute scores, and a large number of PDAMs reporting zero scores. This aspect of the activity is only implemented where the PDAM relies on spring water for their raw water source. As a result, the positive visual in the figure above comes disproportionately from a few PDAMs. CB participants who responded to the mini-survey said that the performance of their PDAMs since working with IUWASH PLUS was either better (over 60 percent) or far better (over 20 percent). This was true whether for both sites supported by IUWASH PLUS-SECO investments and those with IUWASH PLUS investments alone. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 10 21% 61% 16% 3% 23% 66% 9% 2% 0% 10% 20% 30% 40% 50% 60% 70% Far better Better Same Worse IUWASH PLUS-SECO IUWASH PLUS Source: Evaluation team calculations; n=94 We examined the variance in responses from respondents from larger PDAMs (over 100,000 connections; see Annex C for details) compared to those from smaller PDAMs (fewer than 20,000 connections). Around 80 percent of all respondents said their PDAM’s performance was better or far better over the last 12 months, because of participation with IUWASH PLUS-SECO. Those who said “far better” were concentrated in smaller PDAMs—some 40 percent from smaller PDAMs, compared to 15 percent from larger PDAMs. This slightly more enthusiastic response from smaller PDAMs may reflect that the IUWASH PLUS inputs are unique and therefore more memorable and impactful than for larger PDAMs, where other more frequent government and donor activities may have effects as well. The survey also asked participants about their perceptions of specific PDAM performance improvements (see Figure 4). IUWASH PLUS-SECO respondents were more likely to report improved NRW reduction and EE improvement, while IUWASH PLUS respondents were more likely to report improvements in BPs and low-income community outreach. This aligns with the differences in programming between IUWASH PLUS and IUWASH PLUS-SECO programming and mentoring priorities. More IUWASH PLUS-SECO respondents also reported improved performance in reaching full-cost recovery (FCR), though only around a quarter of respondents cited this improvement. Survey respondents reported almost equally on improvements to clean water supply, wastewater treatment,9 and financials. Figure 3. Perceptions of PDAM performance, IUWASH PLUS-SECO vs. IUWASH PLUS 9 Wastewater treatment was not part of the IUWASH PLUS PI but was selected by evaluation survey respondents as one of the areas of improvement they noticed; respondents were permitted to select more than one area of improvement. 11 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 4. Perceived improved aspects, IUWASH PLUS-SECO vs. IUWASH PLUS 21% 23% 30% 43% 45% 45% 68% 74% 28% 45% 31% 34% 28% 59% 38% 72% 0% 10% 20% 30% 40% 50% 60% 70% 80% Reaching full-cost recovery Energy efficiency Financial condition Performance measurement Outreach low income community NRW reduction Business plan and targets Clean water supply IUWASH PLUS-SECO IUWASH PLUS Source: Evaluation team calculations; n=94 Respondents from smaller PDAMs were also more likely to say they saw these improvements in the areas of developing BPs and targets, customer outreach, and NRW reduction, while those from larger PDAMs cited wastewater treatment, reaching FCR, and energy efficiency improvements. Key factors that affected performance improvement or change included having a comprehensive, measurable BP, HR development, and competent leadership, among others (and with some variation between IUWASH PLUS-SECO and IUWASH PLUS sites): Figure 5. Perceived drivers of improvement, IUWASH PLUS-SECO vs. IUWASH PLUS 14% 36% 36% 36% 42% 54% 56% 60% 64% 70% 19% 23% 42% 32% 65% 58% 48% 58% 81% 58% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Access to alternative financing Legislative support Innovative technical approach Collective decision-making Improved staff capacity Strengthened SOPs LG support Competent leadership Human resource development Measurable business plan IUWASH PLUS-SECO IUWASH PLUS USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 12 Female respondents reported experiencing significantly more benefits (per the list of benefits in the table above) than did male respondents; due to the small sample size, it is difficult to conclude that this finding is statistically robust. However, it may reflect different expectations that females and males have of CB opportunities or of their effects on outcomes. It is also likely to reflect differentiated roles, with women trainees coming more frequently from finance offices while male respondents came from a range of roles, including more technical and leadership positions. More research will be necessary to understand this finding in depth. EQ1a: What is needed to replicate the use of USAID IUWASH PLUS’ performance index to improve performance of other PDAMs? Replicating the performance index, to a small or large degree, is now in the hands of the GOI. As IUWASH PLUS comes to a close, the tool itself and the principles behind it are available to the GOI to use – in whole or in part. The goal of replication actually arrives at an opportune moment, when the GOI’s agency that administered its own index has been shuttered. As a result, IUWASH PLUS and any successor programs have a window in which to support the GOI in developing and implementing its own index, thereby helping to improve PDAM performance across the country. Currently the GOI’s plans for an institutional home for a future, nationwide index. The evaluation team heard conflicting preferences from MOHA and MPWH on where the index would be housed, or whether it was important to have one consolidated index to serve multiple purposes. The GOI may select the Keuangan Daerah (KEUDA) office of the Ministry of Home Affairs (MOHA) to develop and administer their index, for several purposes, including benchmarking, performance appraisal, and credit approval. IUWASH PLUS, A Possible office under the Directorate of Water Supply has been deemed too small to fully cover the functions carried out by BPPSPAM in the past. The GOI appears to clearly understand the value of intensifying support to PDAMs, continuing to monitor and provide support to improve PDAM outcomes on an eventual index to be determined. USAID, GOI partners, and some PDAMs have suggested that the GOI should include the PI or parts of it in the GOI’s eventual instrument. While PDAMs will comply with whatever measure the GOI uses, they offered praise of the IUWASH PLUS PI for being, among other things, “simple and operational,” “comprehensive and fair,” and “objective,” with “links to field conditions,” and “accurate for wastewater-related performance.” Additionally, they said, it produces “clear and useful reports” and encourages “two-way discussion and communication.” Several respondents said they would continue to use the PI even after the IUWASH PLUS activity ends in 2022. What will be necessary for these benefits to be replicated is that the GOI takes on board all or part of the IUWASH PLUS index in its deliberations and development of the new tool they will use going forward. To support this, the evaluation team offers a set of recommendations about how to offer support and thereby be involved in the upcoming GOI process. It is also important to note that the IUWASH PLUS index had gaps compared to the GOI’s prior instrument. Respondents noted that the IUWASH PLUS PI lacks a component related to handling user complaints about water quality. One respondent suggested there may be a need to specify the PI for smaller versus larger PDAMs. The IUWASH PLUS team stated that short-term priorities, financial capability, skilled staff, and operation systems of smaller (sometimes weaker) PDAMs often differs considerably from those at larger (sometimes stronger) PDAMs. PI scores at smaller, weaker PDAMs were routinely much lower than those in medium or large PDAMs, as shown in the table below: 13 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 6. Average overall PI scores by PDAM size 66.1 54 34.6 74.4 64.7 36.1 0 10 20 30 40 50 60 70 80 Large Medium Small Average start Average end Average gains in the smaller PDAMs were smaller too; this could indicate that the process gains – for which respondents from smaller PDAMs were more enthusiastic in our survey – were not adequately captured or weighted. The target for PDAM PI gains was 10% of remaining capacity, and on average both large and medium sized PDAMs overachieved by around double. By contrast, smaller PDAMs were far from reaching that target on average. Given the distinct variation shown here, it is worthwhile to consider whether the PI is as sensitive a tool as it needs to be, for the smaller PDAMs. GOI partners will likely be open to support and inputs from IUWASH PLUS and USAID in their upcoming decision-making on this matter, though the process will likely outlast IUWASH PLUS. Additionally, the GOI will surely wish to analyze the data from the final round of BPPSPAM index data collection to understand PDAM progress, gaps, and patterns. It may also be helpful to see the 2017–2020 BPPSPAM data in light of data from the same timeframe from the PI, which would provide an external comparison point on strengths and areas for improvement of each. All of these issues provide entry points for IUWASH PLUS, USAID, and successor programs to become involved and use the experience of the IUWASH PLUS PI to inform the GOI’s tool design and implementation. GOI partners have shown some of the necessary political will to move forward with such a process. They will also need a legal umbrella and an institutional home that can support the PDAMs to use the eventual tool. The latter will be a challenge given the resource and time requirements implied; some representatives of likely agencies indicated low willingness to take on these tasks. In terms of the index itself, the indicators will need to be acceptable across ministries and others, like creditors. Many of these aspects will require funding and attention that the GOI might not allocate, or fully allocate, without USAID support. In addition to the political considerations and opportunities noted here, replication will benefit from taking advantage of IUWASH PLUS’ other lessons learned, per the section above on EQ1 success drivers. New PDAM leadership that brought attention to SOPs, better communication with and commitment from LGs, and targeted support for improvements using a tool like the PI were all cited as necessary to replicate the model elsewhere. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 14 EQ1b: To what extent has USAID IUWASH PLUS assistance prepared PDAMs to manage both water supply and wastewater services as mandated by RPJMN 2020–2024? Based on the 2020–2024 RPJMN, there is a target to combine drinking water and wastewater management in PDAMs. However, this remains optional rather than obligatory. At present, there are different programs and activities underway that have yet to be consolidated into one agency purview. Sanitation is still most often managed by LG-level institutions, though IUWASH PLUS has made strides in helping its partner PDAMs take steps toward shared management. IUWASH PLUS supported two PDAMs (of the total of 35) with merged drinking and wastewater management services to implement L2T2 (scheduled desludging service) program and the wastewater tariff. In a third PDAM, the merging is not yet complete, so IUWASH PLUS continues to help in the creation of the necessary LG regulation. Meanwhile, IUWASH PLUS has supported most of the other PDAMs in a series of steps to prepare for this dual role, as follows: Figure 7. Process steps for sanitation services to merge into PDAMs Source: Evaluation team construction The visual above shows the basic steps necessary for water and sanitation services to merge under the PDAM. Where sanitation was managed by an LG institution, IUWASH PLUS proposed forming a Unit Pelaksana Teknis Daerah (UPTD), or Regional Technical Implementing Unit, for wastewater services. In these cases, the services are offered to citizens without payment. Forming a UPTD establishes a dedicated unit to support activities and plans, and while not strictly necessary, does streamline the later steps including necessary appraisals as this process continues. Where this UPTD already existed, IUWASH PLUS supported PDAMs to begin desludging services, known as L2T2. With L2T2 services in place, IUWASH PLUS proposed forming a transparent, flexible and non-profit Badan Layanan Umum Daerah (BLUD) or Regional Public Service Agency to manage wastewater, and provided CB and TA to calculate the wastewater tariff and develop a tariff regulation. The BLUD has flexibility to implement sound business practices that can improve service, and to use profit generated from drinking water to finance the waste management project. Where the BLUD, desludging and tariff regulation were in place, IUWASH PLUS proposed merging the BLUD into the PDAM (as mandated by the government’s master plan) which would separate it 15 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV from the LG and unite it with the PDAM’s Badan Usaha Milik Daerah (BUMD) structure – a Regional owned enterprise. There are details and variations that could occur within this stepwise process, and increasing complexity through the steps, but on the whole, the team finds that IUWASH PLUS has pursued the most straightforward and logical manner of supporting PDAMs to merge wastewater under their management. Today IUWASH PLUS is supporting three regions to form the BLUD. As a result of their work with the remaining supported PDAMs, there is greater uniformity and increased understanding of the steps necessary for merging wastewater services, though appropriate domestic water regulations and tariffs are lacking. IUWASH PLUS has also developed tools as part of its collaboration with the GOI, that the latter can now use to determine PDAM eligibility for this process. A report on how IUWASH PLUS facilitated this process in Kota Malang – which has progressed the furthest – is available to share with other LGs as they proceed, and IUWASH PLUS suggested to the evaluation team that they could provide lessons learned to support this process. Evidence like that presented here, and more detailed IUWASH PLUS experiences in this integration, provide important inputs to the GOI for their own planning around integrating these services under one roof, CONCLUSIONS • Improved performance, measured using the PI, under IUWASH PLUS has been driven by a suite of promising and replicable characteristics: o Continuous communication between LGs and PDAMs o LG commitment—including financial commitment to—PDAM successes o In several cases, good, new leadership in the PDAM, using SOPs, data and prioritization to good effect o Mentorship (TA and CB) with the PI as a tool to improve performance • Improvements on the administrative sub-index have been an important part of average PI score gains, along with HR reforms and those in operations to a lesser extent. Finance and coverage (“service”) sub-indices have not improved much on average, if at all, while raw water scores improved but for only a small group of PDAMs. • The IUWASH PLUS PI has been an important success for the activity and the participating PDAMs. While this index served activity purposes, the GOI BPPSPAM index remained in place for GOI needs like benchmarking and ranking. Now that BPPSPAM is defunct, there is an opportunity to work with the GOI on a new national index that serves multiple purposes like performance appraisal and improvement and benchmarking. IUWASH PLUS and its successor could conceivably have significant input into the new index based on the successes of the activity PI. There are many hurdles to overcome, however, such as the legal regulations, an institutional home, mentoring resources, and an evidence base for a new index. In addition, the costs to scale up IUWASH PLUS’ intensive TA program are significant and will be burdensome to GOI efforts. • Key operational, human resource, and financial characteristics appeared to differ between smaller and larger PDAMs, in line with findings in other sections of this report. PI scores for smaller, weaker PDAMs were lower on average and improved less. • IUWASH PLUS’ efforts toward merging drinking water and wastewater under the PDAM’s management have been broad and in line with PDAM needs; most are not yet ready for this, but IUWASH PLUS support has brought them closer to this goal by close technical assistance on these issues. There is no GOI obligation to merge these functions in PDAMs, leaving the IUWASH PLUS effort unsustainable. PDAMs will need regulations to make the USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 16 route to follow clearer. Experiences from IUWASH PLUS will provide inputs for GOI decision-making on these issues. RECOMMENDATIONS • Commitment from PDAMs and LGs is critical for success and should be a criterion for participation in future USAID-supported activities in this sector. • Build on the success of the PI. For the remainder of the activity, and in any ongoing programming, USAID should target collaboration around the development of a GOI index applicable nationwide for performance appraisal, improvement, and benchmarking. IUWASH PLUS can be of service with solid analysis of the last BPPSPAM data (2020); inputs from what worked in the IUWASH PLUS PI; design support to ensure the index meets the needs of different agencies and the PDAMs themselves; and other technical supports. • IUWASH PLUS should work with GOI in the final months of the activity to pass on lessons learned around mentoring, such as the types of experts and TA that are most beneficial, cost estimates and ways to minimize costs at scale, guidelines for the TA, and how TA might be applied differently for PDAMs with different needs. This should be closely integrated with work to support a future GOI PI. • IUWASH PLUS should develop the lessons learned in merging wastewater management under the PDAM into a document that can be shared with GOI prior to the activity’s end. • USAID should also support GOI to develop regulations, technical guidelines, and IUWASH PLUS-type pilots to help merge water supply and wastewater services, if this continues to be a priority for the Mission and the GOI. • Hold a roundtable with IUWASH PLUS TA teams on the topic of specifying the PI for smaller and larger PDAMs, given that smaller, more remote PDAMs had sometimes quite different needs. • Indicators and targets in the PI could be scaled for smaller PDAMs, to allow for more indication of process advancement. EQ2 NON-REVENUE WATER AND ENERGY EFFICIENCY INTERVENTIONS EQ2: Are IUWASH PLUS activities resulting in sufficient reduction of NRW and EE improvements to put PDAMs on a pathway to business viability? To date, the reduction of NRW and improvements in EE are promising but insufficient for business viability. This work was concentrated in twelve sites of the activity’s total of 35 (five IUWASH PLUS sites and seven for IUWASH PLUS-SECO.) IUWASH PLUS-SECO interventions were somewhat more intensive, with the provision of equipment, action plans following training, and increased technical assistance. They also conducted a customer water meter survey. Our evaluation sample included three of the IUWASH PLUS sites and four of the IUWASH PLUS-SECO sites, so we looked at seven of the twelve. National targets for NRW are 20 percent, and NRW reduction remains a priority at national level and across agencies (while EE was said not to have emerged as a national priority). In project PDAMs, just as in PDAMs across the country, water loss remains consistently higher than 30 percent. Reducing water loss should result in lower water production, and lower operational cost. This would improve cash flow, and if connections are increased and the network expanded, revenue will subsequently increase, which will be a pathway to business viability. Using reduction of NRW as an example, the diagram that follows shows this iterative process, once sites are selected: 17 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 8. Iterative process of NRW reduction in PDAM Reduce commercial and technical losses Less production is required, reducing operational cost Cash flow improves, and production increases Connections increase, and network expands Revenue increases, increasing viability Diagnose commercial and technical losses (equipment, training) This is a simple diagram. But in actuality, PDAMs operate in complex institutional, technical and socioeconomic environments, which can (and did) negatively affect PDAM achievements. For example, Covid-19 meant that fewer potential clients added household connections than might have been possible under stronger economic conditions (among other effects: please see EQ2d, below). The NRW/EE projects are piloted and remain in early stages, covering only pilot areas of the PDAMs’ service areas. The focus on business viability is not always a priority for the LGs that fund PDAMs, or consistently across PDAM technical teams, who have other targets and priorities. Some PDAMs may not have a significant potential customer base in which to grow, which will ultimately limit their ability to expand revenue: though they can certainly be better at source conservation by doing this work, they may not have as much potential for long-term business viability. As a result, PDAMs interviewed mostly reported that the impact on revenue of reducing NRW remains minimal, and had no revenue data. A former IUWASH PLUS leader told the evaluation team, “It requires a serious change of attitude of PDAM management to prioritize NRW reduction, from reducing the production when not needed, buying enough quality water meters, constantly measuring and monitoring flow and pressure through the whole system, and investing in their staff.” That reducing NRW and improving EE (which is reported to be simpler and more easily manageable for PDAMs) should have the effects noted does not mean that they always will have those effects. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 18 On the positive side, early data do point to increased household connections10 as a result of the NRW and EE interventions – which over time should equate to increased revenue. Parts of this cycle do seem to have the intended effects. It is difficult to estimate the timeline or precise conditions under which the desired business viability will be seen. As the processes are cyclical and subject to interruptions as described above, and potentially others, PDAMs will need ongoing support to reach these goals and to approach business viability. Notably, GOI partners have indicated a willingness to find the funding to provide such equipment, and to mandate an agency to provide ongoing technical assistance and mentoring. They also report a need to know more about the types of experts, costs and time required. Reducing NRW and improving EE makes better use of the existing treatment plant, which can mean postponing new production unit investment – easing PDAM financial burdens. Both NRW and EE interventions begin with an assessment of the existing system and a determination of pilot areas. IUWASH PLUS and IUWASH PLUS-SECO then measure NRW and conduct an energy consumption audit, and finally prepare a feasibility study and implementation plan. IUWASH PLUS is implementing NRW reduction programming in five sites and EE improvement programming in four sites. IUWASH PLUS-SECO is implementing NRW work in seven sites and EE work in five. Non-Revenue Water For NRW, the assessment includes evaluating the pipe network and choosing pilot areas. To set the baseline, they subtract the amount of water billed from the amount produced in a given month. They then conduct field inspections on the condition of the pipe network and read consumer meters, then analyze NRW issues and causes. The feasibility study then shows the scale of water loss and assesses the technical and financial feasibility of the planned implementation. IUWASH PLUS and IUWASH PLUS-SECO work with the PDAMs to set targets and budget for implementation. COVID restrictions affected some of the implementation, given the need to consistently measure and monitor flow and pressure throughout the system. NRW consists of commercial and technical loss,11 with different tactics and benefits of reducing each. IUWASH PLUS-SECO respondents reported that the focus was first on reducing commercial loss, which is less costly but complicated to implement, because it involves management practices in different units or divisions, not all of whom are supportive in loss reduction. To work with this issue, IUWASH PLUS used its integrated model that involved regular monthly meetings among related units and even an NRW reduction task force. Still, regularity in meetings was uneven across PDAMs. While the evaluation team has no direct evidence of a lack of champions for the effort, we speculate that PDAMs may need to task a dedicated, high-level leader to overcome inertia and keep NRW reduction as a top priority, during and after implementation. 10 Increase in net connections occurred not only in the 12 NRW/EE PDAMs, but across almost all supported PDAMs, indicating that any intervention impact is due not only to NRW/EE work. The increase varied across PDAMs from fewer than 100 units to over 15,000 units. 11 Commercial loss is due to illegal connection and consumption, illegal uses, meter inaccuracy, inaccurate meter reading, and mismanagement of data. Technical loss is due to losses in aged transmission, distribution pipes, and service pipes. 19 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV For IUWASH PLUS-supported PDAMs, support is in the form of capacity building during a project to reduce NRW in a pilot area, applying SOPs to establish a district metering area (DMA) that can isolate and accurately measure a given area. The pilot project also includes a survey of informal consumption and testing the accuracy of customer meters, both of which revealed Iosses. As a result, PDAMs replaced inaccurate and aged meters and predicted issues and causes. Other implementation under this rubric includes installation and application of a geographic information system (GIS) to map the pipe network; financial and economic analysis and report; tariff adjustment; customer reclassification; and establishment of NRW task force. IUWASH PLUS-SECO implemented in their PDAMs similarly, but also provided master meters and data loggers, equipment for loss detection, and customers’ smart meters. Further, they carried out a survey to determine the conditions of customer water meters for 32,000 customers in six PDAMs. They found anomalies in over 4,500 units, or 14.6 percent of occupied homes, about half of these in Kota Surakarta. Where customers continued using water, this was classified as an illegal or informal connection, since their use was not properly recorded and billed. IUWASH PLUS-SECO recommended immediately replacing anomalous meters, then repairing leaky pipes, and other remedial actions. Some PDAMs took actions that resulted in gradual reduction of commercial losses. Kab. Sukoharjo, for example, replaced 560 meters, allocated funds to replace 1,500 more, and installed three master meters with data logger using its own funds and three that were funded by the project. Impacts in reduced commercial losses are yet to be known or reported. Physical losses remained problematic due to ongoing drainage due to pipe damage in some construction areas. The project’s Year Four Annual report includes no figures for NRW reductions or EE improvement. Per IUWASH PLUS interviews, the average NRW of their five sites was 47 percent at first measurement, and 40 percent after the capacity building – which they credit primarily to reducing commercial losses – and which they expect to improve upon wider implementation of their NRW measures. For technical losses, PDAMs have committed to purchase master meters and prepare detailed engineering designs, and to develop DMAs in other areas, which they report they will implement themselves. Energy Efficiency IUWASH PLUS implements an energy efficiency in four PDAMs and IUWASH PLUS-SECO implements EE programming another four. Using a ratio of energy costs to operational costs they selected pilot areas and conducted a feasibility study in those areas, ultimately calculating a break￾even point and preparing a conservation plan to reduce energy consumption with no-cost, low cost, or medium/high-cost options outlined. Interventions included pump rehabilitation or replacement, installation of capacitor banks, replacement of bearings and other pump accessories, and others. The implementers and PDAMs prepared targets for reducing energy usage in terms of savings in kilowatt-hours. IUWASH PLUS has not yet issued a report on progress of energy efficiency investments. Initial information suggests reduction in energy usage of 33 percent at Kab. Karawang, of just under 7 percent in Kab. Bogor, and of 17 percent in Kab. Sukoharjo. A PDAM respondent in the latter site confirmed this report, and committed to continue the EE activities across the region. They invested in two pumps that complement the two they received from IUWASH PLUS-SECO. The implementers and PDAMs expect these positive figures to improve still further in the coming months as monitoring continues. Respondents from the implementing partners suggest that EE work is considered secondary to the work of NRW, but that it is also easier to manage at PDAM level, including measurement and USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 20 implementation. At the same time, EE has not yet emerged as a national concern, and is only a concern in PDAMs that use significant energy levels for pumping. Working on EE is also less complicated than NRW programming, but is still an important component for PDAM improvements. Performance and planning NRW and EE activities appear to have a generally positive effect on the operational sub-index12 of the PI, which includes several indicators that these activities directly affect. Improving performance on this sub-index supports eventual business viability. The graphs below track that effect in the seven IUWASH PLUS-SECO PDAMs (at left) and the five IUWASH PLUS PDAMs (at right). The trend is more mixed for PDAMs supported by IUWASH PLUS, and more consistently positive for IUWASH PLUS-SECO-supported PDAMs. With five and seven sites respectively, the trend is not strongly substantiated, but worth pursuing in further implementation to contribute to the evidence base around what works in the sector. Figure 9. Operational indicator by site and IUWASH PLUS-SECO vs. IUWASH PLUS - 5.0 10.0 15.0 20.0 25.0 IUWASH PLUS-SECO Base PY2 PY3 PY4 - 5.0 10.0 15.0 20.0 25.0 IUWASH PLUS Base PY2 PY3 PY4 Source: IUWASH PLUS PI scores for 12 sites PDAM respondents said that project support with business plans and feasibility studies were hands￾on and particularly beneficial as the basis for implementing an NRW/EE program. It should be noted, however, that when new PDAM leaders arrived during IUWASH PLUS, their BPs were often thoroughly retooled—something to prepare for when PDAM leaders leave. EQ2a: In what ways have USAID IUWASH PLUS interventions contributed to the improved capacity of PDAMs to reduce NRW and increase EE? The IUWASH PLUS team has selected, in most cases, larger and healthier PDAMs, which has helped them to get satisfactory results in relatively less time. In 2019 BPPSPAM PDAM status data, 85 percent of IUWASH PLUS-supported PDAMs were “healthy” (28 of 33). Four were deemed “less 12 The operational sub-index includes six indicators. The first is percent NRW, which is weighted at 8% of the overall PI; the others are 4% of the PI. These five are: having an NRW unit/program; functioning master meter; progress on household connection meter replacement; GIS on customer spatial data; and connection to computer-based applications. 21 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV healthy” and one “sick.”13 A large majority (79 percent) of supported PDAMs had over 20,000 connections in the 2019 data and so were relatively large. Respondents working with PDAMs said communication with larger, healthier PDAMs is easier. The opposite is also true: smaller, weaker and more remote PDAMs had lower levels of performance at the outset, improved less, and continue to have significantly lower scores on average. The NRW and EE interventions were primarily carried out in stronger PDAMs: capacity gains should be understood in this context. Among the capacity building activities provided by IUWASH PLUS, respondents said that preparing business plans was one of the most important. BPs were generally prepared at a high level of quality with significant collaboration to encourage PDAM ownership. The BP serves as a planning tool for the PDAM and the basis for preparing the annual budget plan of the company, called the Rencana Kerja dan Anggaran Perusahaan (RKAP). Participants praised other capacity building around reducing NRW and improving EE, particularly those trained by IUWASH PLUS-SECO (given their focus on these issues). Please see more on capacity building topics in the response to EQ3 on the HR roadmap below. Also cited by respondents as a useful CB topic was the use of GIS applications for measuring the condition and efficiency of assets, such as the piping network system, to call for quick response in case of any system disturbance. These GIS systems combine maps and a related database. For smaller PDAMs, this process was reported to have been challenging, in some cases even after training. The difficulty appears to lie in the lack of computers and computer literacy among staff at smaller PDAMs. One important difference between IUWASH PLUS and IUWASH PLUS-SECO sites was the latter’s provision of equipment to their PDAMs, including master meters, data loggers, smart meters and pumps. Respondents found this equipment to be effective in reducing NRW and improving EE. Using the equipment gave teams hands-on technical practice above and beyond the CB support provided to all IUWASH PLUS sites. IUWASH PLUS support included installing an ultrasonic flow meter for their PDAMs, to measure water production accurately; this portable flow meter will be granted to Akatirta upon project completion. A respondent from Akatirta confirms that it has already been loaned to a non-project PDAM, at no cost apart from a nominal maintenance fee. PDAMs will install new main meters with data loggers with their own funds, according to respondents, but they will continue to use the portable terameter to check the accuracy of customer meters. IUWASH PLUS also introduced the use of open-source hardware (OSH) on real-time water pressure and level sensor. OSH was used by PDAMs as a tool to monitor water pressure and level in real time. The use of OSH reduced monitoring costs significantly. Planning, budgeting, capacity building and equipment support were the keyways the interventions helped PDAMs improve their capacity to continue reducing NRW and improving EE after IUWASH 13 BPPSPAM scores based on an index of performance indicators in finance, services, operational and human resources with scaled scores equal to: “healthy” for a score over 2.8, “unhealthy” from 1.8 to 2.8, and “sick” if less than 1.8. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 22 PLUS ends. The equipment in particular allows better measurement to diagnose issues and identify potential solutions. EQ2b: In what ways have USAID IUWASH PLUS interventions influenced local government investment for water utilities to reduce NRW and increase EE? IUWASH PLUS interventions have facilitated and encouraged LGs to allocate equity funding (Penyertaan Modal Pemerintah Daerah—PMPD) to PDAMs. To those LGs participating in GOI-funded performance-based grants (PBG) to reduce NRW and/or improve EE, local government funding is required to participate in grants financed by the World Bank’s National Urban Water Supply Project (NUWSP). With multiple requirements for participation, the process to gain PMPD took longer than planned, as it demanded LG PMPD regulation (Perda), local council approval, a letter of interest, the required feasibility study documents, BP, tariff, and financial reports, among other requirements. Out of the 12 PDAMs that IUWASH PLUS assisted with NRW (eight of which were also supported on EE) to prepare to implement the PBG program, seven LGs have committed (three under IUWASH PLUS support and four under IUWASH PLUS-SECO support) and made PMPD available, while five were still in process. The next step is for MPWH to process the grant applications and submit it to the Ministry of Finance (MOF), which will issue a grant recipient letter indicating commencement of PBG implementation. With or without the PBG, Government Regulation (Peraturan Pemerintah) 54/2017 obliges LGs to provide equity to PDAMs, particularly when these are rated unhealthy. PMPD is also used by PDAMs for operational needs, such as for chemicals, production and staff salaries. Beside the NRW/EE program, PDAMs use LG equity funds for Water Hibah (a connection program for B40 groups) or for PBGs, in compliance with MOF requirements. To some PDAMs, the requirement to have local government funds available to participate in the grants program adds a difficulty that could be avoided if that requirement were deferred until the time when the grant is to be released, so that LGs need not reserve those funds indefinitely during the rather long application process. This falls under GOI policy and authority, and represents the current agreement between MPWH and MOF. It is not likely that LGs invest in PDAMs expressly for the sake of reducing NRW or improving EE, as the EQ seems to signify. The LGs and PDAMs interviewed for the evaluation tended to report that LGs appreciated the professionalization and improvements they saw as a result of IUWASH PLUS interventions, but that political concerns about tariffs were more important for elected LG members. Of course, this would vary significantly across LGs. EQ2c: To what extent have IUWASH PLUS activities established the foundations on which PDAMs can continue to make improvements in NRW and EE beyond IUWASH PLUS? All interviewed PDAMs stated that they will continue to implement NRW reduction, with measures varying from one PDAM to another; some expressed the need to have continued support from IUWASH PLUS or other donors to implement NRW and EE strategies. Most PDAM respondents further stated that their staff are generally capable of carrying out NRW reduction programming, but that they lack in commercial analysis of NRW reduction. IUWASH PLUS TA has worked to fill in this need for support, but several PDAM respondents continued to raise this concern. The IUWASH PLUS team itself shared concerns about NRW programming sustainability, as did central ministries. GOI respondents said they wanted a model of NRW programming to encourage other PDAMs to tackle NRW. Others wished the NRW reduction program had started earlier and 23 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV nationwide. It is important to note, however, that the IUWASH PLUS-SECO model included both equipment costs and much higher technical assistance costs, in sites that were relatively close to Jakarta. Members of the IUWASH PLUS team estimated the costs to replicate this NRW/EE work at levels that are higher than GOI will be able to spend, meaning that it will be necessary to seek significant cost savings through economies of scale, sharing equipment, peer-to-peer business planning or other support, streamlining capacity building, and other efforts. EQ2d: Did NRW and EE remain prioritized areas of investments for PDAMs, even in times of fiscal constraints due to COVID-19? In interviews, all PDAM respondents said they would continue to reduce NRW and improve EE. However, pandemic realities forced LGs to refocus budgets, which affected the release of equity funds to PDAMs, varying in severity from one LG to another. Some funds are released in stages rather than in one tranche; in other cases, targets have had to be adjusted. Alternatively, the equity funding was in competition with the Water Hibah project the LG also wished to fund, as in one case among our sample. Among the PDAM survey respondents, the great majority reported that the COVID-19 pandemic had affected PDAM performance; around 30 percent noted PDAM performance was “deeply affected.” Only eight percent said performance was not affected, as the figure below shows. Also visible below is the similarity of responses from IUWASH PLUS-SECO and IUWASH PLUS sites. Figure 10. Experience of COVID-19 impact, IUWASH PLUS-SECO vs. IUWASH PLUS Source: Evaluation team calculations; n=89 These responses imply significant budget impact from the redirection of funds during Covid-19. At the same time, IUWASH PLUS reports on PDAM budget allocations; USAID and SECO can see in those figures a more direct answer to this question. When IUWASH PLUS staff were asked about impacts of Covid-19 on NRW and EE programming, they referred to their as yet unpublished report which states that capacity building continued, though virtually, and field surveys continued by following the appropriate health protocols. Their report goes on to state: Covid-19 impacted PDAM operations in areas such as decreased water revenue, the inability to add customers because of reduced marketing, and making estimated rather than actual meter readings. The revenue decrease affected cash on hand, further lowering the collection rate and postponing capital expenditures (capex) and budgeted investment costs. In the short and medium term, the 33% 30% 31% 61% 58% 60% 6% 9% 8% IUWASH PLUS-SECO IUWASH PLUS Total Not affected Affected Deeply affected USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 24 minimal cash balance has continued to allow PDAMs to survive through the pandemic. In the longer term, however, the delay in capex spending will eventually cause delays in service coverage. (Excerpted from IUWSH PLUS working paper titled: “Financial Impact of Covid-19 to PDAM”, the evaluation uses Covid Assessment Tool developed by WB. EQ2e: Have the measurements and baseline studies supported the commitment and ownership of the PDAMs to work on NRW and EE? Measurements and baseline studies have supported PDAMs in several ways. Respondents said that accurate measurements revised their previous NRW baseline figures upward. In the absence of a master meter, the NRW figures were formerly estimated quite inaccurately. Where no meter existed in the past, the practice was to assume water loss by calculating production through pump curve. Where meters did exist, they were not calibrated and did not have data loggers installed. Accurate measurement will be key to reducing technical and commercial loss in well-apportioned DMAs covering 500 to 1,500 customers; these are to be established after completing digitized mapping and metering production. The higher but more accurate measurement was dispiriting to some PDAMs, particularly where LG support was tentative or reluctant. It implies more future work to reduce the NRW but also ushers in accurate measurement to show eventual gains more reliably. Five-year targets for NRW reduction and EE improvement were set jointly with the respective PDAMs, included in their BPs, and approved by LGs. The targets are set with consideration of the capacity of PDAMs to allocate budget and human resources. Only PDAMs that use energy for their pumps consider EE improvement a priority. Efficient use of energy is indicated by the ratio of energy costs and operational costs. If the ratio is less than 15 percent (in accordance with BPPSPAM criteria), energy usage is considered efficient. Another important measure in targeting energy efficiency improvement is specific energy consumption (SEC), which is projected to reduce during the implementation period. According to respondents from IUWASH PLUS and IUWASH PLUS-SECO, the progress on EE improvement is still being monitored and a report is yet to be issued. Nevertheless, from information provided, the preliminary findings indicate promising results. PDAM respondents confirmed that the efficiency improves gradually and that measurements are yet to be carried out. EQ2f: What is the likelihood that measurement equipment purchased by the project will be used to undertake NRW and EE improvements independent from the project in the future (in the supported PDAMs or in PDAMs outside the project)? It is likely that the seven IUWASH PLUS-SECO-supported PDAMs will use project-provided measurement equipment to continue to undertake NRW and EE improvements in the future. The equipment, particularly the master meter with data logger, has helped these PDAMs accurately measure production capacity to ensure accurate NRW and energy usage measurements, which are essential steppingstones for NRW reduction and EE improvement program in the years to come. As noted above, IUWASH PLUS-SECO provided their PDAMs master meters, equipped with data loggers to record real-time water flow (NRW and EE) and pumps (EE alone). These meters were equipped with data loggers to record real-time water flow. When IUWASH PLUS-SECO introduced OSH on water pressure sensors as described above, PDAM staff was subsequently trained, resulting in sensors functioning well. Seven PDAMs will use master meters under IUWASH PLUS-SECO to 25 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV gather accurate NRW data for performance-based grant implementation related to the NRW/EE program. These meters and pumps were permanently installed and put into operation. Other tools for reducing technical losses include leak detection surveys and ultrasonic flow meters. These were loaned to PDAM staff for use in the pilot areas14 under cooperation with Akatirta (Water Academy of Technology Tirta Wiyata, based in Magelang). Upon completion, the equipment will be returned to Akatirta for ongoing CB. An Akatirta respondent confirmed that the equipment can be used by any PDAM upon request, and that in fact they have loaned it to a non-project PDAM already. The cost for the loan is only a small maintenance fee. EQ2g: To what extent was the IUWASH-SECO component successfully integrated into the overall IUWASH PLUS program and made efficient use of existing structures, networks, and processes? Interviews within the implementing and funding teams have confirmed the parallel and complementary support that IUWASH PLUS and IUWASH PLUS-SECO have provided for NRW reduction and EE improvement. Although the activities are carried out in parallel in the two sets of sites, and despite the difference of equipment provision with IUWASH PLUS-SECO sites, both are implemented by the same team with the same goals. They have shared experts and knowledge, such as on financial aspects. Integration between IUWASH PLUS and IUWASH PLUS-SECO was reported to be a win-win and positive: USAID obtained broader coverage, and IUWASH PLUS-SECO stepped in with low overhead cost and implemented its vision in tandem, rather than starting from zero. Where there were challenges at the outset, these were mostly due to unfamiliarity, as reported by respondents. These smoothed out over time as the teams came to know and understand one another and ensure needs were met. CONCLUSIONS • Despite the important supports provided, NRW/EE interventions have not been able to bring these PDAMs onto a pathway to business viability. Although the intervention has resulted in improved cash flow and increased nominal house connections, PDAMs did not significantly increase revenue. For replicating in other sites, in addition to the support for reducing NRW and increasing EE, new connections could be an intermediate target, along with network expansion, to increase PDAM revenue and bring them closer to business viability. • IUWASH PLUS and IUWASH PLUS-SECO interventions have contributed to reducing NRW and improving EE through technical assistance and capacity building. In some PDAMs, the operational sub-index improved, and in others, there has been a net increase in household connections, in part through legalizing informal and illegal connections. However, as stated above, these have not yet led to increased revenue. • IUWASH PLUS and IUWASH PLUS-SECO have provided support to PDAMs for multiple steps that prepare them for greater viability: preparing BPs, advising on tariff adjustments, studying feasibility, reclassifying customers, surveying customer satisfaction, applying GIS, measuring NRW, and financial reporting. The PDAMs used the business plans in particular as 14 Pilot areas are located in all seven PDAMs supported by IUWASH PLUS-SECO: Kota Depok, Kab. Bogor, Kab. and Kota Magelang, Kab. Karawang, Kota Surakarta, and Kab. Sukoharjo. In Kabupaten PDAMs, pilot areas are only in selected zones. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 26 a strategic document in five-year planning and as the basis for annual budget plans. These have equipped PDAMs to implement the NRW/EE program sufficiently. However, reducing NRW and improving EE are cumbersome and long-term tasks that require consolidated efforts of PDAM management, and it takes time to reduce NRW even to standard estimates of 1-2 percent per year. • Interventions have succeeded in encouraging LGs to increase financial support to PDAMs, resulting in increased equity funds as a statement of LG commitment to support PDAM management improvement. LGs value professionalization, but do not specifically focus on NRW or EE. • In particular, the IUWASH PLUS-SECO support with equipment and tools (master meter, data logger, leak detection survey, and smart customer meter) was critical for accurately measuring NRW and energy usage in the seven IUWASH PLUS-SECO sites. However, this is very costly and would be a challenge to implement even in all the IUWASH PLUS sites. • GOI partners wish to support these improvements, including by finding block grant funding for the required equipment, but to expand this nationwide will involve significant expense. • Most PDAMs also organized a dedicated NRW reduction task force, though meetings are not regular in all PDAMs. • Significant work remains to bring the remaining 23 IUWASH PLUS-SECO PDAMs to the level of the 12 treated with NRW and EE support. RECOMMENDATIONS • For future interventions, activities should identify and incentivize a champion (a person or unit/division) within a PDAM who will consistently prioritize the NRW/EE program. • Also, for future programming, all PDAMs with which USAID activities work should benefit from an intensive NRW/EE program, as have the treated sites. Moreover, relative improvements in the IUWASH PLUS-SECO versus the IUWASH PLUS sites call for a preference for the IUWASH PLUS-SECO approach and technical equipment over the TA/CB-only approach used by IUWASH PLUS. However, IUWASH PLUS, USAID and the SECO team must examine scaling options for equipment sharing, cascading training and TA, peer-to-peer learning, streamlining capacity building, and other cost savings that would make this recommendation more achievable. Invest in cost-benefit research taking into account the projected savings, revenue gains and economies of scale, to justify greater GOI investment in the sector. • In order to bring the PDAMs closer to business viability, it is necessary to count new additional connections as an intermediate target, and examine network expansion. In using NRW reductions to support this aim, PDAMs will also have to be supported to reduce physical losses as well as commercial losses. • Most LGs do not comply with the current requirement to allocate equity fund in advance of Water Hibah/PBG program implementation. To avoid delays, it is recommended that the compliance for equity fund allocation be deferred to the time of grant release. Consider, too, bringing together MPWH and MOF on this issue to seek a solution that works for both ministries. EQ3: HUMAN RESOURCES ROAD MAP EQ3: In what ways has USAID IUWASH PLUS’s training and human resources development roadmap contributed to the improved PDAM performance? What accounts for successes? What accounts for failures? 27 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV The IUWASH PLUS HR development roadmap is an incipient program built since September 2020 at the request of the GOI. IUWASH PLUS officially launched the roadmap in June 2021. It supports the GOI’s regulation to standardize competencies in the sector, particularly in the smaller PDAMs. It marks a change in focus for the GOI’s MPWH, which before had focused more on infrastructure rather than HR. Since then, IUWASH PLUS and its GOI partners refined the roadmap and released an updated version in September of the same year. It is still not formally implemented by the GOI, as it awaits the stipulation of the MPWH regulation, and therefore cannot yet contribute to PDAM performance (per the EQ text). IUWASH PLUS’ GOI partners related that they will continue to check on the progress of legalizing the Road Map and then operationalizing its plans. IUWASH PLUS has worked on the roadmap since 2020 alongside government and other partners. At present, IUWASH PLUS continues to operationalize the roadmap through developing an occupational map and certification scheme for trainings, as part of the suite of guidelines and tools included in the roadmap. The IUWASH PLUS team is developing the map and retooling training models to be competency-based. All the while, IUWASH PLUS has also led significant CB work of various types as a major component of its work, using technical practitioners and administrative, business and regulatory experts to support PDAM staff. IUWASH PLUS has used a demand-driven training design, in which they built customized modules and materials to meet a PDAM’s specific needs and a facilitator’s expertise. This model approaches TA in many ways, as it provides direct assistance and peer examples for PDAM problems and concerns. However, neither a fully customized nor a fully standardized model is likely to work across the range of PDAMs, if the expectation is for the modules to continue be used after the program’s end. This does seem to be the case, as IUWASH PLUS is currently making these adaptations. IUWASH PLUS-SECO training reflects some of the best practices from early IUWASH PLUS capacity building, as the training design reflects learning from IUWASH PLUS’ early years in the adapted design. For example, IUWASH PLUS-SECO trainings have the hands-on, practical benefit of particular tools that PDAMs need to improve service delivery and billing, reduce NRW, and increase EE. IUWASH PLUS-SECO training also included the Performance Improvement Action Plan (PIAP— more on this below), tools and equipment, and some standardized modules. Training focus areas align with the survey findings about training topics, disaggregated by type of training participation, as the figure shows below: USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 28 Figure 11. Training topics, by IUWASH PLUS-SECO vs. IUWASH PLUS 47% 63% 56% 53% 45% 48% 48% 20% 22% 21% 14% 17% 16% 32% 26% 8% 9% 9% 18% 25% 22% IUWASH PLUS-SECO IUWASH PLUS Total Business Plan NRW Energy efficiency Performance index Watersource vulnerability Alternative financing Other Source: Evaluation team calculations; n=94 Around a fifth of respondents selected “Other” when asked which training they received. They cited training on tariff calculation, GIS mapping and meter accuracy, customer relations, optimization of the water distribution system, and SOP development. IUWASH PLUS-SECO trainees were slightly more likely to be trained on NRW and EE topics, which were priorities, than on BPs, the PI, or water source vulnerability, which were prioritized elsewhere. IUWASH PLUS trainees were much less likely to have received training on energy efficiency, and twice as likely to have received training on water source vulnerability than IUWASH PLUS-SECO trainees. This is not unexpected and reflects the demand-driven nature of programming, where the PDAMs received support on different topics from among the activity’s full menu of themes. On the administration and financial side, IUWASH PLUS respondents said they appreciated training on BPs and finances that involved multiple PDAM divisions. They further mentioned the development of SOPs, particularly in smaller PDAMs, to improve the quality of work. IUWASH PLUS-SECO and IUWASH PLUS trainees were tested before and after training on knowledge outcomes. The IUWASH PLUS-SECO PIAP mentioned above was a group measurement tool to examine team or organizational outcomes. Immediately post-training, teams completing a PIAP were assigned a project to put the learning into immediate practice and convert training gains into organizational impacts. Completing this step was necessary to receive their certificate and was closely related to certain PDAM functions or tasks, making it highly implementable. Apart from this use of the PIAP, which was confirmed by the survey as well, the evaluation team saw no evidence of PDAMs using the PIAP approach for capacity building at the organizational level. Satisfaction. Trainees were largely content with training, according to the evaluation survey. All respondents said the training was either “highly relevant” (40–44 percent) or “relevant” (the remainder) to their jobs. Negative responses were few and far between, but we can distinguish 29 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV between “highly relevant/satisfied” and “relevant/satisfied.” When asked if they were satisfied with certain aspects of training, respondents answered very favorably, with some patterns in the variations: Table 3. High relevance and high satisfaction, sex and IUWASH PLUS-SECO vs. IUWASH PLUS WOMEN MEN IUWASH PLUS-SECO IUWASH PLUS n=12 n=7815 n=38 n=56 Highly relevant 40% 44% 43% 44% Very satisfied with: Training material/content 42% 32% 22% 42% Training method 42% 31% 19% 42% Real-life examples used in teaching 33% 32% 19% 42% Availability of equipment for practice 33% 29% 23% 34% Trainer quality 42% 42% 25% 53% Source: Evaluation team calculations; n=94 As Table 3 shows, women and IUWASH PLUS respondents reported being “very satisfied” than men and IUWASH PLUS-SECO respondents, though when combined with the “satisfied” responses (not shown), there is not much difference between the respondent types. Across respondent types, the trainers received the most “highly satisfied” ratings out of all aspects we queried in the survey. The greatest dissatisfaction, which was still only around 5 percent, was with the availability of equipment for practice. Around 2 percent were dissatisfied with training material and methods, and use of real￾life examples. Overall, satisfaction is quite high. Interviews with PDAM management confirmed that leaders valued the training highly, perhaps more so in smaller PDAMs. Survey respondents from smaller PDAMs were much more likely to say the training was highly relevant: 75 percent, compared to 35 percent from respondents in larger PDAMs. Slightly more from the smaller PDAMs (93 percent) said they had already applied what they learned from the trainings, compared to those from larger PDAMs (85 percent). The overwhelming majority was positive in both cases. Before COVID-19, most training was face-to-face or offline. With the effect of the pandemic, trainings went online. Average offline trainings had more participants and more hours than average online training activities. Most who took part in offline training reported “up to 50 hours” of training activities with IUWASH PLUS. Those reporting online training (67 percent) frequently reported fewer than 20 hours’ total training time. Activity intensity decreased when online, which was not uncommon or unwarranted, given the challenges of online learning. The respondent total is greater than the sample size because some reported both online and offline training experiences. 15 Four respondents did not select gender. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 30 Figure 12. Offline capacity-building activities were more intensive than online 26% 67% 49% 22% 25% 12% Offline Online More than 50 hours 20 - 49 hours Less than 20 hours Source: Evaluation team calculations; n=94 Applying changes in the workplace. Given the requirement of the PIAP, it would stand to reason if IUWASH PLUS-SECO respondents frequently reported putting training into practice. In the event, however, both IUWASH PLUS-SECO and IUWASH PLUS respondents reported significant use of the training in their PDAMs. Both groups also perceive improvements in personal skills and knowledge. Over two-thirds (70 percent) of the respondents said they need additional training. Interviews also garnered information on the application of training in the workplace, where IUWASH PLUS PDAM staff reported changes in work culture and mastery of applying SOPs, the latter particularly in smaller PDAMs. Respondents credited IUWASH PLUS for these important qualitative changes, along with more quantifiable improvements which, of course, varied by PDAM: • One PDAM said it could finally increase the water tariff with TA from IUWASH PLUS. • Another reduced financial loss from over IDR2 billion to IDR700 million by the end of 2020: “Prior to IUWASH PLUS, we could not even finance our operational cost because of the low level of income, high level of NRW and poor EE, which was below 50 percent.” • A third said it had simplified the payment system using the Siska application and improved SOPs like reducing customer complaint handling from up to a week down to one day. • Larger PDAMs reported more HR planning after participation with IUWASH PLUS and in￾service trainings. Smaller PDAMs could be said to benefit even more from IUWASH PLUS￾SECO trainings, given their very low level of previous access to CB. Sustainability. Sustainability planning is logically local, but as such, reports from PDAMs as to what they would be able to sustain were uneven. Respondents mentioned several challenges to sustainability including whether PDAM leadership is on board, and the impact of Covid-19 on PDAM ability to invest. While it is clear that respondents valued capacity building, the degree to which different training topics are institutionalized, and therefore lend themselves to sustainability, is variable. Most PDAM respondents reported that they expect further support from donors and the GOI to help them put their current capacity building gains to use. Most frequent training topics were business plans, NRW/EE, the performance index, water source vulnerability, and financial topics (in that order). Sustainability of the latter two are discussed in more depth in their respective sections (EQ4 and EQ5, below). Since all the PDAMs now have business plans, sustainability of that training will involve updating and ensuring that the business plan is carried 31 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV out – which, per the response to EQ1 above, would benefit greatly from ongoing GOI mentorship since it touches most areas of PDAM operations. Respondents from PDAMs trained on NRW/EE activities said they will continue to reduce NRW and increase EE because these benefit their operational and financial performance; this is visible in some PDAMs’ business plans. One smaller PDAM reported creating a working group to continue the GIS work begun as a result of this training. As noted in the response to EQ1, some PDAM respondents said they would continue to use the PI to improve their own scores, independent of any GOI performance measure. Others said that could prove difficult, if and when the GOI begins to administer a new index of its own, because they would then be duplicating effort. Sustainable application of the PI was probably the least certain because it would be additional to requirements and because its effects, if any, on PDAM’s operational and financial performance would be indirect. When asked what kind of training they still needed, survey respondents from smaller versus larger PDAMs varied in their responses.16 Respondents from larger PDAMs requested more training on NRW reduction, increasing service coverage, financing, best practices, and solutions for challenges like raw water supply that crosses regions. They also requested opportunities for a comparison study or a visit between PDAMs. These advanced and technical requests are qualitatively different to those requested from smaller PDAMs, where requests were for training on capacity building in finance, provision of direct financing, and provision of facilities and tools. The survey responses reflected a similar inclination the team had from our interviews. Interviewees from larger PDAMs were focused on improving the company profile in terms of business areas and financing for new projects. Interviewees from small PDAMs were more inward-focused and expressed more appreciation of small gains in organizational arrangements and business continuity. Training model. It is unclear if there was an underlying model for the IUWASH PLUS-SECO trainings that contemplated the best methods for reaching adult learners, taking learners beyond knowledge and skills acquisition to usage and reflection or self-efficacy, and tracking and ensuring use of the learning. IUWASH PLUS did not have in place a formal system for monitoring the trainees’ progress after training, or for linking training gains to organizational improvements. IUWASH PLUS￾SECO sites used the PIAP to conduct follow-up with participants on action plans, a useful adaptation of the original design. GOI partners agreed that follow-up on effects on trainees and on organizations would be necessary for scaling the Road Map across the nation. In examining the outcomes of the training through the survey data, the CB appears adept at bringing trainees to a higher level of knowledge of their work but not as proficient at usage and reflection. In Figure 10 below, IUWASH PLUS-SECO and IUWASH PLUS respondents report they understand their field of work better, but only around 40 percent feel more confident or capable (which means some 60% do not). 16 Please note here that the survey is not representative, nor is the sample particularly large: while this list is interesting for its variance, it does not speak for PDAMs nationwide, for which a needs assessment would be more appropriate. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 32 Figure 13. Personal changes as a result of training, IUWASH PLUS-SECO vs. IUWASH PLUS 34% 49% 43% 37% 43% 41% 94% 80% 86% IUWASH PLUS-SECO IUWASH PLUS Total More capable of doing my work More confidence Understand my field of work better Source: Evaluation team calculations; n=90 Interestingly, those responding from smaller PDAMs seem to have developed greater feelings of confidence and capability compared to those responding from larger PDAMs. The sample size is not large enough or representative enough to call this a definitive finding, but it may be that in a smaller PDAM setting, each individual’s comparatively greater level of responsibility means that knowledge increases translate quickly into use and feelings of self-efficacy. Organizational gains. About 87 percent of respondents stated the IUWASH PLUS trainings have made a difference to their organizations. Having a BP to guide the PDAM strategically was by far the most frequent change respondents reported (for both IUWASH PLUS-SECO and IUWASH PLUS); this parallels BPs being the most frequently cited training. IUWASH PLUS respondents seem to perceive more organizational change than do IUWASH PLUS-SECO respondents; however, IUWASH PLUS-SECO participants were more likely than IUWASH PLUS to cite an increase in PDAM energy efficiency and increased ability to develop alternative financing. Note that 12 responses were missing for this question, which perhaps indicates slightly less confidence in organizational changes. 33 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 14. Organizational changes as result of training, IUWASH PLUS-SECO vs. IUWASH PLUS 74% 81% 78% 48% 53% 51% 39% 30% 33% 42% 40% 41% 29% 45% 38% 16% 15% 15% IUWASH PLUS-SECO IUWASH PLUS Total Having a business plan to guide PDAM strategically Reduced NRW Increased energy efficiency Increased capability to measure PDAM performance Having an action plan to deal with watersource vulnerability Increased capability to develop alternative financing Source: Evaluation team calculations; n=82 Successes and failures. We cover these in the Conclusions section that follows the discussion of sub-EQs, to avoid repetition. EQ3a: Are there plans to continue the completion of the PDAM human resources development roadmap? MPWH reviewed the roadmap and is expected to issue a related ministerial regulation. However, there is a possibility for more delays due to the change of leadership in the relevant directorate at MPWH. Without a ministerial regulation, the relevant Directorate General cannot expect financial support or the involvement of other government agencies in implementing the roadmap. The IUWASH PLUS team notes the need for MPWH buy-in in this uncertain environment, and say they plan to ensure the new leader is aware of the roadmap and on board with implementation. In addition to the preparation of the roadmap, IUWASH PLUS-SECO supports the development of an occupational map and studies potential training centers. The IPs also assisted in creating guidelines and tools for the assessment of PDAM training centers. Finally, IUWASH PLUS is establishing a training portal based in Persatuan Perusahaan Air Minum Seluruh Indonesia (PERPAMSI), with a database on competency-based and non-competency-based trainings, developed by PERPAMSI under IUWASH PLUS funding. It is likely that the development of the training portal and database will continue post-IUWASH PLUS, considering that the portal is part of the plan to build the e-training system for PDAM. It was not part of the roadmap itself, but will relate to it in terms of infrastructure support for capacity building. EQ3b: In what ways has USAID IUWASH PLUS helped with identification of PDAM training centers? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 34 IUWASH PLUS is currently developing guidelines for PDAM training centers that address Indonesia’s training qualifications (KMP—Kualifikasi Mutu Pelatihan). IUWASH PLUS has also completed a mapping study of PDAMs and other institutions with training centers or high potential. IUWASH PLUS respondents report that some PDAMs improved their training centers by referring to the KMP in the IUWASH PLUS-supported guidelines. PDAMs with training centers are Palyja in DKI Jakarta, Samarinda, Pontianak, Surabaya, Malang City, and Bandung Regency. However, only Palyja’s training center is certified. Non-PDAM institutions that have training centers are Yayasan Pendidikan Tirta Dharma (YPTD) PERPAMSI and Balai Teknik Air Minum dan Sanitasi (BTAM), the latter with technical training for operators and financial staff. YPTD has signed a memorandum of understanding with MPWH to support BTAM as a center with competency-based trainings. EQ3c: To what extent does the roadmap present a feasible and systematic capacity strengthening model for utilities? Will IUWASH PLUS-SECO training materials and trainers support this? Are the NRW/EE field examples helpful? If adopted, the roadmap, occupational map, and competency certification would change PDAMs’ CB approach from the national level. Individuals will be required to have relevant competencies to occupy certain positions, a move respondents praised for improving the quality and standards of HR. Most IUWASH PLUS modules were developed for particular PDAM needs, rather than being competency-based (with reference to MPWH Regulation No. 15 Year 2018 requiring this feature). This means that other potential users (like GOI or USAID) cannot easily apply these again without adaptation to fit these requirements. IUWASH PLUS has promoted some modules that are competency-based to the Directorate of Drinking Water and BTAM for ongoing use. IUWASH PLUS-SECO is in the process of redeveloping some of the modules to align with national standards. These include Occupational Safety and Health (Keselamatan dan Kesehatan Kerja or K3), NRW, EE and asset management. The K3 module has been piloted in some of the IUWASH PLUS-SECO PDAMs, and is currently being reviewed for publication. The remaining modules will be piloted in November 2021 to January 2022. The plan is to handover the competency-based modules to the BTAM and PDAM training centers. For the particular technical trainings related to the NRW/EE field examples, respondents cited the need for specialized equipment for the trainings. The examples would be helpful in this case, if indeed the PDAM has the necessary equipment. CONCLUSIONS The strength of the IUWASH PLUS CB lies in its relevance to PDAM business processes, as well as well-prepared training materials, training methods and trainer quality that participants appreciated. IUWASH PLUS has also tried to link its CB with organizational impact through action planning, though only for IUWASH PLUS-SECO trainee teams. However – except for an increase in knowledge and better job understanding – it was difficult to trace higher-order, longer-term outcomes from existing program documentation, since there is no system for that, or for relating the training to any outcomes that might be identified. Our survey respondents reported that HR improvements were major drivers of performance improvements – something neither IUWASH PLUS nor USAID would know were it not for the evaluation. A system to measure gains from training would be a complex undertaking, one that would unfold over time as capacity matures, and it is ultimately the responsibility of the GOI. However, given IUWASH PLUS capacity building investments, support for a system that could be passed on to PDAMs for this purpose is lacking. 35 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV This makes it challenging to assess in what ways and to what extent these activities contributed to changes at the organizational level, if at all. Detailed conclusions include: • Resources, TA, and best practices differ for larger and smaller PDAMs, which therefore need the training model most appropriate for their size, relative levels of resources and customers, technical advancement, and unique environments. IUWASH PLUS’ efforts to differentiate levels within PDAMs is important, but there is also a need to ensure the trainings take into account the differences across PDAMs with greater and lesser levels of development and capacity. • While progress monitoring and the action plans were more intensive for IUWASH PLUS￾SECO trainings, there was little if any follow-up on IUWASH PLUS trainees. There is no information readily available on individual or group outcomes from training, apart from the action plans, and capturing this would have been difficult as there was no database on trainees. • The action plans capture group development in a given project, but not how individuals have grown as a result of training. In both cases, accessible data on participants, training types, durations, etc., would have been a help to the IUWASH PLUS team. • If IUWASH PLUS has a model behind the training design for how adult learners will learn, master, and utilize the training, that might be tracked and tested, it is not apparent from the way IUWASH PLUS and IUWASH PLUS-SECO have undertaken the trainings or monitored gains and gaps. • The development of fit-for-purpose modules was a boon for PDAMs whose needs were addressed in a customized fashion. Across the board, training was appreciated and valued, and there is at least some evidence of positive outcomes on a case-by-case basis. CB and TA both moved PDAMs up the scales of the sub-indices of the PI. However, the custom-built modules must now be aligned with the competency model demands of government HR and CB standards. • The HR roadmap is still a work in progress. IUWASH PLUS is nearly out of time to take actions to embed the model with GOI partners. This would have meant more certain sustainability; as it stands, the sustainability of the model is in question. Understandably this is a complex and multi-year effort – but there is reason to be concerned that it will not survive the close of IUWASH PLUS. • Overarching enabling factors included the commitment of PDAM leadership, sending the right staff to given trainings, and encouraging real change post-training. Adapting the IUWASH PLUS model for IUWASH PLUS-SECO, with the provision of tools and equipment, the use of the PIAP, and the standardization of some models, worked to produce convincing technical results. RECOMMENDATIONS • IUWASH PLUS should capture implementer staff knowledge around the key differences between smaller and larger PDAMs in intervention design. Smaller and larger may not be the only or definitive axis, but rather weaker and stronger. Smaller project PDAMs tended to be weaker and larger ones stronger, in this case. These distinctions could be helpful for the GOI in its use of the roadmap, as well as for any follow-on USAID activity in the sector. IUWASH PLUS and USAID should keep this on the GOI’s radar and ensure that the follow￾on program takes this into close consideration. • Base any training modules on the occupational map and Indonesian training qualification standards fit to the competencies required for PDAM staff roles. Customized training will still be needed, but in future programming, the formal modules should always meet national USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 36 standards. At the same time, the approach cannot be “one size fits all” because of the previous recommendation, above. GOI and future USAID programs should use TA to fill gaps and adapt modules to meet PDAM needs. GOI and USAID programming in the sector must take into account the stark differences between smaller and larger, or weaker and stronger, PDAMs for program and institutional decision-making. • USAID should require future activities to plan and monitor training gains better. This includes maintaining a current and thorough database of trainees (including turnover), tracking both individual and organizational change, and identifying ways to embed such tracking within the operationalized roadmap in the future. • IUWASH PLUS and its successor need to collaborate with GOI partners in their continuing advocacy for the issuance of a ministerial regulation. IUWASH PLUS should also seek buy-in from MPWH on the PDAM HR roadmap and the completion of key components before the close of the contract. EQ4: SPRING VULNERABILITY ASSESSMENTS EQ4: What evidence is there that spring vulnerability assessments have resulted in improved sustainable management of raw water sources and used by drinking water providers? Raw water scarcity, particularly in the dry season, is growing alongside population increases, land development and changing land use, more demand for (and decreased availability of) raw water for drinking water, reduced water catchment areas, and climate change. Water resource conservation efforts are needed through the spring vulnerability assessments, the construction of infiltration wells, and additional measures. At present, these efforts are limited in scope in Indonesia. The IUWASH PLUS activity used these methods with 14 PDAMs in KKMA. The urgency of the KKMA work is highlighted by the GOI’s own projections for extreme dry climate conditions in the coming years, as the figure below shows from GOI’s RPJMN IV 2020–2024. The drought will impact all raw water sources in the absence of GOI planning. There is little evidence that the KKMA assessments have resulted in improved management, though respondents spoke anecdotally of improvements to source supply in the 14 sites where IUWASH PLUS undertook this work. Unfortunately, this was not confirmed with reliable before-and-after measurements, which would have been both challenging to undertake and very worthwhile as evidence for future work. Further, the PDAM teams reported not constructing as many infiltration ponds as IUWASH PLUS recommended, so it is not known what the full effect might have been in terms of water supply. Additionally, they are working on this issue without the necessary GOI regulations in place, so whether this will result in ongoing source management is in doubt. 37 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 15. GOI projection for extreme dry climate events, 2020–2025, RPJMN Source: RPJMN IV 2020–2024 (revised August 20, 2019, page 210) EQ4a: To what extent are there plans to replicate USAID IUWASH PLUS spring vulnerability assessments in other spring catchment areas? Current GOI plans to replicate spring vulnerability assessments in other spring catchment areas are not fixed. GOI partners have requested before-and-after data on the infiltration ponds, as evidence on the value of these simple interventions. At present, an LG regulation on the topic is in process, with the support of IUWASH PLUS but also COVID-related delays. If finalized, it will regulate the agency in charge of catchment protection to build and maintain more infiltration wells, ensure communication across related agencies, and allot budget to the effort. IUWASH PLUS also helped to establish a Mayor’s or Regent’s decree for the implementation of infiltration wells, including the cooperation of the private sector – this also supports replication. These efforts at present are not yet broadly applicable, but GOI might spread them more widely if political will and resources are priorities. Given the approaching drought cycle (per the GOI’s own analysis, shown in the figure above), it is reasonable to assume the GOI will be seeking ways to resolve water limitations in the coming years, and the KKMA program provides achievable results. MOEF respondents say they will apply some of the IUWASH PLUS methods in delineating recharge areas for springs, as a result of the KKMA program. In this way, the spring’s recharge area becomes a priority for intervention. In 2021, MOEF has compiled ten spring point profiles and delineated the recharge area for the restoration of springs. In the future, this could be expanded and replicated in other areas, such as those with water crises, like Java. Another option comes from PERPAMSI, which reported replicating the KKMA program in other PDAMs using a corporate social responsibility (CSR) scheme, if and when PDAMs ask for support to do so. EQ4b: To what extent has the addition of infiltration ponds increased drinking water source yield or reduced flooding, if at all? Among the evaluation sample, all four KKMA-participating PDAMs reported water increases following their participation and the construction of infiltration wells. Respondents confirmed this increase was not up to the prior level of water quantity, but they were convinced of increase in water supply. No baseline measurements were taken, so any increase cannot be quantified. Though USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 38 baselines would be challenging for these sources, requiring trends data and statistical comparison, this would be a significant support to the GOI in the form of evidence for their own planning. Additionally, respondents noted they had not constructed as many infiltration wells as were recommended by IUWASH PLUS, which may mean there are more gains to be realized. They say they will replicate the KKMA program in their own PDAMs. If and when they do increase the ponds to the recommended level, it would have been of even greater interest to have strong baseline trend data. In this way, the project would have formed a kind of “natural experiment” in which different levels of implementation could be measured against one another, to assess costs, benefits, and effectiveness. The lack of before-and-after measurements means we cannot answer this question confidently. The evaluation team found no evidence to support a reduction in flooding. EQ4c: What project-supported WRM models, if any, are the most promising for the sustainability and resilience of drinking water sources? Respondents praised the KKMA model as a sound method for these purposes. Outside of this, some non-project-supported models emerged from interviews inside and outside the activity. The MOEF is also developing the “rainwater-friendly village” program, which is to be implemented in six provincial watershed control and protected forest areas (representing MOEF in all provinces) in Java Island in a first phase. The program proposes to construct rainwater harvesting installations, infiltration wells, rainwater reservoirs, and bio pore holes, and to conduct environmental rehabilitation. Another example of GOI work in this vein includes using funding from DKI Jakarta to install ponds in Kota Bogor, which thereby reduced flooding in Jakarta. World Bank and GOI partners suggested that the spring vulnerability assessment program also support deep well, artesian well, and river sources. To support water safety SDGs, they also suggested the program should align with GOI’s water safety plan, denominated RPAM. CONCLUSIONS • GOI and PDAM respondents agree that the KKMA work was successful, and one agency is working to replicate following the IUWASH PLUS strategy. However, the fact that this is not yet official policy means sustainability and replication are an open question. • Operationalized GOI regulations with defined responsibilities to replicate KKMA nationwide do not yet exist. • The lack of before-and-after data from the effort is a critical gap that makes definitive decision-making on the topic much riskier. RECOMMENDATIONS • Propose the IUWASH PLUS raw water source programming model to RPAM to support GOI’s efforts to meet SDGs in the sector. With evidence of success, this may encourage fostering relevant regulations for nationwide use of these water supply tools. • IUWASH PLUS should support a KKMA action plan with the GOI and help them identify and test reliable tools to measure before and after implementation. Where data do exist, share these with Bappenas (including any necessary caveats) as part of the evidence base on the value of infiltration ponds. 39 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV • IUWASH PLUS and future implementers should support GOI and LGs to develop water resource conservation regulations involving relevant parties, including interagency communications and the budget. EQ5: ALTERNATIVE FINANCING EQ5: In what ways have IUWASH PLUS’s activities contributed to PDAMs’ ability to secure alternative financing? One purpose of this evaluation, as seen in EQ5, is to assess IUWASH PLUS’ contribution to improving WASH financing policies and regulation and increasing private sector investment. Regulations on the drinking water supply system allow private sector support if the state- or regionally-owned enterprises are unable to finance implementation. This government-owned service provider must have the relevant permits and prioritize people with low incomes. Permissible cooperation types include investment in raw water and production units, or distribution units, which the relevant public body then operates and manages. It is also permissible to invest in operation and maintenance technology to seek effective and efficient implementation with a performance-based contract mechanism. Alternative finance generally refers to financial channels, processes, and instruments that emerge outside of the traditional government budget system, such as a cooperation scheme with the private sector. Partnership schemes and private sector involvement can be categorized as one of the following: build-operate-transfer (BOT) models, turnkey projects, or performance-based contracts. The evaluation team looked at these models, and at IUWASH PLUS-SECO support to preparing PDAMs for PBGs, but not at commercial lending to PDAMs, which were not seen in any case. However, the evaluation team found that a different definition was common across IUWASH PLUS actors and beneficiaries; namely, alternative financing is any source outside the current LG funding – including additional funding from the LG, or grants like PBGs that are helpful but not sustainable financing, per se. Many interviewed PDAMs were unaware of the option of financing outside their LG budget funds. Given that, it is no surprise that for many PDAMs, profit is not a priority. With the equity injections from the LGs, PDAMs can continue running even at a loss. This makes encouraging PDAMs to take on alternative financing more difficult. Given the broad and deep needs for capital expenditure (CapEx) in the sector, identifying a range of sources of capital for PDAMs, including the private sector, is likely necessary. At the same time, to secure alternative financing and attract investors, PDAMs must be creditworthy. Any non￾performing loans (NPL) must be restructured to encourage partnership investment and commercial bank lending. In other ways, the evaluation team found, many PDAMs were not ready yet, particularly those denoted "unhealthy" by the BPPSPAM index. IUWASH PLUS, therefore assisted PDAMs to develop BPs and determine tariffs and adjustments for FCR (either bulk water tariff that PDAM should pay or tariff adjustment for drinking water the PDAM sells). Such adjustments promised to improve PDAMs’ financial performance and creditworthiness, bringing them a step closer to readiness for alternative financing. Additional work particularly with IUWASH PLUS-SECO sites involved supporting PDAMs in NRW reductions and EE improvements and in feasibility studies (please see EQ2 for more information) that would make them eligible for the PBGs and create a virtuous circle for continuous improvement of NRW and EE. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 40 IUWASH PLUS supported PDAMs in annual tariff increase negotiations with the LGs. Tariffs have the largest impact on PDAM business viability, but these are still within the authority of the Regent/ Governor, and LGs in turn need APBD approval for capital injections to PDAMs from the Dewan Perwakilan Rakyat Daerah (Regional Parliament). IUWASH PLUS directly strengthened PDAM capacity in financial analysis based on national audit standards. Using the IUWASH PLUS PI, the activity helped PDAMs build capacity around upstream to downstream performance issues, including financial parameters. However, only eight17 PDAMs have improved on the financial performance sub-index of the PI. (This sub-index includes indicators for FCR, the current ratio, and billing efficacy. FCR is the most heavily weighted in the sub-index and across the entire PI.) Figure 13 shows greater performance change from top to bottom: Gresik had a nearly 12-point improvement, but the four PDAMs with the next greatest change all decreased in their financial PI scores. (For ease of reading, the twenty PDAMs that did not change are not in the chart.) It is promising to see the eight PDAMs with increases (in red), but the performance of the eleven that worsened (in blue)) is more troubling. Those that improved are mostly medium or large and mostly located in Java with one exception. Governance index gains are strong for some, but not for Gresik, which had the highest gains on the PI in this sub-index – perhaps because it was deeply involved in various alternative financing schemes. 17 Among 30 PDAMs with four years’ data. We excluded those starting in 2020, one of which has improved since then. 41 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 16. Financial performance on the PI (base to PY4) Financial Performance Index PY4 - Base -10.0 -6.0 -2.0 2.0 6.0 10.0 14.0 Deli Serdang City Tangerang Regency Pematang Siantar City Probolinggo Regency Ternate City Tebing Tinggi City Sukoharjo Regency Bekasi City Sidoarjo Regency Barru Regency Depok City Malang City Salatiga City Lumajang Regency Sibolga City Karawang Regency Probolinggo City Bulukumba Regency Gresik Regency It is noteworthy that FCR, despite being critical for PDAM financial performance, creditworthiness, and revenue, and the most heavily weighted element of the PI, is not always a priority for LGs. LG respondents appreciated the professionalization and improvements they saw as a result of IUWASH PLUS interventions, but FCR was not the focus. Political issues around tariffs, customer categorization and election seasons make the FCR a sensitive topic for LGs. Indeed, some would rather try to increase the subsidy rather than increase tariffs. This would vary significantly across LGs, who would take into account both political concerns and the ability of the communities, specifically B40 households, to pay increased tariffs. Besides PDAMs' internal issues such as political directors' matters, there are complex tariff issues, their adjustments, and LG policies, which conflict between increasing the drinking water tariff or giving greater capital injections to the PDAMs. Those challenges might contribute to the low or absent gains in financial management for some PDAMs, especially where the drinking water tariff cannot cover PDAMs' operational expenditures. Where PDAMs were ready to secure alternative financing, IUWASH PLUS assisted in the phases from the development of the FS through the tendering process. IUWASH PLUS facilitated existing or planned projects with completion of readiness criteria, especially on the procurement processes such as pre-qualifications, requests for proposals, or other tender documents. This only occurred in some PDAMs, of course, based on readiness. Some examples include Perumda Giri Tirta Gresik Regency with its BOT scheme for a water treatment plant project with a capacity of 1000 lps USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 42 through the B2B cooperation scheme of Sistem Penyediaan Air Minum (SPAM) Bendung Gerak Sembayat. EQ5a: In what ways have USAID IUWASH PLUS interventions contributed to the development of the alternative financing guidelines? IUWASH PLUS supported the development of MPWH Regulation 29/2020 regarding the Provision of Recommendations and Technical Guidelines for Investment Feasibility in drinking water supply. This regulation was made to smooth the GOI’s provision of guarantees and loan interest subsidies. The impact of this regulation cannot yet be identified since it is still new and there is no evidence yet of being applied commercially. USAID IUWASH PLUS engaged with PT Sarana Multi Infrastuktur (PT SMI) in establishing the CapEx loan facility provided to LGs, PDAMs, and the private sector in relation to increasing service coverage for water access. IUWASH PLUS assisted PT SMI in assessing PDAM financial conditions as an off-taker. The IUWASH PLUS consultant analyzed and provided a financial-related recommendations viewpoint. From those real cases, PT SMI developed a guideline document to assess PDAMs’ ability as an off-taker independently, based on project field experience and assistance from IUWASH PLUS. It is hoped that this on-the-job CB, without more formal training on the subject, will allow PT SMI to replicate what they learned. IUWASH PLUS-SECO is currently developing a guideline template in the form of the PDAM Board of Directors Regulation on Drinking Water Supply B2B scheme. The goal is to foster good corporate governance in the implementation of cooperation, such that the PDAM has legal certainty and is transparent, competitive, and accountable. IUWASH PLUS plans to finish this work in November 2021 and socialize it in workshops with PDAMs in January 2022, before closing. In other areas of financing, IUWASH PLUS also assisted Bappenas with a policy paper related to microfinance conditions in Indonesia and what steps to take to expand the use of microfinance in the sector. The MOF acknowledged the paper, and then developed ultra-microfinancing facilities for the microfinance institutions (MFIs) to access working capital for drinking water and sanitation at lower interest rates. Besides conducting socialization and forming a local campaign group to promote hygiene habits, IUWASH PLUS also assisted households and local communities in obtaining microfinance loans to pay for septic tanks and/or house connections for water. Although the financing values were considered small, connecting households to MFIs aids the government’s obligation to improve safe access to water and sanitation. This effort to fulfill household demand for WASH services and products independently is also an alternative financing scheme in the view of the GOI, as it required no or limited government budget. In speaking with MFIs that may wish to participate in such schemes, IUWASH PLUS learned this was their first exposure to this type of water and sanitation financing. For MFIs to expand their services to the WASH business, they need to work with service and product providers to provide or build safe drinking water and sanitation facilities financed by the Water and Sanitation Micro Financing package. Most MFIs also do not have their own builders and construction equipment. IUWASH PLUS introduced them to the materials and providers in the sector, trained them to technical and national standards, and worked with them to develop the financing scheme. 43 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Over time, IUWASH PLUS created a drinking water and sanitation toolkit that can be used by Smallholder Credit/Financing Bank (a form of MFI, known for their acronym in Bahasa BPR), cooperatives, or other interested groups. IUWASH PLUS also cooperated with the Indonesian Monetary Authority at the provincial level to engage with regional-level finance institutions. IUWASH PLUS encouraged them to disseminate information to financial institutions in their area for the microcredit program. Many challenges remain, especially on sustainability. Since IUWASH PLUS only trained several providers/contractors, engaging with another contractor would be challenging for the MFIs. As of December 2020, 27 MFI partners have agreed to work with IUWASH PLUS to expand WASH financing options at the household level. One MFI claimed it had been unable to replicate the IUWASH PLUS work and said it will not continue to focus on the WASH sector without assistance to conduct surveys and recommend potential customers. As a result, MFIs make up a small percentage of new water connection loans, with 381 households in IUWASH PLUS’s tallies as of September 2020. We might also attribute part of this low achievement to the COVID-19 pandemic that caused a financing downturn for the MFIs. Another challenge involves how MFIs engage with PDAMs. If they collaborate, both parties could reduce their operational costs in collecting joint installments from customers. This would reduce the NPL risk of MFIs since PDAMs could directly cut water flow to customers who have not paid. From the interviews with PDAMs, although IUWASH PLUS has initiated discussions on these potential collaborations and benefits, many PDAMs are indeed reluctant to work with microfinance because, they said, they didn’t need it, and it would be complicated. Therefore, currently, MFIs prefer to focus on sanitation rather than the water sector. Furthermore, MFIs are limited in technical and marketing skills, as well as raising public awareness, especially in reaching the B40 category of customers. For their part, MFIs are reluctant because habits (such as defecating in the river) are difficult to change by means of education alone. Making latrines is not a priority for the households. Also, the COVID-19 pandemic makes it more difficult to market latrines to the public. Based on the interview with Islamic-based cooperative Koperasi Simpan Pinjam dan Pembiayaan Syariah (KSPPS) Syirkah that has 18 field officers (six in each of three branch offices), only 20 households have installed new latrines so far. In KSPPS Bakti Huria, around 70 households have been served with the assistance of IUWASH PLUS. EQ5b: In what ways have USAID IUWASH PLUS interventions helped PDAM with business-to￾business preparation? There are two cooperation mechanisms in the water sector between PDAM and business entities, namely PPP and B2B cooperation. The GOI says PPPs provide infrastructure that meets national standards, in which risk is shared between government and private sector partners. A B2B mechanism, on the other hand, is a collaboration between the drinking water service provider and business entities that does not require support from the central or regional government. In terms of B2B cooperation, financing and risks are borne by the business parties, as the government is not involved. The GOI has established the PPP scheme, including its regulations, toolkits, and guidelines. The B2B scheme was essentially unregulated until 2010, when it was generally recognized that in the absence of viable financing mechanisms and lengthy PPP requirements, B2B represented the best option to inject sorely needed investment into the water supply sector within a reasonable time frame. However, to date, GOI has not fully developed the B2B mechanism and PDAM guidelines. Still, the USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 44 GOI (in its 2020-2024 infrastructure plan) proposes increasing private investment dollars to make up the infrastructure shortfall. B2B and PPPs will be necessary to reach the GOI’s ambitious goals for drinking water and sanitation services, but to achieve that, PDAMs and LGs need to commit to this. For those reasons, IUWASH PLUS-SECO is currently developing a guideline in the form of PDAM Board of Directors Regulation on SPAM B2B mechanism. This regulation will guide standard procedures in the implementation of B2B cooperation for PDAMs and collaborators in SPAM implementation within the PDAM. However, since this is to be launched later in 2021, IUWASH PLUS contributions to PDAM B2B preparation are as yet non-existent. In some PDAMs, IUWASH PLUS assisted in the FS and tendering process of their B2B. In B2B and some interfaced national PPPs (Umbulan and Wosusukas Water Supply Projects), IUWASH PLUS assisted in the bulk water tariff calculation to be paid by the PDAMs. In other PDAMs, procurement teams are hesitant to handle the related large budget involved with a partnership, due to the legacies of difficult audits. IUWASH PLUS’ work with financial CB may have helped some achieve more confidence, though this was not mentioned specifically in interviews and is not readily apparent in the survey results. GOI partners agreed that this area of performance must be a focus of implementing the HR Road Map, given the potential for expanding financing, particularly through B2B mechanisms. IUWASH PLUS claimed that a total of five private sector partners engaged with PDAMs to invest in building new treatment plants in municipalities such as Medan, Bekasi, Bogor, and Gresik. Three business entities were interviewed related to their experience on IUWASH PLUS’ contribution to PDAM's B2B preparation; they are currently partnering with PDAM Tirtanadi Medan on treatment plants. Medan needs around 3,000–4,000 cubic liters of additional water supply in the coming years, which the government cannot achieve alone. Alternative financing is reportedly necessary. IUWASH PLUS, in assisting PDAMs, may well speed up the B2B implementation process, while the private sector can also help speed up PDAM services. In Medan, investors build water treatment plants through the BOT scheme where their investment returns will be from the bulk water bought by PDAM. By doing so, current PDAM funds can be allocated to build pipe networks. As a result, the distribution system will be ready when the plant is ready. According to one private sector respondent, without IUWASH PLUS, the B2B program might not be finished because the PDAM did not have reliable HR to assess the project's financial plan and model. The respondent also pointed to the fair value offered in the partnership contract and to IUWASH PLUS support in negotiations. A third private sector partner reported that IUWASH PLUS provided only TA and not real-life practice in the field, and that therefore the impact of IUWASH PLUS was not enough for them. IUWASH PLUS did not link donor programs and private sector/investors, according to IUWASH PLUS respondents, but PDAM interest could be kindled. To take best advantage of the studies carried out by donors, continue the programmining initiated by donors, and accelerate their own program implementation, LGs may link PDAMs to the private sector. For purposes of transparency, LGs should use an open ‘beauty contest’ model for selecting private sector partners. For the B2B project with PDAM Tirtanadi Medan, the FS phase took two years, delayed even with IUWASH PLUS. It progressed with the appointment of a new President Director of the PDAM. Clearly, having good leadership (and good decision-making) is crucial for successful partnerships. CONCLUSIONS 45 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV • IUWASH PLUS assisted in alternative financing processes from the time of FSs through the tendering process for the small number of PDAMs that were ready to secure alternative financing and contribute to the success of the partnerships. However, alternative financing per se was well outside the capabilities and interests of most PDAMs. • IUWASH PLUS worked on improving PDAM’s capacity in financing, but the PI scores on the finance sub-index do not show strong improvement across the PDAMs. They appear to remain far from seeking most types of alternative financing. • B2B options were not explored widely, but there are significant opportunities and GOI appetite to include this (alongside PPP, MFI, and other options) to reduce the WASH infrastructure financing gap. LGs and PDAMs seem well behind the interest and capacity levels necessary to carry these to fruition, in most geographic areas. • Greater use of alternative financing will depend on PDAM leadership making the decision to seek it. The GOI has put in place the requisite PPP regulations, as well as a PPP toolkit and guidelines. • IUWASH PLUS-SECO supported creating guidelines for these types of partnerships but as yet these have not been implemented and cannot be evaluated. RECOMMENDATIONS • LGs should encourage the financial health of their PDAMs and the achievement of FCR, in part so as to attract partnerships and alternative financing. USAID should support PDAMs to improve their financial performance and creditworthiness as a steppingstone to PPP/B2B readiness, and to seek and develop opportunities for alternative financing. Engaging with MFIs as a start may accelerate WASH access goals and increase household connections (and therefore revenue) while reducing the burden of operational expenses by collaborating on installment collection. • To reach the ambitious GOI goals for private sector investment in WASH infrastructure, make B2B and PPPs a top priority as well. Work to build LG and PDAM interest in these arrangements, and their capacities to execute them, which is at present mostly very low. • Support GOI efforts to improve policy and regulations toward the business enabling environment in the WASH sector, to facilitate PPP and B2B processes. • GOI and donors should enhance CB on alternative financing for PDAMs and LGs. CB, TA, and toolkits and guidelines should be combined for ease of understanding and multiple exposures to the subject. • The development of alternative financing guidelines should cover at least the detailed B2B processes, updated regulations and mechanisms of partnership schemes, risks and rewards of partnerships, case studies on some crucial issues that may occur during partnership processes, and the like. Also, a follow-up with the socialization and workshops to get more inputs from the stakeholders related to the developed guidelines before being published is also important in ensuring clarity and accommodating their needs. 6. CROSS-CUTTING THEMES: GENDER AND MEASUREMENT The evaluation team’s consideration of two cross-cutting themes is offered in this chapter: gender and measurement. We also discuss the overlap between the two: measurement of gender. The considerations offered here might be used to spark discussion on these topics within IUWASH PLUS-SECO, USAID, or other stakeholders. These observations are offered without conclusions or USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 46 recommendations, so as not to invite dispute about the evaluation’s purview, but instead to offer experiences and interpretations that may be of service in current and ongoing programming. Gender. The technical nature of the evaluation questions contained little space for examining gender. Still, the importance of gender to the sector and to IUWASH PLUS’ interventions merit at least this brief discussion of the evaluation team’s review of documents, inclusion in interviews, and analysis. IUWASH PLUS has undertaken robust research around gender in the sector, including the household and community levels, and fed that information into a Gender Strategy and specific training content, such as with community water groups. IUWASH PLUS also trains institutions on gender-responsive planning and budgeting for which some government offices are responsible— including a gender-sensitive budget—and IUWASH PLUS is supporting piloting in ten districts. IUWASH PLUS members at regional offices are part of an activity-wide gender working group, to discuss and test solutions where sought. Average PDAM scores in the Social Inclusion category on the Governance Index do not change much over time; there are exceptions, like Kab. Karawang, and Kota Malang, but on the whole, the variations from baseline to end line are small. When an index (or sub-index) performs in this way, it is often because the selected indicators are not sensitive enough to detect change, not operationalized well, or not aligned with what the activity works on. IUWASH PLUS also trained PDAMs in gender awareness. Respondents noted that trainees were often women and from the finance division, a suggestion that was borne from the results of the online mini-survey (see EQ3 for details). Women were rarely found in leadership and technical positions at PDAMs. This condition, broadly true of the sector in Indonesia and worldwide, makes it more of a challenge to ensure that PDAM planning and decision-making include attention to gendered impacts of the sector. Interestingly, IUWASH PLUS’ training numbers show broadly equitable inclusion of women and men in training sessions (and at the same time very uneven survey response rate—only about 14% of sampled women completed the survey, versus 25% of sampled men). Though it was not a focus of this evaluation, it may be useful to dive into the question with IUWASH PLUS of how they got such high participation from women in training, when the sector has built-in bias toward men. Perhaps this high participation can be capitalized on to add gender perspectives into planning throughout PDAMs and other institutions receiving training. At the same time, the low survey response rate among women raises another question the evaluation team cannot answer, but that bears looking into: If enough women were involved (per training records), why were they so much less likely to respond? In the same vein, only 22 of the evaluation interviews were with women, out of 70 total. Of these 22, two were from GOI, five from IUWASH PLUS, three from donors, two from MFIs, and five were from among our 11 sampled PDAMs (from which we interviewed 22 people). In addition, we interviewed one LG representative who is a woman, among 12 interviewees at that level. These figures indicate that those who speak for the PDAMs (and, presumably, make more PDAM decisions) are overwhelmingly male. While this is not the same as tracking gender-responsive PDAM decision-making, it usefully illustrates that the composition of power in the sector probably continues to lack an equity focus. An array of findings in the mini-survey appear to confirm differences between men and women IUWASH PLUS trainees, such as the following: 47 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 17. Training topic participation, by gender 55% 50% 54% 49% 33% 47% 26% 8% 23% 15% 25% 17% 27% 17% 26% 9% 0% 8% 22% 25% 22% Male Female Total Business Plan NRW Energy efficiency Performance index Watersource vulnerability Alternative financing Other Source: Evaluation team calculations; n=90 Men were far more likely than women to have trainings on technical topics, like NRW, EE, KKMA, and financing, while women’s training was centered more on the preparation of BPs and interpreting and using the PI. Totals are greater than 100 percent because respondents could select more than one type of training they attended. Since only 12 women completed the survey, it is not possible to extrapolate from their responses that all trained women had similar experiences, but it is notable that fewer women found the trainings “highly relevant to their work” than did the men (40 percent to 44 percent, respectively). None of either gender said it was not relevant to their work. At the same time, women respondents were more likely to express satisfaction with various aspects of training, like the trainer, the quality of materials and exercises, and the training methods. Measurement, including measurement of gender. The evaluation team noted the exemplary effort made by the IUWASH PLUS team to capture data on newly connected households and to determine through extant trends the percentage of new connections that they should attribute to the activity. This work is laudable and met an important USAID requirement: namely, that IUWASH PLUS shows the connection between the activity’s work and real change for households, including poor households. The ultimate decision to apply a 40 percent figure of the nominal increase is justifiable; so, too, would have been a decision to use more specific local percentages by site, while involving more work for the team. In either case, the evaluation team finds this a logical response to USAID’s need to report on an indicator like this. That said, the evaluation team’s monitoring, evaluation, and learning experience offers a different and perhaps useful perspective. The IUWASH PLUS activity provides primarily TA, CB, and at IUWASH PLUS-SECO sites, new PDAM equipment. The logic chain from these inputs to the outputs of household connections is complicated, and risks overstating (or even understating) IUWASH PLUS’ contribution. This poses a reputational risk for USAID, in that external readers may be inclined to question the link between inputs and results. This concern is set in the context of a very challenging set of indicators—in itself very demanding (with 30 IUWASH PLUS and 19 IUWASH PLUS-SECO indicators). Germane to this issue as well is the set of challenges IUWASH PLUS has faced in meeting its B40 targets, like targeting in mixed catchment areas, and COVID movement restrictions, among others. To their credit IUWASH PLUS USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 48 have been working to mitigate these issues with monitoring and evaluation tools that ultimately can be passed to the GOI to help them identify B40 households as well. This is a significant effort and makes up for some of the existing data gap on B40 households.18 Still, IUWASH PLUS was on track to deliver only half of their target for B40 households in its annual report in September, 2020. At the same time, history shows that female-headed households (FHHs) are likely to be among the poorest HHs in a given country. Though the activity’s monitoring, evaluation, and learning plan cites the value of providing estimates of the gender breakdown of the activity’s high-level results (DAI 2019, p. 14), and of disaggregating the key access indicators, the mode of estimation of the figures does not actually allow for gender breakdowns, because the indicator is not a tabulation of new connections to which IUWASH PLUS-SECO contributed. Instead, it is a calculation of a likely percentage of connections to which the activity’s work contributed: a fundamental distinction. Because of this, the focus of the indicator is on global outcomes rather than on the outcomes of actual households in gaining (or failing to gain) access. This distinction can affect aspects of programming beginning with targeting (e.g., selecting stronger PDAMs or those with more household connections which are more likely to help them meet their overarching goals) and continuing throughout the life of the project. It almost certainly turns attention from B40 households and from FHHs, given the high target figures—a concern the evaluation team feels is justified, since IUWASH PLUS has met or will meet its overarching targets but, as noted above, will meet only about half of targets for B40 access.19 The household connections indicator, though well-intentioned and responsibly pursued, does not provide the implementer or USAID with data that helps it focus better and improve outcomes. Instead, it provides—superficially—proof of achievement but without the substance that would make the data useful for informing programming. The Activity Monitoring, Evaluation and Learning Plan (AMELP) promise of gender breakdowns appears unmet, and if this is true, it’s understandable given the surfeit of indicators to report on. However, this is the kind of data that might be useful, particularly around gendered aspects of the sector and gendered effects of programming. This is especially true since measurement of gender in the project is limited to disaggregation of training participants by gender. The United State Government (USG) Standard Indicator “Number of people trained to advance outcomes consistent with gender equality or female empowerment through their roles in public or private sector institutions or organizations” is counted, along with a count of participants in particular trainings on gender. There is no outcome level indicator on gender, despite the salience of gender for the sector. The evaluation team noted some additional gaps in measurement and some potential areas that would have benefited from measurement, with the goal of having evidence to inform IUWASH PLUS’ learning and adaptation. 18 The GOI also has difficulties with data about B40 households, and their Central Bureau of Statistics does not offer LG￾level disaggregation (though it is unknown if the paid portal at Susenas might offer this dataset.) 19 The challenges in sanitation also include low priority from GOI, low capacity of existing treatment plants, and low interest from among B40 households, for whom the cost is prohibitive. 49 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV • Lack of before-and-after measures taken in the spring vulnerability assessments, without which any evaluation of the value of the undertaking can only be subjective. • Testing variation patterns that emerged as a result of the demand-driven intervention model. With 35 PDAMs, there may be useful lessons on what combinations and sequences of TA and CB are most helpful; even without a formal study of these variations, the evaluation team has concluded that adding equipment to the intervention, as IUWASH PLUS-SECO did, resulted in more durable gains. A well-designed study on this topic, using developmental evaluation or configurational comparative approaches, might have served both IUWASH PLUS and the wider evidence base. • Differences between smaller and larger PDAMs (and/or possibly those that are wealthier and more central, versus those that are poorer and further removed from the capital) is likely an important category for understanding what works and what doesn’t. Documenting those differences would be of benefit to the sector now and in the future. Finally, measurement of training has always been challenging, and it is no different for IUWASH PLUS. It is relatively easy to count trainees and much harder to quantify the gain from training. Reports to USAID’s TraiNet are not deeply demanding, and for that reason, IUWASH PLUS did not develop a database of trainees across its training types—nor was there follow-up with trainees (for which a database would be necessary). Even turnover went unmonitored, so we cannot gauge the degree to which training remained in situ to promote improved institutional performance. The IUWASH PLUS-SECO PIAP, discussed with EQ3 above, was a useful adaptation that should have helped this issue somewhat, but only on a group level. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 50 ANNEX A. EVALUATION MATRIX: GETTING TO ANSWERS Evaluation Questions Data Collection Methods Sampling or Selection Analytical Methods Source(s) Method 1. What is driving PDAM performance improvements and what is the role of the BPPSPAM and IUWASH PLUS indices in performance improvement? • NUWAS /BPPSPAM frameworks and ratings • IUWASH PLUS index and ratings • Relevant PDAM data: operations, HR, finance • Data review • Full data sets as available • Review index formulation; review index results and trends as available; comparison with water utility transactional data, as available • Review index, formulation and trends as available. • Pattern and divergence, and content analyses • Context-mechanism analysis • World Bank NUWAS project • PDAM and GOI • Former BPPSPAM staff • Community of practice (PERPAMSI, Pokja AMPL) • Interviews • Purposive with relevant stakeholders • Identified through IP lists • What is needed to replicate the use of USAID IUWASH PLUS’ performance index to improve performance of other PDAMs? • To what extent has USAID IUWASH PLUS assistance prepared PDAMs to manage both water supply and wastewater services as mandated by RPJMN 2020-2024? • Tim Teknis • MPWH-DitPSPAM and MOHA (Keuda) • MPWH-DitPSPAM and MOHA (Keuda) RDPA/Bappenas • PDAMs – with a possible comparison site TBD • Review secondary data, to be confirmed with interviews • Purposive among relevant stakeholders 2. Are IUWASH PLUS activities resulting in sufficient reductions of NRW and EE improvements to put PDAMs on a pathway to business viability? • PDAM data on NRW and EE • PDAM actors, capacity building participants • Review secondary data • Interviews • Purposive among relevant stakeholders • Transactional analysis – before and after • Pattern and divergence, and content analyses 51 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Evaluation Questions Data Collection Methods Sampling or Selection Analytical Methods Source(s) Method • In what ways have USAID IUWASH PLUS interventions contributed to the improved capacity of PDAMs to reduce non-revenue water and increase energy efficiency? • In what ways have USAID IUWASH PLUS interventions influenced local government investment for water utilities to reduce NRW and increase energy efficiency? • To what extent have IUWASH PLUS activities established the foundations on which PDAMs can continue to make improvements in NRW and EE beyond IUWASH PLUS? (recent addition) • Did NRW and EE remain prioritized areas of investments for PDAMs, even in times of fiscal constraints due to COVID-19? • Have the measurements and baseline studies supported the commitment and ownership of the PDAMs to work on NRW and EE? • What is the likelihood that measurement equipment purchased by the project will be used to undertake NRW and EE improvements independent from the project in the future (in the supported PDAMs, or in PDAMs outside the project)? • To what extent was the IUWASH PLUS-SECO component successfully integrated into the overall IUWASH PLUS program and made efficient use of existing structures, networks, and processes? • IUWASH PLUS and other donor data and reports, KIAT ADB • LGs, IUWASH Plus • Business development services firms • PDAM actors, capacity building participants • Review secondary data, to be confirmed with interview • Sample LGs for data review and interviews • Strong performers and “growth” PDAMs • Data review 3. In what ways has IUWASH PLUS’ training and human resources development roadmap contributed to the improved PDAM performance? What accounts for successes? What accounts for failures? • Tim Teknis • MPWH-DitPSPAM • MOHA • PERPAMSI • Capacity building participants • Interviews • If feasible, mini-survey of capacity building participants • Purposive sample in line with selection for EQ2 • Pattern and divergence, and content analyses USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 52 Evaluation Questions Data Collection Methods Sampling or Selection Analytical Methods Source(s) Method • Are there plans to continue the completion of the PDAM human resources development roadmap? • In what ways has IUWASH PLUS helped with identification of PDAM training centers? • To what extent does the roadmap present a feasible and systematic capacity strengthening model for utilities? Will IUWASH PLUS-SECO training materials, trainers and field examples support this? • USAID, MOHA • IUWASH PLUS • PDAM training center staff • Interviews with related stakeholders • Purposive among relevant stakeholders 4. What evidence is there that spring vulnerability assessments have resulted in improved sustainable management of raw water sources and use by drinking water providers? • Spring vulnerability assessment reports • MPWH • MOEF • IUWASH PLUS • PDAMs (Salatiga) • Document review • Interviews • Data review on spring vulnerability, as available • PDAMs with RPAM activity • Pattern and divergence, and content analyses • To what extent are there plans to replicate USAID IUWASH PLUS spring vulnerability assessments in other spring catchment areas? • To what extent has the addition of infiltration ponds increased drinking water source yield or reduced flooding, if at all? • What project-supported WRM models, if any, are the most promising for the sustainability and resilience of drinking water sources? • USAID • MOPWH • MOME • Interviews with relevant stakeholders • Purposive from among relevant stakeholders and decision￾makers 5. In what ways have IUWASH PLUS’ activities contributed to PDAMs’ ability to secure alternative financing? • WASH commercial enterprises • Interview, data review as available • Purposive among relevant stakeholders • Data review 53 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Evaluation Questions Data Collection Methods Sampling or Selection Analytical Methods Source(s) Method • In what ways have IUWASH PLUS interventions contributed to the development of the alternative financing guidelines? • In what ways have IUWASH PLUS interventions helped PDAM with business-to-business preparation? • IUWASH PLUS, MOF, MOHA, PT SMI • PDAMs, PERPAMSI, MOHA LGs, IUWASH Plus • Document review of B2B reporting • Interview with related stakeholders /FGD • Pattern and divergence, and content analyses USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 52 ANNEX B. DATA COLLECTION TOOLS The interview guides in the evaluation involved significant improvisation to explore EQ themes in depth. The types of respondents are as follows: 1. IUWASH PLUS Senior Management 2. IUWASH PLUS technical team members, component leads and staff 3. IUWASH PLUS PPP Lead 4. USAID and SECO technical activity oversight staff 5. Planning and development manager/PDAM 6. COO of Microfinance Institution/ Lender 7. CSR staff from partner companies 1. USAID IUWASH PLUS KII Report - IUWASH Senior Management (COP, DCOP) Leading Interviewer Note Taker Date and Time Place of KII Unique ID Respondent(s) Organisation and Title Email and Phone Number EQ 1: PDAM Performance Index 1.1 Who/what was/is involved in bringing about or influencing/supporting the use of the BPPSPAM and IUWASH PLUS indexes? 1.2 What was significant about their use in PDAM performance improvement? 1.3 What surprised you about what happened? (enabling or inhibiting factors?) 53 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 1. 4 Does achievement/change in performance vary by different factors? Which ones, and why? 1.5 In what ways do you think IUWASH PLUS support might have improved PDAM's ability to manage both water supply and wastewater services as mandated by RPJMN 2020-2024? Why? EQ. 2: Reduction of NRW and EE Improvements in PDAM 2.1 Tell me about the key approaches/steps/strategies made to promote and support the reduction of NRW and EE. 2.2 How satisfied have you been with the reduction of NRW and EE as a result of the program? Were there contextual factors affecting performance? 2.3 What has been done to influence local government investment in water utilities, to reduce NRW and improve EE? How might this support have affected local government? What are the challenges faced with these investments? 2.4 In what ways might the improved performance of PDAMs have resulted in reduction of NRW and EE? What happened to the priority level of these efforts during COVID-related fiscal constraints? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 54 2.5 Have the measurements and baseline studies supported PDAM commitment and ownership to work on NRW and EE? How? 2.6 To what extent will the changes that have occurred be sustained, and how? Will measurement equipment purchased by the project be used to undertake further NRW and EE improvements after IUWASH PLUS has ended? This question is for participating PDAMs after the intervention, but also for other PDAMs. 2.7 Can you talk a bit about the SECO partnership? What were the strengths and weaknesses of their involvement? Was their participation well-integrated in IUWASH PLUS structures, networks, and processes? Why or why not? EQ. 3: Human Resources Development Roadmap to improve PDAM Performance 3.1 What was significant about this training and human resources development roadmap for PDAM performance improvement? 3.2 To what extent does the roadmap present a feasible and systematic capacity strengthening model for utilities? Will IUWASH PLUS-SECO training materials, trainers and field examples support this? If so, how? 55 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 3.3 What changed as a result, if at all? Who? When? Where? 3.4 What has IUWASH PLUS done in identification of PDAM training centers? What was significant about this? 3.5 How confident are you that the human resources development roadmap and effort will continue post-IUWASH PLUS-SECO? Why? EQ.4: Spring Vulnerability Assessments 4.1 Does performance in spring vulnerability issue vary by different factors? Which ones, and why? 4.2 To what extent were/are the changes (especially increase of raw water) sustainable? Why or why not? 4.3 What project-supported WRM models are the most promising for the sustainability and resilience of drinking water sources? EQ. 5: PDAMs’ Alternative Financing 5.1 How were supported PDAMs approached on the issue of alternative financing? What was significant about the financing guidelines and support for business-to-business preparation? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 56 5.2 In the future, is PDAM alternative financing likely to decrease/stay the same/increase? Why? 5.3 How does IUWASH PLUS’s approach differ from that of other funders in supporting performance improvement of PDAM? 5.4 Without IUWASH PLUS support, how would things have been different? (probe about roles of other actors) 5.5 What could be done differently? Main information that should be collective from USAID IUWASH PLUS: 1. The list of requested documents and data 2. Suggested potential KI: 57 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 2. USAID IUWASH PLUS KII Report - IUWASH PLUS Technical Team Members, Component Leads and Staff (EQ1) Leading Interviewer Note Taker Date and Time Place of KII Unique ID Respondent(s) Organisation and Title Email and Phone Number EQ 1: PDAM Performance Index 1.1 Tell me about the background of your organization support to PDAM performance improvement through the IUWASH PLUS. 1.2 How was it related to other WASH sector agenda/reforms, if at all? 1.3 Which (other) programs/agencies/organizations/partners do you engage with and how? 1.4 How does IUWASH PLUS’s approach differ from that of other funders in supporting performance improvement of PDAM? 1.5 How do/would you define success of this program? 1.6 Without this program, how would things have been different? What would have happened anyway? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 58 1.7 Who/what was/is involved in bringing about or influencing/supporting the use of this BPPSPAM and IUWASH PLUS index? 1.8 What was significant about the use of this index in performance improvement? In your opinion, what difference does it make to performance improvement, if any? 1.9 What surprised you about what happened? (enabling or inhibiting factors?) 1.10 Does performance vary by different factors? (location, project, intervention?) 1.11 Does the project refer to the RPJMN 2020-2024 in water supply and wastewater services in the support to PDAM? How? EQ. 2: Reduction of NRW and EE Improvements in PDAM 2.1 Tell me about the key approaches/steps/strategies made to promote and support the reduction of NRW and EE. 2.2 How satisfied were you with the reduction of NRW and EE as a result of the program? Were there contextual factors affecting performance? 2.3 In what ways might improve PDAM performance have resulted in reduction of NRW and EE? What happened to the priority level of these efforts during COVID-related fiscal constraints? 59 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 2.4 Have the measurements and baseline studies supported PDAM commitment and ownership to work on NRW and EE? 2.5 To what extent will the changes that have occurred be sustained, and how? Will measurement equipment purchased by the project be used to undertake further NRW and EE improvements after IUWASH PLUS has ended? This question is for participating PDAMs after the intervention, but also for other PDAMs. 2.6 Can you tell us about the partnership with SECO – what worked, what didn’t, and would you want to repeat the partnership in the future? Why or why not? EQ. 3: Human Resources Development Roadmap to improve PDAM Performance 3.1 Please tell us about the role of the capacity building team. What have been your proudest accomplishments? (Probe for what accounts for successes) 3.2 What have been your most important obstacles and challenges? (Probe for what accounts for failures) USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 60 3.3 Please tell us about the training and human resources development roadmap for PDAM performance improvement. We understand it is newly launched; what are your expectations for the roadmap? (Probe particularly on government support for the roadmap) 3.4 Prior to the launch of the roadmap , what were IUWASH PLUS’ capacity building activities? How were the SECO capacity building activities different? 3.5 What changed as a result of IUWASH PLUS-SECO capacity building, if at all? Who? When? Where? (Probe for “big picture” and some details) 3.6 To what extent can the utilities use the roadmap’s capacity strengthening model? (Probe on what their obstacles will be, probe on resources) 3.7 What will happen with the IUWASH PLUS and SECO training materials, trainers and field examples? How do you know this? (Probe on the use of the NRW/EE examples in particular) 3.8 What has been the particular effect of administrative/business topics (as opposed to the technical topics)? Do you have any examples of these trainings resulting in better management, administration, financing, business to business, business planning, etc.? 61 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 3.9 What has IUWASH PLUS done to identify of PDAM training centres? To what degree will PDAMs be able to access these after the close of the IUWASH PLUS activity? 3.10 In what ways did you coordinate with other teams at IUWASH PLUS-SECO? (Probe on choosing training topics, content for training, other) Was the SECO training integrated with the IUWASH PLUS training? How and how not? 3.11 What efforts have you made to ensure the sustainability of the capacity you have worked to build? What have been the successes and obstacles in sustaining the learning? 3.12 What could be done differently/better? EQ.4: Spring Vulnerability Assessments 4.1 Does performance in spring vulnerability issue vary by different factors? What? Why? 4.2 To what extent were/are the changes (especially increase of raw water) sustainable? Why or why not? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 62 EQ. 5: PDAMs’ Alternative Financing 5.1 How were supported PDAMs approached on the issue of alternative financing? What was significant about the financing guidelines and support for business to business preparation? 5.2 Which sources of finance do PDAMs engage with and how? 5.3 In the future, will PDAM alternative financing likely decrease/stay the same/increase? Why? 5.4 What other information do you think would be helpful for us to consider? (additional comments, names of other people to interview, documents, datasets) Main information that should be collective from USAID IUWASH PLUS: 1. The list of requested documents: 2. Suggested potential KI: 63 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 3. USAID IUWASH PLUS KII Report - IUWASH PLUS PPP Lead Leading Interviewer Note Taker Date and Time Place of KII Unique ID Respondent(s) Organisation and Title Email and Phone Number 1. In what ways have USAID IUWASH PLUS interventions influenced local government investment for water utilities to reduce NRW and increase EE? What can you say about the SECO contribution to this effort, if anything? 2. In what ways have USAID IUWASH PLUS interventions contributed to the development of the alternative financing guidelines? What programs are planned to apply the alternative financing scheme(s)? 3. In what ways have USAID IUWASH PLUS interventions helped PDAMs with business-to￾business preparation? Has it included technical assistance to develop business plans? If yes, have PDAMs been able to implement the proposed business plan and resolve financial solvency issues? 4. Have you also helped LGs/ PDAMs/ local communities get loans to finance their programs? If yes, what criteria do you recommend when choosing financial institution and other partners? Also, how do you convince the financial institutions to offer low interest rate loans, especially to the B40 communities? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 64 5. Have you made some guidelines or toolkits for financial institutions to develop microfinance products for water, sanitation, and hygiene (WASH) investments? If yes, may we have a copy? 6. Please tell us about the successfully implemented PPP projects and ongoing PPP projects initiated by IUWASH PLUS (project list and current stage/status). 7. How about any PPPs that were not as successful? Any thoughts on why, and how to improve? 8. So far, what challenges have you encountered in implementing PPP scheme(s) for WASH sector improvement? 9. Since the WASH sector is not categorized as commercial, how do you feel it is best to attract private sector involvement (either through PPP, CSR, or other forms of involvement)? 10. How about private sector investment return in PPP projects that were initiated by IUWASH PLUS? Is it using the Availability Payment mechanism? If so, the successful implementation will depend on the government budget which is very limited especially considering the current condition with many refocusing budgets due to COVID-19. How do you weigh or solve this issue? 65 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 11. Is there a plan to perform Spring Vulnerability Assessment in which several PDAMs participated? If so, will funds come from the central/ regional government, PPP, or other funding sources? Main information that should be collective from USAID IUWASH PLUS: 1. The list of requested documents: 2. Suggested potential KI: USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 66 4. USAID IUWASH PLUS KII Report - IUWASH PLUS USAID/SECO Activity Oversight Technical Staff Leading Interviewer Note Taker Date and Time Place of KII Unique ID Respondent(s) Organisation and Title Email and Phone Number 1. How was the IUWASH PLUS’s theory of change related to the USAID development objective? How was it related to wider WASH sector agenda/reforms, if at all? 2. Which (other) programs/agencies/organizations/partners do you engage with and how, and which ones also engage with the WASH sector? How do you work to ensure a lack of duplication of efforts and taking advantage of any synergies? 3. How does IUWASH PLUS’s approach differ from that of other funders in supporting reform of the WASH sector? 4. What has been your experience of the partnership (USAID/SECO)? What are the areas for improvement? 5. How do/would you define success in this program? What areas do you see for improvement? 67 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 6. Without this program, how would things have been different? What would have happened anyway? 7. In what ways do you think a sustainability plan has been put in place (and was that with or without donor support)? 8. What were the factors (internal and external) affecting program implementation? 9. How satisfied have you been with the integration of SECO into IUWASH PLUS? Do you think they made efficient use of existing structures, networks and processes? Why or why not? What might be done differently? 10. How satisfied have you been with PDAM performance improvement to date? What are the areas for improvement? 11. How satisfied have you been you with the reduction of NRW and EE so far? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 68 12. What do you think about the training and human resources development roadmap for PDAM performance improvement? What was significant about it? 13. How satisfied were you with the spring vulnerability initiative so far? 14. How do you think about the alternative financing initiative so far? 15. What surprised you about what happened? 16. What can be done differently/better? Main information that should be collective from USAID IUWASH PLUS: 1. The list of requested documents: 2. Suggested potential KI: 69 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 5. USAID IUWASH PLUS KII Report - PDAM Leadership, Planning and Development Manager Leading Interviewer Note Taker Date and Time Place of KII Unique ID Respondent(s) Organisation and Title Email and Phone Number EQ 1: PDAM Performance Index 1.1 Can you explain the role and involvement of this Institution/PDAM/LG/Directorate with the IUWASH PLUS project? 1.2 Do you think that a performance index is still needed to improve PDAM performance? If yes, what kind of performance index? If no, why not (and skip question 3)? 1.3 Is there still a need for roles such as BPPSPAM and IUWASH PLUS in assessing PDAM performance? 1.4 With the dismissal of BPPSPAM, will it affect this Institution/PDAM/Local Government/Directorate? Do you think that the performance index made by IUWASH PLUS is sufficient? Why or why not? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 70 1.5 In your opinion, are PDAMs ready to manage drinking water supply and wastewater services as mandated in the 2020-2024 RPJMN? 1.6 Please tell us about the assistance from IUWASH PLUS to prepare PDAMs to manage both drinking water and wastewater services. Is it sufficient? What else is needed? EQ. 2: Reduction of NRW and EE Improvements in PDAM 2.1 Please tell us your assessment of Non-Revenue Water and Energy Efficiency in PDAMs in Indonesia. 2.2 What might be affected by the conditions of NRW and EE if no repairs are made? 2.3 Who should be responsible for the condition of NRW and EE in PDAMs? 2.4 In your opinion, is the current assistance form IUWASH PLUS adequate to reduce NRW and increase EE? 71 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 2.5 What effect, if any, does IUWASH PLUS have on local governments providing capital participation to reduce NRW and increase EE? 2.6 Were you aware of any differences between IUWASH PLUS and SECO funding for these NRW and EE efforts? EQ. 3: Human Resources Development Roadmap to improve PDAM Performance 3.1 What can you tell about the condition of human resources in PDAM? Is the capability of human resources in PDAM sufficient, both hard skills and soft skills? What capacities might need strengthening? 3.2 Do you have a roadmap to improve the capability of human resources in the PDAM that you lead? Can you tell me about that roadmap, please? (Probe for quality, utility, who made it, how realistic…) 3.3 Can you give your opinion on what could be a measure of success or failure of human resources? 3.4 In your case, does IUWASH PLUS help you to identify what type of training an employee should receive, and where to do that training? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 72 3.5 What is your opinion of the human resource development roadmap made by IUWASH PLUS for your knowledge of human resources, and for planning to add/remove employees in PDAM? EQ.4: Spring Vulnerability Assessments 4.1 Currently, the issue of dryness of raw water sources occurs in almost all regions. Has there been any effort to start preventing the vulnerability of raw water sources, especially those from springs? If so, was any such effort connected to IUWASH PLUS? 4.2 One of the programs from IUWASH PLUS is Spring Vulnerability Assessment which is participated by several PDAMs. Is there a plan to replicate this program? If there is a plan, will the funds come from the regional or central government or other funding sources? 4.3 For PDAMs participating in the Spring Vulnerability Assessment program, what is your assessment of the program proposed by IUWASH PLUS in the PDAM raw water source that you lead? Will you replicate this program with other raw water sources? EQ. 5: PDAMs’ Alternative Financing 5.1 Another alternative financing to develop PDAM business is the plan of the Government of Indonesia. IUWASH PLUS contributes to developing alternative financing guidelines, what do you think about this guide? What has been the effect of the guide for you, if any? 73 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 5.2 Alternative financing could be through micro credit, Business to Business, instalment financing, and others. What has been your experience with IUWASH PLUS in planning and preparing alternative financing programs? Please tell us your experience with this project. General/Additional Questions for Planning Manager 1. How big is the impact of NRW and EE efficiencies on your revenue? To what degree do those improvements support your company on a pathway to business viability, if at all? 2. How is the progress of your business plan? Do you get any assistance in developing your business plan from USAID, in what ways? Then, have you got more clear goals than before? 3. Have you been able to implement the proposed business plan and solved your financial solvency issues (if any)? 4. What do you think about the Sustainability Checklist from the IUWASH PLUS? And how do you rate your company’sinancial, operational, and management sustainability? 5. What would you expect from USAID support assistance to further improve your company performance? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 74 Main information that should be collective from USAID IUWASH PLUS: 1. The list of requested documents: 2. Suggested potential KI: 75 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 6. USAID IUWASH PLUS KII Report - COO of Microfinance Institution/ Lender Leading Interviewer Note Taker Date and Time Place of KII Unique ID Respondent(s) Organisation and Title Email and Phone Number 1. Please tell us about how you and IUWASH PLUS worked together. (Probe for degree to which they were supported, for example in developing the loan facility system/ guideline/ toolkits.) 2. What are the types of lending products that have been developed? 3. Have you developed the unsecured debt scheme (without collateral), especially for B40 communities? What could make this more effective and widespread, in your opinion? 4. What do you think about the impact of your loan towards the increasing service coverage for water and sanitation access? 5. How much interest fee do you offer? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 76 6. What is the percentage of loan performance from B40 communities? Main information that should be collective from USAID IUWASH PLUS: 1. The list of requested documents: 2. Suggested potential KI: 77 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 7. USAID IUWASH PLUS KII Report – CSR Manager Leading Interviewer Note Taker Date and Time Place of KII Unique ID Respondent(s) Organisation and Title Email and Phone Number 1. In what sector or program do you usually spend your CSR funds? What makes you interested in that program? 2. How much do you usually allocate the funds for CSR programs (every year)? 3. How was partnership with USAID developed and in what ways partnership will be improved in the future? 4. How many percentages of CSR funds have been shifted to the WASH sector development (compared to before the partnerships with USAID)? What programs are they? 5. Do you have any ongoing plan to use the CSR funds in the WASH sector (probably for next year's plan)? USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 78 6. Do you happen to be interested in the CSR for reducing the NRW and increasing the EE? If yes, how would you rate the current performance/ intervention of IUWASH PLUS in reducing NRW and increasing EE? 7. In what ways USAID support assistance help accelerate the GOI program in water and sanitation sector? And how do you rate your company's contribution to this acceleration? 8. Do you happen to have any plans for initiating investment in the WASH sector (probably through the PPP scheme)? And why is that? 9. What would you expect to improve/increase from USAID support assistance in the future? Main information that should be collective from USAID IUWASH PLUS: 1. The list of requested documents: 2. Suggested potential KI: 79 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV ANNEX C. SITE AND SAMPLE SELECTION Site Selection The evaluation team selected ten sites, later adding an eleventh to ensure coverage of EQ5, from among IUWASH PLUS’ geographic areas of work (as the figure below shows and in various Excel spreadsheets provided by IUWASH PLUS). Jawa Barat, DKI Jakarta, and Tangerang district 1. DKI Jakarta 2. Bekasi city 3. Depok city 4. Bogor city 5. Karawang district 6. Bogor district 7. Tangerang district 1. Magelang city 2. Salatiga city 3. Surakarta city 4. Sukoharjo district 5. Magelang district 6. Sragen district 7. Wonosobo district Central Java East Java 1. Surabaya city 2. Probolinggo city 3. Malang city 4. Sidoarjo district 5. Gresik district 6. Probolinggo district 7. Lumajang district 8. Malang district 5. Maluku Tengah district 6. Ternate city 7. Jayapura city 8. Jayapura district Blue color: new city (had been added in 2nd year of the project) Light brown color: terintegrasi dengan project USAID-SECO Partnership Program The evaluation selected from this list using the following criteria: • IUWASH PLUS and IUWASH PLUS-SECO sites • Sites with activities associated with our EQs • We remove the criterion of selecting two PDAMs in each of four provinces, in favor of flexible geographic spread (since we are not likely to travel due to COVID restrictions). • Eastern region to see sites where socio-economic characteristics are different to Western • One site each in North Sumatra and East Java • A mix of “best practice” and “growth” sites, per average performance index scores • Understand other donors’ work in selected PDAMs, like the NUWSP The final selection was: USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 80 PDAM PROVINCE RELATED ACTIVITIES FUNDING/ DONORS Kota Surakarta Central Java NRW, EE, Biz Plan USAID/SECO Kab. Magelang Central Java NRW, EE, Biz Plan, KKMA, micro-credit USAID/SECO, NUWSP PBG Kota Depok West Java NRW, EE, micro-credit USAID/SECO, NUWSP PBG Kab. Karawang West Java NRW, EE, Biz Plan USAID/SECO, NUWSP PBG Kota/Kab. Jayapura Papua NRW USAID Tirtanadi (Kota Medan) N. Sumatra KKMA, B2B, PPP, micro-credit USAID Kota Pematang Siantar N. Sumatra NRW, KKMA, Biz plan USAID, NUWSP PBG Kab. Probolinggo East Java KKMA USAID Kab. Sidoarjo East Java NRW, EE USAID Kab. Gresik East Java EE, PPP, micro-credit USAID, NUWSP matching grant Kab. Barru S. Sulawesi Biz plan USAID Kabupaten Gresik was added as a site mid-course during the evaluation, to ensure coverage of EQ5, because of its particular experience with alternative financing. Sample Selection The team carried out an online survey with a sample of training participants randomly selected from lists compiled by the IUWASH PLUS team. The survey sample was selected based to capture capacity building experiences from PDAM staff who work in IUWASH PLUS sites and IUWASH PLUS-SECO sites. The survey focuses on staff perception about the training and technical assistance they received during the IUWASH PLUS project implementation and how these might have affected their individual and organizational performance. The questionnaire was developed in KoBo Toolbox and a link delivered through WhatsApp and email. The evaluation team asked IUWASH PLUS for their lists of trainees. This information was available in hard copy, PDF versions of sign-in sheets, and trainees number in the thousands. That means there was no searchable or traceable database of trainees. In order to select from the trainees at random, it was necessary to ask IUWASH PLUS to reformat the information in editable software. We delimited the sampling frame to trainees with at least one training funded by IUWASH PLUS￾SECO and active PDAM staff. IUWASH PLUS eventually informed us there were 873 that met our criteria from IUWASH PLUS-SECO sites and 1,569 from IUWASH PLUS sites. We sampled a little over 10% from these lists, for a total target sample of 250 respondents, including resampling when respondents could not be contacted or did not respond after three contact attempts (including both evaluation team and IUWASH PLUS support to encourage participation). The moderate response rates of some 35% to 43% are shown in the following table by category. 81 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Table 4. Moderate response rates for the PDAM trainee online survey CATEGORY SAMPLE SIZE RESPONSE RATE (PERCENTAGE) IUWASH PLUS-SECO 89 38 (43%) Female 7 Male 29 Unknown 2 IUWASH PLUS 161 56 (35%) Female 5 Male 49 Unknown 2 Source: Evaluation team calculations Though the sample was random, and response rates are moderate, the degree to which the sample can be said to be representative is unknown. This is because the non-response rate is high enough to introduce significant uncertainty. Trainee information from larger and smaller, or more central and more remote PDAMs, indicate that in smaller and more remote sites, people were less likely to report having an email address. We selected these people all the same, and contacted them via text message to ask that they complete the survey anyway – through a mobile-enabled version. We conducted additional analysis based on the size of the PDAM from which respondents reported. The size ranges we used come from an “unpublished” method from BPPSPAM, in which PDAMs with 20,000 or fewer connections are called “small”, those with over 100,000 connections are called “large”, and those in between (20,000 to 100,000 connections) are neither. This large category in between the extremes includes 11 of the PDAMs in our sample, and acts as a buffer to amplify differences between the small and large PDAMs which we detected as well in interviews. 85 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV ANNEX E. LIST OF DOCUMENTS AND DATA SOURCES IUWASH PLUS CVAs 1. Kajian Kerentanan Mata Air Ake Gaale, Kota Ternate – Provinsi Maluku Utara 2. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Binangun, Kota Batu – Provinsi Jawa Timur 3. Kajian Kerentanan Mata Air Citrosono, Kabupaten Magelang – Provinsi Jawa Tengah 4. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Clumprit, Kota Malang – Provinsi Jawa Timur 5. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Eremerasa, Kabupaten Bantaeng – Provinsi Sulawesi Selatan 6. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Kalitaman Kota Salatiga – Provinsi Jawa Tengah 7. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Lotonglotong, Kabupaten Bulukumba – Provinsi Sulawesi Selatan 8. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Ciburial, Kabupaten Bogor – Provinsi Jawa Barat 9. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Nagahuta, Kota Pematangsiantar – Provinsi Sumatera Utara 10. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Sibolangit – Provinsi Sumatera Utara 11. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Sumbertopo Kabupaten Lumajang – Provinsi Jawa Timur 12. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Tancak Kabupaten Probolinggo – Provinsi Jawa Timur 13. Kajian Kerentanan Mata Air dan Rencana Aksi Mata Air Tangkil Kota Bogor – Provinsi Jawa Barat 14. Kajian Kerentanan Mata Air Tuk Pecah Kota Magelang – Provinsi Jawa Tengah IUWASH PLUS WORK PLAN 1. IUWASH PLUS Work Plan PY1 2. IUWASH PLUS Work Plan PY2 3. IUWASH PLUS Work Plan PY3 4. IUWASH PLUS Work Plan PY4 IUWASH PLUS ANNUAL REPORT 1. Annual Progress Report 1 2. Annual Progress Report 2 3. Annual Progress Report 3 4. Annual Progress Report 4 IUWASH PLUS QUARTERLY PROGRESS REPORT 1. Quarterly Progress Report 1 2. Quarterly Progress Report 2 3. Quarterly Progress Report 3 4. Quarterly Progress Report 4 USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 86 5. Quarterly Progress Report 5 6. Quarterly Progress Report 6 7. Quarterly Progress Report 7 8. Quarterly Progress Report 8 9. Quarterly Progress Report 9 10. Quarterly Progress Report 10 11. Quarterly Progress Report 11 12. Quarterly Progress Report 12 13. Quarterly Progress Report 13 14. Quarterly Progress Report 14 15. Quarterly Progress Report 15 16. Quarterly Progress Report 16 17. Quarterly Progress Report 17 18. Quarterly Progress Report 18 19. Quarterly Progress Report 19 IUWASH PLUS STUDI KELAYAKAN NRW DOCUMENTS 1. Laporan Study Kelayakan Program Peningkatan Kapasitas/Pelatihan untuk Penurunan NRW di PDAM Tirta Waesai Kabupaten Barru 2. Laporan Study Kelayakan Program Peningkatan Kapasitas/Pelatihan untuk Penurunan NRW di PDAM Giri Tirta Gresik 3. Laporan Study Kelayakan Program Peningkatan Kapasitas/Pelatihan untuk Penurunan NRW di PDAM Kota Probolinggo 4. Laporan Study Kelayakan Program Peningkatan Kapasitas/Pelatihan untuk Penurunan NRW di PDAM Tirta Bulian Kota Tebing Tinggi 5. Laporan Studi Kelayakan Program NRW PDAM Kota Depok, April 2020 6. Laporan Studi Kelayakan Program NRW PDAM Tirta Tarum Kabupaten Karawang, Agustus 2020 7. Laporan Studi Kelayakan Program NRW PDAM Tirta Gemilang Kabupaten Magelang, April 2020 8. Laporan Studi Kelayakan Program NRW PERUMDA Air Minum Kota Magelang, April 2020 9. Laporan Studi Kelayakan Program NRW PERUMDA Air Minum Kabupaten Sukoharjo, April 2020 IUWASH PLUS AUDIT EFISIENSI ENERGI DOCUMENTS 1. Laporan Audit Efisiensi Energi PDAM Tirta Kahuripan Kabupaten Bogor, Juni 2020 2. Laporan Audit Efisiensi Energi PERUMDA Air Minum Tirta Makmur Kabupaten Sukoharjo, Juli 2020 3. Laporan Audit Efisiensi Energi PERUMDA Air Minum Toya Wening Kota Surakarta, Juli 2020 4. Summary of Efficiency Energy Audit, IUWASH PLUS-SECO, June 2021 (in Bahasa Indonesia) IUWASH PLUS TECHNICAL DOCUMENTS 1. Laporan Feasibility Study Efisiensi Energi PDAM Giri Tirta Kabupaten Gresik 2. Laporan Feasibility Study Efisiensi Energi PDAM Kota Probolinggo, Oktober 2020 3. Capacity Building Roadmap for Human Resources of Drinking Water Companies (PDAMs)/ Water Supply BUMDs 4. Laporan Hasil Performance Improvement Action Plan (PIAP) Sampai dengan Bulan Juli 2021 87 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 5. Laporan Pemutakhiran (Update), Target, Kegiatan, Jadwal, dan Sumber Pendanaan Studi Kelayakan Program NRW, Juni 2021 6. PDAM Nominal Increase Assessment 7. Peraturan Menteri Pekerjaan Umum dan Perumahan Rakyat Republik Indonesia Nomor 29 Tahun 2020 Tentang Pemberian Rekomendasi dan Pedoman Teknis Kelayakan Proyek Investasi di Bidang Sistem Penyediaan Air Minum 8. Peraturan Perdirjen P.10 /PDHSAL/SET/KUM.1/5/2019 Tahun 2019 Tentang Perlindungan Mata Air 9. Updated FS for NRW supported by IUWASH PLUS-SECO (in Bahasa Indonesia) IUWASH PLUS EVALUATION REFERENCE MATERIALS 1. IUWASH PLUS Midterm Evaluation Report 2019 2. Activity Monitoring and Evaluation Plan (AMEP) Third Revision 2019 IUWASH PLUS PDAM BUSINESS PLAN 1. Rencana Bisnis Kabupaten Tahun 2020-2024, Kabupaten Sragen 2. Rencana Bisnis PDAM Tirta Gemilang Tahun 2020-2024, Kabupaten Magelang 3. Rencana Bisnis PDAM Tirtanadi Tahun 2021-2025, Provinsi Sumatera Utara 4. Rencana Bisnis PERUMDA Tahun 2019-2023, Kabupaten Probolinggo OTHER DOCUMENTS 1. Gender Strategy 2. Behavior Change Strategy 3. Formative Research Final Report 4. Peta Jalan Pengembangan Kapasitas SDM BUMD AM_full ver PU_cetak 1 juni 21 5. Peta Jalan Pengembangan Kapasitas SDM BUMD Air Minum 20210409 EN VER IUWASH 6. Peta Jalan Pengembangan Kapasitas SDM BUMD Air Minum 20210531 ver IUWASH 7. PPT Bappenas_Diskusi Teknis KKMA-RA 8. PPT IUWASH PLUS Pelembagaan KKMA-RA 9. PDAM Index as September 2021 10. CDCS USAID Indonesia September_2025 11. Innovation in WASH Impact Measures: Water and Sanitation Measurement Technologies and Practices to Inform the Sustainable Development Goals 12. USAID WASHTA Indonesia Report, January 2009, Final 13. BPPSPAM PDAM records on performance index scores 14. USAID Evaluation Reference Materials 15. USAID Evaluation Policy 16. How-to-note: Preparing Evaluation Reports 17. Dataset IUWASH PLUS Performance Index 18. Dataset IUWASH PLUS Governance Index 19. Dataset IUWASH PLUS Sanitation Index 20. Dataset MFI Participants 21. Recapitulation of the Result of Supports on NRW and EE (unpublished report from IUWASH PLUS, in Bahasa Indonesia) 22. Summary Report – Survey of 32,000 Customer Water Meters, IUWASH PLUS-SECO, 2020 (in English) USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 88 ANNEX F. EVALUATION STATEMENT OF WORK Statement of Work: Final Performance Evaluation of Indonesia Urban Water, Sanitation and Hygiene Penyehatan Lingkungan Untuk Semua (IUWASH PLUS) I. Introduction The USAID Indonesia Urban Water, Sanitation and Hygiene Penyehatan Lingkungan Untuk Semua (IUWASH PLUS) is a five year-and-eight month activity (June 22, 2016 - February 21, 2022) designed to assist the Government of Indonesia (GOI) in increasing access to water supply and sanitation services as well as improving key hygiene behaviors among urban poor and vulnerable populations. Implemented by DAI Global LLC (under USAID contract AID-497-TO-16-00003), IUWASH PLUS works with governmental and donor agencies, the private sector, NGOs, communities and others to achieve the following "high level" results; (i) Increased access to improved water service quality to at least 1.1 million urban residents, of which 500,000 are from the poorest 40% of the population; and (ii) Increased access to safely managed sanitation services for at least 500,000 urban residents. To ensure that improvements in access to WASH services are sustained, USAID IUWASH PLUS is guided by a development hypothesis that focuses on strengthening service delivery systems, so they can more effectively reach the poorest and most vulnerable segments of the population. In order to achieve this objective at scale, the program undertakes activities through four interrelated components, including: 1) improving household WASH services; 2) strengthening city WASH institutional performance; 3) strengthening the WASH financing environment; and 4) advancing national WASH advocacy, coordination and communication. In support of these components, USAID IUWASH PLUS also implements a Local Sustainability and Innovation Component (LSIC) that is designed to stimulate WASH innovations that strengthen community, private sector and government WASH service provision. Based on the USAID-SECO Cooperation Arrangement signed on February 20, 2019, the IUWASH PLUS work included a scope of work of $4,499,887 funding by the Swiss State Secretariat for Economic Affairs (SECO) to address two technical challenges confronting municipal water utilities that are especially impactful: (1) high rates of non-revenue water (NRW); and (2) poor energy efficiency (EE). The activities consist of in-depth mentoring and technical assistance for a comprehensive and measurable non-revenue water (NRW) reduction and energy efficiency (EE) improvement in seven water utilities (PDAMs) among a subset of the IUWASH PLUS PDAM partners in West Java and Central Java provinces. Section III B.6 of the USAID-SECO Cooperation Arrangement lists an independent final evaluation of the USAID IUWASH PLUS which will include the SECO financed components, scheduled in July 2021. IUWASH PLUS’ midterm evaluation, conducted in 2019, assessed the project’s approach to reach the bottom 40 percent of the population by wealth (B40), the use of the sustainability checklist, and sanitation marketing. USAID IUWASH PLUS is implementing the recommendation from the midterm evaluation to target a higher percentage of the B40 for increased access to basic or shared sanitation and to improved water services through direct interventions. At the end of Program Year 4, the indirect intervention of USAID IUWASH PLUS to increase access to safely managed sanitation has resulted with 604,670 people or 120% towards the targeted project result (500,000 urban residents with access to safely managed sanitation). However, the indirect intervention of USAID IUWASH PLUS to improve water service resulted in just 902,195 people or 82% from the targeted 89 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV result (1.1 million urban residents). With this updated information, the Mission requested the Monitoring Evaluation and Learning Platform (MEL-P) to conduct a final evaluation to assess and analyze the activity achievements, impacts, and progress towards IUWASH PLUS activity water goals, especially the USAID-SECO scope, and to examine the sustainability and replicability of the activity approaches. In addition, the evaluation will discuss lessons learned and inform USAID’s global learning agenda on urban water interventions. II. Activity Description In consultation with the Government of Indonesia during the site selection process, USAID IUWASH PLUS works with 35 municipalities in its five regions: North Sumatra, West Java-Jakarta￾Tangerang District, Central Java, East Java, South Sulawesi and Eastern Indonesia. In the Water Supply component, USAID IUWASH PLUS is working with 32 PDAMs in 35 municipalities. (Note the activity is not working with the PDAMs in Jakarta and Surabaya cities and PDAM Jayapura serves both Jayapura city and district.) To address the performance of water utilities in Indonesia, USAID IUWASH PLUS provides technical assistance to 32 PDAMs to establish regular monitoring of overall performance through a Performance Index (PI) benchmark. The PI was established to determine utility performance and to identify key weaknesses that prevent them from achieving service excellence. By capturing weaknesses linked directly to performance, the PI also outlines a pathway to guide improvements for the utility's governance institutions and management. The index consists of 18 key performance indicators across the following categories: (1) Financial performance; (2) Operational performance; (3) Service Coverage; (4) Human Resources Management; (5) Business planning and standard operating procedures, and the (6) Safeguarding of Raw Water sources. For each of these topics, detailed criteria are identified, measured and scored, with a total score of 100 points. Calculated annually, the index results are shared with national and local government stakeholders and nearby utilities to incentivize improvements and increase accountability. The USAID-SECO partnership through USAID IUWASH PLUS works with 7 PDAMs: Depok City, Bogor District, Karawang District (West Java), Surakarta City, Sukoharjo District, Magelang City, Magelang District (Central Java). The SECO-financed initiative focused on reduction of NRW (six PDAMs, added by the seventh PDAM in Program Year 2), increased energy efficiency (four PDAMs and added the fifth PDAM in Program Year 5), and related capacity building programs on various key technical and non-technical topics. The results of the initiative will provide an evidence base and example for other utilities—both under the USAID IUWASH PLUS activity and beyond—as they seek to address their own NRW and EE challenges and implement interventions such as the formation of dedicated NRW/EE teams, the adoption of standard operating procedures for NRW reduction and EE, the creation of GIS maps and databases for pipe networks, energy efficiency audits, and water balance measurement. To assist with water services expansion, USAID IUWASH PLUS collaborated with local governments and PDAMS to more effectively utilize public funds, as well as to mobilize additional public and private funds. USAID IUWASH PLUS also partnered with the national ministries (National Planning Agency/ Bappenas and Ministry of Public Works and Housing/ MPWH) to strengthen the enabling environment for water financing and with PT. Sarana Multi Infrastruktur to establish the capital expenditure loan facility to local government, PDAMs, and the private sector. It is important to note that USAID IUWASH PLUS is also affected by the COVID-19 crisis which limited the implementation of field activities and turned most of the technical assistance to virtual USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 90 mode. Many of USAID IUWASH PLUS local government partners had to modify their budget allocation for water to respond to the COVID-19 pandemic. III. Background and Development Hypothesis To ensure that improvements in access to WASH services are sustainable, IUWASH PLUS is guided by development hypothesis that focuses on strengthening the systems to deliver these services as a whole, which includes reducing barriers to access to the poorest and most vulnerable and strengthening the underlying enabling environment. The project is divided into four components – household WASH services and products (Component 1), city/district WASH institutions (Component 2), national WASH financing (Component 3), and National WASH advocacy and coordination (Component 4). IV. Information Sources USAID suggests the following materials for the IUWASH PLUS Midterm Evaluation desk review: 1. IUWASH PLUS Contract and technical program description from the Project Agreement 2. IUWASH PLUS AMEP 3. IUWASH PLUS Quarterly and Annual Progress Reports 4. IUWASH PLUS Work Plans 5. IUWASH PLUS Mid Term Evaluation 6. USAID-SECO funding relevant document 7. Other relevant project document and reports from GOI and partners, including local and national government, water utility, local NGOs and donor organizations 8. Other IUWASH PLUS technical materials, e.g., case studies, factsheets, infographics, profiles, program briefs, technical briefs, et al. V. Evaluation Purpose, Audience and Intended Uses The purposes of this final evaluation are to: (1) assess the performance of USAID-SECO partnership program; (2) measure the project contribution for improving PDAM improved performance; (3) assess IUWASH PLUS’s contribution to improvement of groundwater recharge and ensuring access to safe drinking water for the community; and (4) assess IUWASH Plus’s contribution in improving WASH financing policies and regulation and increasing private sector investment in the sector. This evaluation will provide information to the Mission on the challenges faced, opportunities, and lessons-learned during implementation, which will support future program development With the exclusion of any procurement sensitive sections, USAID intends to disseminate the report widely with stakeholders such as government agencies and NGOs, USAID implementing partners, and donors. The evaluation report will contribute to evidence, as well collaboration and learning priorities of USAID’s global learning agenda on urban Water, Sanitation and Hygiene. This final evaluation will be a performance evaluation as defined in the USAID Evaluation Policy (Annex A). All evaluation materials will be posted to the USAID Development Exchange Clearinghouse (DEC). VI. Evaluation Questions To guide this evaluation, USAID has identified key questions (below) and lines of inquiry in Annex B 91 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV 1. What is driving PDAM performance improvements and what is the role of the BPPSPAM and IUWASH PLUS indices in performance improvement? 2. Are IUWASH PLUS activities resulting in sufficient reduction of NRW and energy efficiency improvements to put PDAMs on a pathway to business viability20? 3. In what ways has USAID IUWASH PLUS’s training and human resources development roadmap contributed to the improved PDAM performance? What accounts for successes? What accounts for failures? 4. What evidence is there that spring vulnerability assessment resulted in sustainable increases of raw water for drinking water providers? 5. In what ways have IUWASH Plus’s activities contributed to PDAMs’ ability to secure alternative financing? VII. Gender Consideration In accordance with USAID’s Automated Directive System (ADS) 201 point 7, the research design for this evaluation will consider gender-specific of IUWASH PLUS. The evaluation team will explore gender aspects of the activity per the questions and data sources in Annex C. VIII. Data Collection and Analysis Methods The evaluation team will propose a comprehensive Evaluation Design and Work Plan to address the evaluation questions, including setting criteria for site visits and field data collection. The objective will be to maximize the evaluation team’s ability to develop evidence-based findings, conclusions, and recommendations that address the purpose and objectives of this evaluation. Within this Design, the evaluation team will propose the best and most rigorous methods for data collection appropriate to address the evaluation questions. The evaluation team will conduct a Team Preparation Meeting during the first week in country to meet with USAID, discuss the evaluation SOW, and prepare detailed data collection and analysis methods. Annex D provides an illustrative “Illustrative Analytical Framework for the Evaluation” matrix, including a range of potential data collection methods that may be suitable for each evaluation question. This matrix will be further developed by the evaluation team in the Evaluation Design. IX. Deliverables The evaluation team will be responsible for the following deliverables. Specific due dates will be proposed in the Evaluation Design, following the TPM. 20 Business viability in this case refers to the potential for PDAM to recover cost for their Non-Revenue Water/Energy Efficiency investment and to increase revenue from service expansion as a result of the amount of water that could be saved and redistributed to new customers. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 92 Deliverable Estimated Due Date 1. Evaluation Design and Work Plan draft, including detailed research methodology, drafts of data collection instruments, sampling plan, and implementation plan & schedule Jul 1, 2021 2. Final Evaluation Design and Work Plan Jul 15, 2021 3. Oral presentation(s) to USAID of key findings and any preliminary conclusions and recommendations. Sep 3, 2021 4. Draft Evaluation Report Sep 18, 2021 5. Final Evaluation Report Oct 28, 2021 X. Reporting and Dissemination The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note￾preparing-evaluation-reports). Evaluation team members will be provided with the USAID’s mandatory statement of the evaluation standards they are expected to meet (see Annex A). XI. Team Composition The suggested composition of the IUWASH Plus Indonesia final evaluation team would be six people: • Team Leader, international, Evaluation Expert • Performance Based Grant Specialist, Indonesian • Water Engineering and WASH Expert Specialist, Indonesian • Private Sector Specialist, Indonesian • Monitoring and Evaluation Specialist, Indonesian • Research Assistant, Indonesian Each Team member will have writing responsibilities for the Evaluation Report draft and final version, per assignments by the Team Leader. All team members should have the following qualifications: • Strong knowledge of Indonesia and the Indonesian WASH sector. • Expertise in program evaluations, including qualitative & quantitative evaluation practices. • Knowledge of USAID programming practices. • For the Team Leader, prior successful experience in leading evaluation or research teams. • Excellent writing and inter-personal communication skills. XII. USAID Participation Regular communication between the evaluation team and the designated USAID Activity Manager will be essential to the successful execution of the IUWASH PLUS final evaluation. The evaluation team will keep USAID apprised of changes and developments that necessitate any significant decision-making or modification of the approved evaluation design. Possible USAID participation in the data collection phase of the evaluation will be discussed in the TPM, prior to the start of fieldwork. 93 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV XIII. Scheduling and Logistics USAID Indonesia has requested MEL-P to finalize the SOW and implement this final evaluation of IUWASH Plus, including handling all logistics. The chart below presents an estimated timetable for this task. Estimated IUWASH PLUS Final Evaluation Timeline Task/ Deliverable Schedules Feb Mar Apr May Jun Jul Aug Sep Oct Start Date End Date SOW and Budget Approval and recruitment Feb 22, 2021 May 24, 2021 Evaluation Preparation and TPM May 25, 2021 Jun 19, 2021 Desk Review May 26, 2021 May 29, 2021 Evaluation Design & Work Plan Jun 22, 2021 Jul 15, 2021 Fieldwork & Data Analysis Jul 26, 2021 Aug 21, 2021 In-Country USAID Debrief(s) Aug 27, 2021 Sep 3, 2021 Report Writing Sep 3, 2021 Sep 18, 2021 Draft Evaluation Report & Other Deliverables Sep 18, 2021 Oct 19, 2021 Final Eval Report & Deliverables Oct 28, 2021 Oct 28, 2021 USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 94 Annex A: USAID Evaluation Policy Appendix I USAID EVALUATION POLICY, APPENDIX 1 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. 95 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Annex B: Evaluation Questions and Lines of Inquiry Evaluation Question Lines of Inquiry 1. What is driving PDAM performance improvements and what is the role of the BPPSPAM and IUWASH PLUS indices in performance improvement? • What is needed to replicate the use of USAID IUWASH PLUS’ performance index to improve performance of other PDAMs? • To what extent has USAID IUWASH PLUS assistance prepared PDAMs to manage both water supply and wastewater services as mandated by RPJMN 2020-2024? 2. Are IUWASH PLUS activities resulting in sufficient reduction of NRW and energy efficiency improvements to put PDAMs on a pathway to business viability21? • In what ways have USAID IUWASH PLUS interventions contributed to the improved capacity of PDAMs to reduce non-revenue water and increase energy efficiency? • In what ways have USAID IUWASH PLUS interventions influenced local government investment for water utilities to reduce NRW and increase energy efficiency? 3. In what ways has USAID IUWASH PLUS’s training and human resources development roadmap contributed to the improved PDAM performance? What accounts for successes? What accounts for failures? • Are there plans to continue the completion of the PDAM human resources development roadmap? • In what ways has USAID IUWASH PLUS helped with identification of PDAM training centres? 4. What evidence is there that spring vulnerability assessment resulted in sustainable increases of raw water for drinking water providers? • Are there plans to replicate USAID IUWASH PLUS spring vulnerability assessments in other spring catchment areas? 5. In what ways have IUWASH Plus’s activities contributed to PDAMs’ ability to secure alternative financing? • In what ways have USAID IUWASH PLUS interventions contributed to the development of the alternative financing guidelines • In what ways have USAID IUWASH PLUS interventions helped PDAM with business to business preparation? 21 Business viability in this case refers to the potential for PDAM to recover cost for their Non-Revenue Water/Energy Efficiency investment and to increase revenue from service expansion as a result of the amount of water that could be saved and redistributed to new customers. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 96 Annex C: Illustrative Gender Aspects of Evaluation Questions Evaluation Question Possible Gender Disaggregation Required 1. What is driving PDAM performance improvements and what is the role of the BPPSPAM and IUWASH PLUS indices in performance improvement? • Gender disaggregated perception data from different stakeholder groups related to PDAM performance improvement 2. Are IUWASH PLUS activities resulting in sufficient reduction of NRW and energy efficiency improvements to put PDAMs on a pathway to business viability22? • Gender disaggregated perception data from different stakeholder groups related to NRW reduction and energy efficiency improvements 3. In what ways has USAID IUWASH PLUS’s training and human resources development roadmap contributed to the improved PDAM performance? What accounts for successes? What accounts for failures? • Gender disaggregated perception data from different stakeholder groups related to the training and human resources development roadmap 4. What evidence is there that spring vulnerability assessment resulted in sustainable increases of raw water for drinking water providers? • Gender disaggregated perception data from different stakeholder groups related to the spring vulnerability assessments 5. In what ways have IUWASH Plus’s activities contributed to PDAMs’ ability to secure alternative financing? • Gender disaggregated perception data from different stakeholder groups related to the PDAMs ability in securing alternative financing 22 Business viability in this case refers to the potential for PDAM to recover cost for their Non-Revenue Water/Energy Efficiency investment and to increase revenue from service expansion as a result of the amount of water that could be saved and redistributed to new customers. 97 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Annex D: Illustrative Analytical Framework for the Evaluation: Getting to Answers Evaluation Question Data Source Data Collection Methods Sampling or Selection Plan Data Analysis Methods 1. What is driving PDAM performance improvements and what is the role of the BPPSPAM and IUWASH PLUS indices in performance improvement? • M&E plan • Annual and Quarterly Report • IUWASH PLUS technical report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and qualitative analysis 2. Are IUWASH PLUS activities resulting in sufficient reduction of NRW and energy efficiency improvements to put PDAMs on a pathway to business viability23? • M&E plan • Annual and Quarterly Report • IUWASH PLUS technical report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and qualitative analysis 23 Business viability in this case refers to the potential for PDAM to recover cost for their Non-Revenue Water/Energy Efficiency investment and to increase revenue from service expansion as a result of the amount of water that could be saved and redistributed to new customers. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 98 3. In what ways has USAID IUWASH PLUS’s training and human resources development roadmap contributed to the improved PDAM performance? What accounts for successes? What accounts for failures? • Annual and Quarterly Report • GOI and partner documents (including local & national government, water utility) • IUWASH PLUS technical report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and qualitative analysis 4. What evidence is there that spring vulnerability assessment resulted in sustainable increases of raw water for drinking water providers? • Annual and Quarterly Report • GOI and partner documents (including local & national government, water utility, local NGOs and donor organizations) • IUWASH PLUS technical report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and qualitative analysis 6. In what ways have IUWASH Plus’s activities contributed to PDAMs’ ability to secure alternative financing? • Annual and Quarterly Report • IUWASH PLUS technical report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and qualitative analysis 99 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV ANNEX G. EVALUATION TEAM AND LOGISTICS Team Composition Evaluation team members are listed in the following table, highlighting each member’s key skills. The section following provides a review of each member’s qualifications and expected areas of focus. Table 5. Evaluation Team Members and Skill Sets TEAM MEMBER POSITION SKILLS & EXPERIENCE EVALUATION WATER AND WASH PBG PRIVATE SECTOR LOCAL GOVERNMENT Keri Culver Team Leader, focus on methodology and gender in infrastructure X X X Risyana Sukarma Performance-based grants specialist X X X X Popy Indrawati Water engineer and WASH lead X X Hartono Kurniawan Private Sector Specialist, with WASH focus X X X Kwan Men Yon MEL Specialist X X X Siti Nur Aini Research Assistant Retno Sri Handini MEL-P Quality control, oversight, USAID relationship Irma Sitompul MEL-P Support for logistics, scheduling and data management Evaluation Team Bios and Roles Keri Culver: Team Leader, responsible for direction of the effort, writing, interface with USAID, and analysis. Keri has extensive experience in evaluation methodology and the conduct of evaluations across sectors and geographies, and brings particular experience with gender, infrastructure, and public-private partnerships to this evaluation. She has more than 20 years of development evaluation experience and is a leader in evaluation education and training. Keri recently served as Team Leader for the independent evaluation of TradeMark East Africa, a USD $500m multi-donor effort to improve trade and trade outcomes for women and vulnerable groups in six countries. Risyana Sukarma: Performance-Based Grant (PBG) Specialist. He is an Indonesian Water and Sanitation Specialist with more than forty years of experience in the field. He was involved in the final evaluations of several water and sanitation projects financed by various bilateral agencies. He holds a sanitary engineering degree from the Institute of Technology, Bandung, Indonesia, and a diploma in hydrology from the International Institute for Hydraulic and Environmental Engineering, Delft, The Netherlands. He is currently working as a Short-Term Consultant in the World Bank and intermittently working as consultant assisting the Bappenas National Secretariat on SDG reporting. He worked as a government official for nineteen years, in charge of water supply planning and development, and joined the World Bank for eleven years as an operations officer on water, sanitation and flood control projects. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 100 Popy Indrawati: WASH and water specialist with experience in operations and performance. She is highly experienced in water supply operations and management, following work with Pam Jaya, Thames Pam Jaya and Aetra Air Jakarta for 28 years. She received a postgraduate in Sanitary Engineering from IHE-Delft, The Netherlands. She joined with BPPSPAM-MPWH as a board member following early retirement from Aetra Air Jakarta, to assess PDAM performance, facilitate PDAM performance improvement, provide recommendations to the Central and Local Governments to improve PDAM performance and maintain the balance between PDAMs and their customers, manage PPP processes and a Center Of Excellence, working with BPPSPAM-MPWH for 2.5 years. She advises the World Bank’s implementation of their WASH intervention (NUWSP), to support the project management unit to deliver the project. Hartono Kurniawan: Private Sector Specialist providing technical insight on the role of the private sector in WASH service delivery. He is an active consultant in drawing up and reviewing project feasibility studies and business cases. He has extensive experience working in and with Indonesian ministries and institutions (Bappenas, MOT, MPWH), donors (AusAid and KOICA), and multi￾national/private companies. He is also highly experienced with infrastructure and Public-Private Partnership schemes in Indonesia. He consulted on the trilateral project between the Governments of Indonesia, the Netherlands and South Korea, called the National Capital Integrated Coastal Development. He was also actively involved in another consultancy project with PT SMI, PT PII (Indonesia Infrastructure Guarantee Funds), GIZ GmbH, KonsulThink, ADHIKARI, LEMTEK, among others. Kwan Men Yon: MEL specialist supporting data collection design and implementation, analysis and report writing. Men Yon has a combined 14 years of experience in designing and administering research activities and has applied various evaluation and knowledge generating methods, including outcome mapping, outcome harvesting, positive deviance approach and double-loop learning in research activities. In his recent assignment as MEL specialist for a US$25 million USAID-funded civil society support program, Men Yon oversaw a baseline survey with 1,970 respondents covering 32 districts in Indonesia. Retno Handini: Evaluation Lead for MEL-P. Oversight, support and inputs, and quality control. She is highly experienced in evaluation design, research, implementation, analysis and report writing. Experience includes WASH evaluations and serving for five years as an M&E Specialist for the World Bank’s Public Sector Governance Practice in Jakarta. Dini holds advanced degrees in economics and has in-depth knowledge of Indonesian governance from assignments carried out for the World Bank, GTZ, CIDA and AusAid. Irma Sitompul: MEL-P Project Officer supporting all aspects of the evaluation’s operations. A decade of strong experience in project and office operational and community development project management, with several international humanitarian organizations. She has provided support to programs on activity planning, logistic arrangements, budget management, compliance, administration and filling system. Irma adapts well to changing evaluation and contextual conditions and will support the evaluation to run smoothly. She was assisted by Siti Nur Aini, who was contracted for this evaluation in particular. 101 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Level of Effort The following table summarizes the LOE for the evaluation, by team position and task. Table 6. Estimated Level of Effort per Team Member EVALUATION TEAM ACTIVITIES TEAM LEADER WASH AND WATER SPECIALIST PBG SPECIALIST PRIVATE SECTOR SPECIALIST MEL SPECIALIST Preparation Planning & Coordination 5 3 3 3 4 Research & Literature Review 4 3 3 3 2 Methodology Development 4 1.5 1.5 1.5 3 Field Work Finalize Evaluation Design 5 0.5 0.5 0.5 1 Data Collection – Jakarta + Fieldwork 4 18 18 18 14 Real-time Analysis 7 5 5 5 7 USAID Presentation 3 0.5 0.5 0.5 0.5 Analysis and Reporting Additional Data Collection & Analysis 2 2 2 2 2 Report Writing - Draft 13 5 5 5 5 Report Finalization 4 0.5 0.5 0.5 0.5 TOTAL 51 39 39 39 39 The MEL-P support to the team included a research assistant along with oversight and quality control from MEL-P leadership. Deliverables The evaluation team’s deliverables were as follows: 1. Evaluation Design and Work Plan, to include evaluation methodology and instruments that will be applied to answer evaluation questions; evaluation schedule; list of people/groups to be interviewed; timeframe; and draft schedule of field activities. 2. Debriefing, PowerPoint presentation of preliminary findings with bulleted response to evaluation questions, and discussion on development of recommendations. 3. Draft Evaluation Report, clearly describing findings, conclusions, and recommendations. USAID will provide comment on the Draft Evaluation Report within ten working days of submission. 4. Final Evaluation Report following USAID standard evaluation report format and branding guidelines, within 10 working days of receiving Mission comment on the draft report. The format of the final report is provided below. The report will be submitted in English, electronically. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 102 In addition to these formal deliverables, the team presented the evaluation design and work plan in a session with USAID to solicit feedback and discuss nuances of the plan. We communicated throughout the evaluation to provide progress updates, opportunities and challenges, any requested revisions to tools or schedule, and other issues. The final Evaluation Report does not exceed 30 pages (excluding Executive Summary, table of contents, references, and annexes), and adheres to requirements as stipulated in the evaluation statement of work (Annex F). The major headings of the report include: i. Executive Summary ii. Background iii. Methods and limitations iv. Findings v. Conclusions vi. Recommendations vii. References viii. Annexes In accordance with AIDAR 752.7005, USAID/Indonesia will make the final Evaluation Report publicly available through the Development Experience Clearinghouse (DEC) following approval by the designated COR. Mandatory Reference All members of the evaluation team have been provided with USAID’s mandatory reference of the evaluation standards they are expected to meet, as the following text box shows, along with USAID’s conflict of interest statement. MANDATORY REFERENCE FOR ADS CHAPTER 201 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT 1. Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. 2. Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. 3. The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. 4. Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. 5. Evaluation methodology should be explained in detail and sources of information properly identified. 6. Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 7. Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. 8. Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. 9. If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. 10. If recommendations are included, they should be supported by a specific set of findings and should be action￾oriented, practical, and specific. 103 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV ANNEX H. RESULT FRAMEWORK FOR IUWASH PLUS IUWASH PLUS develops its work through the following development hypothesis: To ensure that improvements in access to Water, Sanitation and Hygiene (WASH) services are sustainable, IUWASH PLUS is guided by a development hypothesis that focuses on strengthening the systems to deliver these services as a whole, which includes reducing barriers to access to the poorest and most vulnerable and strengthening the underlying enabling environment. This hypothesis and the IUWASH PLUS activity act across four components as specified above, and their 2019 AMELP shows the following results framework/theory of change for the activity: Figure 18. IUWASH PLUS Results Framework/TOC USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 104 ANNEX I. SURVEY DATA ANALYSIS Composition of the Sample: Demographics Nearly half of respondents were technical (46 percent) or administrative staff (22 percent). More than two-thirds of respondents were on their sixth year or more as PDAM staff. This might indicate that IUWASH PLUS training were attended by more mid- to senior-level staff. Note that from the list of sample respondents, 23 people have entered retirement age when contacted for this survey. Figure 19. Respondents by PDAM job type Source: Evaluation team calculations; n= 87 Figure 20. Respondents by duration as PDAM staff Source: Evaluation team calculations; n= 90 105 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV The vast majority of respondents were university graduates and were not civil servants or state workers in status. About 22 percent of respondents had a high school diploma or less education. Figure 21. Respondents by employment status and educational attainment Source: Evaluation team calculations; n= 90 Capacity building: Relevance, level of satisfaction and perceived changes Overall, capacity building related to the preparation of business plans (BPs) is the most frequently attended training activity for both female and male staff (see Figure xx below). However, women were much less likely to attend trainings in field or technical activities (NRW, EE and water source vulnerability or KKMA) and more likely to be trained in BPs, the performance index, or administration. No female respondents participated in alternative financing training. Respondents who do not mention gender were excluded from this calculation; totals are greater than 100% because respondents could select more than one training type they had attended. Figure 22. Training topic participation, by gender 55% 50% 54% 49% 33% 47% 26% 8% 23% 15% 25% 17% 27% 17% 26% 9% 0% 8% 22% 25% 22% Male Female Total Business Plan NRW Energy efficiency Performance index Watersource vulnerability Alternative financing Other Source: Evaluation team calculations; n= 90 The training that IUWASH PLUS-SECO respondents said they attended most was NRW training, ahead of the preparation of business plans, as the figure below shows. More IUWASH PLUS-SECO respondents (proportionally) participated in NRW and EE activities while IUWASH PLUS trainees were much more likely to attend trainings on the preparation of business plans and KKMA. It is interesting that the percentage of respondents who take part in the PDAM performance index activity is relatively small, perhaps because this activity is carried out only for a certain period each USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 106 year. Alternative financing activities were the least mentioned by respondents, highlighting a lower priority for this issue in relation to PDAM capacity building. Totals are greater than 100% because respondents could select more than one type of training they attended. Interestingly, a quarter or nearly a quarter said they were trained in “other topics,” an unexpected finding given that the training topics listed in the table come directly from IUWASH PLUS-SECO. Respondents may be generous in their recall or perhaps not recalling the topics well. Figure 23. Training topics, by IUWASH PLUS-SECO vs. IUWASH PLUS trainees 47% 63% 56% 53% 45% 48% 48% 20% 22% 21% 14% 16% 17% 32% 26% 8% 9% 9% 18% 25% 22% IUWASH PLUS-SECO IUWASH PLUS Total Business Plan NRW Energy efficiency Performance index Watersource vulnerability Alternative financing Other Source: Evaluation team calculations; n= 94 There were both online and offline training activities, roughly divided between pre-COVID-19 and during COVID-19. Offline training had a higher number of participants and a tendency to have more hours of training than did online training (or other capacity building activities). Some three-quarters of respondents who had taken part in offline training types said they had had up to fifty hours of activities with IUWASH PLUS; however, two-thirds (67%) of those reporting online training cited fewer than 20 hours. The intensity of activities appears reliably to have decreased when carried out online – not uncommon or unwarranted, given the challenges of online learning. Respondent totals are greater than sample size because some reported both online and offline training experiences. 107 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 24. Offline capacity-building activities were more intensive than online 26% 67% 49% 22% 25% 12% Offline Online More than 50 hours 20 - 49 hours Less than 20 hours About a third (35%) of respondents said they were still participating in IUWASH PLUS training activities in 2021. On the other hand, almost the same percentage (34 percent) said the last time they participated was in 2019. Fewer than 10% had not attended IUWASH PLUS trainings since 2018. Figure 25. Most recent IUWASH PLUS training, by year 2% 2% 3% 34% 24% 35% 2016 2017 2018 2019 2020 2021 Source: Evaluation team calculations; n= 80 About 44 percent of male respondents stated that the IUWASH PLUS training was highly relevant for their work, compared to 40 percent of female participants. None said the training was not relevant to their work. Note that 14 respondents – two women and twelve men – did not answer this question. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 108 Figure 26. Relevance of training to their work, by sex 56% 60% 56% 44% 40% 44% Male Female Total Highly relevant Relevant Source: Evaluation team calculations; n= 80 Both IUWASH PLUS-SECO and IUWASH PLUS respondents reported that the training was relevant to their work, in approximately equal measure (43% IUWASH PLUS-SECO and 44% IUWASH PLUS reported “highly relevant”). The remainder of respondents in each group reported the trainings as “relevant.” The relevance of training is of course crucial for capacity building, so the more relevant the training is, the better. Figure 27. Relevance of training to their work, by IUWASH PLUS-SECO vs. IUWASH PLUS 57% 56% 56% 43% 44% 44% IUWASH PLUS-SECO IUWASH PLUS Total Highly relevant Less relevant Relevant Source: Evaluation team calculations; n= 80 Nearly all respondents said they were satisfied or very satisfied with the training material/ content, training method, provision of real-life examples, availability of equipment, and quality of trainers. More women than men said they were “very satisfied”, an unexpected finding since more men than women found the training “highly relevant.” 109 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 28. Satisfaction with training content, by sex 32% 42% 34% 66% 58% 65% 1% 1% Male Female Total Very satisfied Satisfied Disatisfied Very disatisfied Source: Evaluation team calculations; n= 89 Just one IUWASH PLUS-SECO training participant, and no IUWASH PLUS participants, reported dissatisfaction. IUWASH PLUS-SECO training participants were somewhat less likely to be “very satisfied” than IUWASH PLUS respondents. Given that IUWASH PLUS-SECO instituted thoughtful training adaptations with the advent of the SECO partnership, this is a surprising finding and warrants further exploration about the causes for dissatisfaction. Figure 29. Satisfaction with the training content, by IUWASH PLUS-SECO vs. IUWASH PLUS 22% 42% 34% 75% 58% 65% 3% 1% IUWASH PLUS-SECO IUWASH PLUS Total Very satisfied Satisfied Dissatisfied Source: Evaluation team calculations; n= 89 The same response patterns (slightly greater satisfaction from women respondents; slightly lesser satisfaction from IUWASH PLUS-SECO trainees) appear in the remaining charts on satisfaction levels. In the charts that follow, we present the breakdown on satisfaction between IUWASH PLUS￾SECO and IUWASH PLUS. Given that women’s participation in the survey was quite low (only about 11%), we simply note any deviance from this response pattern for breakdown by sex. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 110 Figure 30. Satisfaction with training method, by IUWASH PLUS-SECO vs. IUWASH PLUS 19% 42% 33% 78% 58% 66% 3% 1% IUWASH PLUS-SECO IUWASH PLUS Total Very satisfied Satisfied Dissatisfied Source: Evaluation team calculations; n= 89 One-third of respondents said they were very satisfied with the provision of real-life examples in the IUWASH PLUS training. Note that the percentage of female who stated that they were very satisfied with this aspect was the lowest compared to other aspects in the capacity building activity. Meanwhile, a significantly lower percentage of satisfaction was reported by IUWASH PLUS-SECO participants than for IUWASH PLUS for the provision of real-life examples in training. A very small percentage of IUWASH PLUS-SECO participants said they were not satisfied with this aspect. Figure 31. Satisfaction with real-life examples, by IUWASH PLUS-SECO vs. IUWASH PLUS 19% 42% 33% 78% 58% 66% 3% 1% IUWASH PLUS-SECO IUWASH PLUS Total Very satisfied Satisfied Dissatisfied Source: Evaluation team calculations; n= 89 About 29 percent of male respondents reported being very satisfied with the aspect of the availability of practice equipment. However, this percentage was the lowest compared to other aspects of satisfaction. The percentage of males who are highly satisfied was also lower than females. The percentage of IUWASH PLUS-SECO respondents who reported being very satisfied with equipment provision was lower than IUWASH PLUS participants, which is interesting given the relatively larger investment of equipment for IUWASH PLUS-SECO PDAMs. Both IUWASH PLUS￾SECO and IUWASH PLUS participants reported dissatisfaction with the provision of practice equipment. IUWASH PLUS respondents’ satisfaction with the provision of equipment was the lowest 111 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV compared to other aspects. This finding demonstrates the complexity of equipment provision in training. Figure 32. Satisfaction with practice equipment, IUWASH PLUS-SECO vs. IUWASH PLUS 23% 34% 30% 71% 62% 66% 6% 4% 5% IUWASH PLUS-SECO IUWASH PLUS Total Very satisfied Satisfied Dissatisfied Source: Evaluation team calculations; n= 88 The level of satisfaction with the trainer was relatively the highest compared to other aspects. Both female and male participants reported being very satisfied with the trainer, higher than with other aspects of satisfaction. No respondents said they were not satisfied with the trainer’s performance. In that case, it can be said that trainers were the most appreciated aspect of the IUWASH PLUS training delivery. IUWASH PLUS respondents were more satisfied with the trainers than were IUWASH PLUS-SECO respondents. Figure 33. Satisfaction with trainer(s), IUWASH PLUS-SECO vs. IUWASH PLUS 25% 47% 58% 52% 47% 58% IUWASH PLUS-SECO IUWASH PLUS Total Very satisfied Satisfied Dissatisfied Source: Evaluation team calculations; n= 89 USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 112 Across the board, the percentage of IUWASH PLUS-SECO respondents who were very satisfied with various aspects of the IUWASH PLUS training was lower—ranging from 19 to 25 percent— than IUWASH PLUS’s which was 34 to 47 percent. Although this survey cannot be said to be representative, the finding might be usefully interpreted in terms of expectations: IUWASH PLUS￾SECO participants may have higher expectations of capacity building delivery, given that the IUWASH PLUS-SECO PDAMs were relatively larger PDAMs than IUWASH PLUS. Nearly all participants (96%) stated they had applied the new knowledge or skills. Among the few who said they had not used the knowledge, the reason given was mainly job transfers. Both IUWASH PLUS-SECO and IUWASH PLUS respondents, at almost the same frequency, reported developing an action plan after training. IUWASH PLUS-SECO respondents were required to do so, in order to obtain their training certificate. Respondents who do not make an action plan said this was because there was no obligation to do so. Figure 34. Action plan development, IUWASH PLUS-SECO vs. IUWASH PLUS 11% 11% 11% 89% 89% 89% IUWASH PLUS-SECO IUWASH PLUS Total Develop action plan No Yes Source: Evaluation team calculations; n= 89 All respondents who answered this question said they had implemented the action plan, but 17 percent did not answer. Nearly all respondents reported personal improvement in knowledge and skills as a result of training delivery, with none answering “no,” and only a handful answering “don’t know.” Of the latter, there were slightly more IUWASH PLUS respondents (3) who answered “don’t know.” How does the increase in knowledge and skills bring about change at the personal and organizational levels? We found that the change most cited by respondents at the personal level was a better understanding of the field of work. This relates to increasing knowledge and skills. However, higher order changes, like capability and confidence in doing the job, were reported much less frequently. This might indicate that training has helped achieve the increasing knowledge and understanding, but changes at the level of behavior and practice were more elusive. Similar findings were seen for IUWASH PLUS-SECO and IUWASH PLUS. 113 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 35. Personal changes as a result of training, IUWASH PLUS-SECO vs. IUWASH PLUS 34% 49% 43% 37% 43% 41% 94% 80% 86% 3% 6% 5% IUWASH PLUS-SECO IUWASH PLUS Total More capable of doing my work More confidence Understand my field of work better Other Source: Evaluation team calculations; n= 90 About 87 percent of respondents stated the IUWASH PLUS trainings have made a difference to the organization. Having a business plan to guide the PDAM strategically was by far the most frequent change respondents reported (SECO and IUWASH PLUS); this parallels BPs being the most frequently cited training. IUWASH PLUS respondents seem to perceive more organizational change than do IUWASH PLUS-SECO respondents; however, IUWASH PLUS-SECO participants were more likely than IUWASH PLUS to cite an increase in PDAM energy efficiency and increased ability to develop alternative financing. Note 12 missing responses for this question, perhaps indicating slightly less confidence in organizational changes. Figure 36. Organizational changes as result of training, IUWASH PLUS-SECO vs. IUWASH PLUS 74% 81% 78% 48% 53% 51% 39% 30% 33% 42% 40% 41% 29% 45% 38% 16% 15% 15% 3% 11% 8% IUWASH PLUS-SECO IUWASH PLUS Total Having a business plan to guide PDAM strategically Reduced NRW Increased energy efficiency Increased capability to measure PDAM performance Having an action plan to deal with watersource vulnerability Increased capability to develop alternative financing Other USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 114 Source: Evaluation team calculations; n= 82 Over two-thirds (70%) of respondents said they need other kinds of training, with 16 percent saying they didn’t know if they did, and 13 percent saying they did not need more training. Respondents from smaller PDAMs were somewhat more likely (85%) than those from larger PDAMs (79%) to report needing more training to be able to do their jobs well. PDAM Performance Improvement Most respondents, both IUWASH PLUS-SECO and IUWASH PLUS, reported that PDAM performance was very much better or better than in the previous 12 months. The percentage of IUWASH PLUS-SECO respondents stating PDAM performance was much better or better was slightly lower than IUWASH PLUS. On the other hand, the percentage of IUWASH PLUS-SECO respondents saying PDAM’s performance was the same was slightly higher than that of IUWASH PLUS. A very small percentage of IUWASH PLUS-SECO and IUWASH PLUS respondents said that PDAM’s current performance was worse. Figure 37. Perceptions of PDAM performance, IUWASH PLUS-SECO vs. IUWASH PLUS 21% 61% 16% 3% 23% 66% 9% 2% 0% 10% 20% 30% 40% 50% 60% 70% Far better Better Same Worse IUWASH PLUS-SECO IUWASH PLUS Source: Evaluation team calculations; n= 94 The survey identified some differences in participants’ perceptions of PDAM performance improvement between IUWASH PLUS-SECO and IUWASH PLUS respondents for four areas: NRW reduction, energy efficiency, development of business plan and customer outreach especially to low￾income communities (see Figure xx). Far more IUWASH PLUS-SECO respondents reported improved PDAM performance in NRW reduction and energy efficiency; on the other hand, more IUWASH PLUS respondents said that performance was improving in the areas of business plans and low-income community outreach – all in line with IUWASH PLUS-SECO and IUWASH PLUS programming and mentoring priorities. More IUWASH PLUS-SECO respondents also reported improved performance in reaching full-cost recovery (FCR). IUWASH PLUS participants were more likely to report improved performance measurement, but IUWASH PLUS-SECO and IUWASH PLUS respondents reported almost on equally that clean water supply, wastewater treatment, and financial conditions had improved. 115 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV Figure 38. Perceptions of improved performance, IUWASH PLUS-SECO vs. IUWASH PLUS 21% 23% 30% 43% 45% 45% 68% 74% 28% 45% 31% 34% 28% 59% 38% 72% 0% 10% 20% 30% 40% 50% 60% 70% 80% Reaching full-cost recovery Energy efficiency Financial condition Performance measurement Outreach low income community NRW reduction Business plan and targets Clean water supply IUWASH PLUS-SECO IUWASH PLUS Source: Evaluation team calculations; n= 94 About 70 percent of respondents said human resource development was the factor affecting performance improvement or change. Other factors that are most frequently mentioned were comprehensive and measurable business plan, competent PDAM leadership, strengthening of guidelines and standard procedures, support from local government and improved staff capacity. However, there were differences in the factors selected the most by IUWASH PLUS-SECO and IUWASH PLUS respondents. Most IUWASH PLUS-SECO respondents stated human resource development as a factor driving performance improvement, while IUWASH PLUS respondents suggested the factor of comprehensive business plan. The improved staff capacity factor was also significantly more reported by IUWASH PLUS-SECO respondents than IUWASH PLUS. Meanwhile, IUWASH PLUS respondents valued more factors of support from local government, better communication with the district leader and legislative support. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 116 Figure 39. Perceptions, factors affecting performance, IUWASH PLUS-SECO vs. IUWASH PLUS 14% 36% 36% 36% 42% 54% 56% 60% 64% 70% 19% 23% 42% 32% 65% 58% 48% 58% 81% 58% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Access to alternative financing Legislative support Innovative technical approach Collective decision-making Improved staff capacity Strengthened SOPs LG support Competent leadership Human resource development Measurable business plan UWASH PLUS-SECO IUWASH PLUS Source: Evaluation team calculations; n= 94 The vast majority of respondents reported that the COVID-19 pandemic had affected PDAM performance, with only eight percent saying performance was not affected. About a third called this “greatly affected” in IUWASH PLUS-SECO sites, and 30 percent in IUWASH PLUS sites. Figure 40. Experience of COVID-19 impact, IUWASH PLUS-SECO vs. IUWASH PLUS 33% 30% 31% 61% 58% 60% 6% 9% 8% IUWASH PLUS-SECO IUWASH PLUS Total Not affected Affected Deeply affected Source: Evaluation team calculations; n= 89 When asked about the benefits the trainees experienced, the two benefits most cited were trainings that were suitable to their needs (SECO 89%, IUWASH PLUS 68%), and technical assistance (both around two-thirds). Other benefits that were reported the most, respectively, were reduced NRW (61% of IUWASH PLUS-SECO participants, and 28% of IUWASH PLUS participants) and 117 | REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS USAID.GOV development of strategic documents (i.e., business plans) (22% IUWASH PLUS-SECO participants, 53% IUWASH PLUS). If we group the outcomes by more immediate gains (process gains) and longer-chain improvements (outcomes), respondents tended to cite more of the immediate gains. Figure 41. Perceived gains from interventions, IUWASH PLUS-SECO vs. IUWASH PLUS 0% 13% 17% 53% 25% 15% 28% 21% 26% 13% 64% 68% 3% 19% 11% 22% 17% 31% 61% 8% 25% 11% 69% 89% Other Development of guideline documents (i.e, training manuals, work manual, toolkit, etc) Development of necessary documents (i.e, feasibility studies, environmental studies, etc) Development of strategic documents (i.e, business plan) Facilitating better linkage between multi-stakeholders Increased energy efficiency Reduced NRW Increasing watersource availability Innovative work method Demonstration or experimental projects Technical assistance Trainings that are suitable to needs IUWASH PLUS-SECO IUWASH PLUS Source: Evaluation team calculations; n= 89 Female respondents reported experiencing significantly more benefits than did male respondents; this may reflect different expectations that females and males have of capacity building opportunities. However, due to the small sample size, it is difficult to conclude that this finding is robust. USAID.GOV REPORT FINAL PERFORMANCE EVALUATION OF IUWASH PLUS | 124 ANNEX K. COMPARISON OF BPPSPAM AND IUWASH PLUS PI List of Performance Index BPPSPAM and IUWASH PLUS BPPSPAM IUWASH PLUS 1 Financial Aspect 1 Financial Aspect 1.1 Rentability 1.1 Full cost recovery 1.1.1 ROE 1.2 Current Ratio 1.1.2 Operation Ratio 1.3 Collection Efficiency 1.2 Liquidity 1.2.1 Cash Ratio 1.2.2 Collection Effectiveness 1.3 Solvency 2 Service Aspect 2 Service Aspect 2.1 Coverage ratio 2.1 Connection Increase 2.2 Customer growth 2.3 Level of complaint settlement 2.4 Customer water quality 2.5 Water consumption (domestic only) 3 Operational Aspect 3 Operational Aspect 3.1 Production efficiency 3.1 NRW Level 3.2 NRW 3.2 NRW Program 3.3 Operating hours/day 3.3 Production water meter replacement 3.4 Customer connection pressure 3.4 Customer water meter replacement 3.5 Water meter replacement 3.5 Customer/pipe spatial data 3.6 MIS application connected 4 HR Aspect 4 HR Aspect 4.1 Ratio of number of employees/1000 customers 4.1 Employee Ratio 4.2 Ratio of employee training/competency improvement 4.2 Training Ratio 4.3 Training costs/employee costs 4.3 Training budget Ratio 5 Administration Aspect 5.1 Business Plan 5.2 Conformity yearly budget plan and actual spending 5.3 SOP of financial/administration 5.4 SOP of Customer relation 5.5 SOP of Production 5.6 SOP of Distribution 6 Raw Water Aspect 6.1 Raw water program 6.2 Regulation/budget for raw water protection program U.S. Agency for International Development/ Indonesia Jl. Medan Merdeka Selatan 3-5 DKI Jakarta, Indonesia