FINAL PERFORMANCE EVALUATION LEBANON WATER PROJECT (LWP) DAI GLOBAL LLC FINAL REPORT May 2021 Prepared by Social Impact, Inc. for the United States Agency for International Development under USAID Contract/Order No. AID-268-C-15-00001 – Performance Management and Support Program for Lebanon (PMSPL II). This document is not available in print. Documents of this nature are made available to the public through the Development Experience Clearinghouse (DEC) repository at dec.usaid.gov. For additional information, please contact: Social Impact, Inc. Corporate Office 2300 Clarendon Boulevard Suite 10300 Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com Or Social Impact, Inc. Arz Street Librex Bldg. Bloc B – 3rd Floor Zalka, Metn, Lebanon Tel: +961-1-879260 DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. COVER PHOTO: Water pumps installed by LWP at Jradeh pumping station. PHOTO CREDITS: All photos by PMSPL II. ABSTRACT USAID’s Lebanon Water Project (LWP) is a $75-million activity implemented by DAI Global LLC. It was launched in October 2015 and will conclude in July 2021. LWP’s purpose is to improve: i) Service delivery and resource management of Lebanon’s five public-water utilities; ii) Civic engagement in water management and citizens’ perception of water utilities’ role in providing a necessary and valuable service; and iii) Private-sector involvement in water management. This final performance evaluation analyzed the effectiveness of LWP using key informant interviews, site visits, and a document review. The evaluation found that LWP’s construction activities, management plans and tools, and training courses had improved Lebanon’s public water establishments. The degree of impact varies across establishments and activities. Delays have occurred in some construction projects resulting in handover to the regional water establishments during the last year of the project, hence affecting the transfer of knowhow to the establishments. Several projects were still ongoing at the time of the evaluation. Many of these delays could have been corrected during the last eighteen months of the project, had the economic, political, and COVID crisis not crippled the country since October 2019.The enterprise resource planning (ERP) activity could not be completed at the North Lebanon Water Establishment (NLWE). LWP collaborated with another on-going USAID-funded project, Community Support Project (CSP), with USAID’s approval, to continue working on the pending tasks at NLWE, to establish ERP support after the LWP project closes on July 29, 2021, to help ensure the sustainability of ERP accomplishments. LWP has established a diverse and wide-ranging grants program covering communities, agriculture, and industry with more than 32 grants. Media coverage has been substantial, with national campaigns reaching over 1.17 million people. Several public-private partnerships (PPPs) were signed with two Regional Water Establishments (RWEs) for fees collection. Tender documents have been prepared and suggested for selected RWEs for model water supply PPPs, but no partnerships have been signed to date given the current economic crisis. External contextual factors emerged in October 2019 and continue to date. They have affected LWP’s performance, causing additional delays in implementation and a major shift in the priorities of the water establishments whose main concern became the day-to-day sustainability of their operations. The evaluation provides several recommendations to guide future projects in the water sector based on LWP’s successes and challenges as well as local opportunities and constraints. i | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT CONTENTS ABSTRACT................................................................................................................................I CONTENTS ..............................................................................................................................II TABLES...................................................................................................................................III FIGURES..................................................................................................................................III ACRONYMS............................................................................................................................ IV EXECUTIVE SUMMARY........................................................................................................... VI Evaluation Purpose......................................................................................................................................... vi Introduction and Project Background....................................................................................................... vi Evaluation Questions..................................................................................................................................... vi Evaluation Design and Methods................................................................................................................. vii Key Findings and Conclusions.................................................................................................................... vii Recommendations......................................................................................................................................... xii INTRODUCTION ......................................................................................................................1 Project Description ........................................................................................................................................ 1 Problem Statement ......................................................................................................................................... 2 Development Hypothesis.............................................................................................................................. 3 Critical Assumptions ...................................................................................................................................... 3 EVALUATION PURPOSE AND QUESTIONS ..............................................................................4 Purpose.............................................................................................................................................................. 4 Evaluation Questions...................................................................................................................................... 4 EVALUATION METHODS AND LIMITATIONS...........................................................................5 FINDINGS AND CONCLUSIONS ...............................................................................................6 Question 1........................................................................................................................................................ 6 Sub-Question 1.1 Findings..................................................................................................................... 6 Sub-Question 1.1 Conclusions...........................................................................................................18 Sub-Question 1.2 Findings...................................................................................................................21 Sub-Question 1.2 Conclusions...........................................................................................................23 Sub-Question 1.3 Findings...................................................................................................................23 Sub-Question 1.3 Conclusions...........................................................................................................27 Question 2......................................................................................................................................................27 Findings.....................................................................................................................................................27 Conclusions.............................................................................................................................................32 RECOMMENDATIONS.............................................................................................................33 ANNEXES...............................................................................................................................35 Annex A: Inception Report and Scope of Work...................................................................................36 Annex B: Detailed Evaluation Design and Methodology .....................................................................37 Annex C: Evaluation Schedule and Persons Interviewed ....................................................................44 Annex D: Documents Reviewed...............................................................................................................46 Annex E: Data Collection Instruments....................................................................................................48 Annex F: Details of LWP Major Activities..............................................................................................54 ii | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT Annex G: Conflict of Interest Disclosures..............................................................................................56 Annex H: ET Members ................................................................................................................................57 TABLES Table 1. Distribution of KIIs by stakeholder...............................................................................................................5 Table 2. List of site visits..................................................................................................................................................6 Table 3. Data source matrix .........................................................................................................................................39 Table 4. Evaluation Team roles and responsibilities................................................................................................58 FIGURES Figure 1. Lebanon CDCS Development Objective 1 and its Intermediate Results...........................................2 Figure 7. A farmer beneficiary in the Bekaa: “While reducing my water bill, I have also reduced my Figure 8. The winemaker of the cooperative: “Since the introduction of drip irrigation systems, the Figure 2. Qalhat elevated reservoir in Koura DA .....................................................................................................7 Figure 3. Generator installed by LWP in Deychouniyeh wells.............................................................................10 Figure 4. Rehabilitated intake from Litani River to the Khirbet Qanafar Wetland..........................................11 Figure 5. Upgraded Khirbet Qanafar Laboratory.....................................................................................................17 Figure 6. Respondent satisfaction as per baseline and endline CPSS ..................................................................22 pesticide application bill.”...............................................................................................................................................26 quality of the wine produced has improved.”...........................................................................................................26 Figure 9. Control at Ghazzeh pumping station.........................................................................................................29 Figure 10. Achievement status of performance standards.....................................................................................32 iii | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT ACRONYMS A&E Architecture and Engineering ADS Automated Directives System AFD Agence Francaise de Developpement ALI Association of Lebanese Industrialists AUB American University of Beirut BMLWE Beirut and Mount Lebanon Water Establishment BWE Bekaa Water Establishment CCIAZ Chamber of Commerce, Industry and Agriculture CDCS Country Development Cooperation Strategy CEEDD Chaire de l’Education a l’Eco-citoyennete et au Developpement Durable COP Chief of Party COR Contracting Officer’s Representative COVID-19 Coronavirus Disease of 2019 CPSS Citizen Perception and Satisfaction Survey CSC Customer Service Center CSO Civil Society Organization CSP Community Support Program CW Constructed Wetland DA Demonstration Area DAI Development Alternatives, Inc. Global LLC DCOP Deputy Chief of Party DEC Development Experience Clearinghouse DG Director General DMA District Metered Area DO Development Objective E2 Engineer and Educate EBRD European Bank for Reconstruction and Development EIA Environmental Impact Assessment EOI Expression of Interest EOP End of Project ERAC Environmental Review and Assessment Checklist EQ Evaluation Question EQUITM Evaluation Quality Use and Impact ERP Enterprise Resource Planning ET Evaluation Team FGD Focus Group Discussion GIS Geographic Information System ICT Information and Communication Technology IP Implementing Partner IR Intermediate Result IRB Institutional Review Board IRP Incentive Rebate Program IT Information Technology IWA International Water Association KII Key Informant Interview LOE Level of Effort LOP Life of Project LRA Litani River Authority LRC Lebanon Relief Council iv | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT LRBMS Litani River Basin Management Support LWP Lebanon Water Project LWPP Lebanon Water Policy Program LWWSS Lebanon Water and Wastewater Sector Support M&E Monitoring and Evaluation MOEW Ministry of Energy and Water NGO Non-Governmental Organization NLWE North Lebanon Water Establishment NRW Non-Revenue Water NWSS National Water Sector Strategy O&M Operation and Maintenance OECD Organization for Economic Cooperation and Development OU Operating Unit PBC Performance-Based Contract PDOC Project Decreasing Operation Costs PMSPL II Performance Management and Support Program for Lebanon II PODA Project Outside Demonstration Area PPP Public Private Partnership PS Performance Standard PSP Private Sector Participation PV Photovoltaic QA/QC Quality Assurance & Quality Control RAR Rapid Assessment Report RMP Risk Management Plan RWE Regional Water Establishment SI Social Impact, Inc. SLWE South Lebanon Water Establishment SMEs Small and Medium Enterprises TL Team Leader UNICEF United Nations International Children’s Fund USAID United States Agency for International Development VFD Variable Frequency Drive WDM Water Demand Management WE Water Establishment WELI Water Efficiency in Lebanese Industry WISE Water Infrastructure Support Enhancement WQAP Water Quality Assurance Plan WQMP Water Quantity Monitoring Plan WSA Water Security Analysis WWTP Wastewater Treatment Plant v | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of the Lebanon Water Project (LWP) final performance evaluation is to assess to what extent LWP has met its intended objectives and performance standards (PSs) and to provide recommendations for future water activity design. This final performance evaluation is intended for United States Agency for International Development (USAID)/Lebanon, particularly the Program Office and the Local Development Office, DAI Global LLC (DAI), and other parties at the discretion of the Mission. INTRODUCTION AND PROJECT BACKGROUND This draft final evaluation report is submitted to USAID/Lebanon in response to the call for a Final Performance Evaluation of LWP, a $75 million activity that extends from September 2015 until end of July 2021 and is implemented by DAI. The objective of LWP is to increase access to clean, reliable, and sustainable sources of drinking water for Lebanese citizens. LWP focused on improving Lebanon’s capacity in the management of water resources, enhancing the efficiency and sustainability of the public water utilities, and addressing water challenges arising from the impact of the Syrian refugee influx to Lebanon. The activity provided technical and capital assistance to relevant stakeholders and counterparts to enhance the performance of the water sector in Lebanon. LWP also promoted better water governance to help ensure long-term preservation of Lebanon’s water resources. EVALUATION QUESTIONS USAID/Lebanon provided a list of questions that were then incorporated into two main evaluations questions (EQs). These questions and sub-questions are presented below: 1. To what extent have LWP’s interventions achieved the objectives of the Activity as stated in the contract? More specifically: 1.1. In terms of improving service delivery and resource management capacity (result A), the evaluation is to answer whether: 1.1.1. the Regional Water Establishments (RWEs) have become more effective in water delivery. 1.1.2. the RWEs are now better equipped in managing their resources. 1.2. In terms of improving the civic engagement in water management (result B), the evaluation is to answer whether: 1.2.1. the perception and satisfaction of citizens towards RWEs have improved. 1.2.2. the Lebanese citizens are better aware and proactive towards the importance of water management and conservation. 1.3. In terms of improving the private sector involvement in water management (result C), the evaluation is to answer whether: 1.3.1. the targeted private sector is more involved and engaged in water management and conservation initiatives. 1.3.2. The farmers and industries are more involved in water conservation. 2. Did LWP achieve the performance standards relevant to each of the Results A, B and C? vi | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT EVALUATION DESIGN AND METHODS The evaluation was conducted during the Coronavirus Disease of 2019 (COVID-19) pandemic while Lebanon was witnessing a surge in positive cases. Places of business were not completely open, group gatherings were restricted, and a practice of social distancing was in place. In this context, the Evaluation Team (ET) designed this evaluation using a mixed-methods approach to collect and analyze primary and secondary data. Primary data were collected through key informant interviews (KIIs) with LWP project staff, the four RWEs, the Litani River Authority (LRA), project beneficiaries, private-sector partners, and other stakeholders, and through direct on-site observation of implemented activities as needed. Secondary data included the review of project documentation, work plans, progress reports, technical reports, and surveys. An evaluation matrix was used for data analysis to ensure that all data sources are considered. Triangulation of data was used to answer each question with more validity. This approach allowed the ET access to adequate information to assess the LWP activity’s implementation and results against its objectives, scope of work, and deliverables, and to determine whether the set tasks were accomplished, how well the activities were implemented, and to what extent the activities met the targets set in the project documentation. Data collected during this evaluation also provided information on how external and internal factors affected activity implementation and results. The ET consisted of a Local Team Leader/Water Sector Expert, two Local Team Members/Water Sector experts, with technical support from Performance Management and Support Program for Lebanon (PMSPL) II. Other members of the Social Impact (SI) Lebanon office provided ad-hoc inputs. The ET was supported by SI Headquarters members who provided overall guidance on the evaluation and quality control on the deliverables. KEY FINDINGS AND CONCLUSIONS QUESTION 1: TO WHAT EXTENT HAVE LWP’S INTERVENTIONS ACHIEVED THE OBJECTIVES OF THE ACTIVITY AS STATED IN THE CONTRACT? • Sub-question 1.1: In terms of improving service delivery and resource management capacity (result A), have the RWEs become more effective in water delivery (1.1.1) and are the RWEs now better equipped in managing their resources (1.1.2)? Overall, LWP’s activities, management plans, tools, and trainings improved the effectiveness of the water establishments (WEs) in water delivery and rendered them better equipped for managing their resources. The degree of impact varies between RWEs, as the scale and nature of the interventions varied between RWEs, depending on their needs and their commitment. LWP had faced delays in design and implementation of construction projects before the October 2019 Revolution, the economic crisis, the COVID-19 lockdowns, and the Beirut Port blast. The latter further exacerbated these delays. At the time of the evaluation, it seemed to the ET that the extension period was not sufficient for the timely completion of all activities. Delays in completion of some construction activities resulted in some delays in handover. These delays in handover, and most importantly the economic and political crisis that began in October 2019 and was further exacerbated by the COVID pandemic, jeopardizes the effectiveness of the transfer of the projects to RWEs and the latter’s ability to adequately identify pending problems and issues and successfully take over the operation in a sustainable manner . The one-year liabilities period will allow time for the RWEs to ensure the quality of the delivered products, especially that LWP has made arrangements with its subcontractors for the defects liability performance bonds to be transferred to the RWEs. Service Delivery: 1. The closed loop approach in the Demonstration Areas (DAs) was very well received by the RWEs, who are eager to replicate it in other areas. This activity helped RWEs improve service delivery in vii | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT these areas on various fronts: (1) by increasing production through the rehabilitation of sources; (2) by improving supply and reducing technical losses through the installation of new pumps, the installation of chlorinators, the construction/rehabilitation of transmission lines and distribution networks, and the construction/rehabilitation of reservoirs; and (3) by enhancing supply and demand management through the installation of source, district, and customer meters, the updating and consolidation of consumer registers, and the identification of commercial losses through illegal connections. This resulted in the reduction of physical non-revenue water (NRW), which was reported to range between 6 and 41 percent across District Metered Areas (DMA) analyzed in each DA, through the identification and mitigation of technical losses. It also allowed the identification of some commercial losses and led to an increase in subscriptions. However, since LWP could not obtain results on billing and payment by consumers, the calculated NRW was limited to the difference between the supplied water through the distribution networks and the water consumed. 2. The promotion of smart meters and their installation in the DAs, along with the update of the customer registers and the training of staff on meter reading, set the stage for the implementation of consumption-based tariffs by RWEs. BMLWE and North Lebanon Water Establishment (NLWE) already apply consumption-based tariffs in other areas and can swiftly start applying them in the DAs. Until then, subscribers may be consuming more water, as consumption is controlled neither physically (as it used to be with the orifice regulators) nor financially through imposing consumption-based tariffs. The fact that the water is not yet supplied 24/7 may help limit overconsumption. 3. Delays in handover of the DAs resulted in some pending hardware and software issues related to data collection, transfer, and analysis, in particular with regards to water meters and staff not feeling confident to take over. Trained staff leaving the establishments further exacerbated this issue. Delays in handover also led to delays in the planned WDM activities, which are still being finalized. Specifically, for BMLWE, which was the last DA to be handed over, this will limit the extent of follow-up, on-the-job training, and handholding in WDM. While LWP faced many challenges, these can be learned from, as they are real challenges faced by RWEs. 4. RWEs were very satisfied with Projects Outside Demonstration Areas (PODAs), which improved water services for vulnerable Lebanese citizens and refugee communities and enhanced social stability, without imposing additional operation and maintenance (O&M) costs on the RWEs. It was noted in one of the PODAs, that water supply was increased but the RWE was not able to ensure billing, which contributed to an increase in NRW. It is understood that NRW was not the objective of this activity, and that this intervention was a success story in satisfying water needs and diffusing tensions in a vulnerable community, which might outweigh the costs of NRW. Yet, this did not align with other project results. 5. Rehabilitating existing and functional wastewater treatment plants (WWTPs) improved wastewater treatment efficiency and reduced O&M costs without imposing a new burden on RWEs in operating these plants. 6. The installation of drip irrigation in Chabrouh was a very successful project, resulting in a 15% increased efficiency in irrigation systems. 7. Water quality improved with the improvement in infrastructure, including the installation of new chlorinators. RWEs are recently faced with the challenge of providing chlorine gas in light of the current economic crisis. The United Nations International Children’s Fund (UNICEF) is helping in this respect. 8. Project Decreasing Operation Costs (PDOCs) are a timely initiative that will help decrease operating costs on RWEs and promote the sustainability of service delivery, which is a major viii | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT challenge under the current circumstances. While they are expected to decrease O&M costs by 25 percent in the facilities where they are installed, their exact impact could not be calculated yet as this activity is still ongoing. 9. LWP included a training component in all its subcontracts with formal and on-job-training for all its activities, thus training more than 1,150 persons (of which 25 percent were female) with 68 training courses and more than 254 training days. The training provided through LWP was well structured and well received. In many instances Directors General (DGs) and staff asked for additional and follow-up training. The certification training and the leadership training received high praise. However, for some activities like WDM, further training and hand holding that goes beyond water meter reading to water balance calculations and data analysis and follow up is needed. Resource Management: 10. While the business plans were well made, they were not implemented, neither by the departing nor by incumbent DGs. The decision to introduce risk management plans (RMPs) was good, providing a more focused list of priorities for national and international funding. However, RMPs need to be reshuffled following the economic crisis to reflect the new priorities. RWEs are well trained to do that on their own. 11. Enterprise Resource Planning (ERP) as a concept is excellent and has a lot of potential. However, for it to be appreciated by users, it has to be completed and the users should be able to successfully generate all their annual reports from it. For LWP, the bulk of the scope was to implement the ERP at the NLWE, where this system was not available, and to provide high level technical support to ERP support and maintenance subcontractors hired by the other RWEs. However, the new SLWE and BWE DGs appointed in 2018 did not renew the ERP maintenance and support contracts. LWP was not able to complete the implementation of the ERP activity at NLWE, while relying on its current limited inhouse resources and having to accommodate for continuing requests from the RWEs for support in modules that were considered completed. The overall allocated level of effort (LOE) was not sufficient to install a new ERP system for the NLWE and to handle the incremental ERP support requests of the other RWEs, especially the Bekaa Water Establishment (BWE), where software upgrade and data migration were required. Resistance of staff in some RWEs and their turnover following the 2019 posed additional challenges to the LWP ERP team. Adequate management of ERP activities through prioritizing activities towards the completion of the NLWE ERP system and/or procuring additional support could have helped complete this activity. Subcontracting this activity to a third party may have yielded better results, both in terms of tapping into more human resources to complete the project and in terms of ensuring post-LWP sustainability, as ERP support and maintenance knowhow would not be restricted to two inhouse consults. Where the ERP system is complete and functioning smoothly, there is a need for continuous support and maintenance of the system. If the RWEs want to contract out the ERP support and maintenance, they have a limited choice due to limited availability of expertise in Navision in the local market. This jeopardizes the sustainability of the system. Continuous training on the ERP system and other systems is needed since already trained staff are leaving the establishments due to the current crisis. 12. Information and Communication Technology (ICT) assessments and subsequent upgrades were very well received and helped improve the ICT environment at the RWEs for current and future projects. 13. The asset valuation and the financial and performance audits were very beneficial, particularly in RWEs where the ERP system was functioning adequately and where annual reports could be generated smoothly. In RWEs where the work on ERP is still in progress (BWE and NLWE), there are still some issues with assets valuation, particularly updating the list of assets (NLWE) and ix | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT generating the annual depreciation reports (BWE). The RWEs are building on the assets valuation activity and completing their assets lists. 14. Improved billing and collection were successful in the South Lebanon Water Establishment (SLWE) through the partnership agreements with agencies offering bill payment services and the use of the hand-held devices. In BWE, the partnership agreement resulted in an insignificant increase in paid bills and the hand-held devices were not used, pending approval from the Ministry of Finance. In NLWE, the partnership agreements were not signed yet, although they were prepared, and the handheld devices are not in use yet. The approval by the Ministry of Finance to use them in bill collection was issued in 2021. 15. The Assessment of the River Gauging Stations in Lebanon and the Water Quantity Monitoring Plan (WQMP) were adopted by the LRA and endorsed by the Ministry of Energy and Water (MOEW). The recommendations were included in the 2020 National Water Sector Strategy (NWSS) to secure funds to improve river flow monitoring across Lebanon. The BWE Irrigation Master Plan was also an important tool, as acknowledged by the DG where irrigation schemes were assessed and georeferenced for the first time. It will help guide donors in funding future projects. However, currently there is no adequate staff to follow up on it. 16. The Water Security Analysis (WSA) reports will help guide the establishments to better manage their resources, taking into account the five dimensions of water security: (i) domestic, (ii) environment, (iii) economic, (iv) risk management, and (v) water infrastructure, with governance being a cross-cutting dimension. The RWEs are trained and capable of updating their RWE score using a simple tool developed by LWP. A national WSA was prepared for the MOEW upon its request and following the success of the RWE WSAs. The Water Quality Assurance Plans (WQAPs) were developed as part of the WSAs. They have seen some follow-up activities in three of the five establishments in terms of procuring laboratory testing equipment and sub-contracting water testing. However, there is more to these plans that could be enforced if followed up on. • Sub-question I.2: In terms of improving the civic engagement in water management (result B), the evaluation is to answer whether the perception and satisfaction of citizens towards RWEs have improved and whether Lebanese citizens are better aware of and proactive towards the importance of water management and conservation. LWP’s civic engagement work and communication efforts, combined with institutional and infrastructural improvements, contributed towards enhancing the image of the RWEs and bridging the gap between them and their customers, as was evident in the end line Citizen Perception and Satisfaction Survey (CPSS), where the number of subscribers who said were very satisfied with the RWE’s service increased by 17% between 2016 and 2021 to reach 44%. In addition, the number of subscribers trusting RWEs to manage the water services in Lebanon increased by 19% to reach 23%. When considering all respondents and not just subscribers, despite the general dissatisfaction with governmental institutions primarily caused by the recent developments in the country, the level of satisfaction in the services provided by the RWEs in the four DAs slightly increased from 71% in the baseline survey to 74% in 2020. However, a statistical significance testing of the CPSS results was not conducted and would have provided more robust evidence. The civic engagement projects targeted different types of beneficiaries, including households, farmers, industries, students, schools, corporate firms, wineries, and communities. Although activities were small, when added together they provide a nice umbrella effect for water conservation and efficiency. Extensive awareness campaigns were conducted as part of LWP. Although the percentage of respondents remembering these campaigns (16%) and the number of respondents noticing non￾governmental organizations (NGOs) working in the water sector (14%) remained almost the same between 2016 and 2020, survey results showed that 49% of respondents who saw/heard awareness x | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT campaigns to save water had taken related steps at their households and reported mainly “to turn off the faucet when not needed” (38%) or “fix faucet malfunctions immediately to avoid wasting water” (12%).” So, while not many respondents may not have recalled the campaigns when asked, the behaviors of those who did seem to have changed as a result. • Sub question I.3: In terms of improving the private sector’s involvement in water management (result C), the evaluation is to answer whether the targeted private sector is more involved and engaged in water management and conservation initiatives, and whether the farmers and industries are more involved in water conservation. LWP engaged the private sector in water management and conservation through developing different forms of private sector participation (PSP) to (1) improve water service delivery, (2) leverage the funding of innovative water conservation technologies, and (3) improve water usage awareness. Partnerships between RWEs and two payment service providers provided a better payment alternative to water customers. However, the impact varied by establishment. While successful for SLWE, this new tool was not widely adopted by BWE customers who were resistant to change and needed more time to accept to shift from traditional tariff collectors to service providers. The model performance-based framework contracts for public private partnerships (PPPs) prepared jointly by LWP and the various RWEs were viewed positively by DGs. However, the ongoing economic and political situation created a sizeable obstacle to their implementation because of the current inability of the RWEs and some private sector firms to engage in financial borrowing or loan activities. Yet, these contracts can be readily used by RWEs when needed and will help improve the performance of the private sector in PPP contracts. To leverage the funding of water conservation technologies, LWP’s grants program worked with private sector entities from industry (stone-cutting factories) and technology (pump station predictive maintenance). LWP partnered with the Association of Lebanese Industrialists (ALI) to reach out to industries and educate them on possible technologies for water saving and incentivize industrialists to adopt them. Nine stone cutting industries were selected for partnership with LWP which involved them in water conservation and cleaner production processes that resulted in a 67% decrease in water consumption. The Incentive Rebate Program (IRP) allowed 10 farmers to cost-share the installed water-saving irrigation equipment and benefit from the technical support of agricultural suppliers. Furthermore, a grant given to the Chamber of Commerce, Industry and Agriculture (CCIAZ) supported two major farmers and one cooperative in applying water-saving techniques. The individual farmers and industries that received support from LWP have successfully adopted the water conservation methods. However, their role as forerunners and as examples for their neighbors and fellow colleagues to follow is constrained by the current economic crisis. Dealing with farmer cooperatives, municipalities, and professional associations has proven efficient and allowed for good outreach. Hence, promoting new technologies through associations/cooperatives/municipalities can have a wider outreach as compared to targeting individual farmers or industries. To improve water usage awareness, LWP established water stewardship (water conservation, reuse, and treatment) cooperation commitments with companies in the media, banking, manufacturing, and wine industries. This activity was well received by the private sector during the first half of the LOP (pre￾crisis) as they innovated in their awareness campaigns by issuing dedicated newsletters, providing financial initiatives, covering water-related events, etc. QUESTION 2: DID LWP ACHIEVE THE PERFORMANCE STANDARDS RELEVANT TO EACH OF THE RESULTS A, B, AND C? xi | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT Out of the 17 Performance Standards (PSs): - LWP exceeded nine PSs: 1. The number of citizens who benefited from improved water services (772,827 vs. a target of 673,000); 2. Eight context-appropriate sustainable models for improving sanitation service delivery were implemented (target: at least six); 3. The efficiency of irrigation systems in the DAs was improved by 15% via the Chabrouh Project (target: 12%); 4. Several water resources conservation and protection initiatives at the Litani Basin yielding tangible and measurable results (target: at least 1); 5. Eight generators were provided for emergency preparedness to facilities in one or more WEs (target at least five); 6. Eight civil society groups or citizen organizations were engaged in selected areas of intervention (target: at least five); 7. The development of at least several water resources conservation and protection initiatives in the Litani Basin including the treatment of whey water from 14 dairy farms (target: at least one); 8. More than 60% decrease in non-productive wasteful usage of water in both rural and urban settings, particularly in irrigation and in stone cutting industries (target: 15%); and 9. Developing four pilot private sector partnership to improve water usage awareness with media, banking, manufacturing, and wine industries (target at least one), - LWP achieved four PSs: 1. Developing six context-appropriate sustainable models for improving communal water service delivery; 2. Achieving at least a 15% increase in customer satisfaction with service delivery through public water utilities in the selected areas of intervention; 3. Developing a pilot private sector partnership to improve water supply, wastewater, or irrigation service delivery, namely between SLWE and BWE on one hand and two payment service providers on the other to improve bill collection; and 4. Developing one pilot private sector partnership with ALI to leverage the funding of innovative water conservation technologies in nine stone-cutting industries, - LWP partially achieved two PSs: 1. WQAPs, which were developed but not fully enforced by all RWEs and 2. Fifteen percent (15%) reduction in NRW, which could not be assessed in all DAs and did not include quantification of commercial NRW. - LWP was unable to achieve one PS: Fifteen percent (15%) increase in customers who pay bills (only an increase of 8% was achieved before the completion of the DAs and the economic crisis) and 2. - One PS was still in progress at the time of the evaluation and its achievement is yet to be confirmed. It pertains to the twenty-five percent (25%) reduction in O&M costs, as projects were not completed at the time of the evaluation. - One PS could not be assessed due to the lack of available data. It pertains to the reduction in hours of lost service in the selected areas of intervention due to poor utility performance. RECOMMENDATIONS Based on the findings and conclusions presented above, the ET proposes several recommendations for USAID/Lebanon and/or its implementing partners (IPs) for future work in the water sector. What successes can be built on or replicated in the future? 1. USAID/IPs to consider replicating the closed system approach in improving water service delivery and reducing NRW. 2. USAID/IPs to consider replicating the water conservation initiatives with industries and farmers or other heavy water users through partnerships and grants with associations, cooperatives, and/or municipalities. 3. USAID/IPs to consider replicating the leadership training for both female and male staff at RWEs. xii | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT 4. USAID/IPs to consider replicating the PDOCs, as they are much needed during the current crisis to support RWEs in ensuring continuity of service. What challenges should be further addressed or avoided in the future? 5. IPs to plan all construction activities while accounting for delays in both design approval and construction (due to USAID architecture and engineering (A&E) quality assurance & quality control (QA/QC) requirements, Environmental Review and Assessment Checklist (ERAC) and Environmental Impact Assessment (EIA) approvals, as well as unexpected external factors). 6. IPs to plan the completion of construction activities, particularly those requiring extensive data processing post-construction and installation, such as water demand management in DAs at least 1.5 years before the End of Project (EOP) to ensure an adequate handover, training, and transfer of knowledge to staff on equipment operation and water demand management through hand-holding, on-the-job training, and follow-up for at least six months. 7. USAID/IPs to consider following up on the WDM exercises conducted in the LWP DAs to further train RWE staff from all relevant sections—i.e., production, networks and distribution, and billing and customer service—on all elements of WDM and the NRW model and on implementing the developed WDP through hand-holding and on-the-job training. 8. IPs to encourage RWEs to consider dedicating permanent staff for WDM. 9. USAID/IPs, when replicating the closed system approach, to consider that installing smart meters at the consumer level without imposing consumption-based tariffication can result in an unwarranted increase in consumption, which needs to be addressed through technical controls and/or awareness campaigns until the consumption-based tariff is implemented to curb overconsumption. 10. IPs to consider challenges faced by RWEs in imposing consumer-based tariffs and to couple improved water supply projects with improved billing to avoid increasing NRW. 11. USAID/IPs to differentiate in their performance standards between physical and commercial NRW. What local conditions/constraints need to be considered in implementing future projects? 12. Given the current crisis, USAID/IPs to consider placing more focus on stabilization, rather than on development, to ensure that the RWEs are being able to use existing tools, manage what they have, and operate and maintain existing facilities. 13. USAID/IPs to encourage new PPP initiatives that focus on innovative ways to attract the private sector into a productive partnership with the RWEs in light of the current crisis. 14. USAID/IPs to consider investing in building local technical capacities for the development, support, and maintenance of the ERP system to ensure its sustainable use. This could include organizing a formal ERP unit in each RWE and training a joint support team for all establishments. 15. USAID/IPs to consider continuous training and capacity building of RWE staff on all operational aspects with focus on permanent staff, as contractual staff are leaving for better-paid jobs given the current crisis. What local opportunities need to be built upon in implementing future projects? 16. USAID/IPs to encourage the participation of locally-developed innovation and local innovative small and medium enterprises (SMEs) in implementing future projects, such as the Engineer and Educate (E2) activity. xiii | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT 17. USAID/IPs to build upon Lebanon’s excellent academic/research centers when implementing future projects. One example is the Chaire de l’Education a l’Eco-citoyenneté et au Développement Durable (CEEDD). 18. USAID/IPs to benefit from the hundreds of associations/organizations/cooperatives that are active in Lebanon in the fields of water, agriculture, rural development, and/or community support. 19. USAID/IPs to consider implementing awareness campaigns on television to attract a wider range of audiences. Communication budgets should take that into account and ensure broadcasting for several weeks so that advertisement/ awareness campaigns can have an impact and be remembered. Awareness raising on social media and in schools should be continued as the most preferred means in this regard. xiv | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT INTRODUCTION Lebanon Water Project (LWP) is a $75 million activity that extends from September 2015 until July 2021, awarded by USAID, and implemented by DAI Global LLC. The objective of LWP is to increase access to clean, reliable, and sustainable sources of drinking water for Lebanese citizens. LWP focuses on improving Lebanon’s capacity in the management of water resources, enhancing the efficiency and sustainability of the public water utilities, and addressing water challenges arising from the impact of the Syrian refugee influx to Lebanon. The activity provides technical and capital assistance to relevant stakeholders and counterparts to enhance the performance of the water sector in Lebanon. LWP also promotes better water governance to help ensure long-term preservation of Lebanon’s water resources. LWP builds upon previous USAID-funded water development projects: Litani River Basin Management Support (LRBMS), Lebanon Water Policy Program (LWPP), Lebanon Water and Wastewater Sector Support (LWWSS), and Water Infrastructure Support Enhancement (WISE) to achieve its objective to improve water governance and protection of the public water sector in Lebanon. PROJECT DESCRIPTION The objective of LWP is to improve water governance and protection, to ensure reliable and sustainable access to water in Lebanon through improving the service delivery and resource management capacity of five public utilities, and improving civic engagement in water management, and citizen’s perception of water utilities in providing a necessary and valuable service. LWP aimed to achieve this objective through targeted interventions categorized under three purposes, by employing a strategy that brings the public sector, private sector, and civil society together to conserve and improve the management of Lebanon’s water resources. The strategy focuses on improving the capacity of public water utilities, engaging citizens in communicating with public water entities, and promoting citizen behavioral changes to conserve water and protect water quality through media campaigns. Concurrently, LWP will implement activities to increase private sector engagement in finding solutions to water service delivery and water quality problems through partnerships with the public water entities. Results A: Improved service delivery and resource management capacity of the five public water utilities. To achieve Results A, LWP will strengthen the managerial and operational functions of the water entities while providing related upgrades to their infrastructure. The project will support the Government of Lebanon’s newly developed water strategy, which boosts the role of these RWEs and LRA as the legitimate entities to provide efficient services to the population. Purpose B: Improved civic engagement in water management and advanced citizens’ perception of water utilities’ role in providing a necessary and valuable service. To achieve Purpose B, LWP will bolster the legitimacy of the RWEs and LRA by engaging citizens, local authorities, and the commercial sector through improving their understanding and support for steps to improve water management and delivery of water and wastewater services by the water utilities and municipal governments. LWP will also employ a communications strategy to build awareness on water use behavior and to engage civic society in water conservation and management. Purpose C: Improved private sector involvement in water management. To achieve Purpose C, LWP will support and mediate pilot private sector partnerships to improve water and wastewater management, leverage the funding of innovative water conservation technologies, and improve water use awareness. Activities under Purpose C will be cross-cutting along Purpose A and Purpose B, with overlapping outcomes such as increased customers who pay bills, decreased non-revenue water (NRW), and decreased wasteful uses of water. 1 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT day but is as low as three hours per day in some locations. Water utilities have paid virtually no attention to wastewater and irrigation and, although Lebanon’s wastewater network covers 60 percent of the country, less than 10 percent is properly treated. The remaining 90+ percent is generally discharged directly to the environment, causing severe pollution to ground and surface water resources. Surface water resources are heavily exploited, have limited storage, and do not have sufficient environmental protection. There is intense stress on groundwater, which is unsustainably withdrawn to meet ever increasing demands. Water utilities usually operate in financial deficit due to the very low percentage of customer-metered connections (approximately 10 percent nationwide), the low percentage of billing and collection, and the lack of a proper tariff strategy. Gaps in the legal/regulatory framework, the lack of sector reform, the high percentage of NRW, and the lack of a sense of stewardship at the citizen level have entrapped the water sector in a vicious cycle. Citizens have lost faith in the capacity of the water utilities and local municipalities to provide proper services and are reluctant to pay their water bills. Citizens have resorted to other means and methods of securing potable water and disposing of wastewater that harms the environment and people’s health. The private sector has played a very limited role in water and wastewater services delivery. Lebanon recently ratified Law 48/2017 regulating PPPs. Guidelines on administering the new law are yet to be established by the Higher Council for Privatization and Partnership. This should encourage PPPs in the water sector. The Syrian crisis that started in March 2011 has added yet another stress to the already fragile water and wastewater sector. The number of Syrian refugees in Lebanon is more than 1.5 million (40 percent of Lebanon’s 4.1 million pre-crisis population). This refugee influx has seriously affected all of Lebanon's service sectors. DEVELOPMENT HYPOTHESIS LWP is based on the following theory of change: If LWP can improve public water utility service delivery and resource management capacity, advance civic engagement in water management, improve citizen perception of water utilities as providing a necessary and valuable service, and expand private sector involvement in water management, LWP will be able to improve water governance and protection to ensure reliable and sustainable access to water. CRITICAL ASSUMPTIONS The successful achievement of LWP objectives and purposes is based on several critical assumptions which take into consideration the Lebanese political, security, and institutional context. 1. Security situation in Lebanon does not significantly deteriorate. 2. RWEs and LRA are willing participants in project activities. 3. Civil society and the private sector are willing partners for collaboration in project activities. 4. International donors are coordinating and aligning their activities with each other and with RWEs and LRA. Since October 17, 2019, additional risks were presented to the existing context. Latest developments included a revolution (the October Revolution) that paralyzed the country, a financial and economic crisis that saw the local currency devaluating by almost ten-fold, a pandemic (COVID-19), a major disaster (the Beirut port explosion), and a political vacancy (the country has been running without a government for the last 10 months). 3 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT EVALUATION PURPOSE AND QUESTIONS PURPOSE With LWP reaching its end of term on July 29, 2021, its Contracting Officer’s Representative (COR) requested that PMSPL II conduct a final performance evaluation2 to assess the extent to which LWP has met the intended objectives and its performance standards. The evaluation was conducted with adherence to SI’s Evaluation Quality Use and Impact (EQUI™) approach, processes, and protocols. EQUI processes are designed to achieve a quality evaluation that incorporates learning and utilization. Based on EQUI, this evaluation has a utilization focus. In addition to assessing program performance in achieving planned objectives, the evaluation highlights lessons learned and provides recommendations, taking into consideration future programs in Lebanon. Aligned with this mission, SI will continue to actively engage USAID, implementing partners (IPs), and other stakeholders in the evaluation process. This final performance evaluation is intended for USAID/Lebanon, specifically the Program Office, Local Development Office, DAI, and others at the discretion of the Mission. In the spirit of the USAID Evaluation Policy, the evaluation provides USAID/Lebanon with concise, actionable recommendations based on evidence to be used by USAID for future activity design or program work plan improvement. It is anticipated that the results from the evaluation will be used by USAID/Lebanon during its annual Portfolio Review. The report from this evaluation will become publicly available on the Development Experience Clearinghouse (DEC). EVALUATION QUESTIONS The ET conducted meetings with USAID and LWP/DAI during the evaluation’s kick-off in order to determine priorities and add clarity to the EQs based on the Mission’s intended use of the evaluation findings, conclusions, and recommendations. As the mid-term evaluation conducted in 2018 answered questions related to the relevance, effectiveness, and related factors affecting the achievement or non-achievement of the objectives, sustainability and gender integration, this final performance evaluation only looks at the effectiveness of the activity and answers the below questions. 1) Main Question: To what extent have LWP’s interventions achieved the objectives of the Activity as stated in the contract? Sub-questions: 1.1. In terms of improving service delivery and resource management capacity (result A), the evaluation is to answer whether: 1.1.1. the RWEs have become more effective in water delivery. 1.1.2. the RWEs are now better equipped in managing their resources. 1.2. In terms of improving the civic engagement in water management (result B), the evaluation is to answer whether: 2 “A performance evaluation focuses on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and to answer other questions that are pertinent to future program design, management and operational decision making” (USAID Evaluation Policy, 2011). 4 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT connections. RWE DGs and staff also reported an increase in subscriptions due to improved services. However, there were delays in design and construction, identified in 2018 at the midterm evaluation, which were then mainly attributed to the quality assurance/quality control (QA/QC) program. The latter, although very effective and resulting in good designs and good construction, came with a cost in time. This delay was further exacerbated by several external factors starting at the beginning of Year 5 (October 2019 revolution and blocked roads, national economic crisis and devaluation of the Lebanese Pound, COVID-19 pandemic and associated lockdowns starting March 2020, and the Beirut port blast in August 2020) leading to a delayed handover of two DAs towards the end of the project (March 2021 for Koura, and April 2021 for Metn). Hence, at the time of the final evaluation, the ET noted through site visits and interviews with RWE staff that there were still some pending issues to be resolved following the handover, particularly software issues related to meter reading synchronization, transfer of collected data from meter, data processing at RWE headquarters, consolidation of subscribers’ lists, provision of GSM cards by the RWEs, etc. One reason behind these software issues was the high turnover in staff that RWEs have witnessed lately due to the financial crisis, whereby LWP had to do repeat the training to new teams appointed by the RWEs. Figure 2. Qalhat elevated reservoir in Koura DA As per the reviewed LWP project documents, Water Demand Management (WDM) was a key component of the DAs, with the ultimate objective of reducing NRW and increasing hours of service in these areas, through the combined activities implemented in the DAs. However, the delays in infrastructure activities and subsequently project handover as mentioned above, resulted in delays in the completion of the WDM exercise, which was presented to the RWEs in late April 2021. The reviewed draft WDM reports revealed that the exercise was completed for the whole of Zahleh, Jezzine and Koura DAs, while in Metn, the exercise was limited to two villages due to delays in meter installations. The ET noted that the WDM exercise proved to be a challenging and time-consuming exercise. However, the interviews with RWE DGs and LWP staff as well as the reviewed draft WDM reports showed that it was beneficial in allowing the identification of technical losses (network leaks, reservoir overflow, defective valves, etc.), commercial losses (illegal/ unauthorized consumption, etc.), and the analysis of customer consumption behavior. 7 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT With reducing NRW being the primary reason for LWP to choose the closed system approach in DAs, the WDM exercise for each area culminated in the calculation of NRW following the implementation of all the planned interventions. NRW reduction as reported by LWP in the reviewed WDM reports across the District Metered Areas (DMA) analyzed in each DA ranged between 6 and 41 percent, with the highest reduction achieved in Jezzine. The calculated NRW was based on the difference between the supplied water through the distribution networks and the water consumed by the subscribers. Since LWP could not obtain results on collection of payment by consumers, it was not possible to compute the commercial NRW, or the actual revenues forgone by the RWEs at this stage, before implementing the consumption-based tariffs. As for the impact of the DA interventions on hours of service, no data was available at the time of the evaluation, and hence improvements in this respect could not be assessed. Interviews with the RWE DGs revealed that despite the fact the consumption-based tariffs have not been implemented yet, LWP, through the installation of the smart meters, the update of the consumer registers, and the various awareness campaigns and town hall meetings discussed in the sub question 1.2 below have set the stage for the implementation of consumption-based tariffs and provided the necessary tools for that. Billing based on consumption remains an unresolved issue at the national level, particularly with regards to agreeing on a tariff structure that promotes conservation, while ensuring that it is progressive and assures equity and social justice, and is beyond the control of USAID/ IPs. Some DGs explained to the ET that they were planning to start this implementation after a period of communicating to the consumers the volumes of water they are using and the expected consumption￾based tariff they would accrue as per their consumption behavior and which the RWE staff expect to be equal to or less than what the consumers are paying now in the majority of the cases. Interviewed RWE staff confirmed that LWP trained designated staff on meter reading. LWP also provided a WDM Plan to all RWEs. However, various interviewed RWE staff expressed to the ET the need for more training on updating the customer register, data processing using the software installed at the headquarters, using the collected and processed data to conduct similar WDM exercises and following up on the results, and on operating some of the installed equipment where handover was late (Metn). Finally, some additional findings by the ET that influenced the project outcome but that were beyond the control of the LWP team: - Interviews with some RWE DGs and staff and LWP staff revealed that the 24/7 supply, which was a main criterion for the selection of DA could not be achieved due to a combination of factors that are beyond the control of the LWP, such as the delay in the completion of planned water supply augmentation projects, and the intermittent power supply. This has caused problems in the functioning of the smart meters. - Interviews with RWE DGs revealed that they understand the importance of consumption-based tariffs but are not able to implement them yet. An RWE head of supply in a DA noted that installing smart meters without the implementation of a consumption-based tariff system tends to promote overconsumption of water. To mitigate this problem, in one DA, orifice regulators are being reinstalled, along with the existing smart meters, to control the volume of supplied water, until consumption-based tariffs are implemented and are capable of influencing consumer behavior. - In another DA, one RWE DG revealed that turbidity in the supplied water is affecting smart meter readings and the trust and acceptance of the consumers of their use. The DG acknowledged that the RWE is responsible for providing turbidity-free water and has pledged to ensure that the storage reservoirs are regularly clean to avoid this problem. 8 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT Water Delivery PODAs In addition to the activities in the DAs, LWP implemented small infrastructure PODAs, 3 in the Bekaa/BWE and 3 in Northern Lebanon/NLWE, with the aim of improving water service for vulnerable Lebanese citizens and refugee communities. Interventions varied by project and included well equipping, pumping station construction/ rehabilitation, reservoir construction/ rehabilitation, transmission line construction, solar plant, chlorinator equipping, etc. Project components are listed in Annex F. One of the projects was not completed yet at the time of the evaluation. RWE DGs and staff expressed their satisfaction with the PODAs as they helped the RWEs improve water service delivery in specific priority areas without imposing significant O&M costs on RWEs. In addition to improving water quantity, site visits and interviews along with project document review revealed that LWP tested the quality of water sources being rehabilitated and ensured chlorinators are supplied. RWEs are responsible for continuously providing chlorine gas. One issue noted in the Ghazzeh PODA during the ET’s site visit was the absence of payment by the users for the water service. While this activity is considered a success story in providing a continuous reliable source of water to a vulnerable community using a solar-powered pumping station, thus relieving community tensions over scarce water supply and saving up to $60,000 annually in fuel cost for BWE, the supplied water is considered as commercial NRW, with no incentive for the community to conserve water. It is understood that NRW was not the objective of this activity, and that this intervention was instrumental in satisfying water needs and diffusing tensions in a vulnerable community, which might outweigh the costs of NRW. Yet, this did not align with other project results, namely to “Reduce NRW” and to “Improve Water Resources Conservation/Protection in Litani Basin.” Sanitary Services LWP completed several reports to help select sanitary interventions, including “Context-Appropriate Sanitation Solutions,” “Phase I: Stakeholder Consultations and Identification of Potential Sanitation Projects,” “Phase II: Recommended Low-Cost Wastewater Interventions,” and “Assessment of Small Wastewater Treatment Plants in Lebanon: Case Study (Elard/Cabinet Merlin).” LWP also identified and obtained BWE’s buy-in on a list of 127 wastewater projects identified by the four municipal unions that signed the Charter for the Protection of the Litani River Basin. While RWEs have the authority and responsibility to provide sewage collection and treatment services, which have traditionally been provided by municipalities, all DGs confirmed that RWEs do not want to take over wastewater treatment plants (WWTPs) unless adequate staffing and funding sources are assured. Accordingly, interviews with RWE and LWP staff confirmed that the selected LWP sanitary interventions helped improve service delivery and reduced pollution entering surface waters or irrigation networks in targeted communities/municipalities, without imposing an additional O&M burden on the RWEs. These included rehabilitating five existing WWTPs and one constructed wetland (CW) with the aim of improving their treatment efficiencies while reducing their O&M costs, extending wastewater networks to link two distinct villages to an existing WWTP, and constructing a new low￾cost CW. The latter is Roum constructed wetland. It was still under construction at the time of the site visit during the evaluation. It is not clear if it will be completed before the End of Project (EOP) and there will be enough time to train designated staff from SLWE on the O&M of this wetland. Operational Costs for RWEs This new task was added through LWP task order modification 8 of May 26, 2020, it was in direct response to the financial and political crisis that has gripped Lebanon since October 2019 and been compounded by the COVID-19 pandemic. Its aim, as explained by interviewed staff from LWP and USAID was for LWP to assist the RWEs and LRA in lowering operational costs, thereby offsetting 9 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT revenue shortfalls. Accordingly, 17 projects decreasing operation costs (PDOCs) were identified and spread across all four RWEs and LRA and divided in two categories: 1. Rehabilitation of eight water facilities (variable frequency drive-VFD, soft starter, and needed rehabilitation/replacement of electromechanical equipment) and 2. Installation of nine photovoltaic (PV) solar panels in water and wastewater facilities, reducing the electricity requirements from generators and the grid. As per an interviewed LWP staff, due to time constraints, the water facilities’ rehabilitation equipment will be provided without installation. As for the PV solar panels, there installation is on-going on a very tight schedule. It is not clear for the ET if this activity will be completed before the EOP and there will be enough time to train designated staff on the O&M of the newly procured and installed equipment. RWE DGs explained to the ET that these activities are very welcome and very much needed during the current economic crisis. Selected solutions will reduce energy costs, improve equipment efficiency, and improve emergency preparedness and service continuity in targeted facilities. Furthermore, a one-year maintenance contract for the new installations will be provided, which will alleviate the financial burden of RWEs for a full year. Yet, the completion of PV panel installation is an issue. While the expected reduction in O&M costs is 25 percent, as per the approved designs, it will not be possible to measure the actual reduction at the time of the evaluation, due to the delay in project implementation. In addition to the above, and following the Beirut port blast in August 2021, LWP responded to Beirut and Mount Lebanon Water Establishment (BMLWE)’s emergency needs by procuring eight generator sets to provide water to the afflicted areas of Beirut (Figure 3) and securing a one-year maintenance contract for purchased generators to ensure proper continuous functioning in the immediate term. BMLWE DG was very thankful to USAID for this initiative. He explained that having operational generators would allow BMLWE to provide a better service through increased service hours. In addition, in the longer term, these new generators will reduce the overwhelming maintenance costs BMLWE is currently incurring to repair existing old generators, hence saving on maintenance budget. Figure 3. Generator installed by LWP in Deychouniyeh wells Irrigation Systems LWP developed the irrigation Rapid Assessment Report (RAR), identifying crops of Canal 900, calculating current irrigation usage and requirements in farmlands served by Canal 900, establishing baselines for measuring improvements in irrigation efficiency, and identifying potential areas for intervention under LWP. However, interviews with LWP staff and LRA DG revealed that the LWP had 10 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT not been able to take the actions outlined in the RAR because local communities of the Litani River Basin shut down Canal 900 in 2016 due to concerns about water quality and safety. Hence, this deliverable was replaced with the rehabilitation of the Khirbet Qanafar Wetland. This CW was put into operation in 2013. As per the site visit conducted by the ET, this CW receives polluted water directly from the Litani Basin at a rate of 3000 m3/day; therefore, treating this water will contribute to pollution reduction in the Litani basin, downstream of the wetland, and increase the awareness of local communities of the importance of wetlands as a wastewater treatment option. Rehabilitation works improved the intake to the wetland and upgraded the pumps, hence significantly reducing O&M efforts (Figure 4). Another activity was the equipping of the Kherbet Qanafar laboratory with analytical equipment to enhance LRA’s performance as a laboratory facility, thereby strengthening LRA’s support to farmers in the Bekaa and to the region as a whole, as well as improving LRA’s capabilities in monitoring industrial and domestic wastewater pollution cases. Figure 4. Rehabilitated intake from Litani River to the Khirbet Qanafar Wetland LWP supported BMLWE with the financing, technical design, and roll-out of drip irrigation systems in farmland located in the village of Faraiya and the Chabrouh dam area. According to the LWP annual progress report, the total number of farmers initially supported reached 67 and the area covered was 192,932 square meters. The use of new irrigation techniques and the installation of a new pressurized system, LWP was able to save almost 0.3 million cubic meters that were originally wasted through traditional irrigation. The interview with LWP staff and BMLWE utility coordinator revealed that following the success of the initiative, LWP received requests from at least ten new farmers for support to switch to drip irrigation and assisted BMLWE with installing the BMLWE newly purchased irrigation systems. With the exacerbation of the financial crisis and the rise in irrigation practices LWP further supported BMLWE in the design and installation of four new on-farm irrigation systems. Training and Capacity Building to Improve Hours of Service LWP has included a training component in all its subcontracts and has included formal and on-job￾training for all its activities. According to the reviewed annual progress report, LWP has trained more than 1150 persons (of which 25 percent were female) with 68 training courses and more than 254 training days. RWEs, during the interviews with the ET, find all the trainings targeting their staff to be well done and are asking for follow-up on most of the trainings. Certification training provided through Arab Countries Water Utilities Association (ACWUA) on water treatment, water distribution (levels 1 and 2), and wastewater treatment was very well received by the 11 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT RWEs, as per the conducted interviews. The original target was 100 persons. To date 195 persons have been trained (17 percent female). The certifications are valid for five years. LWP staff emphasized that participants were well selected, which showed in the training results. All trainees who took the certification training passed and were certified. LWP’s intent was to institutionalize the certificate training program in Lebanon. Because the Ministry of Energy and Water (MOEW) with Agence Française de Développement (AFD) donor support is proposing to establish a water training center, LWP has decided not to continue independent efforts to license the certification program. In addition, LWP engineers developed and delivered pumping station O&M training to a total of 135 engineers and technicians of BMLWE, SLWE, and NLWE, as per the annual progress reports. The training focused on basic and advanced concepts of pump station operation and maintenance. A leadership training was conducted for selected female staff of BMLWE and SLWE, focusing on communication skills and techniques to increase persuasion and influence. Other topics covered included communication, negotiation, leadership methods, change management, public speaking. This training was very well received, as indicated by the BMLWE DG who had asked LWP to offer it to male staff as well. In-person and on-the job training was delayed by protestors blocking roads and the COVID-19 lockdowns. RWEs’ DGs expressed their concern to the ET regarding the fact that trained contractual staff were leaving the RWEs for better-paid jobs, leaving a void in knowledge transfer, and affecting the ability of RWEs to take over some of the activities. As was evident during the site visits and through discussions with RWE staff, delays in handover of DAs and other construction and equipment installation projects will cause delays in some of the training and handholding activities. For example, in the case of WDM at the DAs, the staff are very well trained on meter reading as per the interview with the RWE DGs. However, they need more training on conducting the WDM exercise, processing the collected data, and implementing the WD Plan. • Sub-Question 1.1.2: In terms of improving service delivery and resource management capacity, are the water establishments now better equipped for managing their resources? Sub-Question 1.1.2 also directly relates to Result A, specifically the conducted assessments and the developed plans and tools to support and guide the RWEs and LRA in managing their resources. The ET examined the satisfaction of the RWEs and LRA with these plans and tools and the extent to which they are being used by these establishments to improve their capabilities in resource management. The findings of the ET are presented below by type of tool/ plan. Business/Risk Management/Master Planning The RWEs (BWE, NLWE, SLWE) prepared new business plans (2016–2020) with the support of LWP. Original plans (2013–2017) were prepared for the RWEs under previous GIZ- and USAID-funded projects. The BMLWE DG opted out of working with LWP and updated his own plan. The interviews with the RWE DGs revealed a consensus that the LWP-prepared business plans were well done. However, both DGs and LWP confirmed that the business plans in their current form were not being used by the RWEs because their limited staff and resources made implementation difficult, and because many actions were contingent on decisions that needed to be taken by the MOEW or the COM and were thus beyond the control of the RWEs. Accordingly, RWE DGs and LWP decided not to update these business plans as originally planned, but to develop risk management plans (RMPs), which define WE risks and mitigation measures. RMPs were developed for SLWE, NLWE, BWE, and LRA in close coordination with them, as explained by RWE DGs and LWP staff. They provided each RWE with a list of prioritized risks and mitigation actions to submit to donors for funding. As a follow-up, the World 12 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT Bank has selected several items to fund from the SLWE RMP. In the interview, NLWE DG told the ET he was not satisfied with the provided Excel sheets and believed more baseline info should have been added. All RWE DGs indicated that following the latest economic crisis facing the country, the pre￾identified risks and priorities have changed, as the RWEs became more concerned with their day-to-day sustainability and are unable to plan. Nevertheless, RWEs confirmed that they are trained on updating these plans and capable of doing that. LWP dropped the preparation of master plans for water and for wastewater. Several RWEs indicated they already had these master plans, so an LRA water quantity monitoring plan for all rivers in Lebanon, a BWE irrigation master plan, and WDM plans for all DAs were prepared instead. The LRA WQMP for all rivers in Lebanon was completed with the Assessment of River Gauging Stations in Lebanon and adopted by the LRA. To assist LRA in its water management, LWP conducted a comprehensive exercise to assess the status of LRA’s existing gauging stations and measurement locations along all rivers of Lebanon and provided recommendations for improvements. The collected data were uploaded into the Geographic Information System (GIS) geodatabase and LWP trained the LRA staff on its update. As indicated by LRA to the ET, this is a major achievement as it allows them to understand the current situation and capacities in monitoring rivers. It also provided an investment guide to LRA and interested donors in order to improve LRA’s monitoring capacities across the rivers of Lebanon and was adopted by the 2020 National Water Sector Strategy (NWSS). Furthermore, the LRA built on the geodatabase established by the LWP to link to it the collected flow monitoring data. The BWE DG expressed his satisfaction with the irrigation master plan, which provided a comprehensive geodatabase of the Bekaa irrigation schemes and identified capital investments and institutional improvements needed to assist BWE with improving its management of the irrigation sector. However, he commented that the appropriate BWE staff to support the irrigation study was limited. Currently, it is put on hold due to current the economic crisis and the shift in focus of BWE, as all RWEs, on maintaining potable water service delivery. However, he acknowledged that it is there when needed. The WDM Plans were discussed above on NRW. These were submitted towards the end of the project due to delays in the handover of the DAs and in the completion of the demand management exercise. Although on the job training was conducted throughout the exercise, more training and follow-up on these plans are needed for their adoption and implementation by the RWEs. Enterprise Resource Planning (ERP) and Costing and Operational Analyses The Microsoft Dynamics-NAV ERP software platform is a flagship activity for USAID with the water establishments, designed for midsize organizations to give managers access to information that enables them to monitor unit operations and track financial and operating performance. ERP was implemented under the previous USAID project LWWSS at BMLWE, BWE, SLWE, and LRA by Engineering Design & Manufacturing (EDM), which provided design, implementation, and technical support for one year. The modules implemented were based on the needs and requirements of each RWE according to its business process. LRA did not renew its license. The other three RWEs did. For LWP, the bulk of the scope was to implement the ERP at the NLWE, where this system was not available, and to provide general technical support for users at BWE and SLWE to ensure the proper operation of the ERP. According to the LWP staff interviews, several new features have been added at the request of the various DGs even though they were outside the project scope. The interviews conducted with the RWEs’ DGs and financial departments’ staff revealed that the ERP was being used to different degrees at each RWE. ERP is being fully used in SLWE with 65 adequately trained users. LWP worked on providing several system enhancements and features, based on requests by the DG. LWP also procured the NAV development license for SLWE to be able to take over the support and maintenance in-house as requested by the DG. The Cost Center feature was still ongoing 13 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT at the time of the evaluation. The interview with the SLWE finance department revealed that the SLWE are having problems in support and maintenance and in adding new features as they do not have the in￾house capability and contracting an external party is expensive. While the evaluation team noted the presence of an IT/Development consultant at the SLWE, interviews with the SLWE DG and staff revealed that the IT staff is not capable of doing this inhouse as this requires specialized expertise as well as legal and accounting knowledge. There is a need for transferring the ERP programming knowledge and experience to the SLWE inhouse IT staff, particularly that two of the previously ERP experienced staff left the establishment. Note that this issue faced by SLWE with the ERP was mainly caused by the recent economic crisis and is beyond LWP control. At the BWE, as per the project annual progress reports, the LWP redesigned the billing and collection module and upgraded the accounting module. LWP supported the generation of the 2017 financial statements directly from the ERP system. The LWP then implemented the ERP upgrade to the NAV 2018 version by June 2019. LWP provided continuous on-the-job training, bug fixing, module efficiency testing, financial data reconciliation for quality assurance, and report generation and verification. However, following the migration of the data to the new version, many problems appeared, and according to KIIs, to date, financial reports are not error-free. Furthermore, the users need more training on the accounting module. As LWP is reaching its end and phasing out, the ERP development and support to the water establishment was picked up by another USAID-funded activity, the Community Support Program (CSP) project implemented by Chemonics, which contracted the same consultant previously contracted by DAI to support the BWE. At NLWE, the ERP was installed by LWP in March 2018. The political instability in the country starting October 2019, along with the roadblocks and Covid-19 pandemic caused delays in task implementation, particularly those requiring physical presence. As per the interviews with the DG and the finance department, and confirmed by LWP KIIs, the ERP implementation could not be finalized by the LWP, particularly the “Billing and Collection Module,” which was completed in two out of nine branches, namely Tripoli and Bcharre, covering more than half of the NLWE subscribers. The accounting module also could not be fully implemented and tested before the completion of the work in the branches. Thus, the NLWE did not stop working on their existing PIMS and X7 software. They are currently running all three systems together. This generated dissatisfaction at the level of the DG and the Finance Department and negatively influenced their viewpoint of other management tools and plans. As per the KII the completion of the accounting module and billing and collection within the branches will be completed by the LWP ERP consultants, now under the CSP contract. At BMLWE, the finance staff is satisfied with the ERP system and is well trained on using it. BMLWE is in touch a company specialized in ERP maintenance and support. LWP successfully linked the Asset Valuation to the ERP. The interviewed staff indicated that the LWP renewed the ERP license for the BMLWE. LWP had a dedicated team of two persons to install, customize, deploy, upgrade, train, and maintain the ERP software. For LWP, the bulk of the scope was to implement the ERP at the NLWE, where this system was not available, and as per contract to assist each RWE to complete the installation and the use of ERP, according to KIIs this was supposed to be done in coordination with subcontractors hired by the RWEs. Despite the team’s dedication and effort, which was acknowledged by most RWEs, it was not able to complete the implementation of the ERP activity at NLWE, while relying on its current limited inhouse resources and having to accommodate for continuing requests from the RWEs for support in modules that were considered completed. According to the LWP staff interview, outsourcing this activity could have achieved better results in terms of providing more human resources. RWEs acknowledged during their interviews with the ET that they were unable to support ERP on their own due to financial and staff limitations, and that there was not enough expertise in the country that could be procured to support and maintain the system. 14 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT RWEs are requesting more training and support and the building of a maintenance and support inhouse team. LWP was proposing an ERP support unit jointly funded by all the RWEs after EOP. However, this was put on hold due to the current economic crisis. In addition to the ERP, the LWP undertook several Information and Communication Technology (ICT) activities to help improve the ICT environment at RWEs. These were guided by an ICT assessment conducted for BWE, SLWE, and NLWE. LWP conducted an ICT assessment for LRA’s headquarters and branches. As per the interview with the LRA DG and utility coordinator, this assessment served as the basis for the planned IT infrastructure upgrades for LRA, funded by UNICEF, and is part of LWP’s continued coordination with UNICEF on IT activities. Now, a new ERP, based on Oracle, will be installed at the LRA through an EU fund. LWP completed NLWE’s governance manual in Y5-Q1 and continued supporting NLWE throughout Y5 in implementing the governance guides because they are crucial for establishing and institutionalizing efficiency at NLWE and helping it meet its goals. These have been developed in accordance with the Organization for Economic Cooperation and Development (OECD) guidelines. Asset Valuation Asset valuation provided the RWEs with asset information and helped improve asset management and institutionalize asset documentation and valuation in the RWE decision making process. Field mapping of all fixed assets and asset valuation reports were prepared for NLWE and SLWE and converted into GIS databases. At BWE the asset documentation prepared in 2013 under another project was updated through desktop review. All asset valuation reports were integrated into the ERP. On-job-training on GIS and asset valuation was provided. A standardized asset management manual was prepared for all RWEs. SLWE and NLWE DGs, during their interview with the ET, both indicated having some outstanding comments on (i) the asset data collected by an LWP subcontractor and (ii) valuation assumptions. Nevertheless, both reported being able to generate financial statements reflecting depreciation values. As for BWE, and as per the interview with the finance department, they updated internally the asset list generated by LWP but did not integrate it into the ERP. The BWE was not able to generate the 2019 annual fixed asset and depreciation report following ERP data migration. For NLWE, the 2019 annual fixed asset report was generated, yet the NLWE finance department reported not being trained on updating the asset list. It was noted by the ET that the benefits of this activity are better appreciated at the RWEs which are being able to use it successfully within the ERP to generate the annual fixed asset and depreciation report. Financial and Performance Audits According to the projects annual progress reports, financial and performance audits were done for the utilities that have the staff in place, starting with NLWE, SLWE, and later adding BMLWE. LWP prepared preliminary audit reports, three-year strategic audit plans, and the annual audit plan for 2018 for NLWE and SLWE and for 2019 for BMLWE. A financial audit and a performance review were conducted for SLWE for 2015, 2016, and 2017. LWP also completed SLWE’s final 2018 audit report and submitted it to the DG. LWP assisted the NLWE finance team with closing its financial statements from 2005 to 2018 and making necessary adjustments. The completion of the annual financial and audit reports for the RWEs is a significant achievement requiring commendable effort and commitment by both the LWP and the RWEs, and has established the grounds for proper financial and accounting books. LWP also conducted a review and audit of BMLWE’s and BWE’s main financial statements and accounts and presented them with a report that identified gaps and weaknesses in their workflow and procedures, suggesting a list of corrective measures to be adopted. All RWEs were recently provided with a standardized Audit Guide. According to the interview with their finance departments, SLWE and BMLWE reported being very happy with the outcome of this activity and very satisfied with the training 15 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT and capacity building received throughout the project. At NLWE, the finance department indicated that this activity was not completed. The 2019 audit report was not generated yet. A general training was done but was not followed up on. LWP developed a financial model for LRA. The financial model consists of a projection of LRA's revenues and expenses, derived from calculating the water balance for all LRA’s water sources, broken down by profit and cost centers. This information allows LRA to assess the profitability and impact of new projects and operational decisions on its sustainability. As such, the financial model is key for optimizing LRA's operation and improving its financial planning and decision-making process. The interviews with LRA staff revealed that this model was very well received by the LRA DG and that LRA staff were well trained and capable of utilizing it. The LRA plans to link this model to its ERP once developed. GIS LWP has met its GIS deliverables. It conducted a GIS assessment for three RWEs (BWE, NLWE, SLWE) and invested in the purchase, licensing, and configuration of GIS software and servers for the same three RWEs. The LWP server equipment and GIS licensing will serve as the basis for forming the EU-funded GIS units for the three RWEs. Asset data were integrated into the GIS systems and on-the-job-training was provided to SLWE and NLWE. For LRA, LWP gave first-level GIS training, helped install a gauging station GIS, prepared a water quality monitoring station map, and conducted an IT assessment. Improving Billing and Collection LWP introduced handheld bill collection devices and portable printers at BWE, NLWE, and SLWE and trained RWE staff on their use. LWP completed the installation of the device software and synchronization of billing data with the ERP billing and collection in SLWE and BWE. Once operational, these devices will feed collected data directly into the RWE ERP systems avoiding human error and strengthening financial management, transparency, and sustainability. Interviews with BWE staff revealed that BWE is not using these devices yet, while interviews with SLWE staff revealed that SLWE is using them for other purposes such as warehouse management. The approval by the Ministry of Finance to use them in bill collection was issued in 2021. LWP assisted the RWEs in establishing partnership agreements with agencies that offer bill payment services to support this task. BWE signed a partnership with a payment service provider in October 2016, followed by SLWE. By the end of June 2019, SLWE customers had started paying their bills at this firm’s counters. Replicating this initiative, SLWE signed a similar agreement, in October 2019, with another money transfer firm to offer SLWE subscribers a wider network totaling more than 1,500 points of sale nationwide where they can pay their bills. LWP supported this SLWE initiative by integrating the payment firm’s system into the ERP system and the Finance Department. With the combined points of sale of these two payment service providers, the establishment will be facilitating payment of subscriptions, thus improving collection rates, reducing overdue bills, and improving cash flow. As for NLWE, the contracts are ready, but NLWE’s finance department indicated to the ET that they wanted direct communication of collected bills from the two firms to NLWE’s headquarters before launching it, which was not part of the LWP scope. This activity proved most successful with the SLWE. Collection rates via the first payment firm at BWE are still very low, only 40 subscribers in 2020. The actual impact of these initiative to improve billing and collection is difficult to assess given the current economic crisis and the abstention of subscribers from paying their bills. WQAP 16 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT WQAPs are an integral part of the Water Security Analysis (WSA) developed by LWP in collaboration with the American University of Beirut (AUB) for NLWE, SLWE, BMLWE, and BWE with LRA. The WSA provides a tool for RWEs to use in addressing water threats and future actions/mitigating measures with government and municipal officials, and other stakeholders. The WSA will help guide the establishments better manage their resources, taking the five dimensions of water security, namely (i) domestic, (ii) environment, (iii) economic, (iv) risk management, and (v) water infrastructure, with governance being a cross-cutting dimension. LWP staff indicated that a simple tool was further developed to help RWEs update their WSA score. The establishments confirmed that they were trained on using this tool and are capable of applying it. Following the positive appreciation by MOEW, LWP extended the initial scope of AUB’s activities to develop a National WSA, which was completed and presented to the MOEW. WQAPs for all four RWEs were completed in collaboration with AUB to improve processes and procedures for enhanced water quality testing. LWP also provided hands-on laboratory training to the staff of all four RWEs. Through this training, the RWEs have come to understand the importance of applying standard operating procedures in water quality testing and the impact of proper water testing from certified laboratories. According to the LWP staff, the Ministry of Energy and Water requested copies of the LWP WQAPs to be used as supporting data for the development of the first water testing plan for Lebanon and for the refurbishment of the RWEs laboratories. The WQAP process led by LWP facilitated two service contracts, one between BMLWE and AUB and one between SLWE and AUB, for the university to provide water quality testing and analysis at their laboratory. The agreement between AUB and BMLWE was signed in mid-2019. As for SLWE, this was put on hold due to the financial crisis and the establishments’ budget shortfall. The WQAP also led to the upgrading of LRA’s Khirbet Qanafar Laboratory facility (Figure 5) through the procurement of much needed equipment as part of LWP. BWE and NLWE did not follow up on the use of the WQAPs. More training and funding in the future are needed to promote their use. Figure 5. Upgraded Khirbet Qanafar Laboratory Miscellaneous 17 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT LWP has utility coordinators assigned and sitting at each RWE to help monitor, implement and report the project’s activities. LRA has a utility coordinator assigned and visiting LRA two to three times a week. All RWEs/LRA DGs and staff are working well with the coordinators. RWEs are currently suffering from the economic crisis in the country and the devaluation of the Lebanese Pounds. Both their revenue and purchasing power decreased significantly. They are having difficulty in purchasing equipment and spare parts, as well as consumables (chlorine gas, lab reagents, oil, etc.) in USD. Their current priority is to perform day-to-day operation and maintenance of utilities to ensure service delivery. UNICEF is supporting the WEs by funding equipment maintenance and provision of some consumables such as chlorine gas and laboratory reagents. SUB-QUESTION 1.1 CONCLUSIONS 1. Overall, LWP’s activities, management plans, tools, and trainings have improved the effectiveness of the WEs in water delivery and have rendered them better equipped for managing their resources. 2. The degree of impact varies between RWEs, as the scale and nature of the interventions varied between RWEs, depending on their needs and their commitment. 3. LWP had a very effective quality assurance/quality control (QA/QC) program that resulted in good designs and good construction but came with a cost in time. LWP had faced delays in design and implementation of construction projects before the October 2019 Revolution, the economic crisis, COVID-19 lockdowns, and Beirut Port blast. The latter further exacerbated these delays. The extension period was not sufficient for the timely completion of all activities. 4. Delays in completion of construction activities resulted in delays in handover, risking the effectiveness of the transfer of projects to RWEs and their ability to adequately identify pending problems and issues, and successfully take over the operation in a sustainable manner. It should be noted that since October 2019, LWP struggled with some RWEs to get them to properly support in a timely manner, the hand over procedures of construction projects. 5. The one-year liabilities period will give time for the RWEs to ensure the quality of the delivered products. LWP did work with the RWEs and their subcontractors to ensure that the performance bonds or other protective measures were put in place to ensure the RWEs had the full one-year defects liability period for all projects turned over to them. Service Delivery: 6. The closed loop approach in the DAs was very well received by the RWEs, who are eager to replicate it in other areas. This activity helped RWEs improve service delivery in these areas on various fronts. It resulted in the reduction of physical NRW through the identification and mitigation of technical losses. It also allowed for identifying some commercial losses and led to an increase in subscriptions. The calculated NRW reduction is mostly physical in nature and while it considered the subscriptions, did not account for commercial billing, in terms of actual payments. 7. The promotion of smart meters and their installation in the DAs, along with the update of the customer registers and the training of the staff on meter reading set the stage for the implementation of consumption-based tariffs by RWEs. BMLWE and North Lebanon Water Establishment (NLWE) already apply consumption-based tariffs in other areas and can swiftly start applying them in the DAs. Until then, subscribers may be consuming more water as consumption is neither controlled physically (as it used to be with the orifice regulators) nor financially through imposing consumption-based tariffs. The fact that the water is not yet supplied 24/7 may help limit overconsumption. 18 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT 8. Delays in handover of the DAs resulted in some pending hardware and software issues related to data collection, transfer, and analysis, particular with regards to water meters and staff does not feel confident to take over. Trained staff leaving the establishments further exacerbated this issue. 9. Delays in some handovers also led to delays in the planned WDM activities, which are still being finalized. Specifically, for BMLWE, which was the last DA to be handed over, this will limit the extent of follow-up, on-the-job training, and handholding in WDM. While LWP faced many challenges, these can be learned from as they are real challenges faced by RWEs. 10. RWEs were very satisfied with PODAs, which improved water services for vulnerable Lebanese citizens and refugee communities without causing additional O&M costs for the RWEs. It was noted in one of the PODAs, that water supply was increased but the RWE was not able to ensure billing, which contributed to an increase in NRW. It is understood that NRW was not the objective of this activity, and that this intervention was a success story in satisfying water needs and diffusing tensions in a vulnerable community, which might outweigh the costs of NRW. Yet, this did not align with other project results. 11. The installation of drip irrigation in Chabrouh was a very successful project. 12. Rehabilitating existing and functional WWTPs improved wastewater treatment efficiency and reduced O&M costs, without introducing a new burden on WEs in operating these plants. 13. The training provided through LWP was well structured and well received. In many instances DGs and staff were asking for additional & follow-up training. The certification training and the leadership training received high praise. 14. For some activities like WDM, although on the job training was conducted throughout the exercise, further training and hand holding that goes beyond water meter reading to water balance calculations and data analysis, and WDM plan implementation is needed. 15. Water quality improved with the improvement in infrastructure, including the installation of new chlorinators. However, RWEs are faced with the challenge of providing chlorine gas given the current economic crisis. UNICEF is helping in this respect. 16. PDOCs are a timely initiative that will help decrease operating costs on RWEs, and promote sustainability of service delivery, which is a major challenge under the current conditions. Their exact impact cannot be calculated, as this activity is still ongoing. Resource Management: 17. Several plans and management tools have been prepared by LWP, previous USAID projects, and other donors; however, they sit idly on the shelf. The main reason given is the lack of capable staff. Several plans developed by LWP have seen follow-up activities. 18. The business plans, although considered to be well done, were not implemented, neither by the departing nor by incumbent DGs, because of limited RWE staff and resources, and because many actions were contingent on decisions that needed to be taken by the MOEW or the COM and were thus beyond the control of the RWEs. 19. The decision to introduce RMPs was a good one, providing a more focused list of priorities for national and international funding. RMPs need to be reshuffled following the economic crisis to reflect the new priorities. 20. The Assessment of the River Gauging Stations in Lebanon and the WQMP were adopted by LRA and endorsed by the MOEW. The recommendations were included in the 2020 NWSS to secure funds to improve river flow monitoring across Lebanon. 19 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT 21. The BWE Irrigation Master Plan was an important tool, as irrigation schemes were assessed and georeferenced for the first time, as acknowledged by the DG. It will help guide donors in funding future projects. There are currently no appropriate staff to follow up on it. 22. The WDM plans require training and follow-up by dedicated staff so they can be put into good use within the DAs and in other areas. 23. WQAPs have seen some follow-up activities in three of the five establishments in terms of procuring laboratory testing equipment and sub-contracting water testing. However, there is more to these plans that could be enforced if followed up on which requires government commitment and funding to achieve the ultimate objectives. 24. Enterprise Resource Planning (ERP) as a concept is excellent and has a lot of potential. However, for it to be appreciated by users, it has to be completed and the users should be able to successfully generate all their annual reports from it. For LWP, the bulk of the scope was to implement the ERP at the NLWE, where this system was not available, and to provide high level technical support to ERP support and maintenance subcontractors hired by the other RWEs. However, the new SLWE and BWE DGs appointed in 2018 did not renew the ERP maintenance and support contracts. 25. LWP was not able to complete the implementation of the ERP activity at NLWE, while relying on its current limited inhouse resources and having to accommodate for continuing requests from the RWEs for support in modules that were considered completed. The overall allocated LOE was not sufficient to install a new ERP system for the NLWE and to handle the incremental ERP support requests of the other RWEs, especially the BWE, where software upgrade and data migration were required. Resistance of staff in some RWEs and their turnover following the 2019 posed additional challenges to the LWP ERP team. 26. Adequate management of ERP activities through prioritizing activities towards the completion of the NLWE ERP system and/or procuring additional support could have helped complete this activity. Subcontracting this activity to a third party may have yielded better results, both in terms of tapping into more human resources to complete the project and in terms of ensuring post-LWP sustainability, as ERP support and maintenance knowhow would not be restricted to two inhouse consults. 27. Where the ERP system is complete and functioning smoothly, there is a need for continuous support and maintenance of the system. If the RWEs want to contract out the ERP support and maintenance, they have a limited choice due to limited availability of expertise in Navision in the local market. This jeopardizes the sustainability of the system. 28. Continuous training on the ERP and other systems is needed as already trained staff are leaving the establishments due to the current crisis. 29. The ICT assessments and subsequent upgrades were very well received and helped improve the ICT environment at the RWEs for current and future projects. 30. The asset valuation and the financial and audit performance audit were very beneficial, particularly in RWEs where the ERP system was functioning properly and where annual reports could be generated smoothly. In RWEs there were still issues with the ERP, and these activities were considered as incomplete. The RWEs are building on the asset valuation activity and completing their assets lists. 31. Improved billing and collection were successful in SLWE through the partnership agreements with agencies offering bill payment services and the use of the hand-held devices. In BWE, the partnership agreement resulted in an insignificant increase in paid bills and the hand-held devices were not used, pending approval from the Ministry of Finance. In NLWE, the partnership 20 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT agreements were not signed yet, although they were prepared, and the handheld devices are not in use yet. The approval by the Ministry of Finance to use them in bill collection was issued in 2021. SUB-QUESTION 1.2 FINDINGS • Sub-Question 1.2: In terms of improving the civic engagement in water management (Result B), the evaluation is to answer whether: 1.2.1 The perception and satisfaction of citizens towards water establishments have improved To advance citizens’ perception of the water utilities’ role in providing a necessary and valuable service, LWP conducted many activities and trainings throughout its lifetime. These included website creation, the production of a total of 10 corporate and awareness raising 2D videos, and one long flagship 3D video that went live on the RWEs’ social media platforms. According to LWP staff, the 3D video, along with the teachers’ guide that LWP developed, proved to be an excellent awareness tool for school children and their parents. Refer to Annex F for more details. LWP assisted the communication staff at BMLWE, BWE, NLWE, and SLWE through enhanced collaboration and the development of an informational pamphlet/booklet on water meters that was distributed in the DAs. LWP assisted NLWE and SLWE with their Facebook campaigns on maintenance, water quality, and hotlines availability. LWP boosted Facebook posts for SLWE and NLWE. In addition, LWP offered training on communications and customer service to more than 60 RWE staff to enhance customer relations skills and rehabilitated the Jezzine Customer Service Center (CSC) within the Jezzine DA. LWP also prepared several media campaigns. A Short Message Service (SMS) text campaign aimed to remind water establishment subscribers to pay their bills (at BWE, SLWE and NLWE). In the Bekaa, this was coupled with banners hung in municipalities, a radio campaign, and billboards promoting bill payments and water conservation along all the main Bekaa roads. The impact of the above-mentioned activities could only be assessed through the results of the baseline and end-line Citizen Perception and Satisfaction Surveys (CPSSs) carried out in 2016 and 2020. The latter survey covered LWP’s DAs, namely Koura, Metn, Jezzine, and Zahle, with a total of 1,600 respondents (49.9% female, 50.1% male) including both subscribers and non-subscribers, equally distributed among the DAs. Overall, 74% of respondents said they were satisfied with water services, compared to 71% in 2016, and 59% said they had submitted claims or requests for repairs to the RWEs, compared to only 28% in 2016, with 81.5% reporting satisfaction with the responses and actions taken concerning their claims (Figure 6). More specifically, the number of subscribers that are very satisfied from the RWE service increased by 17% points between 2016 and 2021 to reach 44% and the number of subscribers trusting RWEs to manage the water services in Lebanon increased by 19% points to reach 23%. Finally, 56% of the respondents noticed improvements in their RWE (infrastructure, services, customer services) over the past two years. The end-line CPSS further analyzed the satisfaction question results by DA and revealed that Metn residents were the least satisfied with the water services provided, while Jezzine residents were the most satisfied, followed equally by Zahle and Koura residents. The lowest satisfaction rate with the responses and actions taken by the water service providers concerning requests to fix water leaks, repairs, and problems was recorded in Metn (60%), and the highest satisfaction rate was scored in Jezzine (95%). Note that the end line survey did not test for the statistical significance of the results to ensure that the reported changes were not due to chance. 21 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT FIGURE 6. RESPONDENT SATISFACTION AS PER BASELINE AND ENDLINE CPSS Subscribers trust very much the RWEs to manage the water service Subscribers very satisfied from the RWE service Respondents reporting claims to RWEs Respondents satisfied with water services 71 28 27 4.1 74 59 44 22.8 2020 2016 0 20 40 60 80 100 Respondent Satisfaction as per Baseline and Endline CPSS 1.2.2 the Lebanese citizens are better aware and proactive towards the importance of water management and conservation. LWP made efforts on multiple fronts to enhance citizen awareness and proactiveness on water management and water conservation. In addition to the videos and media campaigns mentioned above that also targeted water conservation, several initiatives were launched in participation with Civil Society Organization (CSOs), which contributed to awareness raising and to improve civic engagement in water management and advance citizen perception of the water utilities. LWP established a diverse and wide￾ranging innovation and research grants program, covering communities, agriculture, and industry. CSO grantees received $1.2 million to increase civil society’s engagement in improving water use behavior. The projects addressed households, farmers, industries, students, schools, corporate companies, wineries, and communities, and intervened in different areas of Lebanon. The list of grantees and their activities are presented in Annex F. LWP concentrated its efforts on the Litani River and engaged media personalities to increase visibility and publicity around the Litani protection and conservation efforts. They also organized a workshop to mobilize the greatest number of municipalities in developing a Charter for the Protection of the Litani River Basin. This initiative has engaged 28 municipalities, 4 unions of municipalities and the BWE. Interviews with the BWE and LRA revealed that the BWE DG recognized the importance of such an initiative in the region, while the LRA DG did not believe in the potential impact of the Charter initiative and considered that the signatory municipalities were not taking the necessary actions they committed to in the Charter to protect the Litani River. Finally, LWP organized a clean-up day on the Litani River with the participation of the surrounding communities to raise awareness on the importance of the protection of the river. In addition, and in support to investments in infrastructure works in the DAs, LWP organized several town hall meetings in all DAs to engage active community members from different backgrounds in 22 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT discussions related to citizens’ responsibility towards water as a natural resource and their duties. They have also participated in summer festivals in Jezzine DA and water festivals held in collaboration with Lebanon Relief Council (LRC) in NLWE. The impact of the above activities could only be assessed via the end line CPSS. The survey results showed that 49% of respondents who saw/heard awareness campaigns to save water had taken related steps at their households and report mainly “to turn off the faucet when not needed” (38%) or “fix faucet malfunctions immediately to avoid wasting water” (12%). More female respondents (55%) than male (41.3%) reported having taken steps to save water in their households. It is worth noting that the percentage of respondents remembering awareness campaigns remained almost the same between 2016 (12%) and 2020 (16%). Even though the campaigns conducted by LWP were reported to reach 1 million Lebanese citizens, few reported remembering them. Finally, the percentage of the number of respondents noticing any increase in NGOs implementing water saving initiatives over the past two years remained low and almost the same between 2016 (18%) and 2020 (14%). More males than females (18.7% vs. 9.6%) reported an increase in the number of NGOs implementing water saving initiatives, although both scored low rates. Note that the CPSS was conducted in DAs only, while NGOs worked both within and outside DAs. Again, the end line survey did not test for the statistical significance of the results to ensure that the reported changes were not due to chance. SUB-QUESTION 1.2 CONCLUSIONS LWP’s civic engagement work and communication efforts, combined with institutional and infrastructural improvements, contributed towards enhancing the image of the RWEs and bridging the gap between them and their customers, as was evident in the end line CPSS, where the number of subscribers that were very satisfied with the RWEs’ service increased by 17% points between 2016 and 2021 to reach 44%, and the number of subscribers trusting RWEs to manage the water services in Lebanon increased by 19% points to reach 23%. When considering all respondents, and not just subscribers, despite the general dissatisfaction with governmental institutions primarily caused by the recent developments in the country, the level of satisfaction in the services provided by the RWEs in the four DAs slightly increased from 71% in the baseline survey to 74% in 2020. Yet, statistical significance testing of CPSS results was not conducted and would have provided more robust evidence. The civic engagement projects targeted different type of beneficiaries, including households, farmers, industries, students, schools, corporate firms, wineries, and communities. Activities were small but when added together they provide a nice umbrella effect for water conservation and efficiency. Yet, despite the extensive awareness campaigns conducted as part of LWP, the percentage of respondents remembering them and the number of respondents noticing NGOs working in the water sector remained almost the same between 2016 and 2020. This may be partially attributed to the fact that NGOs were working both within and outside DAs, while the CPSS was conducted only in DAs. SUB-QUESTION 1.3 FINDINGS • Sub-Question 1.3 In terms of improving the private sector involvement in water management (Result C), the evaluation is to answer whether: 1.3.1 The targeted private sector is more involved and engaged in water management and conservation initiatives. LWP engaged the private sector in water management and conservation through developing different forms of PSPs to (1) improve water service delivery, (2) leverage the funding of innovative water conservation technologies, and (3) improve water usage awareness. LWP secured PSP contracts between BWE and SLWE and two payment service providers to facilitate the payment process to customers, thus improving collection rates, reducing overdue bills, improving cash flow, and reducing 23 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT fraud through increased transparency This PSP was more successful with SLWE than it was with BWE, and it did not materialize with NLWE, as described earlier under sub-question 1.1.2. LWP created a strategic model performance-based framework for the RWEs to support PPP interventions. The model performance-based framework contract for PPPs was viewed positively by DGs during their interviews with the ET. However, the ongoing economic and political situation created a sizeable obstacle to PPPs because of the current inability of the RWEs and some private sector firms to engage in financial borrowing or loan activities. Yet, the DGs confirmed that this tool could be readily used by RWEs when needed and would help improve performance of the private sector in PPP contracts. The developed performance-based contract models with detailed scopes of work and tender documents for BWE, NLWE, and SLWE are as follows: • BWE: a six-year Performance-Based Contract (PBC) for the O&M of three WWTPs. • NLWE: a six-year PBC with a private sector operator to manage NLWE’s daily operations, maintain and develop Tripoli’s water infrastructure, hold communication campaigns to raise awareness on water citizenship and responsible water use, and transfer its knowledge to NLWE’s staff through training and other methods. The NLWE DG confirmed to the ET that the NLWE Board of Directors officially approved the Tripoli PPP initiative and supporting documents with European Bank for Reconstruction and Development (EBRD) financing through private equity loans and submitted it to MOEW for approval. However, this is currently put on hold given the current economic crisis. • SLWE: LWP worked closely with SLWE to prepare an Expression of Interest (EOI) for private companies specializing in solar energy in an effort to reduce SLWE’s energy costs. SLWE has put this initiative on hold because of the current economic crisis and the COVID-19 pandemic. • BMLWE: LWP assisted BMLWE in conducting a bid for a complete feasibility study for the Hrach Hydropower Plant. An entity was selected to conduct the study which will provide necessary information on the technical, financial, and economic return of the project, especially in terms of offsetting the energy costs of BMLWE’s nearby pumping station. According to BMLWE advisor, having a clear financial picture on this endeavor will increase BMLWE’s chances on venturing into a PPP with a private sector entity. The interviews with RWE DGs revealed that RWEs appreciated the effort put by the LWP into the PBCs and had learned a lot from this exercise. However, the completion of the above PPP efforts is hindered until Lebanon achieves greater economic stability. To leverage the funding of water conservation technologies, the LWP grants program worked with private sector entities from industry (stone-cutting factories) and technology (pump station predictive maintenance). LWP partnered with the ALI to reach out to industries to educate them on possible technologies for water saving and incentivize them to adopt them. Nine stone cutting industries under ALI’s Water Efficiency in Lebanese Industry (WELI) program were selected for partnership with LWP. Results showed a decrease of 67% in water consumption in the stone/rock-cutting plants that had adopted water conservation and cleaner production processes. The industries contacted and visited by the ET expressed their satisfaction with the installed systems for various reasons: (1) through water conservation, the need for pumping well water was minimized, which helped reduce their energy bill and their use of corrosive saline water (that results from salt-water intrusion in coastal areas), (2) the recycled water has better quality (low turbidity and suspended solids), (3) the volume of sludge produced was reduced and could be safely disposed of in controlled dump sites, and (4) the system is easy to maintain, thus ensuring its sustainability. Under the predictive maintenance grant for Water Conservation awarded to Engineer and Educate (E2) firm, LWP and the E2 engineers completed the installation of all the predictive maintenance software, 24 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT hardware, and equipment at the Jezzine Pump Station. However, the field visit revealed issues with the functioning of the predictive maintenance panel following the replacement of the pumps in the station. To improve water usage awareness, LWP established water stewardship (water conservation, reuse, and treatment) cooperation commitments with companies in the media, banking, manufacturing, and wine industries. This activity was well received by the private sector during the first half of the life of the project (pre-crisis) who innovated in their awareness campaigns by issuing dedicated newsletters, providing financial initiatives, covering water-related events, etc. 1.3.2 The farmers and industries are more involved in water conservation. LWP worked on improving the private sector’s involvement in water management and conservation with a focus on farmers and industries. With farmers, the project adopted various mechanisms to reduce wasteful irrigation water use through farmer outreach and the implementation of projects with tangible results. The Incentive Rebate Program (IRP) allowed 10 farmers to cost-share the installed irrigation equipment and benefit from the technical support of agricultural suppliers. An interview with one of the biggest carrot growers in the Bekaa revealed the successful installation of a drip irrigation network on 20 Ha of land, which led to 60% water saving on average compared to previous years. As part of the IRP, five farmer field days were organized and attended by more than 100 farmers with the objective of encouraging farmers to switch to water-saving irrigation technologies and to engage with the private sector. Partner agriculture suppliers showcased the outcomes of experiments to introduce drip irrigation, mini-sprinklers, hydroponics, Nutrient Film Technique, AutoPot Watering System, etc. to growers of different produce types. However, one of IRP’s beneficiary farmers showcasing his water-saving technique revealed to the ET that without financial support, other farmers could not adopt these technologies on their own, even if they wanted to. Additionally, a grant was given to CCIAZ which supported two major farmers and one cooperative in applying water saving techniques. A greenhouse farmer, visited by the ET, cost-shared an energy efficiency system with LWP and CCIAZ, where a PV panel was installed on his farm to power his two wells; in addition, he received four tensiometers to monitor the water content of the soil (Figure 7). He reported a 50% reduction in water consumption, after he was able to schedule his irrigation rounds every four days (compared to two days in the past) and shorten his irrigation time from 2 to 1.25 hours. With the water saved, he was able to irrigate an additional 2,000 m2 of land. In addition to reducing water consumption, he benefitted from a decreased workload on him and his family, less stress related to when he is able to irrigate (since he has the PV), and reduced pesticide application since the leaves of tomato plants became drier and thicker and hence less likely to be infested. 25 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT Figure 7. A farmer beneficiary in the Bekaa: “While reducing my water bill, I have also reduced my pesticide application bill.” Furthermore, LWP and CCIAZ teamed up with a local grape cooperative that produces wine. According to the interviewed beneficiary farmer members of the cooperative, before LWP, irrigation was a lengthy process that required much effort and resources (the farmers were oftentimes not able to irrigate their vineyards). With the introduction of drip irrigation connected to an irrigation pond established by a previous USAID-funded project, the region has seen a 70% increase in irrigated lands. Not only water consumption has decreased, but also other collateral benefits were observed including decreased farmer workload, increased grape production, and improved wine quality. Figure 8. The winemaker of the cooperative: “Since the introduction of drip irrigation systems, the quality of the wine produced has improved.” Finally, the Chabrouh project, which was discussed earlier under sub question 1.1.2, convinced more than 100 farmers to shift to water-saving irrigation materials. A new irrigation schedule for the farmers was set up in collaboration with the municipality. At the industrial level, the nine stone-cutting industries under ALI’s WELI program successfully conserved water as previously discussed under sub question 1.3.1. 26 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT SUB-QUESTION 1.3 CONCLUSIONS 1. LWP promoted the concept of water stewardship, namely water conservation, reuse, and treatment with key players from the private sector. The private sector has leveraged up on this and innovated in its awareness campaigns by issuing dedicated newsletters, providing financial initiatives, covering water-related events, etc. 2. PPPs with the two payment service providers provided a better payment alternative to water customers. However, BWE customers have not accepted widely this new tool, as people need more time to accept to shift from traditional tariff collectors to service providers. 3. The model performance-based framework contract for PPPs was viewed positively by DGs. However, the ongoing economic and political situation created a sizeable obstacle to PPPs because of the current inability of the RWEs and some private sector firms to engage in financial borrowing or loan activities. Yet, these contracts can be readily used by RWEs when needed and will help improve the performance of the private sector in PPP contracts. 4. Farmers and industries that received support from the project have successfully adopted the water conservation methods. However, their role as forerunners and as examples for their neighbors and fellow colleagues to follow is constrained by the current economic crisis. 5. Dealing with cooperative of farmers, municipalities, and professional associations have proven efficient and allowed for good outreach. Hence, promoting new technologies through associations/cooperatives/municipalities can have a wider outreach as compared to targeting individual farmers or industries. QUESTION 2 DID LWP ACHIEVE THE PERFORMANCE STANDARDS (PS) RELEVANT TO EACH OF THE RESULTS A, B AND C? The ET verified the performance metrics through various sources of data, including LWP’s Monitoring and Evaluation (M&E) reporting on performance indicators, KIIs with project stakeholders, and a document review, namely the annual LWP progress reports, the baseline and end line CPSSs, and the WDM reports. FINDINGS PS1: At least six hundred seventy-three thousand (673,000) Lebanese citizens benefit from improved water services as a result of the project. Lebanese citizens benefitting from improvements in water and wastewater services in DAs and in PODAs had reached 533,686 by the end of year 5. This number is expected to reach 772,827 as the populations in the Metn and Koura areas are added following the handover that took place in March and April 2021.  This PS was exceeded. PS2. At least a fifteen percent (15%) increase in the number of customers who receive and pay bills in the selected areas of intervention. RWEs reported the registration of new subscribers within the DAs, for example in Jezzine (359) and in Koura (150). According to the M&E performance indicator table, the baseline percent of the subscribers who receive and pay bills was 78% at the onset of the LWP. By year 5, it had reached 86% in response to the improvement in service delivery within the DAs. However, this decreased again to 80% in year 6 due to the political and economic crisis. 27 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT  This PS was not achieved. PS3. Four (4) WQAPs in place and enforced for each water supply provider in the selected areas of intervention. Four WQAPs were developed as part of the WSA conducted for BMLWE, NLWE, SLWE, and BWE with LRA. Trainings were conducted for laboratory staff at the 5 establishments. The WQAP process facilitated a service contract between BMLWE and AUB for the university to provide water quality testing and analysis at their laboratory. SLWE tried to do the same but could not pursue with the contract due to the economic crisis. LRA upgraded the Khirbet Qanafar Laboratory with the support of the LWP. BWE and NLWE did not take follow up actions. Hence, it can be concluded that the WQAPs were developed but they were partially enforced.  This PS was partially achieved. PS4. At least a fifteen percent (15%) reduction in the number of hours of lost service in the selected areas of intervention due to poor utility performance, including, but not limited to, inefficiencies in the collection and distribution systems. The improvement in water production in DAs and the decrease in technical losses are expected to result in decreased lost service hours as more water becomes available. Until year 5, the percentage reduction ranged between 8 and 13 percent. The percentage needs updating and verification following the handover of the DAs of Metn and Koura. This data was not available at the time of evaluation.  This PS could not be assessed at this stage. PS5. At least six (6) context-appropriate sustainable models for improving sanitation service delivery, including environmentally responsible sewage management, developed, piloted, and evaluated in selected areas of intervention. LWP rehabilitated three WWTPs in the Chouf area, namely Moukhtara, Mrousti, and Jbaa and two WWTPs in Jezzine, namely Bteddine El-Laqsh and Haitoura, and the Kirbet Qanafar CW. LWP extended the wastewater network in Kherbit Qanafar (1.7 km - additional 15,000 people connected) and Jebb Jannine (2.5 km - additional 15,000 people connected) to the Joub Jannine WWTP. The construction of a new CW in Roum is still ongoing, with handover expected in mid-June 2021. These models did not impose additional O&M requirements on the RWEs and are expected to be sustained. Hence, at the time of the evaluation, eight models had been implemented.  This PS was exceeded. PS6. At least six (6) context-appropriate sustainable model for improving communal water service delivery developed, piloted, and evaluated in selected areas of intervention, including vulnerable communities with a high number of Syrian refugees. LWP implemented small infrastructure PODAs: 1. BWE/ Ghazzeh: Well equipping, pumping station and elevated reservoir, solar plant, connection pipeline. No bills collected from this area. 2. BWE/ Baaloul 2.8 Km network to connect Baaloul households to the network. 3. BWE/ Barr Elias well equipping, pumping station, connecting pipeline to the UNICEF intermediate reservoir, rehabilitation of existing pump line to the Barr Elias village, sector zoning. 28 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT 4. NLWE/ Rahbeh: Deepening of an existing well, well equipping, transmission line from well to existing reservoir, rehabilitation of existing reservoir. 5. NLWE/ Rachaine: Existing reservoir rehabilitation, pumping station construction, lift line from pumping station to reservoir, transmission line from spring to pump station. 6. NLWE/ Bazaoun: Equipping the existing well, construction of control room, and construction of 1,000 Lm/ pumping line from the Bazaoun well to the Bazaoun existing reservoir. This has not been completed yet. RWEs are satisfied with the infrastructure work done, and with impact on the communities. One model, Bazaoun was not yet completed at the time of evaluation.  This PS was achieved pending the completion of Bazaoun project. Figure 9. Control at Ghazzeh pumping station PS7. At least a twelve percent (12%) improvement in the efficiency of irrigation systems in the selected areas of intervention where appropriate and feasible. The Chabrouh Irrigation Project showed a 15% increased efficiency in irrigation systems according to Annex A of Year 5 Annual Report.  This PS was exceeded. PS8. At least a fifteen percent (15%) decrease in the percentage of NRW in the selected areas of intervention. The activities implemented in the intervention areas contributed directly towards significant reductions in technical losses (physical NRW), and indirectly through the identification of illegal connections and the increase in subscriptions (commercial NRW). Due to the delay in construction activities and subsequently the WDM exercise, and due to the inability to obtain data on billing and payment from the establishments, only physical NRW could be computed for all areas except for Metn, where it was restricted to one village. The reported decreases in NRW as per the draft WDM reports were as follows. The ranges reflect the different values in the different DMAs: 29 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT - NLWE: Baseline: 54%.; End-line: 26-48%; Reduction: 6-28% - SLWE: Baseline: 64%.; End-line: 23-39%; Reduction: 25-41% - BMLWE: Baseline: 48%.; End-line: 18-24%; Reduction: 24-30% Due to the fact that NRW could not be computed for all Metn area, and that it did not include commercial NRW.  This PS was partially achieved. PS9. At least one water resources conservation and protection initiative at the Litani Basin yielding tangible and measurable results. Several initiatives were undertaken in the Litani Basin, including the CCIAZ project with 14 dairy farms located in the Litani Basin to improve (1) the treatment of whey water produced and reduce the BOD load into the river, (2) the wastewater connections in Joub Jannine and Khirbet Qanafar to the Joub Jannine WWTP, and (3) the rehabilitation of the Khirbet Qanafar Wetland.  This PS was exceeded. PS10. At least twenty-five percent (25%) reduction in O&M costs across fifteen (15) selected facilities. Seventeen activities aimed to decrease O&M costs of targeted facilities, focusing on offsetting electricity needs, as electricity is a key cost driver. PDOCs are spread across all four RWEs and LRA and are divided in two categories: - Eight water facilities rehabilitation (VFD, soft starter and needed rehabilitation/replacement of electromechanical equipment). This entailed procurement of equipment without installation due to time constraints. - Nine PV solar panels installation in water and wastewater facilities. This task has not been completed yet. While the total number of targeted facilities exceeds the standard, the activities in more than half of the facilities were at their early stages of installation and are at risk of not being completed before the end of the project. As for the percent reduction in O&M costs, the feasibility studies of the selected activities ensure the achievement of a 25 percent reduction. However, no measured data is available yet due to the delay in completion.  This PS was still in progress at the time of evaluation and its achievement is yet to be confirmed. PS11. At least 5 facilities in one or more RWEs have critical supplies for emergency preparedness. Eight generators were installed and are operational for BMLWE with a one-year service and spare parts contract.  This PS was exceeded. PS12. Engage at least five (5) civil society group or citizen’s organization in selected areas of intervention to identify and implement locally appropriate solutions for improved water use behavior for environment and water resources protection. LWP engaged eight civil society groups. 30 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT  This PS was exceeded. PS13. At least a fifteen percent (15%) decrease in nonproductive (wasteful) uses of water in both rural and urban settings in the selected areas of intervention. Through the incentive-rebate grants program, and in collaboration with ALI and private sector suppliers of agricultural equipment, LWP raised awareness about the importance of conserving water in agriculture (with three private firms) and industries (with ALI) and introduced new water-saving technology in both irrigation and stone-cutting industries. Results showed a 67% decrease in water consumption in stone/rock-cutting plants that had adopted water conservation and cleaner production processes. Furthermore, the installation of drip irrigation on a 20 Ha of land planted with carrots by one of the largest producers in the Beqaa has allowed for an average 60% water saving compared to previous years.  This PS was exceeded. PS14. At least a fifteen percent (15%) increase in customer satisfaction with service delivery through public water utilities in the selected areas of intervention. Based on the Baseline and the Endline CPSS: - 74% of respondents said they were satisfied with water services, compared to 71% in 2016. - 59.3% of respondents said they had submitted claims or requests for repairs to the RWEs, compared to only 27.6% in 2016. - The number of subscribers who said were very satisfied with RWEs’ service and actions taken in response to claims and requests increased by 17% between 2016 (26.67%) and 2020 (44.11%). - 56.4% of respondents noticed improvements in their respective RWE (infrastructure, services, customer service). - 18.7% increase in customers’ trust in the RWEs to manage the water services in Lebanon. Despite the dramatic increase in resentment of the public towards the government and all government services since October 2019, the overall increase in customer satisfaction with service delivery was 3 percent and the increase in satisfaction with response to repairs reached 17 percent.  This PS may be considered as achieved. PS15. Develop a pilot PSP to improve water supply, wastewater, or irrigation service delivery LWP succeeded in mediating a contract between a payment service provider and SLWE. The contract offers SLWE customers the chance to pay their water bills via one of 600 points of sale, scattered all over the country. Several PPP framework contracts were prepared for RWEs but did not materialize given the crisis.  This PS was achieved. PS16. Develop at least one pilot private sector partnership to leverage the funding of innovative water conservation technologies. LWP partnered with ALI to perform outreach to industries to educate them on possible technologies for water saving and incentivize industrialists to adopt them. Accordingly, three factories benefitted from 31 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT RECOMMENDATIONS Based on the findings and conclusions presented above, the ET proposes several recommendations for USAID and its IPs for future work in the water sector. What successes can be built on or replicated in the future? 1. USAID/IPs to consider replicating the closed system approach in improving water service delivery and reducing NRW. 2. USAID/IPs to consider replicating the water conservation initiatives with industries and farmers or other heavy water users, through partnerships and grants with associations, cooperatives, and/or municipalities. 3. USAID/IPs to consider replicating the leadership training for both female and male staff at the RWEs. 4. USAID/IPs to consider replicating the PDOCs, as they are much needed during the current crisis to support RWEs in ensuring continuity of service. What challenges should be further addressed or avoided in the future? 5. IPs to plan all construction activities while accounting for delays in design approval and in construction (due to USAID A&E QA/QC requirements, ERAC and EIA approvals, as well as unexpected external factors). 6. IPs to plan the completion of construction activities, particularly those requiring extensive data processing post-construction and installation, such as water demand management in DAs at least 1.5 years before EOP to ensure an adequate handover, training, and transfer of knowledge to staff on equipment operation and water demand management through handholding, on-the-job training, and follow-up for at least six months. 7. USAID/IPs to consider following up on the WDM exercises conducted in the LWP DAs to further train RWE staff from all relevant sections—i.e., production, networks and distribution, and billing and customer service—on all elements of WDM and the NRW model, and on implementing the developed WDP through hand-holding and on-the-job training. 8. IPs to encourage RWEs to consider dedicating permanent staff for WDM. 9. USAID/IPs to consider that installing smart meters at the consumer level without imposing consumption-based tariffication can result in an unwarranted increase in consumption, which needs to be addressed through technical controls and/or awareness campaigns until the consumption￾based tariff is implemented to curb overconsumption. 10. IPs to consider the challenges faced by RWEs in imposing consumer-based tariffs and to couple improved water supply projects with improved billing to avoid increasing NRW. 11. USAID/IPs to differentiate in their performance standards between physical and commercial NRW. What local conditions/constraints need to be considered in implementing future projects? 12. Given the current crisis, USAID/IPs to consider placing more focus on stabilization, rather than on development, to ensure that the RWEs are able to use existing tools, manage what they have, and operate and maintain existing facilities. 13. USAID/IPs to encourage new PPP initiatives that focus on innovative ways to attract the private sector into a productive partnership with the RWEs, in light of the current crisis. 33 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT 14. USAID/IPs to consider investing in building local technical capacities for the development, support, and maintenance of the ERP system to ensure its sustainable use. This could include organizing a formal ERP unit in each RWE and training a joint support team for all establishments. 15. USAID/IPs to consider offering continuous training and capacity building to RWE staff on all operational aspects with a special focus on permanent staff since contractual staff are leaving for better-paid jobs due to the current crisis. What local opportunities need to be built upon in implementing future projects? 16. USAID/IPs to encourage the participation of locally-developed innovations and local innovative SMEs in implementing future projects, such as the E2 activity. 17. USAID/IPs to build on Lebanon’s excellent academic/research centers (such as the CEEDD) for implementing future projects. 18. USAID/IPs to benefit from the hundreds of associations/organizations/cooperatives that are active in Lebanon in the fields of water, agriculture, rural development, and/or community support. 19. USAID/IPs to consider implementing awareness campaigns on TV to attract a wider range of audiences. Communication budgets should take that into account and ensure broadcasting for several weeks so that advertisement/ awareness campaigns can have an impact and be remembered. Awareness raising on social media and in schools should be continued as the most preferred means in this regard. 34 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT ANNEXES Annex A: Inception Report and Scope of Work Annex B: Detailed Evaluation Design and Methodology Annex C: Evaluation Schedule and Persons Interviewed Annex D: Documents Reviewed Annex E: Data Collection Instruments Annex F: Details of LWP Major Activities Annex G: Conflict of Interest Disclosures Annex H: ET Members 35 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT ANNEX A: INCEPTION REPORT AND SCOPE OF WORK Inception Report: [redacted] Scope of work: [Attached] 36 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT i | LWP F NAL PERFORMANCE EVALUATION SOW USA D.GOV LEBANON WATER PROJECT FINAL PERFORMANCE EVALUATION Scope of Work January 2021 This document is available in printed or online versions. Documents are also made available through the Development Experience Clearing House (dec.usaid.gov). Additional information can be obtained from: Social Impact, Inc. Corporate Office 2300 Clarendon Boulevard Suite 10300 Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com Social Impact, Inc. Arz Street L brex Bldg. Bloc B – 3rd Floor Za ka, Metn, Lebanon Tel: +961-1-879260 USA D.GOV LWP F NAL PERFORMANCE EVALUATION SOW ii CONTENTS INTRODUCTION 1 AUDIENCE AND INTENDED USES 1 PROJECT IDENTIFICATION 1 PROJECT DESIGN 2 INSTITUTIONAL CONTEXT 2 THEORY OF CHANGE 2 PROJECT INTENDED RESULTS / SCOPE OF THE PROJECT 3 CRITICAL ASSUMPTIONS 4 EXISTING INFORMATION 4 EVALUATION OBJECTIVES AND QUESTIONS 4 OBJECTIVES 4 EVALUATION QUESTIONS 4 EVALUATION DESIGN AND METHOD 5 ILLUSTRATIVE DATA COLLECTION METHODS AND TOOLS 5 DATA ANALYSIS METHOD 6 FINAL PERFORMANCE EVALUATION DELIVERABLES 11 REPORTING GUIDELINES 12 TEAM COMPOSITION 13 PROVISIONAL EVALUATION MANAGEMENT 14 ANNEX 1: LWP LOGICAL FRAMEWORK 17 ANNEX 2: LWP PERFORMANCE INDICATORS Y5 RESULTS 21 ANNEX 3: LWP PERFORMANCE STANDARDS 28 iii | LWP F NAL PERFORMANCE EVALUATION SOW USA D.GOV ACRONYMS ADS Automated Directives System CAS Central Agency of Statistics CDCS Country Development Cooperation Strategy CFR Code of Federal Regulations CLA Collaborating, Learning and Adapting COP Chief of Party COR Contracting Officer’s Representative CPSS Customer Perception and Satisfaction Survey CSO Civil Society Organization DAI Development Alternatives, Inc. DEC Development Experience Clearing House DG Director General DO Development Objective DQA Data Quality Assessment EMMP Environmental Mitigation and Monitoring Plan ERP Enterprise Resource Planning FGD Focus Group Discussion FY Fiscal Year GE/FE Gender Equality and Female Empowerment GIS Geographic Information System GOL Government of Lebanon HO Home Office HOE Haouch El Omara IP Implementing Partner IM Implementing Mechanism IR Intermediate Result KII Key Informant Interview LRBMS Litani River Basin Management Support LRA Litani River Authority LWP Lebanon Water Project LWPP Lebanon Water Policy Program LWWSS Lebanon Water and Wastewater Sector Support LOE Level of Effort M&E Monitoring and Evaluation MENA Middle East and North Africa MOU Memorandum of Understanding NGO Nongovernmental Organization NRW Non-Revenue Water PIRS Performance Indicator Reference Sheet PMP Performance Management Plan PMSPL Performance Management and Support Program for Lebanon PPP Public-Private Partnership USA D.GOV LWP FINAL PERFORMANCE EVALUATION SOW iv PSP Private Sector Partnership RWE Regional Water Establishment RF Results Framework SI Social Impact SOW Scope of Work TAMIS Technical and Administrative Management Information System USAID United States Agency for International Development USG United States Government WISE Water Infrastructure Support Enhancement WQAP Water Quality Assurance Plan WSPAI Water Service Provider Assessment Index WWTP Waste Water Treatment Plant 3 | LWP FINAL PERFORMANCE EVALUATION SOW USA D.GOV LWP is based on the following theory of change: If LWP can improve service delivery and resource management capacity of the public water utilities; improve civic engagement in water management, advance citizens’ perception of water utilities’ role in providing a necessary and valuable service; and improve private sector involvement in water management, then LWP will be able to improve water governance and protection to ensure reliable and sustainable access to water in Lebanon. PROJECT INTENDED RESULTS / SCOPE OF THE PROJECT The objective of LWP is to improve water governance and protection to ensure reliable and sustainable access to water in Lebanon through improving the service delivery and resource management capacity of five public utilities; and improving civic engagement in water management and citizen’s perception of water utilities in providing a necessary and valuable service. LWP will achieve this objective by employing a strategy that brings the public sector, private sector, and civil society together to conserve and improve the management of Lebanon’s water resources. The strategy focuses on improving the capacity of public water utilities, engaging citizens in communicating with public water entities, and promoting citizen behavioral changes to conserve water and protect water quality through media campaigns. Concurrently, LWP will implement activities to increase private sector engagement in finding solutions to water service delivery and water quality problems through partnerships with the public water entities. Purpose A: Improved service delivery and resource management capacity of the five public water utilities To achieve Purpose A, LWP will strengthen the managerial and operational functions of the water entities, while providing related upgrades to their infrastructure. The project will support the Government of Lebanon’s (GoL’s) newly developed water strategy that boosts the role of these RWEs and LRA as the legitimate entities that should provide efficient services to the population. Purpose B: Improved civic engagement in water management and advanced citizens’ perception of water utilities’ role in providing a necessary and valuable service To achieve Purpose B, LWP will bolster the legitimacy of the RWEs and LRA by engaging citizens, local authorities, and the commercial sector by improving their understanding and support for steps to improve water management and delivery of water and wastewater services by the water utilities and municipal governments. LWP will also employ a communications strategy to build awareness on water use behavior and to engage civic society in water conservation and management. Purpose C: Improved Private Sector Involvement in Water Management LWP will support and mediate pilot private sector partnerships to improve water and wastewater management, leverage the funding of innovative water conservation technologies, and improve water use awareness. Activities under Purpose C will be crosscutting along Purpose A and Purpose B, with overlapping outcomes such as increased customers who pay bills, decreased non-revenue water, and decreased wasteful uses of water. The Performance Data Table can be found in Annex 2. USA D.GOV LWP F NAL PERFORMANCE EVALUATION SOW 4 LWP CRITICAL ASSUMPTIONS The successful achievement of LWP objective and purposes is based on several critical assumptions which take into consideration the Lebanese political, security, and institutional context. 1. Security situation in Lebanon does not significantly deteriorate 2. RWEs and LRA are willing participants in project activities 3. Civil Society and private sector are willing partners in collaboration for project activities 4. International donors are coordinating and aligning their activities with each other, and with RWEs and LRA EXISTING INFORMATION - Country Development Cooperation Strategy (CDCS) USAID/Lebanon - Water Project PAD - LWP Contract and contract modifications - LWP Work Plans - LWP M&E Plan - LWP Periodic reports (Quarterly Reports, Annual Reports, IPPT, etc.) - Citizens perception and satisfaction survey – Baseline Survey - Data Quality Assessment (DQA), Project Site Visit Reports (PSV), and Data Verification (DV) reports conducted by Social Impact. - PMSPL II construction management reports of LWP projects - LWP Mid-term performance evaluation EVALUATION OBJECTIVES AND QUESTIONS OBJECTIVES While the objective of the mid-term evaluation that was conducted in 2018 focused on validating the LWP theory of change and on providing recommendations for mid-course corrections to reach the overall planned LWP targets, the objective of this LWP final performance evaluation is to assess the extent to which LWP has met the intended objectives and its performance standards. The evaluation will answer the below evaluation questions. Meetings with the Mission and the stakeholders will be held prior to the Evaluation kick-off to determine priorities, and to add clarity to the evaluation questions based on the Mission stakeholders intended use of the evaluation findings, conclusions and recommendations. EVALUATION QUESTIONS As the mid-term evaluation conducted in 2018 answered questions related to the relevance, effectiveness and related factors affecting the achievement or non-achievement of the objectives, sustainability and gender integration, this final performance evaluation is only required to look at the effectiveness of the activity, and answer the below questions on effectiveness. 1. To what extent have LWP’s interventions achieved the objectives of the Activity as stated in the contract? More specifically: 5 | LWP FINAL PERFORMANCE EVALUATION SOW USA D.GOV 1.1. in terms of improving service delivery and resource management capacity (result A), the evaluation is to answer whether: 1.1.1. the water establishments have become more effective in water delivery; 1.1.2. the water establishments are now better equipped in managing their resources. 1.2. in terms of improving the civic engagement in water management (result B), the evaluation is to answer whether: 1.2.1 the perception and satisfaction of citizens towards water establishments have improved; 1.2.2. the Lebanese citizens are better aware and proactive towards the importance of water management and conservation. 1.3. In terms of improving the private sector involvement in water management (result C), the evaluation is to answer whether: 1.3.1. the targeted private sector is more involved and engaged in water management and conservation initiatives? 1.3.2. The farmers and industries are more involved in water conservation? 2. Did LWP achieve the performance standards relevant to each of the Results A, B and C (Annex 3)? In accordance with ADS 205.3.8.2 and USAID youth policy, this evaluation is to adequately capture the situations and experiences of youth and both males and females. EVALUATION DESIGN AND METHOD This evaluation will rely on review of project documentation (e.g. RFA, CA, work plan, quarterly reports, annual report, and technical reports) and data collected from interviews with the Mission, project staff, project beneficiaries and other stakeholders. Site visits, direct observations, and other mixed methods outlined in the evaluation matrix below will be employed to inform the evaluation. These approaches should allow the evaluation team adequate information to assess the project’s implementation and results against its objectives, scope of work, deliverables, work plan, etc. and to determine whether project activities are progressing as planned, how well activities are being implemented and to what extent they meet with targets set in project documentation. Data collected during this evaluation should also provide information on whether the target audience was reached, and how external and internal factors affected project implementation and results. ILLUSTRATIVE DATA COLLECTION METHODS AND TOOLS The evaluation is expected to use a mixed methods approach. The methodology for this evaluation will draw upon qualitative methods, including a desk review, remote key informant interviews (KII), online FGD and direct site observation. Acknowledging the fact that this evaluation is being conducted during a pandemic period and while Lebanon is witnessing a surge in the official number of COVID-19 confirmed cases, a Social Impact Internal Review Board needs to be undertaken as face-to-face Key Informant Interviews and On site Direct Observation can be anticipated, for use in-person data collection approaches during the COVID-19 pandemic. SI’s IRB will review the application to determine whether the procedures effectively safeguard participants and whether the informed consent process is appropriate. Methods that will be used for gender sensitive data collection are the following: USA D.GOV LWP F NAL PERFORMANCE EVALUATION SOW 6 - Desk review: The evaluation team will do a systematic literature search to identify evaluations, tools and other documents on water sector projects; the evaluation team will also review existing documents including CDCS, RFA, contract agreement, contract modification, initial assessments, periodic reports (annual, quarterly, other) and all other reports produced by USAID, Social Impact, and LWP as well as previous evaluation reports of similar projects to determine service delivery mechanisms, quality of project results, alignment with CDCS, validity of development hypothesis, quality of program management and the progress against project targets. The evaluation team will also review data from water service providers, as available. - Interviews of LWP and USAID Key Staff: The team will design an interview matrix and meet remotely with the activity staff and conduct key informant interviews (KII) with a mix of male/ female key personnel, such as USAID Local Development Office, LWP COR, DAI/LWP COP and DCOPs, DAI/LWP M&E team, DAI/LWP technical team and engineers, private sector engagement specialist, to obtain more detailed, in-depth understanding about specific issues. Discussions will focus on how the interventions were selected, designed and implemented, how the gender gaps were identified and addressed, as well as suggestions for improving the LWP activity outcome during the remaining LOP. Interviews with other key stakeholders will also be conducted using a structured questionnaire as a guide during interviews. - Remote KII and FGD with LWP beneficiaries and stakeholders: The Evaluation Team will conduct remote KIIs and online FGDs (as feasible) with selected water sector actors and the project beneficiaries (water utilities employees, training recipients, NGOs representatives, private sector entities, farmers trained on irrigation techniques, etc.) Discussions will be derived from the initial findings of KIIs. The sampling for the FGD will be determined based on a random sample of the different beneficiaries involved in the activity across Lebanon. An invitation for a Zoom or TEAMS or Whatsapp FGD will be sent to selected individuals after a quick initiation call. The final sampling process will be determined by the TL evaluation expert and the local expert based on discussion with USAID and LWP team. -Direct site’s observations: Following the IRB recommendations, the evaluation team will undertake field visits to activity sites (i.e. water establishment offices and water stations) utilizing a direct observation guide to obtain a first-hand experience of the activity outcomes. DATA ANALYSIS METHOD The data gathered from the KII and the FGD will be used as the basis for data analysis. Techniques in data analysis will be gender sensitive and include review of existing documentation obtained, conceptualizing coding and categorizing of information obtained in the KII and FGD, examining relationships between the different activity components, activities and achievements as revealed by stakeholders, and authenticating conclusions by triangulation of data obtained from different sources. In conformity with ADS 201.3.5 and 201.3.6.8, the evaluation team will use relevant monitoring data and feedback from beneficiaries already collected by implementing partners as data sources, when appropriate. The evaluation team will prepare a data source matrix (also called an evaluation matrix) that clarifies how the data will be collected. It will include the evaluation questions, the evaluation tool(s), data source(s) and analysis plan for each question. An illustrative version of the matrix is provided below. The technical expert will consider the illustrative evaluation matrix and propose a final matrix for consensual agreement by the evaluation team. To the extent possible given time and budget, the methods used or even evaluation questions may be revised in response to preliminary findings or obstacles that prevent timely implementation of the proposed method. For example, evaluation questions that require comparison to a prior point in time may need to be rephrased if it is not possible to obtain reliable prior data to measure 7 | LWP FINAL PERFORMANCE EVALUATION SOW USA D.GOV a change over time. This matrix will ensure that a multitude of data sources are considered, and the team will be able to use triangulation of data to answer each question with enriched validity. All the data collected from different sources will be reviewed for reliability and validity, and findings will be compared using multiple methods, data forms and sources of data. Data will be organized and disaggregated by gender, to answer specific evaluation questions in the final report. Differentiation between the findings, interpretations, judgments, recommendations, and the specific sources of evidence supporting these, should be made clear in the final evaluation report. POTENTIAL LIMITATIONS AND BIASES The evaluation team should be aware several noteworthy risks to the data collection and analysis and should be prepared to mitigate those risks: • Recall Bias: Recall bias is a common evaluation problem. LWP beneficiaries may respond to questions posed by the evaluation team with answers that blend their experiences into a composite memory. Water utilities representatives who may have benefited from similar activities implemented by other donors may not distinguish them as separate activities. Additionally, depending on when beneficiaries participated in LWP, perceptions of events may change over time, and the ability to remember specific details may fade. The evaluation team could mitigate this risk by attempting to interview individuals most actively involved in the activity and ask specific questions to frame the USAID assistance. • Response Bias: Response bias is the risk that key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. Beneficiaries may believe that negative evaluation findings could result in reduced assistance from USAID/Lebanon. The evaluation team should be especially cognizant of the risk of response bias. The evaluation team will mitigate the risk of response bias by utilizing data triangulation to note discrepancies in responses through different data sources (KIIs, FGDs, document review) and preparing follow-up questions for informants and FGD participants that encourage more specific responses. For example, evaluators will ask respondents for a specific example to back up certain responses. • Selection Bias: Selection bias is an inherent risk when implementers facilitate contact with activity beneficiaries. To eliminate selection bias, the evaluation team should select the target beneficiaries without input from the implementer, thereby ensuring that the sample is not skewed. The ET should be aware of the risk of selection bias due to invited interlocutors being unfamiliar with or unable to access technology used for Internet-based meetings and is prepared to use telephone calls with individuals if invited individuals are unable to participate in remote FGDs. • Remote Data Collection: The COVID-19 pandemic will restrict the ability of the evaluation team to meet with individuals for KIIs and FGDs. Because face-to-face contact is not possible, the team will conduct remote interviews with key informants and FGDs. As the Team Leader might not travel to Lebanon, he/she would conduct interviews by telephone or Skype with staff from the Mission, implementers and beneficiaries. 9 | LWP FINAL PERFORMANCE EVALUATION SOW USAID.GOV management and conservation? (Result B) Primary Sources - Key Informant Interviews (KIIs) with USAID and LWP key staff, technical experts and USAID stakeholders - FGDs with group of citizens (if feasible) 1.3.1. Is the targeted private sector more involved and engaged in water management and conservation initiatives? (Result C) -LWP Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) - KII with USAID staff, stakeholders and technical experts Secondary Sources - Desk review of project documents Primary Sources - Key Informant Interviews (KIIs) with USAID and LWP key staff, technical experts, industrialists beneficiaries of the activity, association of Lebanese industrialists, and USAID stakeholders - Qualitative analysis -Quantitative analysis of LWPs indicators 1.3.2. Are the farmers and industries more involved in water conservation? (Result C) -RFA -LWP Contract -LWP Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) - KII with USAID staff, stakeholders, and technical experts Secondary Sources - Desk review of project documents Primary Sources - Key Informant Interviews (KIIs) with USAID and LWP key staff, technical experts, farmers beneficiaries of the activity and USAID stakeholders - Qualitative analysis -Quantitative analysis of LWPs indicators 2. Did LWP achieve the performance standards relevant to each of the Results A, B and C Are the activity assumptions still valid? -RFA -LWP Contract -LWP Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) - Citizens perception and satisfaction survey. - KII with USAID staff, stakeholders and technical experts Secondary Sources - Desk review of project documents including the Citizen Perception and Satisfaction Survey. Primary Sources - Key Informant Interviews (KIIs) with USAID and LWP key staff, technical experts and USAID stakeholders - Qualitative analysis - Quantitative analysis of LWP indicators 11 | LWP F NAL PERFORMANCE EVALUATION SOW USA D.GOV FINAL PERFORMANCE EVALUATION DELIVERABLES - Briefing: The evaluation team will provide a briefing to USAID prior to the start of the evaluation and at the end of the evaluation. The initial briefing will focus on presenting the evaluation team plan for conducting the evaluation. Discussions with the Mission on the proposed methodologies for the evaluation will result in an Inception Report (a deliverable) that states the agreed methodologies. - Inception Report: PMSPL II team will prepare an Inception Report that is the result of kickoff meetings with the Mission and stakeholders. The Inception Report makes final the methodologies, tools, and evaluation work plan and time schedule. The Inception Report will be submitted to the COR at USAID/Lebanon for approval. Approval of the Inception Report signals agreement by the Mission and the evaluation team to use the proposed methodologies and tools with adherence to the work plan and time schedule. The Inception Report is Annex A of the evaluation report. - Interim Findings Presentation: Toward the end of the field work, prior to the development of the Draft Report, the team will carry out a preliminary findings presentation at USAID to discuss findings, conclusions and recommendations. - Draft report: A draft report of the findings and recommendations will be submitted to USAID COR clearly describing findings, conclusions, and recommendations, within two weeks of completing the data collection. USAID will provide comments on the draft report within two weeks of submission. ADS 201 directives state that draft reports “must undergo a peer review organized by the office managing the evaluation”’, and that “reports must be shared with implementing partners of the projects or activities addressed in the evaluation.” The evaluation team will consider USAID comments and the comments of other stakeholders and revise the draft report accordingly and as appropriate for an independent evaluation. Where differences of opinion exist, these will be presented in a Statement of Differences Annex. - Final Report: The final evaluation report will be provided to USAID in MS Word and Adobe PDF within 15 calendar days following receipt of final comments from USAID. The report will include all of the components outlined in the “Structure of the Evaluation Report”, but not necessarily in the order specified below. The report will not exceed 25 pages (excluding appendices). Appendices should at a minimum include the Inception Report for the evaluation; a list of individuals interviewed; a complete description of the methodology used for the evaluation; and any data collection used. The report will be submitted in English, electronically in MS Word format and compliant with USAID Graphic Standards. Utilization and Dissemination Plan: The final approved evaluation report will be shared with USAID/Lebanon staff members. The report will follow USAID branding requirements. Evaluations at USAID should be transparent: Findings from evaluations will be shared as widely as possible with a commitment to full and active disclosure. ADS 201.3.5.15 operationalizes this principle by requiring Operating Units to plan for evaluation dissemination: “Missions and Washington OUs must plan for dissemination and use of the planned evaluation. Missions or Washington OUs should consider how key partners and other development actors would best receive evaluation information and plan their communications products accordingly.” A dissemination plan will be prepared and filled prior the kick off the evaluation and upon the approval of the SOW. USA D.GOV LWP FINAL PERFORMANCE EVALUATION SOW 12 As a part of the post Evaluation action plan, the evaluation team will follow up to facilitate and enhance use through one or more of the following steps: o Distr bute the final approved evaluation report to DAI, LWP and other stakeholders at the discretion of USAID/Lebanon. o Uploaded the final approved evaluation report to the DEC for public access o Develop a utilization plan with primary intended users o Adapt and reformat findings for different audience. o Watch for emergent opportunities to reinforce findings o Continue to build evaluation capacity during the follow up process. o Conduct a client satisfaction survey, o Conduct a tracer survey. REPORTING GUIDELINES USAID’s Evaluation Policy requires that all evaluation SOWs include USAID’s Criteria to Ensure the Quality of the Evaluation Report. The LWP final performance evaluation team is advised to incorporate these guidelines in their report where relevant and applicable to the evaluated contract agreement. Guidelines on the Structure of the Evaluation Report The findings from the evaluation will be presented in a draft report at a full briefing with USAID/Lebanon and key stakeholders. The format for the evaluation report is as follows: 1. Abstract of not more than 250 words briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions. 2. Glossary & Acronyms 3. Table of Contents (1 pp.); 4. Executive Summary: that summarizes key points (purpose and background, evaluation questions, methods, findings, and conclusions). (2 to 5 pp.); 5. Introduction: purpose, audience, and synopsis of task (1 pp.); 6. Background: Brief overview of LWP activity context, USAID/Lebanon program strategy and activities implemented in response to the development problem, a brief description of LWP, and purpose of the evaluation (3 pp.); 7. Methodology: Describes evaluation methods, including constraints and gaps (1 pp.); 8. Key Findings and Conclusions for each evaluation question; including a reference to the data quality of the evidence provided (10 pp.); 9. Issues: provide a list of key technical and/or administrative issues, if any (1 pp.); 10.Recommendations: Identify list of recommendations based on the evaluation findings. 11.Annexes: (A) Scope of Work/Inception Report, (B) Persons Interviewed, (C) B bliography, D) Data Collection Tools, (E) Data Analysis, (F) Focus Group results (if applicable), (G) AOB, (H) Statement of Differences 13 | LWP F NAL PERFORMANCE EVALUATION SOW USA D.GOV TEAM COMPOSITION USAID’s ADS requires that at least one member of every evaluation team be an evaluation specialist. The PMSPL II team will provide evaluation specialization support to the evaluation team backstopped by Social Impact’s home office. The team will include two experts, one of whom will act as the Team Leader. 1) Team Leader/ Water Sector Expert with a professional background in water utility and resources management or civil engineering in the water sector, the expert should preferably have work experience in water sector development strategies and development programs in developing countries. This person shall be responsible for coordinating and directing the overall evaluation, including preparation and submission of the draft and final evaluation report. He/she should have a minimum of 10 years’ experience in the design, implementation, and/or evaluation of foreign assistance water sector projects with organizations like USAID, the United Nations, or the World Bank. He/she should have an advanced degree in civil engineering, Mechanical Engineering, Environmental Engineering, Public Administration, or in another relevant area and with sufficient knowledge of gender sensitive evaluations. The team leader should have excellent written and oral communication skills in English (45 days LOE). 2) Lebanon Water Sector Expert (local) with at least 7 years of experience in general water sector situation of Lebanon, the national water policies, and the institutional context. He/She should have an advanced degree in Water resources management, water engineering or other related fields. Knowledge of USAID and other international donors that provide support to the water sector is preferable. He/ She is expected to develop gender sensitive data collection tools, conduct KII and FGDs, and participate in the analysis and drafting of the evaluation report. He/She is also expected to work with the Team Leader on other aspects of the evaluation (42 days LOE). The evaluation team will consist of a Team Leader/Water Sector Expert, Local Water Sector expert with technical support from PMSPL II evaluation team. PMSPL II staff works alongside the technical expert/evaluator and forms a part of the evaluation team. SI HO has budgetary provision for additional technical evaluation support that might be needed and providing quality assurance for the evaluation through SI’s EQUI (Evaluation, Quality, Use and Impact) initiative. PMSPL II resident staff and HO staff participate in drafting, reviewing and editing the draft final report before circulating to the stakeholders for comment and review. Preparation of the evaluation report is the responsibility of the team leader/technical expert. Final editorial responsibility and quality control for the report is with Social Impact. Comments received are to be address by the team leader and the technical expert supported by the PMSPL II staff. The draft evaluation report is to be reviewed by USAID/Lebanon and, at the request of USAID/Lebanon, a broad range of stakeholders. Comments received after the budgeted end date for this evaluation, may be addressed in a Statement of Differences Annex at the discretion of USAID/Lebanon. PMSPL II proposes to recruit technical experts to collaborate in this evaluation as described above. PMSPL II will certify that there is no conflict of interest or potential conflict of interest with respect to the performance of this evaluation on the part of the contractor and the contractor’s team members. Social Impact will guarantee that substitutions will not be made for individuals proposed as team members without the approval of USAID. USA D.GOV LWP FINAL PERFORMANCE EVALUATION SOW 14 PROVISIONAL EVALUATION MANAGEMENT Pre Field-Work: PMSPL II staff identify and recruit the technical experts and assign a team leader, obtain key documents, establish key contact, liaison with LWP to set up interviews with project staff and prepare interview schedule with LWP stakeholders and beneficiaries. Evaluation Kick-off (Weeks 1-3): PMSPL II COP meets remotely with the Team Leader, Local Expert and PMSPL II evaluation team to assign roles and respons bilities, agree on the evaluation work plan, list of people to be interviewed, outline of the final evaluation report. During this period the evaluation team will develop the interview guide, the focus group discussion guide and all the necessary forms for collection of informant narratives, design of field work, and strategy for qualitative analysis takes place. Technical advice and counsel might be sought from SI HQ at this time. The pre-field work should be accomplished prior to the data collection process. The team will meet remotely with USAID Program Office and COR for an in-country briefing to agree on the evaluation details, methodology and reporting guidelines. Additional documentation may be requested at that time. Logistics for the field visits will also be made final. The Team Leader will present an Inception report underlying a detailed methodology and data collection plan and evaluation matrix to be approved by USAID COR prior to the start of the field work. Field Work (weeks 4 to 6): Interviews with key stakeholders including, USAID key staff, LWP staff, partners and sub-contractors will be carried out. The team will also interview project stakeholders and beneficiaries whose work have been impacted by project activities. The evaluation team will also carry out focus group discussion with project beneficiaries and stakeholders. Post Field-Work (weeks 7-9): The Team Leader is expected to write the final evaluation report including background, setting and institutional context, data analysis, findings and recommendations. The local expert will provide support where applicable in the drafting and writing of certain sections in the report. Integration of the results and analysis from the different focus group discussions will be the responsibility of the team leader. Where applicable, PMSPL II team will provide support to the evaluation team. An out-brief presentation to the Mission will be carried out prior to the submission and presentation of the completed draft final report. The purpose of the presentation is to highlight some of the major findings, conclusions and recommendations based on the field work carried out. It will help underline any concerns or issues by USAID staff members and stakeholders to be included in the draft as well as final report to be presented. Review Draft Report (week 10-11): USAID and stakeholders are requested to review the draft report and prepare comments within two weeks of the submission of the final evaluation report. Respond to Draft Report Comment (weeks 12-13): PMSPL II responds to comments from USAID and stakeholders revising the report as necessary. Final Report Preparation and Quality Assurance: SI HO applies EQUI quality assurance steps to the draft report. The final report will be submitted no later than two weeks following receipt of final comments from USAID/Lebanon. This includes time needed by SI HQ to edit and review the report prior to submission as a final evaluation report to USAID/Lebanon. The deadline for final report submission is End of April 2021. 15 | LWP F NAL PERFORMANCE EVALUATION SOW USAID.GOV Table 2. LWP Evaluation Illustrative Work Plan LWP Evaluation Workplan.xlsx USA D.GOV LWP F NAL PERFORMANCE EVALUATION SOW 28 ANNEX 3: LWP PERFORMANCE STANDARDS Performance Standard – Result A IMPROVE SERVICE DELIVERY AND RESOURCE MANAGEMENT CAPACITY 1 At least six hundred seventy-three thousand (673,000) Lebanese citizens benefit from improved water services as a result of the project 2 At least a fifteen percent (15%) increase in the number of customers who receive and pay bills in the selected areas of intervention 3 Four (4) Water Quality Assurance Plans (WQAPs) in place and enforced for each water supply provider in the selected areas of intervention 4 At least a fifteen percent (15%) reduction in the number of hours of lost service in the selected areas of intervention due to poor utility performance, including, but not limited to, inefficiencies in the collection and distribution systems 5 At least six (6) context-appropriate sustainable model for improving sanitation service delivery, including environmentally responsible sewage management, developed, piloted, and evaluated in selected areas of intervention 6 At least six (6) context-appropriate sustainable model for improving communal water service delivery developed, piloted, and evaluated in selected areas of intervention, including vulnerable communities with a high number of Syrian refugees 7 At least a twelve percent (12%) improvement in the efficiency of irrigation systems in the selected areas of intervention where appropriate and feasible 8 At least a fifteen percent (15%) decrease in the percentage of NRW in the selected areas of intervention 9 At least one water resources conservation and protection initiative at the Litani Basin yielding tangible and measurable results 10 At least twenty-five percent (25%) reduction in O&M costs across fifteen (15) selected facilities. Performance Standard – Result B IMPROVE CIVIC ENGAGEMENT IN WATER MANAGEMENT 1 Engage at least five (5) civil society group or citizen’s organization in selected areas of intervention to identify and implement locally appropriate solutions for improved water use behavior for environment and water resources protection 2 At least a fifteen percent (15%) decrease in nonproductive (wasteful) uses of water in both rural and urban settings in the selected areas of intervention 3 At least a fifteen percent (15%) increase in customer satisfaction with service delivery through public water utilities in the selected areas of intervention Performance Standard – Result C IMPROVE PRIVATE SECTOR INVOLVEMENT IN WATER MANAGEMENT 1 Develop a pilot private sector partnership to improve water supply, wastewater, or irrigation service delivery 2 Develop at least one pilot private sector partnership to leverage the funding of innovative water conservation technologies 3 Develop at least one pilot private sector partnership to improve water usage awareness ANNEX B: DETAILED EVALUATION DESIGN AND METHODOLOGY DATA COLLECTION METHODS AND TOOLS The methodology for this evaluation has drawn upon qualitative methods, including a desk review, remote KIIs, and direct site observation. Given the current COVID-19 pandemic situation in Lebanon, a SI’s IRB has been consulted where face-to-face KIIs and On-site Direct Observation were anticipated, for in-person data collection approaches during the COVID-19 pandemic. SI’s IRB has reviewed the application to determine whether the procedures effectively safeguard participants and whether the informed consent process is appropriate. Methods that have been used for gender sensitive data collection are the following: - Desk review: The evaluation team has reviewed existing documents including the CDCS, contract agreement, contract modification, initial assessments, periodic reports (annual, quarterly, other) and all other related reports, such as pre- and post- customer survey and citizen perception reports, produced by USAID, SI, and LWP, as well as data from water service providers, as available. - Interviews of LWP and USAID Key Staff: The team has designed a structured interview guide and met remotely with the activity staff and conducted KIIs with a mix of male and female key personnel, such as from the USAID Local Development Office, LWP COR, DAI/LWP Chief of Party (COP) and Deputy Chief of Party (DCOP), DAI/LWP M&E team, DAI/LWP technical team and engineers, private sector engagement specialist, to obtain a more detailed, in-depth understanding about specific issues. Discussions were focused on how the interventions were selected, designed and implemented, how the gender gaps were identified and addressed, as well as suggestions for improving activity outcomes in future programs. Interviews with other key stakeholders have also been conducted using a structured questionnaire as a guide during interviews. - Remote KIIs with LWP beneficiaries and stakeholders: The ET has conducted remote KIIs with selected water sector actors and the project beneficiaries (water utility Directors General and their staff, other water utilities employees, training recipients, municipalities, NGOs representatives, private sector entities, farmers trained on irrigation techniques, Lebanese Association of Industrialists, and others). KIIs have been held with the water utility Directors General and their staff to better understand their key problems and issues, how LWP assisted in addressing these issues, and what issues they still needed support in in the future. For other project beneficiaries, such as farmer groups, industries, municipalities, utilities customers, the KIIs helped inform on how effective LWP has been in working with these beneficiaries, and the outcomes and sustainability of the LWP activities. Focus Groups Discussions (FGD) have been avoided and replaced by KIIs due to anticipated challenges in administering this method online, in light with the COVID-19 pandemic. - Direct site observations DOs: Following the IRB recommendations regarding COVID-19 risk mitigation, the ET has undertaken field visits to activity sites (i.e., water establishment offices, water stations, wastewater treatment plants, agricultural sites, industries, etc.) utilizing a direct observation guide to obtain a first-hand experience of the activity outcomes. Where possible, KIIs have been combined with site visits and held in the staff offices, on demo sites, and at field sites. This allowed the ET to see first-hand the activities carried out by the project and their results. Given COVID-19 considerations and the large number of varied activities implemented as part of the LWP, field visits have been restricted as much as possible and sites to be visited have been selected based on the following: (1) one field visit per type of activity (installation of district and household meters and chlorinators; equipping well, rehabilitating reservoirs, installing water transmission line, pumping station; CW; irrigation network); (2) selected sites to be located in different areas across Lebanon, one per RWE (BMLWE, BWE, NLWE, SLWE, LRA); (3) selected sites to involve projects that 37 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT directly link to performance standards; and (4) visited sites to include successful interventions and challenging ones. ETHICAL CONSIDERATIONS Evaluators informed key informants that their comments will remain anonymous, and that any attribution will be indirect. The ET administered informed consent scripts and asked for respondents’ verbal consent to ensure that respondents understand the purpose and voluntary nature of the KII. All participants have been informed about any potential risks and been notified that they can withdraw from the evaluation at any time without penalty. KII participants have also been informed that there are no right or wrong answers, and there will be no negative consequences for sharing candid information about how DAI/LWP was designed or implemented. The ET addressed questions from the respondents and were prepared to stop KIIs should the participant request to do so. The safety and confidentiality of respondents is of utmost priority for SI. Additionally, meetings took place in safe locations. The ET produced reports that aggregate data and omit personal identifiers. No individual data have and will be shared outside the ET. DATA PREPARATION The privacy of all participants who took part in the data collection have been respected throughout the evaluation. All notes from KIIs and site visits, and any identifying information have been stored in password-protected folders that are only accessible to the ET and will not be made publicly available. SI used SharePoint to store all data and reports and it is accessible only to authorized personnel on this evaluation. SI use several data security measures, including the physical security of any hard-copy materials and the secure storage, transfer, destruction, and access of digital materials. DATA ANALYSIS METHOD The data gathered from desk review, the KIIs, and the site observations was used as the basis for data analysis. Techniques in data analysis included review of existing documentation obtained and categorizing of information obtained in the KIIs and DOIs, examining relationships between the different activity components, activities and achievements as revealed by stakeholders, and authenticating conclusions by triangulation of data obtained from different sources. Basic quantitative tabulations and means for data obtained on performance indicators and performance standard achievement have been done in Excel. In conformity with Automated Directives System (ADS) 201.3.5 and 201.3.6.8, the evaluation team used relevant monitoring data and feedback from beneficiaries already collected by implementing partners as data sources, when appropriate. The ET prepared a data source matrix (also called an evaluation matrix) that clarified how the data were collected. It included the evaluation questions, the evaluation tool(s), data source(s) and analysis plan for each question. An illustrative version of the matrix is provided in Table 3. This matrix ensured that a multitude of data sources were considered, and the team was able to use triangulation of data to answer each question with enriched validity. All the data collected from different sources were reviewed for reliability and validity, and findings were compared using multiple methods, data forms and sources of data. 38 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT aware and B- Clean-up Day (B) Desk review - Desk review of LWP desk review, and site proactive C- no of student research projects in (C) KIIs with two universities documents visit notes) towards the water (D) Survey results - Categorization by importance of D- Citizen perception survey results Primary Sources theme, region, water management and conservation? a-2016 pre-survey b- 2019 post-survey - KIIs with USAID and LWP key staff, technical experts, and LWP beneficiaries and gender - Triangulation of findings from different sources 1.3.1. Is the targeted A- no of PPP projects (A) - KIIs with WEs Secondary Sources Qualitative analysis private sector B- no of companies that engaged in - KIIs with LWP staff - Desk review of LWP (synthesis of KII, more involved water stewardship campaigns for (B) - KIIs with a bank and a TV station documents desk review, and site and engaged in their employees (C) -KIIs with WEs visit notes) water management and conservation initiatives? C-no of service contracts with RWEs to reduce NRW KIIs with LWP staff Primary Sources - KIIs with USAID and LWP key staff, technical experts, and LWP beneficiaries - Categorization by theme, region, and gender - Triangulation of findings from different sources 1.3.2. Are the farmers A- No. of farmers implementing water (A) & (B) KIIs with farmers Secondary Sources and industries conservation initiatives - KIIs with CCIAZ - Desk review of LWP more involved in B- Ability of farmers to sustain the - Site visit to farm documents water initiatives (C) & (D) KIIs with Industries conservation? C- No. of industries implementing water conservation initiatives D- Ability of industries to sustain the initiatives - KIIs with CCIAZ - Site visit to industry Primary Sources - KIIs with USAID and LWP key staff, technical experts, and LWP beneficiaries 2. Did LWP achieve the Refer to Performance Standard Table Refer to Performance Standard Table Refer to Performance performance standards Standard Table relevant to each of the Results A, B and C ? 41 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT LIMITATIONS AND BIASES USAID/Lebanon and other donors have provided assistance in the water sector in Lebanon for many years. The fact that multiple donors and implementers have been actively working in this sector presents challenges for making judgments about attribution. Given that this limitation is understood by USAID and the ET from the outset, the evaluation focused less on attribution and more on implementation, approach, and contributions of the LWP activity. The ET was aware of several noteworthy risks for data analysis and took every action possible to mitigate those risks: • Inability to attribute change to LWP: The evaluation questions were framed as comparisons (e.g., are they more effective, better equipped, have things improved), but the design and scope of this evaluation cannot permit robust analysis of change that is attributable to LWP alone. This is because the ET does not have a comparison group, and in most cases, baseline data are not available for comparison of pre-LWP conditions noted in the evaluation questions. The ET did its best to leverage both existing data sources and stakeholder perceptions and recalled about how things have changed to estimate change. The ET also addressed contextual changes in Lebanon external to the activity that might have influenced LWP outcomes, such as the October Revolution, the devaluation of the Lebanese currency, the Beirut Port Explosion on August 4, 2020, and the COVID-19 pandemic. • Response Bias: Response bias is the risk that key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. For example, a training participant may provide positive remarks about an activity because s/he would like to attend such training in the future. Respondents may believe that negative evaluation findings could result in reduced assistance from USAID/Lebanon. Alternatively, given the prevailing situation in the country and the overall revolt against public utilities and public services, respondents may be inclined towards providing overall negative feedback towards improved services, without differentiating between LWP-related activities and overall public services. The ET has been especially cognizant of the risk of response bias. In order to mitigate bias, at the beginning of each KII it has been clarified that the way the respondent provides answers will not influence the possibility to obtain new assistance in the future. Along similar lines, consent scripts at the beginning of all qualitative events explained to respondents that their answers are completely confidential, increasing confidence in the honesty of responses. Further, to the extent possible, qualitative events was separated by gender to reduce gender-related response bias. The ET additionally mitigated the risk of response bias by utilizing data triangulation to note discrepancies in responses through different data sources (KIIs, document review, including the pre- and post- LWP citizen perception surveys) and preparing follow-up questions for informants that encouraged more specific responses. For example, evaluators asked specific questions about desired alternative courses of actions and asked respondents for a specific example to back up certain responses. In addition, the ET addressed contextual changes in Lebanon external to the activity that might have influenced LWP outcomes, such as the October Revolution, the devaluation of the Lebanese currency, the Beirut Port Explosion on August 4, 2020, and the COVID-19 pandemic. • Remote Data Collection: The COVID-19 pandemic restricted the ability of the evaluation team to meet with individuals for KIIs. Because face-to-face contact was not possible in most cases, the team conducted remote interviews with key informants. Whenever the connection allowed, the online meetings were held with cameras on, to establish good rapport with respondents and capture their facial reactions to questions. In case of poor internet connection influencing the clarity of the conversation and the quality of the collected data, KIIs were 42 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT followed up with telephone calls to ascertain some collected data. Site visits (upon IRB approval) were also used for this purpose and helped overcome technology-related issues. DISSEMINATION AND UTILIZATION Following fieldwork, the ET presented initial findings and conclusions during an out-brief at USAID/Lebanon. The ET proposed to the Mission to present preliminary findings to LWP in a separate meeting. USAID will provide comments on this present draft report. The ET will revise the draft report to fully reflect USAID comments and suggestions, and the final evaluation report will be submitted within 15 days of receiving feedback from USAID on this draft report. The final approved evaluation report will be shared with USAID/Lebanon staff members. The report will follow USAID branding requirements. Evaluations at USAID should be transparent. Findings from evaluations will be shared as widely as possible with a commitment to full and active disclosure. ADS 201.3.5.15 operationalizes this principle by requiring Operating Units (OUs) to plan for evaluation dissemination: “Missions and Washington OUs must plan for dissemination and use of the planned evaluation. Missions or Washington OUs should consider how key partners and other development actors would best receive evaluation information and plan their communications products accordingly.” Upon instruction from USAID, SI will submit the final evaluation report (excluding any potentially procurement-sensitive information) to the DEC for dissemination among implementing partners, stakeholders, and the general public. The DEC submission must be within three months of USAID’s approval of the final report. Any personally identifying information of evaluation respondents will be appropriately de-identified from data and from the evaluation report per SI guidelines prior to submission to the DEC to ensure respondent confidentiality. As a part of the post Evaluation action plan, the ET will follow up to facilitate and enhance use of evaluation findings through one or more of the following steps: - Distribute the final approved evaluation report to DAI, LWP and other stakeholders at the discretion of USAID/Lebanon. - Upload the final approved evaluation report to the DEC for public access - Develop a utilization plan with primary intended users - Adapt and reformat findings for different audiences - Watch for emergent opportunities to reinforce findings - Continue to build evaluation capacity during the follow up process - Conduct a tracer survey. After the conclusion of the field work, the Evaluation Team presented initial findings and conclusions during a remote out-brief with USAID/Lebanon on April 19, 2021. 43 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT ANNEX D: DOCUMENTS REVIEWED SI: 1. LWP Midterm Evaluation Final Report 2. LWP Final Evaluation Scope of Work 3. Data Quality Assessment (DQA), Project Site Visit Reports (PSV), and Data Verification (DV) reports conducted by Social Impact. 4. PMSPL II construction management reports of LWP projects 5. USAID/Lebanon Citizen Perception Survey (CPS) - December 2019 USAID: 1. Country Development Cooperation Strategy (CDCS) USAID/Lebanon 2. USAID/Lebanon Statement of Objectives for the LWP Final Evaluation Assessment 3. RFTOP No.SOL-268-15-000002 Lebanon Water Project (LWP) Lebanon Water Project (LWP): 1. LWP Contract-AID-268-N-1-15-0001 and contract modifications (9) 2. Quarterly and Annual Reports with Selective Annexes (Y1-Q1, Q2, Q3, Y1-Annual Report, Y2-Q1, Q2, Q3, Y2-Annual Report, Y3-Q1, Q2, Q3, Y3-Annual Report, Y4-Q1,Q2,Q3, Y￾4 Annual Report, Y5-Q1,Q2,Q3, Y5-Annual Report, Y6-Q1,Q2) 3. Annual Work Plans with selective Annexes (Y1, Y2, Y3, Y4, Y5, Y6) 4. Monitoring and Evaluation Plan (May 2017) + updates 5. Citizen Perception and Satisfaction Survey Analysis (2016) 6. Gender Strategy RWE 1. GIS Assessment Report- Final 2. Final Financial Audit Manual 3. Final Performance Audit Manual 4. Fixed Assets Manual- Final 5. Internal Audit Manual 6. Water Quality Monitoring Methodology 7. Pump Operation and Maintenance Manual BWE 1. Business Plan 2016-2020 - Revised 2. Assets and Cost Centers 3. Fixed Assets Report 4. Internal Audits 5. Memorandum of Understanding LRA 1. Gauging Stations Summary-Final 2. Litani WQM-Final-clean 3. Assessment of Information Communication Technology 46 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT 4. Executive Plan 5. Yazda Post installation Audit Report 6. Irrigation Rapid Assessment Report NLWE 1. Fixed Assets Report 2. Business Plan 3. GIS Training 4. Assets and Cost Centers 5. Fixed Assets Report 6. Internal Audit 7. ERP Training Report SLWE 1. Assets Summary List 2. Business Plan 2016-2020 3. Risk Mgt and Executive Plans 4. 5-Y Master Plan 5. Fixed Assets Report 6. Manpower Planning 7. GIS Training 8. Assets and Cost Centers BMLWE 1. Fixed Assets Report 2. Assets and Cost Centers 3. Internal Audit 47 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT ANNEX E: DATA COLLECTION INSTRUMENTS KEY INFORMANT INTERVIEWS PROTOCOL *Names and position titles will be documented in a separate document to protect anonymity Key Informant Interview Data Collection Interview Code: Date of Interview: Time Start: Team: Time End: Interviewer(s): Location: Note taker(s): Category of Interviewee: No. of Interviewees _______ M _____ F______ Provided Consent: Y or N Notes to Facilitator/Interviewer The interview should be led by the interviewee, and only guided by the interviewer. The interview should be a relaxed conversation. The topics reported per respondent category (refer to separate files) present a list of topics to be covered during interviews. They also include concrete questions to guide the conversation and to cover each topic. The reason for focusing on topics rather than on questions is that the way in which a question would need to be formulated varies considerably depending on the interviewee’s specific knowledge, attitude and circumstances. The listed topics will allow the interviewer to achieve clear predetermined interview goals, but the actual questions will be adapted –as one does in a normal conversation- to the conversation tone, interviewee’s attitude, and knowledge. It is to be expected that the same key issue may naturally be transformed into several questions with one interviewee and only into one question with another. Informed Consent Agreement Good morning and thank you for meeting with us today. Our names are _______________ and we work for an independent organization called Social Impact. As mentioned during our interview request, we are working with USAID to conduct the final evaluation of the Lebanon Water Project LWP. The evaluation is intended to provide an informed assessment of LWP’s achievement to date and learn lessons for new projects if needed. It is very important for you to know that this study will not determine if you will receive any services/support/funding now or in the future. The way you answer our questions will not determine if you will receive any kind of government or NGO assistance now or in the future. There is no payment or direct benefit to you or your community for your participation in this study, besides toward the goal of improving this project or new project. Also, we do not anticipate any major risks to you for participating other than losing time you could spend on other things. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project. However, these documents can only tell us so much. We would like to speak with you today to hear about your experience, in your own words, in order to help 48 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT us better understand how these projects look and function “on the ground.” The interview will take approximately 40 minutes. It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable doing so. If you decide to take part, your responses will be kept strictly confidential. This means that your name will not be mentioned anywhere in the report, and will not be provided to anyone, including USAID/LWP or anyone in your community. Specifically: • We will collect individuals’ names and the organizations they belong to on a separate form. A description of the characteristics of the participants will be made available as an annex to the final evaluation report, but names and positions will not be disclosed nor will be associated to any particular findings or statements in the report. • We may include quotes from respondents in the evaluation report, but will not link individual names, organizations, or personally identifiable information to those quotes, unless express written consent is granted by the respondent. Should the team desire to use a particular quote, photograph, or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. • All data gathered will be used for the sole purposes of this evaluation and will not be shared with other audiences or used for any other purpose. Your participation in this key informant interview is voluntary and if you do not feel comfortable answering a particular question, please let us know and we will simply go on to the next question. You may also request to end the interview at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. Do you consent to participate in this interview? By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation and voluntarily consent to participate. Do you have any questions for us before we get started? If you have questions or concerns about the study after we leave today, you can contact Joanna Khater from Social Impact on 03-273 540 or on irb@socialimpact.com. Do you agree to participate? CONSENT STATEMENT: I understand and agree to participate in this study. Respondent provided consent: Yes_____ No_____ [The consent form was also translated into Arabic] 49 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT SAMPLE KII QUESTIONS BY RESPONDENT CATEGORY USAID: 1. What was the degree of enthusiasm & cooperation of WE DG’s throughout the implementation of the LWP? 2. What were the key issues faced by RWE that affected LWP performance? (staffing, budgeting) 3. How well has LWP met its deliverables in terms of quality, deadlines and budget? 4. How have the current crises (October revolution, LBP devaluation, COVID-19 pandemic, Beirut Explosion) affected LWP’s overall performance? 5. What were the successful initiatives that USAID would like to replicate in the future? 6. What were the challenging initiatives that USAID would prefer to avoid in the future? Water Establishments DGs and Staff: 1. How was the interaction with the USAID LWP project? 2. How was the project able to help you solve your problem/issues? 3. How many staff have received training from LWP? a. In ERP b. In GIS c. In Business Planning d. In Internal Auditing e. Laboratory Testing, etc. 4. Was there a change in the performance of your Establishment after the trainings received? a. Any difficulties encountered? b. Any success stories? 5. How many Lebanese citizens have benefitted from improved water services after LWP within your WE? How do you estimate this? 6. What is the increase of in the number of customers who receive and pay bills in the DA within your WE? 7. Have the water quality and laboratory testing improved within your WE? 8. To your knowledge was there an improvement in non-revenue water in the intervention areas? 9. Were you able to decrease O&M costs on the selected facilities? 10. What has been the impact of the financial audits and asset valuations provided by LWP? Do you consider yourself better equipped to manage your assets and facilities? 11. Success story to share 12. What Project activities worked/didn’t work? Why? a. Capacity Building/Management 50 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT b. Business Plans c. ERP d. Internal Audit e. Budget f. Cost Tariff Model g. Equipment h. Field projects i. Awareness campaigns 13. Have you discussed possible follow-up project with USAID? Other donors? LWP COP & Staff 1. To what extent did LWP’s interventions meet the objectives of the Activity as stated in the contract; more specifically: a. How did the various management plans and tools developed under LWP (Business Plans, ERPs, Asset Valuation & Management, Financial and Performance Audit, GIS for asset management and operation), and the associated trainings affect resource management capacity of the public water utilities? b. How did the construction activities, activities in intervention areas, management plans, as well as trainings provided to the WE staff affect the service delivery? c. How did the grants provided to NGOs/CSOs, as well as the implemented awareness and media campaigns contribute to civic engagement in water management and how did it affect citizens’ perception of water utilities’ role in providing a necessary and valuable service? i. How are you measuring the change brought forward by these grants? d. How did the initiatives implemented with the private companies influence the overall involvement of the private sector in Water Management? i. How did you measure change on this front? ii. How did you evaluate the involvement and engagement of the private companies? 2. How did the external contextual factors (Covid-19 pandemic, October Revolution, Devaluation of LBP, and others) affect the activity’s performance? If factors were identified to hinder the activity’s potential to achieve its objective, how those factors were mitigated/addressed? 3. What internal factors contributed to the achievement (or non-achievement) of the activity’s objective? 4. Can you provide us with success stories you recommend for replication in the future? 5. Were there major challenges, leading to failures in certain areas, can you please elaborate? 6. How are you ensuring the sustainability of the initiatives implemented by LWP following the end of the Activity? Are there mechanisms put in place to ensure that the WE will maintain the systems put in place by LWP and that the water savings techniques introduced by the Activity will continue to be used by the water users? 51 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT 7. What are your overall recommendations, based on the LWP experience, for the success of future activities in the water and wastewater sector? 8. To what extent was your involvement with municipalities for the implementation of project activities? 9. Some performance standards still not reached: a. (A1) 673,000 citizens benefitting from improved services b. (A2) 15% increase in the number of customers who receive and pay bills c. (A3) WQAPs- Enforced? 10. Some performance standards with no measurable data yet a. (A4) hours of lost service b. (A8) decrease in NRW c. (A10) reduction in O&M costs d. (B3) increase in customer satisfaction with service delivery 11. Questions on indicators (usually with M&E)- the achievement (non-achievement) of several targets have not been reported yet. Some are indicated to be reported in Y6. Are there interim data that can be shared with us to help assess achievement of these indicators and subsequently some of the standards? a. Assessing the actual percentage of water customers indicating satisfaction with their water service? How is this being done? b. The number of entities with enhanced capacities? c. Increase in the % of cost recovery in DA? d. What is the number of new subscribers to the public water services (the target is 2,700; the number achieved is 1,111 as per the end of year 5) e. Decrease in the % of non-revenue water in DA? f. Decrease in the % of lost service hours in DA? g. What is the unit for the target volume to be treated (51,000…) h. Decrease in the % of technical losses of water in DA? 12. Are the following pumping station operating? If not, what is the expected starting date for their operation? Are there any problems that might delay the start of their operation? a. Chekka/Jradeh b. Rachaaine/Rahbe c. Ghazzeh/Barr Elias 13. What were the results of the Farmer’s Field Days and Clean-up Days? Are these activities improving the citizens’ perception? Farmers: 1. What is the name of your organization? 52 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT 2. Who do you represent? 3. Characteristics of system; a. Major problems in system b. Link with LRA c. Women farmers? Role of women in agriculture? 4. Roles & resp of LRA: a. Who is your contact with LRA? b. Do you work with MOA or other organizations? c. Are you satisfied with your relationship with LRA? d. Do you feel that your relationship with LRA changed after the LWP initiative? 5. What training and/or support have you received from USAID LWP Project? 6. Are you confident you can proceed on your own? a. What elements learned during trainings with LWP are you applying? b. What water conservation and management practices are you applying? 7. What do you need to be successful? a. What type of support are you receiving from the private sector in terms of water management and conservation, if any? b. What are the steps that you took to improve water management and conservation on your lands? What do you still see room for improvement? Industries: 1. What is the name of the Factory? 2. What is your main activity that requires heavy water consumption? 3. How did you used to procure water? and how did you used to dispose sludge water? 4. What are the challenges associated to water supply and water discharge? 5. What did the LWP installed on your factory? 6. What changes have you noticed? 7. Are you encountering difficulties in running and maintaining the system? 8. Do you recommend installing this system to other peers in the future? 53 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT ANNEX F: DETAILS OF LWP MAJOR ACTIVITIES Demonstration Areas: (DAs) The DAs were selected to be model supply service areas, having closed systems that are fully metered, with a potential of 24/7 household water availability, minimum NRW losses (less than 25 percent), and a high fee collection rate (greater than 85 percent). The selected DAs along with a summary of the implemented activities included: - SLWE- Jezzine (27 villages): rehabilitation of 3 springs, 4 wells, 2 pumping stations, and 11 reservoirs, supply and installation of 6 chlorination systems, construction of 3 new reservoirs, of 10 Km of transmission lines, construction/ and or rehabilitation of 40 Km distribution networks, installation of 65 source and district meters, and 3,900 smart customer meters, rehabilitation of service center - North Lebanon Water Establishment (NLWE)- Koura (5 villages): installation of 18 new pumps in the Jradeh pumping station and rehabilitation of well, construction of Qalhat elevated reservoir, construction of a 7-Km distribution network, rehabilitation of Fiaa-Jradeh 6 Km transmission line, supply and installation of 1 chlorination system, installation of 25 source and district, and 1000 customer meters, preparation of SCADA foundation - Beirut and Mount Lebanon Water Establishment (BMLWE)- Metn (27 villages): rehabilitation of 10 reservoirs and installation of 55 source and 24 district meters, 2 chlorination systems, and 28,000 customer meters - BWE- Haouch El Omara, Zahleh (1 village): Installation of 40 district meters and 10 chlorinators Projects Outside Demonstration Areas (PODAs): - BWE/ Ghazzeh: Well equipping, pumping station and elevated reservoir, solar plant, connection pipeline. No bills collected from this area. - BWE/ Baaloul 2.8 Km network to connect Baaloul households to the network. - BWE/ Barr Elias well equipping, pumping station, connecting pipeline to the UNICEF intermediate reservoir, rehabilitation of existing pump line to the Barr Elias village, sector zoning. - NLWE/ Rahbeh: Deepening of an existing well, well equipping, transmission line from well to existing reservoir, rehabilitation of existing reservoir. - NLWE/ Rachaine: Existing reservoir rehabilitation, pumping station construction, lift line from pumping station to reservoir, transmission line from spring to pump station. - NLWE/ Bazoun: Equipping the existing well, construction of control room, and construction of 1,000 Lm/ pumping line from the Bazaoun well to the Bazaoun existing reservoir. This has not been completed yet. - BMLWE/ Chabrouh: Roll-out of drip irrigation systems in farmland located in the village of Faraiya and the Chabrouh dam area. Activities to advance citizens’ perception of the water utility's roles and to promote water conservation: 54 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT - Creation of a website for MOEW and for SLWE. - Production of 4 short corporate videos with RWEs on water distribution, water conservation, infrastructure work, and customer service centers (CSC) to promote their work to customers, improve their perception by customers, and enhance awareness as to water conservation. - Production of 6 short awareness videos and one long flagship 3D video that went live on the RWEs’ Facebook, Instagram, YouTube, and Twitter pages reaching hundreds of thousands of views. LWP disseminated the 3D video through its own efforts and through various NGO partners who have been working with schools in the Bekaa, North Lebanon, and Metn. - Assistance to the communication staff at BMLWE, BWE, NLWE, and SLWE through enhanced collaboration and the development of an informational pamphlet/booklet on water meters. This pamphlet was distributed in the DAs to provide accurate information to customers of the RWEs as construction works and meters installation went forward. - Assistance to NLWE and SLWE with their Facebook campaigns on maintenance, water quality, and hotlines availability and boosting of their Facebook posts. - Preparation of several media campaigns. Short Message Service (SMS) text campaign was aiming to remind water establishment subscribers to pay their bills (at BWE, SLWE and NLWE). In the Bekaa, this was coupled with banners hung in municipalities, a radio campaign to encourage citizens to pay their bills, and billboards promoting bill payments and water conservation along all the main Bekaa roads, running for about 10 days. - Establishment of a diverse and wide-ranging innovation and research grants program, covering communities, agriculture, and industry. There were eight grantees from the civil society: o Grantee 1 worked on encouraging water conservation and community liaison through awareness-raising in Haouch el Oumara, Bekaa. o Grantee 2 worked on improving water availability, quality, and uses in agriculture and industry in the Bekaa (14 dairy industries, 8 grape farmers, 2 greenhouse and open-air farmers). o Grantee 3 worked on raising awareness among the local communities in Jezzine and national authorities about sustainable use and management of water. o Grantee 4 worked in Qobayat, it connected a hill-lake to farmlands in the Akkar area and created a water governance committee to ensure that water is being used properly. o Grantee 5 promoted NLWE’s image through awareness campaigns in the Koura district focusing on the duties of consumers and citizens. o Grantee 6 raised awareness about water conservation and drinking quality among targeted communities (Beqaa and Northern Lebanon), working with cooperatives, households, and schools, and reached 2,190 students including 30 Syrian refugee children. o Grantee 7 worked on raising awareness among youth and adults (students and their teachers) about sustainable water use and management. This included the development of 9 2D videos on water and wastewater. o Grantee 8 contributed to 13 episodes on water and energy conservation broadcasted during primetime (before the 8:00 pm news) on a national TV station, reaching 1.6 million viewers. 55 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT ANNEX G: CONFLICT OF INTEREST DISCLOSURES [redacted] 56 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT ANNEX H: ET MEMBERS For this performance evaluation, the ET consisted of a Local Team Leader/Water Sector Expert, two Local Team Members/Water Sector experts, with technical support from PMSPL II evaluation team. Other members of the SI Lebanon office provided ad-hoc inputs. The ET was supported by SI’s HQ members who provided overall guidance on the evaluation and quality control on the deliverables. A short description of each team member and their position is below, followed by a summary in Table 4. The Team Leader (TL) for this evaluation is Dr. Rania Maroun, a senior environmental and water resources specialist, with a strong inter-disciplinary background and over 20 years’ experience in water resources management, water, sanitation and hygiene (WASH), and environmental and social assessments, for a wide range of donors, including USAID, UNICEF, UNDP, and World Bank. As team leader, Dr. Maroun is responsible for evaluation design and implementation. Mr. Karim Amatouri, Lebanon expert in water and wastewater related design infrastructure, supported the ET in conducting the evaluation, including developing gender sensitive data collection tools, conducting KII, direct observations, and FGDs, and participated in the analysis and drafting of the evaluation report, with focus on the infrastructure design components of the LWP. Mr. Georges Gharios, expert in general water and irrigation sector situation of Lebanon, the national water policies, and the institutional context, supported the ET in conducting the evaluation, including developing gender sensitive data collection tools, conducting KII, direct observations, and FGDs, and participated in the analysis and drafting of the evaluation report, with focus on the farmers and industries interventions of the LWP. Ms. Joanna Khater, Evaluation Project Manager and member of PMSPLII staff, supported the ET in conducting the evaluation. She oversaw the conduct of KIIs, and other field work tasks, data analysis, and reviewed report preparation. She has participated in multiple evaluations of the water sector in Lebanon. Mr. Harvey Herr and Ms. Carine Khoury are respectively the COP and the DCOP of PMSPL II and both provided ad hoc inputs during the whole evaluation. Ms. Leslie Greene Hodel, Technical Director at SI HQ in Washington, DC, provided overall guidance on the evaluation and quality control on the deliverables. PMSPL II resident staff and HO staff participated in drafting, reviewing, and editing the present draft final report before circulating to the stakeholders for comment and review. Preparation of the evaluation report is the responsibility of the team leader/technical expert. Final editorial responsibility and quality control for the report is with SI. Comments received are to be addressed by the team leader and the technical experts supported by the PMSPL II staff. 57 | LWP FINAL PERFORMANCE EVALUATION USAID.GOV MAY 2021 – FINAL REPORT Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA 22201 Suite 1000 Tel: (703) 465-1884 Fax: (703) 465-1888 www.socialimpact.com