March 31, 2015 This publication was produced for review by the United States Agency for International Development by International Development Group LLC. LEARN EVALUATION AND ANALYSIS PROJECT-II Liberia Municipal Water Project Mid-Term Performance Evaluation Final Evaluation Report Learning, Evaluation and Analysis Project-II (LEAP-II) Liberia Municipal Water Project (LMWP) Mid-Term Performance Evaluation Final Evaluation Report   Contract Number: AID-OAA-I-12-00042/AID-OAA-TO-14-00046 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Page 1 EXECUTIVE SUMMARY Introduction USAID is currently implementing the four-year Liberia Municipal Water Project (LMWP), a unique project opportunity and a foundational investment that not only addresses urban water supply infrastructure needs, but also sets the stage for a community-based and a decentralized approach for water supply management that can foster sustainable operation and maintenance (O&M) for the recommended capital improvements. The primary objective of LMWP is to support the design, tendering, execution and operation of water supply infrastructure improvements in the three Liberian county capitals – Robertsport (Grand Cape Mount County), Sanniquellie (Nimba County), and Voinjama (Lofa County). The contractor for LMWP, Tetra Tech, is tasked with assisting local and national authorities in developing plans for water supply and sanitation improvements, overseeing construction of this capital improvement project (CIP), supporting initial operations of water supply infrastructure improvements, and establishing local capability to sustainably operate and maintain constructed water supply systems. The CIP is separate and distinct from LMWP, and will be contracted by USAID separately. The primary design criteria for the improved infrastructure is that by the end of LMWP, over 90 percent of the population in each city will have improved water supply access1 and improved infrastructure will be handed over to a locally-based operator capable of financially and technically sustaining water supply services. USAID is embarking on a mid-term evaluation to determine whether the assistance it has provided through LMWP is meeting its objectives, and if adjustments are needed as USAID advances plans to begin capital works in line with the plans developed by LMWP for the three cities. Accordingly, the evaluation addressed the following three key areas: (1) Effectiveness of infrastructure planning/construction oversight approach, (2) Effectiveness of institutional framework and capacity building, and (3) Overall project positioning and strategy for phase-out of USAID assistance. The evaluation team consisted of a Team Leader/Utilities Expert, a Utilities Expert, and a Liberian Administrative/Logistics Expert. Purpose and Study Questions of the Evaluation The purpose of the evaluation was to provide a detailed picture of the major accomplishments, strengths and weaknesses of LMWP since inception, indicating any recommended changes in the project’s implementation approach and USAID’s project design/programming approach to assure successful completion of urban water improvements in line with LMWP’s objectives. Furthermore, the evaluation was designed to include specific recommendations for successful completion of the four-year base contract in September 2015 and the two option years in the event they are exercised by USAID, and recommendations on any key issues/changes regarding the implementation approach and programmatic priorities for the said option years. The evaluation also sought to identify priority areas that should be the focus of possible future programming in the Liberian urban water supply sector in support of the Mission’s overall development objectives outlined in the Liberia Country Development Cooperation Strategy (CDCS) for 2013-17. 1 “Improved water supply access” is defined as regular household access to a water source, a distribution system, or a delivery point, which by the nature of its design and construction is likely to protect the water source from external contamination, in particular from fecal matter, and can be reached by the household in a round trip of 30 minutes or less. Improved water supply sources are: piped water into dwelling, plot, or yard; public tap/standpipe; tube well/borehole; protected dug well; protected spring; or rainwater collection. Unimproved water supply sources are: unprotected dug well; unprotected spring; cart with small tank/drum; tanker truck; surface water (river, dam, lake, pond, stream, canal, irrigation channel); and, bottled water. Page 2 The primary audience for this evaluation is USAID for the purposes of validating the status of planning (both infrastructure design, and viability of plans to establish sustainable Government of Liberia (GOL) management entities) prior to initiation of the large-scale construction phase. Secondary audiences are USAID’s GOL partners and stakeholders, principally the Liberia Water and Sewer Corporation (LWSC), to provide recommendations related to their role in ensuring successful conclusion of the project and handover of infrastructure to financially and operationally capable local entities. The evaluation addressed the following three key questions:  Effectiveness of infrastructure planning/construction oversight approach: To what extent was appropriate due diligence done in the design and how well is the project positioned (in terms of management/staffing structure) to support the construction phase?  Effectiveness of institutional framework and capacity building: How effective is LMWP's current and proposed institutional and capacity-building approach in achieving the stated objectives?  Overall project positioning and strategy for phase-out of USAID assistance. Given delays in the construction design and implementation phase, the original timeframe underlying the project design (e.g. finish construction during LMWP 4-year base period, and then support O&M for 2 additional years) is no longer viable and it is likely that construction will just be finishing by the end of the LMWP's sixth year. Furthermore, much uncertainty remains around what exactly it will take for the outstations to reach operational cost recovery and how long this will take - estimates range from 1-3 years. How should USAID planning proceed? Approach and Methodology The approach to the LMWP mid-term evaluation included a desk review, key informant interviews, and a field visit to Liberia from November 24 to December 15, 2014. Field activities included visits to the cities of Robertsport, where components of LMWP have been and are currently being implemented, and Kakata, where LMWP has established a Local Steering Committee (LSC) as a model for local stakeholder engagement in managing an outstation outside LMWP’s mandate. The key informant interviews covered a wide array of sources, but were mainly focused on LWMP key stakeholders including USAID, LWSC, county and city authorities and the LSC in Robertsport, LWSC office in the city of Kakata, and Tetra Tech, the LWMP contractor. The evaluation team, in consultation with USAID/Liberia, utilized a Risk Assessment Framework (RAF) to evaluate potential risks and impacts of recommended CIP improvements including analysis of the O&M cost recovery, capacity building and institutional strengthening. Evidence, Findings and Analysis of the Study Questions Effectiveness of Infrastructure Planning/Construction Oversight Approach To what extent was appropriate due diligence done in the design and how well is the project positioned (in terms of management/staffing structure) to support the construction phase? LMWP has designed the CIP to meet the primary design criteria that by the end of LMWP (1) over 90 percent of the population in each city will have improved water supply access, and (2) improved Page 3 infrastructure will be handed over to a locally-based operator capable of financially and technically sustaining water supply services: Appropriate due diligence conducted including a comprehensive situational analysis. LMWP developed plans for water supply in a very structured and logical manner. LMWP conducted a comprehensive situational analysis including reviews and assessments of existing infrastructure and water sources, assessments of capacity at both the national and local levels, and the socio-economic environment when it comes to water usage, practices, and willingness to pay. The situational analysis revealed that LWSC has no presence locally except for a caretaker to oversee the remnants of the facilities for encroachment and bush clearing. A design that will result in improved water access for over 90 percent of the population in each target city. From the situational analysis, LMWP developed a baseline and the basis of design reports detailing what was to be improved, the water sources, types of treatment, new facilities, etc. The design took into account treatment options, distribution, and the energy source which is a hybrid system of solar power and diesel generators. LMWP evaluated a range of investment options to select from among alternative options, a CIP that best fits the USAID budget constraints while ensuring the key objectives of the project are met. LMWP conducted an informal value engineering (VE) analysis both internally within Tetra Tech and externally with relevant stakeholders, which resulted in design modifications. This process included engineering review by five USAID engineers, some of whom reviewed multiple iterations of drafts and provided multiple rounds of comments. Improved infrastructure will be handed over to a locally-based operator capable of financially and technically sustaining water supply services. LMWP has made considerable progress toward this goal. LMWP determined that local governments have no role in construction, management, and operation of piped water services. This led LMWP to the establish of LSCs and a national Transition Working Group (TWG) whose key function was to explore various options and provide recommendations on the appropriate institutional framework for the management of the outstations based on international best practices and local realities. LMWP introduced the idea of an internally delegated area management contract (IDAMC) to allow LWSC to “outsource” operation of the outstations to a chosen entity. The IDAMC has been fully embraced by LWSC which is an important accomplishment of LMWP. The framework is supported by memoranda of understanding (MOUs) drafted by LMWP which describe the roles and responsibilities for successfully completing the project, and serve as a vehicle to ensure the GOL is aware of and plans for their role in planning for the long-term viability of the systems after USAID phase-out. While the proposed CIP design appears to be well-suited to achieve the project objectives in terms of effectiveness of institutional framework and capacity building, appropriate due diligence conducted including the comprehensive situational analysis for the design, and overall project positioning, the following areas of the design need further exploration and evaluation: Assessment of existing infrastructure asset inventory (condition assessment). The existing infrastructure is over 20 years old. LMWP informed the evaluation team that no design drawings or as￾built drawings are available for the target cities’ water infrastructure. LWSC has not provided drawings and LMWP was unable to contact the original German designers, which poses significant challenges to conducting a structural analysis. As a result, LMWP assessed the condition of the existing infrastructure for inclusion in the CIP (e.g., elevated water concrete tanks that are about 100 feet in height) at the three project sites primarily via visual inspections. Incorporating the existing infrastructure in the CIP could result in a catastrophic failure if there are structural issues that a visual inspection cannot capture. A catastrophic failure of the vertical structures such as the elevated tanks could result in a prolonged loss of Page 4 and disruption to service, and may potentially cause injuries and human loss depending on the timing and mode of failure and if the failure occurs when people are in proximity to the structure. Catastrophic failure of such assets can occur because of insufficient design criteria, poor quality construction, and pre￾mature aging of the infrastructure due to lack of routine inspection and maintenance that may collectively contribute to a poor or non-response mechanism to dynamic loads such as seismic activities. A more elaborate application of condition assessment tools and techniques such as mathematical modeling and when and if applicable destructive testing to structural components (i.e., vertical structures such as the elevated tanks) is warranted to ensure that these types of infrastructure are integrated into proposed new assets only when their structural and functional conditions are compatible to the design and construction criteria of the new proposed infrastructure. Application of formal value engineering/value analysis (VE/VA). LMWP informed the evaluation team that it did not formally apply an analytical tool such as VE/VA, as is suggested by international best practices, in arriving at the proposed alternative investment strategies and design schemes including their associated costs and trade-offs. While VE/VA is typically applied in the design process, the evaluation team recommends applying VE/VA at the construction phase. The costs for design and construction materials are a large portion of the construction budget, and applying VE/VA at the construction phase will potentially maximize the value for the budget. VE/VA continues during construction because a construction contractor's practical experience and purchase options can often generate substantial savings. During the construction phase, a construction contractor can propose a value engineering change to construction requirements, materials, or methods, and the construction contractor shares in the savings. The change may reduce the cost of construction or the life-cycle cost of the project without compromising performance, design quality, safety, functionality, or ease of maintenance. Typically, the construction contractor gets a share of construction cost savings for fixed price contracts, but can be different for incentive-based contracts. Ambitious assumptions around the time required to reach O&M cost recovery. The ability of the systems, once operational, to reach O&M cost recovery will be critical for sustainability. In its financial and economic analysis, LMWP employed appropriate tools and methodologies. However, LMWP informed the evaluation team that it did not perform any sensitivity analysis to the financial projections and analysis for O&M cost recovery to show what happens if the CIP does not meet some of the overly optimistic assumptions. While the assumptions in LMWP’s projections related to operational costs are reasonable, several of the assumptions related to revenue growth are ambitious and unreasonable:  Payback period: The payback period (O&M cost recovery) is overly optimistic and ambitious, given that it is assumed that demand approximately quadruples in the first three or four years depending on location. There is no justifiable reason for the sudden increase in demand provided. The assumed collection rate of 90 percent is reasonable given that the systems will be cash-and￾carry; however, the 90 percent recovery of accounts in arrears appears aggressive as experience from LWSC suggests otherwise.  Affordability analysis: The evaluation team conducted affordability analyses for Voinjama, Robertsport and Sanniquellie from 2016-2031. All three analyses show that the household water bill per month exceeds the willingness to pay sometime over the next five years making the projected revenues unrealistic.  Risk analysis and mitigation: LMWP did not perform any risk analysis or attendant mitigation strategies. Cost benefit analysis (CBA) findings include: Page 5  Discount rate: The basis for the discount rate is not clear. The costs are discounted at three percent while the benefits are discounted at two percent. An internal memo from USAID2 shows the agency typically uses a higher rate. Using a higher discount rate will lower the cost-benefit ratios which appear unreasonable due to ambitious benefits projections described below.  Benefit projections: LMWP identifies mostly social benefits (opportunity costs) which are difficult to quantify and can swing widely depending on the assumptions. LMWP’s approach is grouping together disparate impacts (fewer hospital visits, fewer lost work hours, increased economic activity due to extended life expectancy, etc.) and adding the “benefits” to arrive at the project benefit values. These impacts may or may not be additive but there is no sensitivity to see what variations in the assumption will bring.  Hurdle rate: There is no hurdle rate to meet (internal rate of return) for the project. Given these factors, projected O&M cost recovery period of one to three years is not realistic. The evaluation team conducted an illustrative sensitivity analysis for Voinjama. The evaluation team used less aggressive assumptions including a more gradual rise in demand and revenue (adjusting demand volume to 20 percent per year from 2018 instead of quadrupling demand, and reducing head office allocation payments by 40 percent). This resulted in an O&M cost recovery of seven years. LMWP needs to conduct sensitivity analyses for all the three cities to determine what set of options, e.g. growing demand gradually, limiting head office allocation payments, revisiting other O&M cost components etc., works best for O&M cost recovery. Management/staffing structure to support the construction phase. The project resources for capacity building appear adequate but the resources including staffing for the construction phase are inadequate. Given the lack of adequate capacity at LWSC, and in anticipation of procurement and construction challenges in Liberia with limited construction capacity particularly as Liberia continues to deal with the recent Ebola outbreak, the proposed construction oversight staffing plan, although still evolving and being revised, is inadequate to minimize potential risk of timely and quality construction and completion of the proposed CIP. Effectiveness of Institutional Framework and Capacity Building How effective is LMWP's current and proposed institutional and capacity-building approach in achieving the stated objectives? LMWP’s current and proposed institutional and capacity-building approach are effective for achieving the stated objective of ensuring that improved infrastructure will be handed over to locally-based management technically and financially able to sustain water supply service improvements in the three cities. Strong institutional framework. LMWP successfully established a strong institutional framework. LMWP introduced the idea of an Internally Delegated Area Management Contract (IDAMC) to allow LWSC to “outsource” operation of the outstations to a chosen entity. The framework is supported by memoranda of understanding (MOUs) drafted by LMWP between: (1) LMWP and the three targeted cities, (2) LMWP and the LWSC, (3) USAID and Ministry of Finance (MOF) (which is in draft form). The main purpose of the MOUs is to describe roles and responsibilities for successfully completing the project, and serve as a vehicle to ensure the GOL is aware of and plans for their role in planning for the long-term viability of the systems after USAID phase-out. The MOUs drafted by LMWP that support the IDAMC detail the background and context, the objectives, the responsibilities of the parties involved, and show the necessary information typical of MOUs. The signing of the city specific MOUs indicates good 2 CBA Review by USAID EG Jan 2014 – comments, Page 6-7. Page 6 faith from the stakeholders – LWSC, Cities, LMWP – to see the CIP through. While the IDAMC and overall the agreements do provide a viable framework the following issues need to be addressed:  The city-specific MOUs are valid through September 30, 2015, which corresponds to the base period of the LMWP contract. This date should be modified to at a minimum cover the construction phase.  The draft USAID-MOF MOU should explicitly address the party responsible for O&M support if the projected O&M recovery period does not materialize prior to the phase-out of USAID assistance post-construction. While the MOU implicitly states that the GOL will provide the necessary subsidies to operate the systems until O&M cost recovery is achieved, the evaluation team recommends that the MOU explicitly state this.  The draft USAID-MOF MOU shows a Robertsport tariff of US$0.045 per gallon and a common Sanniquellie and Voinjama tariff of US0.035 per gallon in the computations for O&M costs and subsidies. LMWP used a common tariff of US$0.04 in its financial projections for the cities and should ensure the computations use the same inputs. The MOF has a state owned enterprise (SOE) monitoring unit which monitors the operational, technical, and financial details of the SOEs and holds them responsible and accountable. However, the LWSC does not fall under this unit. To improve the effectiveness of the institutional framework, it is recommended that the MOF hold LWSC to the same level of accountability. Effective capacity building at LSCs and LWSC. The establishment of and capacity building of the LSCs at the Robertsport and Kakata outstations have been among the key achievements of LMWP toward establishing an institutional framework for outstation management. In addition to improving the management capacity of the LSCs, LMWP has fostered pride in local ownership, and the LSCs are willing to take tariff challenges/changes to the citizens directly as well as to market water. These efforts should be expanded to other outstations at the appropriate time. The evaluation team also recommends that LMWP continue to expand its capacity building efforts to include asset management aspects of the O&M: project management, capacity building and training by the construction contractor, building decision support systems, and vocational training. LMWP has the appropriate tools and plans to build the needed capacity at LWSC. LMWP can replicate the success of the LSCs capacity building effort at LWSC. A major hurdle has been LWSC’s inability to provide staff for training as the LWSC is understaffed and staff often do not have time for training and required to perform their duties instead. LWSC needs additional resources to strengthen its technical staff. The support must come from LMWP or GOL (or both). Additional analytical work is required on market segmentation, costing of future network improvements, procurements mechanisms, etc. On customer segmentation, LWSC must have the ability to identify which market segments to pursue to boost its revenue. It will need tools that provide decision support systems such as dashboards of key performance indicators to facilitate decision making. There is a need for a communications specialist at the LWSC, which currently does not exist, to provide education and awareness, (e.g., the benefits of well water versus piped water) after the CIP system has been installed. This will be an important part of LWSC’s marketing drive to boost revenues for its services as well as those of the outstations. Procurement continues to be a challenge at LWSC even without the added burden of the outstations. LWSC would thus benefit from procurement mechanisms and systems with the appropriate training. LMWP’s plan to monitor and asses the progress of its capacity-building work is still evolving. LMWP has developed capacity building tools which have not yet been fully aligned to the capacity building work Page 7 plan. The capacity building plan has to be assessed first to establish a baseline before progress can be measured. Overall Project Positioning and Strategy for Phase-out of USAID assistance Given delays in the construction design and implementation phase, the original timeframe underlying the project design (e.g. finish construction during LMWP 4-year base period, and then support O&M for 2 additional years) is no longer viable and it is likely that construction will just be finishing by the end of the LMWP's sixth year. Furthermore, much uncertainty remains around what exactly it will take for the outstations to reach operational cost recovery and how long this will take - estimates range from 1-3 years. How should USAID planning proceed? Lack of qualified and responsive local contractors, and environmental uncertainty regarding health, transportation infrastructure, etc., have already exacerbated the uncertainty and additional cost associated with construction activities in Liberia. The application of VE/VA prior to and during construction, a two￾stage procurement process, performance based contracting (PBC) with provisions for incentives, and evaluating and examining project fast tracking are among the options that when employed, can minimize the risk of construction cost overruns. LWSC figures prominently in the success of the CIP. It is facing enormous challenges in providing the required services under its mandates. The current performance data suggest that LWSC can, with adequate capacity and policy support and over the course of LMWP implementation including USAID’s two-year O&M support, successfully ensure that the new systems reach O&M cost recovery. To this end, it is critical to establish and support a functional and professional monitoring and evaluation (M&E) unit at LWSC. LWSC also would benefit from continued institutional capacity support which could be accomplished through utility-to-utility training via twinning programs. To improve operational efficiency, LWSC must establish decision support systems (DSS) and the related tools. LWSC will require additional resources from GOL or LMWP (or both) to be able to provide construction support and to fulfill the objectives in its MOU. Over the long term, LWSC would benefit from a training center to facilitate training at a minimum for technicians, supervisors and mid-level managers. LMWP should explore supporting the revitalization of the existing training center. The CIP does not focus on sewerage. Therefore, a complementary investment in sewage is necessary once the CIP is operational for approximately five years. To increase the likelihood of the GOL meeting objectives around operational capacity and cost recovery, USAID needs to communicate to the MOF how long it is willing to support O&M. This is critical to include in the USAID-MOF MOU as the MOU should be the overarching framework defining each party’s obligations. Recommendations The evaluation team, in consultation with USAID, identified three levels of priority recommendations to reflect the degrees and levels (1-3) of importance (high-low) of the proposed recommendations to allow timely and concurrent, yet priority-driven improvements to the project. Determination of the priority level is based on the severity of impact on cost, schedule, timing for implementation, sustainability, safety and technical performance. Page 8 Priority Level 1 1. Conduct and apply the principles of VE/VA to the design and construction of proposed improvements prior to and during construction. For any given project, a typical VE/VA study reveals the optimum blend of scheduling, performance, constructability, maintainability, environmental awareness, safety, and cost effectiveness. Applying VE/VA during construction will potentially maximize the value for the budget as the costs for design and construction materials are a large portion of construction budget. 2. Re-evaluate the economic and financial analysis including identifying options through sensitivity analysis. A “financial analysis” usually focuses narrowly on the costs incurred to build and operate a project, and the revenues that then accrue to the project implementers. An “economic analysis” adds to this more broadly societal benefits such as job creation or water access, and societal costs such as environmental impact. The revenue projections for the CIP are aggressive and overly optimistic. The evaluation team recommends that LMWP revisit the assumptions, discount rates, period to recovery of O&M costs, etc. and analyze various scenarios to develop options for functionality at optimal cost. The CBA should be revised to adjust the previous projected 4:1 benefit-cost ratios to reflect adjusted benefit values. 3. Conduct a two-stage procurement process. A two-stage procurement process encourages a contractor to convert from a low-risk, cost-plus-fee reimbursable pre-contract service agreement to a cost-certain and time-certain contract in which the contractor bears the design and construction risk. 4. Institute performance-based contracting (PBC) with provisions for incentives. The concept of PBC is centered on a contract instrument that defines performance expectations in terms of outcomes or results as opposed to methods, processes, systems or broad categories of work activity. The PBC concept describes the work in terms of what the required output is supposed to be rather than how the work is to be accomplished. Another component of PBC is that the contractor bears responsibility for assuring quality performance. Based on the incentive structure, a PBC allows the contractor to employ innovative techniques to yield cost/time savings. 5. Evaluate and examine project fast tracking and crashing the construction schedule. Fast tracking and crashing are two project compression techniques used to shorten project schedules. In light of timing challenges and citizens’ anxiety about getting access to potable water, LMWP should check these project compression techniques to see if the project could be completed faster at minimal or no additional cost. 6. Reevaluate and revisit resource planning including the proposed staffing plan and the project/construction management tools after incorporating VE/VA process to also include construction procurement scenarios/options. LMWP’s proposed staffing for the construction phase appears inadequate for the anticipated construction challenges. The evaluation team recommends that LMWP re-evaluate its resource planning post a VE/VA study to ensure a robust staffing plan. 7. Support LWSC to further recognize and overcome external and internal institutional challenges such as staffing, marketing, customer base, efficient billing, and revenue collections - LWSC should champion the execution of the provisions in all the agreements. The success of the CIP post construction hinges on LWSC’s ability to manage its operations with limited resources. Helping LWSC with systems to improve revenue generation while improving efficiency will enhance its ability to manage the CIP. Page 9 8. Utilize local construction capacity where feasible particularly for pipe laying. The evaluation team and LMWP jointly believe that there are aspects of the project, such as pipe laying, that local contractors can reasonably handle to minimize cost and provide an opportunity for capacity building. 9. Revise, refine and expand the current capacity building work plan to include project management, capacity building and training by construction contractor, building decision support systems, including through approaches such as vocational training and revitalizing the existing Training Center. LMWP’s capacity building efforts have garnered positive reviews and should be expanded. The current plan catalogued several areas for support and the needed intervention and LMWP developed its initial plan accordingly. However, LMWP should enhance its capacity building plan by including on-the-job training, training by the contractor, building decision support systems, and should explore revitalizing the existing training center to facilitate training technicians, supervisors and some middle managers. 10. The MOU with MOF and LWSC should be the overarching framework to hold LWSC responsive and accountable. The draft USAID-GOL MOU details the objectives and responsibilities of all parties – LWSC, MOF and USAID. However, the MOU does not explicitly state who will be responsible for providing the needed financial support should the O&M cost recovery not occur in the projected time. The evaluation team recommends that the MOU be revised to address this. Priority Level 2 1. On-the-Job Training (OJT) construction management capacity building (i.e., learning while doing). The direct benefits of OJT are that the trainees will be very familiar with what goes into a water infrastructure system, running and maintaining the system, and troubleshooting problems. The experience will facilitate running O&M after the construction. LMWP and the contractor should work on modalities to explore the best way to execute this. 2. Support establishing a functional and professional M&E unit at LWSC. The LWSC lacks an M&E unit to provide the tools and performance reports management needs to run and improve operations. A professional M&E unit will provide an avenue for continuous learning and the details/tools for LWSC management to manage priorities: what works well, what requires tweaking and what changes are necessary for effective operations. Many of the M&E activities should be carried out at the outstation level given the agreed upon institutional framework where the outstations are independently operated. 3. Need for more analytical work around market segmentation, costing of future distribution network improvements, social marketing and communications and procurement support. LWSC needs the ability to determine profitability by market segment and needs the appropriate tools to conduct such analytical work. 4. Support the establishment of a decision support system (DSS) for senior and mid-level management for efficient operation at LWSC. A DSS is a computer-based application that collects, organizes and analyzes business data to facilitate quality business decision-making for management, operations and planning. DSS analysis helps companies to identify and solve problems, and make decisions. LWSC requires such a tool to facilitate efficient operational management. 5. Expand capacity building to LSCs. LMWP’s capacity building efforts at both Robertsport and Kakata have been very effective and should be expanded. Page 10 Priority Level 3 1. Support LWSC to replicate the LSCs in all other outstations to promote confidence in LWSC central management. Local control and “ownership” of the water infrastructure via the LSCs has had a very positive effect and engendered much optimism. Replicating the LSCs at all outstations will not only facilitate their management but also send the message that the outstations are all equally important. 2. Support innovative institutional capacity building approaches that promote South-to-South learning, e.g. utility-to-utility training via twinning programs. Exposure is critical for LWSC managers so that ideas on operating a successful utility will not appear abstract to them. Twinning programs with other successful utilities using the IDAMC concept will facilitate understanding and the workings of such a concept. 3. LMWP should establish and monitor a baseline capacity-building to measure success/progress over time. Since LMWP has developed several capacity building tools and has already started implementation, it is imperative for it to have baseline so that it can measure the effectiveness of the capacity building efforts. Page 11 TABLE OF CONTENTS Executive Summary ...................................................................................................................... 1 Introduction ............................................................................................................................... 1 Purpose and Study Questions of the Evaluation .................................................................... 1 Approach and Methodology ..................................................................................................... 2 Evidence, Findings and Analysis of the Study Questions ...................................................... 2 Effectiveness of Infrastructure Planning/Construction Oversight Approach ......................... 2 Effectiveness of Institutional Framework and Capacity Building .......................................... 5 Overall Project Positioning and Strategy for Phase-out of USAID assistance ....................... 7 Recommendations ..................................................................................................................... 7 Acronyms and Glossary .............................................................................................................. 13 Introduction ................................................................................................................................. 14 Purpose and Study Questions of the Evaluation....................................................................... 15 Limitations to the Study .............................................................................................................. 16 Approach and Methodology ....................................................................................................... 16 Desk Review ............................................................................................................................. 16 Key Informant Interviews and Field Visit ............................................................................ 17 Methodology ............................................................................................................................ 17 Evidence, Findings and Analysis of the Study Questions ........................................................ 18 Risk Assessment Framework (RAF) Matrix ........................................................................ 18 Risk Assessment Framework Results ................................................................................... 22 Analysis of the Study Questions ............................................................................................. 24 Effectiveness of Infrastructure Planning/Construction Oversight Approach ....................... 24 Effectiveness of Institutional Framework and Capacity Building ........................................ 32 Overall Project Positioning and Strategy for Phase-out of USAID Assistance Upon Completion of the Project ..................................................................................................... 35 LMWP’s Performance against Contract Requirements ..................................................... 36 Conclusions .................................................................................................................................. 37 Recommendations ....................................................................................................................... 38 Priority Level 1 ........................................................................................................................ 39 Priority Level 2 ........................................................................................................................ 43 Priority Level 3 ........................................................................................................................ 44 Appendices ................................................................................................................................... 46 Appendix 1. Evaluation scope of work .................................................................................. 46 Appendix 2. List of Documents Consulted ............................................................................ 52 Appendix 3. List of Individuals and Organizations Contacted ........................................... 54 Appendix 4. Technical Topics Including Study Methodology ............................................ 60 Appendix 5. Schedule of Activities ........................................................................................ 62 Appendix 6. Questionnaires and other evaluation tools ...................................................... 64 Infrastructure Planning & Design Evaluation ....................................................................... 64 Contractor Questionnaire ...................................................................................................... 66 USAID Questionnaire ........................................................................................................... 67 GOL Entities Questionnaire ................................................................................................. 69 Other Donors Questionnaire (AfDB, World Bank, etc.) ...................................................... 70 Page 12 Appendix 7. Evaluation Team Composition ......................................................................... 71 Appendix 8. Debriefing Presentation .................................................................................... 72 Page 13 ACRONYMS AND GLOSSARY AfDB African Development Bank CBA Cost benefit analysis CDCS Country Development Cooperation Strategy CIP Capital Improvement Project CBA Cost benefit analysis DDS Decision Support Systems DPA Destructive physical analysis DRT Design Review Team EG Economic Growth GOL Government of Liberia GPCD Gallons per capita demand GPD Gallons per day HH Household IDAMC Internally Delegated Area Management Contract LEC Liberia Electricity Corporation LEAP-II Learning Evaluation and Analysis Project-II LMWP Liberia Municipal Water Project LOE Level of effort LSC Local Steering Committee LWSC Liberia Water and Sewer Corporation M&E Monitoring and Evaluation MHI Manitoba Hydro International MOF Ministry of Finance MOU Memorandum of Understanding NWRSB National Water Resources and Sanitation Board O&M Operations and maintenance OTJ On-the-Job PBC Performance based contracting PFMRAF Public Financial Management Risk Assessment Framework PMP Performance monitoring plan QBS Quality-based selection RAF Risk Assessment Framework RFP Request for proposals SOE State owned enterprise SOP Standard operating procedure SOW Scope of work SWOT Strengths weaknesses opportunities threats analysis TOR Terms of reference UWSSP Urban Water Supply and Sanitation Project WB World Bank WBS Work Breakdown Structure VE/VA Value Engineering/Value Analysis USAID United States Agency for International Development Page 14 INTRODUCTION Civil war has wreaked economic havoc as Liberia, once a very peaceful and prosperous country on its way to a middle-income status, is now one of the poorest countries in the world. Though the economy has significantly improved over the past decade post-war, its progress towards economic recovery has been further derailed by the recent Ebola crisis. More than two decades of civil unrest coupled with limited, aging and outdated water supply infrastructure has prompted USAID to make a strategic investment in urban water to provide health and socio-economic benefits in Liberia. However, there is no agency responsible for the water sector in Liberia. The Executive Order for the National Water Resources and Sanitation Board (NWRSB) was issued in early 2014. However, the board has not yet met. The Water Supply and Sanitation Commission (WSSC) which is also expected to be formed through Executive Order remains outstanding as well. Additional challenges include the remoteness of the three target cities, poor road networks compounded by an intense rainy season, the lack of a functional electricity grid, etc. It is in this challenging environment that USAID is poised to bring the Liberia Municipal Water Project (LMWP) to fruition. LMWP is a four-year project to support the design, tendering, execution and operation of water supply infrastructure improvements in the three Liberian county capitals – Robertsport (Grand Cape Mount County), Sanniquellie (Nimba County), and Voinjama (Lofah County). It is a unique project opportunity and a foundational investment that not only addresses urban water supply infrastructure needs, but also sets the stage for a community-based and a decentralized approach for water supply management that can foster sustainable operation and maintenance of the recommended capital improvements. LMWP was designed in parallel to and in cooperation and coordination with other efforts by Government of Liberia (GOL) and international development partners to build and when applicable strengthen the institutional capacity of the Liberia Water and Sewer Corporation (LWSC), to sustainably operate and manage the capital investment. The Contractor, Tetra Tech, is tasked with assisting local and national authorities in developing plans for water supply and sanitation improvements, overseeing construction of this CIP, supporting initial operations of water supply infrastructure improvements, and re-establishing local capability to sustainably operate and maintain constructed water supply systems. The primary design criteria for the improved infrastructure is that by the end of the project, over 90 percent of the population in each city will have improved water supply access3 and improved infrastructure will be handed over to a locally-based operator capable of financially and technically sustaining water supply services. It is also expected that, through the Contractor’s collaboration with other USAID programs, the improved water systems will provide substantial health benefits and increased business opportunities in the three cities. Given that the LMWP is a model foundational investment project set in a post-conflict environment, USAID is embarking on a mid-term evaluation to determine whether the assistance it has provided through LMWP is meeting its objectives (see Appendix 1 for evaluation scope of work and Appendix 7 for the team composition). The timing of this evaluation is propitious for making adjustments as USAID 3 “Improved water supply access” is defined as regular household access to a water source, a distribution system, or a delivery point, which by the nature of its design and construction is likely to protect the water source from external contamination, in particular from fecal matter, and can be reached by the household in a round trip of 30 minutes or less. Improved water supply sources are: piped water into dwelling, plot, or yard; public tap/standpipe; tube well/borehole; protected dug well; protected spring; or rainwater collection. Unimproved water supply sources are: unprotected dug well; unprotected spring; cart with small tank/drum; tanker truck; surface water (river, dam, lake, pond, stream, canal, irrigation channel); and, bottled water. Page 15 advances plans to begin capital works in line with the plans developed by LMWP for the three cities accelerate over the coming months. PURPOSE AND STUDY QUESTIONS OF THE EVALUATION The purpose of the evaluation is to provide a detailed picture of the major accomplishments, strengths and weaknesses of LMWP since inception, indicating any recommended changes in the project’s implementation approach and USAID’s project design/programming approach to assure successful completion of urban water improvements in line with LMWP’s objectives:  Assess the existing water supply situation, environmental context, demand for improvements, and institutional capacities.  Develop recommendations, for USAID and GOL approval, on sustainable improvements in water supply service over the short, medium and long term.  Develop recommendations on the respective roles of various GOL entities including LWSC, public works, and county and municipal government. Coming to agreement with government officials on an institutional framework to manage proposed water supply infrastructure improvements on a sustainable basis.  Complete designs, providing procurement documents and oversight, and providing construction oversight on agreed short- and medium-term water supply improvements. The Contractor is not carrying out actual construction, and instead USAID will engage a construction contractor directly. The LMWP Contractor will oversee the construction procurement process and provide construction supervision.  Support operation and maintenance of improved water supply infrastructure on a transitional basis.  Build the capacity of the GOL, Liberia Water and Sewer Company (LWSC), county governments, municipal authorities, and the selected local water service management to effectively play their role in the agreed institutional framework by the end of the contract period. As the LMWP is moving into the construction phase in an environment that lacks adequate institutional and project delivery capacity, periodical formal and programmatic performance evaluation of the construction and operation and maintenance (O&M) of the LMWP is essential to not only minimize the risk of construction overrun, but also encourage and foster sustainability of the project. Therefore, the performance evaluation of LMWP revolved around examining three overarching questions: 1. Effectiveness of infrastructure planning/construction oversight approach: To what extent was appropriate due diligence done in the design and how well is the project positioned (in terms of management/staffing structure) to support the construction phase? 2. Effectiveness of institutional framework and capacity building: How effective is LMWP's current and proposed institutional and capacity-building approach in achieving the stated objectives? 3. Overall project positioning and strategy for phase-out of USAID assistance. Given delays in the construction design and implementation phase, the original timeframe underlying the project design (e.g. finish construction during LMWP 4-year base period, and then support O&M for 2 additional years) is no longer viable and it is likely that construction will just be finishing by the end of the LMWP's sixth year. Furthermore, much uncertainty remains around what exactly it will take for the outstations to reach operational cost recovery and how long this will take - estimates range from 1-3 years. How should USAID planning proceed? Page 16 Furthermore, the evaluation was designed to include specific recommendations for successful completion of the four-year base contract in September 2015 and the two option years in the event they are exercised by the USAID contracting officer, and recommendations on any key issues/changes regarding the implementation approach and programmatic priorities for the said option years. Finally, the evaluation is to identify priority areas that should be the focus of possible future programming in the Liberian urban water supply sector in support of the Mission’s overall development objectives outlined in the Liberia Country Development Cooperation Strategy (CDCS) for 2013-17. (See Appendix 5 for the schedule of activities of the evaluation.) The primary audience for this evaluation is USAID for the purposes of validating the status of planning (both infrastructure design, and viability of plans to establish sustainable GOL management entities) prior to initiation of the large-scale construction phase. Secondary audiences are USAID’s GOL partners and stakeholders, principally the LWSC, to provide recommendations related to their role in ensuring successful conclusion of the project and handover of infrastructure to financially and operationally capable local entities. This evaluation addresses the specific concerns of the stakeholders and focus on the project’s relevance and quality-at-entry, efficacy, efficiency, approach to institutional development impacts, sustainability, as well as the performance of the LMWP contractor (Tetra Tech) to deliver the project. LIMITATIONS TO THE STUDY The large amount of material to review in a limited time and logistics in the field were key challenges to the study. The key informant interviews in Monrovia including field visits to cities of Robertsport and Kakata that took place from November 25 through December 15, 2014, were diverse and iterative and were performed under a very tight schedule, requiring frequent inquiries to secure meetings and interviews with key individuals. However, the interviews were instrumental in helping the evaluation team to not only verify and seek clarifications of the key findings of the desk review, but also identify areas of the planning and design processes, and the construction phase that require improvements prior to and during construction. The processes leading to the evaluation of the project’s viability and related work have been iterative in nature and required careful due diligence to account for and meet industry standards of best planning and engineering practices, cost effectiveness, functionality and value for money. APPROACH AND METHODOLOGY The approach to the performance evaluation of the LMWP included a desk review, key informant interviews, and a field visit to Liberia. Desk Review The desk review consisted of a select review of the background information and LMWP’s deliverables to verify conformance with the scope of work (e.g., deliverables), industry standards, and help the evaluation team to formulate a questionnaire for the key informant interviews in the field (see Appendix 3 for the list of documents consulted). The evaluation team performed a review, and when applicable, key assessments of selected background information to form initial opinions about the degrees to which the proposed infrastructure design is well-suited to achieve project objectives in a cost-effective and sustainable manner; the extent to which LMWP conducted appropriate due diligence in the design process, and how well the project is positioned (in terms of management/staffing structure) to support the Page 17 construction phase. The desk review also formed the framework for an iterative cross referencing of the key informant interviews from the field with the quality and quantity of the background information. The evaluation team reviewed LMWP’s deliverables and cross referenced them against the expected deliverables as outlined in the Objectives and Scope of Work (Tasks 1-8). The review process included interactions with the interviewees focused on: (1) methodology employed thorough the planning and design process; (2) adequacy and consistency and the sources of field data and information; (3) key planning and design criteria in selecting the short-long term water and sanitation infrastructure improvements schemes and options; (4) integration/synergy (benefits from combinations of two or more options); (5) the contribution of the LMWP contractor to delays in targeted water supply service improvements; (6) assessment of current national and local capacities including private sector and public￾private partnership: (7) problems, issues, challenges and opportunities; (8) the use of value engineering/value analysis in facilitating and supporting optimum design vis-à-vis cost and best value for the design options; (9) capacity building needs assessment and capacity building threshold; and (10) cost effective synergies with other projects. Key Informant Interviews and Field Visit The evaluation team traveled to Liberia from November 24 through December 15, 2014. The purpose of the key informant interviews and field visit to Liberia was to: (1) clarify and verify the planning and design criteria that have guided the proposed implementation process vis-à-vis constructability and sustainability; (2) explore and examine the lessons learned including potential risks to successful implementation and operation and maintenance of the proposed improvements; and (3) build on the project’s accomplishments to make recommendations for sustainable operation and maintenance of the proposed systems. The evaluation team conducted over 20 key informant interviews and consulted over 15 organizational stakeholders including: USAID, LWSC, county and city authorities and the LSC in Robertsport, LWSC office in the city of Kakata, World Bank, AfDB, and Tetra Tech, the LWMP contractor. Field activities included visits to the cities of Robertsport, where components of LMWP have been and are currently being implemented, and Kakata, where LMWP has established Local Steering Committees (LSCs) as a model for an outstation outside LMWP’s mandate. (See Appendix 2 for the list of individuals and agencies contacted and Appendix 6 for questionnaires and other evaluation tools used to collect data in key informant interviews). The evaluation team presented the results of the field work to USAID on December 10, 2014 (see Appendix 8 for the debriefing presentation). Methodology The evaluation team, in consultation with USAID/Liberia, selected the risk assessment framework (RAF) to analyze the potential risks and impacts of recommended CIP improvements including analysis of the O&M cost recovery and sustainability, capacity building and institutional strengthening (see Appendix 4 for a discussion of the three analytical methods considered). The primary purpose of risk assessment is to evaluate the consequences if an investment decision or action fails. Decision makers typically want to avoid new investments or projects when the threats are catastrophic or when they are deemed as not being cost effective. The goal is to analyze the risks and rewards of a decision using data. Risk assessment reduces the need for intuitions and instincts. A RAF is a strategy for prioritizing and sharing information about the risks to infrastructure investment. A good RAF organizes and presents information in a way that both technical and non-technical personnel can understand. Page 18 According to the USAID Public Financial Management Risk Assessment Framework (PFMRAF), “a classification of Critical requires stringent mitigating measures only if there is a high probability of success.” Otherwise, USAID should terminate exposure by delivering the assistance through other means. Risk assessment is neither a certification nor a seal of approval. It is a measurement of the risks presented by a particular implementing mechanism deployed to achieve a given development objective relative to the delivery approach to be used. “Identification” neither eliminates nor mitigates the risk. Concrete actions will be required for mitigation. The three overarching evaluation questions: (1) Effectiveness of infrastructure planning/construction oversight approach; (2) Effectiveness of institutional framework and capacity building; and (3) Overall project positioning and strategy for phase-out of USAID assistance, served as the guide to identify the focus areas that are most applicable to meet the key project objectives: viability (design options and value for money) and sustainability (cost recovery and institutional capacity of the LWSC to operate and maintain the CIP post construction and start-up and commissioning). The evaluation team assessed and assigned a classification to eight focus areas: 1. Existing structure (e.g., elevated tanks) that have been proposed to remain as part of the CIP, 2. CIP value for money, 3. Management/staffing structure to support the construction phase, 4. Construction cost overrun, 5. LWSC’s resources, 6. LMWP capacity building of LSCs, 7. LMWP capacity building of LWSC, and 8. O&M cost recovery. Within each focus area, the evaluation team evaluated processes and procedures to assess risks according to a rating system developed by the evaluation team, adopted from USAID’s PFMRAF framework. For each focus area, the evaluation team determined the probability of that event occurring and the severity of impact of the occurrence to arrive at a risk rating. The risk classification for the eight focus areas is presented in the findings section below summarized by the five PFMRAF criteria used in this evaluation. EVIDENCE, FINDINGS AND ANALYSIS OF THE STUDY QUESTIONS Risk Assessment Framework (RAF) Matrix This section presents the results of the RAF. First the definitions of risk classification and probability are presented in Tables 1 and 2 below. The RAF Summary Chart showing the classification and probability of risks is presented in Table 3 below. The numbers on the RAF Summary Chart correspond to the ID numbers on the RAF Table (Table 4). The RAF table includes mitigation measures which are discussed in detail in the Recommendations section of the report. Following the tables and charts is detailed discussion of the RAF results. Page 19 Table 1: Risk Classification by RAF Criterion Classification USAID Stage 2 Risk Mitigation Guidance Critical “Critical requires stringent mitigating measures only if these have a high probability of success. Otherwise, we will terminate our exposure by delivering the assistance through other means. In rare cases where an effective transfer of risk mechanism exists and is deemed effective, we will consider transfer of the risk, albeit with a risk assessment of the ability of the transferor to deliver on its obligation.” High "High requires serious mitigating measures to treat the risk to avoid possible catastrophic and other major failures.” Medium “Medium requires mitigating measures but these may be periodic.” Low “Low requires monitoring and audit, but treatment of specific risks may be required if they can lead to Medium risk conditions.” Table 2: USAID Rating System: Probability Probability Quantitative Qualitative Remote Less than a .25 probability. An adverse event is rare or would only occur in exceptional circumstances. There is little or no experience of a similar failure. Occasional Probability lies between .26 and .50. An adverse event might occur because the conditions for it exist, but controls exist and are effective. Probable Probability lies between .51 and .75. An adverse event will likely occur because the controls are inadequate or are applied inconsistently. Frequent Probability lies between .76 and .99. An adverse event is expected to occur. There is near certainty of occurrence because the controls do not exist or are ineffective. Page 20 Table 3: RAF Summary Chart4 4 The numbers in the circles represent the Focus Area ID from Table 4 below. The first number is the evaluation question (1, 2, or 3). The second number after the decimal point refers to the focus area. Catatastrophic High Critical Critical Critical Serious High High Critical Critical M arginal Medium Medium High High Neglegible Low Low Medium Medium Remote Occasional Probable Frequent Probability Impact 1.1 1.4 1.3 2.2 2.3 2.1 1.2 3.1 Page 21 Table 4: RAF Matrix Existing structure (e.g., elevated tanks) that have been proposed to remain as part of the proposed CIP 1.1 The condition assessment of the few existing infrastructure was performed primarily through visual inspections without use of analytical tool (i.e., computer models). Probable Catastrophic Critical Apply computer models to simulate structural stability for seismic and wind loadings; and when and if applicable, perform destructive testing to ensure material integrity. CIP value for money 1.2 Lack of adequate design options and iterations, and trade offs Occasional Marginal Medium Conduct Value Engineering & Value Analysis to simulate alternative design options and associated cost implications. Management/staffing structure to support the construction phase 1.3 The resource planning including the proposed staffing plan is still evolving and has yet to be finalized. Occasional Marginal Medium The resource planning including the proposed staffing plan is still work in progress. However, the project and the proposed construction management tools should be re￾visited and re-evaluated after incorporating VE/VA process, to also include construction procurement scenarios/option. Construction cost overrun 1.4 Lack of qualified and responsive local contractors; and environmental uncertainty vis-à-vis health, transportation infrastructure, etc., Probable Serious Critical Conduct VE/VA prior to and during construction; Two-stage procurement process; Performance based contracting with provisions for incentives; Evaluate and examine project fast tracking. LWSC's resources 2.1 LWSC's HR constraints and limitations to dedicate staff to LMWP trainings and review processes including at the construction phase. Occasional Marginal Medium Utilize the MOU with MoF to allocate additional dedicated resources specific to LMWP; and when and if applicable, LMWP provide the support resources. LMWP Capacity Building (CB) of LSCs 2.2 The CB of the LSCs have been well received and proven effective Occasional Marginal Medium Continue and expand the CB of the LSCs to asset management aspects of O&M. LMWP Capacity Building of LWSC 2.3 LWSC is understaffed and can not fully engage and commit to CB programs. Occasional Marginal Medium Utilize the MOU with MoF to allocate additional dedicated resources specific to LMWP; and when and if applicable, LMWP provide the support resources. 3 Overall project positioning and strategy for phase-out of USAID assistance O&M Cost Recovery 3.1 Appropriate tools and methodologies have been employed without sensitivity analysis; cost assumptions are reasonable but revenues are unreasonable and ambitious; O&M recovery becomes a problem and the 1-3 year cost recovery period is not realistic; and best estimate for O&M cost recovery is 7 years. Probable Serious Critical Re-evaluate and simulate the financial analysis including the CBA under various options and assumptions to identify the range of cost and financial trade offs and implications. Overarching Questions Focus Area Effectiveness of institutional framework and capacity building Probability of Impact Risk Recommended Mitigation Measures Occurrence Key Observations Effectiveness of Infrastructure Planning/ Construction Oversight Approach 1 Reference ID Questions 2 Page 22 Risk Assessment Framework Results The evaluation team noted several key observations which impact the risks across the three study questions, principally the need for: 1. Further refinements of the design options to minimize potential cost overrun during construction; 2. Re-evaluating the economic and financial analysis to develop a more realistic understanding of operation and maintenance cost recovery; and 3. Building on current successes with capacity building and institutional strengthening at the LWSC. Existing Infrastructure Asset Inventory (Condition Assessment): As LMWP evolved from that of conceptual design options at the planning stage, to design schemes that fit USAID’s budgetary constraints, LMWP evaluated and designed a CIP package that retains components of the existing water supply systems in the three cities of Robertsport, Sanniquellie, and Voinjama. At the core of continued use of an existing infrastructure is its structural integrity, functionality and operability. To determine the condition of an existing infrastructure component like water supply systems, data are collected through observation, direct inspection, indirect monitoring, and mathematical modeling or a combination of all to establish a baseline data. The data when properly analyzed can reveal structural and operational conditions including potential modes and causes of failures for continued and intended use. The existing infrastructure is over 20 years old. In discussions with the LMWP design team in US and in Monrovia, the evaluation team was informed that the bulk of the condition assessment of existing water supply infrastructure in the three cities was performed through visual inspection only, and that the lack of access to as-built and as constructed plans and documents posed a significant challenge to a formal and a more thorough condition assessment of components of the existing system. Incorporating the existing infrastructure in the CIP could result in a catastrophic failure if there are structural issues that a visual inspection cannot capture. A catastrophic failure of the vertical structures such as the elevated tanks could result in a prolonged loss of and disruption to service, and may potentially cause injuries and human loss depending on the timing and mode of failure and if the failure occurs when people are in proximity to the structure. Catastrophic failure of such assets can occur because of insufficient design criteria, poor quality construction, and pre-mature aging of the infrastructure due to lack of routine inspection and maintenance that may collectively contribute to a poor or non-response mechanism to dynamic loads such as seismic activities. A more elaborate application of condition assessment tools and techniques such as mathematical modeling and when and if applicable destructive testing to structural components (i.e., vertical structures such as the elevated tanks) is warranted to ensure that these types of infrastructure are integrated into proposed new assets only when their structural and functional conditions are compatible to the design and construction criteria of the new proposed infrastructure. For example, in destructive testing, or (destructive physical analysis, DPA) tests are carried out to the sample's failure, in order to understand a sample's structural performance or material behavior under different loads. These tests are generally much easier to carry out, yield more information, and are easier to interpret than nondestructive testing. CIP Value for Money: Utility managers such as water and sewerage networks in general have two key challenges for ensuring that investment in water assets delivers best value for money: (1) how best to mitigate the deterioration of existing assets, and (2) how to select the best options to deliver improvements to service. Adopting investment practices that adhere to the principle of “value for money across the entire life cycle of an asset” can ensure sustainable and equitable development. As a principle, the shorter the payback period, the more attractive the investment is. Page 23 The objective of the LMWP is sustainability through O&M cost recovery, and adequate institutional and human resources capacity to operate the CIP post start-up and commissioning, particularly when the proposed LMWP CIP has not been discounted (i.e., no expected payback period on the initial cost of construction and other soft costs such as capacity building). Therefore, it becomes critical to ensure that whatever is invested will yield value-for money vis-à-vis donor-funded investment. The evaluation team noted that throughout the planning and design processes and as the project evolved, LMWP evaluated a range of investment options to select from among alternative options, a CIP that best fits the USAID budget constraints while ensuring the key objectives of the project are met (e.g., number of people served by the service, water quality, reliability of supply, efficient energy requirements etc.). However, the evaluation team noted also that LMWP did not formally apply industry standard decision support systems such as value engineering/value analysis (VE/VA) to more effectively simulate alternative investment strategies and design schemes including their associated costs and trade-offs. This becomes particularly important in light of the risk of construction cost overrun. Management/Staffing Structure to Support the Construction Phase: The primary consideration of the staffing plan for project management is to determine the specific skill sets required for completing project deliverables. The staffing plan entails drawing up a time schedule for specific skill requirements based on the project schedule and task plan. One important phase in developing a project management staffing plan is undertaking a skills inventory based on knowledge, skills, and abilities of the human resources of the organization and matching the same with the skill sets required. Given the lack of adequate capacity at LWSC, and in anticipation of procurement and construction challenges in Liberia with limited construction capacity particularly as Liberia continues to deal with the recent Ebola outbreak, the evaluation team considers the proposed LMWP construction oversight staffing plan, although still evolving and being revised, inadequate to minimize potential risk of timely and quality construction and completion of the proposed CIP. The project/construction management tools should be re-visited and re-evaluated after incorporating VE/VA process, to also include construction procurement scenarios/option. This should include reconciling the required skills with existing skills. This entails having a clear understanding of the available staff and contribution of LWSC to the construction phase, and if the availability meets the requirements, preparing a comprehensive schedule by assigning time￾specified duties and responsibilities to the available staff. Construction Cost Overrun: Construction cost overruns have a broad range across the project spectrum, and can be attributed to many factors including but not limited to: (1) errors in budgeting/estimating a project; (2) mathematical errors-transcribing, pressing wrong keys, omissions and miscalculations; (3) plans and specifications-errors, omissions, vague drawings and scope in the plans and specifications; (4) estimators inexperienced in the field of expertise, estimating programs and unique bid requests by the client; and (5) lack of knowledge by the contractor in new locations. Another major problem that often can bring about cost overruns is working in a new environment, in this case Liberia with all its socio￾economic and environmental challenges post recovery from the civil war. Consideration to resources (personnel, material and equipment), site conditions, weather, accommodations, and safety requirements are indispensable. Costs required beyond the scope of work are most often attributed to: conditions unknown to the contractor including requests by client clearly not within the scope of work, and client failure to fulfill commitments according to specifications. Lack of qualified and responsive local contractors; and environmental uncertainty vis-à-vis health, transportation infrastructure, etc., have already exacerbated the uncertainty around the cost of construction activities in Liberia. The application of VE/VA prior to and during construction; a two-stage procurement process; performance based contracting (PBC) with provisions for incentives; and evaluating and examining project fast tracking are among the options that when employed, can minimize the risk of construction cost overrun (these are discussed in more detail in the Recommendations section of the report). The VE/VA in the construction phase is conducted with the goal of eliminating unnecessary costs while maintaining the project’s Page 24 function, quality and owner’s vision by providing creative construction solutions such as: employing efficient construction technology, alternative construction materials, minimizing pipe depth, eliminating/substituting more commonly available construction material for hard to find items, flow fill of abandoned utilities in lieu of removal, etc., all to provide opportunities to eliminate unnecessary costs. LWSC’s Resources: LWSC is facing enormous challenges in providing the required services under its mandates. Its management culture and operational structure, appear to be heavily centralized and are vulnerable to lack of sufficient and efficient resources and technical and managerial capacity to prepare for and operate and manage a capital investment. It is also understaffed in some critical areas so much so that LWSC has had to pull staff from LMWP’s capacity building training. According to a high-level LWSC official, LWSC did not receive a subsidy from the GOL in 2014. The current performance data suggest that LWSC can, with adequate capacity and policy support and over the course of LMWP implementation including USAID’s two-year O&M support, achieve O&M cost recovery of the CIP￾constructed water systems. LWSC requires additional resources from GOL or LMWP to be able to provide construction support and to fulfill the objectives in its MOU. LMWP Capacity Building of LSCs: The establishment of and capacity building of the LSCs at the Robertsport and Kakata outstations have been among the key achievements of LMWP toward establishing an institutional framework for outstation management. Both outstations gave LMWP excellent reviews on its capacity building efforts. The capacity building and training have gone so well that the LSCs have requested more training. LWSC, which was initially reluctant and skeptical about the IDAMC concept, has now fully embraced the model. The LSCs have been enthusiastic about championing the cause of the outstations since they now have the tools to provide effective oversight of the water infrastructure O&M. LMWP must continue to expand its capacity building efforts to include asset management aspects of the O&M. LMWP Capacity Building of LWSC: LMWP has the appropriate tools and plans required to build adequate or needed capacity at LWSC. While LMWP can replicate the success of the LSCs capacity building effort at LWSC, the major hurdle has been LWSC’s inability to provide staff for training. LWSC needs resources to strengthen its technical staff. The support must come from either LMWP or GOL. O&M Cost Recovery: Ensuring that the new outstations reach O&M cost recovery is critical for sustainability. In its financial and economic analysis, LMWP employed appropriate tools and methodologies without sensitivity analysis to test any of the parameters. The cost assumptions are reasonable but the revenues are unreasonable and ambitious, if not unrealistic. Hence the projected O&M cost recovery period is not realistic. A key issue was that the revenue projections did not take into account affordability (willingness to pay). LMWP has to re-evaluate and analyze the financial details of the CIP. Analysis of the Study Questions This section presents findings for each study question followed by an analysis of LMWP’s performance against the requirements in Section C of the contract. Effectiveness of Infrastructure Planning/Construction Oversight Approach Is the proposed infrastructure design well-suited to achieve project objectives in a cost-effective and sustainable manner, to what extent was appropriate due diligence done in the design, and how well is the project positioned (in terms of management/staffing structure) to support the construction phase? Page 25 The proposed CIP design is well-suited to achieve the project objectives; however, cost-effectiveness of the design and sustainability need further exploration and analysis. Additional staffing is required to support the construction phase; however, the Contractor indicated that resource planning (which includes staffing) is a work-in-progress and will be revised. a. Do the overall plans for each city (Water Master Plans and designs developed by LMWP, including costs and Cost Benefit analyses) appear reasonable and indicate value for money in terms of achieving project objectives and ensuring judicious use of US Government funds? Are the proposed design approach and cost reasonable in comparison to similar projects (in Liberia or other settings) and taking into consideration the size of the target populations, institutional sustainability issues, etc.? Analysis of Water Master Plans and Designs The overall plans appear reasonable and contain the components typically found in a Water Master Plan5 :  Characteristics of the existing system (to establish a baseline),  Water Demand Projections (based on population projections and other needs),  Water Supply Capacity (based on water sources - ground water, surface water, etc.),  Water Infrastructure Selection (showing water system components),  Financial Summary (displaying cost/benefit analysis),  Sustainability (to minimize costs by optimizing investment choices, improving efficiency, strategic planning, etc.), and  Plan Implementation (to ensure that the project team is using appropriate project control tools based on best practices, e.g. change management, risk management, communications management, budget control, development of reliable cost and scheduling estimates, feedback loop, etc.) are present. However, as noted above the plans lack a VE/VA. Engineering principles and design standards make it imperative to employ VE/VA to identify and verify potential design refinements vis-à-vis functionality, and reliability at optimal costs. Often value engineering reduces costs by eliminating unnecessary practices throughout the planning, design and implementation phases of a project. It usually covers various dimensions of the project including but not limited to: (1) choice of materials; (2) function-driven improvements (needs versus wants); (3) energy efficiency; and (4) modularity. LMWP did not employ a formal VE/VA process to simulate alternative design options to ensure value for function while minimizing costs (e.g., selecting the most cost effective materials and balancing recommended solutions for restorations, upgrades, and replacement/new improvements). It is prudent to employ all tools to manage risks given the challenging environment. The VE/VA is one such tool to potentially improve the recommended CIP by developing alternative design options that facilitate and support decision-making relative to cost, value and function. The CIP involves both rehabilitating a few existing infrastructure components and installing new ones. It was thus quite critical that LMWP inspect existing structures to assess their efficacy. Engineering norms require conducting such inspections with a focus on safety and structural integrity. The evaluation team was informed by LMWP that it assessed the condition of the existing candidate infrastructure for inclusion in the CIP (e.g., elevated water concrete tanks that are about 100 feet in height) at the three project sites primarily via visual inspections only, without employing analytical or other appropriate 5 Draft Robertsport Water Master Plan August 2012, Draft Sanniquellie Water Master Plan November 2012, Draft Voinjama Water Master Plan December 2012. Page 26 physical testing tools to evaluate structural integrity. A few of the existing assets are over 20 years old, and any structural failure of such assets could have catastrophic consequences. Visual inspections for horizontal structures may at times prove appropriate and sufficient if as-built drawings/records are available and properly reviewed for conformity to the original design and specifications. However, elevated structures usually can pose a risk from seismic and other loadings. Hence notwithstanding their expected design lives in the 25 to 50-year range, it is prudent and necessary for LMWP to rigorously evaluate and analyze such assets for structural integrity and functionality. Cost and Sustainability Analysis In 2012, LMWP conducted a socio-economic study in developing the Master Plans which showed the willingness to pay for water per household comprising five people, ranged from US$12.00 to US$23.00 per month for Robertsport6 , US$12.50 to US$25.00 for Sanniquellie7 , and US$7.29 to US$14.58 for Voinjama8 . The same study also showed that households could most likely pay more for water as there was some surplus (net) income after paying for typical household expenses. Notwithstanding the economic and financial analysis as they relate to sustainable O&M (i.e., cost recovery) the analysis require further refinement to reflect affordability and willingness to pay. It should be noted that other than limited vendor sales in Robertsport, there is no significant precedent of payment for water services in the target cities. This creates challenges in predicting demand and willingness and ability to pay which increases the importance of conducting a sensitivity analysis of the projections. Sustainability is driven by financial analysis buoyed by realistic and conservative assumptions which take into account drivers of projected revenues and costs. The current cost model and the analysis require further refinements to better understand the sensitivity of the results granted the following:  Payback period: The payback period (O&M cost recovery) is overly optimistic and ambitious, given that it is assumed that demand approximately quadruples in the first three or four years depending on location. There is no justifiable reason provided for the sudden increases in demand. The assumed collection rate of 90 percent is reasonable given that the systems will be cash-and-carry; however, the 90 percent recovery of accounts in arrears appears aggressive as experience from LWSC suggests otherwise.  Affordability analysis: The evaluation team conducted affordability analyses for Voinjama, Robertsport and Sanniquellie from 2016-2031. All three analyses show that rates become unaffordable (the willingness to pay amount per household exceeds the monthly household bill) sometime over the next five years making the projected revenues unrealistic and unsustainable. Affordability Analyses The evaluation team conducted affordability analyses for Voinjama, Robertsport and Sanniquellie from 2016-2031 based on the data provided in LMWP’s projections for each city. Assumptions and results are shown in Tables 5-10 below. The willingness to pay is US$7.29-$25.00 (2012) per month per household of five people based on location. The evaluation team assumed an inflation rate of three percent to increase the willingness to pay amount from 2012 to 2031. All three analyses show that the total household expenditures at the projected demand levels become unaffordable sometime over the next five years making the projected revenues unrealistic. The yellow highlights in the projection tables indicate when monthly household bill exceeds the willingness to pay. It should be noted that this occurs when the more than quadrupling in demand occurs. 6 Draft Robertsport Water Master Plan, August 2012, Page 96. 7 Draft Sanniquellie Water Master Plan, November 2012, Page 62. 8 Draft Voinjama Water Master Plan, December 2012, Page 49. Page 27 Voinjama Analyses of the data show that from 2016 to 2018, the willingness to pay amount per household (Column 9) exceeds the monthly household bill (Column 8) (see Table 6 below). However, from 2019 to 2031, the opposite is true making the projected revenue unrealistic. Also, the average total water demand (Column 4) in gallons per day (gpd) more than quadruples from 2018 (26,811 gpd) to 2019 (111,820 gpd). Unless there is a justifiable reason for this jump, the 2020 volume is not reasonable making the related cash flow unrealistic. Table 5: Voinjama Assumptions Proposed Tarriff $0.04 Household Size 5 Minimum willingness to pay per household per month (2012) $7.29 Maximum willingness to pay per household per month (2012) $14.58 Inflation 3% Source: Socioeconomic Study, Voinjama Master Plan Table 6: Voinjama Projections Tetra Tech's Projections Evaluation Team's Analysis 1 2 3 4 5 6 7 8 9 10 Year Service Area Population Percent Coverage Total Daily Flow (gpd) Production Capacity Capacity Utilization Per Capita Demand (gpcd) Household (HH) Bill per Month Willingness to Pay per Household a Month (max) Is Willingness to Pay greater than monthly HH bill? 2016 16,100 90% 25,598 342,451 7.47% 1.59 $9.54 $16.41 YES 2017 16,300 90% 26,201 342,451 7.65% 1.61 $9.64 $16.90 YES 2018 16,500 91% 26,811 342,451 7.83% 1.62 $9.75 $17.41 YES 2019 16,700 92% 111,820 342,451 32.65% 6.70 $40.17 $17.93 NO 2020 17,000 93% 115,040 342,451 33.59% 6.77 $40.60 $18.47 NO 2021 17,200 94% 141,329 342,451 41.27% 8.22 $49.30 $19.02 NO 2022 17,400 95% 169,533 342,451 49.51% 9.74 $58.46 $19.59 NO 2023 17,600 96% 198,871 342,451 58.07% 11.30 $67.80 $20.18 NO 2024 17,900 97% 230,649 342,451 67.35% 12.89 $77.31 $20.79 NO 2025 18,100 98% 262,469 342,451 76.64% 14.50 $87.01 $21.41 NO 2026 18,300 99% 292,553 342,451 85.43% 15.99 $95.92 $22.05 NO 2027 18,600 100% 324,979 342,451 94.90% 17.47 $104.83 $22.72 NO 2028 18,800 100% 328,474 342,451 95.92% 17.47 $104.83 $23.40 NO 2029 19,100 100% 333,715 342,451 97.45% 17.47 $104.83 $24.10 NO 2030 19,300 100% 337,210 342,451 98.47% 17.47 $104.83 $24.82 NO 2031 19,600 100% 342,451 342,451 100.00% 17.47 $104.83 $25.57 NO Column Notes 1 Service year 2 Service Area Population 3 Percent of service area population covered 4 Total Daily Flow in gallons per day 5 Production capacity (designed capacity) in gallons per day 6 Capacity Utilization - (4)/(5) 7 Per Capita Demand - (4)/(2) 8 Household (HH) bill per month - (7)*30*Proposed Tariff*HH size Page 28 9 Upper limit of Willingness to pay 10 Affordability - If (9) is greater than (8), then YES, else NO Robertsport Analyses of the data show that for 2016 and 2017, the willingness to pay amount per household (Column 9) exceeds the monthly household bill (Column 8) (see Table 8 below). However, from 2018 to 2031, the opposite is true making the cash flows unrealistic. Also, the average total water demand (Column 4) in gallons per day (gpd) more than quadruples from 2017 (6,604 gpd) to 2018 (29,977 gpd). Unless there is a justifiable reason for this jump, the 2018 volume is not reasonable making the related cash flow unrealistic. Table 7: Robertsport Assumptions Proposed Tarriff $0.04 Household Size 5 Minimum willingness to pay per household per month (2012) $12.00 Maximum willingness to pay per household per month (2012) $23.00 Inflation 3% Source: Robertsport Master Plan Table 8: Robertsport Projections Tetra Tech's Projections Evaluation Team's Analysis 1 2 3 4 5 6 7 8 9 10 Year Service Area Population Percent Coverage Total Daily Flow (gpd) Production Capacity Capacity Utilization Per Capita Demand (gpcd) Household (HH) Bill per Month Willingness to Pay per Household a Month (max) Is Willingness to Pay greater than monthly HH bill? 2016 4,100 90% 6,447 97,081 6.64% 1.57 $9.43 $25.89 YES 2017 4,200 90% 6,604 97,081 6.80% 1.57 $9.43 $26.66 YES 2018 4,300 91% 29,977 97,081 30.88% 6.97 $41.83 $27.46 NO 2019 4,300 92% 53,169 97,081 54.77% 12.36 $74.19 $28.29 NO 2020 4,400 93% 57,746 97,081 59.48% 13.12 $78.74 $29.14 NO 2021 4,500 94% 64,810 97,081 66.76% 14.40 $86.41 $30.01 NO 2022 4,600 95% 69,305 97,081 71.39% 15.07 $90.40 $30.91 NO 2023 4,700 96% 74,589 97,081 76.83% 15.87 $95.22 $31.84 NO 2024 4,800 97% 79,472 97,081 81.86% 16.56 $99.34 $32.79 NO 2025 4,900 98% 85,191 97,081 87.75% 17.39 $104.32 $33.78 NO 2026 5,000 99% 87,817 97,081 90.46% 17.56 $105.38 $34.79 NO 2027 5,100 100% 90,478 97,081 93.20% 17.74 $106.44 $35.83 NO 2028 5,200 100% 92,252 97,081 95.03% 17.74 $106.44 $36.91 NO 2029 5,300 100% 94,026 97,081 96.85% 17.74 $106.44 $38.02 NO 2030 5,400 100% 95,800 97,081 98.68% 17.74 $106.44 $39.16 NO 2031 5,500 100% 97,574 97,574 100.00% 17.74 $106.44 $40.33 NO Sanniquellie Analyses of the data show that for 2016 and 2017, the willingness to pay amount per household (Column 9) exceeds the monthly household bill (Column 8) (see Table 10 below). However, from 2018 to 2031, the opposite is true making the cash flows unrealistic. Also, the average total water demand (Column 4) Page 29 in gallons per day (gpd) more than quadruples from 2017 (21,700 gpd) to 2018 (91,398 gpd). Unless there is a justifiable reason for this jump, the 2018 volume is not reasonable making the related cash flow unrealistic. Table 9: Sanniquellie Assumptions Proposed Tarriff $0.04 Household Size 5 Minimum willingness to pay per household per month (2012) $12.50 Maximum willingness to pay per household per month (2012) $25.00 Inflation 3% Source: Sanniequellie Master Plan Table 10: Sanniquellie Projections Tetra Tech's Projections Evaluation Team's Analysis 1 2 3 4 5 6 7 8 9 10 Year Service Area Population Percent Coverage Total Daily Flow (gpd) Production Capacity Capacity Utilization Per Capita Demand (gpcd) Household (HH) Bill per Month Willingness to Pay per Household a Month (max) Is Willingness to Pay greater than Monthly HH bill? 2016 13,600 90% 21,386 305,760 6.99% 1.57 $9.44 $28.14 YES 2017 13,800 90% 21,700 305,760 7.10% 1.57 $9.43 $28.98 YES 2018 14,100 91% 91,398 305,760 29.89% 6.48 $38.89 $29.85 NO 2019 14,300 92% 113,163 305,760 37.01% 7.91 $47.48 $30.75 NO 2020 14,500 93% 135,929 305,760 44.46% 9.37 $56.25 $31.67 NO 2021 14,800 94% 160,800 305,760 52.59% 10.86 $65.19 $32.62 NO 2022 15,000 95% 185,774 305,760 60.76% 12.38 $74.31 $33.60 NO 2023 15,300 96% 213,199 305,760 69.73% 13.93 $83.61 $34.61 NO 2024 15,600 97% 242,015 305,760 79.15% 15.51 $93.08 $35.64 NO 2025 15,800 98% 270,536 305,760 88.48% 17.12 $102.74 $36.71 NO 2026 16,100 99% 278,486 305,760 91.08% 17.30 $103.78 $37.81 NO 2027 16,400 100% 286,541 305,760 93.71% 17.47 $104.83 $38.95 NO 2028 16,600 100% 290,035 305,760 94.86% 17.47 $104.83 $40.12 NO 2029 16,900 100% 295,277 305,760 96.57% 17.47 $104.83 $41.32 NO 2030 17,200 100% 300,518 305,760 98.29% 17.47 $104.83 $42.56 NO 2031 17,500 100% 305,760 305,760 100.00% 17.47 $104.83 $43.84 NO Analysis of LMWP Cost Benefit Analysis (CBA) The standard practice in CBA is to:  Ensure the data and assumptions underlying the estimation of costs and benefits are reliable and realistic.  Identify risks e.g. examining each variable to assess the level of uncertainty involved.  Use risk assessment techniques to assess the level of risk and the impact of risk on project performance including such techniques as: o Sensitivity analysis - to test the extent to which the outcome of the cost benefit analysis is sensitive to changes in the values of the input variables. o Scenario analysis - to test the best case, worst case scenarios for example. Page 30 o Expected values - to determine the probability-weighted average value of all possible outcomes. o Monte Carlo analysis - to model the effects of key variables on the net present value (NPV) of a given proposal.  Devise a risk management strategy including measures to contain, avoid and mitigate risks, as appropriate and communicate the risk management to the relevant stakeholders. In the LMWP CBA9 , the benefits LMWP identifies are mostly social (opportunity costs) which are difficult to quantify and can swing widely depending on the assumptions. The benefits costs comprise the following opportunity costs related to access to better quality water:  Health care costs associated with diarrheal disease,  Transportation costs per visit to the hospital,  Lost income due to illness,  Lost education value (due to outpatient and hospitalization episodes),  Lost income from parent due to child illness (due to outpatient and hospitalization episodes),  Lost convenient time spent collecting water, and  Economic value due to projected (extended) life expectancy. LMWP groups together disparate impacts (fewer hospital visits, fewer lost work hours, increased economic activity due to extended life expectancy, etc.) and adding the “benefits” to arrive at the project benefit values. These impacts may or may not be additive but there is no sensitivity to see what variations in the assumption will bring. The disparate benefits from having access to potable water are difficult to quantify. Pertaining to lost income due to illness, the model estimates 40 percent of the population is affected and the opportunity cost is based on US$7.00 per day as the labor rate. One key question to ask is whether those in the work force are as likely to get sick as those out of it. If the answer is no, then the model may be overestimating the related opportunity costs. If for example, the people getting sick are primarily the elderly and young people (both less likely to be economically active), then the model overestimates the opportunity cost. Another example is value for time lost collecting water. Those who do most of the water collection may not be able to readily find jobs at the minimum wage (estimated at US$4.00 per day in the model) meaning the opportunity cost may not exist. This thus overestimates the lost time value. The analysis also shows lost value per household and lost value per individual. This may be double counting as individuals form part of a household. The model shows no rationale for using US$7.00 per day for lost income due to illness and US$4.00 per day for lost time spent collecting water. LMWP should have checked the assumptions and run sensitivities to see the impact of changes in the inputs. LMWP also discounted the benefits, with an implicit optimism bias given the assumptions, at two percent (resulting in a higher present value) and the costs at three percent resulting in a lower present value. There is no sensitivity analysis conducted to show what happens if the project does not meet some of the overly optimistic assumptions. LMWP did not perform any risk analysis or attendant mitigation strategies. Had LMWP appropriately quantified and discounted the benefits, the 4:1 benefit-cost ratios would have been lower. b. Are the proposed design approach and cost reasonable in comparison to similar projects (in Liberia or other settings) and taking into consideration the size of the target populations, institutional sustainability issues, etc.? 9 20131120 CIP_CBA plus Sanni Cost Service Spreadsheet, Benefit Values tab Page 31 LMWP is a unique long-term project that integrates soft and hard aspects of development in an economically fragile state. The designed CIP is a very distinctive project in its scale in an environment beset with uncertainties exacerbated by the ongoing Ebola crisis. Given this, there is no sufficient baseline data readily available for comparison. Nonetheless, the cost, while conservative, is relatively reasonable. c. How much variability/uncertainty exists around the current cost estimates and what could be done (in terms of RFP or contract provisions, phasing of works, etc.) to minimize risk of cost overruns while ensuring delays are avoided and GOL/target city expectations are met. Given that baseline data do not exist, a numeric value for the variability/uncertainty of costs cannot be determined. Accurate estimation of construction costs is heavily dependent on the availability of quantity and quality historical cost data, the level of professional expertise, the existing local construction capacity, site requirements and bidding and labor climate. Variability and uncertainty in cost estimates arise from various factors including the prevailing environment and related perceived risks, capacity and logistics challenges, and likely bidders. Given all the current prevailing challenges, it is not surprising that LMWP used conservative cost estimates to address and mitigate risks. Implementing a two-stage procurement process is one way to minimize the risk of cost overruns. This is discussed in detail in the Recommendations section below. c. How well has LMWP conducted due diligence in analyzing and understanding sustainability issues facing the Capital Improvement Project, e.g. are projected revenue/cost assumptions, and the timeline to reach cost recovery realistic and reasonable? LMWP used appropriate tools and methodologies in conducting its due diligence to understand sustainability issues facing the CIP. The methodology fell into three categories namely (1) engineering, (2) institutional, and (3) socio-economic. On the engineering front, LMWP assessed existing structures, developed master plans, assessed O&M capacity, and procurement and construction systems. On the institutional front LMWP reviewed the policy and institutional framework, assessed capacity needs, analyzed MOUs and developed capacity building plans and delegated performance contracts. As for socio-economic, LMWP conducted a socio-economic study via a survey10, assessed the cost of service including rate setting and attendant monitoring and evaluation functions, and evaluated capacity building for O&M. LMWP then used the data to develop cost and revenue projections. The cost assumptions are reasonable but the revenues are ambitious and unreasonable due to the reasons enumerated above. The O&M cost recovery period of one to three years is not realistic based on the current LMWP data and analysis. In LMWP’s cash flow projections, there is a line item for head office allocation. The allocation amounts begin in 2016 for all three cities even though it is more reasonable to delay the payments for the first few years. In Voinjama projections, the amounts are US$185,138, US$184,067, US$182,849 and US$189,948 for 2016, 2017, 2018 and 2019. For Voinjama, the evaluation team conducted an illustrative sensitivity analysis. The evaluation team adjusted the head office allocation payments to 60 percent of the indicated projected values in first four years of operation (2016-2019) and revised the demand volume to 20 percent per year from 2018 which leads to O&M cost recovery in seven years. It should be noted that different sets of assumptions may yield different cost recovery periods. LMWP needs to conduct sensitivity analyses for all the three cities to find the combination of assumptions e.g. growing demand gradually, limiting head office allocation payments, revisiting other O&M cost components etc., and 10 Baseline Survey Report Sanniquellie, Voinjama and Robertsport May 9, 2012. Page 32 scenarios to make informed estimate of the cost-recovery period. Then it is critical to determine how to address subsidies for the years until cost recovery occurs. USAID, LWSC and GOL must come to an agreement. d. Are project resources well-positioned (staffing, resources, staff positions seconded to LWSC, etc.) to provide the requisite construction oversight, and capacity-building support to LWSC, efficient and effective to ensure quality and value to USAID and LWSC? LMWP’s resources for capacity building appear adequate but its resources including staffing for the construction phase are inadequate. Having the appropriate resources and deploying them diligently are critical for success of the CIP. LWSC must be at the core of the success of this project but its limited resources pose a challenge, if not a risk to begin assuming the ownership of the project. LWSC needs to secure the necessary resources to fulfill its expected role in the construction phase. Who will be responsible to provide those resources must be determined. The current construction phase staffing requires a careful review to ensure maximum utilization of senior construction management staffing. LMWP informed the evaluation team that resource planning, including the proposed staffing plan, is a work-in-progress. However, it is prudent for LMWP to re-visit and re￾evaluate the project/construction management tools after incorporating a VE/VA process, which takes into account construction procurement scenarios/options and the required resources for execution. Effectiveness of Institutional Framework and Capacity Building How effective is LMWP's current and proposed institutional and capacity-building approach in achieving the stated objectives, e.g. ensuring that "improved infrastructure will be handed over to locally-based management technically and financially" to sustain water supply service improvements in the three cities? LWMP’s proposed institutional approach is effective. The MOUs detail the background and context, the objectives, the responsibilities of the parties involved, and show the necessary information typical of MOUs. The MOU with Robertsport is working well as all parties are living up to their responsibilities. However, while the agreements provide a viable framework, several weaknesses need to be addressed which are discussed in detail below. LMWP’s capacity-building approach has worked well in both Robertsport and Kakata where participants spoke highly of the training LMWP has provided. Furthermore, operations at both stations have been running smoothly due to LMWP’s capacity building and training. For LWSC, LMWP conducted an assessment and employed previous studies to tailor its capacity building efforts to critical areas. a. Do the agreements established to date with the GOL regarding the institutional framework (particularly the city specific MOUs and the draft USAID-Ministry of Finance MOU) provide a viable framework for project implementation, and sustainability post-close-out? Is anything missing? LWSC has been a responsive partner to all the frameworks. The signing of the city specific MOUs indicates good faith from the stakeholders – LWSC, cities, LMWP – to see the CIP through. While overall the agreements do provide a viable framework the following issues need to be addressed:  The city-specific MOUs are valid through September 30, 2015, which corresponds to the base period of the LMWP contract. This date should be modified to at a minimum cover the construction phase. Page 33  The draft USAID-MOF MOU should explicitly address the party responsible for O&M support should the projected O&M recovery period not materialize.  The draft USAID-MOF MOU shows (on Page 9) a Robertsport tariff of US$0.045 per gallon and a common Sanniquellie and Voinjama tariff of US$0.035 per gallon in the computations for O&M costs and subsidies. LMWP used a common tariff of US$0.04 in its financial projections for the cities and should ensure the computations use the same inputs. The MOF has a state owned enterprise (SOE) monitoring unit which monitors the operational, technical, and financial details of the SOEs and holds them responsible and accountable. It is recommended that the MOF hold LWSC to the same level of accountability. b. Are there important lessons learned from the past decade of institutional reforms at Liberia Electricity Corporation and other electricity service expansion efforts in Liberia that should being informing LMWP’s work with LWSC or broader USAID engagement in water service delivery? The Liberia Electricity Corporation (LEC) went through reforms spearheaded by Manitoba Hydro International (MHI) via a five-year management contract set to expire in 2015. One of the incentives for a performance bonus for MHI was tied to the number of customer connections. The number of LEC customer connections went from 2,500 in 2010 to 29,000 in 2014. MHI did instill a level discipline at LEC which boosted the latter’s performance thus opening doors for donors and partners to commit more funds to the organization. MHI brought a level of confidence to LEC as well while providing tools to manage customer accounts. MHI also introduced pre-paid meters to enhance revenues. For customers without pre-paid meters, LEC is very quick to stop services for non-payment issues. LWSC can take two key lessons from LEC’s reform: 1. LWSC must focus on expanding its customer base via new connections while ensuring that all existing customers are in its database and that they pay their bills on time or risk disconnection. It could also explore pre-paid services for water. 2. LWSC must have the tools for effective customer management. LMWP can explore the option of using an external management contractor for customer management. c. Are LMWP's interventions to build the capacity of LWSC at the central level relevant and strategic, and effective to support project objectives, particularly as interim measures given the delays in expanding operational field presence into Sanniquellie and Voinjama? LMWP’s intervention to build LWSC’s capacity at the central level is both relevant and strategic. It is relevant due to the fact that a weak LWSC with limited resources poses a risk to the CIP and achieving O&M recovery. It is also strategic in the sense that LWSC needs resources to achieve its goals. Plotting the right strategy requires the allocation of the appropriate resources and the capacity to perform. The planned interventions require revisions and refinements in preparation for construction management and monitoring as well as O&M. The current capacity building work plan needs to be expanded. Illustrative activities to expand capacity building include project management, capacity building and training by the construction contractor, building decision support systems, vocational training, and revitalizing the existing training center. d. How well are the interim capacity-building efforts proceeding in Kakata and Robertsport as testing grounds for the proposed outstation institutional framework? The interim capacity building efforts in Kakata and Roberstport have been very effective and should be expanded to other outstations at the appropriate time. LMWP has provided training in asset management, Page 34 business planning, and O&M among other topics. Both the Kakata and Robertsport outstations spoke highly of the training provided. In Kakata, the Office In Charge, Ms. Nyenekon B. Snoh Barcon stated “I personally appreciated the training from Tetra Tech to enable us to monitor revenues and expenses, improve services and take corrective actions11.” She was discussing several LWMP capacity building exercises including computer training, and training in cost of services, O&M services, business plan development, performance targets, etc. She also expressed an interest in the introduction of plumbing technology and providing training for her technicians. LMWP’s training has increased the LSC’s and LWSC’s demand for further training and has given them the ability to identify areas of weakness where further capacity building is required. The training has also contributed to a sense of pride of ownership of the water facility (that is “not controlled by Monrovia”). Praise for LMWP’s training goes beyond the LSCs and it should be noted that a Deputy Managing Director at LWSC stated enthusiastically that the LSCs have been the best concepts to encourage local participation and a sense of ownership of the water facility. e. How effective is LMWP’s approach to monitor and assess the progress and results of its capacity-building work? LMWP’s plan to monitor and asses the progress of its capacity building work is still evolving. LMWP has developed capacity-building tools which have not yet been fully aligned with the capacity building work plan. The capacity building plan has to be assessed first to establish a baseline before progress can be measured. f. Is there a viable plan for helping LWSC to reach cost recovery in outstation operations? LMWP has strategically been positioned to support LWSC to reach operational cost recovery. The current plan includes a sustainability framework that generates monthly reports, focused training on asset management, business plan, cost of service training, IDAMC, and a set of operational tools. The tools include a cost data tracker to track operational costs and revenues; an O&M manual which provides standard operating procedures (SOPs) to run the plants (completed for Robertsport and in the works for Kakata); Sustainability Monitoring & Report which LMWP, LWSC and the LSCs dissect monthly for improvement opportunities; asset management; and a business plan. The tools and continued training provide the LSCs the needed knowledge to oversee O&M details for the water facilities. g. Does LMWP have an adequate approach to catalyze/enable continued LWSC investment in and expansion of the distribution network through private/household connections once the currently envisioned Capital Improvement Project is completed? LMWP’s approach is more than adequate as it continues to build LWSC’s capacity, provide training on cost of service, customer segmentation, etc., and has the same framework for all outstations. LWSC is committed to pursuing private customers through marketing services, advertisements, and community outreach, etc. but lacks the tools and resources e.g. connections need meters costing about US$75 to US$100 each (funds which LWSC does not have). LWSC should learn from the LEC and focus on marketing and expanding household and commercial connections as soon as the USAID-funded portion of the CIP is completed. Discussions with a high-level LWSC official indicate that plans are already in process to follow this course. Once the CIP system is in place, LWSC can seek other sources of financing, including user fees, to finance the meters for customers to facilitate customer acquisition. 11 Ms. Snoh Barcon stated this at her office in Kakata during a Key Informant Interview with the evaluation team on December 5, 2014 Page 35 h. Is more analytical work needed around market segmentation, costing of future distribution network improvements, social marketing and communications, setting up LWSC procurement mechanisms and systems that will be needed for O&M, e.g. to procure pipes, parts, services, etc.? Additional analytical work is required on market segmentation, costing of future network improvements, and procurements mechanisms. On customer segmentation, LWSC has to be able to identify which market segments to pursue to boost its revenue. It will need tools such as DSSs which can provide dashboards of key performance indicators to facilitate decision making. Behavior change is critical and LWSC has to be ahead of the curve even before the CIP is completed. There is a need for a communications specialist to provide education and awareness to consumers (e.g., the benefits of well water versus piped water). According to interviews conducted with the LWSC, procurement continues to be a challenge even without the added burden of the outstations. Interviews with LMWP project staff reinforced this point. LWSC struggles with procurement irregularities, management turnover, and a litany of problems outlined in the 2012 corporate audit12 of LWSC which are indicative of systemic issues that are not easily remedied in a short timeframe. LWSC will thus benefit from strengthening its procurement mechanisms and systems, including through appropriate training. Overall Project Positioning and Strategy for Phase-out of USAID Assistance Upon Completion of the Project Given delays in the construction design and implementation phase, the original timeframe underlying the project design (e.g. finish construction during LMWP 4-year base period, and then support O&M for 2 additional years) is no longer viable and it is likely that construction will just be finishing by the end of the LMWP's sixth year. Furthermore, much uncertainty remains around what exactly it will take for the outstations to reach operational cost recovery and how long this will take - estimates range from 1-3 years. In light of this situation: a. What options exist and what is recommended for ensuring achievement of the project objectives as quickly and efficiently as possible (follow on project, expanded operations support built into the construction contract, change/scale-back of the design, etc.)? Is a follow-on project needed and if so what should the focus, objectives, and level of effort/investment look like? The recommended options for ensuring achievement of project objectives include conducting a VE/VA to compare various options to meet the project objectives CIP at optimal cost, instituting a performance￾based contract, re-evaluating the economic and financial analysis, and project fast tracking and crashing. These options are discussed in detail in the Recommendations section below. LWSC figures prominently in the success of this project. It is critical to establish and support a functional and professional monitoring and evaluation (M&E) unit at the LWSC. LWSC also needs continued institutional capacity support which could be accomplished through utility-to-utility training via twinning programs for example. To improve operational efficiency, LWSC must establish DSS and the related tools. Over the long-term, LWSC needs a training center to facilitate training at a minimum for technicians, supervisors and mid-level managers. LWSC could seek additional support (such as from AFDB or WSP) to develop a combined training center to build staff capacity leveraging inputs from various donors/projects. 12 A Corporate Audit of Liberia Water and Sewer Corporation - Global Business Solutions, Inc., December 2012 Page 36 The CIP does not focus on sewerage. Therefore, a complementary investment in sewage is necessary once the CIP is operational for approximately five years. b. Is there anything USAID or LMWP can do differently in the interim to increase the likelihood of the GOL meeting objectives around operational capacity and cost recovery, or create better incentives for this to occur, e.g. what could we create in terms of benchmarks/ establishing a timeline for gradual phase-out for subsidies, and how / in what venue to negotiate this, etc.? To increase the likelihood of the GOL meeting objectives around operational capacity and cost recovery, USAID needs to communicate to the MOF how long it is willing to support O&M. The USAID-MOF draft MOU should be the overarching framework defining each party’s obligations. However, the MOU does not explicitly state who will be responsible for covering O&M costs should the projected O&M recovery period not materialize. It should be revised to address this issue. As mentioned above, there are currently eight SOEs that fall under the SOE Monitoring Unit and are held to account. The GOL and MOF should hold LWSC to the same standards (financial, technical, operational benchmarks) as other SOEs. c. What can be done to ensure a closer working relationship between LMWP and LWSC and promote host country ownership of outstation operation? Initial resistance of LWSC to the LMWP project reportedly arose from comparison of the LMWP with the AfDB-funded model for the UWSSP project. UWSSP provides direct budget support to the LWSC and the LWSC initially expected LMWP to employ the same project model. LMWP and LWSC now have a good working relationship but there is still room for strengthening the relationship and partnership. Embedding people at LWSC and focusing on small victories – success of quick impact projects, taking study tours together, sharing plans, etc. will also foster a closer working relationship between LMWP and UWSSP. d. Is the project sufficiently integrating/coordinating with AfDB investments through UWSSC? There is coordination between USAID (LMWP) and AfDB (UWSSP) at the top to share planning details, and there are indications that this coordination and cooperation will continue. LMWP helped LWSC to establish the LSC in Kakata – an AfDB-supported town – and there are plans to help with establishing LSCs at other outstations. LMWP is now coordinating plans and sharing details with UWSSP to avoid duplication and overlaps. LMWP’s Performance against Contract Requirements This section presents a brief review of LMWP’s performance against the eight tasks in Section C of the contract. Aspects of these are discussed in the findings section above and summarized by task here. Task 1: Situational Analysis LMWP conducted a comprehensive situational analysis including reviews and assessments of existing infrastructure and water sources, assessments of capacity at both the national and local levels, and the socio-economic environment when it comes to water usage, practices and willingness to pay. Task 2: Plans for Staged Water Supply and Sanitation Infrastructure Improvements LMWP developed plans for the water supply in a very structured and logical manner. From the situational analysis, it developed a baseline and the basis of design reports detailing what was to be improved, the water sources, types of treatment, new facilities, etc. and options to meet the project objectives. Page 37 Task 3: Institutional Framework for Water Provision Based on the situational analysis, LMWP determined that local governments have no role in construction, management, and operation of piped water services. Furthermore, LWSC has no presence locally except for a caretaker to look over the remnants of the facilities for encroachment and bush clearing. This led LMWP to the establishment of LSCs and a national Transition Working Group (TWG) whose key function was to explore various options and provide recommendations on the appropriate institutional framework for the management of the outstations based on international best practices and local realities. LMWP also introduced the idea of an IDAMC to allow LWSC to “outsource” operation of the outstations to a chosen entity. In addition, LMWP drafted MOUs to ensure stakeholders knew their responsibilities based on the objectives to be achieved. Task 4: Capital Works LMWP has designed the capital works to meet the project objectives. The design took into account treatment options, distribution, and the energy source which is a hybrid system of solar power and diesel generators. The design was done with an eye towards financial sustainability; however, the evaluation team recommends that LMWP revise the economic and financial analyses to address the issues with O&M recovery and affordability discussed above. Task 5: Transitional Management of Water Supply Improvements LMWP has already put a plan in place for a team to manage the water improvements post-construction to allow for a smooth transition to the selected operator. The team comprises three system overseers/managers, three operations consultants, three maintenance/mechanic consultants and three electrical power supply/solar consultants. Each of the three LSCs will thus have a manager, an operations consultant, a maintenance/mechanic consultant, and an electrical power supply/solar consultant to provide oversight of the respective teams running operations and maintenance of the water systems. The budget for this oversight has already been submitted to USAID. Task 6: Capacity Building LMWP has introduced important tools and concepts:  Tools: LSCs the Design Review Team (DRT) MOUs; Transitional Working Groups (TWG); Engineering Working Group.  LWSC to decentralize and delegate O&M responsibilities to the city and county officials and outstations (e.g., joint control over revenue) which created a sense of local ownership to provide best services. Task 7: Coordination with Other USAID Programs LMWP is successfully coordinating with other USAID programs. It is also coordinating with other water sector programs such as the AfDB’s UWSSP. Task 8: Planning and Reporting LMWP has developed the following monitoring tools: sustainability & monitoring framework; M&E framework; sustainability monitoring tools (protocols, reporting templates, schedule); and financial procedures and guidelines. CONCLUSIONS LMWP has in light of many socio-economic, environmental and institutional challenges and constraints of a post-conflict country like Liberia, accomplished considerable key objectives of an enabling environment as envisioned and articulated in the original and evolving objectives of the project. Page 38 USAID is implementing LMWP in a very difficult environment characterized by a lack of local construction capacity, a major health crisis (Ebola), human resource challenges, institutional capacity limitations, etc. which all adversely impact the project execution if LMWP does not take an appropriate mitigation strategy to reduce any related risks. Regarding the effectiveness of infrastructure planning/construction oversight approach, LMWP employed all the necessary tools and key concepts in designing the CIP. However, because it did not apply a VE/VA through the design, the CIP may not necessarily represent value for money. The current financial and economic analysis put the projected one to three year period to reach O&M cost recovery in serious doubt which adversely impacts the prospects for the project’s sustainability. The inadequate management/staffing structure to support the construction phase could pose safety problems and lead to improper construction/contract management and cost overruns. The lack of resources at LWSC poses similar problems. Pertaining to the effectiveness of institutional framework and capacity building, LWMP’s current and proposed institutional approach has been effective. The effectiveness of the current approach of building consensus through TWGs and LSC meetings, formalizing understandings through MOUs, and providing complementary training and technical assistance, has been one of the major highlights of the project. The tools, concepts and capacity building programs have been the foundation of LMWP’s success. Thus, the model should be replicated at LWSC and the outstations. While the city-specific MOUs and the USAID￾MOF draft MOU are an important start, LMWP can make them even more effective by addressing the key weaknesses discussed in the findings section above. Regarding the overall project positioning and strategy for phase-out of USAID assistance upon completion of the project, LMWP should consider employing the principles of VE/VA to the construction phase of the project with goal of eliminating unnecessary costs while maintaining the project’s function, quality and owner’s vision through creative construction solutions. Regarding the likelihood of the GOL meeting objectives around operational capacity and cost recovery, USAID must make its intentions on the amount of time it is willing to support O&M very clear and communicate them to the MOF. All the challenges notwithstanding, LMWP is a foundational investment integrating both hard soft aspects of development which will bring dividends to Liberia when it comes to participatory and iterative processes in project development, development of capacity building tools, attitudinal changes on paying for utility services, creation of an enabling environment, etc. RECOMMENDATIONS The evaluation team, in consultation with USAID, identified three levels of priority recommendations to reflect the degrees and levels (1-3) of importance (high-low) of the proposed recommendations to allow timely and concurrent, yet priority driven improvements to the project. Determination of the priority level is based on the severity of impact on cost, schedule, timing for implementation, sustainability, safety and technical performance. Priority Level 1 recommendations have a high impact on costs, scheduling, timing, safety or technical performance. Most, if not all of the recommendations, should take place pre￾construction to mitigate risk, improve costs, scheduling and technical performance. Priority Level 2 recommendations have a medium to low impact on costs, schedule or performance. They can be implemented (medium-term) after the construction phase except for the on-the-job training which can occur during construction. In addition, they have little or no impact on safety. Priority Level 3 recommendations focus on long-term sustainability and can be implemented after construction. Page 39 Priority Level 1 1. Conduct and apply the principles of value engineering/value analysis (VE/VA) to the design and construction of proposed improvements prior to and during construction. The purpose of VE/VA is to simulate alternative design options to ensure value for function while minimizing cost (e.g., selection of most cost effective material and balancing recommended solutions for restorations, upgrades, and replacement/new improvements). The implementation of the recommended CIP could potentially be improved by employing the principles of VE/VA during construction phase of the project with goal of eliminating unnecessary costs while maintaining the project’s function and quality through creative construction solutions. For any given project, a typical VE/VA study reveals the optimum blend of scheduling, performance, constructability, maintainability, environmental awareness, safety, and cost effectiveness. The evaluation team recommends that LMWP consider the application of a VE/VA prior to and during the construction phase of the project to incorporate the potential cost implications of the recent Ebola outbreak in the region and to mitigate the concomitant health risks at project sites. VE/VA continues during construction because a construction contractor's practical experience and purchase options can often generate substantial savings. During the construction phase, a construction contractor can propose a value engineering change to construction requirements, materials, or methods, and the construction contractor shares in the savings. The change may reduce the cost of construction or the life-cycle cost of the project without compromising performance, design quality, safety, functionality, or ease of maintenance. Typically, the construction contractor gets a share of construction cost savings for fixed price contracts, but can be different for incentive-based contracts. The proposed CIP involves both the rehabilitation of a few of the existing water supply infrastructure (e.g., intake weir, elevated tanks, slow sand filtration basins, etc.) as well as the installation and construction of new facilities. Since as-built/as-constructed documents were not available and LMWP conducted the condition assessments of the existing infrastructure visually, the VE/VA process should also be incorporated into re-evaluating the condition assessment of the infrastructure components to be retained as part of the hybrid infrastructure. The consequences of not conducting VE/VA are a missed opportunity to gain possible substantial cost savings (reduced capital costs) and improve the project schedule. The consequence of not undertaking structural analysis of the existing structures as part of the VE/VA study is potential failure of the existing elevated water tanks if integrity and functionality are not established. 2. Re-evaluate the economic and financial analysis including identifying options through sensitivity analysis. A “financial analysis” usually focuses narrowly on the costs incurred to build and operate a project, and the revenues that then accrue to the project implementers. An “economic analysis” adds to this more broadly societal benefits such as job creation or water access, and societal costs such as environmental impact. The revenue projections for the CIP are aggressive and overly optimistic. The evaluation team recommends that LMWP revisit the assumptions, discount rates, period to recovery of O&M costs, etc. and analyze various scenarios to develop options for functionality at optimal cost. The current cost model and analysis require further refinements to better understand the sensitivity of the results. The following revisions should be made:  The basis for the discount rate is not clear and it should be revised.  Calculate the hurdle rate to determine the minimum rate of required return on the investment.  LMWP must revisit its demand assumptions and collection rates to determine a realistic payback period. Page 40  Analyses of LMWP’s water demand and willingness to pay data show that depending on the location, beginning in either 2018 or 2019, the rates become unaffordable (for each household, the projected willingness to pay amount per month exceeds the projected monthly household bill) making the cash flow projections unjustifiable from 2018/2019 to 2031. The LMWP CBA includes mostly social benefits (opportunity costs) which are difficult to quantify and can swing widely depending on the assumptions. LMWP must ensure that the data and assumptions underlying the estimation of costs and benefits are reliable and realistic. Additionally, the CBA does not address any risk issues. LMWP should identify risks e.g. examining each variable to assess the level of uncertainty involved, and use risk assessment techniques to assess the level of risk and the impact of risk on project performance (such as sensitivity analysis, scenario analysis, expected values and Monte Carlo analysis. LMWP should devise a risk management strategy including measures to contain, avoid and mitigate risks, as appropriate and communicate the risk management to the relevant stakeholders. The consequences of not taking this action are unrealistic revenue projections which jeopardize the CIP’s sustainability and unjustifiable cost-benefit ratios which results in higher benefits than they should be due to a lower discount rate and unrealistic cash flows. This is high a risk item due to the likely adverse impact on O&M sustainability. The evaluation team suggests that LMWP implement this recommendation to ensure the project is on solid financial ground when it comes to O&M sustainability before proceeding further. 3. Conduct a two-stage procurement process. The key issue for a project sponsor is to encourage a contractor to convert from a low-risk, cost-plus-fee reimbursable pre-contract service agreement to a cost-certain and time-certain contract in which the contractor bears the design and construction risk. Hence the advent of two-stage tendering. The first stage is used for determining responsiveness to the request for offers and for clarifying and reaching agreement on the technical specifications. In this first stage, bidders are requested to submit a technical proposal with their best solution for fulfilling the requirement. The proposal is evaluated and scored, and the firm or top two firms with the highest ranked technical proposals are invited for discussions with the purpose of reaching agreement on the proposed technical solution. This stage of the procurement will encourage the contractor(s) to develop plans for:  Maximizing the utilization of local construction capacity;  Understanding, incorporating, and mitigating Ebola-related health risks; and  Developing a work breakdown structure (WBS) that allows incremental decomposition of the project into phases, deliverables and work packages that will identify and mitigate potential risk of procuring and importing the construction material from outside Liberia. Stage two is for receiving the final offer of the technically highest ranked firms. This process is very similar to the quality-based selection (QBS) procedures used to request proposals from consulting firms when the terms of reference (TOR) are difficult to define under highly specialized and complex requirements. In this second stage, if an agreement is reached, the technical specifications or TOR are finalized and the highest ranked firms are invited to first submit financial proposals based on the agreed technical solutions, and then to contract negotiations. Key Advantages of the two-stage tendering process are that:  It is a more flexible approach to awarding contracts because it allows participation of prospective bidders in the definition of the technical specifications and scope of work; Page 41  The preferred bidder is more likely to have a good understanding of the requirement, which potentially reduces risks in the implementation of the contract;  Prospective bidders are able to make suggestions for improvement of the technical specifications and scope of work of the assignment, through their technical proposal and clarification discussions;  The technical approach and methodology can be adjusted to suit the agreed technical specifications and scope of work;  The early involvement of prospective bidders in the definition of the technical specifications and scope of work can minimize risk;  A financial proposal is submitted only after reaching agreement on the technical specifications and scope of work;  A contract is negotiated on the basis of the agreed technical specifications and scope of work; thereby reducing the second stage tendering time; and  More certainty regarding the qualifications of the preferred bidder. Key disadvantages of the two-stage tendering process are extended procurement lead-time due to the two-stage submission process and the potential intractable negotiations. 4. Institute performance-based contracting (PBC) with provisions for incentives. The concept of performance-based contracting is centered on a contract instrument that defines performance expectations in terms of outcomes or results as opposed to methods, processes, systems or broad categories of work activity. The PBC concept describes the work in terms of what the required output is supposed to be rather than how the work is to be accomplished. Another component of PBC is that the contractor bears responsibility for assuring quality performance. Based on the incentive structure, a PBC allows the contractor to employ innovative techniques to yield cost/time savings. Since the contractor’s compensation is tied to the achievement of the prescribed outcomes or results under PBC, it is quite critical for the project sponsor to develop formal and measurable performance (i.e. delivery, cost, and technical performance) standards, including surveillance plans, to facilitate assessment of the contractor’s performance. Given the prevailing atmosphere, LMWP can benefit from a PBC with the right cost incentives (share of savings, profit, adjusted fee, etc.), delivery incentives (reward for early completion; penalty for late completion) and multiple incentives (combination of cost and other incentives) which can serve as motivation for outstanding performance in all areas and compel the contractor to make trade-off decisions. Implementing this recommendation can align the interest of the contractor with that of LMWP/USAID and also incent the contractor to be creative in the construction phase to save time and money. 5. Evaluate and examine project fast tracking and crashing the construction schedule. Fast tracking and crashing are two project compression techniques used to shorten project schedules. In fast tracking, one reviews the critical path to find out which sequential activities can be performed parallel or partially parallel to each other. The benefit of fast tracking is that it does not cost any extra money; however, it comes with some increase in risks, because the contractor is performing many activities, which were originally planned in sequence, in parallel. In crashing, one adds extra resources to the project to compress the schedule. The technique involves reviewing the critical path to see which activities can be completed by adding extra resources. The notion it to try to find the activities that can be reduced the most by adding the least amount of cost and then applying the crashing technique. In light of timing challenges and citizens’ anxiety about getting access to potable water, LMWP should check these project compression techniques to see if the project could be completed faster at minimal or no additional cost. Page 42 6. Re-evaluate and re-visit resource planning including the proposed staffing plan and the project/construction management tools after incorporating VE/VA process to also include construction procurement scenarios/options. LMWP’s proposed staffing for the construction phase appeared inadequate for the anticipated construction challenges that may result in cost-overrun. Even though LMWP advised the evaluation team that the staffing plan was a work-in-progress, LMWP should re-evaluate its resource planning post a VE/VA study to ensure a robust staffing plan, the appropriate project/construction management tools, and construction procurement options aimed at maximizing costs with no loss in functionality. Taking this action pre-construction will ensure LMWP has allocated the appropriate resources to see the construction through. Neglecting this action and planning details in an ad hoc manner will result in possible delays with adverse cost implications. 7. Support LWSC to further recognize and overcome external and internal institutional challenges such as staffing, marketing, customer base, efficient billing, and revenue collections - LWSC should champion the execution of the provisions in all the agreements. LWSC’s internal and external challenges are numerous and LMWP should prioritize its institutional capacity building efforts to address such challenges. Staffing shortages are a key internal challenge as LWSC has had difficulty providing people for LMWP’s capacity building training. This is an ongoing challenge as LWSC endeavors to shuffle staff to manage operations. LMWP is aware of these challenges from its assessment of LWSC during the development of the former’s capacity building strategy. Even though a strategy was developed in collaboration with LWSC, there is a perception that LMWP needs to do more. The success of the CIP post construction hinges on LWSC’s ability to manage its operations with limited resources. Helping LWSC with systems to improve revenue generation while improving efficiency will enhance its ability to manage the CIP after installation. LWSC should recruit pertinent staff from its resources, GOL sources or other donors’ resources. 8. Utilize local construction capacity where feasible particularly for pipe laying. USAID’s attempt earlier to solicit bids for other local construction projects showed the lack of local construction capacity as bidders’ responses displayed their inability to read specifications and other bid documents correctly, pricing discrepancies, misunderstanding of contract documents, etc. This is not surprising since local contractors lack the experience and have limited or no access to financing for project execution. In spite of the limitations, the evaluation team and LMWP jointly believe that there are aspects of the project, such as pipe laying, that local contractors can reasonably handle to minimize cost and provide an opportunity for capacity building. 9. Revise, refine and expand the current capacity building work plan to include project management, capacity building and training by construction contractor, building decision support systems, including through approaches such as vocational training and revitalizing the existing training center. LMWP’s capacity building efforts have garnered positive reviews and should be expanded. In developing its capacity building plan, LMWP employed various tools including off-site managerial retreat, self-assessments, previous studies of LWSC, etc. to understand focus areas for intervention. The plan catalogued several areas for support and the needed intervention and LMWP developed its initial plan accordingly. However, LMWP should enhance its capacity building plan by including on-the-job training, training by the contractor, building decision support systems, and revitalizing the existing training center to facilitate training technicians, supervisors and some middle managers. 10. The MOU with the MOF and LWSC should be the overarching framework to hold LWSC responsive and accountable. The draft USAID-GOL MOU details the objectives and responsibilities Page 43 of all parties – LWSC, MOF and USAID. However, the MOU does not explicitly state who will be responsible for providing the needed financial support should the O&M cost recovery not occur in the projected time. The evaluation team recommends that the MOU be revised to address this. Priority Level 2 1. On-the-job training (OJT) construction management capacity building (i.e., learning while doing). On-the-job training provides a hands-on experience to build construction management capacity which is sorely lacking in Liberia. OJT is often one of the best methods for knowledge transfer since it involves learning by doing under the tutelage of an expert. OJT can happen in a variety of ways from the trainee shadowing an experienced colleague, to a supervisor observing and providing feedback, and coaching. The direct benefits of OJT in this case are that the trainees will be very familiar with what goes into a water infrastructure system, running and maintaining the system, and troubleshooting problems. The experience will facilitate running O&M after the construction. LMWP and the contractor should work on modalities to explore the best way to execute this. 2. Support establishing a functional and professional M&E unit at LWSC. LWSC lacks resources that a typical water utility needs to operate efficiently. There is no M&E unit to provide the tools and performance reports management needs to run and improve operations. A professional M&E unit will provide an avenue for continuous learning and the details/tools for LWSC management to manage priorities: what works well, what requires tweaking and what changes are necessary for effective operations. Such a unit in conjunction with planning can:  Coordinate formulation and preparation of annual plans, business plans and medium-term strategic plans;  Provide technical guidance and support for institutionalization of strategic planning and budgeting process;  Participate in analysis of outsourcing of non-core functions (Private Sector Participation);  Monitor implementation of annual plans, business plans and medium-term strategic plans;  Prepare periodic performance reports;  Collect, study and analyze statistics needed in the formulation and implementation of plans and budgetary proposals;  Participate in preparing plans, programs and budgetary activities of LWSC and establishment of performance targets and indicators;  Provide technical support including institutionalization of M&E process within LWSC;  Conduct research and impact studies of plans, projects and programs undertaken by LWSC;  Undertake service delivery surveys to collect stakeholders’ and clients’ views on services rendered by the LWSC; and  Coordinate mid-year and annual performance reviews. It should be noted that many of these activities should be carried out at the outstation level given the agreed upon institutional framework where the outstations are independently operated. 3. Need for more analytical work around market segmentation, costing of future distribution network improvements, social marketing and communications and procurement support. LWSC needs the ability to determine profitability by market segment, and the appropriate tools to conduct such analytical work. Currently, evidence of profitability of various market segments is more anecdotal and ad hoc than analytical. Customers must be educated to see the benefits of paying for Page 44 treated water versus untreated “free” well water. A communications specialist is needed at LWSC to provide education and awareness, (e.g. benefits of well water versus pipe-borne water). LWSC needs procurement mechanisms and related training to facilitate its operations. 4. Support the establishment of a Decision Support System (DSS) for senior and mid-level management for efficient operation at LWSC. A DSS is a computer-based application that collects, organizes and analyzes business data to facilitate quality business decision-making for management, operations and planning. Well-designed DSS applications are systems and subsystems that help businesses make decisions based on data that is culled from a wide range of sources: raw data, documents, internal/external databases, employees, management, executives and business models. A key component to any DSS is business intelligence reporting tools, processes, and methodologies. DSS analysis helps companies to identify and solve problems, and make decisions. LWSC requires such a tool to facilitate efficient operational management. For example, a DSS could include systems or sub-systems to provide a dashboard of indicators and key drivers for a water utility to show pertinent details at a glance. Some of the key benefits of a DSS include speeding up the decision-making process, increasing organizational control, expediting problem solving in an organization, helping automate managerial processes, improving personnel efficiency, and eliminating some unnecessary value chain activities. For LWSC, a DSS will help it to make better decisions by providing a flexible tool for analysis while compelling it to better manage its data. Unlike the other SOEs under the tutelage of the MOF, LWSC is not held accountable for its operational, technical and financial performance. Using its diplomatic clout and relationship with GOL, USAID should work with the MOF to ensure that LWSC, like other SOEs that MOF monitors, is held accountable and responsible. 5. Expand capacity building to LSCs. LMWP’s capacity building efforts at both Robertsport and Kakata have been very effective. Both outstations spoke highly of the training provided (asset management, business plan, O&M, etc.). Expanding capacity building training to other LSCs will benefit LWSC and allow the LSCs to provide better oversight of the O&M functions. The notion here is to replicate a successful model that all key stakeholders are very happy with. Without the expansion, the other LSCs will not benefit from the positive experience of the successful LSCs and LMWP’s knowledge of the pertinent programs and capacity building exercises to facilitate O&M management. Priority Level 3 1. Support LWSC to replicate the LSCs in all other outstations to promote confidence in LWSC central management. Local control and “ownership” of the water infrastructure via the LSCs has had a very positive effect and engendered much optimism. Replicating the LSCs at all outstations will not only facilitate their management but also let the outstations know they are all equally important. 2. Support innovative institutional capacity building approaches that promote South-to-South learning, e.g. utility-to-utility training via twinning programs. Exposure is critical for LWSC managers so that ideas on operating a successful utility will not appear abstract to them. Twinning programs with other successful utilities using the IDAMC concept will facilitate understanding and the workings of such a concept. This includes how the successful utilities use key operational indicators such as total connections, service coverage, non-revenue water, water losses per connection per day and labor productivity (staff per 1,000 connections) in IDAMC. Sending LWSC personnel to successful water utilities using a similar operational model is a great learning experience to see how such utilities work. Page 45 3. LMWP should establish and monitor a baseline capacity-building to measure success / progress over time. Since LMWP has developed several capacity building tools and has already started implementation, it is imperative for it to have a baseline so that it can measure the effectiveness of the capacity building efforts. Page 46 APPENDICES Appendix 1. Evaluation scope of work I. BACKGROUND OF PROGRAM TO BE EVALUATED In September 2011, USAID awarded a $14.5 million, four-year (with an option to extend for two years and up to a total estimated cost of $19.7 million) to Tetra Tech, Inc. for implementation of the Liberia Municipal Water Project (LMWP). The objective of LMWP is to support the design, tendering, execution and operation of water supply infrastructure improvements in three Liberian county capitals - Robertsport, Sanniquellie, and Voinjama. The Contractor is tasked with assisting local and national authorities in developing plans for water supply and sanitation improvements, overseeing construction of this Capital Improvement Project, supporting initial operations of water supply infrastructure improvements, and re￾establishing local capability to sustainably operate and maintain constructed water supply systems. The primary design criteria for the improved infrastructure is that by the end of the project, over 90% of the population in each city will have improved water supply access13 and improved infrastructure will be handed over to a locally-based operator capable of financially and technically sustaining water supply services. In tandem with and immediately following construction, the LMWP Contractor, Tetra Tech, is required to develop local capacity to maintain, manage and operate USAID-funded water system improvements in the three cities and create conditions favorable for further expansion of such improvements after the end of the project. It is also expected that, through the Contractor’s collaboration with other USAID programs, the improved water systems will provide substantial health benefits and increased business opportunities in the three cities. The LMWP Contractor is responsible for the following tasks in each of the three cities:  Assessing the existing water supply situation, environmental context, demand for improvements, and institutional capacities.  Developing recommendations, for USAID/GOL approval, on sustainable improvements in water supply service over the short, medium and long term;  Developing recommendations on the respective roles of various GOL entities including LWSC, public works, and county and/or municipal government. Coming to agreement with government officials on an institutional framework to manage proposed water supply infrastructure improvements on a sustainable basis.  Completing designs, providing procurement documents and oversight, and providing construction oversight on agreed short- and medium-term water supply improvements. The Contractor is not carrying out actual construction, and instead USAID will engage a construction contractor directly. The LMWP Contractor will oversee the construction procurement process and provide construction supervision.  Supporting operation and maintenance of improved water supply infrastructure on a transitional basis. 13 “Improved water supply access” is defined as regular household access to a water source, a distribution system, or a delivery point, which by the nature of its design and construction is likely to protect the water source from external contamination, in particular from fecal matter, and can be reached by the household in a round trip of 30 minutes or less. Improved water supply sources are: piped water into dwelling, plot, or yard; public tap/standpipe; tube well/borehole; protected dug well; protected spring; or rainwater collection. Unimproved water supply sources are: unprotected dug well; unprotected spring; cart with small tank/drum; tanker truck; surface water (river, dam, lake, pond, stream, canal, irrigation channel); and, bottled water. Page 47  Building the capacity of the GOL, LWSC, county governments, municipal authorities, and the selected local water service management to effectively play their role in the agreed institutional framework by the end of the contract period. II. PURPOSES AND USES OF THE EVALUATION The primary purpose of the evaluation is to determine whether the assistance provided by USAID through LMWP is meeting the objectives outlined in Section I and the contract document. Additionally, the evaluation should provide a detailed picture of the major accomplishments, strengths and weaknesses of the project since inception, indicating as well any recommended changes in LMWPs implementation approach and/or USAID’s project design / programming approach to assure successful completion of urban water improvements of in line with the objectives outlined in section I. This should include specific recommendations for successful completion of the four-year base contract in September 2015 as well as the two option years in the event they are exercised by the USAID contracting officer, including any key issues / changes recommended to the implementation approach and programmatic priorities for the option years. Finally, the evaluation will identify priority areas that should be the focus of future programming in the Liberian urban water supply sector in support of the overall objectives of the Liberia Municipal Water Project and of the Liberia Country Development Cooperation Strategy (CDCS) for 2013-17. The timing of this evaluation is propitious for making project corrections as USAID advances plans to begin capital works in line with the plans developed by LMWP for the three cities accelerates over the coming months. III. PROPOSED EVALUATION QUESTIONS 1. Effectiveness of infrastructure planning / construction oversight approach: Is the proposed infrastructure design well-suited to achieve project objectives in a cost-effective and sustainable manner, to what extent was appropriate due diligence done in the design, and how well is the project positioned (in terms of management/staffing structure) to support the construction phase? In answering this overarching question, the Evaluation Team should consider: a. Do the overall plans for each city (Water Master Plans and designs developed by LMWP, including costs and Cost Benefit analyses) appear reasonable and indicate value for money in terms of achieving project objectives and ensuring judicious use of US Government funds? Are the proposed design approach and cost reasonable in comparison to similar projects (in Liberia or other settings) and taking into consideration the size of the target populations, institutional sustainability issues, etc.? b. How much variability / uncertainty exists around the current cost estimates and what could be done (in terms of RFP or contract provisions, phasing of works, etc.) to minimize risk of cost overruns while ensuring delays are avoided and GOL / target city expectations are met. c. How well has LMWP conducted due diligence in analyzing and understanding sustainability issues facing the Capital Improvement Project, e.g. are projected revenue/cost assumptions, and the timeline to reach cost recovery realistic and reasonable? d. Are project resources well-positioned (staffing, resources, staff positions seconded to LWSC, etc.) to provide the requisite construction oversight, and capacity-building support to LWSC, efficiently and effectively to ensure quality and value to USAID and LWSC? Page 48 2. Effectiveness of institutional framework and capacity building: How effective is LMWP's current and proposed institutional and capacity-building approach in achieving the stated objectives, e.g. ensuring that "improved infrastructure will be handed over to locally-based management technically and financially" to sustain water supply service improvements in the three cities? In answering this overarching question, the Evaluation Team should consider: a. Do the agreements established to date with the GOL regarding the institutional framework (particularly the city specific MOUs and the draft USAID-Ministry of Finance MOU) provide a viable framework for project implementation, and sustainability post-close-out? Is anything missing? b. Are there important lessons learned from the past decade of institutional reforms at Liberia Electricity Corporation and other electricity service expansion efforts in Liberia that should being informing LMWP’s work with LWSC or broader USAID engagement in water service delivery? c. Are LMWP's interventions to build the capacity of LWSC at the central level relevant and strategic, and effective to support project objectives, particularly as interim measures given the delays in expanding operational field presence into Sanniquellie and Voinjama? d. How well are the interim capacity-building efforts proceeding in Kakata and Robertsport as testing grounds for the proposed outstation institutional framework? e. How effectively is LMWP’s approach to monitor and assess the progress and results of its capacity-building work? f. Is there a viable plan for helping LWSC to reach cost recovery in outstation operations? g. Does LMWP have an adequate approach to catalyze/enable continued LWSC investment in and expansion of the distribution network through private/household connections once the currently envisioned Capital Improvement Project is completed? h. Is more analytical work needed around market segmentation, costing of future distribution network improvements, social marketing and communications, setting up LWSC procurement mechanisms and systems that will be needed for O&M, e.g. to procure pipes, parts, services, etc.? 3. Overall project positioning and strategy for phase-out of USAID assistance upon completion of the project. Given delays in the construction design and implementation phase, the original timeframe underlying the project design (e.g. finish construction during LMWP 4- year base period, and then support O&M for 2 additional years) is no longer viable and it is likely that construction will just be finishing by the end of the LMWP's sixth year. Furthermore, much uncertainty remains around what exactly it will take for the outstations to reach operational cost recovery and how long this will take - estimates range from 1-3 years. In light of this situation: a. What options exist and what is recommended for ensuring achievement of the project objectives as quickly and efficiently as possible (follow on project, expanded operations support built into the construction contract, change / scale-back of the design, etc.)? Is a follow-on project needed and if so what should the focus, objectives, and level of effort/investment look like? b. Is there anything USAID or LMWP can do differently in the interim to increase the likelihood of the GOL meeting objectives around operational capacity and cost recovery, or create better incentives for this to occur, e.g. what could we create in terms of benchmarks/ establishing a timeline for gradual phase-out for subsidies, and how / in what venue to negotiate this, etc.? Page 49 c. What can be done to ensure a closer working relationship between LMWP and LWSC and promote host country ownership of outstation operation? d. Is the project sufficiently integrating / coordinating with AfDB investments through UWSSC? IV. EVALUATION METHODOLOGY The Evaluation Team will use the following methodology to conduct the evaluation, organized below by each of the three key evaluation components noted in section III. 1. Effectiveness of infrastructure planning / construction oversight approach. USAID expects this will be assessed largely through document review of key formative assessments, plans, and infrastructure designs, and key informant interviews in Monrovia.  Document review to include: o Situational analyses and master plans for water provision for each town, including cost benefit analyses o Detailed cost and revenue projection data from LMWP o USAID FAA Section 611(e) certification document o Environmental and Social Impact Assessment for capital works o LMWP Monthly construction reports (suggest review of 1-3 illustrative monthly reports from 2014) o Reference documents:  LMWP technical specifications, engineering design drawings, and tender documents and cost estimates for the water treatment plans and piped water distribution systems  Engineering designs and tender documents for African Development Bank (AfDB) Urban Water Supply and Sanitation Project (UWSSP), which is rehabilitating water infrastructure in three other Liberian towns (for comparison)  Key informant interviews o Tetra Tech engineering team and management team in Liberia and at the headquarters level o USAID engineering staff at the field and headquarters o LWSC leadership and engineering staff o Counterparts at the African Development Bank (AfDB) and key staff on the Urban Water Supply and Sanitation Project (UWSSP) in Monrovia o GOL counterparts in the Ministry of Lands, Mines and Energy (MLME) and the Liberia Environmental Protection Agency (EPA) o Outstation operators in 1-2 towns (Robertsport, Kakata suggested) 2. Effectiveness of institutional framework and capacity building:  Document review to include: o LWSC draft corporate strategy o LWSC corporate audit report (GBSI 2013) o Liberia Water Supply and Sanitation Policy (2009) and 2013 Sector Performance Report o LMWP deliverables related to institutional framework Page 50  Framework Options Assessment Report  LWSC Capacity-Building Plan  Sustainability Monitoring Plan  MOU between Tetra Tech and LWSC, MOU with outstations, and draft USAID￾LWSC-MOF MOU  LMWP business plan template and draft Robertsport business plan  LMWP Internally Delegated Area Management Contract (IDAMC) template  LMWP Sustainability Monitoring Plan Template and monthly reports for Robertsport (suggested sample of June-August 2015)  Cost of Service / Tariff Model – SUWASA project reports  LMWP Outstation audit template  Key informant interviews o AfDB and World Bank Water & Sanitation Program (WSP) water advisors o LWSC managing director (MD) and senior management in Administration, Technical Services, Finance, and Commercial departments o Members of the Local Steering Committee (LSC) in LMWP towns (Robertsport recommended) o Liberia Electric Corporation and USAID Mission staff familiar with LEC reform (and review of any relevant documents outlining the process / lessons of LEC reform)  Site visits o Given construction has not yet begun and there are neither existing systems nor LMWP or LWSC project staff in Sanniquellie and Voinjama, it is suggested that the team visit Robertsport (an LMWP focus town) and Kakata (a town near Monrovia with a currently operational water system, for institutional context / understanding) to view existing systems and meet with key informants 3. Overall project positioning and strategy for phase-out of USAID assistance.  Document review, in addition to documents noted for component 2: o USAID/Liberia CDCS o USAID Water and Development Strategy o Liberia WASH Strategy (reference) o Program and design documents related to the AfDB Urban Water Supply and Sanitation Program  Key informant interviews o Senior officials from LWSC and GOL counterparts o AfDB and World Bank Water & Sanitation Program (WSP) advisors o USAID staff and infrastructure/water advisors General / cross-cutting documents:  LMWP contract Statement of Work  LMWP Performance Monitoring Plan (PMP)  LMWP Annual Reports for FY13 and FY14  LMWP Annual Work Plans for 2013-14 and 2014-15  USAID/Liberia CDCS  USAID Water and Development Strategy Page 51 Note: If this is not feasible due to in-country travel limitations related to Ebola response, consultations with relevant staff may be done by phone or in-person in Monrovia. However, the Mission anticipates that travel should be possible to both Robertsport and Kakata. The contractor must engage a local counterpart / consultant to make all necessary logistical arrangements, including vehicle rentals, arranging meetings, site visits, etc. USAID will provide technical feedback and advice regarding key informants and the site visit itinerary, and review of key documents. USAID will also facilitate appointments with senior GOL counterparts as may be required during the course of the evaluation. USAID staff from the LMWP project will accompany the Evaluation Team as appropriate. Page 52 Appendix 2. List of Documents Consulted Overview Documents Overview Presentation, May 16, 2014 Snapshot: Ebola Not Stopping Efforts to Advance Safe Water Access in Robertsport, August 2014 Factsheet, October 30, 2014 Presentation to USAID Liberia Health Team, September 25, 2014 Task 1: Situational Analysis Situational Analysis Report, May 15, 2012 Task 2: Plans for Staged Water Supply and Sanitation Infrastructure Improvements Robertsport Master Plan, August 31, 2012 Voinjama Master Plan, December 16, 2012 Sanniquellie Master Plan, November 15, 2012 Task 3: Institutional Framework for Water Provision MOU between LWSC, Voinjama City Government, and USAID, July 23, 2014 MOU between LWSC, Sanniquellie City Government, and USAID, July 23, 2014 MOU between LWSC, Robertsport City Government, and USAID, March 20, 2014 MOU between Tetra Tech and USAID, June 20, 2013 Framework Options Report, June 2012 Fifth Transitional Working Group Workshop Report, May 9, 2013 LMWP Business Plan Template, April 24, 2014 LWSC Internally Delegated Area Management Contract for Operation and Maintenance of Water Supply Template, May 8, 2014 MOU between GOL and USAID (Draft), October 2014 Robertsport Business Plan and Cost Projections, July 22, 2014 Task 4: Capital Works Task 5: Transitional Management of Water Supply Improvements Operator Auditor Report: Robertsport Outstation, October 31, 2014 Sustainability Monitoring Report, July 2014 Sustainability Monitoring Report, August 2014 Task 6: Capacity Building Final Capacity Building Plan, April 17, 2014 Action Memorandum for Assistant Administrator for Africa, August 14, 2014 WASH Sector Performance Report, GOL, 2013 Cooperation Letter to LWSC from USAID and AfDB, September 30, 2013 Liberia’s Second Annual Water, Sanitation & Hygiene Joint Sector Review: Event Report, World Bank, UNICEF, May 2014 Draft Sustainability Monitoring Plan, April 23, 2014 Terms of reference: Management, Training and Commercialization Consultancy Services Task 7: Coordination with Other USAID Programs N/A Task 8: Planning and Reporting Page 53 LMWP Year 3 Work Plan, March 2014 LMWP Year 4 Work Plan, September 2014 LMWP PMP, August 2014 LMWP Year 2 Annual Report (Draft), November 4, 2013 LMWP Year 3 Annual Report (Draft), November 1, 2014 Page 54 Appendix 3. List of Individuals and Organizations Contacted No Name Position Organization Phone # Email: Date USAID Washington DC 1 Anthony Kolb Urban Health Advisor USAID 202-712-0258 akolb@usaid.gov Nov. 21, 2014 USAID Liberia 2 Courtney Babcock M&E Officer, Prog. & Proj Dept Office USAID 231- 776777300 cbabcock@usaid.gov Nov. 25, 2014 3 Ben Zinner Health System Sub-Team Leader USAID 23177- 6777109 Nov. 25, 2014 4 Luis Velazquez Engineering Officer, Econ. Growth Office USAID 231- 776777000 Lvelazquez@usaid.gov Nov. 25, 2013 STAFF OF Tetra Tech USAID/LMWP 5 Matt R. Harder, P.E Construction Project Manager Tetra Tech, USAID/LMWP 231- 770373658 mharder@tetratech-lmwp.com Nov. 26, 2013 6 Elmos B. Glay Deputy Chief of Party Tetra Tech, USAID/LMWP 231- 886550962 eglay@tetratech-lmwp.com Nov. 26, 2014 7 Alioune Fall Chief of Party Tetra Tech, USAID/LMWP 231- 777332712 afall@tetratech-lmwp.com Nov. 26, 2014 8 Magdalene Matthews Program Manager & Water Quality Specialist Tetra Tech, USAID/LMWP 231- 770373626 mmatthews@tetratech￾lmwp.com Nov. 26, 2014 STAFF OF LWSC 9 Himmieh Kuoduor DMD-Fin LWSC 886778864 Kuoduor1998@gmail.com Nov. 26, 2014 10 Frankie N. Cassell DMD-TS LWSC 886453554 Fnatcass54@gmail.com Nov. 26, 2014 Page 55 No Name Position Organization Phone # Email: Date 11 Gabriel Flaboe, Sr. UWSSP Manager LWSC 886513239 lwscproject@gmail.com Nov. 26, 2014 12 Christian Yeakula Outstation Manager LWSC 888335950 cnyeakula@yahoo.com Nov. 26, 2014 RICHARD ENGINEERING 13 Walker Richards Civil Engineer Richards Engineering 231- 777377724 wrrmonrovia@yahoo.com Dec. 1, 2014 STAFF OF Tetra Tech USAID/LMWP 14 Alioune Fall Chief of Party Tetra Tech, USAID/LMWP 231- 777332712 afall@tetratech-lmwp.com Dec. 1, 2014 15 Matt R. Harder, P.E Construction Project Manager Tetra Tech, USAID/LMWP 231- 770373658 mharder@tetratech-lmwp.com Dec. 1, 2014 TETRA TECH LEAD ENGINEERS Conference Call) 16 Thomas Keeffe, P.E. Sr. Water Resource Project Manager Tetra Tech, MA 508.903.2090 thomas.keeffe@tetratech.com Dec. 1, 2014 17 Dave Favazza Dec. 1, 2014 18 Natalie Brown Dec. 1, 2014 MINISTRY OF PUBLIC WORKS 19 Min. George W.K. Yarngo Assistant Minister, Bureau of Community Services MPW 231-777-511- 171 Dec. 2, 2014 20 G. Woloba Karwee Director/Rural Water Division MPW 231-886-455- 530 Dec. 2, 2014 21 Bearford O. Weeks MPW Consultant MPW 231-886-514- 005 Dec. 2, 2014 LAND, MINES AND ENERGY 22 Jerferson Willie Director of Hydrology MLME 231-777-164- 821 jeffersonw.wylie@yahoo.com Dec. 2, 2014 STAFF OF Tetra Tech USAID/LMWP Page 56 No Name Position Organization Phone # Email: Date 23 Matt R. Harder, P.E Construction Project Manager Tetra Tech, USAID/LMWP 231- 770373658 mharder@tetratech-lmwp.com Dec. 3, 2014 24 Elmos B. Glay Deputy Chief of Party Tetra Tech, USAID/LMWP 231- 886550962 eglay@tetratech-lmwp.com Dec. 3, 2014 25 Safaa Fakorede Utility Operations & Institutional Specialist Tetra Tech, USAID/LMWP 231-770-214- 932 sfakorede@tetratech-lmwp.com Dec. 3, 2014 26 Alioune Fall Chief of Party Tetra Tech, USAID/LMWP 231- 777332712 afall@tetratech-lmwp.com Dec. 3, 2014 MEETING WITH SUPERINTENDENT, GRAND CAPE MOUNT COUNTY 27 Abraham B. Bockarie Assistant OIC-LWSC LWSC 231-886-377- 895 abrahambockarie@gmail.com Dec. 4, 2014 28 D. Ignatious Roberts Correxpondent/MICAT Ministry of Information 231-886-597- 236 Dec. 4, 2014 29 Matt Harder, P.E Construction Project Manager Tetra Tech, USAID/LMWP 231- 770373658 mharder@tetratech-lmwp.com Dec. 4, 2014 30 Col. AL Nimely Wisnor County Commander Immigration 231-886-632- 305 Dec. 4, 2014 31 Mohamed A. Paasewe Superintendent 231-886-517- 418 m_paasewe@yahoo.com Dec. 4, 2014 32 Sylvester V. Lormie Director/CSIO/LISGIS LISGIS 231-886-528- 233 s.vokerlormie@yahoo.com Dec. 4, 2014 33 Eric V. Pinney County Inspector Ministry of Justice 231-886-742- 745 Dec. 4, 2014 34 Lorenzo D. Smith Site Engineer Tetra Tech, USAID/LMWP 231-886-550- 096 smithlorenzo20333@yahoo.com Dec. 4, 2014 35 Anne Ogbigbo HRO/Acting Head of Office UNMIL 231-886-236- 368 ogbigbo@un.org Dec. 4, 2014 Page 57 No Name Position Organization Phone # Email: Date MEETING WITH LOCAL STEERING COMMITTEE, GRAND CAPE MOUNT COUNTY 36 Abraham Bockarie Assistant OIC-LWSC LWSC 231-886-377- 895 abrahambockarie@gmail.com Dec. 4, 2014 37 Baindu Sombai Committee Member LSC 231-886-761- 223 Dec. 4, 2014 38 Fatu Williams Secretary and Gender MGD and LSC 231-776-355- 141 Dec. 4, 2014 39 James G. Kiazolu - l City Mayor City Council 231-777-169- 925 Dec. 4, 2014 40 B. Mohamed Pusah RLSC/Communication LSC 231-880-565- 390 Dec. 4, 2014 41 Ethel McCrimada Committee Member LSC 231-886-900- 109 Dec. 4, 2014 42 Lorenzo Smith Site Engineer Tetra Tech, USAID/LMWP 231-886-550- 096 smithlorenzo20333@yahoo.com Dec. 4, 2014 43 Matt Harder, P.E Construction Project Manager Tetra Tech, USAID/LMWP 231- 770373658 mharder@tetratech-lmwp.com Dec. 4, 2014 44 Paul W. Bropleh I Committee Member LSC 231-777-023- 238 Dec. 4, 2014 CONFERENCE CALL WITH AfDB FROM GHANA 45 Mwasambili Rees Mwasambili Rees AfDB 233-269-872- 483 r.mwasambili@afdb.org Dec. 5, 2014 MEETING WITH LWSC OFFICER IN CHARGE, KAKATA CITY 46 Nyenekon B. Snoh Barcon Officer in Charge LWSC 231-886516- 127 nyenekonbsbarcon@gmail.com Dec. 5, 2014 WB 47 Joseph Collins World Bank/WSP Personnel World Bank 231-886-634- 518 jcollins1@worldbank.org Dec. 8, 2014 LEC 48 J.Ivan Sims IT Manager LEC 231-886-510- 558 isims@lecliberia.com Dec. 9, 2014 Page 58 No Name Position Organization Phone # Email: Date 49 Fobay F. Dorbor Human Resources Manager LEC 231-886-541- 042 fobaydorbor@yahoo.com Dec. 9, 2014 50 Joseph T. Mayah Deputy Chief Executive Officer LEC 231-886-511- 941 jmayah@lecliberia.com Dec. 9, 2014 EPA 51 Johansen T. Voker Coordinator, Synagetic Project EPA 231-886-520- 042 vokerj@yahoo.com Dec. 10, 2014 52 Jonathan W. Davies Coordinator, Bio diversity Program EPA 231-886-523- 544 jwdavies.epalib@gmail.com Dec. 10, 2014 PRESNTATION AT USAID 53 Ben Zinner Health System Sub-Team Leader USAID 23177- 6777109 Dec. 10, 2014 54 Luis Velazquez Engineering Officer, Econ. Growth Office USAID 231- 776777000 Lvelazquez@usaid.gov Dec. 10, 2014 55 Barbara Dickerson Deputy Mission Director USAID Dec. 10, 2014 56 Courtney Babcock M&E / Program Officer USAID Dec. 10, 2014 57 David Wounuah Civil Engineer, EG team USAID Dec. 10, 2014 58 John Mark Winfield Mission Director USAID Dec. 10, 2014 59 Tara Milani Health Team Leader USAID Dec. 10, 2014 60 Sophie Parwon Deputy Health Team Leader USAID Dec. 10, 2014 61 Marc Douglas Program Officer USAID Dec. 10, 2014 62 Mervyn Ferroe Supervisory Program Officer USAID Dec. 10, 2014 63 Edgar Thornton Program Officer USAID Dec. 10, 2014 Page 59 No Name Position Organization Phone # Email: Date 64 Gerald Smith Supervisory Contracting Officer USAID Dec. 10, 2014 MEETING WITH GAC DIRECTOR 65 Dec. 10, 2014 66 Gabriel Flaboe, Sr. UWSSP Manager LWSC 231-886-513- 239 lwscproject@gmail.com Dec. 11, 2014 67 Frankie N. Cassell DMD/Technical Services LWSC 231-886-453- 554 Fnatcass54@gmail.com Dec. 11, 2014 68 Ben Zinner Health System Sub-Team Leader USAID 231-777-465- 893 bzinner@usaid.gov Dec. 11, 2014 69 M. Hue Coleman LWSC/LMWP Liason LWSC 231-886-514- 316 hnecoleman@yahoo.com Dec. 11, 2014 70 Churz Sherman Administration Manager LWSC 231-880-789- 846 zsherm21@gmail.com Dec. 11, 2014 71 Charles B. Allen Jr. Managing Director LWSC 231-886-510- 375 chabural@yahoo.com Dec. 11, 2014 MEETING WITH AfDB/UWSSP TEAM 72 Gabriel Flaboe, Sr. UWSSP Manager LWSC 231-886-513- 239 lwscproject@gmail.com Dec. 11, 2014 PRESENTATION AT TETRA TECH 73 Alioune Fall Chief of Party Tetra Tech, USAID/LMWP 231- 777332712 afall@tetratech-lmwp.com Dec. 12, 2014 74 Matt R. Harder, P.E Construction Project Manager Tetra Tech, USAID/LMWP 231- 770373658 mharder@tetratech-lmwp.com Dec. 12, 2014 75 Safaa Fakorede Utility Operations & Institutional Specialist Tetra Tech, USAID/LMWP 231-770-214- 932 sfakorede@tetratech-lmwp.com Dec. 12, 2014 Page 60 Appendix 4. Technical Topics Including Study Methodology The review of the background information (the desk study), and the knowledge gained from and observations noted during key informant interviews in the field in Liberia, prompted the need for a framework within which to not only identify the strengths and weaknesses of the LMWP project, but also employ an analytical tool to evaluate the overall project’s performance, and facilitate risk and consequences based decision making associated with the recommended capital improvement plans vis-à￾vis project’s sustainability and viability. The evaluation team explored a range of industry analytical tools that can best capture the characteristics of the risks and potential consequences/impacts of recommended design and operation and maintenance options. It narrowed down the options to: Strength, Weaknesses, Opportunities and Threats (SWOT) Analysis, a Logical Framework Matrix (LogFrame), and Risk Assessment Framework Matrix (RFA). SWOT Analysis SWOT analysis is in general a way of taking an assessment of a current position and identifying internal and external factors that establish that position and can lead to change. It is an exercise in decision￾making that relies on specific data to generate a strategic plan. SWOT analysis is most often applied in a variety of business contexts. It can also be used to analyze unrelated situations where project strategic decision-making is important. SWOT analysis is normally performed during the initial project start-up phase so that the elements of the analysis can form the basis of the project plan, but it can also be used later in the project if the project is running into difficulties with scheduling, deliverables or budget and needs to be brought back on track. For example, if a certain key activity in the project requires improvements and revisions, a SWOT Analysis can be used to assess the risks and the opportunities of improvements in order to help with the resource planning. SWOT Analysis is one of those tools that is in the toolbox of every planner and evaluator who is involved in project design, implementation, and evaluation. SWOT Analysis is the most renowned tool for audit and analysis of the overall strategic position of a business and its environment. It views all positive and negative factors inside and outside the decision framework that affect the course of action. Critics of SWOT however, claim that it hampers performance while others claim that the findings of a SWOT Analysis are difficult to integrate into overall strategic decisions. It is a strong tool, but it involves a great subjective element. It is best when used as a guide, and not as a prescription. Logical Framework Matrix (LogFrame) The Logical Framework Approach (LFA) is a highly effective strategic planning and project management methodology with wide applications. It is particularly valuable for water management and sanitation projects, especially because water ― the resource base ― has diverse and competing uses. It comprises an integrated package of tools for analyzing and solving planning problems and for designing and managing their solutions (the approach). The product of this analytical approach is the LogFrame (the matrix), which summarizes what the project intends to do and how, what the key assumptions are, and how outputs and outcomes will be monitored and evaluated. According to the World Bank, “the Logical Framework has the power to communicate the essential elements of a complex project clearly and succinctly throughout the project cycle. It is used to develop the overall design of a project, to improve the project implementation monitoring and to strengthen periodic project evaluation.” It provides a set of interlocking concepts which are used as part of an iterative process to aid structured and systematic analysis of a project or program idea. LFA is best started early in activity design, and should be thought as an “aid to thinking.” The key disadvantages of the LogFrame are:  Focusing too much on problems rather than opportunities and vision Page 61  Organizations may promote a blueprint, rigid or inflexible approach, making the LogFrame a straitjacket to creativity and innovation  Limited attention to problems of uncertainty where a learning or adaptive approach to project design and management is required  The strong focus on results can miss the opportunity to define and improve processes. Risk Assessment Framework (RAF) Matrix The primary purpose of risk assessment is to evaluate the consequences if an investment decision or action fails. Decision makers typically want to avoid new investments or projects when the threats are catastrophic or when they outweigh values for money. The goal is to analyze the risks and rewards of a decision using data. Risk assessment reduces the need for intuitions and instincts. A risk assessment framework (RAF) is a strategy for prioritizing and sharing information about the risks to infrastructure investment. A good RAF organizes and presents information in a way that both technical and non￾technical personnel can understand. The evaluation team in consultation with USAID/Liberia discussed and selected from among three options, the Risk Assessment Framework to further facilitate decision making associated with evaluating potential risks and impacts of recommended CIP improvements including analysis of the O&M cost recovery, capacity building and institutional strengthening. Page 62 Appendix 5. Schedule of Activities Activity Week 1 2 3 4 5 6 7 8 9 10 11 11/2‐ 11/8 11/6‐ 11/15 11/16‐ 11/22 11/23‐ 11/29 11/30‐ 12/6 12/7‐ 12/13 12/14‐ 12/20 12/21‐ 12/27 12/28‐ 1/3 1/4‐ 1/10 1/11‐ 1/17 Preparation for Field Work Telephone in‐brief with USAID/Liberia, evaluation team and IDG home office staff Finalize evaluation questions Conduct document review Schedule key informant interviews Develop and submit draft detailed study plan outlining evaluation approach, methodology, and draft schedule of field activities Submit final Study Plan Develop and submit draft outline of the Evaluation Report outlining key issues/emerging themes to be addressed in each section Develop and submit draft questionnaires and other tools and instruments to be used in key informant interviews and for collecting data Submit final questionnaires and data collection tools/instruments Briefing in IDG home office on evaluation and medical precautions Field Work In‐depth Analysis, Follow‐up and Synthesis Meet with USAID/Liberia and evaluation team to finalize schedule of field activities Page 63 Activity Week 1 2 3 4 5 6 7 8 9 10 11 11/2‐ 11/8 11/6‐ 11/15 11/16‐ 11/22 11/23‐ 11/29 11/30‐ 12/6 12/7‐ 12/13 12/14‐ 12/20 12/21‐ 12/27 12/28‐ 1/3 1/4‐ 1/10 1/11‐ 1/17 Conduct key informant interviews with USAID staff and project implementers, partners, and beneficiaries to confirm project results, identify constraints to the project being more successful in achieving desired results, and stakeholder views on ways in which assistance could be more effective in achieving expected/desired results Finalize the outline of the Evaluation Report Conduct site visits to Robertsport and Kakata Meet with USAID/Liberia to provide briefing and PowerPoint presentation on findings and recommendations Produce Final Report Write Evaluation Report Submit Draft Evaluation Report Incorporate comments and feedback from USAID/Liberia Submit Final Evaluation Report Present Final Evaluation Report to USAID/Liberia Page 64 Appendix 6. Questionnaires and other evaluation tools Infrastructure Planning & Design Evaluation Task 1: Situational Analysis (Contractor to answer questions about methods, etc.)  Methodology used - Adequacy and consistency and the sources of field data: review of literature from prior assessments at a national level including situation analysis documents (i.e., lessons learned); national and USAID Liberia water and sanitation country strategy reports, if any; stakeholder consultation; meetings, country interviews, and indicator monitoring if any; questionnaires; discrepancies; constraints to and limitations of the study. Task 2: Plan for Staged Water Supply & Sanitation Infrastructure Improvement (Contractor to answer questions on planning, selection criteria, infrastructure planning, etc.)  Key planning criteria in selecting the short-long term water & sanitation infrastructure improvements schemes and options: Cost Recovery and motivations for sustainability approach (economic, financial, environmental, social and political challenges, demands, sources of supply options; service requirements and standards; operational performance objectives; social objectives; environmental objectives; financial objectives; workplace health and safety objectives.  Project Selection Criteria: Methodologies employed (e.g., multiple-criterion or decision making approach, and decision matrix); scaling, rating or ranking of alternatives;  Evaluation of various decision factors: integration/synergy (benefits from combinations of two or more options); Input/output characteristics; functionality; technical soundness; constructability; local capacity for construction, operation & maintenance considerations  Communication and Information Dissemination and approval process: Stakeholder involvement (GOL, LWSC, Ministry of Health, Planning authorities; USAID; consensus versus agreement￾based decisions.  The significance of the failure to deliver the service need; the extent of the planning issues to be considered; complexity of the issues to be addressed; risks associated with meeting the service need; costs of project; and potential benefits arising from greater investment in planning.  Managing competing priorities Task 3: Institutional Frameworks for Water Provision Institutional frameworks for sustainable sanitation and water management are a crucial pre-requisites for the implementation of any sanitation and water management intervention measure, as they are the basis for their success and sustainability. Basically, the framework is determined by national, provincial and local policies and legislation (‘legal framework’) that constitute the “rules of the game”. It also includes the institutions and organizations (‘institutional framework’) with forums and mechanisms, information and capacity building, created to establish these “rules of the game” and to facilitate and exercise stakeholder participation (GWP 2008).  Assessment of current national and local capacities including private sector and public-private partnership: Problems, issues, challenges and opportunities;  Public funding for water & sanitation improvements: construction & operation & maintenance;  Opportunity and prospect for Promotion, Extension and Community Mobilization;  Private Sector Involvement: “Water Means Business”;  Current Policy for the relative roles of the government, community and the consumers in Liberia. Literature and experience suggest that the community's role has seldom goes beyond the digging of trenches;  Strategy: Need versus demand; Page 65  Community Management and Self-Empowerment ;  Communication and Information Dissemination and approval process: Stakeholder involvement (LoG, LWSC, Ministry of Health, Planning authorities; USAID; consensus versus agreement￾based decisions. Task 4: Capital Works  Was Value Engineering/Value Analysis employed to facilitate and support optimum design vis-à￾vis cost and best value for the design options?  Was knowledge of construction materials, constructability, labor market, capacity of local construction market; inflation, design-build versus design-bid-build were considered in the design process?  Did LMWP assess the host country procurement mechanism for construction?  Was a construction contractor ever involved in the design process?  How have risks and uncertainties in construction and construction management supervision been incorporated in the design process?  Any design and specification standards?  Any environmental threshold issues? Task 6: Capacity Building  Needs assessment – What was done to develop a baseline?  Gap Analysis – What was done to develop the gaps and how did the gaps influence capacity balding plans and training?  Training options – Were there various training options to address capacity gaps?  On-the-Job training capacity building – Any experience with on-the-job-training capacity building?  Target capacity – Is there a target capacity building plan?  Gender considerations – Any gender considerations for capacity building?  Logistic challenges – What are the logistics challenges to capacity building? Task 7: Coordination with other USAID programs  Cost effective synergies with the 5-year WASH Program; and the 5-year Eco-system service Program: Any Lessons learned from coordination with other programs? Page 66 Contractor Questionnaire a) What do you see as the key accomplishments to date? b) What are the key challenges you have faced to date and continue to face? c) What are the key weaknesses of the project? d) What has caused the delays in the project from your perspective? e) Do you see any corrective actions to bring the project back on track and is that even possible? f) What are the lessons learned to date? g) How was the project staffed at the onset? Kindly provide us an organizational chart of the field office. h) Did you have the capacity and resources to manage all components of the tasks at hand? i) Have you had difficulty recruiting technical and other staff for project? If Yes, how would you address that if the options years were to be granted? If No, how have you been able to achieve that granted the capacity challenges in the country? j) What has been your experience with GOL entities as credible and viable partners – think of LWSC primarily? k) Any thoughts about the project design, i.e. anything you would have done differently? l) Have there been any scope issues? m) Is the improved water supply access achievable in the three designated cities? Page 67 USAID Questionnaire 1. What was the rationale for the LEAP-II project ? o Agency initiative? o Demand driven by and request from GoL? o A complement to and in coordination with other development partners like the WB or African Development Bank? o Project development process? o Degrees of collaboration with GoL at the initial stages of project development? o Willingness of GoL for participation in, and or facilitation of the project development and implementation process? o Any signed agreements with GOL? o How well were institutional and capacity constraints understood in the early stages of the project development? Were these treated as risks, or challenges with opportunities for an enabling environment for progress? o What is USAID Liberia’ s project management structure for managing this project? 2. LMWP Performance o What were the key challenges as understood by USAID in the timely and efficient delivery of the project by the Contractor? o Is USAID satisfied with the project’s progress to date? What appears to be areas of success and areas for improvements and concerns? o What has caused the project to fall almost 2 years behind schedule? o How satisfied is USAID with Contractor’s staffing in the field? o What are USAID’s concerns, if any, with the Contractor’s project management approach and methodology? 3. Collaboration and role of GOL (i.e., LWSC) o Any documentation and or an agreement on the role that GOL should have played in facilitating and supporting the project? o How has the relationship been with GoL since the project was initiated? o How have GOL project concerns been disseminated to USAD management? Has there been a direct channel? Is there a single point of contact (champion) on the GOL side? What role has the Contractor played in the process? o Has GOL expressed satisfaction or concerns about project’s progress? If so what are they? o In the opinion of USAID, has the GOL played a constructive role in the project process? o What conditions should have prevailed, if applicable for GOL, to have better facilitated and supported the project? o Overall, how satisfied is USAID with the course of the project? o What lessons has USAID learned to date? o Would USAID have done anything different in designing this project granted what it knows now? o What would USAID do different should the project proceed to the next stages? a) Was the project consistent with the national water supply and sanitation sector policies, strategies and priorities of the USAID and GOL and relevant to the needs of the target population? b) To what extent was the project design rigorous, adequate and relevant to achieve the project objectives? Page 68 d) Is the proposed infrastructure design well-suited to achieve project objectives in a cost-effective and sustainable manner, and how well is the project positioned (in terms of management/staffing structure) to support the construction phase? Factors affecting sustainability include but are not limited to: management and financing model; non-revenue water; informal water suppliers; hygiene behavior; and effectiveness of partnership? e) To what extent will the improved water systems likely provide substantial health benefits and increased business opportunities; gender outcomes and environmental protection in the three cities through the contractor’s collaboration with other USAID programs? f) When constructed and transferred to GOL, what is the likelihood that the project facilities delivered will be operated and maintained over the life of the project and yield expected benefits to the target population stipulated in the project objectives? Page 69 GOL Entities Questionnaire a) What do you see as the key accomplishments to date? b) What are the key challenges to this project? c) What are the key weaknesses of the project? d) What has caused the delays in the project from your perspective? e) Do you see any corrective actions to bring the project back on track and is that even possible? f) What are the lessons learned to date? g) Has GOL been a good partner to this project? Does GoL have the capacity to meet its obligation for this project? h) What are GOL’s concerns, if any? i) What was LWSC’s expectation for this project? j) Was there a perception that it was going to be similar to the AfdB’s UWSSP project? k) How does this project fit into LWSC’s strategic plan? l) Has LWSC been a viable and credible partner to the Contarctor on this project? m) What has been your experience with GOL entities as credible and viable partners – think of LWSC primarily? n) Any thoughts about the project design, i.e. anything you would have done differently? o) Have there been any scope issues? p) Is the improved water supply access achievable in the three designated cities? q) What are the key lessons learned in LEC’s reform project? r) What were the challenges to LEC’s reform project? s) How can those lessons be used to help the water sector in its own reform plan? t) What are your thoughts on Public-Private Partnerships (PPPs)? Page 70 Other Donors Questionnaire (AfDB, World Bank, etc.) a. Please provide a brief take on this project? b. What do you see as the key accomplishments to date? c. What are the key challenges of this project? d. What are the key weaknesses in the project? e. What has caused the delays in the project from your perspective? f. Do you see any corrective actions to bring the project back on track and is that even possible? g. Are you willing to provide more resources as needed to see the project through? h. What has been your experience with GOL entities as credible and viable partners in project execution? i. Any thoughts about the project design, i.e. anything you would have done differently? Page 71 Appendix 7. Evaluation Team Composition Team Leader/Utilities Expert: Azad Mohammadi The Team Leader/Utilities Expert provided overall leadership, management, and technical direction of the evaluation and served as the primary point of contact with USAID and other counterparts. He managed the work of the Utilities Expert and Administrative/Logistic Expert. He led evaluation work on water and sanitation service delivery/utility management and infrastructure system operations and maintenance. He was responsible for the preparation and submission of all deliverables working closely with the Utilities Expert. Utilities Expert: Sam Koduah The Utilities Expert led evaluation work on business/organizational performance improvement and capacity-building. He worked closely with the Team Leader/Utilities Expert to produce project deliverables. Administrative/Logistics Expert: Sheku Daboh The Administrative/Logistics Expert scheduled meetings and interviews, arranged transportation, and provided other administrative support. COP/Contract Manager: Elly Preotle The COP was responsible for monitoring the overall management and technical performance of the evaluation and assured that the evaluation team receives the resources and other support it requires to function effectively. Senior Economist: David Snelbecker The Senior Economist contributed to and ensured the quality of the study plan, questionnaires/instruments, and the evaluation report Appendix 8. Debriefing Presentation Page 72 Page 73 Page 74 Page 75 Page 76 Page 77 Page 78 Page 79 Page 80 Page 81 Page 82 Page 83 Page 84 Page 85 Page 86 Page 87 Page 88 Page 89