NORTHERN UGANDA WATER SUPPLY SERVICES PROJECT (NUWATER) END-OF-PROJECT EVALUATION FINAL REPORT 21st November, 2011 This document was produced for review by the United States Agency for International Development (USAID/Uganda). It was prepared by The Mitchell Group, Inc. for the Uganda Monitoring and Evaluation Management Services, herein referred to as UMEMS, Contract Number 617-C-00-08-00012-00 EVALUATION REPORT OF THE NORTHERN UGANDA WATER SUPPLY SERVICES PROJECT (NUWATER) Prepared for United States Agency for International Development USAID/UGANDA Team Members: Barney P. Popkin Lucrezia Koestler Patrick Jangeyanga The Mitchell Group, Inc. 1816 11th Street N.W. Washington, D.C. 20001 DUNS#: 17528-5121 TIN#52-1467119 Telephone: (202) 745-1919 Facsimile: (202) 234-1697 Internet: www.the-mitchellgroup.com 2 NUWATER End of Project Evaluation Acknowledgement The Evaluation Team is very grateful for the generous cooperation received from water consumers of Kitgum and Pader who participated in focus group meetings. The Team is grateful as well to the many agencies and stakeholders who were interviewed and assisted the evaluation process. DISCLAIMER The author‘s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 3 NUWATER End of Project Evaluation Table of Contents ABBREVIATIONS AND ACRONYMS........................................................... 6 EXECUTIVE SUMMARY............................................................................. 7 1 - INTRODUCTION, BACKGROUND AND PURPOSE...............................11 1.1 Main Project Activities ................................................................................................ 11 1.2 Purpose of the Evaluation .......................................................................................... 12 1.3 Evaluation Team......................................................................................................... 13 1.4 The Country Context .................................................................................................... 13 2 - STUDY APPROACH AND METHODS...................................................16 2.1 Limitations of the Methodology ..................................................................................... 17 3 – FINDINGS ...........................................................................................18 3.1 Improved Access to Water in Kitgum and Pader....................................................... 18 3.1.1 NUWATER Performance Indicator Results ............................................................................. 18 3.1.2 Stakeholders‘ Assessments........................................................................................................20 3.1.3 Customers‘ Assessments ........................................................................................................... 21 3.1.4 Conclusions ............................................................................................................................... 22 3.2. How realistic and appropriate was the design of the project? ................................... 22 3.2.1 Contracting a Private Operator .................................................................................................. 24 3.2.2 Incentive-Based Contracts ......................................................................................................... 25 3.2.3 Conclusions Regarding Project Design and Assumptions ........................................................ 27 3.3 Were the appropriate stakeholders involved in the program and to what extent did the program promote better coordination and collaboration amongst them?.......................... 27 3.3.1 Gender Issues ............................................................................................................................ 29 3.3.2 Capacity Building of Stakeholders ............................................................................................ 29 3.3.3 Conclusions on Participation, Collaboration and Coordination ................................................ 32 3.4 The effectiveness of the program management and its effect on the program outcomes ............................................................................................................................................... 32 3.4.1 USAID Management ................................................................................................................. 37 3.4.2 WASH Consults Management .................................................................................................. 38 4 NUWATER End of Project Evaluation 4 - CONCLUSIONS....................................................................................40 4.1 Management of NUWATER Project .............................................................................. 40 4.2 Contracts and Water Management Models ................................................................... 42 4.3 The Problems with Infrastructure .................................................................................. 43 4.5 Capacity Building ........................................................................................................... 43 4.6 Sustainability ................................................................................................................... 43 5 - LESSONS LEARNED............................................................................44 5.1 Project Design ................................................................................................................. 44 5.2 Technical Capacity ......................................................................................................... 44 5.3 Implementation ............................................................................................................... 44 5.4 Involvement of Stakeholders and Beneficiaries ............................................................ 44 5.5 Incentive-Based Contracts .............................................................................................. 45 5.6 Sustainability ................................................................................................................... 45 6 – RECOMMENDATIONS........................................................................46 7 - APPENDIXES.......................................................................................48 APPENDIX A: NUWATER Evaluation Scope of Work ................................................................... 48 APPENDIX B: NUWATER Project Context..................................................................................... 55 APPENDIX C: Study Approach and Evaluation Methodology ......................................................... 60 APPENDIX D: List of Stakeholders Interviewed .............................................................................. 64 APPENDIX E: Questionnaire for Stakeholders ................................................................................. 68 APPENDIX F: Focus Group Questionnaire for Water Consumers ................................................... 74 APPENDIX G1: Focus Group Water-Customer Meeting Attendance Sheet– Pader ......................... 75 APPENDIX G2: Focus Group Water-Customer Meeting Attendance Sheet – Kitgum .................... 77 APPENDIX H: Record of Focus Group Meeting with Water Customers in Kitgum ........................ 80 APPENDIX I: Record of Focus Group Meeting with Water Customers in Pader ............................. 84 APPENDIX J: Focus Group Voting and Comments – Kitgum and Pader ......................................... 87 APPENDIX K: Report on Technical Site Visits to Kitgum and Pader Water Systems ..................... 89 APPENDIX L: Evaluation Findings Summary .................................................................................. 91 APPENDIX M: NUWATER Project Final PMP Indicator Matrix .................................................. 117 APPENDIX N: WASH Consults Ltd Production Data .................................................................... 122 APPENDIX O: Photographic Evidence from Site Visits Conducted............................................... 123 APPENDIX P: Bibliography ............................................................................................................ 126 5 NUWATER End of Project Evaluation List of Tables Table 1: Summary of Evaluation Findings .................................................................................... 8 Table 2: Answers to Key Evaluation Questions ........................................................................... 40 6 NUWATER End of Project Evaluation Abbreviations and Acronyms ADS USAID‘s Automated Directive System APWO Association of Private Water Operators of Uganda ARD ARD, Inc., a Burlington, Vermont-based development consulting firm COP Chief of Party COTR USAID Contractor Officer‘s Technical Representative DLG District Local Government DWD Uganda Ministry of Water and Environment‘s Directorate of Water Development ESU NWSC‘s External Services Unit GOU Government of Uganda GPOBA Global Partnership on Output Based Aid IDP Internally Displaced Person KI Key Informant LRA Lord Resistance Army M&E Monitoring and Evaluation MOU Memorandum of Understanding NGO Non-governmental organization NSSF National Social Security Fund NUTI USAID/Uganda‘s Northern Uganda Transition Initiative NUWATER Northern Uganda Water Supply Services Project NWSC Uganda National Water and Sewerage Corporation O&M Operations and maintenance OBA Output Based Aid PMP Performance Monitoring Plan or Preventive Maintenance Plan PO Private Operator (local water management firms) RGC Rural Growth Center SOW Scope of Work SWAP Sector Wide Approach TC Town Council TMG The Mitchell Group TOT Training of Trainers UGX Ugandan Shillings UMEMS Uganda Monitoring and Evaluation Management Services Project URA Uganda Revenue Authority USAID United States Agency for International Development USD United States dollar UWIN www.waterintegritynetwork.net VAT Value added tax WIN World Bank‘s Water Integrity Network WIN-A Water Integrity Network - Africa WIN-S Water Integrity Network Secretariat WSP World Bank‘s Water and Sanitation Program 7 NUWATER End of Project Evaluation Executive Summary The Northern Uganda Water Supply Services Project (NUWATER) was a three year (10th June 2008 – 6th June 2011), $3 million USAID/Uganda-funded program created to increase access to water in Kitgum and Pader towns by improving the urban water supply systems using private service providers for operation and maintenance of the water systems. Both of these towns are located in Northern Uganda and only recently emerged from a long conflict. Specifically, NUWATER was responsible for improving the quality of services such that more customers would have access to clean water and that the incentive-based system would have moved significantly toward financial sustainability, if not full financial sustainability. Further, it was expected that local institutions would be capable of sustaining the operating contract model beyond the life of NUWATER. NUWATER was implemented by ARD, Inc., a Burlington, Vermont-based subsidiary of the publicly traded Tetra Tech, Inc. Uganda National Water and Sewerage Corporation (NWSC) was hired as a local expert firm on a consultancy basis to provide technical expertise in water systems and utilities management. The NUWATER project evaluation was undertaken between May and June 2011 to answer four key evaluation questions pertaining to project achievements and impact and to obtain lessons learned from the project to inform future USAID water services projects in Uganda: 1) To what extent did the project meet its goal of improving access to water in Kitgum and Pader? 2) How realistic and appropriate was the design of the project? 3) Were the appropriate stakeholders involved in the program and to what extent did the program promote better coordination and collaboration amongst them? 4) Analyze the effectiveness of the program management and its effect on the program outcomes. The Evaluation Team answered these questions through compiling and reviewing 86 documents, in addition to emails and cell phone text messages; conducting over 46 one-on-one, semi￾structured and confidential interviews with major stakeholders; facilitating two focus group meetings for water customers as beneficiaries in Kitgum and Pader; undertaking field site visits of the water utility infrastructure in both towns; and holding internal discussions, reviews and critiques and peer reviews to prepare this report. In addition, the team participated in USAID briefing and debriefing meetings and incorporated many of the comments in its Inception Report and this Evaluation Report Generally, the Evaluation Team has found that NUWATER fell significantly short of all of its performance targets. A straightforward analysis of planned versus actual accomplishments as measured in the NUWATER Performance Monitoring Plan (PMP) shows that the project did not achieve the majority of its performance targets as set out in the project PMP (See Appendix M for the project‘s final PMP Indicator Matrix). The table below summarizes our conclusions. 8 NUWATER End of Project Evaluation Table 1: Summary of Evaluation Findings Key Evaluation Questions Evaluation Outcome 1. To what extent did the project meet its goal of improving access to water in Kitgum and Pader? There was an improvement in access to water in Kitgum but supply was intermittent and unreliable. Water access in Pader increased from a zero baseline. 2. How realistic and appropriate was the design of the project? The project design was neither realistic nor appropriate and should have been modified early in the project life to achieve more practical and long-lasting results. Key assumptions made about Pader purchasing power and the existence of a working system in Kitgum did not prove to be correct. 3. Were the appropriate stakeholders involved in the program and to what extent did the program promote better coordination and collaboration amongst them? The appropriate agency stakeholders were involved with the program but the program had poor communication and collaboration between local authorities and the project staff and timely communication between USAID Washington and ARD. Project beneficiaries were not adequately involved, mobilized or sensitized. 4. Analyze the effectiveness of the program management and its effect on the program outcomes. ARD, USAID, WASH, and Pader water management lacked clear time lines. NUWATER did not report most of its PMP data or set targets for key indicators. The project team lacked decisive problem solving which hindered program outcomes. Many of the unmet expectations of the NUWATER project were the result of USAID‘s inexperience in the Ugandan water sector, lack of appropriately qualified supervisory staff and the imposition of a USAID/Washington-generated project design based on incorrect data and assumptions such as the length of time needed to observe significant and sustained locally led private sector management. It fell short of achieving its contracted tasks in incentive-based private-water operator services, capital infrastructure works, and human resources capacity building. The Evaluation Team concludes that more could have been done within the given timeframe and budget had more timely problem-solving taken place and more project resources were devoted to increasing the water kiosk network. The incentive contracts were sound in principle, but USAID and NUWATER learned late in the process that they were not appropriate in Pader, which did not have any house connections and no interested operator. Also, NUWATER failed to build capacity in Kitgum to understand and manage the contracts in future for sustainability. NUWATER lost time waiting for MOUs to operate, time that could have been used for planning, 9 NUWATER End of Project Evaluation needs assessments and baseline surveys. NUWATER and the private operator in Kitgum blamed the failure to achieve objectives mainly on the production capacity. According to the Evaluation Team, this should have been the first priority of NUWATER, and then some of the project‘s budgeted resources could have been reprogrammed toward improving the systems‘ production capacity. Those relatively minor infrastructure improvements that the project undertook in Pader occurred only late into the project in December 2010. The management team had limited time between then and the end of the project in early June 2011 to follow-up, track partner performance and build local private sector and government capacity. The Evaluation Team is therefore concerned that the management model chosen in Pader may prove to have been inappropriate, and it could create further management problems for the community in the future. The NUWATER Team‘s community capacity building effort was mostly limited to on-the-job training for the private operator and the possibility for Town Council officials and Water Board members to sit in on the monthly M&E meetings. In Pader, the water management contractor did not receive any of this follow-up or training. This leaves the water managers, private operator, Town Councils, Districts and Water Boards with challenges that they have to address on their own when NUWATER ended. In hindsight the project should have clearly outlined an in-service training plan with local authorities on each new topic prior to rolling out new approaches. Only one workshop on the billing software was carried out.Also, the centralized management approach of NUWATER gave the local stakeholders limited responsibility or financial control during the three years of project operation, instead of gradually transferring knowledge and responsibility in order to strengthen local management to assume full operational management of new water systems following the life of the project. In addition to a lack of capacity, the main challenges to sustainability are financial. Neither Pader nor Kitgum towns can sustain operations and maintenance of their water systems with the current price structure. None of the towns have funds for emergency repair or spare part stores. In Pader, the situation is especially difficult and water managers have not been paid for several months. In Kitgum, some money was saved from return to project sales proceeds. As a result of the NUWATER subsidies, limited funds are available for operational costs, but these funds will soon run out and there is no clear plan in place to finance water operations. The most important lessons learned from the NUWATER project experience are summarized below:  Project design needs to be based on feasible timeframes, reliable data and reflect conditions on the ground, including household disposable income  Project design needs to factor in timelines for local administrative and regulatory steps such as ―gazetting‖, and institutional factors such as the need for MOUs  Water experts are needed with the requisite technical backgrounds on the contract team,USAID and within the local government and private sector to ensure a rapid start-up and consistent management and informed follow-up with partners  USAID needs to supervise more closely and decisively and to insist upon adequate monitoring and reporting 10 NUWATER End of Project Evaluation  Capacity building is crucial to sustain water operations beyond the life of project by the local private sector operators.  Beneficiaries including water consumers need to be involved in and committed to water supply projects  The time period for incentive contracts for the private sector need to be longer, less complex and provide ―real‖ motivation  The management approach by the contractor needs to be conducive to encouraging local partners to assume responsibility USAID should in the future consider working more closely with MWE, NWSC and/or community-based organizations to develop any future project design In general, community￾based organizations/NGOs can implement small-scale water projects faster than water/utilities generation agencies. The Evaluation Team recommends that USAID hire a local water engineering manager to design and monitor future water projects. USAID should set up an independent technical working group comprised of Ugandan water utilities to carry out timely peer review and quality assurance/ control processes, similar to that required in the United States for water utility projects. USAID should build into future water projects study tours to successful water projects in neighboring countries to observe best practices in water supply management. Going forward in Kitgum and Pader, a comprehensive, professional assessment and design for water services should be initiated which should include a market analysis of consumer demand and willingness and ability to pay for services at various price points. The Evaluation Team recommends that community mobilization be integrated into the infrastructure development and that technical and managerial support be extended to the private operator or manager of the water system at least in the transition period. For the current water system in Pader a private operator and a more informal, locally-trained operator would be better because the volume of business is not large enough to be profitable. Depending on the size, potential market and local government subsidy of the new infrastructure development in Pader, a private operator might be outsourced to carry out the services for the community. Kitgum will largely benefit from the new infrastructure development. However, the current operator, WASH Consults, reported to the Evaluation Team that it is not interested in continuing operations in Kitgum. With a new and better infrastructure in place, it may be possible to attract more bidders to participate in the procurement process. However it is crucial to work closely and collaboratively with MWE play its important role in overseeing this process and to gain its support for the project. Also, even if a new private operator is contracted, the Water Board, Town Council, District Water Office and probably the private operator will still need capacity building and follow-up in order to properly oversee and monitor the contract using NUWATER evaluation and monitoring tools and software. In addition, due to the high levels of poverty in both towns, it is important to include many more public water sources in the any future design and not to solely rely on household residential water connections. Most people are more likely to buy water from the kiosks, especially in Pader, because they cannot afford to be connected. The Evaluation Team understands that these lessons have been incorporated into the new design. 11 NUWATER End of Project Evaluation 1 - Introduction, Background and Purpose Implemented by ARD, Inc., in association with the External Services Unit (ESU) of the National Water and Sewerage Corporation (NWSC) of Uganda, the Northern Uganda Water Supply Services (NUWATER) activity was a three-year, $3 million USAID/ Uganda-funded program created to assist Kitgum and Pader towns to re-establish their water supply systems using incentive-based management contracts with private service providers. The project began in June 2008 and ended in early June 2011. Incentive-based contracts, modeled on the well established and successful model already used by the Directorate of Water Development (DWD) and the National Water and Sewage Corporation (NWSC), were used in order to provide financial incentives to the private operator if certain targets involving non-revenue water, production levels and collection rates were achieved for three consecutive months. The main stakeholders were the local Town Councils (TC) of Kitgum and Pader, the District Local Government (DLG), which is responsible for the procurement procedures of the Town Councils and technical support, and the private operator, in this case only WASH Consult in Kitgum because a private operator in Pader was never contracted. Other key stakeholders at the national level were the Urban Department in the DWD under the Ministry of Water and Environment (MWE) and the External Services Unit (ESU) of NWSC, the para-statal that runs the water systems in the major towns in Uganda. Originally the NUWATER Project was to address the water supply in three towns in Northern Uganda, an area that only a short while before had emerged from a long confict: Kitgum, Pader and Aloi. An early NWSC assessment in 2008 determined Aloi was not a good candidate for an incentive-based contract and it was dropped from the project‘s workplan. Specifically, NUWATER was responsible for improving the quality of services so that more customers would have access to clean water and that the incentive-based system would have moved significantly toward financial sustainability, if not full financial sustainability. Further, it was expected that local institutions would be capable of sustaining the operating contract model beyond the life of NUWATER (USAID Program Description excerpted from the USAID-ARD Contract, 2007:6). NUWATER proposed 12 indicators in its Performance Management Plan (PMP) in terms of which to measure the results sought. The PMP and associated data are summarized in Appendix M and NUWATER‘s performance in respect of collecting and reporting is discussed in greater detail in Section 3.4. 1.1 Main Project Activities Detailed accounts of the project‘s activities can be found in the ARD quarterly and annual reports. The activities below were all confirmed by the Evaluation Team during the site visits and through stakeholder interviews. In Kitgum, infrastructure improvements included repairing some pumps, procuring a stand-by generator, procuring and installing bulk meters and procuring household water meters for the subsidized connections. In addition, an office was rehabilitated and a number of repairs and improvements were carried out on the water system. Two hundred and seventy new households were connected in the project period. In addition, a MOU was 12 NUWATER End of Project Evaluation signed with MWE and later with Kitgum Town Council, a private operator was contracted through the District Procurement Office and the private operator‘s performance was followed up monthly for 15 months by a M&E team from NWSC. The Water Board and selected Town Council officials participated in monthly meetings, and were given training in the use of billing software that was specially designed for the project. Towards the end of the project, NUWATER procured the design of the new infrastructure project1 , and supervised the drilling of three new production wells. In Pader, a pump was replaced and a new generator purchased for the main production well. Five water kiosks and the pipeline were rehabilitated and put back in service. The project facilitated the ―gazetting‖ process that allowed the TC to become a Water Authority with powers to contract a private operator, and tried twice to procure a private operator through the District, with no success. Therefore, an interim management solution was chosen whereby TC officials assumed the management of the system as an extra job, managing the kiosk attendants and the pump attendant and receiving a small bonus payment. Toward the end of the project, NUWATER supported the consultants that were to design the new infrastructure project, and supervised the drilling of two new production wells. 1.2 Purpose of the Evaluation USAID/Uganda contracted The Mitchell Group (TMG) to provide an evaluation of the NUWATER Project. The four key evaluation questions outlined in the SOW are listed below (See Appendix A for the full SOW): 1) To what extent did the project meet its goal of improving access to water in Kitgum and Pader? 2) How realistic and appropriate was the design of the project? 3) Were the appropriate stakeholders involved in the program and to what extent did the program promote better coordination and collaboration amongst them? 4) Analyze the effectiveness of the program management and its effect on the program outcomes. The evaluation was carried out between May 23 and June 22, 2011 with field work in Kitgum and Pader conducted between May 29 and June 8, 2011. 1 A number of challenges in achieving project goals were attributed to the poor state of the water infrastructure in Kitgum and Pader. Only limited funds were available for infrastructure improvement under the NUWATER project, so USAID decided to approve a full rehabilitation of the water system in Kitgum and the construction of a new water system in Pader after the end of NUWATER. 13 NUWATER End of Project Evaluation 1.3 Evaluation Team The Evaluation Team consisted of the following experts:  Barney P. Popkin, Team Leader with 30 years international experience in assessing water projects for USAID  Lucrezia Koestler, Water Engineer, resident in Uganda for six years and  Patrick Jangeyanga, Research Assistant. The Evaluation Team was supported by UMEMS Chief of Party Patricia V. Rainey, Operations Manager Nestore Jalobo, and Monitoring and Evaluation Specialist Stanley Golooba Lukenge. The final report is a result of a substantial modification due to considerable comments from USAID and ARD to the initial draft report submitted by the Evaluation Team. The final report was re-worked by Patricia Rainey, with additional inputs from Stanley Gooloba of UMEMS Project, Lucrezia Koestler and Patrick Jangeyanga from the Evaluation Team and Lans Kumalah from TMG Inc. 1.4 The Country Context Appendix B supplies a fuller description of Uganda‘s water delivery system and conditions in post-conflict Northern Uganda. The water sector in Uganda is managed by the MWE through the DWD. A Sector-Wide Approach has been adopted whereby donors put their money into a basket fund which is distributed to local governments through the central government and spent on projects following agreed priorities and objectives. However, some donors and NGOs prefer to carry out interventions directly. In the rural water sector, money is channelled from the central government to the District Water Offices that are responsible for implementation. Rural water supplies are normally boreholes or shallow wells with hand pumps, protected springs or small gravity flow schemes or piped water schemes. In the urban sector, operation and maintenance was handed over to the private sector in 2001. The national water parastatal, NWSC, manages the 23 biggest towns (GOU, 2010). Around 80 other small towns hire their own private operators. In order for a Town Council (TC) to contract a private water operator, the Ministry needs to ―gazette‖ the town into a Water Authority. This gives the TC (or the sub-county) the authority to contract a private sector operator and take over ownership of the assets of the water system on behalf of the central government. The Ministry signs a performance agreement with the Water Authority. Thereafter, the Water Authority signs a management contract with the private operator. The contract is supervised by a Water Board, which consists of the Town Clerk (secretary), the Mayor and three board members selected in the community to represent the water users, one of whom one is elected as the Chairman. Being gazetted also has financial benefits, because the TC will benefit from a Conditional Grant to help with operations and maintenance. 14 NUWATER End of Project Evaluation According to the Ministry, about 70% of the private operators are ―breaking even‖ in their operations. Many un-gazetted Rural Growth Centers (RGC) and small towns have simple piped water systems, often only with a few dozen connections and a few public stand posts that require some limited form of management. Rather than find and contract private operators, these RGCs normally train local people to manage the system on a profitable basis, under supervision of a Water Committee or a Water Board. The small companies are often not registered and operate informally. They only have a few employees. However, it is more realistic that local people manage the water systems in small towns where the consumer base is not enough to create sufficient revenue for a private sector company to make profits. The piped water schemes have their umbrella organizations in Western, Mid-Western and Eastern Uganda (GOU, 2010) that provide technical and management support. However many water systems are too expensive to operate to be profitable. ―Breaking even‖ means the operators are covering their operating expenses such as fuel, power, chemicals, salaries and small repairs. However the responsibility for large repairs, replacements and emergency breakdowns still theoretically lies with the Ministry. The margins are low, and some operators operate several towns in the same area in order to benefit from economies of scale. Under a new Output-Based-Aid (OBA) program funded by GPOBA (Global Partnership on Output-Based Aid), subsidies are provided to 11 small towns (Azuba et al, 2010). Slowly, the Ministry is starting to realize that operating water systems is not very attractive for a private company, especially because consumers are not willing to pay and assets are often in a poor condition and need frequent repair. The responsibility for repairs is often not clear. In addition, the private operator has to apply for financial support from the central government, something that takes time and is unreliable. The incentive-based contract designed by the USAID project is similar to the OBA program. However, it is unlikely that a system can be ―break even‖ or become profitable after only three years. As a result, the OBA program recently agreed to extend its contracts to seven years and some even to ten years. The main characteristics of the NUWATER operating environment included:  The relatively high levels of poverty and the ongoing process of reconstruction of Northern Uganda after a war that lasted almost two decades. In contrast to the rest of the country, Northen Uganda did not register any major improvements in household well-being during the 1990s. For example, the national poverty head count index declined from 56% in 1992 to 34% in 1999 while for Northern Uganda it only declined from 72% to 64% over the same period (UBOS: 2003). Even when other dimsensions of welfare are considered, the region still performs poorly compared to the rest of the country. For instance, the infant mortality infant mortality rate for northern Uganda is about 20% higher than the national average (UBOS: 2001)2 2 Poverty under Conflict: The Case for Northern Uganda, by Sarah Sewannyana etal, 2007 15 NUWATER End of Project Evaluation "People have been living in survival mode and have not snapped out of it. If they see something they grab it" NUWATER Stakeholder  The specific roles of the towns Kitgum and Pader doubling as Internally Displaced Persons (IDP) camps during the war;  The culture of passivity and dependency on aid agencies in the communities due to the long time spent in the camps. A study of 112 residents of four camps for internally displaced Ugandans done in 2009, respondents described four main response strategies: assistance from others (including neighbours, relatives, agencies and community organisations); work and income generation; personal characteristics (e.g. determination, ability to work hard); and social support. The study observed that the fact that assistance from others was identified as the primary strategy perhaps also reflects the level of dependency that has been created amongst IDPs in Northern Uganda3  The feeling of neglect and mistrust of people in Northern Uganda toward the central government. In its June 2009 publication ―Contributing to a Peace Economy in Northern Uganda, A Guide to Investors, International Alert pointed out that the prolonged duration of the LRA conflict and the widening gap between north and south have been compounded by instances of government forces failing to adequately protect civilians during the war, creating a deeply-felt mistrust of the government in the region. Representation of Northerners among top public sector bodies is very low compared to that of other regions, leading some Northern politicians to complain of systemic discrimination and some even calling for the secession of a ―Nile Republic‖. This feeling of marginalisation is compounded by perceptions that certain politically well-connected individuals and military personnel actively exploited the conditions of the conflict in order to amass personal wealth.  The high levels of corruption in public procurement both at local and central government level, as well as the high risk for political interference, especially in the process of contracting private operators; Public procurement is one of the sectors most affected by corruption in Uganda. According to the 2007 African Peer Review Mechanism Report, Uganda loses USD 258.6 million annually through corruption and procurement. The report further estimates that if the country could eliminate corruption in public procurement, it would save USD 15.2 million a year. In the assessment of the country‘s Auditor General, procurement accounts for 70% of public spending, of which an estimated 20% is lost via corruption.  Urban/rural distinctions in Uganda and the fact that Pader is a small and new urban centre; and  The general rules of operations of water utilities in small towns in Uganda and which stakeholders are involved. 3 Coping with displacement: problems and responses in camps for the internally displaced in Kitgum, by Rebecca Horn, 2009 16 NUWATER End of Project Evaluation 2 - Study Approach and Methods The Evaluation Team used several approaches and methods for this evaluation, following the approach and methods defined in the TMG technical proposal to USAID as validated in the evaluation team‘s inception report to USAID. These methods included:  Compiling and reviewing over 86 documents and several e-mail and mobile phone text￾message correspondence to obtain background information about the NUWATER project throughout the course of the evaluation but particularly in the start-up phase.  Reviewing the Kitgum baseline survey commissioned by NUWATER in late 2009, one year into project implementation as a source of objective and quantifiable data.  Conducting and reviewing over 46 one-on-one, semi-structured, confidential interviews with major stakeholders to answer the four key evaluation questions and associated sub￾questions and to obtain insights on project history, recommendations and lessons learned.  Facilitating two focus group meetings with water customers in Kitgum and Pader.  Conducting technical site visits to the Kitgum and Pader water systems.  Holding internal discussions, reviews and critiques, and peer reviews to prepare this report.  Participating in USAID briefing and debriefing meetings and incorporating many of its comments and suggestions into the Inception Report and this Evaluation Report. The Evaluation Team administered a questionnaire designed to elicit key stakeholder reaction to the NUWATER project. The questions were designed to obtain a simple yes/no response from stakeholders to the four main evaluation questions during the fieldwork process. To obtain more detailed information, 29 sub-questions were formulated (see Appendix C). These formed the content of the questionnaires developed and administered to various categories of stakeholders (Appendices E and F). The Evaluation Team comprising three members moved together to conduct all the interviews in the two towns over a period of 11 days. Comprehensive notes were taken by each team member and these were compared each evening to ensure that nothing was missed. After each day of fieldwork, the Evaluation Team filled in the data summary sheet that was circulated amongst the team members to ensure complete documentation of the responses (Appendix L). The stakeholder questionnaires took on average 90 minutes to administer. Stakeholders were identified in advance on the advice of the USAID Mission and NUWATER COP. Focus Groups Discussions were held with approximately 80 people in Kitgum and 40 people in Pader (Appendices G1 and G2). The sessions were guided by a set of semi-structured questions (Appendix F). These meetings were held to obtain data on Key Evaluation Question No. 1,4 as well as to determine what water sources were in use before and after NUWATER, water-systems management, benefits and problems, and other relevant issues (see Appendices H, I and J for a summary of the findings). 4 ―To what extent did the project meet its overall goal of improving access to water in Kitgum and Pader?‖ 17 NUWATER End of Project Evaluation The Evaluation Team consistently confirmed information from consumers and stakeholders by carrying out field inspections of infrastructure, where possible. Three site visits over three days were conducted and photographic evidence assembled (Appendices K and O). Other claims were confirmed by examining monthly and quarterly reports in detail. Claims or information obtained from only one source were not considered valid for a finding in this study, but might have been used as a quote or to illustrate a point. All positive and negative responses were recorded and have been factored into this report. Further details on the methodology appear in Appendix C. 2.1 Limitations of the Methodology A major lacuna in the methodology was the absence of hard data in the NUWATER performance reports. Only two quarterly and two annual reports were submitted to USAID by NUWATER Project. NWSC/ESU carried out monthly evaluations of the Private Operator in terms of the agreement between itself and NUWATER and summarized these in monthly and quarterly performance reports of the PO. These reports provided quantitative data on water production, non-revenue water, billing, collections and connections. However, NUWATER only submitted performance indicator data to the USAID/Uganda database on two occasions – in FY2011 Quarter 2 for one indicator and at the end of FY2011. Baselines were inconsistently recorded in their PMPs and largely not entered into the database. This hampered the ability of the Evaluation Team to get a handle on what changes occurred in the course of the project and to verify claims made. The Evaluation Team was therefore forced to rely on interview data with stakeholders and consumers. The qualitative component of the evaluation and related fieldwork were limited primarily by time and resource constraints. It was only possible to spend a short amount of time in the field as the NUWATER project closed on 6th June 2011 but the Evaluation Team was only mobilized on 23rd May and travelled to the field on 29th May, 2011. Additionally, local stakeholders‘ and beneficiaries‘ comprehension of the research questions was sometimes limited by language barriers and cultural factors such as the challenge of understanding abstract concepts such as ―assumptions‖ and ―design‖. The Evaluation Team remedied this by taking sufficient time to explain the questions before requesting a response. The Evaluation Team is aware of the difficulty in drawing statistical conclusions from a small sample that was unevenly distributed over the different stakeholders. However, the aim of providing statistical information in the form of percentages of respondents was to quickly grasp the general trend in the responses rather than present a statistically significant result. 18 NUWATER End of Project Evaluation 3 – Findings The main objective of this report is to answer the four questions below: 1. To what extent did the project meet its goal of improving access to water in Kitgum and Pader? 2. How realistic and appropriate was the design of the project? 3. Were the appropriate stakeholders involved in the program and to what extent did the program promote better coordination and collaboration amongst them? 4. Analyze the effectiveness of the program management and its effect on the program outcomes. An overview table summarizing the information from the Evaluation Team‘s interviews with stakeholders appears as Appendix L, organized by category of respondent - USAID, NUWATER, MWE and NWSC, Local Government, Water Board, Private Operator and Water Users - while the outcomes of the focus group discussions appear as Appendices H, I and J. 3.1 Improved Access to Water in Kitgum and Pader In Uganda, ―access‖ in urban settings is defined by the percentage of the population within 0.2km from an improved water source. International definitions, such as the definition of access in the Millennium Development Goals (MDGs) include aspects of both quantity and quality. In Uganda, these aspects are measured by separate indicators (MWE, 2010). Since both these aspects are crucial for the satisfaction of consumers, the Evaluation Team decided to use a combination of these indicators and thus interpreted ―access‖ to mean a continuous and sufficient water supply of adequate quality within a convenient distance from the household as NUWATER itself did not appear to define these key concepts in their planning documentation. In addition to reviewing documentation, the Evaluation Team interviewed stakeholders on the issue of improved access referring to the seven sub-questions related to this key evaluation question (Appendix C). Water consumers were also questioned in focus group settings about improved access (Appendix F). After a review of the NUWATER performance indicators and data, the findings from the stakeholder interviews and focus group discussions are summarized before each evaluation question is discussed in more detail. 3.1.1 NUWATER Performance Indicator Results The Evaluation Team approached this question by first examining the performance data submitted by NUWATER in the Mission‘s database that is managed by the UMEMS Project5 . The numbering system for indicators used in this report follows that of the USAID/Uganda database. The relevant indicator is Indicator No. 4.1 in the NUWATER PMP - Number of people 5 The Uganda Monitoring and Evaluation Management Services Project (UMEMS) provides technical support to the Uganda Mission and all IPs on performance management 19 NUWATER End of Project Evaluation in targeted areas with access to improved drinking water as a result of USG assistance (ARD 2009). As Appendix M shows, NUWATER fell short of its targets in this regard. NUWATER reported that 12,740 people were given access to clean water over the life of the project, 66% short of the life-of-project target of 37,739 entered into the database. Note that this is a different target from that set out in the 2009 version of their PMP (ARD, March 2009: no page number but 62,080 persons) and their 2010 PMP (ARD July 2010: no page number but 28,539 persons). Varying baseline numbers were also reported by NUWATER – 13,410 in the 2009 PMP versus 6,623 in the 2010 version. It thus appears that only by July 2010 did NUWATER get a handle on the actual situation on the ground in Kitgum and Pader with respect to the services provided from their water systems and realized that the Pader system was non-existent. Unfortunately, Indicator No. 4.1 was also never selected by the Mission for a Data Quality Assessment and thus the data was not verified by UMEMS. In the light of the deficiencies of the data for the next indicator, the Evaluation Team doubts that the data for this key indicator reported by NUWATER is valid. Data on a related indicator - number of water points constructed or rehabilitated (Indicator No. 1.1.2.1), showed the NUWATER Project fell short of its life-of-project target here as well. Only 270 such points were constructed or rehabilitated, representing only 20% of the targeted 1,300. In 2011 only 28 new connections were added (USAID database record, May 2011). These targets had also been revised downwards from the 2009 PMP targets as had the baseline value for Kitgum from 841 to 721 (the last number being confirmed by the Kitgum baseline survey) although the USAID database shows zero. UMEMS‘ Data Quality Assessment of this indicator conducted in August 2010 concluded that the data (242 in FY2010) were not valid as NUWATER included as connections items such as water storage tanks, generators and solar systems. UMEMS recommended that NUWATER revise its data collection tool to ensure that only water points were counted per the requirements of this indicator‘s particular definition and to change the data in the database but this was not done. However, from the monthly M&E reports prepared by NWSC/ESU the Evaluation Team was able to reconstruct the number of connections made between August 2009 and December 2010. There were 89 connections made during this period. If we add the base number of connections (35) reported by NWSC/ESU and the 28 connections installed in 2011 (generously assuming that they were all connections) the total is only 135 or 10% of the targeted 1,300. Although included in their Performance Management Plan, NUWATER did not report any data on their indicator for improved water quality (Indicator No. 1.1.4) although targets were set so it is also not possible to know if water quality improved in the life of the project. However, from the Kitgum baseline survey commissioned by NUWATER one year into the project, we can be reasonably certain that at that time the quality of water in Kitgum at least was good after a previously inconsistent record (ARD Baseline Survey Report, 2010). Although targets were set for increased water availability (Indicator 1.1.6), no data were ever reported on this important performance indicator, nor any baseline measure taken and documented. It was thus difficult for the Evaluation Team to determine from other documentary sources if water availability had improved since the commencement of the NUWATER Project. The Evaluation Team reviewed water production statistics supplied by WASH Consult for 2010 but concluded that the data were likely invalid and unreliable (See Appendix N). A review of the 20 NUWATER End of Project Evaluation NWSC/ESU monthly reports showed that by November 2010, the average number of hours pumping per station increased from a base of 12 hours in 2009 to 15 in November 2010 against a target of 18 hours (thus 81% of the target was met) but it was not a consistent rise. 3.1.2 Stakeholders’ Assessments Since each stakeholder respondent was asked the same questions, it is possible to summarize the findings for each of the questions asked in the semi-structured interviews carried out with stakeholders such as GOU, MWE, NWSC, NUWATER, WASH, USAID, UMEMS, TCs, DLGs and Water Boards and local water operators. In all, 46 stakeholders were interviewed, including relevant NUWATER and USAID staff (Kampala 15; Lira 2; Kitgum 14; Pader 15). A large majority of respondents believe access to water improved (74%). In Kitgum, 86% of stakeholders interviewed (14 in total) reported that water access had improved as a result of the NUWATER Project. In Pader, overall 73% responded likewise. All four officials interviewed at Kitgum Town Council affirmed that access to water has improved despite only 270 (a high estimate in the light of the previous discussion of indicator results) of the 1,200 connections committed to in the MOU with the Town Council having been installed (Kitgum TC and ARD 2008). They also reported that connections were subsidized at 59,000 Uganda Shillings and this was affordable for households. Two out of the 3 leaders of Kitgum district believed that access has not improved and attributed this to low production. One official noted: ―The water supply situation in Kitgum is greatly undermining the access to water in the town with some connections standing as dry connections”. At the time of this evaluation water rationing was being exercised whereby the town is divided into seven service areas and each rationed water once a week. In Pader, 50% (two out of four) of district officials interviewed said NUWATER did not improve access to water but only one-third of the TC officials agreed. According to the Pader TC, currently only 25% of the town‘s population has access to water kiosks because the network is very limited (in 2007, the PA Consulting Feasibility Study documented that 10,000 urban residents in Pader were served by just 3 boreholes). With a current population of about 12,000 people, this means that around 3,000 people have increased access. They also noted, however, that the water kiosks are concentrated within the middle of the town thus not serving the majority of the town‘s population that is located in the outskirts. The TC officials stressed that water quality had improved significantly although there was no hard data to back up this claim as no water quality monitoring appears to have been undertaken. All four WASH Consult (PO) staff interviewed disagreed that access had improved. The PO reported that some new connections had not received water for the six prior months and that the "We are grateful because when NUWATER came the water system was almost dead. NUWATER brought it back to life" Kitgum Stakeholder "NUWATER came as a savior when the system was at the verge of breakdown" Kitgum Stakeholder 21 NUWATER End of Project Evaluation system still operates with the five pumping stations it had before NUWATER. The Private Operator in Kitgum also pointed out that some pumping stations were non-functional. In addition, it noted that low pressure in the system has greatly affected the water system in Kitgum. The same is reported with the Pader system. However, all five kiosk attendants who were interviewed in Pader agreed that NUWATER had improved access to water. 3.1.3 Customers’ Assessments The focus groups involved 42 beneficiaries in Pader and 81 in Kitgum. The detailed findings from the focus group meetings are captured in Appendices H and I. In Kitgum, 59% stated that access to water had improved while in Pader 76% said access to water had improved through the NUWATER project. About one-fifth of water users in Pader did not vote. The reason for the much higher ―yes‖ vote in Pader was likely because there was no previous functioning water system at all and that NUWATER improved the situation considerably. In Kitgum, people had the experience of a more continuous water supply before NUWATER‘s arrival and, according to the focus group participants in the first year that WASH operated the system the water flow was continuous. However, now, of the focus group respondents who are connected, 31% get water only once a month, 47% get water once every two weeks and only 13% get water once a week. A large number of participants in the focus group meetings were also residents of areas that were completely unserved by the water system, or that had pipelines but had received no water for years due to damaged pipes or other problems. During the beneficiaries‘ focus group meeting in Kitgum, the Evaluation Team learned that no one in the group of over 80 water customers received water on a daily basis. The only functioning Kitgum water kiosk was non-operational at the time of the focus group meeting as water was being rationed. In addition, the Team met several people in Kitgum hotels who said they did not have water in Kitgum for several weeks. This illustrates that even if access was somewhat improved, the situation at the end of the project was uneven and many challenges remain to meet consumer and commercial business needs. When NUWATER ended in June 2011, subsequent discussions held by the Evaluation Team with the private operator, customers and field inspections revealed there was extensive unaccounted-for-water from unrepaired distribution leaks and other losses in both towns (non-revenue water was at 59% in November 2010 in Kitgum when NWSC ceased its monitoring of the project) and continued water rationing in Kitgum. A review of the NWSC M&E reports for the period August 2009 to November 2010 substantiates the information provided by focus group participants. Service reliability varied between 9 and 15 hours pumping daily for that period against a target of 18 and from a base of 12 hours while the volume of water supplied varied from a low of 9,709 m3 to a high of 16,860 m3 from a base of 15,000 m3 and against a target of 31,583 m3 . In Kitgum, an important concern raised during the focus group meetings was that there were essentially no functional water kiosks since the only one in operation was rationed. The implication of this is that people who cannot afford house connections but still want to and can afford to buy water are excluded from the piped water system and have to wait in line at the boreholes. In general, water users tend to place a high value on convenience with time being important to most water users. If they can afford it, a household appeared to prefer a direct 22 NUWATER End of Project Evaluation household water connection to walking more than 30 meters to a kiosk or waiting in line at a free well or borehole. This became clear from discussions with water users at the kiosks, kiosk attendants and the focus groups and is known to be especially true for mothers with large families, those impacted by war, disabled people, childhood households, and HIV/AIDS households. Given the degree of poverty in this region of Northern Uganda, there will always be a large percentage of people who cannot afford to pay for connected or kiosk water. The feasibility study conducted by PA Consulting (USAID, June 2007: 115) also pointed to the low level of willingness to pay and used very low values for monthly contributions from consumers in their costings. Such people may be more likely to use other water sources such as rainwater and river water, shallow wells and free boreholes with hand pumps, even if they must walk to them or wait in line. These consumers who now use other water sources will only use direct household connections if they are highly subsidized. The NUWATER Project did not establish how much it would cost to connect these underserved potential customers. Another issue cited in the focus group discussions about improved access was the low income level of consumers in the two towns. This was felt by the participants to result in people defaulting on their monthly water bills. This was substantiated also in the NWSC M&E monitoring reports in which it is documented that total arrears incurred during the time WASH Consult operated the Kitgum water system increased from 28,549UGX to 61,548UGX over the period August 2009 and November 2010. The existence of free alternative sources of water also undermined the project‘s ability to collect payments for overdue bills. 3.1.4 Conclusions From the above, the Evaluation Team concludes that NUWATER Project did increase access to water to a limited extent but far below expectations and agreed targets. Due to the low and unreliable water production levels and the practice of rationing it is difficult to calculate the number of people with increased access because an unreliable and intermittent supply does not qualify as improved supply. 3.2. How realistic and appropriate was the design of the project? This section commences with a summary of the views of respondents about the design assumptions of the NUWATER Project and then analyzes two key components of the design – private operators and incentive-based contracts. Telephonic interviews with USAID/Washington staff who had been involved in the design of the NUWATER Project underscored the fact that the project was based on an existing NWSC model that had been successfully implemented in other parts of Uganda. The success of the model was widely acknowledged and the Evaluation Team was referred to several published studies about the model. PA Consulting was hired in 2007 to conduct a feasibility study to examine existing capacity to deliver services, potential interest from bidders, demand and investment needs. They 23 NUWATER End of Project Evaluation worked closely with NWSC whom they sub-contracted to assist them. Their report characterises the water supply systems in Northern Uganda rather optimistically (USAID, June 2007:12-15, 30-31). For example Kitgum‘s system is characterized as follows: System is fairly well developed, has a lot of suppressed demand with the potential for growth with minimum refurbishment required”. The PA Consulting Team‘s assessment of Pader was short on detail but more accurately described it as requiring a lot of investment. The report also clearly lays out the legal and institutional framework. A major assumption made in their proposed design that has subsequently proved incorrect was that after three years the town system would be able to finance their own O&M costs (USAID, June 2007:21). While the report lists the private operators that had been involved in water management in small towns no attempt appears to have been made to determine their level of interest in running water supply systems in the short-listed towns, let alone their capacity to do so (USAID, June 2007: 105). The capacity of the local governments to supervise such a contract was determined to be low; hence the consultants‘ recommendation that a contractor be hired to perform this function (USAID, June 2007: 19). The consultants stated that the model would work in Northern Uganda but did not provide any evidence of how they arrived at this conclusion, referencing only the positive attitudes expressed by civic leaders to the involvement of the private sector in water provision (USAID, June 2007: 14, 15). It is possible that they relied too heavily on the input of NWSC whose area of expertise is with large towns. However, PA Consulting may possibly have had reservations about the workability of themodel as they recommended that USAID establish three pilot projects to test the three models they proposed for each of a town, RGC and village situation. USAID‘s subsequent contract with NUWATER did not reflect the experimental nature of the endeavor however. NUWATER Task 1 related to the design, award and management of water operating contracts. This task included the extension of safe water delivery services to 15,073 additional new customers in up to three towns with different water delivery services including: Kitgum with a centralized water system and an established town; Pader, a new town growing out of a Regional Growth Center (RGC) and without a centralized water supply; and Aloi, a former Internally Displaced Persons (IDP) camp and RGC with a water supply system. Following an initial field visit by NWSC in July 2008, it was decided that Aloi should be dropped from the project due to its small size and the lack of a functioning water system (ARD, October 2008). Since the project did not have a strong hardware component, the town was judged not to be viable for incentive￾based private operator contracts. In the light of subsequent experience in Pader, this proved to be a wise decision. From the Evaluation Team‘s interviews with knowledgeable stakeholders about the realism of the design and two of the key assumptions, namely a functioning water system and level of private sector interest in bidding for the contract, we conclude that the design was not realistic for either Kitgum or Pader which were emerging from a long conflict. Even though 64% of respondents in Kitgum felt that the NUWATER project design assumptions were realistic, this was qualified by respondents with statements about the dire state of the pre-existing system which was on the verge of collapse – functioning but at very low production levels. Those closest to the situation on the ground in Kitgum, namely five of the seven Kitgum officials who were interviewed and who felt that the project‘s design assumptions were true qualified their 24 NUWATER End of Project Evaluation assessment in this way. Respondents further from the situation such as those from MWE/DWD (50%) and NWSC (100%) thought the project‘s assumptions were correct for Kitgum where there was already an existing water system, and POs were willing to run it, given the location of the town. In retrospect, NUWATER staff noted that the project‘s assumption of an existing water system, even in the case of Kitgum, did not consider all relevant factors such as the production and water storage capacity of the existing system. Finally, the Private Operator in Kitgum also agreed that while the assumptions were true, the poor level of infrastructure on the ground was not taken into account with the Kitgum system needing a high level of infrastructure investment if production was to be boosted. For Pader however, only 20% of all respondents interviewed felt that the NUWATER project design was appropriate and realistic. According to Town Council officials, again the respondents closest to the ground, Pader had no functioning water system when the NUWATER Project started but three different systems were built during the time the town also served as an IDP camp. The design initially assumed that these three systems could be integrated into one but NUWATER reported this was technically impossible and not viable due to the low yield of the production boreholes. Only one system (called the ―DWD System‖ by local officials) was therefore rehabilitated by NUWATER and this system has no private connections. NUWATER rehabilitated five water kiosks, bought a new generator and a pump and helped carry out repairs on the pipeline. Only four water kiosks were operating at the time of the evaluation. For respondents from MWE/DWDS and NWSC, their negative responses to the design questions were largely explained by them as relating to Pader having no history of ever paying for water. Most of the Pader TC officials interviewed (5 out of 8) concurred with the views of MWE/DWD and MWSC in this regard but added that there was also no functioning system in place at the start of the NUWATER Project. 3.2.1 Contracting a Private Operator Task 1 required NUWATER to award operating contracts that reflect the type of community and available infrastructure. The project design assumed several private water utility management/operational firms would be interested, qualified and immediately available to bid on the contract for operation and maintenance (O&M) of the three towns‘ water systems. In fact, very few were interested, and most of them were disqualified because of their tax situation. None of the members of the Association of Private Water Operators in Uganda (APWO) bid on Kitgum or Pader. The Evaluation Team obtained inconsistent data on the number of interested operators in Kitgum and Pader but it appears that in Kitgum only four or five firms were interested and in Pader only one or two firms expressed an interest in the project. About half of the firms were disqualified due to missing or forged documents. It is unclear why only a few contractors were interested in managing Kitgum but possible reasons included anticipated low financial margins as well as limited national capacity, poor condition of the infrastructure and political issues. MWE blamed the poor reputation of the Kitgum Town Council regarding its relationships with private operators whereas the Kitgum Town Council blames the former operator (Trandint Ltd) for spreading false rumors after its contract was terminated. However, a survey carried out by APWO showed that operators thought 25 NUWATER End of Project Evaluation the towns were too far from their area of operation and that they did not want to challenge Trandint which also holds the chairmanship of APWO (APWO, April 2009). Kitgum Water Board members and NUWATER officials blamed the lack on enthusiasm on the interests of MWE officials in the private operator companies. The Evaluation Team was not able to determine which if any of these aforementioned factors accounted for the lack of interest. For Pader, local government officials said that they could understand why no private operator was interested in a system with four kiosks and no private connections, a conclusion that the Evaluation Team finds reasonable. NUWATER also did not take into account the fact that Pader was not gazetted and thus not even authorized by the MWE to contract an operator. The authorized list of gazetted towns is readily available from the MWE but a failure to obtain it resulted in ARD spending several months going through the gazetting process in Pader. Task 3, Planning and Reporting, per Statement of Work (C.4), was thus poorly implemented by ARD, which failed to adjust its work schedule to the constraints it encountered and to obtain a USAID contract modification to reflect these constraints. USAID staff interviewed also concluded that institutional risks were not properly assessed in the case of Pader and the former COTR reported having warned that the capacity of private operators in a small town recovering from a war situation would be very limited, and that water infrastructure was lacking. USAID and NUWATER modified the design after the situation on the ground was better known, but no formal contract modification was made. The changes consisted of approving extra money for infrastructure development (later halted) and a change in management system in Pader after no private operator showed interest. 3.2.2 Incentive-Based Contracts The incentive-based contracts were intended to encourage the private operators to manage the project well and several tools were used to incentivize the private operator. The first was to give a subsidy based on collections so as to motivate the contractor to collect as many payments as possible. The second was to provide for a ―bonus payment‖ if the contractor was able to achieve certain targets such as non-revenue water levels, collection rates and production levels over three consecutive months. In addition, the contract provided subsidies for new connections in order to motivate both the operator and the end-users to increase the number of connections and thereby the potential revenue and financial sustainability of the intervention. The NUWATER Project also helped the private operator pay utilities such as power bills and security costs. In principle, the incentive-based contract is a good initiative and should have attracted many private operators, especially because of the subsidy on collections to help the operator covering operating costs. Unfortunately, complicating assessment of this component of the design of NUWATER, neither targets nor data were ever provided by NUWATER for one of the key performance indicators (Indicator No. 1.2) for this result – Level of performance bonuses earned and provided to operators. The same situation pertains to Indicator No. 1.1 – Percentage reduction in the ratio of subsidies to operation and maintenance costs and Indicator 1.1.2 – Increase in collection rate. A related indicator for which data and targets are available (Indicator No. 1.1.1.1) – number of 26 NUWATER End of Project Evaluation incentive-based contracts tendered and awarded for implementation – shows that NUWATER only achieved one out of a targeted four such contracts. However, this output level indicator is not very helpful in evaluating the effectiveness of this aspect of the project design. The more qualitative investigations of the Evaluation Team lead it to conclude that the incentive￾based contracts largely failed in Pader and Kitgum. There appeared to be three major reasons for this: 1. Kitgum Town Council officials admitted to the Evaluation Team that they had provided out-dated data on the production levels of the wells during the assessment undertaken in 2007. The data came from the Austrian development cooperation that rehabilitated the system in 2001 but production had already significantly decreased by the time NUWATER was designed. As the bonus payments were based on targets established from this incorrect baseline, the private operator never managed to even come close to the targets. 2. One of the main assumptions made in the design was that a large number of new household connections would be made through OBA support. For example, the manager of the private operator had calculated in his bid for the contract that the projected increase in connections would compensate for the reduction in subsidy over three years. However, NUWATER fell very short of its target for new connections (see previously analyzed performance indicator in this regard). Pader‘s Town Council officials noted that the contract was completely unrealistic in their situation because of the near complete absence of household connections at the start of the project. The procurement was agreed to because they hoped NUWATER would construct the connections. Two reasons were given for the slow progress in installing connections: NUWATER had to stop their work on connections because of the low production capacity and the delay in procuring household meters (Owot and Okaronon, April 2010). The meters were finally received on April 30, 2010 (eight months after the private operator starting operations) and then there was a further delay in the private operator in installing them. 3. Related to the issue of metering, the system in Kitgum had no functioning bulk meters making it difficult to calculate water production which was the base measure for the incentive system. This is mentioned in every monthly report made by the NWSC Monitoring Team from August 2009 to August 2010 (Owot and Okaronon, October 2009, Odonga and Owot, November/December 2009, Owot and Okaronon December 2009, January 2010, February 2010, March 2010, April 2010, May 2010, July 2010, August 2010a, August 2010b, October 2010). During this period, NUWATER delayed in procuring new bulk meters, which are readily available in stores in Kampala and which, once installed had to be exchanged after only few days in operation due to bad quality. An additional aspect of the incentive-based system related to collection rates. Here the private operator in Kitgum complained there were no clear procedures on how to handle the debts of the previous private operator (power and salaries) and the arrears in water bills. This led to confusion with different interpretations made by the NWSC ESU team, the Town Council, the Private Operator and NUWATER. For example, after classifying payments of water bills first as new bills and then excess money to cover arrears, this was then changed six months into the contract 27 NUWATER End of Project Evaluation so that payment of bills first covered arrears and only then new bills. This changed the incentive structure drastically for the private operator since the subsidy on arrears was only 50% whereas the subsidy on new bills was 95%. Considering the significant amounts of arrears left by the previous operator, this new solution was not perceived as fair to the private operator. This explanation might be behind the perception of USAID officials noted that even the 95% subsidy for collections did not seem to have enough effect on the private operator to maintain a high level of collection of bills, and that the efforts made by the private operator to increase collections were disappointing and inconsistent. The issue was reported upon in detail by the NWSC team in February 2010 (Owot and Okaronon, March 2010). Finally, MWE pointed out that the duration of the contract was too short. It stated that normally the contracts range between 7 to 10 years where there is no infrastructure and 2 to 5 years where there is some infrastructure. 3.2.3 Conclusions Regarding Project Design and Assumptions From the information above, the Evaluation Team concludes that NUWATER operated from an incomplete and inaccurate planning information base even for Kitgum and was thus not able to engage in realistic planning and target settings. NUWATER does not appear to have accurately or comprehensively assessed the viability of the existing water infrastructure upon start-up. The private operator and the Town Council officials said NUWATER should have conducted pumping tests at the start of the project in order to adjust the targets. ARD should have documented the status of water systems in the first weeks of project start-up and submitted a revised workplan in accordance with the realities in the towns. The project seems only to have got a handle on the true situation on the ground by July 2010. Although the NUWATER project had limited funds to improve infrastructure under its Task 2, Capital Investment Works, per Statement of Work (C.4), NUWATER was instructed by the USAID Mission wait until the USAID-MWE Memorandum of Understanding (MOU) had been signed and after selecting a private operator before assessing the town‘s water infrastructure needs. NUWATER mentioned in interviews with the Evaluation Team that capital investments had been halted by USAID in March 2010 and this crippled progress on increasing production capacity in the two towns. NUWATER also confirmed that it had asked USAID for more funds for infrastructure development once it realized the infrastructure was in a much poorer state than anticipated but that only a fraction of this was approved. Greater involvement of the relevant GOU authorities at the time that USAID designed the project would likely have resulted in a more realistic design. 3.3 Were the appropriate stakeholders involved in the program and to what extent did the program promote better coordination and collaboration amongst them? As it typical of water sector development projects, a wide range of stakeholders need to be coordinated and involved to ensure project success. Stakeholders range from those with in-depth technical knowledge of water engineering and utility requirements, such as NWSC, to the water consumer and their representatives who may have little knowledge. None of the stakeholders 28 NUWATER End of Project Evaluation interviewed felt that any important stakeholder had been omitted from involvement in NUWATER although most felt that the level of involvement, coordination and collaboration could have been vastly better than what they experienced. MWE/DWD, the central government department, felt that it was not adequately consulted in either the conceptualization or implementation of the NUWATER project. MWE appears to have been involved only at the stage of the initial negotiations for the MOU and in a limited manner through its Water and Sanitation Development Facility North that is based in Lira and which was involved in the design of the new infrastructure. MWE also supported the gazetting process of Pader. Most development partners in Uganda work closely with and through the MWE/DWD and support on-going Ministry work. USAID‘s way of doing business does not include the provision of budget support as outlined in the earlier section on project context. DWD staff interviewed suggested that USAID start participating in the coordinated planning process for the water sector along the same lines as other development partners. It was their view that if USAID had engaged more closely with MWE/DWD from the outset, then problems such as locating a qualified private operator would have been avoided because, in their experience, private operators tended to be more confident about bidding on projects where the central government plays an active role (the Evaluation Team feels that the lack of financial viability of the Kitgum and Pader projects were more likely the critical factors behind the lack of expressed interest). USAID expressed concerns that the NUWATER COP had not coordinated adequately with the engineers at DWD. However, qualifying somewhat these perceptions of MWE/DWD is the fact that the Evaluation Team also determined that MWE/DWD had a presence in Kitgum before the NUWATER project but left as soon as it learned that USAID project was going to the area! MWE/DWD also said that NUWATER failed to keep them informed about project progress to the extent that at a certain point they had to call for regional meeting to ascertain what activities were taking place on the ground. As an example they claimed that they had received no information about the upcoming closure of the NUWATER project despite this date being documented in the MOU that they signed. All Kitgum and Pader Town Council and District officials interviewed agreed that the appropriate stakeholders were identified for this type of project but that the cooperation between them was not satisfactory. Districts were not much involved in the execution of the NUWATER project except in the procurement process. In Pader especially, cooperation was sub-optimal. The District Water Officer claimed not to have ever met anyone from NUWATER and had no idea what the project was doing. Kitgum District officials said the NUWATER team should have had a greater presence on the ground in order to facilitate communication and learning: ―When everything is managed from Kampala, communication becomes difficult” it said. The Pader Town Council expressed the same opinion. The Water Board was constituted in Pader but after an initial meeting, it never sat again. It felt neglected and said it could have played an important role as an intermediary between customers and the interim management but it was never called upon. In Kitgum, the Water Board was more involved through the monthly and quarterly monitoring and evaluation exercises that were undertaken by the NWSC/M&E Unit. However, it expressed a need for capacity building to better understand the reports generated. In Kitgum, one Town Council official complained that the whole system was run by WASH Consults and 29 NUWATER End of Project Evaluation NUWATER alone and that the TC and the LDG were left out. However, during the NWSC ESU M&E Team‘s work that improved somewhat, the official said. In addition, the Evaluation Team was told that the relationship between NUWATER and the private operator was tense. One Town Council official in Kitgum said that shouting and arguing occurred during joint meetings to such an extent that some TC officials feared attending the NUWATER meetings. The Evaluation Team met with many complaints about the NUWATER COP‘s management style but acknowledges that these were difficult to verify. However, in the question of management style, perceptions may be considered valid data (see section 3.4). In the Evaluation Team‘s focus group discussions, consumers also expressed concern about a lack of communication from the NUWATER Project. Consumers wanted more information about the project, especially in frustrating situations such as when the subsidies were cut for new connections in Kitgum. Also, they had no knowledge of how meters are read and how bills are calculated, and why a service charge had to be paid even if no water had been received during the billing period. These situations, they said, have led to suspicion and resistance to pay water bills. 3.3.1 Gender Issues In accordance with USAID‘s recognition of the importance of gender issues in development, the contractor must, where possible, identify, disaggregate, and report on, all gender related inputs, outputs and outcomes. Any other relevant gender-related implications or opportunities in the project must also be addressed and reported. In a context where women are the primary collectors and users of water, attention to gender is generally considered paramount in the design and implementation of water supply projects. This was confirmed by NUWATER‘s own Kitgum baseline survey (ARD, November 2009: 25). However, the Evaluation Team did not find any documents addressing gender issues in the NUWATER project, either with respect to planning or reporting although NUWATERS‘s Technical Clarification Response (ARD, 15th May 2008:7) documents that they would include amongst their activities gender-sensitive performance targets and incentives for contractors, gender mainstreaming training for Water Boards, Town councils and District Councils and public outreach activities specific to gender issues. There were also no performance indicators relating to gender issues in the project‘s PMP although this was also promised in the Technical Clarifications Response through the provision of a gender consultant who was to survey gender issues and then to establish gender indicators and targets. 3.3.2 Capacity Building of Stakeholders A central component of the project design was the building of the capacity of project stakeholders, in particular the local authorities, to effectively manage and supervise the incentive-based contracts with the private operators. Indeed, two of the performance indicators in NUWATER‘s PMP relate to capacity building: Indicator No. 1.1.1 – Improvement in record keeping standards and Indicator No. 1.2.1 – Number of local government officials trained on contract management. In respect of the first indicator, targets were set but no data ever provided 30 NUWATER End of Project Evaluation while for the second indicator, NUWATER‘s data in the USAID database shows that NUWATER only achieved 26% of its target (10 out a targeted 38 officials trained). NUWATER‘s first Annual Report for the period June 2008 – September 2009 acknowledges that a great deal of capacity building support was required and documented that the project was working with NWSC/ESU and APWO to finalize a training program for the Private Operator and local councils that would encompass water utilities technical operations, contract management, commercial and financial management and asset maintenance and management (ARD, September 2009). The plan appears never to have been submitted to USAID/Uganda for review or approval. Furthermore, NUWATER had committed to providing the Kitgum TC and Pader Water Authority ―formal training‖ per the Kitgum-ARD and Pader-ARD MOUs of December 10, 2008. USAID also requested a detailed plan for capacity building when a no-cost extension was requested by ARD toward the end of the project lifespan but NUWATER did not provide this to the satisfaction of the USAID Mission. The COTR was concerned that the NUWATER Team had not invested adequately in capacity and did not have an adequately detailed capacity building plan. In an interview with the Evaluation Team, the NUWATER COP agreed that capacity building was one of the project‘s objectives but admitted that the only training offered to Town Council officials and Water Board members was in the form of sitting in on evaluation meetings and that the only training provided to the PO was through the M&E exercises. These were carried out by NWSC from August 2009 to November 2010, after which no plan was put in place to replace the services of the NWSC‘s M&E Unit. District officials in Kitgum said they wanted more capacity to be built in the District Water Office so that they could later give technical support to the town engineers. The only documentation of the M&E exercises is in the form of the monthly reports from NWSC. These included some recommendations for further capacity building, but they were mostly not acted upon according to respondents; nor could the Evaluation Team find any evidence that the trainings were delivered. The reports also did not contain a description of who was trained in what topic during these on-the-job trainings. Finally, the M&E exercises only took place in Kitgum and Pader thus never benefited from them. The Evaluation Team confirmed that the Private Operator in Kitgum benefited from the work of the NWSC M&E team and that a quarterly evaluation of the PO was carried out and documented. The Private Operator in Kitgum expressed satisfaction with the follow up provided by NWSC but still wanted more formal training on specific topics such as water utilities management and financial management. The only formal training workshop that took place and for which the Evaluation Team could find evidence was on the billing software in May 2011, the last month of the project‘s operation. The billing software users claim there are ongoing problems with implementing the software. The Evaluation Team further confirmed that the Town Clerk and on occasion the Mayor participated in the monthly M&E meetings held by NWSC between August 2009 and November 2010. Town engineers and technical staff did not report having received any training except for an introduction to the new billing software in May 2011, the month before the project closed out. Town Council officials said that more formal training in water utility management and M&E would have been welcome. In Pader, TC officials noted that the meetings organized by NUWATER mostly consisted of ―presentations‖ and were not exercises that promoted learning. The Water Board in Kitgum was happy to be involved in the NWSC-led M&E exercises but 31 NUWATER End of Project Evaluation never received any training on how to understand the reports or findings and how to take a more active role. The Water Board in Pader reported that it never received any training. NUWATER said it had asked NWSC to carry out capacity building interventions but that NWSC had asked for extra payment for this and therefore the request was dropped. NWSC staff reported that their role under the contract was to identify and highlight training needs. The Evaluation Team was not able to determine which of these pieces of information was valid. Moreover, according to the private operator in Kitgum, water managers and kiosk attendants in Pader and Water Board members in Kitgum, NUWATER did not work to build sustainability but rather undermined it with the approach it used. Instead of giving the local entities increasing responsibility NUWATER controlled everything tightly up until the end of the project. For example, NUWATER required approval of even a $12 (or 28,000 Uganda Shillings) replacement tap or a few hours of pipe-excavation labor to go through its Kampala Office once the Field Coordinator had physically verified the need. One official was worried that all the capacity would leave when NUWATER ended which would not be good for the sustainability of the project. In addition, as the site visits to Kitgum and Pader showed, NUWATER did not provide manufacturers‘ manuals, operations and maintenance manuals, checklists and guidelines to the water system operators. During the Evaluation Team‘s site visits to WASH in Kitgum and the Town Water Authority in Pader, it was clear that neither body evinced good knowledge about utility administration, management, engineering design, infrastructure, O&M, community mobilization and sensitization or customer service. Finally, the Evaluation Team reviewed NUWATER‘s work in respect of improving community understanding and acceptance of piped water supply systems. This had been highlighted as important in the feasibility study (USAID, June 2007: 115-116). This is especially true for Pader which experienced a high level of water diversion. Both in Kitgum and Pader, Town Council officials, customers and the private operator reported that in the past pipes had been damaged by people purposefully hammering nails in them and children vandalized solar panels which were not protected by the community. The Evaluation Team also found a newly drilled well already destroyed. In their Year 1 Annual Report, NUWATER reported that a draft Public Awareness Campaign strategy document had been prepared in February 2009 and would be implemented in Year 2 of the project. The year 2 Annual Report indicated that activities had been conducted from November 2009 through September 2010 except for June, July and August. These included door￾to-door sensitization, radio talk shows and holding of a Town Hall meeting. Training of the operator on door-to-door sensitization and limited in-house customer care training were also carried out. No numbers of people reached or dates upon which activities took place were recorded in the NUWATER Report. Several beneficiaries in the focus groups said they had no knowledge of the campaign and in Kitgum most beneficiaries said they had only heard about the water system once on the radio. In Pader, beneficiaries said the Public Outreach Specialist only “…we are worried about the new project because they are pegging, but people may not allow lines to pass through their land if no sensitization is done‖ Pader Stakeholder 32 NUWATER End of Project Evaluation “talked to a few people‖ and did nothing more. The depth and reach of the public awareness campaign cannot therefore be assessed. In short, there was no documented plan for capacity building activities building up to a hand over of key actions to the stakeholder organizations. There is a dearth of documented evidence of the capacity building interventions undertaken with evidence located for the billing software training and some of the community sensitization work only. With no formal capacity assessment undertaken at the beginning of the project, there is therefore also no documented evidence to show whether the level of knowledge gained by the various parties increased as a result. 3.3.3 Conclusions on Participation, Collaboration and Coordination The Evaluation Team concludes that while the appropriate stakeholders were all nominally included in the implementation of the project, NUWATER could have done a much better job of coordinating with them and building their capacity to manage the project once it closed out. Gender considerations and NGO involvement were not in any shape or form considered by NUWATER while its capacity building efforts cannot be construed to constitute a meaningful or serious capacity building plan to ensure even a minimal level of sustainability. 3.4 The effectiveness of the program management and its effect on the program outcomes The Team interpreted this question as relating to the management by NUWATER, USAID and WASH Consult and each is addressed below. Respondents seemed to use this question as an opportunity to vent their feelings about NUWATER. The Evaluation Team however feels that the perceptions the various actors had of each other each in the context of this particular evaluation question constitute valid data although the Evaluation Team did try to the extent possible to obtain and confirm with examples given by actors. Degree of On-Site Supervision One particular question that USAID/Uganda asked the Evaluation Team to address was the extent to which the NUWATER Team provided on-site supervision and management versus administering the project from their Kampala office. In the course of its field work, the Evaluation Team heard consistently and frequently from all the stakeholders - the Town Councils in Kitgum and Pader, Kitgum District, water managers in Pader, Private Operator in Kitgum and USAID itself – that there was limited NUWATER field presence. According to NUWATER, they were supposed to have a field office in Kitgum with a resident Field Coordinator, from which operations in Pader would also be coordinated. The NUWATER Field Coordinator was hired late in the life of the project and then only at the insistence of USAID, a factor that contributed to a lack of initial on-site supervision and which was a reason why rehabilitation activities started late. NWSC staff interviewed said the NUWATER Field Coordinator was acting as ―Technical Advisor‖ to the COP and thus spent much of his time in Kampala, instead of being on the ground. Pader‘s local government officials also cited the limited presence of the Field Coordinator claiming he took a long time to respond to requests and 33 NUWATER End of Project Evaluation thus caused delays to repairs of taps or pipes. The Private Operator in particular complained that NUWATER did not have a fully functional field office from which to coordinate activities in a timely manner. The Evaluation Team confirmed that NUWATER supposedly occupied one of the rooms in the Private Operator‘s building. The team also discovered that NUWATER field staff never occupied it on a full-time basis and had no work station there as all the office space was occupied by the private operator‘s staff. Further, the Evaluation Team did not find any NUWATER staff in that office at any point during their one-week visit to Kitgum. The Evaluation Team discovered that the Field Coordinator had rented a house in Kitgum for only four months. However, ARD later supplied information from which the Evaluation Team calculated that the Field Coordinator spent about 75% of his time in the field. In the light of these conflicting claims, the Evaluation Team is unable to draw any firm conclusions regarding this aspect of NUWATER‘s management that is in any event better suited to an audit than an evaluation. Financial Management Controls Most respondents appreciated the financial controls instituted by NUWATER. At Kitgum TC, 25% of officials thought NUWATER was well managed although it poorly coordinated with the Private Operator and they supported the COP‘s requirement that he approve all financial transactions as a good practice to emulate. Members of the Kitgum Water Board were divided on the question of overall management effectiveness, with a majority (2 out of 3) agreeing that NUWATER introduced a very strict system of financial management. NUWATER Staff Expertise The Evaluation Team reviewed the CVs of NUWATER staff that were made available to them and interviewed NUWATER staff. Two out of three NUWATER staff interviewed had some experience with the Uganda water sector. However, the COP had no water engineering experience although his experience in water utility management appeared to be good on paper. The engineering expertise of the Field Coordinator was perceived as low by the Evaluation Team‘s Water Engineer. This was assessed by asking technical questions during the site visits and when in the field. Chief of Party’s Management Style The management style of the NUWATER COP came in for widespread and vociferous criticism. All four NWSC staff interviewed asserted that NUWATER was managed poorly and behind this assessment appears to be their dislike of the COP‘s modus operandi. They described the COP as “reactive, defensive, vindictive and likes operating behind people‟s backs; a fault finder.” NWSC‘s role was to supply local expertise and water utilities management expertise to the NUWATER team but they felt that they failed to work well with the NUWATER COP. NWSC staff reported that the COP did not react well to any observations in their reports that did not concur with his own perceptions and that he ―hid behind USAID to explain his failures‖. They concluded that their expertise was not appreciated or able to benefit the project. Their assessment is corroborated by a USAID/Washington respondent who characterized NUWATER‘s project 34 NUWATER End of Project Evaluation management style as being in ―open conflict with NWSC. Indeed, NWSC was not happy with the situation as their views were not taken into account”. Fifty percent of officials in Kitgum thought the NUWATER management style stifled the PO thereby affecting the quality of services. The relationship between NUWATER and the Private Operator was tense. One Town Council official in Kitgum described shouting and fighting during joint meetings to an extent that some TC officials feared attending the NUWATER meetings. Several interviewees reported shouting matches and loud arguments, harassing phone calls and threats that frightened some respondents and led to poor cooperation, coordination, and project progress USAID staff also found the COP difficult to deal with and unwilling to assume responsibility for project results to the extent of refusing to set targets for some of the performance indicators in the NUWATER PMP. It is the view of the Evaluation Team that many of the contested findings presented in this Evaluation Report are likely due to the bad will created by NUWATER through its COP‘s style of managing. In addition, the COP was largely responsible for project delays, poor quality performance management and reporting and failure to grasp the importance of capacity building as detailed in the sections below. Project Delays Although NUWATER was a three-year project, not many activities took place in the field between July 2008 and February 2009. All stakeholders interviewed agreed that the project was not implemented in a timely fashion. According to USAID, that was because it was not comfortable with the current private operator in Kitgum, Trandint Ltd. However, the Town Council and MWE were reluctant to terminate the contract with Trandint, and MWE wanted an MOU signed between USAID and MWE before separate MOUs were signed with the respective Town Councils. USAID agreed that it asked NUWATER not to carry out any activity in this period in order to put pressure on the Town Council, and that initially it had planned to give an extension in the project period to compensate for this. However, when the problems in project implementation became evident, USAID decided to terminate NUWATER on the original contract end date. NUWATER said the main project delay was created at the beginning of the project and that its hands were tied due to the USAID decision. The Field Coordinator started work on the project only in December 2009 after USAID requested that such a person be hired and therefore not much was done on the technical side before this time. Added to this was a change in the NUWATER COP at the outset of the project as the slated incumbent was not available and a replacement had to be found. Once the COP was hired NWSC asserted that he took a long time trying to understand the context and the procurement and other water systems processes. From a review of the monthly monitoring reports prepared by NWSC‘s M&E Unit between August 2009 and November 2010, the Evaluation Team was able to find documented proof of delays on the part of NUWATER in respect of:  Providing billing software. This was requested by NWSC in every monthly report from August 2009 onwards. The software appears to have been provided by the time of the November 2010 M&E visit when it was reported that it was not being used by the WASH staff (15 months later). Training was only provided in May 2011. 35 NUWATER End of Project Evaluation  Sourcing and providing water connection meters. This was mentioned in every monthly report from August 2009 and appears to have been resolved only by the time of the April 2010 monitoring visit by NWSC (eight months later).  Faulty bulk meters to measure water production were mentioned in the monthly reports from November 2009 onwards. Project Monitoring and Reporting Both USAID/Uganda and UMEMS reported a lack of timely reporting on project indicators and stated that reports and data had to be requested many times before they were received. A review of the two quarterly and two annual NUWATER reports revealed that they were short on detail for the capacity building, public awareness and gender components of the project. The NUWATER Program Description did not provide the usual USAID Results Framework model but rather a set of tasks that ARD was to accomplish even though the results sought from the project were very clearly stated. ARD‘s first PMP likewise did not clearly articulate the results although a later version of the PMP did provide a results framework. A review of the Project‘s 2009 version of its PMP (ARD, March 2009) revealed that NUWATER did not consider itself responsible for results at the highest level of the project‘s results framework and by implication for the major performance indicators relating to subsidy reduction, performance bonuses and non-revenue water (non-revenue water was dropped from the July 2010 version of the PMP and replaced with an indicator measuring increased access). This stance appears to have been adopted because NUWATER was claimed to be a ―hypothesis model-pilot‖ type of project (ARD, 2009:1) and because the indicators did not lend themselves to performance monitoring or accountability as they were ―elastic‖. However, nowhere in the USAID program description is NUWATER conceptualized as a pilot project and in fact, the expectation was that NUWATER would follow the “well established and successful management contracts used by the DWD and NWSC” (USAID-ARD Program Description excerpted from the contract; 2008:6). The only variable that could be construed as different from prior work using this model is that NUWATER was to be implemented in the post-conflict environment of Northern Uganda. If USAID had intended a model be tested, it would not then have held the contractor responsible for “the achievement of tangible results (which form) the essence of the contract and that USAID will judge the contractor‟s success in the contract based on whether or not the tangible results are achieved” (USAID-ARD Program Description excerpted from the contract, 2008:12). The tangible results sought by USAID in the NUWATER Project were improved quality of service, expanded customer base and evidence of financial sustainability (USAID-ARD Program Description, 2008:6). In any event, NUWATER never set any targets for the higher level indicators, nor did it ever provide any performance data in terms of which the model‘s viability could be tested. As documented in other sections of this report, NUWATER either failed to set targets and/or did not supply data for more than half the PMP indicators (6/12). UMEMS reported frustration in their efforts to get NUWATER to comply with Mission M&E requirements. Targets for indicators related to capacity were never adjusted to more realistic production levels by carrying out pumping tests and results were affected by the eight-month delay in installing the first bulk 36 NUWATER End of Project Evaluation meters needed to measure production capacity accurately. The failure to provide data for at least some of the performance indicators is strange as it was available from the NWSC monthly M&E Reports e.g. non-revenue water, household connections, billing etc. NWSC in fact provided some useful advice to the Operator on how to obtain better data for some of these indicators e.g. pumping hours, water supplied etc. All data supplied in the M&E reports was verified by the NWSC Team e.g. each connection was physically inspected and billing cross-checked. These high quality data could easily have been adapted for performance reporting by NUWATER. Plans to conduct annual water quality testing and customer satisfaction surveys never materialized and it is doubtful if internal data validation reviews involving ARD‘s Senior Technical Advisor ever took place given the parlous state of monitoring data for the project. Certainly the quarterly reports on the PMP were not submitted to USAID/Uganda per the M&E Work Plan embedded in the PMP. The baseline survey was supposedly conducted in both towns in November-December 2009 but neither USAID/Uganda nor the Evaluation Team was ever availed of the report for the Pader survey. How the data provided by the Kitgum baseline survey were used to inform the July 2010 version of the PMP is not clear and a baseline survey conducted 18 months into a three-year project implementation period is of limited value in any event. Coverage of Contract Scope NUWATER‘s Additional Contract Requirements (C.5), per Statement of Work (C.4), included several issues that ARD appears to have failed to conduct entirely. These are gender considerations; establishing, collaborating with NGOs; and training to build local training and implementation capacity, including training- of- trainers. See Section 3.2 above for more detail on these aspects of their program. 37 NUWATER End of Project Evaluation 3.4.1 USAID Management The NUWATER Project benefited from relatively consistent USAID management of the contract with a change in COTR only occurring toward the end of the project. The majority of USAID staff interviewed felt that USAID/Uganda Mission did not have sufficient capacity in the water sector to oversee NUWATER effectively. The USAID/Washington respondent echoed this assessment. The Mission had little history of supervising such technically complex projects as NUTI operated relatively independently of the Mission for the time it was active in Uganda in the water sector. The only other water sector projects were two small grants awarded in 2010 to local organizations. NUWATER was felt to have been imposed upon the Mission by USAID/Washington and thus also had little support from the Mission. Although USAID Staff at the USAID/Gulu Office tried to supervise NUWATER they also did not have the requisite expertise. Kitgum Water Board members picked up on this and felt that USAID needed to hire water sector experts to oversee NUWATER. The USAID Engineer in Gulu felt he was brought on board too late to be able to make a real impact on the project. Most of the external stakeholders of the project felt that USAID/Uganda did not have a sufficiently high on-the-ground presence. Five out of the seven Kitgum officials interviewed by the Evaluation Team claimed that USAID rarely visited the project sites to verify project activities. The officials thought USAID could have done better by visiting the project more often and by streamlining internal decision-making. The PO said better results would have been achieved in the project if USAID had held ARD accountable and pushed harder to overcome the delays experienced as a results of NUWATER‘s management. Greater water sector expertise on the part of USAID/Uganda might also have resulted in a more timely project and contract modification and appropriate performance monitoring. At a minimum, USAID/Uganda should have insisted on greater accountability from NUWATER by insisting that targets be set for its PMP indicators and on better quality and timely reporting, especially given that it had entered into a contractual relationship with ARD. However, the decision to require that NUWATER hire a Field Coordinator was a good one even if it came late in the project.Similarly, its request that the project hire an M&E Officer was in the right direction even though it came late and was not followed through. Finally, on recognizing the constraint that water production posed for the project, USAID did procure a new design and after that a new water infrastructure contract in 2011. The quality of oversight extended to non-engineering matters as well. Monthly and quarterly reports from NWSC ESU M&E team to NUWATER for several months (August 2009 - November 2010), indicated that there were financial problems in the management of bank accounts, billing and collections, and Value Added Tax payments. ESU M&E recommended specific actions to improve financial management. The Evaluation Team‘s discussions with USAID officials indicated USAID was unaware of these issues and took little or no action. A key error on the part of USAID/Uganda was to delay the NUWATER project while negotiating a MOU with the Government of Uganda when other modalities for working existed and were used confidently by other development partners while the bureaucracy took its course. 38 NUWATER End of Project Evaluation The Evaluation Team concluded that USAID did not exercise its oversight responsibilities adequately and allowed the contractor to evade its responsibility for delivering the tangible results required by the contract. 3.4.2 WASH Consults Management WASH Consults was formerly a water systems construction firm and the new staff hired specifically for the NUWATER project by WASH Consults did not have any water utilities management experience. The Evaluation Team asked simple engineering questions of three technical staff members of WASH Consults and it was apparent that they had extremely limited knowledge of basic hydraulics and maintenance procedures for urban water network. For example, WASH technical staff members were unaware of the role of water pressure in managing the distribution system and were unaware of the engineering design requirements to increase water pressure and water flow. This lack of expertise also became clear from a review of monthly reports from NWSC M&E Team to NUWATER (August 2009 to November 2010). The NWSC ESU M&E Team that supported WASH Consults produced reports that were well-written, detailed, technically sound with helpful recommendations. Highlighted in every report the Evaluation Team reviewed are issues with financial management - failure to bank on a daily basis, missing receipts, inaccurate billings, failure to reconcile bank statements and cash cheques being made out when not permitted to do so. One example that the Evaluation Team tried to verify was the claim by the NUWATER COP that the Town Clerk in Kitgum withdrew 2 million Uganda Shillings in early 2011 without the authorization of NUWATER or the private operator. When this was questioned, the TC replied that this money was used to purchase land for the new wells that were being drilled for the expansion of the water system. However, it was not possible to verify this. ESU also provided a special training exercise to improve WASH staff capacity in financial management and reconciliation of bank statements in May 31, 2010, and provided informal capacity building in human resources. The reports abound with examples of poor workmanship. For example, 86 customers paid for new connections between August 200 and December 2009 but were not connected because there were no water meters. Later, three of the customers put pressure on the operator to connect them without meters which is not a sound financial practice. When the connections were verified by the ESU M&E team, the Team found the connections were sub-standard, e.g., wrong pipe diameters, inadequate depth of trenches. The Evaluation Team received information from respondents about the poor standard of operations, which may or may not be attributable to the current WASH and Pader Water Authority operators. For example, during field visits and discussions with operating staff, the Evaluation Team documented the following poor practices: a meter was installed backwards in Pader; wrong types and sizes of water meters are being used in Kitgum; several consumer meters were installed at too shallow a level in Kitgum to meet national standards; a meter was removed without connecting pumps to the distribution systems in Pader; a missing roof in a Kitgum wellhouse; a Kitgum pump burned out at the KTI pump station in September 2010 because its electrical connection bypassed the electrical stabilizer and received high voltage;, pumping stations and elevated tanks and outside meters left unlocked and unsecured in Kitgum and Pader; 39 NUWATER End of Project Evaluation new wells in Kitgum and Pader not protected or secured from vandalism. There were also routine delays in finding and fixing leaks, repairing vandalized pipes, and replacing broken taps because of the lack of spare parts and timely on-site inspection and approvals that were to be provided by NUWATER‘s Field Coordinator. A lack of time and resources meant that the Evaluation Team could not verify each claim but a report detailing its site visits documented and photographed many similar shortcomings (Appendix K and O) Given its experience as a construction sub-contractor to the USAID/Uganda‘s Northern Uganda Transition Initiative (NUTI), WASH should have been cognizant of better operations, including O&M, preventive maintenance, spare parts, and security. As at May-June 2011 when the Evaluation Team conducted its site visits, WASH was not able to show an O&M manual, Preventive Maintenance Plan, guidelines, checklists, work-flow-process protocols and related documents generally expected of utility managers. Neither WASH in Kitgum nor the Pader Water Authority had a spare parts collection, equipment warehouse or bone yard, which are generally expected of utility managers. WASH did not seem to grasp the importance of mobilizing and sensitizing the community to protect water sources and distribution systems from vandalism and abuse or did not have the time or skills to do so while Pader made repeated requests for such support and received a little support from NUWATER. Based on the monthly reports from August 2009 to November 2010, the NWSC M&E team found several cases in Kitgum of cutting corners in costly operations, such as payment in rentals, vendors, day laborers, National Social Security Fund (NSSF), VAT, licensing fees, and other taxes, fees, expenses, and direct costs.. The M&E Team repeatedly reported there may be investigations and penalties with serious consequences through the NSSF and the Uganda Revenue Authority (URA). Based on a review of records, the Evaluation Team confirmed in June 2011 that WASH Consults did not have a record of remitting any NSSF payments. It is common practice in Uganda for many organizations to try to avoid the NSSF, VAT, withholding tax and other taxes in general. However, avoidance of legal financial requirements is not a good practice for a utility company and may lead to more corruption, forced utility shut downs or closure, and non-sustainable utility services. For example, in February 2010, the URA blocked the escrow account of the Kitgum water system. Ono a more positive note, WASH Consults did from time to time meet targets in respect of average response time for technical complaints and leaks and bursts although it did not fare so well consistently in respect of timely responses to billing complaints. WASH was pumping at 80.6% of its target and had met 99.8% of its target for metering efficiency as at November 2010, the last time NWSC/ESU monitored the project. No other targets were met. Since the beginning of 2011 without the oversight of NWSC/ESU, WASH Consults has experienced a relapse on all indicators such as non-revenue-water, production and collections. Overall, the Private Operator performed very poorly and clearly required a significant amount of support to build its capacity in all areas of water utility management. 40 NUWATER End of Project Evaluation 4 - Conclusions The NUWATER project evaluation was undertaken to answer four key evaluation questions and determine lessons learned from the project. The overall assessments are summarized in the table below and a more detailed exposition follows. Table 2: Answers to Key Evaluation Questions Key Evaluation Questions Evaluation Outcome 1. To what extent did the project meet its goal of improving access to water in Kitgum and Pader? NUWATER improved access to water but could have and should have done much more and sooner. 2. How realistic and appropriate was the design of the project? The project design was not realistic or appropriate, and should have been modified early to achieve more practical and long-lasting results. 3. Were the appropriate stakeholders involved in the program and to what extent did the program promote better coordination and collaboration amongst them? The appropriate agency stakeholders were involved with the program but the beneficiaries were not involved, mobilized or sensitized; the program was marked by poor communication and collaboration. 4. Analyze the effectiveness of the program management and its effect on the program outcomes. ARD, USAID, WASH, and Pader water management entity management were not effective, which hindered program outcomes. 4.1 Management of NUWATER Project There were significant shortcomings in the management of the NUWATER Project on the part of all parties. Despite their lack of expertise in the water sector, USAID could have been more active and could have corrected the performance problems that ARD was having in implementing NUWATER. USAID should also have been more active on the ground in order to interact more frequently with other stakeholders to ascertain the truth of allegations made by the different parties. At a minimum, USAID could have demanded better quality and more timely and quantitative reporting and performance monitoring. The biggest shortcoming on the part of USAID was to halt activities while waiting for the MOUE. A short-term action plan comprising items such internal procurement and design of mobilization campaigns and capacity building programs should have been instituted at this stage to maximize the time available. Finally, if USAID was dissatisfied with ARD‘s performance, it should have changed contractors or changed key ARD in-country staff as soon as the problems became apparent. USAID/Uganda‘s lack of experience in the water sector led to a number of problems that could otherwise have been avoided: 41 NUWATER End of Project Evaluation  Early recognition of the true situation on the ground in Pader would have resulted in a more drastic and timely change in project design with a focus on rehabilitating and expanding the kiosk network and not promising house connections that could not be made  Alternative models could have been chosen that were more suitable for a town the size of Pader such as a local management group supported by the District and an umbrella organization  Small improvements in the infrastructure such as drilling new production wells could have been done in the first months of the project and in this way, all the subsequent problems arising from poor production capacity could have been avoided  USAID should have seen the need for community mobilization in both towns and insisted that the contractor carry out and document such activities  USAID should have recognized that the management style of NUWATER was not conducive to sustainability because it did not give local stakeholders the possibility to be actively involved  USAID should have taken capacity building more seriously and insisted upon a structured program of activities. ARD, the USAID implementer of NUWATER, failed to meet expectations overall. It fell short of achieving its contracted tasks in incentive-based private-water operator services, capital infrastructure works, and human resources capacity building due to poor headquarters and in￾country management, with the latter being particularly inflexible. There were also delays in gazetting and procuring private utility operators. Other activities it could have undertaken in the interim include procurement (especially of water meters), and monitoring and evaluation and financial management, as is commonly done in Uganda by non-governmental organizations (NGO) and others working in the water sector while MOU negotiations are ongoing. Although it was difficult to obtain definitive evidence, ARD COP‘s aggressive, argumentative, accusative and bullying management style embarrassed some respondents at best and intimidated others at worst. This had knock-on effects on the levels of collaboration, partnering and USAID‘s own image amongst stakeholders. NUWATER staff could have increased the impact of the project considerably by undertaking the following actions:  NUWATER should have asked for an early modification of the contract, taking into account the fact that Pader had no functioning system suitable for management by a PO  If technical staff were there from the beginning of the project, targets would have been changed after pumping tests in order to make them more realistic both for M&E and for the incentive contracts  NUWATER would then have recognized that with relatively small and cheap interventions such as drilling new production wells next to old wells many future problems could have been avoided  Mistakes such as procuring faulty meters would have been avoided  More technical capacity and management capacity could have been built in the PO and management of Pader if NUWATER staff had these skills 42 NUWATER End of Project Evaluation  More skilled staff would have recognized the need for capacity building and community mobilization  A better management style would have gradually transferred capacity to the local stakeholders, taking into account the advice from NWSC, MWE and TC, and in this way made the intervention more sustainable. 4.2 Contracts and Water Management Models The concept of using of incentive-based contracts was good but the lack of capacity of private operators in the Ugandan water sector was not properly taken into account. The contract was therefore too complex both for the TC to manage and for the operators to bid on. It assumed that the financial management practices of the private operator were impeccable and that rules were followed, something that was not the case in practice. Also, the lack of policy in Uganda on how to deal with previous operators‘ debts caused problems that minimized the incentives for the private operator. The contracts should never have been signed based on outdated production figures and NUWATER should have conducted pumping tests to verify production levels before signing the contract with the private operator. Failure to do so reduced the incentives for the operator, rather than increasing them. In addition, the contract was completely unrealistic in Pader because of the absence of house connections. This should have been recognized by NUWATER immediately and appropriate amendments made (see further below). With regard to the low level of interest on the part of private operators, this was likely due to local political factors, and it is difficult to determine if NUWATER could have done anything differently to affect this. The Evaluation Team believes the project achieved the least impact and lost most opportunities in Pader. Pader is a RGC not a town (see Appendix B), and had no operating water system. It is common for RGCs in Uganda, and also small, un-gazetted towns, to adapt more ―informal‖ management systems for their water systems. This is because a system that does not have private connections but only public stand-posts, rarely achieves sufficient margin to motivate a private company. Instead, local technicians or individuals are trained to manage and operate the system under the supervision of the Town Council and the Water Board (or a similar entity) and supported by the District and various umbrella organizations that have been created in different parts of Uganda to this end. Given that Pader was not going to be attractive for a private operator, this approach could have been chosen instead. However, NUWATER pushed ahead with what transpired to be an unnecessary and lengthy gazetting process. The interim management system chosen by NUWATER was also not ideal. In addition, the stakeholders involved only received minimal training and follow-up. Having TC officials directly manage the water system without the involvement of the community is risky in terms of transparency and conflicting interests. In addition, people generally do not trust government officials because of the widespread corruption in local governments, so they are reluctant to pay for water when they know the TC is managing the system. Experiences from elsewhere in Uganda show that local governments should generally only perform a supervisory role when it comes to water management in order to enhance sustainability. 43 NUWATER End of Project Evaluation 4.3 The Problems with Infrastructure Many of the problems in the project were blamed by NUWATER and the private operator on the lack of functioning infrastructure. In Kitgum, production levels were the main constraint to achieving new connections, an increase in revenue and customer satisfaction. However, the Evaluation Team concluded that this problem could have been swiftly dealt with at the start of the project within the limited budget for infrastructure improvements available for the project. ARD could have conducted an early technical survey and inventory of the towns‘ water utility systems, and moved quickly to upgrade it, procure bulk meters, and make additional repairs, promote community mobilization and sensitization, prior to getting the operator on-board. New production wells drilled in the last months of the project (but not connected at the time of this evaluation) showed that production levels could have been increased three or four-fold by drilling new wells close to the existing wells. In Pader, at a minimal cost, impact could have been increased many times by just expanding the existing network with 5-10 new kiosks. All this would have been possible within existing budgets and time frame and it was difficult for the Evaluation Team to understand why NUWATER did not prioritize this and implement it within the first months of the project. While it is clear that in the long run both towns need a significant upgrade in their water systems, these small interventions would have probably led to an achievement of targets and a much greater impact. 4.5 Capacity Building There was a complete lack of focus on capacity building even though it was a key project objective. Formal capacity building was completely absent although it was desperately needed by the private operator in Kitgum that had no prior utility management experience and the Water Boards that had a poor understanding of overseeing contracts of this nature. In Pader, activities on the water system started late so there was almost no monitoring or follow up for a town that desperately needs increased capacity in water management. 4.6 Sustainability The Evaluation Team concludes that the project was not implemented in a way to enhance sustainability. There was insufficient focus on capacity building as explained above and the NUWATER management style was over-controlling and failed to assign responsibility to the private operator or the water board. WASH Consults was found to have limited capacity to act as a private operator, both technically and in water utilities management. In the Ugandan context, however, WASH Consults was not doing too badly and performance improved steadily until the NWSC ESU M&E team was taken off the project in late 2010. Since the beginning of 2011, WASH Consults has experienced a relapse on all indicators such as non-revenue-water, production and collections (See data in Annex N). The new infrastructure planned by USAID will increase the customer base and also the production and storage capacity, but with the current level of knowledge and experience it is not likely that WASH Consults will be able to successfully manage this unless technical and managerial back stopping is provided by an external entity and the TC and Water Board are trained and facilitated to play their supervisory role. 44 NUWATER End of Project Evaluation 5 - Lessons Learned This project has produced a number of lessons learned at all levels that are important to take into account for future projects. The main lessons are summarized below. 5.1 Project Design It is important that the project design reflect the real conditions on the ground and that the terms and concepts used are clearly understood by people outside the Ugandan context. Concepts like ―rural‖ and ―urban‖ can mean something entirely different in Ugandan and American contexts. The project design was based on a number of assumptions that were simply not true on the ground. These include beliefs that Pader was a functioning town and ready to contract a private operator (i.e. it was gazetted), that the towns had functioning water systems and that there were private operators interested in running them. The production capacity of Kitgum was based on outdated data and led to the calculation of incorrect targets and the wrong incentive payments. In addition, there were institutional factors such as a need for an MOU with central government and delays in procurement that should have been anticipated and integrated into early work plans. 5.2 Technical Capacity The NUWATER Project experienced a lack of expertise in almost all subjects necessary to successfully implement this project - engineering, water utilities management, community mobilization and project management. It is therefore extremely important that in future the staff in USAID and all the staff of the contractor including local staff are highly qualified. It is possible to sub-contract expertise from NWSC. The feasibility study noted the capacity gap in local government and these need to be properly recognized from the start of the project, and then addressed with well managed and results-based capacity development programs that are approved within technical proposals, work plans and PMPs. 5.3 Implementation Implementation is a complex task that does not necessarily follow project plans. USAID, the contractor and the other stakeholders need to be flexible and adapt project implementation to the situation on the ground. This is only possible with good communication. Where the project design is shown to be difficult to implement in its original form, it is necessary that the contractor present a logical case and ask for the necessary modifications in a timely manner. Thorough technical assessments and baseline studies need to be carried out early in the life of the project and not when the ―going gets tough‖. At the same time, when delays such as the signature of an MOU or delays in the procurement of a private operator, USAID needs to work with the implementor to develop a short-term action plan so that activities that are able to be carried out are implemented and in this way move the project forward. It is also necessary that the COTR closely follow up the progress of the project and demand explanations for delays in reporting, spending and progress. 5.4 Involvement of Stakeholders and Beneficiaries This project shows how important it is for sustainability to actively engage with stakeholders. Stakeholders should not only be ―involved‖, but encouraged to take a leading role in the project. 45 NUWATER End of Project Evaluation In order to motivate stakeholders, capacity building is a powerful tool and it gives the stakeholders the necessary knowledge to carry out their role effectively. In a short project like this, emphasis should be put on transferring as much responsibility as possible to the stakeholders at an early stage. Capacity building needs to be taken seriously and included in performance monitoring. A group that was largely neglected by NUWATER was the water consumer despite their being key to sustaining good management of the water system, to increase willingness to pay, to avoid vandalism and to create accountability and transparency. Any project that involves water supply should include activities for beneficiaries in order to ensure that support is built from the ground upwards. 5.5 Incentive-Based Contracts The idea of incentive-based contracts is a good one and other versions of this are being implemented with success through the MWE. However, this project shows that they are definitely not suitable for smaller towns or RGCs such as Pader that have difficulties attracting a private operator in the first place. Also for a town like Kitgum, the complexity of the contract was too challenging both for the private operator and the Town Council. In addition, in order to work, incentives need to be ―real‖ and not unachievable goals based on inaccurate data. Another important lesson is that with limited capacity in customer care, even with a 95% subsidy on collections, this alone is not enough to motivate the operator to increase collections. The incentives need to be accompanied by capacity building, monitoring and follow up. A three year project duration is too short to create a real incentive and for the operator to reach a level of revenue to become self-sustaining. It is also necessary to keep in mind that, according to MWE, only 70% of small town operators break even. Some water systems are simply too costly to operate for a company to make profits. Even in highly developed countries, most water utilities are subsidized operations, so complete sustainability is unlikely to be achieved. 5.6 Sustainability In a project that aims at the same time to invest in capital infrastructure and instil sustainable community-based and private sector management models, three years is too short a time to register significant accomplishments. However, this should not stop the project management from working toward the maximum sustainability possible. This includes proper involvement of stakeholders at all levels and giving them responsibility and capacity to stand alone after the project ends. It also means giving stakeholders simple tools and routines to undertake as well as providing for checks and balances, especially when dealing with financial issues. It is also necessary that stakeholders are involved in the construction, maintenance and follow-up of new infrastructure and that they are given the necessary manuals and capacity to operate and manage the system. 46 NUWATER End of Project Evaluation 6 – Recommendations The Evaluation Team has grouped its recommendations into the following themes: 6.1 Improving Capacity for Managing Water Sector Projects The Evaluation Team recommends USAID ensure that its water sector projects are designed, procured, managed, implemented, and monitored and evaluated by experienced, proactive professionals in water utilities, through an independent, transparent and timely peer review and quality assurance/ control processes, similar to that required in the United States for water utility projects. Where such capabilities do no exist in the Mission, USAID could consider sub￾contracting the management to a qualified sub-contractor that does have the expertise. USAID could also have drawn on the expertise of the DWD and NWSC to review infra-structure related project design work 6.2 Future Design & Implementation Models The demand for kiosk and connected water is high but because of the high levels of poverty in the more rural towns of Uganda, low-cost kiosk water and subsidized household and yard connections are very attractive to potential water customers and should be continued. To facilitate timely and appropriate water sector interventions in Uganda, the Evaluation Team also recommends that USAID work directly with MWE, NWSC and/or community-based organizations to develop any future project design, check its assumption, and implement the project. In general, well-established national NGOs can implement small-scale water projects faster than government agencies while smaller local partners can also assist with community mobilization efforts and with their knowledge of local conditions, contribute to selected aspects of the design of water sector projects. 6.3 The new Infrastructure Project Going forward in Kitgum and Pader, a comprehensive, professional assessment and design for water services should be initiated. The Evaluation Team understands that a USAID-funded capital investment project is currently underway. It is highly recommended that community mobilization be integrated into the project and that technical and managerial support be extended to the private operator or manager of the water system at least in the transition period. Although for the current water system in Pader a private operator is not suitable and a more informal, locally-trained operator would be better, depending on the size and potential of the new infrastructure development in Pader, a private operator might be attracted. It could take a long time until the water system reaches a critical mass of private connections in order to make operations profitable due to the poverty in the area. Therefore, it is important to integrate as many public water sources as possible and ensure they are adequately managed with water meters and transparent routines. 47 NUWATER End of Project Evaluation Kitgum will largely benefit from the new infrastructure development. However, the current operator, WASH Consults, reported to the Evaluation Team that it was not interested in continuing operations in Kitgum. With a new and better infrastructure in place, it may be possible to attract more bidders to the procurement process. However it is crucial to work closely and collaboratively with MWE as its plays its important role in overseeing this process and to gain its support for the project. Also, even if a new private operator is contracted, the Water Board, Town Council, District Water Office and probably the private operator itself will still need capacity building and follow-up in order to properly oversee and monitor the contract. In addition, beneficiaries in Kitgum are willing to be involved and should be given this chance. This is important because unless consumers are included in the process, it will take a long time to turn the poor reputation of the water management around in order to attract new customers, motivate people to pay bills, and have illegal connections reported. 6.4 Centrally-designed versus Locally-designed Mechanisms This recommendation is related to 6.1 above in that, while it is tempting to resort to a centrally￾designed project where the Mission lacks the capacity, the Mission should not assume responsibility for supervising the project without support from USAID/Washington. At a minimum, experts from USAID/Washington should visit the field periodically and review project progress reports and work plans and provide on-going support to Mission staff if they lack the technical background to supervise the project. 48 NUWATER End of Project Evaluation 7 - Appendixes APPENDIX A: NUWATER Evaluation Scope of Work I. BACKGROUND The Northern Uganda Water Supply Services (NUWATER) activity is a 3-year, $3 million USAID/Uganda-funded program created to assist Kitgum and Pader towns to re-establish their water supply systems, using incentive-based management contracts with private service providers. The project began in June 2008 and is scheduled to end June 2011. NUWATER is a three-year contract that started in June 2008, with the goal of improving access to water in Kitgum and Pader towns through improving the urban water supply systems using incentive-based contracts with private service providers for operation and maintenance of the water systems. Originally the project was to address the water supply in three towns: Kitgum, Pader and Aloi. An early assessment by the contractor determined that Aloi was not a good candidate for an incentive-based contract that would depend on user fees to sustain the system. The life of project work plan that the contractor submitted to USAID in August 2008 identified the four major elements:  Provide capital improvements to ensure that water supply systems are viable financial entities with the water production capacity and customer service base necessary to recover operational costs  Engage private sector contractors through management contracts that provide sufficient economic incentive to provide improved services  Provide Output Based Aid (OBA) for customers for establishing new connections in order to increase water availability and operator revenues (from water subsequently sold), and  Provide capacity building support and training in contract management to local government so that they can manage these contracts once the project is completed. The work plan goes on to identify the following indicators linked to USAID contractor performance: Indicator 3.1: Number of contracts tendered and issued, awarded and completed (with reporting on overall quality of performance); Indicator 3.2: Number of connections/points rehabilitated, expanded, and/or new connections/points established (and reporting on other system improvements made); Indicator 3.3: Number of capacity building and/or outreach activities facilitated (on customer responsibility, management/billing/accounting systems, contract management, etc.). 49 NUWATER End of Project Evaluation II. PURPOSE OF THE EVALUATION As the activity draws to an ending, USAID Uganda is commissioning this evaluation is to better understand the overall lessons learned and impact from the activity to date, including but not limited to, what USAID and partners are obtaining from this activity, what is going well and what is not and how can lessons be applied in future programming. More specifically, USAID/Uganda requires the Evaluation Contractor (the ―Contractor‖) to design and implement an evaluation of NUWATER. The evaluation will serve the following purposes: (1) provide lessons learned for USAID, and, the Government of Uganda (GOU), and other development partners supporting the water sector; (2) assess the existing and/or potential ability of key successes to be replicated, (3) uphold an institutional commitment of measuring program results; and (4) provide practical lessons for current and future water sector partners in developing and implementing sustainable water supply systems. Furthermore, the evaluation shall discuss and analyze program performance and success but should also address opportunities missed or accomplishments that fell short of potential or expectations, as this information can be a useful tool in informing future USAID/Uganda programming after NUWATER closes, particularly in the context of the Mission‘s implementation of the Country Development Cooperation Strategy (CDCS). Part of the CDCS, Collaborating, Learning and Adapting (CLA), specifically requires the use of a methodology to improve development efforts through increased coordination and collaboration, testing of promising new approaches, and adaptation of approaches when appropriate to improve effectiveness. The evaluation should take this effort into account when analyzing the evolution of NUWATER‘s strategy and approach and should include these findings in its recommendations for the Mission. Based on the key findings and recommendations, the evaluation will inform USG and key stakeholders on future programming and collaboration. III. EVALUATION QUESTIONS The Automated Directive System (ADS) 203.3.6.1 requires that an evaluation is conducted when there is a distinct and clear management need to address an issue. This review is to critically examine the overall NUWATER project progress/impact to date. The evaluation methodology and process shall address the questions outlined below: 1. To what extent did the project meet its overall goal of improving access to water in Kitgum and Pader? The Contractor should review the program performance and establish the extent to which the intended goal and results have been met. How does the current water supply situation in each town compare to what it was prior to the NUWATER activity? Analysis of performance should also establish the degree to which the program has established and or supported mechanisms and institutions that will guarantee sustainability of the services provided or benefits realized to date. What were the success factors and challenges that have hindered the achievement of the results? 50 NUWATER End of Project Evaluation 2. How realistic and appropriate was the design of the project? Here the evaluator should review the project conception, and design whether it was responsive to the need at that time, the extent to which the proposed implementation approach was realistic and appropriate. Analysis of the project components and their ability to lead to the desired goal. To what extent did this design influence the outcomes? The development hypothesis of the project was identified as: ―Well designed incentive-based contracts have the capacity to ensure recovery of costs associated with water supply service provision and will result in significantly improved services to customers, resulting in long-term sustainability of the systems serving these customers.‖ Did this prove true? If not, why? Did the incentives in the contract really serve as incentives to the private operator? What is required to ensure the incentive based contracts result in improvements in services (compare with other output based aid (OBA) related contractors in Uganda. 3. Were the appropriate stakeholders involved in the program and to what extent did the program promote better coordination and collaboration between them?. The evaluation should establish and assess the extent to which the critical stakeholders and inter-relationships were correctly identified, structured and involved in the delivery and management of the activity. To what extent did the program address gender issues in both the participation and enjoyment of benefits? The Contractor should recommend how the stakeholder involvement could have been managed differently in relation to the positive realization of the program results. 4. Analysis of effectiveness of the program management and its effect on the program outcomes. Did the contractor have sufficient capacity and management systems in place? How responsive was USAID? What could USAID have done differently to manage the project IV. EVALUATION METHODOLOGY The evaluation team will be required to propose a clear methodology to answer all the evaluation questions, utilizing both quantitative and qualitative methodologies such as focus groups, structured interviews and/or questionnaires, as appropriate. Preliminary analysis and review of relevant documentation will be conducted to analyze the program design, implementation mechanisms and document achievements registered. In analyzing program performance and determining the lessons and best practices, the evaluator will gather views and inputs of critical stakeholders who have been involved in the delivery and benefitted from its services. These stakeholders include but not limited to the program management team, sub-grantees, USAID, local governments, private water operators, the water management board and selected end beneficiaries of the program. Other field observations and systems check shall be conducted to ascertain the functioning of the system against national and international performance standards and use of services by beneficiaries. With regard to data quality, the evaluation team is expected to be familiar with USAID data quality standards for objectivity, validity, reliability, precision, utility and integrity and be able to 51 NUWATER End of Project Evaluation apply them in the final report, by identifying such data limitations as may exist with respect to these standards (ADS 78.3.4.2 - http://www.usaid.gov/policy/ads//500/578.pdf) and ADS 203.3.5.1- http://www.usaid.gov/policy/ads/200/203.pdf). Findings of the evaluation will be shared within USAID and other development parties in the government, donor agencies and civil society to inform better urban water management delivery in Uganda and other developing countries. V. PROGRAM INFORMATION The following information documents and sources are available and relevant to the review: USAID:  Original Request for Proposal  USAID program and financial reporting requirements NUWATER:  Agreement and other amendments/modifications  Annual and quarterly reports  Annual work plans and Performance Management Plans  Baseline survey reports  Data Quality Assessments  Relevant training and activity reports  Internal assessments and reviews  Individual contracts and agreements between USAID and sub-grantees  Other background materials such as relevant policy documents, sector strategies (Water and Sanitation sector Performance Report 2008, 9, 10; Urban Water Supply and Sanitation sector Strategy, OBA guides, etc.) VI. EVALUATION TEAM COMPOSITION The evaluation team will be comprised of two experts and a research assistant. The team will have prior experience in public-private partnerships and urban water supply projects with a focus on output-based aid and strengthening local government capacity. One staff member from USAID/Uganda will also participate. The team should possess the skills and experiences below: Team Leader  Qualifications and demonstrated experience in the design and management of evaluations  Demonstrated 5-10 years of experience with public-private partnerships in social services delivery, water and sanitation services preferable. Extensive experience in Uganda and or Africa will also be considered.  Solid understanding of the decentralized service delivery systems in Uganda.  USAID programming experience is desirable. National Expert 52 NUWATER End of Project Evaluation  Solid experience in engineering and urban water supply systems in Uganda.  Experience in working with output-based aid. Research Assistant VII. DELIVERABLES The evaluation team is expected to deliver the following outputs to USAID/Uganda: Deliverable Level of Effort Total 1. In-brief meeting for an introduction of the evaluation team, discussion of the SOW and initial presentation of the proposed evaluation work plan. 1 day x 2 persons 2 2. An inception report submitted to USAID within one week after the in-brief. The report will include:  A detailed work plan showing a timeline for each evaluation activity to be undertaken, including field work.  Methodology detailing sub-grantees and field sites to be visited, data collection instruments. 2 days x 2 persons 4 3. Field work/Data collection 7 days x 3 persons 21 4. Oral debriefing to USAID, NUWATER and selected partners to present key findings prior to submission of draft report. 1 day x 2 persons 2 5. Draft evaluation report in both hard copies (2) and one electronic copy for review by USAID. *Please see the Illustrative Report Outline at the end. Requirements for the evaluation report are also attached. 3 days x 2 persons 6 6 Dissemination meeting for program stakeholders (Attendants will be agreed upon with USAID. Tentative venue for the meeting in Northern Uganda) 2 days X 2 persons 2 6. Final evaluation report in both hard copies (5) and one electronic copy incorporating feedback from USAID. 1 day x 1 person 1 VIII. SCHEDULE The evaluation will begin on or before May 15, 2011 and will require approximately 30 working days of effort. In addition to time in the NUWATER office in Kampala, it is proposed that team members will spend time with the sub-grantee, and in Kitgum and Pader with the Town Councils and the private water operator. A draft report will be submitted to USAID prior to the departure of the evaluation team leader and a final report provided to USAID no later than June 15, 2011. IX. ROLES & RESPONSIBILITIES UMEMS: Provide quality assurance of the process and products before delivery to USAID 53 NUWATER End of Project Evaluation Select and contract the evaluators Manage the evaluation process Provide briefings to team; organize consultant participation Provide logistical support for the evaluation team including office space and transport Submit evaluation report to USAID USAID: Have a USAID staff member to participate in the evaluation Appoint a point of contact for the assignment to coordinate USAID inputs Approve the evaluation team, methodologies and work plan Participate in briefings Review inception and draft evaluation reports and provide feedback Sign off on final report NUWATER: Participate in final review of the inception, draft and final reports Provide relevant documents as needed Provide assistance with setting up meetings and interviews Partners/Sub grantees: Provide relevant documents as needed Participate in meetings and interviews as needed Other roles and responsibilities reviewed in line with the level of participation deemed as necessary. ILLUSTRATIVE REPORT OUTLINE  Cover page (Title of the study, the date of the study, recipient‘s name, name(s) of the evaluation team.  Preface or Acknowledgements (Optional)  Table of Contents  List of Acronyms  Lists of Charts, Tables or Figures [Only required in long reports that use these extensively]  Executive Summary [Stand-Alone, 1-3 pages, summary of report. This section may not contain any material not found in the main part of the report] Main Part of the Report 1. Introduction/Background and Purpose: [Overview of the final evaluation. Covers the purpose and intended audiences for the final evaluation and the key questions as identified in the SOW) 2. Study Approach and Methods: [Brief summary. Additional information, including instruments should be presented in an Annex]. 54 NUWATER End of Project Evaluation 3. Findings: [This section, organized in whatever way the team wishes, must present the basic answers to the key evaluation questions, i.e., the empirical facts and other types of evidence the study team collected including the assumptions] 4. Conclusions: [This section should present the team‘s interpretations or judgments about its findings] 5. Recommendations: [This section should make it clear what actions should be taken as a result of the study] 6. Lessons Learned: [In this section, the team should present any information that would be useful to people who are designing/manning similar or related new or on-going programs in Uganda or elsewhere. Other lessons the team derives from the study should also be presented here.] Criteria to Check the Quality of the Evaluation Report  The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  Evaluation reports shall address all evaluation questions included in the scope of work.  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the USAID technical officer.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people‘s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 55 NUWATER End of Project Evaluation APPENDIX B: NUWATER Project Context Donor interventions in the water sector of northern Uganda are challenged by several factors likely unknown to remote planners. Donors and implementers must be cognizant of several bureaucratic, institutional, historical, cultural and social settings and culture of dependency and constraints therein to best serve donor intentions. These are discussed below as background. A good understanding of the context of the project area is especially important in order to achieve what development agencies often see as their ultimate goal; sustainability. Even if sustainability in itself is almost impossible to achieve through a three-year intervention, there are clearly factors that contribute to a higher impact amongst beneficiaries, as well as greater independence and long-term effects in form of continuous water supply. A better understanding of the local context can also improve project implementation through more realistic planning; the ability to know which activities should happen before others, and how to mitigate risks. There are several aspects peculiar to Uganda and Northern Uganda in that are important to take into account when reading this report. Socio-Economic Factors Uganda is a fairly stable country in East Africa with an average growth rate of 5.2% per year (in 2010) and $1,300 (or 1300 U.S. dollars, USD) per capita (CIA World Fact Book: DATE). However, the North of Uganda suffered an insurgency that lasted almost for two decades, and the situation only started to improve in 2005. From the early 1990s almost the entire population of Acholiland, the home of the main ethnic group of Northern Uganda, was concentrated in camps due to the brutal killing, raping and abductions carried out by the rebel group called the Lord Resistance Army (LRA). The NUWATER project area lies in the heart of Acholi, and Kitgum and especially Pader were the scenes of some of the most brutal massacres and attacks. At its peak, more than 2 million people lived in so-called IDP camps (Internally Displaced Persons). Many IDPs moved to the towns of Kitgum and Pader for security, and the populations of both towns were more than double in 2007 compared to today (USAID, 2008). Because of this situation, the economic development enjoyed by people in the South was not felt in the North. The northern region still lags behind on all development indicators such as education, access to health care and access to water and sanitation. In practice, this translates into extreme poverty and numerous vulnerable groups such as disabled and mutilated people, child￾headed households, orphans, women-headed households and people with mental illnesses and trauma. It is always difficult to assess the level of income in a society, but the average person in Pader still lives on less than one USD a day in cash, although food for consumption is mostly grown in the household field. A businessman or woman running a shop would typically earn between 20 and 30 USD per month in Pader, maybe as much as 50 USD per month in Kitgum6 . A District Water Engineer has a salary of less than 300 USD per month. It is important to keep this in mind in order to understand why replacing a tap that costs 12 USD or an overnight trip from Pader for a meeting in Kitgum or Gulu is not a simple thing. In addition, paying for water 6 The baseline survey (ARD February 2010a) commissioned by NUWATER indicates that 70% of people in Kitgum Town Council live on more than 100,000 shillings a month, something that is not likely to be the case in practice according to the authors and several interviewees. 56 NUWATER End of Project Evaluation quickly eats up the available cash the household has to cover cash expenses such as school fees and medical fees. Cultural Factors The long insurgency and dependency on foreign aid organizations handing out food, water and medicines has also had a great cultural impact. Some people have been in camps without the possibility to work in their fields or to have a job for decades. The life in the camps has made the people passive and unwilling to take action, even if it is to improve their own living conditions. Free handouts have created a culture of dependency. Many people are not willing to pay or work for basic necessities anymore. In addition, the communities are heterogeneous, because people from many different villages came together in the camps. This creates a lack of common community feeling and community organization that is necessary to mobilize people to work for the common good. Another factor is that the Acholi largely feel neglected by the central government. This is because government soldiers failed to give them protection when the LRA carried out their raids. Some historians even claim the government was intentionally prolonging the war for political reasons (Green, 2008). Until today, the Ugandan army has failed to defeat the LRA which is still active in the Central African Republic and the Democratic Republic of the Congo. Northern Ugandans and IDP are therefore suspicious of institutions that represent the central government, such as District Local Governments (DLG) and Town Councils (TC). All this translates into a reluctance to be involved in community projects such as water, and even encourages vandalism and sabotage. In order to change these attitudes so that people can truly benefit from a project and also participate in its success in the long term, it is necessary to carry out specific activities to address this, such as repeated community mobilization activities with follow up and positive results. In addition, it is necessary to improve communication and transparency between consumers and the government, and between consumers and the water management entity. Governance Factors Uganda has a reputation for corruption. According to Transparency International‘s, Corruption Perception Index 2010 Results, Uganda scores a 2.5 value where 9.0-10 is Very Clean and 0-0.9 is Highly Corrupt. Neighboring states score 1.6 for Sudan, 2 for Congo, 2.1 for Kenya, 2.7 for Ethiopia and Tanzania, and 4 for Rwanda. By the same report, Uganda ranks no. 127 out of 178 countries; neighborhood rankings were 127 for Sudan, 164 for Congo, 154 for Kenya, 116 for Ethiopia and Tanzania, and 66 for Rwanda (Transparency International, 2010a). Looking at trends, corruption seems to increase every year. When asked in 2010 how the level of corruption has changed in the last three years, 67% of Ugandans said it had increased (Transparency International, 2010b). In reality, corruption is present in all government agencies, as well as in the private sector. It comes in forms of grand corruption, fraud, misuse of money, abuse of office, bribes, petty corruption and collusion. These practices add on to the inequality in the society, and also fuel the mistrust between the population and the government. A corrupt police and juridical system leads to the fact that impunity is common, especially amongst the big perpetrators. Occasionally, officials are indicted such as the former Town Clerk of Kitgum and the former Water Officer of 57 NUWATER End of Project Evaluation Pader District, but this does not mean that they are convicted. It is not common for external agencies to be expected to take part in these practices, though, because their strict due diligence policies are normally known. However, a donor agency can easily experience delays in administrative procedures that could have been speeded up with a bribe. Most of the corruption taking place on local government level is in the forms of misuse of government funds and accepting bribes. Procurement processes are specifically at risk, and paying 10% of the contract value as a kickback to obtain the contract is common (WIN-S, 2009). It is therefore fair to expect that any additional source of money such as revenues from water users could represent a temptation for underpaid public officials. Grand corruption and collusion is more common on central government level, and the sums of money involved are much larger. In addition, numerous powerful government officials also abuse their office by owning private companies that are awarded government contracts. The selection of private operators for small towns has been identified as a process with high risks of corruption, and in a survey carried out in 2009, 51% of respondents said that political interference was very common in the selection of Private Operators (WIN-S, 2009). Being aware of this culture is important in order to avoid situations where donor funds can be at risk, as well as to train people in sound financial management providing systems and procedures that allow for transparency and checks and balances. Urban and Rural Distinctions The distinction between urban and rural in Uganda can often be fluid. According to the official definition, a place with less than 500 people is considered rural, a place with more than 5,000 people is considered urban and places with populations between 500 and 5,000 are considered Rural Growth Centers (RGC). These are in the transition phase between a village and a town. Small towns have populations up to 15,000 (GOU, 2007). However, a centre with only 5,000 people can hardly be considered urban compared to typical definitions used in the developed world. Small-town TCs rarely assume all the responsibilities a ―town‖ would commonly assume such as responsibility for service provision. In addition, different factors contribute to a gap between theory and practice. One is particular to the North, and is because most small towns were also IDP camps. For example, Pader town had more than 20,000 inhabitants only seven years ago. Now the population is likely to be around 12,000. The second is that RGCs are often made ―TCs‖ for political reasons. Politicians lobby for a place to become a town because this entails a budget and the possibility to hire town officials, and it also has to do with pride. Therefore, several places that should not have been classified as a town are actually towns. Pader is a good example. It was made a town in 2001. However it does not even have a completed town administration building. The town administration‘s capacity is extremely weak. The centre does not have a single paved road. Pader people are mostly living as in a rural trading centre. Kitgum, on the other hand, was founded around 1920 and has proper administration buildings and facilities, as well as a functioning administration. 58 NUWATER End of Project Evaluation The two centers of Kitgum and Pader are therefore very different from each other. This translates into different needs. In a town, people know that they have to pay for water and customer relations campaigns about payments and meter readings might be enough to give them sufficient information to become good customers. In an RGC, the population first has to be sensitized on the purpose of the water system, the reason why buying water is better for their health than walking to the next stream, what happens to the money they pay and why it is necessary to pay for fuel and electricity to make water available. In addition, it is necessary to mobilize the community and create a sense of ownership of the program, so that vandalism is avoided and the water managers have sufficient and support in the community in order to carry out their role. The Water Sector The water sector in Uganda is managed by the Ministry of Water and Environment (MWE) through the Directorate of Water Development (DWD). The sector has a Sector-Wide Approach (SWAP): MWE expects donors to put money in a basket fund which is distributed to local governments through the central government and spent on projects following agreed priorities and objectives. However, some donors and NGOs prefer to carry out interventions directly. In the rural water sector, money is channeled from the central government to the District Water Offices that are responsible for implementation. Rural water supplies are normally boreholes or shallow wells with hand pumps, protected springs or small gravity flow schemes or piped water schemes. In the urban sector, operation and maintenance was handed over to the private sector in 2001. The national water parastatal, National Water and Sewerage Corporation (NWSC) manages the 23 biggest towns (GOU, 2010). Around 80 other small towns hire their own private operators. In order for a TC to contract a private water operator, the Ministry needs to ―gazette‖ the town into a Water Authority. This gives the TC (or the Sub-County) the authority to contract a private sector operator and take over ownership of the assets of the water system on behalf of the central government. The Ministry signs a performance agreement with the Water Authority. Thereafter, the Water Authority signs a management contract with the private operator. The contract is supervised by a Water Board, which consists of the Town Clerk (secretary), the Mayor and three board members selected in the community to represent the water users, out of which one is the Chairman. Being gazetted also has financial benefits, because the TC will benefit from a Conditional Grant to help with operations and maintenance. According to the Ministry, about 70% of the private operators are ―breaking even‖ in their operations. However many water systems are too expensive to operate to be profitable. ―Breaking even‖ means the operators are covering their operating expenses such as fuel, power, chemicals, salaries and small repairs. However the responsibility for large repairs, replacements and emergency breakdowns still theoretically lies with the Ministry. The margins are low, and some operators operate several towns in the same area in order to benefit from economies of scale. Under a new Output-Based-Aid (OBA) program funded by GPOBA (Global Partnership on Output Based Aid), subsidies are provided to 11 small towns (Azuba et al, 2010). Slowly, the Ministry is starting to realize that operating water systems is not 59 NUWATER End of Project Evaluation very attractive for a private company, especially because users are not willing to pay and assets are often in a poor condition and need frequent repair. The responsibility for repairs is often not clear. In addition, the private operator has to apply for financial support from the central government, something that takes time and is unreliable. The incentive-based contract designed by the USAID project is not so different from the OBA program. However, it is unlikely that a system can be ―break even‖ or become profitable after only three years. The OBA program recently agreed to extend its contracts to 7 and some even to 10 years. Many un-gazetted RGCs and small towns have simple piped water systems, often only with a few dozen connections and a few public stand posts that require some kind of management. Rather than find and contract private operators, these RGCs normally train local people to manage the system on a profitable basis, under supervision of a Water Committee or a Water Board. The small companies are often not registered and operate informally. They only have a few employees. However, it is more realistic that local people manage the water systems in small towns where the consumer base is not enough to create sufficient revenue for a private sector company to make profits. The piped water schemes have their umbrella organizations in Western, Mid-Western and Eastern Uganda (GOU, 2010) to provide technical and management support. For a town like Pader, this would have been a much more viable option. 60 NUWATER End of Project Evaluation APPENDIX C: Study Approach and Evaluation Methodology The NUWATER evaluation study approach and methods were defined in the TMG technical proposal to USAID and the evaluation team‘s inception report. The study approach and methodology are key important aspects of the study to ensure that the evaluation adequately responds to the key evaluation questions in the Statement of Work (SOW), and to clearly bring out information that meets USAID‘s management needs for decision-making and the Agency‘s data quality standards for; validity, integrity, precision, reliability and timeliness (ADS 203) and, objectivity and utility (ADS 578) as applicable for evaluations. The Four Key Evaluation Questions in the SOW were: 1. To what extent did the project meet its goal of improving access to water in Kitgum and Pader? 2. How realistic and appropriate was the design of the project? 3. Were the appropriate stakeholders involved in the program and to what extent did the program promote better coordination and collaboration amongst them? 4. Analyze the effectiveness of the program management and its effect on the program outcomes. To help the data collection for the key questions and to focus the research, 29 sub-questions were added: 1. Has access to water generally improved for the population in Kitgum and Pader? 2. Were there clear performance indicators and targets well specified in the PMP? 3. What are the factors that have contributed to the success of the initiative? 4. To what extent can the progress made thus far be attributed to the program‘s intervention rather than external factors? 5. How does the current water supply situation compare to what it was prior to activity implementation? 6. What are the challenges hindering the achievement of the results? 7. What mechanisms and institutions are in place to guarantee sustainability of the services provided and results achieved thus far? 8. Were the two key assumptions – a functioning water system and a private sector interested in bidding for contracts - true? 9. What informed the original conception and design of this project and how responsive was the design was to the need at that time? 10. How did the design of the project influence the results? 11. Was project implementation appropriate and in line with the priority needs of all stakeholders? 12. Assess the extent to which the development hypothesis within the project worked. 13. Did the incentives in the contract really serve as incentives to the private operator? 14. If no, what should be done differently? 15. Were the appropriate stakeholders and their various needs well identified? 16. What was the level of stakeholder involvement in managing the project? 17. Assess the extent to which the project addressed gender concerns in the management and beneficiaries of the project. 18. How best should stakeholders be involved in such projects? 19. How capacity building activities were carried out and were they effective? 61 NUWATER End of Project Evaluation 20. Was the project well managed by NUWATER? 21. Were sufficient funds for project implementation available? 22. Were implementation procedures followed? 23. Were activities implemented in a timely manner? 24. Did the staff have sufficient sector knowledge? 25. Did the project staff have capacity to track the intended results? 26. Assess the extent to which project outcomes were affected by the capacity of the project staff. 27. How responsive was USAID? 28. What could USAID have done differently to manage the project? 29. Was the NUWATER management done in a way to enhance sustainability and build capacity for the future? Methodology The Evaluation Team relied upon both primary and secondary data sources. Primary Sources: The Team obtained useful data about the project through Key Informants (KIs) which included; USAID/Uganda Mission team that was supervising the project, National Government of Uganda (GoU) line Ministry and its Departments (Ministry of Water and Environment - Directorate of Water Development (MWE-DWD), Water Supply Development Facility–North (WSDF-N) and National Water and Sewerage Corporation (NWSC)), Local Governments (District and Town Councils) of Kitgum and Pader. Other KIs included NUWATER management team, Urban Water Boards of Kitgum and Pader Town Councils, the Private Water Operators & Water Consumers of the two towns. The primary data was collected from 46 stakeholder respondents at the various levels and a total of 123 respondents from water users as beneficiaries of the project. The list of people interviewed is attached as Appendix D. The evaluation team targeted three categories of key informants, the first comprised of the implementers of the project and policy makers, and these included USAID and its Contractor (NUWATER management), GoU-Ministry and its departments and the Urban Water Boards. The second category comprised the water consumers and private water operator. The third category was the individuals and officials from private consulting firms that the Evaluation Team considered would provide technical insight into the NUWATER project, and advice on best practices in utility management. Secondary Sources The evaluation relied on two classes of secondary data. First was NUWATER Project-generated literature. This included the project documents, baseline survey reports, annual reports, two versions of the Performance Monitoring Plan (PMPs) and monthly and quarterly M&E reports from NWSC/ESU. The second was literature about water sector in Uganda. A total of about 86 reports and other documents were reviewed. Details of the literature reviewed are included in this report as Appendix P. 62 NUWATER End of Project Evaluation Data Collection Methods Four methods of data extraction were employed, field observations, desk study of documents, in￾depth interviews with Key Informants (KIs) and Focus Group Discussions (FGDs). The Evaluation Team developed illustrative questions and shared with USAID/ Uganda at the inception report, for semi-structured confidential interviews with stakeholders and for public and open focus group discussions with the beneficiaries. These were then incorporated into the primary data collection and evaluation tools. In addition to the methods above, the Evaluation Team conducted field visits of in both Kitgum and Pader to establish the status of the project water infrastructure, confirm access and availability of water to the community and to get a better understanding of the overall project operations and its challenges. Analysis The Evaluation Team is aware of the difficulty in obtaining quantitative data from an evaluation like this; however some statistical analysis was undertaken in order to summarize the findings despite the small sample. However, the Team decided it gives the reader a quick way to appreciate the general impression that was gathered from the data. For the four main evaluation questions, the Team attempted to get yes/no answers from stakeholders. However, some of the questions are broad, and for example on question 3, ―yes but no‖ was the most common answer and this is therefore the answer presented in the conclusions. When yes/no answers were asked for, the evaluation team made sure the respondent had a good understanding of the details of the question (for example, the assumptions behind the design) before an answer was given. Limited statistics on the yes/no questions are given in the findings. People that had no knowledge of the topic considered in the question were rated as ―no comment‖. Consumers were only asked question 1, since it was evident they would not have sufficient knowledge to answer the other three questions in a sensible way. The answers to the 29 sub-questions were tabulated in an analysis matrix to be found in Appendix L. Answers were grouped in seven categories: USAID, NUWATER, MWE/NWSC/DWD respondents, Local Government, Water Board, Private Operator, Kiosk and Pump attendants and Water Consumers. This matrix was used as a basis for the findings section in the report. Where possible, statistics have been given for answers to specific sub-questions. However, since not all questions were asked to each respondent, they are only used to provide evidence for the data and not for comparison. The Evaluation Team always tried to triangulate the information using different methods of data collection and confirmation. For example, some allegations were confirmed through field inspections of infrastructure where possible. Others were confirmed by going through monthly reports in detail. Allegations or information got from only one source have not been considered valid for a finding in this study, but might have been used as a quote or to illustrate another point. Intended Audience 63 NUWATER End of Project Evaluation The Evaluation Report provides an independent assessment of the results achieved with implementation of the NUWATER project. The Evaluation Team identified key program successes and challenges, including opportunities missed or activities that fell short of expectations. This information will be used to inform future USAID/Uganda in programming and designing its future projects in the water sector. Specifically, the evaluation will: a) Provide lessons learned for USAID, and, the Government of Uganda (GOU), and other development partners supporting the water sector, b) Assess the existing and/or potential ability of key successes to be replicated, c) Uphold an institutional commitment of measuring program results and, d) Provide practical lessons for current and future water sector partners in developing and implementing sustainable water supply systems. Study Limitations The main limitations to the study have been time and other resources. It is was not within the scope of this evaluation to carry out detailed interviews with large numbers of beneficiaries, which would have been necessary to capture the impact of the project on different sample groups such as by geographic location, gender, socio-economic status and vulnerability. The sampling method for the focus group was completely random and was based on the interest of customers to show up; therefore the outcomes can be somehow biased. There was also little time to assess the sustainability of the projects more in detail, because technical tests of equipment and capacity tests of stakeholders were not carried out. Several incidents during the evaluation of the project revealed an iceberg of lack of proper communication and coordination, which could not be fully investigated due to the constraints mentioned above. The Evaluation Team is aware of the difficulty in drawing statistical conclusions from a small sample which was unevenly distributed over the different stakeholders, however, as mentioned above, the aim of statistical information is not comparison but for the reader to get a quicker grasp of the general trend in the responses. The Evaluation Team experienced some constraints during the evaluation that impacted on timeframes and access to information. Friday June 3, 2011 was a public holiday in Uganda (Uganda Martyrs Day). According to the field time schedule, the Evaluation Team was meant to conduct interviews in Pader on that day, but that was not possible. As a result the team was forced to conduct several one-on-one KI interviews to catch up with the time schedule, which was rather tiring. Power cuts and poor internet access and telephone network in Kitgum and Pader caused communication challenges amongst evaluation team members and KIs. For example, the network coverage for Warid Telecom (a telecommunication service provider in Uganda) network had not reached Kitgum and Pader. This forced the evaluation Team Leader to share with the Research Assistant the Orange (another telecommunication service provider) internet modem which UMEMS had provided to the Team Leader. The same reasons also forced Evaluation Team to travel back at forth to Gulu town where they could get better access to communication facilities. Travel distances between districts, road conditions and communication with KIs was very challenging. 64 NUWATER End of Project Evaluation APPENDIX D: List of Stakeholders Interviewed Date Institution Sn Name Title Contact Information Place and City of Interview May 23, 2011 UMEMS 1 Stanley Golooba Monitoring & Evaluation Specialist slukenge@ugandamems.com, 0772 413 781 UMEMS, Kampala UMEMS 2 Nestore O. Jalobo Operations Manager njalobo@ugandamems.com, 0772 482 035 UMEMS, Kampala UMEMS 3 Ms. Patricia V. Rainey UMEMS Chief of Party pvrainey@ugandamems.com, 0777 564 215 UMEMS, Kampala May 24, 2011 NUWATER 4 Alioune Fall NUWATER Chief of Party afall@northernugandawater.com, 0772 704 590 UMEMS, Kampala May 26, 2011 NUWATER, Kampala 5 Ms. Leila Mbabazi Project Assistant (Financial) lmbabazi@northernugandawater.c om, llmbabazi@gmail.com, 0772 835 144 NUWATER, Kampala May 27, 2011 Ministry of Water and Environment, DWD 6 Chris Henry Azuba Assistant Commissioner, Urban Water & Sewerage Department chris.azuba@mwe.go.ug, 0772 498 330 Luzira, Kampala USAID/Uganda 7 Sudi Bamulesewa former USAID COTR sbamulesewa@usaid.gov, 256-31-387387 Golf Course Hotel, Kampala USAID/Uganda (also May 10, 2011) 8 Ms. Jenna Diallo NUWATER COTR jdiallo@usaid.gov, 0772 221 669 Golf Course Hotel, Kampala May 30, 2011 NUWATER, Kitgum 9 Sam Otedor NUWATER Field Coordinator 0717 326 293 0712 625 121 Bomah Hotel, Kitgum Kitgum Town Council 10 Richard Ojara Okwera Town Mayor 0772 355 628 Kitgum TC Offices, Kitgum May 31, 2011 Kitgum DLG 11 George Ocaya Procurement Officer 0772 859 623, 0791 855 022 gocaya679@gmail.com Kitgum DLG Offices, Kitgum Kitgum DLG 12 Eugene Oola District Planner 0772 358 696 Kitgum DLG 13 Ms. Rhoda Oroma Former Acting Town Clerk (Now Assistant CAO) Kitgum DLG Offices, Kitgum Kitgum Town Council 14 Ms. Concy Ajok Water Board Member 0782 368 130 Kitgum TC Offices, Kitgum WASH Consults Ltd. 15 Denis Lawoko Managing Director 0772 584 782 WASH Offices, Kitgum Kitgum Town Council 16 Michael Wokorach Acting Town Clerk 0775 115 117 Kitgum TC Offices, Kitgum 65 NUWATER End of Project Evaluation Date Institution Sn Name Title Contact Information Place and City of Interview Kitgum Town Council 17 Benson Atube Acting Urban Water Officer 0772 321 335 Kitgum TC Offices, Kitgum Kitgum Town Council 18 Alexis Alfred Abonga Senior Assistant Engineering Officer 0791 868 340 0772 879 110 Kitgum TC Offices, Kitgum WASH Consult Ltd. 19 Peter Owot Finance & Administration Manager 0792 661 192 WASH Offices, Kitgum Kitgum Town Council 20 Wilfred Ocora Chairperson Water Board 0772 886 048 Kitgum TC Offices, Kitgum Kitgum Town Council 21 Martin Anyalla Water Board Member 0772 916 809 Kitgum TC Offices, Kitgum Kitgum Town Council 22 Samuel Owen Ocaya Town Engineer 0791 836 654 Kitgum TC Offices, Kitgum Pader Town Council 23 Richard Matuwa WSDF-N Engineer 0782 397 860 Pader, Phone Interview June 1, 2011 Pader Town Council 24 Solomon JD Sally Commercial Supervisor/Town Agent 0777 004 973 Pader TC Offices, Pader Pader Town Council 25 Joseph Omona Lapit Current Town Council Mayor 0773 232 145 0757 267 239 Pader TC Offices, Pader Pader Town Council 26 Augustine Orach Lojimoy Deputy Mayor 0777 190 726 Pader TC Offices, Pader Pader Town Council 27 Christopher James Oketayot Town Clerk 0772 461 866 Pader TC Offices, Pader Pader Town Council 28 Ojor Otto JRR Chairman Water Board Pader TC Offices, Pader Pader Town Council 29 Rev. Robin Oginga Odinga Water Board Member 0772 351 406 Pader TC Offices, Pader Pader DLG 30 Charles Otai Chief Adm. Officer (CAO) 0772 640 638 Pader DLG Offices, Pader Pader DLG 31 Samuel Obolo Komakech Head of Procurement & Disposal Unit 0711 231 811 Pader DLG Offices, Pader Ministry of Water & Environment - DWD 32 Robert A. K. Mulema Water & Environment Officer; Focal Point Officer - TSU - 2 0772 555 906 0702 124 039 Pader DLG Offices, Pader Warner Consulting 33 John Charles Okunya Engineer Pader District Local Government 34 Charles Obali Acting Water Officer 0392 948 332 Pader DLG Offices, Pader June 2, 2011 Pader Town Council 35 Patrick Olanya Town Engineer 0713 983 093 Pader TC Offices, Pader WASH Consults Ltd 36 John Paul Okeny Technical Supervisor 0782 341 404 WASH Offices, Kitgum 66 NUWATER End of Project Evaluation Date Institution Sn Name Title Contact Information Place and City of Interview June 6, 2011 Pader Town Council 37 Ms. Akongo Stella Human Resource 0774 299 880 Oasis, Pader Town June 8, 2011 USAID/Gulu Office 38 Gerald Komakech USAID Engineer 0772 221 660 Chobe Safari Lodge, Karuma - Phone interview WASH Consults Ltd 39 Issac Mwaka Former Technical Director waterengineer.uganda@gmail.co m, 0755 029738 Chobe Safari Lodge, Karuma - Phone & email interview USAID/Gulu Office 40 David Mutazindwa Program Management Specialist, Economic Growth Team (SO7) 0772 221 675 Golf Course Hotel, Kampala - Phone Interview Pending, June 15, 2011 DWD 41 Sottey Bamukama Director 0414 5059 50 Kampala June 15, 2011 DWD 42 Chris Tumusiine Principal Engineer, Planning and Development, Urban Water Supplies June 16, 2011 NWSC ESU 43 Ms. Rose C. Kaggwa Senior Manager, External Services, Project Administrator for NUWATER rose.kaggwa@nwsc.co.ug, 0772 425 019 Kampala 44 Charles Odonga Consultant, Project Manager for NUWATER odonga@gmail.com, 0752 554 760 Kampala 45 Jude Mwoga Contract and Incentive Design Expert, Contract Manager for NUWATER Kampala 46 Ambrose Olaa SO9 Democracy and Governance Officer/ Security and Conflict Officer 0772 221 690 Gulu 47 Silver Mughisha Chief Manager, Institutional Development and External Services, Deputy Project Manager for NUWATER silver.mugisha@nwsc.co.ug, 0717 315 109 Kampala 67 NUWATER End of Project Evaluation Acrynoms CAO - Chief Administrative Officer DWD - Ministry of Water and Environment, Directorate for Water Development NUWATER - Northern Uganda Water Services NWSC - National Water and Sewerage Corporate SO - USAID Strategic Objective TC - Town Council TMG - The Mitchell Group UMEMS - TMG‘s Uganda Monitoring and Evaluation Management Services Project WSDF-N - Water Supply Development Facility, North 68 NUWATER End of Project Evaluation APPENDIX E: Questionnaire for Stakeholders NUWATER Project Evaluation Questions for USAID 1. How was the design of the project made, what kind of study took place? What was the process? 2. Is a USAID representative going to work with us on this project, and is the representative going to join us to the field? 3. How do our research questions relate to the indicators in the PMP? 4. Should we include beneficiaries/water users in our research? How detailed do you expect us to present this information? 5. Why was the debriefing workshop requested? Is there any point of carrying this out, knowing the project has ended by that time and USAID is not doing any more infrastructure projects in water? Is there another way of doing this? 6. What were the qualifications of the USAID COTRs? Did they have any experience in water utilities management/water engineering/private sector involvement? Questions for COTR (present and former) 1. Please explain how the subsidies work and how they are calculated, and how the subsidy reduction indicator is calculated. 2. Please explain the bonus incentive system. Is it true that the operators have never qualified for a bonus payment? 3. What is the cost recovery ratio? Why has this indicator been neglected? 4. What is the status on the OBA arrangement, have any connections been subsidised in this way? 5. What were the criteria used to award contracts and what was USAID/ NUWATER‘s role in overseeing the process? 6. Was the project design appropriate? 7. What are the main reasons for the differences between the design of the project and the implementation? 69 NUWATER End of Project Evaluation 8. Were there any formal changes in the contract (Section C)? 9. How was the cooperation with NUWATER? 10. Did NUWATER have sufficient capacity to carry out this contract? 11. Do you think the projects has improved the water access for the population in Kitgum and Pader? 12. How was the decision made to end the NUWATER contract? What was the basis? 13. What is USAID‘s intention for future upgrades to Kitgum and Pader water systems? 14. What is the relationship between NWSC and USAID? Questions for NUWATER 1. In your view, were all the appropriate stakeholders involved in this project? What was the level of stakeholder involvement? What did NUWATER do to improve the cooperation of the stakeholders of the project? 2. What was the responsibility of central/district government and did they comply? 3. Was there any involvement of NWSC? 4. How is the capacity building carried out? (of stakeholders, private operator) are there any detailed reports on this? Do you have any documentation on whether the knowledge has been internalised? 5. In training – did the contractor give pre-tests and post-tests? 6. Did the contractor provide Operations and Maintenance manuals, checklists, guidelines? 7. What institutions/mechanisms have been put in place to ensure sustainability of services provided? 8. How were the public outreach campaigns conducted? Did you see any results? 9. Comment about how gender concerns were addressed in the project. 10. How were the private operator contracts advertised (wording)? 11. What was NUWATER‘s role in overseeing the contract awarding processes? 12. Why did the Pader activities start so late? Explain any reasons for delay in project implementation for Kitgum/ Pader Districts. 13. What is the exact status on Pader water system today (infrastructure and management)? 14. Please explain how the subsidies work and how they are calculated, and how the subsidy reduction indicator is calculated. 70 NUWATER End of Project Evaluation 15. Please explain the bonus incentive system. Is it true that the operators have never qualified for a bonus payment? To what extent did ‗incentives‘ as per original design provide incentives in the project implementation? Is there anything that you feel could have been done differently? 16. What is the status on the OBA arrangement, have any connections been subsidised in this way? 17. Why are the bulk water meters still not operating, yet this problem was early detected? 18. Are there still infrastructure projects that have not yet been finalised? 19. In your view, were the indicators specified in the PMP clearly spelt out? 20. Why was not data given in year 1? 21. What data was collected during the audits, and why were no audits done in year 1? 22. Is it possible to get disaggregated data for Pader and Kitgum on the key indicators (Excel table)? 23. Why was M&E information not shared on in a timely manner? 24. What M&E model was proposed by ARD? 25. Why did NUWATER not hire an M&E specialist like recommended by UMEMS? 26. Why was the baseline study carried out so late? Why was there a discrepancy between the baseline survey and the PMP? 27. Was there a performance indicator such as cost recovery ratio for financial performance? 28. What are the qualifications of NUWATER in water engineering and water utilities management? 29. Who gave NUWATER technical support on the water engineering subjects? 30. Was a technical assessment carried out of the water systems? 31. Who gave NUWATER technical support on water utilities management? 32. In your view, are all project staff having appropriately qualifications and experience to carry out their specific duties and responsibilities. 33. Was the project design appropriate? 34. Are there any specific challenges encountered during the project implementation? 35. What are the main reasons for the differences between the design of the project and the implementation? Were there any modifications in the project designs, and how were these 71 NUWATER End of Project Evaluation modifications carried out? 36. Were there any formal changes in the contract (Section C)? 37. What is your comment about the basis upon which the original project concept was designed, especially when compared to the current outcome/results. 38. How was the management of the project by USAID? Did they provide sufficient oversight and support? Were there any problems in the cooperation? 39. What could USAID have done differently to manage the project? 40. Do the private operators have sufficient incentives to run the systems on a sustainable basis? Why/why not? 41. What is your comment about the success of NUWATER 42. Do you think the projects has improved the water access for the population in Kitgum and Pader? What is your comparison of the current access to water situation with that before NUWATER project 43. What would you have done differently if you could change the project design? 44. What NUWATER staff can we talk to and where are they? 45. Which key informants should we talk to and what are their contacts? 46. What are the contact persons in DWD that we should talk to? Documents missing:  Reports from capacity building activities/meetings (for improved cooperation) etc.  Contracts between the water operator and the water authority/NUWATER  Recent addenda to the contracts with the water operator  MOUs between districts/NUWATER, MWE/NUWATER etc.  Early assessment done deciding to exclude Aloi Questions for DWD 1. What are the responsibilities of the private operator, and what are the responsibilities of government (local and central)? 2. Who is responsible for capital investment? 3. Who is responsible for replacement (in case of large breakdowns)? 4. Are the operations of other small towns water systems in Uganda subsidised? 72 NUWATER End of Project Evaluation 5. Why are there so few companies that are interested in managing small towns and why did so few qualify? 6. What is the common cost recovery ratio for small towns? 7. What are the limitations of the contracts, what are the possibilities of the private operators to do other business and services to complement on the revenue from water sales? 8. How is the risk shared between the private operator and the owner of the system? 9. How long are the contracts normally? 10. How were you involved in this project? 11. How was the cooperation with USAID and NUWATER? 12. Why do you think the private operators are struggling to meet their targets? 13. Why do you think no private operator qualified to be contracted for Pader? 14. Do you think the project has improved access to water for the population in Kitgum and Pader? 15. Are there similar experiences to Kitgum and Pader elsewhere in Northern Uganda? 16. Do you think the project was adequately managed by NUWATER and USAID, what could have been done better? 17. Do you think the officials in USAID and NUWATER had the sufficient capacity to manage this project? 18. What would you recommend as the way forward Questions for NWSC 1. What was your involvement in the NUWATER project, and how did the cooperation start? 2. What were your responsibilities in the project? How often did you visit the project sites? 3. In your opinion, was the project design adequate? 4. What was the impact of the project on the access to water in Pader and Kitgum in your opinion? 5. Did you encounter any challenges during your involvement with the NUWATER project? 6. Was the project managed well? 7. What are your comments on the capacity of the private operator? 8. Please comment on the sustainability of the projects 9. When did your involvement end and why? 73 NUWATER End of Project Evaluation Questions for Local Government Officials/ Stakeholders 1. What was your involvement in the NUWATER project? 2. What are the achievements of the NUWATER project in your opinion? 3. Are you aware of any problems concerning the NUWATER project? 4. Is the water supply service adequate in the town today? Why/why not? 5. Has the access to water improved in the town compared to how it was before? 6. What did you learn from the NUWATER project? 7. Did NUWATER handle the procurement processes adequately? Why do you think so? 8. How were gender issues handled by the project? 9. Generally, what are your suggestions for future improvement? Questions for Private Operators 1. How did you first get involved with the NUWATER project? 2. Was the bidding and procurement process carried out in an appropriate way? 3. What are the main advantages of the contract? 4. Are there any problems that you have experienced with the contract? 5. What are your main challenges in running the water system? 6. How was your cooperation with the water board/town council/NUWATER/USAID/NWSC? 7. Would you like to renew the contract when it runs out? Why? 8. Where do you see this project in the next 5-10 years? 9. What capacity building interventions/plans have you carried out/put in place? 10. How were gender issues handled by the project? 11. What are your suggestions for future improvement? 74 NUWATER End of Project Evaluation APPENDIX F: Focus Group Questionnaire for Water Consumers Question Methodology Purpose What was your water source before the project? Asked the group gauge the proportions that use boreholes/piped water/other sources To get a baseline of the situation before the project What is your water source now? Same as above To find out about the current situation so that it can be compared with the baseline. What do you know about the current management of the water system? Open discussion, writing down key words. The groups were guided with specific questions about how bills are paid, what they do if there is a problem etc. Also asked how information was obtained. To find out what the level of understanding of the management of the system is. This will give the Evaluation Team information about the quality of the customer service of the private operator/town council, and how effective public outreach campaigns were. Name the positive aspects of the project Open brainstorming, writing down key words. After, the key words were then grouped into key areas such as quality, availability etc. To get an idea of the perception of the population, and whether they feel the situation has improved compared to before. Name the negative aspects of the project Same as above If there are any complaints about the current level of service, this is where the users are able to express them. It will indicate the level of satisfaction with the current service levels What are your suggestions for improvement? Open-end probe To help gauge/obtain feedback on how things could have been done differently? 75 NUWATER End of Project Evaluation APPENDIX G1: Focus Group Water-Customer Meeting Attendance Sheet– Pader 76 NUWATER End of Project Evaluation 77 NUWATER End of Project Evaluation APPENDIX G2: Focus Group Water-Customer Meeting Attendance Sheet – Kitgum 78 NUWATER End of Project Evaluation 79 NUWATER End of Project Evaluation 80 NUWATER End of Project Evaluation APPENDIX H: Record of Focus Group Meeting with Water Customers in Kitgum On Tuesday, June 7, 2011, the NUWATER Evaluation Team conducted an open, public meeting for water customers at Kitgum Town at the New Acholi Pub from 11:30 a.m. to 1:30 p.m. In total, 81 water customers attended, consisting of 47 (58%) women and 34 (42%) men. Approximately three women brought their babies with them. Soft drinks were provided. The agenda is listed below. Agenda for Focus Group: Water Customers at Kitgum Town 1. Prayer – led by a local lady 2. Welcome and Introduction of USAID NUWATER Project Evaluation 3. Water Sources: Before and After NUWATER 4. Management of the Water System 5. Benefits 6. Problems 7. Voting and Comments 8. Closure by a lady member of the Kitgum Water Board Water Sources: Before and After NUWATER Before (out of 108 votes by show of hands)  Boreholes – 29 votes (27%)  Open shallow hand-dug wells – 4 votes (4%)  Season river water/ riverbed – 20 votes (19%)  Rain water – 28 votes (26%)  Yard taps in compounds – 9 votes (8%)  Water kiosks – 18 votes (17%)  Household connections – 0 votes After (Now – Today) out of 103 votes by show of hands. Note, the connections for attendees were for yard taps in their business/shops/housing compounds. None of the attendees said they had inside plumbing or household connections.  Boreholes – most of the people use this for drinking and cleaning water – 30 votes (29%)  Open wells – 3 votes (3%)  Yard taps in compound – drinking, unreliable, only sometimes – 11 votes (11%)  River water – washing clothes and drinking – 24 votes (23%)  Rainwater when available – 35 votes (34%)  Water kiosks – rarely have water, kiosk attendants never there, 0 votes Management of the Water System  When started, water was there, now no water 81 NUWATER End of Project Evaluation  Private operator contracted by Town Council to operate water system  Bring bill to house, go to office and pay  Made new connections  Six new connections  Don‘t tell new customers how to pay  Every month meter read  Before private operator, running by Town Council  First year, good water flow  After first year, many weeks without water  Few connections – enough water. Many connections made – not enough  Seven people heard radio announcement only  Heard from other sources never heard directly from private operator o No house-to-house  Ten complaints Received two bills, arrears not cleared Bill without water flowing Brought his neighbor‘s bill (did not respond) Paid amounts not removed from balance on account (until now not solved) Don‘t know how much a unit of water costs Don‘t understand value added tax (VAT) and service fee Don‘t tell us how meter reading is done, don‘t let us see when they do it No meter reading since 2007, pays flat rate of 10,000 UGX, complained but nothing done Tap is leaking, complained, asked to pay 15,000 UGX, doesn‘t know if it is correct Complained about poor availability, bill still the same Air makes meter run when tap is opened, still have to pay bill. Complained Even if no water still have to pay VAT and service fee Paid for connection but still not connected Benefits  Many people were connected this past year  First year, constant flow  First year, kiosks were operating, employing people  The subsidized connections were very affordable at 59,000 UGX  ―The coming of this project has made water very accessible so you don‘t have to move around looking for water‖  ―Those connected to businesses are happy with the water connections‖ - Improving business for businesses that have connected  Saves time  Reduced water-borne diseases  Easy access for disabled  It was better in the first year, but after the first year were problems 82 NUWATER End of Project Evaluation Problems  ―Kiosk attendants are never there. It becomes so unreliable it is unbelievable!‖  Water comes once a month  Not enough water storage  Water comes a few hours at night  Sometimes the water has color - brown – and can‘t be used for drinking or cooking  Very low pressure  Not enough taps  They don‘t have access  Too few pump stations now  ―The water is hard, it takes a lot of soap‖  Water kiosks not operating  ―The water is not enough everywhere‖  In Pongdwongo, no water for two years, main pipeline damaged by road grader  Water never is the whole town  Bills don‘t reflect quantity we use  Gangdyang – no water for two years  Hotels take all the water from the users – bought  Westtland: people down the slope get water, people up don‘t  Not enough pumping capacity  Don‘t tell us how to read meters  Expensive for poor people  ―In a month, sometimes we see water only once‖  How many people have no connected water? 20 votes  How many get connected water once a month? 9 votes  Every two weeks? 11 votes  Once a week? 3 votes Voting and Comments A closed paper ballot was taken for the question: Did NUWATER increase access to water in Kitgum? Participants were asked to code F for female and M for male on their ballots. They were also invited to write any lessons learned, comments and recommendations on the ballot slips. They were encouraged to help each other writing if needed, either in their local language or English. The results and comments are summarized below. Voting Results for Focus Group: Water Customers in Kitgum Town YES NO Total Male 17 14 31 Female 31 5 36 TOTAL 48 19 67 83 NUWATER End of Project Evaluation In total, 67 customers voted: 48 ―YES‖ and 19 ―NO.‖ Comments for Focus Group: Water Customers in Kitgum Town Members of Water Board should communicate with community Kitgum High School, last 2 months, client, more than 1,000 students We are very happy USAID is helping us and we need more help Closure. The lady member of the Kitgum Water Board Ms. Concy Ajok made closing remarks of gratefulness. 84 NUWATER End of Project Evaluation APPENDIX I: Record of Focus Group Meeting with Water Customers in Pader On Monday, June 6, 2011, the NUWATER Evaluation Team conducted an open, public meeting for water customers at Pader Town at the Oasis Hotel from 10 a.m. to noon. In total, 42 water customers attended, consisting of 30 (71%) women and 12 (29%) men. Approximately seven women brought their babies with them. Bottled water and soft drinks were provided. The agenda is listed below. Agenda for Focus Group: Water Customers at Pader Town 9. Prayer – led by a local lady 10. Welcome and Introduction of USAID NUWATER Project Evaluation 11. Water Sources: Before and After NUWATER 12. Management of the Water System 13. Benefits 14. Problems 15. Voting and Comments 16. Closure Water Sources: Before and After NUWATER Before  Open shallow hand-dug wells – widely distributed, polluted, usually boiled  Borehole water – few, scattered, long lines, a lot of crowding and waiting, safer than open wells  River water  Rain water After (Now – Today) out of 26 votes by show of hands  Public taps at kiosks, now first choice – 11 votes (42%)  Borehole, second choice – 6 votes (23%)  Both public tap and borehole – 8 votes (31%)  Rainwater when available – 9 votes (35%) Changes  Prefers public taps, used borehole if no tap water  Tap is faster, can take better care of baby  When there is no money to pay, use the borehole  Tap water is clean, people don‘t fall sick  Boreholes are crowded, people fight; at taps, there are no fights  Venders sell at 300 UGX, tap is only 50 UGX to fill a Jerry can  Most people get their water from both boreholes and public taps 85 NUWATER End of Project Evaluation Management of the Water System  Most people don‘t know anything about it  Paying kiosk attendants  Paying for repairs  Collections are taken to Town Council  Paying for fuel  Going to NUWATER  Money is collected by kiosks supervisor  Use of water meters Benefits  Saves time, boreholes are crowded, fast access at kiosks  Bigger water quantity  Created employment  Cheap  Availability – anytime  Enough water for hygiene  No fighting at water points  Reduced disease  Better image for town Problems  Not free  Quality – hard water, foams when boiling  Borehole water is not hard  Rigid metering/ sales – would like to be able to buy smaller quantities  No help for vulnerable people  Dirt entering pipes  Kiosk attendants are not always there when they should be  Not enough water pressure when all four taps running; when some turned off, pressure surges to remaining taps causing water loss  Taps too high for Jerry cans, causes water loss  Broken taps  No shade for kiosk attendants  No tools for cleaning kiosk for hygiene  Rude kiosk attendants  Spillage of water  Not enough kiosks to serve the whole town  Tap attendants should work in shifts so there is always someone there doing daylight  Self-closing taps are difficult to open 86 NUWATER End of Project Evaluation  Pressure creates water loss Voting and Comments A closed paper ballot was taken for the question: Did NUWATER increase access to water in Pader? Participants were asked to code F for female and M for male on their ballots. They were also invited to write any lessons learned, comments and recommendations on the ballot slips. They were encouraged to help each other writing if needed, either in their local language or English. The results and comments are summarized below. Voting Results for Focus Group: Water Customers in Pader Town YES NO Total Male 6 9 6 Female 26 1 27 TOTAL 32 1 33 In total, 33 customers voted: 32 ―YES‖ and 1 ―NO.‖ Closure The new Pader Town Mayor Joseph Omona Lapit made closing remarks of gratefulness. He asked two women participants what they learned today and confirmed the focus group meeting was helpful to the community. 87 NUWATER End of Project Evaluation APPENDIX J: Focus Group Voting and Comments – Kitgum and Pader Evaluation Question 1: Did NUWATER increase access to water in your town? (Note: Not all focus group attendees voted) VOTING & COMMENTS YES NO Pader Town Council "Because it makes it easy to access water" "If the water was free it would be good" "Water is available to the community, it saves us time to get water, the water is safe" "There is sufficient water and there is no over crowding" "Yes, but poor workers. NUWATER should improve on the supply of water because the water supply is not available all the time. They should also see into helping those who can not get money to buy, for example the poor, the sick etc" Kitgum Town Council "Because we don't have to move very far to access water. But they should increases the production level" "Because the water is not enough, and the bills are too high" "But the water is not reliable. No training for water users on how to read the meter" "The water is not adequate to sort our water problems" "Basing on my view, I see that there is going to be improvement because the start has been with serious training of the beneficiaries… I say yes" "The water is not clean and not available all the time" "Because many house holds were connected but the water flow is very poor" "Because some people get water once a month" "Because I get water twice a week" "Because there is no water in my area" "Yes, it has solved the problem of water in the community, because tap water is purified, safe for drinking and community don't struggle to get tap water" "Advantages sighted for the first year, and afterwards it is NO" "Yes, but for two years we didn't get water" "The pipe is too small. We need big pipe to be put for us so that we get more water" "Yes, but it needs improvement" "Yes, it helps the sick and the weak people" 88 NUWATER End of Project Evaluation "Yes, it improved access for the period of one year only. But the problem came after" "Even if the project has improved access to water in the community but sometimes in most cases water doesn't come" "It was in the beginning of their contract but these days there is no access of water, especially in my area…which is, coming to a month now, going without water" "It has improved because we are safe from guinea worms. But for other areas which are cut off try to improve" "Has brought us ease of access to water in the community. But they should increase production level because water is currently not enough for everyone. Some people still do want to access water in their areas, so they should also increase connections" "But there must be improvement" "But not to the standard because consumers are increasing but supply is low" "Because of new connections" "But they should improve on production; pumping points. Training the water users and more water tanks should be created" 89 NUWATER End of Project Evaluation APPENDIX K: Report on Technical Site Visits to Kitgum and Pader Water Systems Site Visits to Kitgum On 30 May 2011, the Evaluation Team conducted a field site visit to the Kitgum water infrastructure from 3:30 to 5 p.m., accompanied by Sam Otedor, NUWATER‘s Field Coordinator and Benson Atube, Kitgum Town Council‘s Acting Urban Water Officer. Neither has experience in water utility operation and maintenance, water resources engineering design, well or pipeline hydraulics or groundwater hydrology. Sam Otedor is not an experienced water engineer. He did not understand drawdown, specific capacity, sand packs, well hydraulics, booster pumps, pressure gages, water meter mechanics, why old wells produce turbidity and silt and how to prevent or remediate this, and the need to lock and secure wells, well houses and bulk water meters. He was unaware of the fire plug in the system which Benson mentioned (we were told the next day by WASH Consults‘ Managing Director Denis Lawoke there are five fire plugs in the system). Sam‘s confidence in his lack of experience and knowledge of water utility engineering is not warranted. Benson also seems not to know these elementary things. Sam indicated that the current WASH site engineer is not an experienced utility or water engineer. The team observed the following: 1) No locks on any well houses, gates to wells, gate to aboveground reservoir and ground bulk meters boxes 2) None of the three newly drilled wells have sand-pack replacement or water-level measuring ports 3) One newly drilled well was already destroyed, presumably by vandals - it had no gate or protection from vandalism 4) None of the pipe fittings we saw had pressure gauges 5) No booster pumps in the system although there are booster pumps throughout Kitgum for private wells 6) Very poor and unreliable method is being used by water-well driller to estimate well discharge with uncontrolled and non-directed blower and a 20-second timing to fill a small wash pan 7) Water-well driller apparently used coarse sand-gravel mix for well sand-pack material It is not clear to the Evaluation Team why NUWATER waited until the end of the third year of the three-year project to drill new wells. However, if and when the currently drilled three new wells are functioning and tied into a distribution system, which could take months or longer, the new pumping capacity could be significantly increased. On June 7, 2011, the Evaluation Team conducted a second field site visit to the Kitgum water infrastructure from 2 to 3 p.m., accompanied by WASH Consults‘ Managing Director Denis Lawoke and Technical Supervisor John Paul Okeny. During that visit, the Team learned that: 90 NUWATER End of Project Evaluation 1) Four of the five water kiosks are non-functioning and the other has no current water due to water rationing 2) There is one well with a pressure gage in the system but system pressure is not measured or monitored 3) The large-solar field well by the UN headquarters is locked 4) The destroyed well observed previously has been since re-drilled but not yet secured 5) Vandals had destroyed some solar photovoltaic panels which run pumps. The Team also confirmed: 6) NUWATER had not established, equipped, and staffed a Kitgum office 7) Provided little training to WASH other than introductory orientation in billing software 8) WASH did not have an O&M manual, Preventive Maintenance Plan, guidelines, checklists, work-flow-process protocols and related documents and training, not a spare parts collection, equipment warehouse or bone yard which are generally expected of utility managers. On the previous night, the Evaluation Team again visited the Kitgum elevated storage tank complex and confirmed it was still unsecured and not guarded a week after the Team pointed this out to NUWATER Field Coordinator and Kitgum Action Urban Water Officer on May 30, 2011. Site Visits to Pader The Evaluation Team conducted site visits to the Pader Water Authority systems on June 1 and 2 with Pader Water Authority representatives, none of whom are water resources or water utility engineers. The Team noticed the following: 1) Four of the five kiosk attendants and the supervisors were women 2) One of four taps at one kiosk was missing 3) Several kiosks had improvised water funnels cut from liter plastic water-bottles to reduce water spillage and wastage. 4) There was also a removed but not replaced water meter at a pump 5) A newly drilled but non-secured and un-guarded water well 6) A school latrine a few horizontal meters from a new water well 7) An associated new diesel generator with soundproofing but no manufacturer‘s manual or operator training. The soundproofing was likely installed as not to disturb the adjacent school. 8) The Team also confirmed that the Pader Water Authority did not have an O&M manual, Preventive Maintenance Plan, guidelines, checklists, work-flow-process protocols and related documents and training, not a spare parts collection, equipment warehouse or bone yard which are generally expected of utility managers. The Team was told several times that the NUWATER Field Coordinator had come to the well with the now removed water meter unannounced to remove the meter leaving the pipe from the pump disconnected from the water delivery system to the kiosks. 91 NUWATER End of Project Evaluation APPENDIX L: Evaluation Findings Summary EVALUATION QUESTIONS RESPONSES/FINDINGS 1. Q1. To what extent did the project meet its overall goal of improving access to water in Kitgum and Pader? USAID NUWATER MANAGEMENT MWE/DWD/NWSC OFFICIALS LOCAL GOVERNMENT OFFICIALS WATER BOARD MEMBERS PRIVATER WATER OPERATOR, KIOSK & PUMP ATTENDANTS WATER CONSUMERS 1. Has access to water generally improved for the population in Kitgum and Pader? 80% of USAID staff that the Evaluation Team interviewed all said NUWATER project produced more water but not sufficient for the towns; With some little capital investment, NUWATER increased water in the two towns. The evaluation Team discovered that water supply in the systems increased towards the end of project period. More wells were drilled in Kitgum, but not connected to the system by the time of evaluation. All NUWATER staff interviewed (3 out of 3) agreed that access to water has improved considerably. They noted production level as the major challenge in achieving the goal of NUWATER project. 50% of NWSC staff interviewed disagreed that NUWATER improved access to water situation in both Kitgum and Pader towns. For example, they said, by end of 2010 the water supply in Kitgum had not increased as planned. Limited rehabilitation work was done. The target to construct 600 new connections per year was very far from target. 50% of MWE/DWD officials interviewed answered to the affirmative, that NUWATER improved access to water in both towns. 50% however, disagreed that NUWATER did improve access to water situation in Kitgum and Pader. All officials interviewed at Kitgum TC (4 out of 4) affirmed that access to water has improved. They said connections were subsidized at 59,000/= and was affordable for a number of households. 2 out of 3 (66.67%) leaders of Kitgum district disagreed that access has not improved and attributed that to low supply. The Assistant CAO noted: “Water supply situation in Kitgum is greatly undermining the access to water situation in the town with some connections standing as dry connections”. At the time of this evaluation there was water rationing; whereby the town is divided into seven service areas, each rationed for water once a week. In Pader, 50% of district officials said NUWATER Members of the Kitgum Water Board that were interviewed all share the same view on access to water situation in the town. All (3 out of 3) board members agreed that access to water has improved with NUWATER. However, the same respondents also pointed at low production as affecting access to water. Water is not available all the time. 50% of the Pader water board agreed that access to water had improved. The 4 out of 4 WASH Consult (PO) staff interviewed disagreed that access had improved. In terms of new connections, more people have been connected on the system. However, low production is frustrating the people. PO reported that some new connections had not received water for six months. They reported that the system still operate with five pumping stations it had before NUWATER. 5 out of 5 kiosk attendants in Pader agreed that NUWATER has improved access to water. Majority of water consumers interviewed in both towns (59.26% in Kitgum and 76.19% in Pader) confirmed that access to water has improved. The Evaluation team however, observed that Kitgum water consumers are a bit frustrated with NUWATER project. They said water supply and availability is not reliable, though they feel water is brought nearer to them and at affordable pricing. 92 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS has not improved access to water situation. The 66.67% of TC officials agreed that access to water situation has improved. They boasted of the water quality which has improved significantly. They however noted that the water kiosks are only concentrated within the middle/centre of the town, leaving out the bigger areas. 2. Were there clear performance indicators and targets well specified in the PMP? Question was not asked to USAID Question was not asked to NUWATER, but Evaluation Team reviewed the project PMP availed to them by NUWATER team. The PMP had the following indicators categorized in different levels: Composite Indicators (Level 1: Viability of Model) >Operational Support >Performance Bonus >Non-Revenue Water (NRW) Outcome Indicators (Level 2:Utility Performance) >Collection Rate >Response Time 2 out of 4 MWE officials interviewed responded that there were no problems with the targets. The Assistant Commissioner (Urban Water) noted: “These are same kind of targets which are being achieved by private operators in other towns”. They attribute inability of Kitgum PO to meet the targets to the PO‟s lack of capacity to do so. They express the need for DWD to liaise with the local governments (Pader and Kitgum) to design and tender better contracts in future. Question was not asked to LG Officials Question was not asked to Water Board PO agreed that PMP specified clear performance indicators. However, they noted that targets were unrealistic, unachievable due to factors ranging from low production of the system to bureaucratic and administrative delays in decision making processes. The Evaluation Team‟s assessment of the indicators show that all indicators were clear EXCEPT last indicator in Level 4 (Number of rural households benefiting). Question was not asked to Water Consumers 93 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS >Water Quality Standards >Record Keeping & Accountability Standards >Water Availability Output Indicators (Level 3: Contractor Performance) >Government officials trained >Connections/points constructed >Audits performed on PO Output Indicators (Level 4: FACTS Reporting Indicators) >Number of people with improved access to drinking water >Number of rural households benefiting The Evaluation Team‟s assessment of the indicators show that all indicators were clear EXCEPT last indicator in Level 4 (Number of rural households benefiting). The team wondered why this particular indicator was maintained on the list of indicators for reporting on which NUWATER scored zero throughout the project reporting. 3. What are the factors that have contributed to Question was not asked to USAID NUWATER staff interviewed pointed the following factors: DWD staff interviewed pointed that success was achieved Local Government (LG) officials in both Kitgum and Pader expressed that: >There was a lot of >The community is willing to pay for water PO noted: “NUWATER paid outstanding UMEME bills that was a big problem. Now the situation is better, it has Question was not asked to water 94 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS the success of the initiative? >NUWATER staff were fully facilitated >Project had radio talk shows utilized to mobilize community >M&E team had public awareness staff that helped the PO and improved collection. because: >NUWATER involved NWSC in technical supervision. >DWD staff also pointed out they facilitated NUWATER to have Pader gazetted as a water authority. community support to the NUWATER project. because people knew they would get more water as a result. in both towns. Both towns attributed success to NUWATER project interventions such as paying for power bills, providing fuel for generators. improved water supply”. PO also pointed that people are willing to pay water bills. consumers 4. To what extent can the progress made thus far be attributed to the program‘s intervention rather than external factors? Question was not asked to USAID. NUWATER COP reported that: Institutional risk that wasn‟t properly assessed in the original design was overcome. Pader didn‟t have a Water Board and was not gazetted. NUWATER took it upon itself to work with the MWE and the Pader TC to have it gazetted. NUWATER reported having done a complete re-design of the Pader system after the system failed to attract a competent PO, thus enabling the project to be operated by the TC with assistance of the kiosk operators. In Kitgum, NUWATER carried out pre-testing of water points to establish their production capacity because production capacity was below the demand. COP reported that the PO‟s excuses for not meeting targets were placed on production capacity of the Kitgum system. The billing system was not adequate, and in January 2011 NUWATER installed a web-based Billing Software, enabling efficiency The monthly Evaluation Reports produced by the technical team from NWSC-ESU (a sub￾contractor to ARD) helped in monitoring progress and outcome. In Kitgum, respondents expressed that the water system was nearly dead (non-functional) before NUWATER came in. They said the system was supplying less water, but also pointed that the situation has not improved significantly with NUWATER, except for infrastructure (wells, new connections, pumps etc) because water quantity/supply has not been increased to match the demand (new connections). The system should have focused on hardware first. Kitgum water Board members listed a number of interventions that led to success of NUWATER: >Payment of UMEME bills which to them was impossible to off-set >Close monitoring and timely replacement has reduce the rate of systems break down >stringent financial control system put in place led to financial discipline, thus they never experience any financial mismanagement by the PO. > Board got monthly reports. Also attended quarterly workshops to assess water situation; where they board members were educated on areas of success and improvement. There has been some level of capital investment on the system in Kitgum. Well testing to determine their production capacity has also been done. NUWATER cleared outstanding UMEME bills which had resulted into disconnection of some pumps from the main power grid. Question was not asked to water consumers 95 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS 5. How does the current water supply situation compare to what it was prior to activity implementation ? According the COTR: “The reports submitted indicate improvement in water supply”. This view concurs with those of other USAID staff interviewed by the Evaluation Team; whereby 80% thought water situation has improved in both towns as a result of NUWATER interventions. NUWATER management all agreed that water supply situation improved in both Kitgum and Pader. They noted the following: >Pader had 3 pumping station and water situation was bad. NUWATER carried out pump testing and took on one that it rehabilitated and supply is now sufficient >Pader well re-drilled >Kitgum situation was bad. Low pumping level resulting from power cut due to accumulated power bills. NUWATER cleared bills and situation got better. >New wells drilled in Kitgum NWSC staff observed that NUWATER has done very little to improve the water situation in Kitgum and Pader. Former NUWATER Field Program Manager noted: “According to the design, Pader would get 300 new connections, but got none. NUWATER was very far from targets in both Kitgum and Pader.” Low water production to supply areas at a go, thus water rationing, in which one area gets water once a week. All members of the water boards that the Evaluation Team interviewed shared the general view that NUWATER has improved the water situation in Kitgum and Pader. In Kitgum for example, Water Board members agreed that NUWATER came at a time when the town water system was almost non-functional. The town Mayor observed: “…the system was almost dead”. But now, the system has been improved, more wells drilled. However, they further noted that water supply situation is not very different from past. The PO in Kitgum noted that current water supply situation in Kitgum has improved. They pointed that NUWATER cleared all UMEME bills which were a major impediment to supply. Water users/consumers were interviewed about past and current water situations and the Evaluation Team got the following: Water Sources Before NUWATER: Out of 108 votes by show of hands; the following sources were used: Boreholes – 29 votes (27%), open shallow hand-dug wells – 4 votes (4%), season river water/ riverbed – 20 votes (19%), rain water – 28 votes (26%), yard taps in compounds – 9 votes (8%), water kiosks – 18 votes (17%). Water Sources After NUWATER: Out of 108 votes by show of hands; Boreholes water is used by most of the people for drinking and cleaning – 30 votes (29%), open wells – 3 votes (3%), yard taps in compound – drinking, unreliable, only sometimes – 11 votes (11%). Other sources included river water used mainly for washing clothes and drinking – 24 votes (23%), 96 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS rainwater when available – 35 votes (34%).Water kiosks – rarely have water, kiosk attendants never there, 0 votes. The Evaluation Team discovered that there are 5 water kiosks in Kitgum and only one was functional; but had no water most of the time because of rationing. 6. What are the challenges hindering the achievement of the results? USAID staff interviewed expressed concern over lack of capacity at all levels. They said NUWATER was not doing capacity building as was expected of them in the contract. They did not have any plan for capacity building, at least to the extent that they could prove to USAID. This is one of the reasons USAID presented to the Evaluation Team for ending of the NUWATER contract without extension. USAID also reported poor attitude on the part of NUWATER to do the job. NUWATER was managing the project from its Kampala office and USAID thought the NUWATER COP should have had a hands-on by basing in Kitgum. USAID admitted its failure to scrutinize the NUWATER Quarterly Reports to be able to take action in addressing NUWATER management pointed that: >The PO has other businesses and sometimes gives less time to the project. For example slow responses. They suggest that PO should allocate specific staff to the project. >PO lacked capacity, though they proved to be the best bidder. >Low income level of the people leading to default in bill payment >Existence of alternative sources of water undermines project collections of overdue bills (arrears) because people tend to run to the alternative sources which are usually free >NUWATER failed to get PO in Pader because of low production. In Kitgum, well NWSC Staff attributed challenges to: >Lack of delegation on the part of ARD COP, >Lack of flexibility and ingenuity. On the other hand MWE/DWD staff thought that: > NUWATER contract was too short. To them, normally the contracts range between 7 to 10 years (where there are no infrastructure) and 2 to 5 years (where there is some infrastructure). >DWD said lack of information (proper coordination) on progress of activities on the ground was another major challenge of the project; up to the extent that at certain point they had to call for regional meeting to ascertain the TC in Kitgum said the implementation capacity of NUWATER was low and the project promised many unachieved goals. The project should have dealt with the infrastructure challenges first. Also, the problems faced with Trandint, the former operator, resulted in a poor reputation of Kitgum TC and therefore only few qualified operators bid on the contract. The capacity of the current operator is not sufficient and the TC would prefer a more competent PO. The District, on the other hand, said the DWO could have been more involved and capacity could have been built both in TC and District engineers to enhance sustainability. In addition, the presence of the Field Coordinator was insufficient. In Pader, both the District and the TC said the The production level is still low in Kitgum. The town system had low producing wells connected to it. All new and high production wells were not yet connected to the system. The KTC Water Board listed other challenges: a) Problem with DWD, not helping the town as expected b) Power problem, not consistent affecting supplies c) Board members are not trained how to read reports and understand them. d) Water not enough for the population, suburbs have no access. e) Water is pumped in shifts to different areas due to low production capacity in Kitgum. f) High Non-Revenue Water (NRW) g) Some pipes are of Unreliable power supply by UMEME. Some solar systems at pumping stations like YY Okot and KTI were non￾functional. UMEME connections at some stations like K-New were found by the Evaluation team to be non functional due technical problems. Low pressure in the system has greatly affected the water system in Kitgum. The same is reported with the Pader system as well. Public impatience/frustration inadequate water supply. The high expectations for improved water supply services and the new connections subsidy arrangements are slowly degenerating into frustration as NUWATER has not quickly moved to implement Water users pointed a number challenges, some of which could not be verified by the evaluation team. >Quality – hard water, foams when boiling. Takes a lot of soap for washing (Kitgum). >Rigid metering/ sales – would like to be able to buy smaller quantities >No provision for vulnerable people who are unable to get money for purchasing water. >Dirt entering pipes leading to water contamination. > poor communication from PO about meter 97 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS some of the concerns raised in the reports. COTR noted: “…scrutinizing the Quarterly Reports; I think if there was a weakness on USAID‟s part it is not to recognize some of the weaknesses and have them addressed with NUWATER”. USAID also admitted that its previous Team Leader (TL) was weak; she knew all the problems with the project but could not address them. There was poor communication between NUWATER COP and USAID. The COP directed his communication to the Mission Director. “There were many direct communications between COP [NUWATER] and Mission Director by-passing the Team Leader. Operational weaknesses on the part of the PO. For example, meter readers also collect bills, which shouldn‟t have been the case. The meter readers collecting payments without issuing receipts instantly. drilling was delayed until late. If done earlier would bring some tangible results. >Project design was not appropriate because no baseline survey was done to enable them gauge the level of targets to set. >Procurement took too long, a lot of time wasted. activities on the ground. project implementation should rather have focused on training local individuals to run the system than spending time on gazetting and procurement, and that there was no community mobilization, something that will compromise sustainability and the new infrastructure project. small diameters and they need replacement, which was not done by NUWATER. the promised activities. Faulty bulk meters for some Installations. Faulty bulk meters were sited at various pumping stations and at the reservoir. The PO expressed concern about delayed approval of payment by the NUWATER COP thus affecting its operations. Such delays have resulted into delayed payment of the PO staff payment thereby affecting their performance. The PO reported that that they are always reluctant to incur expenses on fixing certain issues due to fear of delayed reimbursement. reading and how bills are calculated > Not enough Kiosk sales in Kitgum due to rationing. Many of the poorer people prefer the Kiosks over private connections, however they are rarely operational > Very unreliable supply in Kitgum due to rationing > Some areas in Kitgum have not had water for years due to un-repaired distribution pipes >Kiosk attendants are not at the taps all the time when they should be. >Low pressure. In Pader they revealed that when all four taps at a kiosk are running there it takes long to fill a jerry can. Taps are too high for Jerry cans, causes water loss. 98 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS 7.What mechanisms and institutions are in place to guarantee sustainability of the services provided and results achieved thus far? The original project design looked at capacity building as a major activity. In the PMP NUWATER project would conduct training of local government officials to take on contract management role. NUWATER seemed to have not been doing this to the satisfaction of USAID. The Evaluation Team discovered that a number of officials got some form of capacity building (through attending workshops and presentations), but they were not able to verify the level of knowledge gained by the officials to take on management role. NUWATER management reported that: >The PO and members of the Water Board in Kitgum have been trained in the Billing System. However, PO expressed worries that the NUWATER Field Coordinator (FC) was the Systems Administrator for the software, and given that the FC was not in Kitgum all the time, the billing software was not being utilized. >The TC officials of both Kitgum and Pader were trained to take over management of the systems. As noted in other parts of this document, the Evaluation Team did not see any documented evidence of such trainings, leave alone determining the extent of knowledge gained. According to MWE officials, the project was not sufficiently anchored in the local government framework, especially MWE, to benefit from its support after the end of the project. MWE should have been more closely involved so that technical support can be provided after the project end. Some TC officials (Acting Water Officer and Town Engineer) got some training in Billing Software. The Town Clerk normally participated during the M&E exercises, however was exchanged several times during project preiod. TC officials said that more formal training in water utility management and M&E would have been welcome and necessary. In Pader only a brief explanation to Kiosk attendants and TC officials was reported. Local engineers and officials lack maintenance manuals and the knowledge of how to maintain machines and infrastructure in Pader. NUWATER conducted some level of training for the Water Board members through presentations. However, the Evaluation Team could not verify the level of knowledge gained to take on management role. WB members participated in M&E exercise meetings but one member in Kitgum said he had difficulties understanding the reports and would like training in that. WB members in Pader never received any on-the-job or formal training. The over-controlling attitude of NUWATER management became clear as WB members expressed concern over what will happen when NUWATER is gone. This shows local control mechanisms were not created or strengthened to continue beyond the project. The Water Board members in both Kitgum have received some minimum level of training to be able to oversee the water systems when NUWATER expires. PO employees in Kitgum were satisfied with the follow up through the NWSC team however agreed formal training would have helped in certain subjects such as financial management. There was no technical training of engineers although the technical capacity of the PO was low, and NUWATER Field Coordinator even gave wrong technical advice in Kitgum or did not inform pump-attendants in Pader when work was to be done. Some of the infrastructure put in place by NUWATER was faulty (bulk meters). Since the increase in consumers in Kitgum was not achieved, revenues are not likely to be able to cover operations and maintenance costs, let alone replacement and rehabilitation in the long run. The PO in Kitgum said he would not re-apply for the contract when it expires. In Pader, the water price is currently not enough to cover operations and maintenance either without subsidies. This question was not asked for the water consumers directly. However, the Evaluation derived some answers to this question from the responses and comments of the water consumers. For example, majority of the water consumers in Pader are comfortable with paying for water. They said the price of shs.50/= is affordable, and they are willing to pay. Q2. How realistic and appropriate was the project design? USAID NUWATER MANAGEMENT MWE/DWD/NWSC OFFICIALS LOCAL GOVERNMENT OFFICIALS WATER BOARDMEMBERS PRIVATER WATER OPERATOR WATER CONSUMERS 1.Were the two key assumptions – a functioning water system and a private sector interested in 40% of USAID staff interviewed responded that NUWATER project design was based on wrong assumptions. NUWATER former COTR observed: “…PROJECT WAS BASED Project design was not appropriate because no baseline survey was done to enable NUWATER agree on realistic targets. NUWATER Program Assistant observed that “… the project was Both MWE/DWD (50%) and NWSC (4/4) thought the assumptions were right for the case of Kitgum where there was already an existing water system, and POs were willing to run it, given the location of the Pader TC officials interviewed (5 out of 8) all disagreed with the assumptions. They said there was no functioning 33.33% of water board members in Kitgum thought the assumptions were true. 50% of Pader water board said the assumptions were true. The PO agreed that assumptions were true, but noted that those assumptions did not take into consideration the level of Question was not asked to the water consumers. 99 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS bidding for contracts - true? ON COMPLETELY WRONG ASSUMPTIONS.”, forcing USAID to think about terminating the project and do a redesign; but realized USAID would spend money without outcome if that happened. USAID had to approve some funding ($500,000) to ARD to do basic small-scale capital investment. 40% thought the assumptions were correct, and 20% had no comment. merely guessing conditions”. The project assumption of an existing water system, as was the case for Kitgum, did not consider other factors like the production capacity and storage of the existing system to enable NUWATER set realistic targets. town. However, they disagreed with the assumptions being true for Pader which had no history of paying for water. system in place. 5 out of 7 Kitgum officials agreed that assumptions were true. They said the town had a system that was at the verge of collapse – functioning but with low production level. infrastructure on the ground. The Kitgum system needed a lot of infrastructure development to boost production. It was assumed that people are willing to pay for water if as a result they receive increased quantity and quality of water, which quantity did not increase significantly. The PO reported that some connections on the system do not receive water for more than six months. According to the PO the design over estimated production level 2.What informed the original conception and design of this project and how responsive was the design was to the need at that time? USAID/DC designed the project. NWSC did a rapid assessment that guided the design. They wanted to support the northern & eastern Uganda towns and then to central Uganda. They put together a $2M DC￾earmark for project to pilot how to use incentive-based systems as model to stimulate utilities to support private operators with incentives. NUWATER reported that original conception and design of project was based on a rapid feasibility study conducted by ARD in collaboration with ARD NWSC was asked by USAID Washington to work with ARD to do a feasibility study. Based on the report of this study USAID increased project funding from $1million to $3million. Officials at DWD feel the design of the project did not let the Central Government (MWE) to play its key role it does in such projects. In-house procedures were ignored by the project. The common practice is that DWD usually signs an agreement and have a leading role in supervising the local governments. But to them NUWATER, as with other USAID projects, ignored them and usually communication was directly between USAID and NUWATER team. The targets set in PMP were unachievable, were unrealistic. Production level was too low. Engineers in Kitgum TC admitted to have given the assessment team wrong and outdated data on which they based the design. The data was from the Austrians that had done rehabilitation in 2001, and the production capacity of the wells had already significantly declined when the assessment took place in 2007. Therefore, the project design was based on entirely wrong data. In Pader, TC officials said the DWD system in Pader was functional at the time of assessment, however it broke down shortly after. Question was asked to Water Board members. PO expressed that design of the project identified the needs on the ground. For example capacity building was catered for in the design, but never implemented effectively. Water production remained a big challenge throughout the project life and NUWATER seemed to have not responding to that during the last months of project end. Question was not asked to the water consumers. 3.How did the design of the project influence the The project design assumed if done well, with minimal capital investment, water utility would become The targets set in PMP were unachievable, were unrealistic. Production level was too low. In the view of DWD, NUWATER did the best in their ability, but the problem was with the design Both Town Councils said the design was good because it promised an increase in private Question was asked to Water Board members. Targets are not realistic, unachievable. Question was not asked to the water consumers. 100 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS results? sustainable. Former COTR reported having warned about capacity of private operators in a small town from war situation being very limited, and that water infrastructure is not there. Institutional risks were not properly assessed. Pader didn‟t have a Water Board and was not gazetted, thus could not contract a Private Operator. USAID and NUWATER did some modifications in the design after the situation on the ground was better known, however no formal contract modification was done. The changes consisted of approving extra money for infrastructure development (later halted) and a change in management system in Pader after no PO showed interest. of the project Officials at DWD feel the design of the project did not let the Central Government (MWE) to play its key role it usually does in such projects. The role of NWSC to try and make things quicker, given its experience, was not properly utilized by ARD. All NWSC staff interviewed (4 out of 4) thought their role was minimized by ARD and thus could not do to their fullest. Work ethics in the districts/towns are totally different from the US standards. DWD feel that because they were not fully involved, especially in the procurement process, NUWATER simply picked the „best‟ bidder who happened to be less experienced in water utility management. connections, however there was too little focus on hardware and therefore the limited results. Water production remained a big challenge throughout the project life. 4.Was project implementation appropriate and in line with the priority needs of all stakeholders? USAID clearly expressed dissatisfaction of implementation and therefore the evaluation. Implementation was hampered by changes in team leader and COTR, and USAID had to make several difficult decisions in the implementation process such as halting implementation until the former PO in Kitgum could be terminated and a new one contracted in an orderly way, halting the spending on infrastructure in 2010, and deciding to close the project without a no-cost extension. These decisions affected the progress of the implementation of the project NUWATER staff say their efforts in implementation were affected by external factors such as political interest of MWE officials in Private Operator firms, reluctance of the TC and MWE to terminate the former PO in Kitgum, time spent with negotiating MoUs and problems in communication and understanding from USAID. The capacity of the local PO, TCs and Water Boards were also extremely low and could not be given much independence. They assure they always tried to do their best. DWD was not interested in cooperating because it was not involved in the design which provided for 70% in technical assistance - DWD wanted 70% for capital investment. DWD holds the assets, Town Council (Kitgum/Pader) provides operator; private operator; consumers as stakeholders were ignored; ARD COP did not coordinate very well with engineers at DWD, met with them. Town Council and District officials complained that the project promised many connections and more water for the people, however very little was achieved. This shows the implementation was poor and therefore NWUATER performed poorly. Also, several parts of the MoUs were not fulfilled at all, such as the promised capacity building. One Water Board member in Kitgum said NUWATER tried their best but it was difficult due to corruption in the Town Council and the lack of capacity and willingness to perform better of the PO. The PO in Kitgum said capacity building was poor, there was too little focus on infrastructure and the infrastructure came too late in the project period. Also, the implementation and management style of NUWATER made operations more difficult and complicated instead of facilitating it. The Field Coordinator was mostly absent. In Pader, the Kiosk attendants, pump attendants and officials managing the system said capacity building was poor and the entire management system chosen was not optimal. In addition, no community mobilization Question was not asked to the water consumers but from their comments it was clear that the NUWATER team did almost no outreach to the consumers. Only a few consumers in Kitgum had heard announcements on the radio, and only about the new infrastructure project. Consumers in both towns were generally thankful that the ET organized a meeting because it was the first 101 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS and also affected its results. was done even after several requests. opportunity for them to air their concerns in the whole project period. 5.Assess the extent to which the development hypothesis within the project worked. USAID officials are concerned because the project did not achieve the expectations; none of the towns are able to manage their water systems on a sustainable basis. This was one of the main reasons why they agreed on the additional infrastructure development project to follow NUWATER. NUWATER is based on a project hypothesis that “well managed incentive-based contracts have the capacity to ensure recovery of costs associated with water supply provision and will results in significantly improved services to customers, and thus result in sustainability of the systems serving those customers.” It was expected that the 3- year intervention would result into the following: >The quality of service provided will be markedly improved; >Systems will be able to reliably serve more customers, and; >Incentive based systems will contribute to water supply system sustainability. Good as they were, not much of expectations from the interventions were achieved. The Evaluation Team discovered that there was poor communication and tense relationship amongst the actors, particularly between ARD management on the project and other actors. Limited capacity building was done to enable sustainable management. Many customers have been connected to the water system but availability of water is not sufficient to DWD and NWSC shared the view that well managed incentive-based contracts have the capacity to ensure recovery of costs associated with water supply provision and results in significantly improved services to customers and subsequent sustainability of the systems. All 4 out of 4 NWSC staff thought the NUWATER experience did not improve quality of service and the systems in both towns have low capacity to serve the customers. For example, there are number of dry connections in Kitgum, and Pader does not serve a large number of the town population. Question was asked to Water Board members. Question was not asked to the water consumers. 102 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS serve them. NUWATER staff also said the contracts were too complex for the private operators in Uganda to understand. 6.Did the incentives in the contract really serve as incentives to the private operator? Question not asked to USAID staff NUWATER management noted that the Kitgum PO‟s capacity is low. They noted that the PO has not been able to meet operational targets due to low capacity. DWD staff confirmed that the targets for the Kitgum PO are the same targets being achieved by other PO operating in other towns. However, they pointed out that 3 years is not enough to achieve this, and that they have now extended contract length in other small towns. NWSC staff thought many of the targets were too high for the PO to achieve, given low production. Pader system could not attract any PO despite the project incentives. Local government officials were asked why only few private operators showed interest in the incentive-based contracts which should normally be attractive because they involve subsidies. Answers showed that there were internal dynamics in the private operator market that resulted in lack of interest from competent private operators. For example, the former PO, Trandint, has a powerful position in APWO, the private operator association, and influenced other members not to bid. Others said the fact that USAID was supporting the project and the presence of subsidies was not clearly communicated in the procurement process. Others also said that MWE officials have personal interests in private operator companies, and since they felt left out in the project they discouraged the companies from bidding. This shows incentive￾based contracts are not sufficient to attract good private sector operators and that local political factors have to be taken into account. The water Board in Kitgum expressed that the incentives of NUWATR project were very attractive. The PO in Kitgum expressed to Evaluation Team that he might not be interested in renewing his contract if he was given the opportunity. They noted the following: >Targets set without looking at the production capacity of the system. Due to low production level PO was not able to meet the targets, not even once. For the same reason, PO never earned any bonuses which to them would be a source of motivation. >Faulty meters (not functional after short period) installed by NUWATER. It was difficult to estimate volume of water pumped and supplied through the system. >Delayed payment resulting into financial difficulties for the PO to run operations smoothly. The PO in Kitgum was asked about some of the incentives, and why for example the subsidy on collections did not motivate the PO to increase collections. There was a significant slump in collections in the beginning of 2011 for example. The PO explained consumers were difficult and often refused to pay. In addition, the decision by NWUATER to first considers payments as arrears and then as payments Question was not asked to the water consumers. 103 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS for new bills, resulting in a lower subsidy, reduced the incentive. In addition, subsidy payments were always done late, therefore a limited effect was felt by the PO. This shows that even monetary incentives need to be properly implemented to function. In addition, the bonus payments were calculated based on outdated production data. This was not adjusted after the start of the project and therefore the PO never achieved a bonus payment. 7.If no, what should be done differently? The design did not entirely take into account the local factors and it was therefore difficult for NWUATER to implement. According to MWE, USAID should align with other donors and provide funding to the basket fund through the SWAP instead of funding individual projects. MWE should have been more closely involved in the inception of the project, not NWSC which is not exactly a central government entity but rather acted as a consultant in this project. NUWATER project design should have focused more on hardware. >Increase production levels and the other variables will respond setting target >Set realistic/achievable targets Question was not asked to the water consumers. Q3. Who was involved in the program and to what extent did the program promote better coordination and collaboration amongst them? USAID NUWATER MANAGEMENT MWE/DWD/NWSC OFFICIALS LOCAL GOVERNMENT OFFICIALS WATER BOARD MEMBERS PRIVATER WATER OPERATOR WATER CONSUMERS 1.Were the appropriate stakeholders and their various needs well identified? According to USAID, DWD was not interested in cooperating because it was not involved in the design which provided for 70% in technical assistance - DWD wanted 70% for capital investment. Appropriate stakeholders ARD was told to use NWSC when they were bidding for this project. NWSC worked with ARD as a sub￾contractor who provided technical assistance (TA) in forms of monthly M&E audits of the project. NUWATER COP would authorize payments for the PO based on the M&E reports submitted. ARD let go/fired NWSC in November 2010 on NWSC designed the Operating Contract based on its own experience and work ethics. They say their conception of how performance would be improved in Kitgum was not utilized by ARD, were seriously constrained and forced to put down their ideas. DWD feel their involvement was not fully incorporated. DWD has about 23 pre-qualified POs that would have been attracted to bid for All (8 out of 8 in Pader and 7 out of 7 in Kitgum) officials interviewed agreed that appropriate stakeholders were involved. They mentioned: >The District (Procurement, Water, CAO) In Kitgum, the WB was actively involved in monitoring the PO. However, in Pader, a WB was established but never involved in the procurement process or the alternative management chosen after the tendering process failed. Although the project design provides for “community outreach campaigns”, this was only carried out in a limited way. In Pader, only one point intervention involving only a few water users was reported by users. In Kitgum, users 104 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS were involved but poorly coordinated. DWD holds the assets, Town Councils (Kitgum) provides operator; private operator; consumers as stakeholders were ignored; ARD COP did not coordinate very well with engineers at DWD. The COP struggled to get USAID￾MWE MOU signed and learned that the NUWATER COP rarely met with DWD - they conflicted. The right stakeholders were involved, but there were poor communications and coordination. The Evaluation Team learned that MWE/DWD had a presence in Kitgum before NUWATER project but left as soon as they learned that USAID project was going to the area. grounds of alleged collusion, failure to check for meter readings, demanding different rate/fee for capacity building. the project operation but because the POs think as long as the central government has no hand in the procurement and other processes they would not participate for fear of local government mistreating them. To DWD, Kitgum already had a bad record of mistreating the previous PO. DWD thought if they had worked together with USAID, the POs response to the bid would have been different since they would be confident that they have the Ministry to fall back to in case USAID pulls out. DWD had piloted a project and claimed to have advised NUWATER on the way to go but NUWATER ignored them. Usually, development partners consult with the MWE/DWD on priority/need of the Ministry, but to them, USAID‟s way of doing things is usually different from other development partners, not only in the water sector. NWSC formed a team that carried out monthly M&E and prepared reports against set targets that were submitted to NUWATER COP. NWSC provided input in designing bid documents and participated in bid evaluation. DWD expressed no knowledge that NUWATER project ends now and that current PO contract ends Feb 2012. >Town Council (Town Clerk, Urban Water, Engineer, Mayor, Water Board) >MWE/DWD/NWSC In Kitgum, the District expressed that the District Water Office should have been more involved to ensure sustainability. There was no needs assessment done of local stakeholders. reported to have heard about the project only once on the radio. No users reported to have received a visit of the public outreach specialist hired to carry out a campaign in Kitgum. Consumers in both towns were generally thankful that the ET organized a meeting because it was the first opportunity for them to air their concerns in the whole project period. 2.What was the level of stakeholder NUWATER sub-contracted NWSC-ESU to provide technical supervision of the NUWATER staff mentioned that it was difficult to give stakeholders autonomy in NWSC was asked by USAID/Washington to work together with ARD to do a At the district level, both Kitgum and Pader District Procurement In Kitgum, the Water Board was involved through participating The PO in Kitgum was a central player in managing the project and was Question was not asked to the water consumers. 105 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS involvement in managing the project? project. The COP would provide administrative guidance and coordinate various stakeholders. project management because of fears of misuse of funds and corruption. Therefore, NUWATER had to take on all the responsibility. feasibility study in 5 towns, which formed the basis for the design for NUWATER. NWSC formed a team for the monthly M&E and prepared reports against set targets the were presented to NUWATER COP. NWSC also participated in the design of operating contracts, procurement, preparing documentation for capital works projects. These roles shifted between the PO and TC (for case of Kitgum) upon procurement of the private operator. ARD was the Contract Manager to oversee and manage the contract between PO and KTC. MWE was involved through the initial MoU negotiations, and in a limited way through its Water and Sanitation Development Facility North based in Lira, which was involved in the new infrastructure design. MWE also supported the gazetting process of Pader. Officers said their offices had full involvement in the procurement processes. The office of the Chief Administrative Officers (CAO) of both districts were reported to have been represented the office of the Town Clerk (TC) of respective town council. Other town council leadership in Kitgum however, mentioned that they were kept out and didn't understand how NUWATER works, not collaboration at senior TC levels. They leaders reported that NUWATER dealt with the Water Board, Urban Water Officer and Town Engineer. The District Water Office was not involved in the project in neither towns. during monthly and quarterly evaluation meetings. They reported that they appreciated this and were learning, however they wanted more training on how to understand the reports and about M&E and utility water management in general. In Pader, the Water Board was not involved at all. supported, until November 2010, by the NWSC-ESU team. This support was highly appreciated because of limited capacity of the PO in water utilities management, engineering, financial management and customer relations. In Pader, some officials from the town council doubled as water managers in the last few months of the project, however they complained about lack of resources and skills to perform well. 3.Assess the extent to which the project addressed gender concerns in the management and beneficiaries of the project. Question was not asked to USAID. However, Evaluation Team discovered USAID SO7 Team Leader and the COTR for NUWATER were both females. Evaluation Team reviewed several project documents but was not able to find topics elaborating how gender considerations would be addressed by the project. Question was not asked to NUWATER management. Evaluation Team reviewed several project documents but was not able to find topics elaborating how gender considerations would be addressed by the project. Question was not asked to MWE/DWD/NWSC The Project did not include women in the key positions at the TC and the Water Board. In Pader for example, the Evaluation Team discovered that all members of the water Board were men. Key positions at the TC (Town Clerk, Water Officer, Town Engineer, Mayor, Deputy Mayor) were all taken by men. Even at the district level, key positions (CAO, Water Officer, District Procurement Question was not asked to the water consumers. 106 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS Officer) were taken up by men. In Kitgum TC, there is only one female on the Water Board and another at the district (Assistant CAO). NUWATER project documents that the Evaluation Team had access to do not include topics elaborating how gender considerations would be addressed by the project. 4.How best should stakeholders be involved in such projects? Question was not asked to USAID DWD staff interviewed suggested that USAID should start participating in designing the MWE/DWD plans. They reported that other development partners who participate in their planning process normally ask the MWE about their (MWE‟s) needs and the MWE plan with them. They asked USAID to do the same in order to respond to needs. Kitgum District said there should be more physical presence of the NUWATER team in order to facilitate communication and learning. When everything is managed from Kampala communication gets difficult. The same was reported from Pader Town Council. Kitgum District also wanted more involvement of District Water Officers in order to provide support to Town Engineers. The Water Board in Pader had potential to create a much needed link between the consumers and the town council. They felt they were neglected and therefore the problems of vandalism etc. Even in the interim management structure, they felt they could have played an important role, but they were never solicited or informed. Consumers wanted more information about the project, especially in frustrating situations such as when the subsidies were cut for new connections in Kitgum. Also, they had no knowledge of how meters are read and how bills are calculated, and why they have to pay a service charge even if they have not received any water the whole month. This lead to suspicion and reluctance to pay. 5.How were capacity building activities carried out and were they effective? The original project design looked at capacity building as a major activity. In the PMP NUWATER project would conduct training of local government officials to take on contract management role. NUWATER seemed to have not been doing this to the In an interview with the Evaluation Team, NUWATER COP admitted that capacity building was part of the project contract. He reported to the team that he had recommended the NWSC M&E team to incorporate one member of the water board, and to carry out quarterly evaluation of the private operator and the Water Board. COP also NWSC noted that training was not their role in the NUWATER contract. NWSC staff reported that their role was to point out training/capacity building needs. However, NWSCM&E team carried out some form of capacity building in the monthly M&E exercise. Former NUWATER Field Program Authorities that required formal training never received them; only Pader received some training when they were being prepared to become a water authority. TC officials said they would have benefited a lot from training in water utilities management and monitoring and evaluation, however they The Water Board in Kitgum was happy to be involved in the M&E exercise, however never received any training on how to understand reports or findings and how to take a more active role. The M&E exercise, according to them, mostly became meetings in which NUWATER criticized The PO in Kitgum was happy with the follow up provided by NWSC but still wanted more formal training on specific topics such as technical issues, water utilities management, financial management and customer relations. The water managers in Pader never benefited from NWSC follow up and seemed to be “learning by doing”. All of Question was not asked to the water consumers. 107 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS satisfaction of USAID. The COTR noted: “They were not doing capacity building. They didn‟t have any plan for capacity building”. USAID asked for a plan for capacity building especially when the no-cost extension was requested by ARD but never felt like this was properly addressed. The Evaluation Team discovered that a number of officials got some form of capacity building (through attending workshops and presentations), but they were not able to verify the level of knowledge gained by the officials to take on management role. reported that having joint meetings with the Private Operator, stakeholders and the water board. The COP admitted that the only form of training offered to Town Council officials and Water Board members were to sit in on evaluation meetings, and that the only training provided to the PO was through the M&E exercises. NUWATER said that they had asked NWSC to carry out capacity building but NWSC had asked for extra/different payment for this and therefore it was dropped. However, from own assessment, the Evaluation Team discovered that the PO benefited from the M&E team, quarterly evaluation of the PO was carried out but there was no documented evidence of the capacity building interventions for the water board and the TC officials. Neither was there any documented evidence of the level of knowledge gained by the various parties. The only workshop that took place was on the billing software in 2011, towards the end of the project. Manager observed that: “The level of training was restricted to whoever would gain from the various workshops”. They reported that authorities that requested formal training never received them, except Pader that got trained when it was being prepared to become a water authority. never got anything. In Pader, TC officials noted that the “workshops” or meetings by NUWATER mostly consisted of “presentations” and not exercises that promote learning. TC officials in Kitgum were happy to have received training in the billing software, however resented that they still do not have access to it. the PO who then defended himself, while the WB was listening. The WB in Pader never received any training. them requested training, both Kiosk attendants (on how to read meters), Pump attendants (on how to maintain and service the generator and pumps, and how to do small repairs) and the management people (on financial management). Q4. Analyze the effectiveness of program management and its effect on the program outcomes. USAID NUWATER MANAGEMENT MWE/DWD/NWSC OFFICIALS LOCAL GOVERNMENT OFFICIALS WATER BOARDMEMBERS PRIVATER WATER OPERATOR WATER CONSUMERS 1.Was the project well managed by NUWATER? USAID staff interviewed noted that NUWATER management, particularly the COP, paid attention only from Kampala. COTR observed: “When we go to the field [Kitgum, Pader] I From the interviews the Evaluation Team had with NUWATER Field Coordinator, the team discovered that the Field Coordinator was hired late and that is why rehabilitation All staff of NWSC that the Evaluation Team talked to (4 out of 4) shared same view that the management of NUWATER was poorly done. The local government officials‟ view were somewhat divided in Kitgum town 33.33% of DLG officials Members of the water board that the Evaluation Team talked to in Kitgum town had divided opinion over this question, with majority (3/4) agreeing. They The PO completely disagree that the project was managed well by NUWATER. They cited delays in decision making, delayed payment resulting into the PO unpaid arrears dating back to Question was not asked to the water consumers. 108 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS heard from multiple sources that the NUWATER field guys are not out there to help fix some problems of the project. Attitude to do the job was poor on the part of NUWATER COP. NUWATER paid too much attention to infrastructure and skipped the other important things”. activities started late. NWSC staff described the management, particularly the COP, as always “reactive, defensive, vindictive and likes passing the back of other people, fault finder”. COP never responded to problems presented to him in time, had problem with paying money because of his poor perception that every Ugandan is corrupt. They reported that NUWATER COP „hid behind USAID to explain his failures‟, meaning he never accepted his failures. They said the NUWATER Field Coordinator was acting as “Technical Advisor” to the COP, and spent much of his time in Kampala, instead of being on the ground (Kitgum). NWSC staff added that “if the project was managed well, the results of evaluation now would be big”. thought NUWATER was managed well. The Kitgum District local government pointed that there was a problem with how communication and coordination was handled by the project – poor coordination with NUWATER staff and the PO. At Kitgum TC, 25% of officials thought NUWATER was managed well. They cited stringent financial procedures involving COP approval of ALL financial transactions as good practice to learn from. 50% of officials in Kitgum thought NUWATER management style wash suffocating the PO thereby affecting quality of services in the town water system. In Pader, 50% of those interviewed pointed that NUWATER was NOT managed well. They cited long time taken to respond to system breakdown, delayed payment for services. NUWATER FC who was the focal person reportedly had no presence in Kitgum, would only visit the area when needed. They claimed that the FC had no house in Kitgum and would sleep in hotels when he came to the area. The officials pointed that noted that NUWATER brought system of financial management that was very strict and restrained financial indiscipline. 1/4 of the board disagreed that the project was managed well by NUWATER. They noted that NUWATER did not meet their expectations, by failing to meet their promises to the town, thus could not view NUWATER having been managed well. Pader water board members all (2/2) disagreed. The Evaluation Team discovered that the Pader Water Board was created in 2009 but has only had one meeting since then, and only 1 time contact with NUWATER or the project. Expected training, briefings etc. Nothing happened. They expressed need technically knowledgeable engineers for technical support, detailed training. August 2010. Other issues cited by Po on NUWATER management included: >Delayed procurement of metres, sometimes allegedly fake/counterfeit meters were procured and wouldn‟t last. Evaluation Team also found several bulk meters at pump stations and the reservoir non-functional. This allegation could probably be true. >NUWATER had NO FULLY FUNCTIONAL FIELD OFFICE to be able to coordinate activities in timely manner. The Evaluation Team took interest in this particular allegation because it had come up several times and from different stakeholders. The team found out that NUWATER occupied one of the rooms on the building where WASH Consult (the PO) sits. However, the team also discovered that NUWATER field staff never sat there full time and indeed not even had a desk or working station there. The Evaluation Team, during their more-than-a- week stay in Kitgum, did not find any NUWATER staff in that office at any point. The PO alleged that the only printer used for printing customer bills was normally carried along by the Field Coordinator in his car disabling their operation. They would only print when the FC brought back the 109 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS NUWATER had very low capacity to handle the project, as seen from the number of staff and their presence in the area of implementation. printer. >Capacity building, which was part of NUWATER contract, was done to the minimal. PO expressed that their team got insufficient training thus lacked capacity to handle a number of issues. PO did not receive any engineering training but reported having received training in accounting procedures. 2.Were sufficient funds for project implementation available? USAID pointed that NUWATER project had an approved budget of $ 3 million for a period of 3 years. NUWATER thought USAID‟s „piece- meal‟ funding style did not make sense. Also, the capital investment funding was halted in March 2010 making further improvements of infrastructure impossible. 4 out of 4 officials at NWSC agreed that NUWATER project had an approved budget with funds sufficient to implement the project. They said poor management on the part of NUWATER (delayed payment for procurement and other services rendered) made it appear as if the project had insufficient funding. All (6 out of 6) officials in Pader TC said that the project had sufficient funds, but wondered why it operated only 5 water kiosks leaving other parts of the town without them. Kitgum TC officials (4 out of 4) all pointed that the project had sufficient funds for implementation. Both towns were disappointed and felt cheated because they had been promised a number of subsidized house connections, and due to the poor management of NUWATER and the PO the population will now never benefit from this money that they feel is rightly theirs. Board members interviewed in Kitgum (3 out of 3) all thought that there were sufficient funds for the project. They say funds were released for specific activities. Pader water board did not have any comment. They said that they were not aware about the total funding that was available for them. PO in Kitgum expressed that funds were not readily available when required, cited unpaid arrears dating as far back as August 2010. Other challenges relating to funds included delayed procurement of spares and other replacements resulting to system being unable to operate. In Pader, a kiosk attendant commented: “We attendants don‟t get our payments in time. We signed an agreement to get pay every forth night, but for two months now we have not been paid. Have no seats at the water points, no shed”. Question was not asked to the water consumers. 3.Were implementation procedures followed? All (4 out of 4) NWSC officials disagreed that NUWATER implementation procedures were followed. The former NUWATER Field Program Manager observed: “We told lies through the COP to the people of Pader on deliverables that never materialized, to an extent that even now I fear to go to Pader”. 5/5 officials in Pader agreed that implementation procedures were followed. They cited a process where the town authority with the help of NUWATER got a Water Authority status. Kitgum officials (6/6) also responded to the Board members pointed that procedures were followed. They however, noted that the project did not move according to schedule of activities, a lot of time was wasted through delays. No answer in Pader PO agreed that procedures were followed to a greater extent. The Evaluation Team noted that there were certain incidences where procedures were not followed. For example, the monthly M&E reports repeatedly highlighted the issue of cash payment (of salaries and wages) to PO staff, instead of Question was not asked to the water consumers. 110 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS According to NWSC staff, NUWATER did not achieve even 20% of its target, was supposed to increase water supply but capital improvements didn‟t' happen until late, nothing on the ground apart from little infrastructure improvements was achieved. affirmative. They said NUWATER followed all procedures by involving stakeholder and ensured that all processes were adhered to. EFT/Cheque transfers. PO attributed this to delayed transfer of its payment, such that by the time the funds are transferred PO would prefer cash payment as quickest mode of payment. The Evaluation team is also concerned that the Taxes and other obligations of the PO have not been met as required, despite several calls in the M&E reports to do so. The PO had arrears of VAT, WHT and NSSF remittances; which are mandatory obligations, something that might result into closure of PO business, thus affecting the Kitgum water operation. 4.Were activities implemented in a timely manner? NUWATER was a 3-yr project but actually ran less than 2 years, started more than 8 months late; then procurement took a lot of time. NUWATER arguments about contract extension were not compelling to USAID. In additional, USAID‟s level of confidence with the contractor was already low. According to the COP, “NUWATER project began in June 2008. But actual field activity began in February 2009”. Causes of delay include preparation of bidding documents, delays in having MOUs signed between various parties. The delays left the project with a 2 ½ year period for implementation. First appointed ARD COP declined his offer at some point and a new one brought on board. The change in COP had a number of start-up issues. The new COP had a legalistic approach and took a lot of time trying to understand IBC, the procurement and other processes involved. A lot of time was wasted as a result of delays at various stages. The bureaucracy in procuring PO in Uganda takes at least six months (according to DWD). There was a lot of bureaucracy with USAID in decision taking. A lot of time spent between USAID and ARD trying to harmonize their budgets. Delay in sorting out issues over MOU between USAID and MWE; nothing could be done on capital improvement before Operating Contract was signed between MWE and town council authority. Officials in both Kitgum and Pader 11/11 disagreed. They cited delays at several stages that left the project with a shorter life. As a result, there were a lot of frustrations for the communities about NUWATER. All (3 out of 3 in Kitgum and 2 out of 2 in Pader) water board members disagreed that activities were implemented in a timely manner. They noted that the project meant for 3 years took less than 2 ½ year on the ground. PO reported delays in many instances, and cited weaknesses in decision making on the part of NUWATER management. As a result, break downs could not be fixed in time. Additionally, the PO noted that it took a long time for them to realize sufficient water production level, and this affected operations in many ways. The PO was specifically frustrated about the fact that the new connections could not be done in time. When they bid on the contract, they had estimated a rapid increase in new customers through the OBA scheme, and therefore also sufficient revenues. However, due to lack of meters and wrong calculations of production capacity, and the failure of Question was not asked to water consumers 111 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS NUWATER to address this early, only 270 were connected in Kitgum over 3 years. 5.Did the staff have sufficient sector knowledge? According to USAID staff that Evaluation Team talked to, USAID/Uganda Mission did not have sufficient capacity in the water sector to handle NUWATER. The Mission Director observed that NUWATER was “a mistake in poor design from implementation by EGAT to impose NUWATER on USAID/Uganda. USAID/Uganda Mission didn‟t have the capacity to manage NUWATER.” 2/3 of the NUWATER staff interviewed had some background experience with the Uganda water sector. They reported to have worked with the NWSC at some point before joining the NUWATER project. The COP has no water engineering experience or utilities management experience. The engineering capacity of the Field Coordinator was perceived as low by the ET when asking technical questions in the field. PO also reported to have received “wrong” technical advice from the Field Coordinator. NWSC put their best people on NUWATER project but failed to work well with ARD COP. NWSC reported on non-revenue water and training needs but ARD COP took no timely actions. NWSC staff interviewed reported that COP did not take well any observations in their reports that did not suit the ARD COP‟s own perception. They described the COP as a “Super Manager” who speaks with arrogance. The Evaluation team learned from staff of NWSC that the NUWATER Field Coordinator acted as „technical advisor‟ to the COP and spent much of his time in Kampala instead of being in the field. Unfortunately, NWSC was not replaced with an equally competent entity when terminated in 2010. Town council officials in Pader noted that they had requested several times for community mobilization but apparently the NUWATER staff had no experience on this and had only once sent in a person from NWSC. Evaluation Team‟s findings indicate that the PO staff had little knowledge in water/ utility management. The Engineers on the team have background in construction engineering and could not well appreciate the technicalities involved in this kind of project. The team observed that the PO staff are in an on-going on job learning process that requires a lot of support and capacity building. Being a water system constructions firm, PO staff also lacked capacity in water utilities management, financial management and customer relations. Technical staff was not assigned full time on the NUWATER project but also worked on other projects. Question was not asked to water consumers 6.Did the project staff have capacity to track the intended results? USAID (and UMEMS) reported lack of timely reporting on project indicators and that reports had to be requested many times until they were finally received. . NUWATER project had a sub-contractor NWSC-ESU that was responsible for technical supervision. The team carried out monthly M&E audits of the PO to track results and made recommendations for improvements. However, the Evaluation Team observed that there was low capacity to track results at all levels. The monthly M&E reports NWSC did monthly M&E to track results. When they contract was ended by the project, no further audits were carried out to track results. MWE reported poor reporting by NUWATER to the Districts and to the Ministry, and at one point they had to call a special meeting to find out what was going on. Kitgum officials (6/7) responded to the affirmative. The explained that monthly audit reports showed that results were being tracked. However, they were not actively involved in the M&E process as to keep up the activities after the project end and requested training in M&E. The Kitgum board pointed that the M&E team carried out monthly audits of the PO. They thought the project had staff with capacity to track results. They reported that the board had learnt from the M&E team but noted that they (Water Board) do not have that capacity. The Evaluation Team discovered that PO staff have forms, check lists and activity logs (designed by the NWSC M&E team) that are used to track results. Question was not asked to water consumers 112 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS done by the NWSC highlighted critical issues which were not always addressed by the parties. In the view of the ET, NUWATER also showed a lack of adapting the indicators to the situation on the ground and providing tools to measure them; for example it took more than 8 months to get the first bulk meters to measure production capacity and all indicators related to capacity were never adjusted to more realistic production levels through carrying out pumping tests. Also, the indicator about “rural households” was not amended even if only partly applicable. This is reflected by the monitoring results, which shows zero values in 2009 on ALL indicators. In addition, base line surveys were carried out late (December 2009 for Pader and February 2010 for Kitgum), after more than half of the designed project period had passed. No answer in Pader 7.Assess the extent to which project outcomes were affected by the capacity of the project staff. If USAID had more experience in the Ugandan water sector, a number of problems could have been avoided; >the recognition of the situation in Pader would have resulted in a more drastic change in project design early on, with focus on rehabilitating and expanding the kiosk network and not promising house connections that could not be made >alternative models could NUWATER FC noted: “The project had no field staff until after about one and a half years”. NUWATER Field Coordinator was reportedly hired late, hence rehabilitation work delayed. He reported lack of capacity on the part of the PO. For example meters procured by NUWATER and delivered to PO took 3 months to install. PO was reported as failing to go out to the field to make collections, with arrear going over Ugx 100 million. 4/4 of NWSC staff pointed that lack of delegation and flexibility on the part of NUWATER COP affected the project negatively. There was lack of ingenuity in NUWATER management. NWSC said they assembled their best team on NUWATER but they were not utilized. DWD reported that their advice to NUWATER on how activities should be carried out was ignored by NUWATER. The following points were noted by various officials in both Kitgum and Pader: >Poor workmanship in Kitgum that lead to re￾excavation of trenches and cutting supply to affected areas >Over billing of customers leading to disconnection from the water system >Poor customer Pader TC officials reported a meter having been installed backwards by the NUWATER team. They also reported poor coordination by NUWATER team, for example, PO had capacity gaps, particularly in water engineering field. Evaluation Team discovered that critical staff, like Technical Supervisor had little knowledge in water engineering. This posed a risk of giving wrong advice on courses of action to run the system. For example, it was reported in the M&E report for January 2010 that the connections made in December 2009 and January 2010 had unsatisfactory depth of excavation (0.3m as opposed to the contractual requirements of 0.8m). The Question was not asked to water consumers 113 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS have been chosen that are more attractive to a town of the size of Pader than using a PO, such as a local management group supported by the District and an Umbrella organization >small improvements in the infrastructure (such as drilling new production wells) could have been done in the first months of the project and in this way, all the subsequent problems arising from poor production capacity could have been avoided >USAID should have seen the need for community mobilization in both towns, and asked the contractor to carry out such activities >USAID should have recognized that the management style of NUWATER is not conducive to sustainability because it does not give local stakeholders the possibility to be actively involved >USAID should have taken capacity building more seriously and pushed for a program and activities carried out Higher capacity in NUWATER staff could have increased the impact of the project considerably, and also the value for money; >NUWATER should have asked for an early modification of contract taking into account the fact that Pader had no functioning system suitable for management by a PO >If technical people were there from the start, targets would have been changed after pumping tests in order to make them more realistic both for M&E and for the incentive contracts >If technical people were there from the start, NUWATER would have recognized that with relatively small and cheap interventions (drilling of new production wells next to old wells) many future problems could have been avoided >Mistakes such as procuring faulty meters would have been avoided >More technical capacity and management capacity could have been built in PO and management of Pader if NUWATER staff had these skills >More competent staff would have recognized the need for complaints responses. Many customer complaints are not responded to, sometimes takes too long – leading to customer frustrations Operator had not satisfactorily rectified the problem of poor quality of workmanship for the new connection effected in November 2009. From the monthly M&E reports, Evaluation Team discovered that PO staff had a capacity gap in their accounting department. They failed to apply tax components of their operations and failed to submit accurate figures of such computations. There were some reported incidences of over -billing of customers that resulted into disconnection from the system. 114 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS capacity building and community mobilization >A better management style would have gradually transferred capacity to the local stakeholders taking into account the advice from NWSC, MWE and TC, and in this way made intervention more sustainable 8.How responsive was USAID? Supervision by USAID was not effective. Supervisors were in Kampala and could really take long to get to the field. NWSC staff reported that USAID/Uganda office had a vibrant, active staff that left - USAID then appointed a new staff to handle NUWATER but the new staff didn't have a full grasp of the project. Things could have been done faster problems avoided. According to the Kitgum officials (5/7), USAID was not very responsive. They said USAID rarely came on the ground to check on project activities. Members of the board all shared the view that USAID response was slow. They cited delays for the project to kick off as having resulted from USAID slow response to the project issues. PO said USAID was too relaxed, was not very involved in the project. PO did not feel the presence of USAID much as they expected. Question was not asked to water consumers 9.What could USAID have done differently to manage the project? Supervision by USAID needs to be improved. USAID/Gulu said they tried as field office, but the main USAID supervisions were in Kampala and could really take long to get to the field. USAID needed to have made regular checks to curb delays with day-to-day complaints. The following suggestions were put forward by staff of NWSC that the Evaluation Team interacted with: >Supervision by USAID needs to be improved. The main USAID supervisions were in Kampala and could really take long to go to the field. >USAID need to make regular checks to curb delays with day￾to-day complaints. The officials thought USAID could have done better if they did the following: >USAID should have been involved in the project by checking activities on the ground more frequently. >They should have been quick in decision making to avoid delays in implementation Board members suggested that USAID should have: >Engaged its own water experts to oversee the project so that it could have responded well to the project demands >Been quick in making decisions to speed up process. No answer in Pader PO noted that USAID could have done better by exercising its oversight role more. They pointed out that USAID was not hard enough on ARD contractor to ensure project was on track. Question was not asked to water consumers 10. Was the NUWATER management done in a way to enhance sustainability and build capacity for the future? According to USAID, the original project design looked at capacity building as a major activity. Accordingly, NUWATER would conduct training of local government officials to take on contract management role. NUWATER seemed to have not been doing this to the NUWATER staff (3/3) reported that the project carried out capacity building at various levels. In Kitgum, the TC and Water Board members were involved in the M&E exercise and attended presentation of the M&E team. NUWATER management viewed on job (hands on) learning as a more reliable way of building capacity of the Water NWSC staff (4/4) said NUWATER management was not done in a way that promoted sustainability and capacity building. They reported that there was lack of delegation on part of COP. Minimal level of capacity building was done. The do not agree that the level of capacity building carried out by NUWATER project enhances sustainability. Pader officials 5/5 thought the project is going to break down as soon as NUWATER closes because of low capacity in terms of finance and technical skills. They said they do not have financial capacity to run the system and meet other operational costs. 4 out of 7 officials in Kitgum board attended monthly presentations of the M&E team. They were happy about the strict financial control done by NUWATER and feared that as soon as NUWATER goes, the Town Council will misuse the water funds. Evaluation Team discovered that capacity building of PO was done to minimal level. PO received on-job training in financial and accounting procedures but none on technical/engineering aspects. PO pointed that NUWATER FC remained Systems Administrator of the new Billing Software and Question was not asked to water consumers 115 NUWATER End of Project Evaluation EVALUATION QUESTIONS RESPONSES/FINDINGS satisfaction of USAID. The Evaluation Team discovered that a number of officials got some form of capacity building (through attending workshops and presentations), but they were not able to verify the level of knowledge gained by the officials to take on management role. authority. In Pader, NUWATER reported that the project helped the TC to become a water authority by involving them to have the town gazetted. Members of the new water board were reportedly trained to take on their roles. However, from the Evaluation Team‟s assessment, not enough training for TC and PO was done in Kitgum. The NUWATER Field Coordinator reported having proposed this to the COP but it was not acted upon. He observed that: “Organizing workshops and trainings for the Town councils would help them appreciate the project more”. The Pader Water Board did not portray any signs of having received any forms of capacity building. In all circumstances, the Evaluation Team did not find and documented evidence of capacity building intervention. There is no evidence of assessment of knowledge gained through the interventions. In addition, the management style of NUWATER was over￾controlling not putting any responsibility on local stakeholders, in this way not building the necessary local framework for sustainability. Kitgum thought NUWATER is not going to be sustainable because the project did not build capacity of the town council to the required level to take over management. In addition, they think the water production level is too low to make it profitable for a PO. 2/7 said they believe that the project will be sustainable, given that it trained some people and the new wells will be connected to the system to boost production level. The Evaluation Team discovered that the in spite of reported trainings and workshops, the Pader Water Board still appeared ignorant of their duties. Pader Water Board was created in 2009 but has only had one meeting since then. Their presentation gave the Evaluation Team an impression that they were not well informed about NUWATER. envisaged serious technical difficulties in using the billing system after NUWATER closes operations. Due to NUWATER subsidies, the escrow account has some reserves that can be used to cover losses in the next few months. However, once this money is gone, it will no longer be profitable to run the Kitgum water system. Therefore, the PO does not wish to renew the contract once it expires in February 2012. 116 NUWATER End of Project Evaluation 117 NUWATER End of Project Evaluation APPENDIX M: NUWATER Project Final PMP Indicator Matrix Indicator Year Baseline 2009 Target 2009 Actual 2010 Target 2010 Actual 2011 Target 2011 Actual LOP Target LOP Total Achieved % Achieved IR. 1: INCENTIVE-BASED CONTRACTS FOR WATER UTILITY MANAGEMENT IMPLEMENTED 1.1. Percentage reduction in the ratio of subsidies to operation and maintenance costs 2011 0 0 0 0 0 0 0 0 0 2010 0 2009 0 1.2. Level of performance bonuses earned and provided to operators 2011 0 0 0 0 0 0 0 0 0 0 2010 0 2009 0 1.3. Number of people in targeted areas with increased access to clean drinking water as a result of usg assistance 2011 0 9,753 6,593 12,913 6,147 15,073 0 37,739 12,740 34 2010 6,623 8,273 9,993 10,913 2009 13,410 16,500 20920 24,660 118 NUWATER End of Project Evaluation Indicator Year Baseline 2009 Target 2009 Actual 2010 Target 2010 Actual 2011 Target 2011 Actual LOP Target LOP Total Achieved % Achieved IR.1.1: WATER SUPPLY QUALITY, QUANTITY AND MANAGEMENT IMPROVED 1.1.1. Improvement in record keeping standards (measured in terms of the number of times operators can produce accurate quarterly reports to authorities) 2011 0 3 0 3 0 3 0 9 0 0 2010 4 4 4 2009 6 8 8 1.1.2. Increase in collection rate (measured as a percentage collected of bills/expected) 2011 7 0 25 0 8 0 40 0 0 2010 75% 85% 90% 2009 80% 85% 90% 1.1.3. Decrease in response time (measured as the average number of days to solve a problem - billing, leak, breakages, etc.) 2011 6 0 6 2 6 0 6 2 33 2010 Average of 5 Average o f4 Average of 3 2009 Average of 5 Average o f4 Average of 3 119 NUWATER End of Project Evaluation Indicator Year Baseline 2009 Target 2009 Actual 2010 Target 2010 Actual 2011 Target 2011 Actual LOP Target LOP Total Achieved % Achieved 1.1.4. Increase in water quality 2011 1 0 2 0 2 0 5 0 0 2010 1 pass 1 re-test in order to pass 2 passes 2 passes 2009 1 pass 1 re-test in order to pass 2 passes 2 passes 1.1.5. Number of audits/M&E performed on operators to ensure accuracy and contract specifications 2011 0 5 0 12 12 12 0 29 12 41 2010 3 3 3 2009 0 8 8 6 1.1.6. Increase in water availability (measured as average number of hours per day that customers can get water from the 2011 12 0 18 0 18 0 16 0 0 2010 6 hours 12 hours 18 hours 18 hours 120 NUWATER End of Project Evaluation Indicator Year Baseline 2009 Target 2009 Actual 2010 Target 2010 Actual 2011 Target 2011 Actual LOP Target LOP Total Achieved % Achieved service) 2009 6 hours 12 hours 18 hours 18 hours IR.1.2: CAPACITY FOR LOCAL GOVERNMENT TO MANAGE COMMERCIAL CONTRACTS IMPROVED 1.2.1. Number of local government officials trained on contract management 2011 0 14 0 12 10 12 0 38 10 26 2010 0 10 10 10 2009 0 10 10 10 IR.1.1.1: TENDERED AND AWARDED INCENTIVE-BASED CONTRACTS AWARDED 1.1.1.1. Number of incentive-based contracts tendered and awarded for implementation 2011 0 2 0 2 1 0 0 4 1 25 2010 2009 IR.1.1.2: PARTIAL CAPITAL INVESTMENT FOR COST-RECOVERY INSTITUTED 1.1.2.1. Number of water points constructed or rehabilitated by USG assistance 2011 0 500 0 400 242 400 28 1,300 270 21 2010 1021 1021 1321 121 NUWATER End of Project Evaluation Indicator Year Baseline 2009 Target 2009 Actual 2010 Target 2010 Actual 2011 Target 2011 Actual LOP Target LOP Total Achieved % Achieved 2009 1,431 2,091 2,351 2011 - Database 2010 - July 2010 PMP 2009 – March 2009 PMP 122 NUWATER End of Project Evaluation APPENDIX N: WASH Consults Production Data Production Data FY 2009 Pumping Station Volume Produced September October November December KTI - 1,984 3,733 5,945 YY 1,440 1,953 1,820 930 Langa Langa 3,600 4,176 3,882 3,732 K-Flag 3,000 3,360 1,163 4,516 K-New 1,800 1,500 1,942 1,741 Total 9,840 12,973 12,540 16,864 Production Data FY 2010 (Note: shaded data suspect) Pumping Station Volume of Water Produced January Feb March April May June July August Sept October Nov Dec KTI 7,668 4,921 4,517 4,517 3,163 3,163 1,578 814 1,029 1,029 1,029 1,029 YY 1,130 705 1,768 1,768 4,930 4,930 4,523 5,185 3,765 3,765 3,765 3,765 Langa Langa 3,402 2,478 450 450 - - 1,000 886 533 533 533 533 K-Flag 3,431 3,252 2,053 2,053 18 18 3,050 3,523 2,293 2,293 2,293 2,293 K-New 2,430 1,622 2,008 2,008 1,331 1,331 2,244 3,757 2,359 2,359 2,359 2,359 Total 18,061 12,978 10,796 10,796 9,442 9,442 12,395 14,165 9,979 9,979 9,979 9,979 Production Data FY 2011 January February March April May June KTI 2143 3,141 572 2,990 3383 2671 YY 6208 5,064 635 3,700 1841 1759 Langa Langa 2046 2,225 569 1,850 1879 962 K-Flag 4148 3,254 644 2,881 1794 1227 K-New 3477 1,975 626 2,604 915 0 Total 18,022 15,659 3,046 14,025 9,812 6,619 123 NUWATER End of Project Evaluation APPENDIX O: Photographic Evidence from Site Visits Conducted New wells drilled but left unprotected from vandalism: Kitgum New wells drilled but left unprotected from vandalism: Pader Pader: Unprotected Water Meter unprotected Kitgum: A gate to one of the pump stations not secured 124 NUWATER End of Project Evaluation Kitgum: Pump house with broken roof Removed but not replaced water meter Missing meter at pump - Pader Pader: Water pump station situated next to school latrine 125 NUWATER End of Project Evaluation Figure 7: June 2, 1011, new diesel generator at pump but no manufacturer’s manual or operator training provided; school latrine a few horizontal meters from new water well - Pader Figure 9: May 30, 2011, Non-secured, un￾guarded, and non-functional water meters and elevated storage tanks (reservoirs), left; June 6, 2011, Again, non-secured and un-locked elevated storage tanks a week after Evaluation Team pointed this out to NUWATER Field Coordinator and Kitgum Action Urban Water Officer - Kitgum Figure 8: May 30, 2011, Destroyed new, high￾capacity water well, Kitgum, reportedly by vandalism; the well was unsecured or guarded; June 2, 2011, Non-secured and un￾guarded, new water well, Pader Figure 12: May 30, 2011, Water well left and water meter right without pressure gage, Kitgum 126 NUWATER End of Project Evaluation APPENDIX P: Bibliography ARD, October 2008. NUWATER Summary Activities, July-September 2008. Prepared for USAID/Uganda _______________, February 2009. NUWATER Implementation Schedule. Prepared for USAID/Uganda _______________, November 2009. NUWATER Quarterly Performance Evaluation Report of WASH Consults Limited for August-October 2009. Prepared for USAID/Uganda _______________. 2009. NUWATER, Performance Monitoring Plan. Prepared for USAID/Uganda _______________, August 19, 2009 - January 14, 2010. RE: Update of NUWATER Activities, Letters to the Ministry of Water and Environment _______________, November 2009. NUWATER, Year 2 Project Work Plan. Prepared for USAID/Uganda _______________, February 2010a. NUWATER, Consultancy Services for the Baseline Survey of Water Supply Systems Services in Kitgum. Prepared for USAID/Uganda _______________, February 2010b. Programme for Care and Marketing Training for NUWATER Project, Kitgum Town Water Supply Project, Proposed Dates, 11th-12th February 2010. Prepared for USAID/Uganda _______________, ARD - REVISED capital INVESTMENT Halted March 2010. Prepared for USAID/Uganda _______________, March 2010. NUWATER, Quarterly Performance Evaluation Report of WASH Consults Limited for the Second Quarter, November 2009-January 2010. Prepared for USAID/Uganda _______________, July 30, 2010. NUWATER, Kitgum Private Operator Performance Report - Year 1. Prepared for USAID/Uganda _______________, November 2010. NUWATER, Year 2 Annual Report. Prepared for USAID/Uganda _______________, December 17, 2010 – March 14, 2011. Chief of Party (COP) Financial Analysis, Pader Town Council Water Supply Services 127 NUWATER End of Project Evaluation _______________, April 13, 2011. Activities carried out by the Field Coordinator in Pader and Kampala, December 2009-April 2011 for NUWATER _______________, May 2011. NUWATER, Quarterly Progress Report, January-March 2011. Prepared for USAID/Uganda _______________, May 2011. NUWATER, Private Billing Software Reference Manual. Prepared for USAID/Uganda _______________, July 30, 2011. NUWATER, Performance Monitoring Plan. Prepared for USAID/Uganda Anyoli, Edward, 10 April 2011. State wants Kitgum SACCO officers charged with fraud. New Vision, Kampala, http://www.newvision.co.ug/D/8/16/751679 Association of Private Water Operators, April 2009. Responses to Informal Survey/Polling of APWO Members in Relation to the Recent Tenders for the Operation of the Piped Water Systems in Kitgum and Pader Districts, Uganda Azuba, Chris, Josses Mugabi and Yogita Mumssem, July 2010. Output-Based Aid for Water Supply in Uganda, Increasing Access in Small Towns. Global Partnership on Output-Based Aid, OBA Approaches, Note Number 35 Baguma, Raymond, October 18, 2010. Uganda gets sh470b for water, energy. New Vision, ReliefWeb report — http://reliefweb.int/node/371439 Baily, Sarah, August 2008. Perceptions of corruption in humanitarian assistance among Internally Displaced Persons in Northern Uganda. Humanitarian Policy Group (HPG) Working Paper, http://www.odi.org.uk/resources/download/1218.pdf Benoni, Anywae, 18 April 2011. Government wants Kitgum SACCO officers charged with fraud. Acholi Times, Gulu, http://www.acholitimes.com/index.php?option=com_content&view=article&id=326:govt-wants￾kitgum-sacco-officers-charged-with-fraud&catid=1:news&Itemid=2 Birungi, Machrine, March 30, 2010. Kitgum‘s Lands Office dogged by corruption scandals. Uganda RadioNetwork, http://ugandaradionetwork.com/a/story.php?s=26492 _______________, Machrine, June 28, 2010. Low targets, goals lost, Kitgum Council faces wrath of public criticism. Uganda RadioNetwork, http://ugandaradionetwork.com/a/story.php?s=27652&PHPSESSID=3f63c6b11516a2b7b879e20 2ed359fc1 Bitek, Oketch, August 31, 2009. Uganda: Kitgum, Pader Districts Get Safe Water. New Vision, http://allafrica.com/stories/200908311077.html 128 NUWATER End of Project Evaluation Businge, Conan and Jeff Lule, May 25, 2009. '30% of Ugandans lack enough food. New Vision, ReliefWeb report — http://reliefweb.int/node/310490 Butler, John W., June 2011. Personal communications. Mr. Butler is the ARD Senior Associate for NUWATER Center for Democracy and Governance, February 1999. A Handbook for Fighting Corruption. Technical Services Series, Washington, DC Central Intelligence Agency, CIA Worldfacts Book, April 28, 2011. Uganda, www.cia.gov Drakun, Chris, 14 April 2010. Museveni Orders Probe into Kitgum NAADS Funds. allAfrica.com, http://allafrica.com/stories/201004150582.html Economist magazine, May 20, 2010. A special report on water: to the last drop, how to avoid water wars Fall, Alioune, May-June 2011. Personal communications, including NUWATER, Activities carried out during the months of July, August and September 2008 and beyond. Mr. Fall is the ARD Chief of Party for NUWATER Garvey, G. and S. Kayaga, 2005. Maximizing the Benefits of Water and Environmental Sanitation: Private sector of water services to small towns in Uganda. 31st Water, Engineering and Development Centre (WEDC) International Conference, Kampala, Uganda, 2005 Government of Uganda, Ministry of Water and Environment, Water and Sanitation, March 27, 2007. Water and Sanitation Sector, District Implementation Manual, Version 1, Volume 1 Main Report _______________, 2009? Sectoral Specific Schedules/Guidelines 2009/10 Final _______________, 2009? Water and Sanitation Sub-Sector Gender Strategy (2010-2015) _______________, Water and Sanitation, Sectoral Specific Schedules/Guidelines 2009/10 Final International Resources Group, March 10, 2011. USAID/Uganda Environmental Threats and Opportunities Assessment. Prepared for USAID/Uganda _______________, Ministry of Water and Environment, September 2010. Water and Environmental Sector Performance Report, Uganda Green, Matthew, 2008. The Wizard of the Nile: the Hunt for Africa‘s Most Wanted, Portobello Books, UK 129 NUWATER End of Project Evaluation Jammal, Yahya and Leroy Jones, October 2006. Impact of Privatization in Africa: Uganda Water. Boston Institute for Developing Economies. Funded by the Norwegian Trust Fund and co-funded by Public-Private Infrastructure Advisory Facility, Managed by the Investment Climate Department and Africa Private Sector and Infrastructure Department of the World Bank Joint Sector Review, October 14-16, 2009. Agreed Minutes, The 2009 Joint Sector Review, Water an Environment Sector, Kampala _______________, Brief of Uganda Annual Joint Sector Review (JSR) of the Water and Environment Sector, Organized by the Ministry of Water and Environment, October 14-16, 2009 – Munyonyo Resort Kamakech, Gerald, October 7, 2010. Kitgum Town Water Supply Report. Email to Sudi Bamulesewa/USAID-Kampala Kitgum District Local Government, November 2007. Bid Evaluation Report, Prepared by Supplies, Services and Revenue Collection Sub-Committee Kitgum Town Water Authority and ARD, December 10, 2008. Memorandum of Understanding (MOU) Kiwawulo, Chris, December 18, 2009. Kitgum Children Hit by Nodding Disease, New Vision, ReliefWeb report — http://reliefweb.int/node/337705 Kwo, Jimmy, 24 April 2007. Kitgum Council to raze kiosks. allAfrica.com, http://allafrica.com/stories/200704231486.html Management Systems International, January 23, 2007. A Rapid Anti-Corruption Assessment Technique for USAID/Africa: Developing a Practical Checklist for USAID Missions in Africa. Prepared for USAID Mathias, Mugisa Isaac, May 28, 2011. Civil servants in Kitgum Town Council unhappy about stalling of Town Council Affairs. Welformers, http://www.weinformers.net/2011/04/30/civil￾servants-in-kitgum-town-council-unhappy-about-stalling-of-town-council-affairs/ Maxwell, Carl, May 19, 2011. Uganda drought. Personal communication National Water Supply and Sewerage Corporation (NWSC), 2009. Kitgum Water Tests _______________, External Services Unit (ESU), April 2010. NUWATER, Quarterly Technical Report, January 2010 - March 2010. Prepared for USAID/Uganda New Vision, May 2, 2010. Former IDPs Demand 350 Boreholes. ReliefWeb report — http://reliefweb.int/node/353353 130 NUWATER End of Project Evaluation Ocowun, Chris, August 18, 2010. Uganda: Kitgum District Gets Piped Water. New Vision. http://allafrica.com/stories/201008190401.html Odonga, Charles and Owot, Francis, November/December 2009. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Months of September and October 2009. NWSC ESU Unit Ocen, George Albert, February 23, 2009. Termination of Contract of Your Service with Kitgum Town Council. Letter to Lule Richard, Managing Director, TRANDINT Limited from Town Clerk, Kitgum Town Council Ocow, Franklin, April 1, 2011. Suspended Kitgum Town Clerk exposed skeletons in the council. Uganda RadioNetwork, http://ugandaradionetwork.com/a/story.php?s=32559 Otema, JB, October 2009. Public Awareness Campaign Strategy Report for Kitgum Town Water Supply: Public Awareness Campaign Monitoring and Evaluation Report for the Month of August 2009 in Kitgum Town Area _______________, February 2010a. Public Awareness Campaign Radio Talk Show Guide for Kitgum Town Water Supply Services: Public Awareness in Kitgum Town Water Supply Area Radio Talk Show in Mighty FM Radio _______________, February 24-17, 2010b. Public Awareness Campaign Report for Kitgum Town Water Supply Services: PAC Report for the Month of February 2010 Otto, Sara Okwaare, Deo Opany, Alfred Omony and Beatrice Adokorach, June 2002. Uganda Participatory Poverty Assessment Process, Kitgum District Report. ActionAid Uganda Owot, Francis and Okaronon, Edmond, October 2009. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of August 2009. NWSC ESU Unit _______________, December 2009. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of November 2009. NWSC ESU Unit _______________, January 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of December 2009. NWSC ESU Unit _______________, February 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of January 2010. NWSC ESU Unit _______________, March 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of February 2010. NWSC ESU Unit _______________, April 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of March 2010. NWSC ESU Unit 131 NUWATER End of Project Evaluation _______________, May 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of April 2010. NWSC ESU Unit _______________, July 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of May 2010. NWSC ESU Unit _______________, August 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of June 2010. NWSC ESU Unit _______________, August 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of July 2010. NWSC ESU Unit _______________, October 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of August 2010. NWSC ESU Unit _______________, December 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of September 2010. NWSC ESU Unit _______________, December 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of October 2010. NWSC ESU Unit _______________, December 2010. Monitoring & Evaluation Report on the Performance on WASH Consults Limited for the Month of November 2010. NWSC ESU Unit Pader Local District Government, April 1, 2011. RE Report on Preliminary Survey Done at Water Points, Letter to the Pader Town Council Pader Town Council, April 4, 2011. RE: Preliminary Survey Report for Water Point Infrastructure at Pader Town Council, Letter to the Ministry of Water and Environment Pader Town Council, April 4, 2011. RE: Value for the Land Surveyed for Pader Town Council Water Infrastructure, Letter to the Ministry of Water and Environment Pader Town Council, February 25, 2011. RE: Land for Town Council Water Infrastructure, Letter to the Ministry of Water and Environment Pader Town Water Authority and ARD, December 10, 2008. Memorandum of Understanding (MOU) _______________, November 2009. MOU Amendment No. 1 Republic of Uganda, April 7, 1997. The Water Act, Cap. 152 _______________, April 7, 1997. The Water Resources Regulations, No. 33/1998 132 NUWATER End of Project Evaluation _______________, April 17, 1997. Water (Waste Discharge) Regulations, No. 32/1998 Richard, Lule, February 2, 2009. Institution of Legal Procedures. Letter to M/S Lukwago and Company Advocates, Kampala, from Managing Director, TRANDINT Limited, Kampala _______________, February 24, 2009. Intervention in Management Contract of Kitgum Town Water Supply. Letter to Justice Faith Mwonda, Inspector General of Government, Kampala, from Managing Director, TRANDINT Limited, Kampala Rukare, Donald, August 2009. Uganda: Risk/Opportunity Mapping Study on Integrity and Accountability in the Water Supply and Sanitation Sector. Prepared for the Water Integrity Network (WIN) and Water and Sanitation Program (WSP). Funded by the Water and Sanitation Program Africa (WSP-AF) of the World Bank _______________, September 2009. Report on the Uganda Water Integrity Workshop: Revising the Anti Corruption Action Plan: 2nd – 3rd September 2009: Golf Course Hotel, Kampala. Prepared for the Water Integrity Network (WIN) and Water and Sanitation Program (WSP). Funded by the Water and Sanitation Program Africa (WSP-AF) of the World Bank Skilling, Heather, May 19, 2011. Personal communication, USAID/DC Swanson, Richard, October 12, 2010. Comments on NUWATER Baseline and Quarterly Reports (October 12, 2010) The Mitchell Group, May 13, 2011. Proposal for USAID/Uganda, UMEMS Technical Proposal for the Evaluation of the NUWATER Project, Volume I: Technical Proposal. Submitted to USAID/Uganda Transparency International, Corruption Perception Index 2010 Results, 2010a. http://www.transparency.org/policy_research/surveys_indices/cpi/2010/results _______________, Global Corruption Barometer Report 2010, 2010b. http://www.transparency.org/policy_research/surveys_indices/gcb/2010/results Uganda National Development Report, 2005. Overview of the Water Sector, Chapter Two (See revision of 2010) UMEMS, NUWATER data extracted from the UMEMS database, May 18th 2011 United Nations Environment Programme, 2010. Africa Water Atlas United States Agency for International Development (USAID), February 1999. Handbook on Fighting Corruption. Technical Publications Series, Center for Democracy and Governance, Bureau for Global Programs, Field Support, and Research 133 NUWATER End of Project Evaluation _______________, 2008. Section C, Description/ Specifications/ Statement of Work, Water Supply Services for Northern Uganda, Contract Number: 617-C-00-08-0014-00 _______________, March 2010. Uganda Water and Sanitation Profile _______________, October 2010. Safeguarding the World‘s Water, 2010 Report on USAID Fiscal Year 2009 Water Sector Activities _______________, March 2011. SUWASA (Sustainable Water and Sanitation in Africa) News. Issue 03 _______________ and the Ministry of Water and Environment, January 22, 2009. Memorandum of Understanding (MOU) on the Northern Uganda Water Supply Services Project (NUWATER) UWIN, Fighting Corruption in the Uganda Water Sector. http://www.waterintegritynetwork.net/page/3175 WASH Consults, November 2009. NUWATER Project, Quarterly Performance Evaluation Report, August - October 2009. Prepared for USAID/Uganda _______________, March 2010. NUWATER, Quarterly Performance Evaluation Report, Second Quarter, November 2009 - January 2010. Prepared for USAID/Uganda Water Integrity Network Secretariat (WIN-S) and the Water and Sanitation Program-Africa (WSP-Africa), August 2009a. Baseline Survey on Integrity in the Uganda Water Supply and Sanitation Sector (WSS), Executive Summary. Prepared for the Water Integrity Network (WIN) and Water and Sanitation Program (WSP). Funded by the Water and Sanitation Program Africa (WSP-AF) of the World Bank _______________, August 2009b. Baseline Survey on Integrity in the Uganda Water Supply and Sanitation Sector (WSS), Main Report. Prepared for the Water Integrity Network (WIN) and Water and Sanitation Program (WSP). Funded by the Water and Sanitation Program Africa (WSP-AF) of the World Bank WaterOrg, May 16, 2011. WaterOrg pioneers microloans for water and sanitation access: launches WaterCredit.org site for cross-sector learning. WaterCredit.org website WGMA, December 2009. Draft-Final Design Report for Detailed Engineering System Design for Pader Piped Water Supply, Pader District, Northern Uganda. NUWATER Project Wikipedia, May 12, 2011. Kitgum, Uganda _______________, May 14, 2011. Pader, Uganda