Evaluation Report Urban WASH, Juba, Southern Sudan 1 SUDAN INFRASTRUCTURE SERVICES PROJECT EVALUATION REPORT FEBRUARY 2010 This publication was produced for review by the United States Agency for International Development. It was prepared by Michael Wolfe, Management Systems International. SUDAN INFRASTRUCTURE SERVICES PROJECT EVALUATION REPORT Management Systems International Corporate Offices 600 Water Street, SW Washington, DC 20024 Contract No. DFD-I-00-05-00251-00, Task Order No. 2 Project Title: Services under Program and Project Offices for Results Tracking (SUPPORT) ACKNOWLEDGEMENT The authors thank the MSI Team in Juba, Southern Sudan, for their logistical and administrative support, allowing the evaluation to occur smoothly. Specific thanks go to San Huston for offering technical coordination, feedback and valuable insights on the urban WASH context in the region. CONTENTS ACRONYMS.......................................................................................................................ii INTRODUCTION.............................................................................................................iii 1. EXECUTIVE SUMMARY ............................................................................................. 1 2. EVALUATION INTRODUCTION AND BACKGROUND ...................................... 8 3. EVALUATION METHODOLOGY............................................................................ 10 3.1 Limitations ..................................................................................................... 12 3.2 Tools Used in Evaluation.............................................................................. 12 4. FINDINGS AND CONCLUSIONS ........................................................................... 15 4.1 LBG’s Performance to Date......................................................................... 15 4.2 LBG’s Monitoring and Evaluation Systems................................................. 24 5. STRATEGIC RECOMMENDATIONS ...................................................................... 26 5.1 Urban Water and Sanitation Infrastructure .............................................. 26 5.2 Enabling Environment .................................................................................. 27 Tables and Figures Table 1 – Summary of Evaluation Factors..................................................................................................... 1 Table 2 – Summary of Key Findings, Conclusions, and Recommendations.......................................... 3 Table 3 – Evaluation Questions and Tools Used ......................................................................................10 Table 4 –Tools Used and Hygiene Improvement Framework Areas ..................................................11 Table 5 –Evaluation Team...............................................................................................................................12 Figure 1 – Map of Infrastructure Locations................................................................................................15 Table 6 – Pipeline work performed by LBG ..............................................................................................17 Table 7 – Community Toilets........................................................................................................................20 Table 8 – PSI Multi Channel Behavior Change Hygiene Approach ......................................................22 Appendices Appendix A. Scope of Work ........................................................................................................................29 Appendix B. Findings, Conclusions and Recommendations..................................................................36 Appendix C. Data Evaluation Schedule......................................................................................................44 Appendix D. List of Key Informants Interviewed....................................................................................46 Appendix E. Records Reviewed...................................................................................................................47 Appendix F-1. Survey of Household Water Supply, Sanitation and Hygiene....................................48 Appendix F-2. Water System Observation Sheet ...................................................................................50 Appendix F-3. Modified IDF Tool (blank)..................................................................................................51 Evaluation Report Urban WASH, Juba, Southern Sudan i Evaluation Report Urban WASH, Juba, Southern Sudan ii ACRONYMS CES Central Equatoria State CHF Community Housing Fund International GOSS Government of Southern Sudan GtZ German Technical Cooperation HH Household HIF Hygiene Improvement Framework IDF Institutional Development Framework JICA Japan International Cooperation Agency KII Key Informant Interview LBG Louis Berger Group M&E Monitoring and Evaluation MDTF Multi Donor Trust Fund MHPPE Ministry of Housing, Physical Planning & Environment MoH Ministry of Health MSI Management Systems International MWRI Ministry of Water Resources and Irrigation NGO Non Governmental Organization POU Point of Use PSI Population Services International SDG Sudanese Pound SISP Sudan Infrastructure Services Project TOR Terms of Reference UNICEF United Nations Children’s Fund USAID/Suda n United States Agency for International Development Sudan Mission UWC Southern Sudan Urban Water Corporation WASH Water, Sanitation and Hygiene INTRODUCTION This evaluation report was prepared following over two weeks of data collection in Juba, Southern Sudan. The product includes a list of recommendations for future project work. Following completion of data collection, a table of Findings, Conclusions, and Recommendations was shared with MSI, USAID/Sudan, LBG (Louis Berger Group), key government staff, and other key stakeholders during an oral presentation. Comments on a draft of this report -- received before the evaluator left Juba -- were considered in drafting the final report. Evaluation Report Urban WASH, Juba, Southern Sudan iii 1. EXECUTIVE SUMMARY The evaluation revealed significant improvements in WASH (Water, Sanitation and Hygiene) conditions as a result of the project. This evaluation also revealed areas that can be improved for future USAID/Sudan WASH program work. Representatives from USAID/Sudan, Louis Berger Group (LBG) and key government departments attended an oral presentation of the evaluation findings and recommendations. Input received from these key stakeholders was incorporated into this report. TABLE 1 – SUMMARY OF EVALUATION FACTORS Factor Summary Effectiveness Water and sanitation infrastructure projects of varying size and complexity were successfully implemented. Hygiene improvement projects were also successfully implemented. The proportions of funds spent in the different types of WASH projects were appropriate, considering the current policy context, the overall WASH conditions in Juba during the duration of the project, and the varying expected durations (short-,medium-,and long-term) of responses to the identified needs. Sustainability The quality of LBG construction work appears to conform to expected technical quality standards. However, in the absence of properly functioning institutions, investments made in the WASH sector are less likely to be sustainable. Further institutional strengthening and capacity building of UWC and MWRI are critical for ensuring sustainable investments and service delivery in the WASH sector. Two major immediate impediments are preventing UWC (Southern Sudan Urban Water Corporation) from operating as a sustainable water utility: 1. UWC lacks necessary autonomy (regulatory, financial management, institutional capacity) making it vulnerable to outside political forces; and 2. Current user revenues are insufficient to cover operating costs. USAID/Sudan should strongly consider supporting institutional strengthening and capacity building of UWC and MWRI (Ministry of Water Resources and Irrigation) by providing further technical assistance in the following areas: • WASH sector policy dissemination, strategy development, and investment planning; • Design and support implementation of a UWC business model based on cost recovery for operations and maintenance costs; • Reform of the UWC tariff policy to cover operating costs and to include institutional users; • Strengthening of UWC finance and administrative systems, including: − Establishment of a UWC finance account separate from the GOSS block account; and − Enabling revenue collection from GOSS ministries, schools, and health centers. In future urban WASH program work, LBG should support UWC institutional strengthening and capacity building in some or all of the following key areas: • Contracts Management – technical oversight and contracts enforcement mechanisms; • Administration and Finance – establishment of formal policies and procedures; Evaluation Report Urban WASH, Juba, Southern Sudan 1 Evaluation Report Urban WASH, Juba, Southern Sudan 2 • Personnel/Human Resources – establishment of formal policies and procedures; and • Communication – establishment of formal policies and procedures (both external and internal). Significant long-term investment in urban water infrastructure is required to improve and expand service delivery to UWC customers. Urban sanitation coverage also remains quite low and further investment in urban sanitation infrastructure is warranted. Further hygiene behavior change programming is recommended to reinforce hygiene behavior change messaging in areas already targeted and to expand into new target urban areas. Table 2 summarizes this evaluation’s summary of key findings, conclusions, and recommendations. For mission management decision-making, those excluded from Table 2 can be considered non-essential. The others on Table 2 have been ranked as essential, meaning needed to obtain maximum results on the project, and vital meaning the project should not go forward without them. A full table of findings, conclusions, and recommendations from the evaluation is included as Appendix B. TABLE 2 – SUMMARY OF KEY FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS No Finding Source Conclusions Recommendations Priority LBG Performance – Heavy Infrastructure 1 LBG installed more pipeline than what was initially proposed. Records The amount of work actually performed by LBG increased to better address the urban water system needs, based upon the conditions encountered during various stages of project implementation. None: contract was established as a ceiling contract. N/A 4 Quality of LBG pipeline construction work appears to be acceptable. Interviews-UWC, Observations LBG effectively manages its subcontractors to ensure good quality work. Construction quality appears to have complied with UWC expectations. Continued use of local contractors for cost-effective network expansions and the creation of a competitive construction environment. This will assist the long-term expansion of urban water utilities in Southern Sudan. Essential 5 Approximately 41% of LBG’s subcontractor budget was spent on heavy infrastructure for urban water. Records The proportion of funds spent on heavy infrastructure water projects is reasonable. These projects are long-term investments in urban WASH infrastructure. Continue funding water infrastructure activities that promote long-term investment. Vital LBG Performance - Light Infrastructure 6 Construction quality of the three community river tanks appears to be acceptable. Some minor improvements can be made in drainage. Observations, Water Systems Sheets LBG manages its subcontractors to ensure acceptable quality work. None N/A 7 Construction quality of the four community toilets appears to be acceptable. Some minor improvements can be made in drainage. Observations, Toilets Observation Sheets LBG has managed its subcontractors to ensure acceptable quality work. None N/A 11 CHF (Community Housing Fund Interviews-CHF Chlorination may not regularly occur at the 3 community tanks. It is critical that LBG successfully transitions these systems over to Essential Evaluation Report Urban WASH, Juba, Southern Sudan 3 Evaluation Report Urban WASH, Juba, Southern Sudan 4 No Finding Source Conclusions Recommendations Priority International) tanker truck chlorinators are also chlorinators at the three community river tank sites. The CHF contract has finished, so chlorinators are no longer receiving payments. payam management, ensuring that user fees cover the costs of continued operations, including regular and consistent chlorination. 16 Approximately 13% of LBG’s subcontractor budget was spent on light infrastructure water projects. These projects are short- to medium￾term response activities. Records The proportion of funds spent on light infrastructure water projects is reasonable. Phase out funding for activities with short- to medium-term investment. Increase focus on more sustainable WASH programming. Vital LBG Performance - Sanitation & Hygiene 17 Responsibility for sanitation is shared among different ministries. No one ministry or government entity has the full responsibility to enact or enforce policies. There reportedly are policies at the payam level requiring construction of latrines at households. Interviews-MHPPE, MWRI, Records Policies may exist but are not being enacted or enforced. Advocate for all sanitation responsibilities to be granted to a single ministry as well as capacity building of that ministry to enact sanitation policies, strategies, and enforcement mechanisms. Essential 18 30% of people surveyed did not have access to improved sanitation. Many households with access to a latrine share the latrine with multiple HH Survey, Observations Many people lack access to improved sanitation facilities. USAID/Sudan should focus its sanitation programming on demand creation and improving the policy environment for the expansion of household sanitation coverage. Vital Evaluation Report Urban WASH, Juba, Southern Sudan 5 No Finding Source Conclusions Recommendations Priority households. 21 Cost of constructing a latrine is higher than most families are currently willing or able to pay. Interviews-multiple sources Households will not construct latrines until construction costs decline or the perceived value increases. Continue awareness raising and demand inducement. Investigate ways of lowering cost through bulk purchasing and bulk labor agreements with local masons. Essential 28 Nearly 70% of people surveyed treat their water with WaterGuard or PUR (a commercial water filtration system). HH Survey PSI has been effective in social marketing of water point of use treatment products. POU water treatment social marketing should be continued and expanded to new geographic areas. Essential 30 Many households have soap. Soap is widely used for washing clothes. Soap is not widely used for hand washing at crucial times. Observations, HH Survey Households already have soap but are not always using it for hand washing at crucial times (before eating and after toilet use). Increase hygiene behavior change messaging emphasizing importance of hand washing with soap. Essential 31 Hygiene promotion activities are not specifically aligned with areas where LBG heavy water infrastructure work is occurring. Observations, Interviews￾PSI, LBG Community members using new water infrastructure will not necessarily have exposure to improved hygiene behavior trainings. Specifically target areas benefiting from water infrastructure with hygiene behavior change trainings. Essential 33 Approximately 46% of LBG’s subcontractor budget was spent on sanitation (11%) and hygiene (35%) projects. Records The proportion of funds that have been spent on sanitation corresponds with the current enabling environment. The proportion of funds that have been spent on hygiene represents a significant investment in promoting hygiene behavior changes. Continue funding hygiene behavior change activities. Increase funding for sanitation activities when enabling environment is improved. Essential LBG Performance -UWC Capacity Building 34 80% of tariffs collected by UWC go into a large GOSS pool. No mechanism exists for UWC to use directly Interviews-MWRI, UWC It is difficult for UWC to receive funds necessary for ongoing operations and maintenance. UWC will remain vulnerable to GOSS political forces, unless sectoral mandates are clarified and UWC capacity is increased. USAID/Sudan should consider providing technical assistance supporting sector policy dissemination, strategy development, investment planning, and institutional Vital Evaluation Report Urban WASH, Juba, Southern Sudan 6 No Finding Source Conclusions Recommendations Priority these funds. UWC is challenged by a weak institutional mandate, inadequate systems and a general lack of capacity. strengthening. The establishment of a UWC finance account separate from GOSS block account is essential for the long term operations of a water utility. 35 UWC does not know the gap between its operating expenses and its tariff revenues, but it is widely thought that revenues from users are insufficient to cover operations and maintenance costs. Interviews-UWC, Records It will be difficult for UWC to sustain operations in the short term without GOSS subsidy funds. UWC needs support developing a comprehensive business model for the long-term sustainability of the water utility. USAID/Sudan should consider providing technical assistance supporting establishment of a working business model for UWC. Such a business model should be based on cost recovery for operations and maintenance costs. Essential 36 Less than 20% of the Juba population is serviced by a UWC connection. Interviews-MWRI, UWC Much further work on the urban water system is needed to service the needs of the population. Focusing on expansion of the network should be prioritized, following successful reduction in non-revenue water loses Essential 41 Formal UWC administrative procedures either do not exist or are dated to colonial times. Informal systems exist. IDF tool, Interviews at UWC, Records Informal systems are being used. Formal administrative policies and procedures should be established. USAID/Sudan should consider supporting capacity building of administrative systems. Vital 42 Formal UWC personnel systems either do not exist or are dated to colonial times. Informal systems exist. IDF tool, Interviews at UWC, Records Informal systems are being used. Formal human resources policies and procedures should be established. USAID/Sudan should consider supporting capacity building of human resources systems. Essential 43 Dissemination of information to the public does not systematically occur. IDF tool, Interviews at UWC, Records Public is ill informed of activities and actions of the UWC. Formal external communications policies and procedures should be established. USAID/Sudan should consider supporting capacity building of communications systems. Essential Evaluation Report Urban WASH, Juba, Southern Sudan 7 No Finding Source Conclusions Recommendations Priority 48 35% of people surveyed expect that government will take care of problems with their water source. HH Survey People expect government to provide water services and take care of problems that arise. This creates an enabling environment for a workable business model. USAID/Sudan should consider providing technical assistance supporting establishment of a working business model for UWC to expand provision of water services and address any problems that arise in services. Essential 49 MWRI has requested support in policy setup, regulatory framework, and institutional development. Interviews-MWRI MWRI currently lacks capacity in these key areas of its responsibility. USAID/Sudan should consider supporting a Technical Advisor to be placed in MWRI office to assist with institutional development and regulatory policy framework of MWRI. Essential LBG Monitoring & Evaluation 54 Beneficiaries of point of use water treatment products are listed in the same table as number of people with improved access to safe water supply. Records Incorrect reporting. Point of use water treatment does not equal increased access to improved water supply. LBG needs to revise its reporting on this standard indicator. Point of use data should appear in a different table than number of people with access to improved water supply. Essential 55 Beneficiaries for community toilets are listed in table for people with improved household access to sanitation. Records Incorrect reporting. The standard indicator for sanitation does not include users of public toilets. LBG needs to revise its reporting on this standard indicator. Users of communal toilets should appear in a different table than number of people with access to improved sanitation. Essential 56 Target beneficiaries for water infrastructure are calculated using SPHERE (Humanitarian Charter and Minimum Standards in Disaster Response) guidelines for maximum number of users. Records SPHERE guidelines are not applicable for estimating number of beneficiaries of water infrastructure. LBG should revise the target beneficiaries based upon calculations of the amount of water being supplied. Essential 2. EVALUATION INTRODUCTION AND BACKGROUND Sudan is the largest country in Africa, borders nine countries, and has a population estimated at 40 million. The current population of southern Sudan is estimated at 10-12 million people, with an estimated four million others displaced to northern Sudan and living as refugees outside the country. Southern Sudan suffered from decades of underdevelopment, war, famine, drought and flood, resulting in the devastation of Southern Sudan’s economic, political and social structures. The southern Sudanese people lack basic health and education services and the infrastructure needed to build a thriving economy and functioning state. Since the mid-1990s, non-governmental organizations, faith-based organizations, and international humanitarian relief agencies have been the prime providers of an array of much needed services. After decades of civil war, parties to Sudan’s north-south civil war signed a Comprehensive Peace Agreement (CPA) in January 2005. The CPA is comprised of six interlocking agreements related to wealth and power sharing, the establishment of the Government of National Unity that provides southern Sudanese with representation in the national government, a resolution of conflicts in the oil-rich border regions between northern and southern Sudan, the creation of a southern Sudanese government and state governments, and the establishment of a six-year interim period after which the citizens of southern Sudan will vote on whether to remain a part of unified Sudan or secede to create an independent state. Southern Sudan achieved some level of security and political stability since the signing of the CPA. However, with the large influx of returning refugees, with little basic infrastructure, and with inadequate employment creation, Southern Sudan remains a fragile region. USAID/Sudan created the Sudan Infrastructure Services Project (SISP) funding mechanism to assist the people of Southern Sudan with increased access to infrastructure. USAID/Sudan works with the Government of Southern Sudan (GOSS) to assist in the formation of core government institutions and systems. Since the signing of the CPA, USAID has funded significant amounts of programming in the Water, Sanitation and Hygiene (WASH) sector. Typically, WASH funding has been divided between urban and rural areas, with the three major cities of Juba, Wau and Malakal being classified as urban centers. Despite each of these cities being garrison towns during the civil war, urban water systems continued operating in limited portions of each town. To date, USAID’s urban WASH program focuses principally on Juba. In 2006, Juba experienced a major cholera outbreak that resulted in over 4000 confirmed cases. In February 2007 USAID performed an assessment of WASH conditions in Juba, which resulted in recommendations for further WASH interventions. In September 2007, Louis Berger Group (LBG) signed Task Order #4 of SISP to develop improved urban access to clean water and improved access to sanitation. The funds committed to date are $6,200,000 of a $14,000,000 ceiling. The program description of Task Order #4 can be found in Annex 1 of the TOR, which is included in Appendix A of this report. The influx of returnees to Juba has overwhelmed the existing limited water and sanitation infrastructure of the city. Most areas of Juba are not serviced by the urban water system. Prior to the start of this project, the majority of the population relied on untreated water obtained directly from the Nile River. Human waste disposal has been largely uncontrolled. Limited access to safe water and sanitation infrastructure, coupled with poor hygiene practices has led to high levels of water borne illnesses. Rapid population growth has exacerbated the situation. LBG’s Juba WASH project activities take place within the context of Juba’s ongoing rapid expansion, which is fraught with complexities and challenges. A mixture of short- to medium-term emergency response activities and longer-term infrastructure development activities were performed to address the various WASH needs. LBG’s project activities have included the following components: • Water infrastructure (long-term improvements to UWC system); Evaluation Report Urban WASH, Juba, Southern Sudan 8 • Quick impact and short- to medium-term water improvement projects; • Sanitation infrastructure – public and household projects; • Hygiene – community-focused behavior change programming; • UWC (Southern Sudan Urban Water Corporation) Capacity Building – institutionally-focused capacity building. The evaluation was performed at approximately the midpoint of a three-year award under Task Order #4. The next phase of the project targets Wau. USAID/Sudan expects the evaluation to provide data to assist the Mission to: • Capitalize on lessons learned and successes for future investments in urban water systems in Sudan; • Ensure that the program is clearly achieving results within the management capabilities of the Juba UWC and that the necessary management tools are in place to accommodate further improvement in the Juba water systems; • Ensure that future urban WASH programming in Southern Sudan would have the maximum possible positive impact on the urban health environment, even considering the endless series of changes and unique challenges to working in Sudan. The objectives of this evaluation are to: 1. Identify and, if possible, measure the impact and successes of LBG’s program activities to date vis-à￾vis the program objectives, namely: a. Building the organizational capacity and development of UWC; b. Improving the distribution of water via the new connections made in Kator and Munuki; and c. Increasing the effectiveness of the UWC in providing improved water quality and access to water to customers in Juba town; 2. Assess the sufficiency of data collection with respect to: monitoring, program design, program input, and program context; and, 3. Make recommendations, as needed and within the context of current program funding, to: a. Improve the management and organizational capacity of the UWC so that it will be able to improve and sustain its service delivery; and b. Strengthen the collection of program input, context, and performance data to enable robust life-of-program performance monitoring and end-of-program evaluations. Evaluation Report Urban WASH, Juba, Southern Sudan 9 3. EVALUATION METHODOLOGY Questions the evaluation sought to answer and the tools used to address the questions are listed in Table 3. TABLE 3 – EVALUATION QUESTIONS AND TOOLS USED No Evaluation Question Tool Used 1 LBG Performance - To Date 1.1 What is the effectiveness and sustainability of LBG's implementation of heavy infrastructure (pipeline extension and urban water system design) for the Juba water distribution system? KIIs (Key Informant Interviews), Records Review, Observations 1.1 Evaluate the cost-benefit of LBG heavy infrastructure programming on the overall water and sanitation situation in Juba. Records Review 1.2 What is the effectiveness and sustainability of LBG's implementation of light infrastructure (ablution blocks and emergency river treatment facilities)? KIIs, Records Review, Water Systems Sheets, Toilet Observations Sheets 1.2 Evaluate the cost-benefit of LBG light infrastructure programming on the overall water and sanitation situation in Juba. Records Review 1.3 What is the effectiveness and sustainability of LBG's Juba household latrine program? Toilet Observation Sheets, KIIs, Records Review 1.3 What is the effectiveness and sustainability of LBG's Juba community hygiene improvement initiative? Household Surveys, KIIs, Observations, Records Review 1.3 Evaluate the cost-benefit of LBG community focused hygiene promotion on the overall water and sanitation situation in Juba. Records Review 1.4 What are the critical institutional constraints on the UWC and government authorities on sustainably and effectively managing the Juba water facility? KIIs, IDF, Records Review 1.4 Has the LBG program design focused on these institutional constraints and if so how successfully have they been addressed? KIIs, Records Review 2 Monitoring and Evaluation Are there gaps in data currently being collected that would need to be filled in order to successfully fulfill LBG’s scope of work? KIIs, Records Review Is the current M&E system sufficient for monitoring implementation and incremental impact, USAID/Sudan/GOSS reporting, and managing program performance? KIIs, Records Review 3 Strategic Recommendations 3.1 What programmatic adjustments should be made to maximize the effectiveness of LBG’s work with the UWC? Covered by 1.4 3.1 In what areas has LBG been particularly effective at contributing to improvements in the Juba water and sanitation? Covered by 1.1, 1.2, 1.3, and 1.4 3.1 In which areas does more focus need to be placed on to improve the effectiveness of water and sanitation programming in Juba? Covered by 1.1, 1.2, 1.3, and 1.4 3.2 What programmatic adjustments should be made to maximize the sustainability of LBG’s work with the UWC? Covered by 1.4 Evaluation Report Urban WASH, Juba, Southern Sudan 10 Evaluation Report Urban WASH, Juba, Southern Sudan 11 No Evaluation Question Tool Used 3.2 What adjustments should be made to address the long term financial sustainability of the UWC? Covered by 1.4 3.2 What adjustments should be made to address how LBG supports the administrative needs of the UWC? Covered by 1.4 3.2 What adjustments should be considered to improve LBG’s support to the management capacity of the UWC? Covered by 1.4 USAID’s Hygiene Improvement Framework provided the analytic optic. The Hygiene Improvement Framework has three core components: 1. Improving Access to Water and Sanitation Infrastructure; 2. Promoting Hygiene Behavior Changes; 3. Strengthening the Enabling Environment (policy framework, institutional capacity). Particular focus was given to examining the effectiveness and perceived sustainability of project activities. Table 4 lists the tools used and the relevant core component of the Hygiene Improvement Framework that the tool examined. TABLE 4 –TOOLS USED AND HYGIENE IMPROVEMENT FRAMEWORK AREAS Tool Name HIF 1 Infrastructure HIF 2 Behavior Change HIF 3 Enabling Environment Effectiveness Sustainability Data Collected From Key Informant Interviews Project stakeholders Records Review Reports, Project Records, Secondary Sources Water Systems Observation Sheets Communities Toilet Observation Sheets Communities Institutional Development Framework UWC staff Household Surveys Communities Observations Project stakeholders, Communities The evaluation was performed in two steps, as follow: 1. Collection of data in Juba (in communities and through interviews with key stakeholders) from August 18 – September 2, 2009; and 2. Receiving input/feedback from USAID/Sudan, LBG, government staff, and other project stakeholders at an evaluation debriefing on September 4, 2009. The evaluation assessed the project’s effectiveness at improving access to safe water, improving access to sanitation facilities, and improving hygiene practices in Juba. Sustainability of project activities was also assessed to determine whether activities made possible by USAID/Sudan in the Juba urban WASH sector were considered likely to continue into the future. Quantitative and qualitative data were obtained. Data were obtained from several different sources using various tools. A description of the tools used in the evaluation follows in section 3.2. Prior to the start of data collection, a half-day team planning meeting was held at the MSI office in Juba to discuss the purposes of the evaluation, review and approve the evaluation tools, and work out a logistical schedule for the evaluation. Table 5 lists the evaluation team. The evaluation schedule as performed is included in Appendix C. TABLE 5 –EVALUATION TEAM Name Affiliation Role Michael Wolfe MSI WASH Technical Consultant Evaluation Leader Sam Huston MSI Water and Sanitation Advisor for USAID Evaluation Coordinator 3.1 LIMITATIONS Limitations experienced during the evaluation were quite minor and are not judged to have significantly affected the findings of the evaluation. The following limitations were experienced: 1. Some government officials originally targeted for interviews were repeatedly unable to keep pre￾scheduled appointments; and 2. Due to limitations in time, it was not deemed worthwhile to inspect all of the slabs for household latrines. 3.2 TOOLS USED IN EVALUATION A description of each of the tools used in the evaluation follows. 3.2.1 Key Informant Interviews Key Informant Interviews were the main source of data in this evaluation. KIIs were conducted with individuals who had direct implementation roles in the project such as LBG staff, USAID/Sudan staff, officials from UWC and MWRI, and other government officials. KIIs were also conducted with selected LBG subcontractors, current and former government officials at various levels, and other active implementers in the WASH sector. The Evaluation Leader conducted the key informant interviews. A list of individuals interviewed is contained in Appendix D. One of the main limitations of KIIs is the potential bias of the individual being interviewed. To attempt to mitigate these potential biases, effort was made to interview a broad range of individuals, both from different organizations and also from within the same organization. Data obtained from interviews was also triangulated with data collected using other tools and from direct observations made in the field. Evaluation Report Urban WASH, Juba, Southern Sudan 12 3.2.2 Records Review Reports produced by others and project records were reviewed. A listing of documents reviewed is contained in Appendix E. 3.2.3 Water Systems Observation Sheets This tool assessed the quality of construction, the reliability of service, and the ability of users to maintain the water systems. Eleven different aspects of each water system were rated, each on a scale of 0-2. Rating of the aspects was somewhat subjective. The water systems observations sheets were completed by the Evaluation Leader. All three community river treatment tank systems were assessed. A blank water systems observation sheet is contained in Appendix F. 3.2.4 Toilet Observation Sheets This tool assessed the quality of construction, the use of the toilet/latrine, and hand washing. The toilet observation sheets were used for the ablution blocks/community toilets and the household latrines. Eleven different aspects of each water system were rated, each receiving either a positive (satisfactory) or negative (unsatisfactory) score. Rating of the aspects was objective. The toilet observation sheets were completed by the Evaluation Leader. All four ablution blocks/community toilets were assessed. Four household latrines were assessed. A blank toilet observation sheet is contained in Appendix F. 3.2.5 Modified Institutional Development Framework Tool This tool assessed 11 different organizational capacity features (e.g. management systems, human resources systems, financial management). Each capacity feature was rated at one of four stages along an organizational development continuum, to assess the current stage of institutional development of the organization. The tool was used by the Evaluation Leader in interviews with senior officials from the UWC. In addition to conducting interviews with senior UWC officials, additional documentary evidence (i.e. records and reports) was sought to assist in completing the IDF. Interviews with other key informants were also used to corroborate information collected from UWC officials. The blank modified IDF tool is contained in Appendix F. Prior to the start of the evaluation, several different tools for assessing organizational capacity were reviewed (Refer to USAID’s Tips No. 15 Measuring Institutional Capacity). The IDF was selected to be the most conducive to assessing the organizational capacity of the UWC. Upon arriving in Juba, it was decided that conducting the full IDF as contained in Tips No. 15 would be too advanced for the current stage of UWC. It was judged that the modified IDF would be more applicable. It is acknowledged that ratings with the modified IDF tool are somewhat subjective. For some of the categories rated, it was judged that the UWC currently had characteristics applicable to more than one IDF stage of development. In these instances, usually the UWC had some characteristics of one stage and some characteristics of another stage, but not a clear weighting in either stage of the IDF. The UWC is in a nascent stage of its development. For some of the IDF categories that were rated, it was judged that even the modified IDF was too advanced for the UWC. 3.2.6 Household Surveys A survey sheet was designed to provide indication of hygiene and sanitation behaviors amongst a small sample of the population. The survey was performed at the individual level, with staff from PSI meeting one￾on-one with community members to fill out the survey sheet. The survey was comprised of questions and specific observations that supported the questions. Prior to data collection the surveys were reviewed and Evaluation Report Urban WASH, Juba, Southern Sudan 13 tested by PSI staff to ensure proper understanding of the survey. A blank survey sheet is contained in Appendix F. Forty household surveys were conducted in the targeted areas in each of the three payams. The surveys were conducted in areas near to the water and sanitation infrastructure facilities constructed by LBG. It is assumed that many of the survey respondents are beneficiaries of the LBG water and/or sanitation infrastructure. The evaluation team sought to survey as many individuals as possible in the time available. The evaluation team also sought to interview different types of individuals within the communities. The survey sheets enabled desegregation between men and women and also between youth, adults, and elderly. Data were directly entered on the questionnaires and was compiled at the end of surveying. It is acknowledged that the number of surveys completed does not provide statistically significant representation of hygiene behavior practices across Juba. However, it is judged that the results from the surveys – in conjunction with other data gathered during the evaluation – provide an indicative snapshot of practices in some of the communities where PSI has targeted behavior change messaging. Results from the survey also corroborate observations made in the field by the Evaluation Leader. Evaluation Report Urban WASH, Juba, Southern Sudan 14 4. FINDINGS AND CONCLUSIONS This evaluation yielded 63 main Findings, Conclusions, and Recommendations, which are contained in Appendix B. Findings in the table and in this section of the report are organized in different categories according to the pertinent evaluation questions under which the finding was made. 4.1 LBG’S PERFORMANCE TO DATE LBG’s project outputs thus far can be grouped into different components. Due to uncertainties at project inception regarding interventions in the WASH sector by other donors and acknowledging Juba’s rapidly changing context, specific project activities were not spelled out by USAID at the beginning of the project. Specific activities developed during the course of the project, upon consultation with various government officials. For the purpose of this evaluation, LBG’s activities to date have been grouped into the following components. • Heavy Infrastructure (pipelines and urban water system design) • Light Infrastructure (ablution blocks and emergency river treatment facilities) • Sanitation (household latrines) • Hygiene (community-focused soft programming) • UWC Capacity Building (institutionally-focused soft programming) Figure 1 shows the locations of the major infrastructure activities that were performed. FIGURE 1 – MAP OF INFRASTRUCTURE LOCATIONS Evaluation Report Urban WASH, Juba, Southern Sudan 15 Evaluation Report Urban WASH, Juba, Southern Sudan 16 LBG completed construction of water and sanitation infrastructure projects of varying size and complexity. Representatives from the UWC and MWRI (Ministry of Water Resources and Irrigation) stated their satisfaction with the quality of the works performed. The infrastructure works that were directly observed during the evaluation appeared to be of acceptable technical quality. LBG sought to involve community members in the decision making and management of the water and sanitation infrastructure. LBG consulted with various government officials of different government levels, departments, and locations throughout the project, depending on the type of infrastructure being constructed. Especially during the beginning stages of the project, LBG made modifications to the proposed works based upon input received from various government officials. LBG involved the UWC in each stage of planning and construction. LBG engaged local level government officials in the long term operations and maintenance of the relevant infrastructure items. LBG was less effective in areas of institutional capacity building of the UWC. Although LBG helped to build capacity of UWC technical staff through on-the-job training during pipeline construction, LBG has not strengthened the institutional operations of UWC. LBG did not systematically seek to strengthen the UWC’s ability to function in areas of administration, financial management, human resources, or communications. It should be noted that these areas of institutional strengthening and capacity building fall outside the traditional strengths and experience of LBG as a large contracting organization. It should also be noted that institutional capacity building activities were not specifically demanded of LBG in the stages of the project performed. Such activities are planned for future stages of the project that are to be performed in Wau. 4.1.1 Heavy Infrastructure – Water Working in close conjunction with the UWC, LBG performed works to improve the Juba urban water system. These works are long term investments in urban WASH infrastructure. Approximately 41% of the LBG subcontractor budget has been spent on heavy infrastructure water works. Given the rapidly changing context of Juba during implementation of the project, and the mixture of needs for both short term emergency water interventions and also longer term investments in urban water systems, it is deemed that a reasonable proportion of project funds have been spent on heavy infrastructure water projects. Representatives from the UWC and MWRI stated their satisfaction with the quality of the pipeline construction work performed. During the evaluation, major aboveground systems components and a portion of the pipeline installation works were directly observed. The portions of the urban water system that were directly observed appeared to be of acceptable technical quality. LBG has effectively managed its subcontractors to ensure good quality work. LBG installed more pipeline than what was initially proposed. The work of LBG was increased to better address the urban water system needs, based on conditions encountered during various stages of project implementation. At the time of the evaluation LBG had completed the following activities in Juba: ABOVE: Removal of old 1930’s pipeline in preparation for the installation of the new pipeline. In Kator: • Installation of 200 mm diameter pipe main lines and 150 mm and 100 mm diameter distribution lines; • Construction of a pump house with two pumps rated at 50 m/hr; • Construction of a generator house with an 88 kVA generator; • Installation of a 10,000 L fuel tank with secondary containment; • Rehabilitation of a 350 m3 underground concrete reservoir; • Rehabilitation of a 250 m3 elevated steel tank; and • Construction of four new tap stands (24 new taps in total), with valve boxes and water meters Note: All of the above items are fenced In Munuki: • Rehabilitation of a 32 m3 elevated steel tank; • Installation of 200 mm diameter pipe main lines; and • Rehabilitation of 8 tap stands (24 new taps total), with valve boxes and water meters. Southern Sudan does not have official construction standards for pipeline installation. Nevertheless, based upon conversations with UWC staff and upon site observations, the quality of LBG’s pipeline construction work appears to be of acceptable technical quality. LBG also gave thoughtful consideration to future expansion and development of the UWC system in Kator. The pump house can accommodate an additional pump. The two existing pumps are designed to be run one at a time, but the system has been designed such that it would be possible to run the pumps in series, if future demands on the system necessitate such operations. These measures enable future expansion and development of the UWC system in Kator, which means increased likelihood of improved sustainability of the investment made to the Juba urban water system. LBG installed far more pipeline than initially proposed, as reflected in the work plans. The following table lists the originally-planed pipeline work and the actual pipeline installations performed by LBG. TABLE 6 – PIPELINE WORK PERFORMED BY LBG Original Plan Actual Performed Description Length (m) Diameter (mm) Length (m) Diameter (mm) Kator Main Line 1000 200 1375 200 Munuki Extension Main Line 0 0 409 200 Kator Distribution Lines 0 0 1889 150 Kator Tap Stands 0 0 541 100 Munuki Main Line 1000 200 1550 200 TOTAL 2000 5764 Evaluation Report Urban WASH, Juba, Southern Sudan 17 Evaluation Report Urban WASH, Juba, Southern Sudan 18 LBG increased its work to better address the urban water system needs, based upon conditions encountered during construction and upon input received by the UWC. In Kator, the original plan did not include the section of main line between the Konyo Konyo underground reservoir and the existing 250 m3 elevated steel tank. Upon further investigation this main line was deemed to be in poor condition, which would have resulted in severe leakages. UWC requested LBG to install an additional 375 meters of main line between the reservoir and the tank. The UWC also requested LBG to install distribution lines and four tap stands (24 taps total) in Kator, to increase access to safe water supply for Kator residents. During construction of the Munuki main line it was discovered that the existing asbestos cement pipes running from the MDTF (Multi Donor Trust Fund) program tank to the proposed start point of the LBG work on the Munuki main line were in worse shape than what was anticipated by the UWC. LBG agreed to replace the existing 550 meters of asbestos cement pipes. LBG’s contract was established as a ceiling contract and the total cost of works performed thus far has been within the established ceiling, so the additional pipeline work that has been performed has not come with additional cost implications for USAID/Sudan. 4.1.2 Light Infrastructure Various quick impact and medium-term light infrastructure projects have been completed to address emergency water needs and improve sanitation in marketplaces. LBG constructed three community tank river treatment facilities and four ablution blocks/community toilets and has rehabilitated 51 boreholes. Emergency chlorination of tanker trucks that withdrew water from the Nile River for distribution throughout Juba has also been completed. Further description of the community tanks, community toilets, boreholes, and tanker truck chlorination follows. Approximately 13% of the LBG subcontractor budget has been spent on light infrastructure projects. These projects were a combination of emergency response quick impact measures and medium-term infrastructure projects. Given the rapidly changing context of Juba during implementation of the project, and the mixture of needs for both short-term emergency water interventions and longer-term investments in urban WASH, it is deemed that a reasonable proportion of project funds have been spent on light infrastructure projects. 4.1.2.1 Community Tanks Three community tank river treatment facilities were installed at locations selected through discussions with officials from MWRI, UWC, Kator Payam, and Juba Payam, upon consideration of available land and water needs of nearby community members. The purpose of the tanks is to provide safe water to people living close to the Nile River. Prior to the installation of the tanks, these community members obtained water directly from the river. Each of the three community tank river treatment systems contains four 10,000 L tanks connected in series. The first tank allows sedimentation. Chlorine is added into the second tank. The systems are designed to allow sufficient chlorine mixing and contact time prior to exiting the tanks at the tap stands. The facility at the Juba bridge has 12 taps. The other two systems have six taps. The construction quality of the three community tank river treatment systems appears to be acceptable. LBG manages its subcontractors to ensure acceptable quality work. During the site visits, it was observed that minor modifications could be made to improve drainage, but it is not deemed that these modifications are ABOVE: Local water user collecting clean w at the community tanks. ater crucial to the operations of the tanks. Results from the Water Systems Observations Sheets are contained in Appendix G. LBG attempted to establish and facilitate water users groups among community members living near the tanks. Informal committees had already existed prior to the start of the project. LBG mobilized these existing committees and conducted trainings in management and operations. Reportedly, interest of the committees to undertake responsibilities for operations and maintenance was rather low. Several times when LBG staff sought out committee members, these could not be located or were no longer interested in managing the water tanks. The committee members were not paid for their work, so had little incentive to donate their time to manage the tanks on behalf of the community. LBG staff sought the assistance of local government officials to help mobilize the community members and create some means of reliability amongst the committees. After repeatedly mobilizing in the communities and training new committee members who would usually not take an active or reliable role in the management of the tanks, LBG concluded that the community members were not genuinely interested in voluntarily taking on the responsibilities involved with managing the tanks. LBG changed approach to have management of the tanks be the responsibility of the payam pubic health office. LBG conducted numerous meetings over a period of several months with officials from both payams to arrange management plans and working business models to facilitate the sustainable operations of the three community tanks. At the time of the evaluation, management of the tanks was transitioning from LBG to the payams and user fees were not yet being collected. However, the payams appeared to have workable management plans and business models based upon collection of fees. Two of the community tanks are located in Kator Payam. One of the tanks is located in Juba Payam. Both payam public health offices planned to begin collecting users’ fees within the upcoming month. Fees collected from users of sanitation or water infrastructure by the payam public health department go into a general payam block account. There is less incentive for the public health department to properly collect fees if the public health department is unable to manage the fees. This could have implications on the sustainable operations and management of the tanks. At the time of the evaluation, chlorination was occurring at the tanks, but there was uncertainty regarding the consistency and regularity of chlorination activities. Prior to August 2009, the chlorinators at the three community river tank sites were paid by CHF, as part of the same subcontract to CHF for payment of chlorinators at the tanker truck filling sites. At the time of the evaluation, the CHF contract had finished, so the chlorinators were not receiving payments. In the absence of proper motivation and management, chlorination may not regularly and consistently occur. 4.1.2.2 Community Toilets Four community toilet and ablution facilities were constructed at marketplaces that were selected by payam officials. The purpose of the community toilets is to increase access to sanitary facilities in public areas. In the early stages of the project, government officials were concerned that poor sanitation conditions in public marketplaces, where large numbers of people are regularly present, could lead to disease outbreaks. Officials from the Ministry of Health requested that community toilets be constructed. Each community toilet has separate blocks for men and women, each block containing toilet stalls and shower stalls. Hand washing facilities are also present. Soap and toilet paper are provided. The construction quality of the four community toilets appears to be acceptable. Table 7 lists data on the four community toilets. Evaluation Report Urban WASH, Juba, Southern Sudan 19 TABLE 7 – COMMUNITY TOILETS Location Payam Female toilet Male toilet Female shower Male shower Ave users/day Atlabara Kator 2 2 2 2 29 Jebel Kator 3 3 3 3 154 Gabat Juba 3 3 3 3 12 Juba Juba 3 3 3 3 61 *Figures are from August 17-21 LBG managed its subcontractors to ensure acceptable quality work. Results from the Toilet Observations Sheets are contained in Appendix G. LBG conducted numerous meetings over a period of several months with officials from both payams to arrange management plans and working business models to facilitate the sustainable operations of the four community toilets. The Kator Payam has experience managing community toilets in the past and has a realistic business plan in place. The Juba Payam has a working business model but no previous experience managing community toilets. The toilets were handed over to payam management within the past one or two months, so it is still the early days of payam management. At the time of the evaluation, a caretaker was observed present at each of the community toilets. Users are charged 1 SDG for use of the facility. The caretaker records number of users, collects fees, stocks consumables supplied, and ensures cleanliness of the facility. The current number of users at the Jebel and Juba toilets is calculated to be sufficient to meet operating expenses. The current number of users at the Atlabara and Gabat toilets is calculated to be lower than the amount needed to meet operating expenses, however these toilets have been open for a short time and it is anticipated that usage will increase over time. During construction (rehabilitation of an existing public toilet) of a fifth toilet in Konyo Konyo market, LBG was informed that the toilet was located in an area scheduled for demolition. LBG had received approval for the site from payam officials, but apparently the payam officials were not properly aware of the plans for demolition made by the Ministry of Housing, Physical Planning and Environment. This is judged to be a failure of communication between government departments during a period of rapid changes in Juba. As demarcation of areas scheduled for demolition is nearly completed, it is not anticipated similar problems will arise in future work in Juba. It is noted, however, that similar challenges could arise in other urban areas that have not yet been formally demarcated. 4.1.2.3 Rehabilitation of boreholes Fifty-one (51) boreholes, located throughout Juba, have been rehabilitated as an emergency-response quick-impact measure. The Central Equatoria State Rural Water Supply Department advised on the locations of boreholes to be rehabilitated. At each of the boreholes, water management committees were trained in hand pump operations and basic repair. User fees are currently not being collected, as the Central Equatoria State Rural Water Supply Department discourages fees being ABOVE: Ablution block containing shower and latrine facilities for both men and woman. Evaluation Report Urban WASH, Juba, Southern Sudan 20 collected from users. This informal policy is not in line with the GOSS Water Policy that was approved in November 2007. 4.1.2.4 Tanker truck chlorination Currently many residents of Juba pay taker trucks to deliver water to their homes and businesses. The source of the water is the Nile River. In the past, the river water was delivered untreated. As an emergency response quick impact measure, LBG established chlorination of the tanker trucks at three sites on the river where tanker trucks were drawing water. Upon further observations and consultation with government officials, it was concluded that the chlorination of tanker trucks should expand to seven locations where the trucks were withdrawing water. LBG subcontracted CHF to establish and manage a tanker truck chlorination program at the seven tanker truck filling sites. As of June 2008, CHF paid teams of chlorinators to be present at the tanker truck filling locations seven days a week from dawn to dusk. Chlorination of tanker trucks was a temporary measure that was intended to occur only until the rehabilitations of the Juba UWC treatment plant had been completed. Rehabilitations of the Juba water treatment plant were funded as part of an MDTF (Multi Donor Trust Fund) program. In March 2009 the rehabilitations to the treatment plant were completed and the capacity of the treatment plant was doubled to 7200 m3/day. Chlorination of the tanker trucks ended in July 2009. Currently, a relatively small percentage of Juba’s population is serviced by a UWC connection. As such, tanker trucks remain an essential component of water delivery services to many areas of Juba. USAID/Sudan has been leading efforts to establish new tanker truck filling stations that take water from the UWC system. When completed, these filling stations would enable the tanker trucks to fill with water that has been treated at the UWC plant. 4.1.3 Sanitation The current policy framework for sanitation is less well defined than for water. Responsibility for sanitation is shared between the Ministry of Housing, Physical Planning and Environment, the Ministry of Health, and the Ministry of Education. The GOSS has assigned responsibility for urban sewerage systems to the MHPPE, although no piped sewerage systems currently exist in Southern Sudan. Currently the MWRI does not have responsibility for urban sanitation, but there is discussion of bringing urban sanitation into the responsibility of the MWRI. There reportedly are policies at the payam level requiring construction of latrines at households. However, these policies are not being enforced. In addition to the lack of a well defined policy environment, there is also a lack of consensus on a strategy and approach for household sanitation. Thus far, the focus of donors in the urban WASH sector has been on water, with relatively little effort on sanitation. Many people in Juba lack access to improved sanitation facilities. Of a small number of households surveyed, less than one-third had any access to improved sanitation. Many of the households that report having access to a latrine share that latrine with multiple households. Many households in Juba are renters. Renters have little incentive to construct household latrines. Currently, the cost of constructing a household latrine is higher than most families are willing or able to pay. Households will not construct latrines until the perceived value of a latrine exceeds the cost. The main factors that are preventing families from constructing household latrines are the following: 1. Lack of awareness of the importance of household sanitation (lack of demand); 2. Perceived high cost of constructing a latrine; and 3. Threats of demolition. Evaluation Report Urban WASH, Juba, Southern Sudan 21 Compass, an LBG subcontractor, implemented a household sanitation project in areas of Kator payam that were more heavily impacted during the 2006 cholera outbreak. As this was the first time implementing a household sanitation project, it was decided to start with 150 slabs to gauge the response of community members. This number of households represents a small portion of the Kator population. Compass staff went house-to-house in Kator raising awareness of the slab program, encouraging people to dig pits, and providing technical advice on the location of a pit latrine. After the pits were dug and prior to distribution of the slabs, the locations were inspected and approved by an officer from the Kator Payam Public Health Department. Local masons were trained in the construction of low cost slabs. Materials for the 150 slabs were provided. The cost of materials for one of the slabs was approximately 230 SDG. The labor cost of the local masons was approximately 120 SDG per slab. Slabs were distributed to the first 150 pits that were dug by families. Some families received more than one slab, due to the large size of the pit that was dug for the large number of users of the pit latrine. Household latrine slab construction quality appears to be of acceptable technical quality. However, no latrine hole covers were observed in use. Flies will not be blocked if covers are not in place and used properly, reducing public health benefits. Also, soap or ash was not observed present near most of the latrines observed. Results from the Toilet Observations Sheets are contained in Appendix G. At the time of the evaluation, all 150 slabs had been distributed to 73 households, but only 87 of the slabs were completed with latrine superstructures. In recent weeks, rumors of demolition have been circulating in the communities where slabs have been distributed. These are areas that have not yet been formally demarcated by the Ministry of Housing, Physical Planning and Environment. Community members are reluctant to construct latrines, as demolition may occur in their areas. Most other areas of Juba have now been formally demarcated, but at the time of inception of the sanitation project, the areas of Kator that were targeted had not been demarcated. Approximately 11% of the LBG subcontractor budget has been spent on sanitation projects. While this represents a relatively small portion of the overall budget, the current policy framework for urban sanitation does not create an enabling environment for large investments in household sanitation. Given the competing funding needs of urban water and sanitation infrastructure, the current policy context, and the lack of consensus on a strategy for urban sanitation, it is deemed that the proportion of funds that have been spent on sanitation reasonably corresponds with current enabling environment. 4.1.4 Hygiene PSI has implemented a multi-channel behavior change methodology for hygiene promotion and social marketing of point of use water treatment products. PSI has targeted locations throughout Juba, using various means of outreach. The following table lists the different aspects of PSI’s multi channel behavior change approach. TABLE 8 – PSI MULTI CHANNEL BEHAVIOR CHANGE HYGIENE APPROACH Outreach Target Audience Location Duration Radio Juba Juba minutes Drama Groups <100 markets, gathering points 1 hr. Small Outreaches 50-60 communities 1 hr. Health Education 70-80 health centers, public places 1 hr. Modular Training 20-40 schools, health workers 10 hrs. Evaluation Report Urban WASH, Juba, Southern Sudan 22 It appears that PSI is effective in social marketing of water point of use treatment products. Nearly 70% of people surveyed treat their water with WaterGuard or PUR. The demand for the products has been such that they are readily available for purchase in stores in many areas of Juba. PSI’s hygiene behavior change messaging is also judged by the evaluator to have been effective. The materials used during the outreaches are judged by the evaluator to be both simple to understand and also effective at conveying a proper message. Over 50% of people surveyed had attended a hygiene promotion session, which indicates that PSI has been able to reach people in the communities with hygiene behavior messages. PSI did not conduct a baseline survey of hygiene behavior practices in Juba at the beginning of the project, and an exhaustive survey of hygiene practices was not conducted as part of this evaluation. Therefore, it is not possible to quantify the change in hygiene practices as a result of PSI’s work. However, many people interviewed indicated that their hygiene practices have improved. Observations made during household surveys corroborate this finding. Some 90% of households surveyed have soap, which is widely used for washing clothes. Soap is not widely used for hand washing at crucial times (before eating and after toilet use). According to survey data and direct observations, many households have water containers used for hand washing near their latrines. People commonly wash their hands with water after using toilets. However, use of soap for hand washing is less common. Washing hands with soap before eating and after toilet use is a very effective way of breaking the fecal-oral transmission route, and therefore preventing water borne diseases. It is an encouraging finding that many families are already in the habit of washing their hands at one of the critical times. The finding that soap is already available for other uses in many households suggests that with some additional messaging on the importance of hand washing with soap at critical times, additional improvements in hygiene behaviors could occur. Hygiene promotion activities have occurred at locations throughout Juba and have targeted a broad segment of Juba’s population. However, the hygiene promotion activities have not been specifically aligned with the areas where LBG’s water and sanitation infrastructure work has occurred. It is possible that community members using the newly constructed water and sanitation infrastructure will not necessarily have exposure to improved hygiene behavior change trainings. Approximately 35% of LBG’s subcontractor budget has been spent on hygiene projects. The proportion of funds that have been spent on hygiene represents a significant investment in promoting hygiene behavior changes. Given the importance of improved hygiene practices as a crucial component of preventing water borne illnesses, it is deemed that a reasonable proportion of project funds have been spent on hygiene projects. 4.1.5 UWC Capacity Building and Institutional Development LBG assisted capacity building of UWC in technical areas by providing on-the-job training of UWC technical staff during construction activities. This increased the skills of the UWC staff responsible for performing maintenance work on the existing UWC system. LBG also sought involvement from the UWC in each stage of planning and construction. LBG has not, however, strengthened the institutional operations of UWC. LBG has not systematically sought to strengthen the UWC’s ability to function in areas of administration, financial management, human resources, or communications. It is noted that institutional capacity building activities were not specifically indicated in LBG’s work plans approved by USAID/Sudan. Currently, less than 20% of Juba’s population is served by a UWC connection. Much further work on the urban water system is needed to service the water needs of the population. There are two major immediate impediments preventing UWC from operating as a sustainable water utility: 1. UWC lacks necessary autonomy (regulatory, financial management, institutional capacity) making it vulnerable to outside political forces; and Evaluation Report Urban WASH, Juba, Southern Sudan 23 2. Current user revenues are insufficient to cover operating costs. There are several critical constraints on UWC and MWRI that impact the ability of these government institutions to effectively and sustainably manage investments in Juba’s urban water infrastructure. In 2008 a Provisional Order was drafted by the Ministry of Legal Affairs and Constitutional Development outlining the framework for UWC to operate as a somewhat independent entity. Passage of the Provisional Order would be a first step in granting UWC the autonomy necessary for a water utility to operate effectively. The Provisional Order has not yet been signed by the President. It is understood through discussions with MWRI staff that the Provisional Order is currently stalled. MWRI is continuing to press the Provisional Order with the President’s office. Current provisions enable UWC to retain 20% of tariff revenues. However, 80% of revenues are directed to the GOSS Ministry of Finance block account. There is currently no mechanism for UWC to directly access these funds. UWC does not know the gap between its operating expenses and its tariff revenues, but it is widely thought that revenues from users are insufficient to cover operations and maintenance costs. Therefore, even if UWC were able to directly manage its tariff revenues, current total tariff revenues collected by UWC may not be sufficient to cover expenses of the utility. This would make UWC dependent upon GOSS subsidies. But, since funding for UWC is vulnerable to GOSS political forces, it is difficult for UWC to rely on receiving the funds necessary for ongoing operations and maintenance. Reform of the tariff system and establishment of a working business model is needed to reduce this vulnerability Currently, UWC Administration does not know with certainty its percent collection of tariffs. UWC has approximately 2000 registered customers and it is estimated by UWC that approximately of 90% households are paying fees to UWC. Fees collection data is not fully collected or monitored. Currently, UWC does not collect any revenue for water it delivers to institutional users, such as government ministries, schools, and health centers which have not been paying UWC tariffs. UWC also has gaps in its institutional capacity. Formal UWC administrative procedures and personnel systems either do not exist or are dated to colonial times. Informal systems exist and are being used. Dissemination of information to the public does not systematically occur. The general public is ill informed of activities and actions of the UWC. According to the GOSS decree, The Functions and Duties of the Government of Southern Sudan, MWRI is responsible to “Oversee the operation of the Water Corporation of Southern Sudan to ensure it performs its functions…” In conjunction with UWC, there is need for institutional strengthening and capacity building of the MWRI. MWRI has specifically requested support in policy setup, regulatory framework, and institutional development, as capacity is currently lacking in these key areas of its responsibility. Maintaining records is currently a problem for MWRI, as it is unable to locate some key regulatory framework documents. MWRI staff is often not aware of annual investment and planned actions in the WASH sector from donors and implementers. MWRI needs to improve internal information sharing from the Budget Sector Working Group processes. Furthermore, MWRI also lacks capacity to manage large numbers of contractors and could benefit from strengthening in its ability to coordinate and direct implementers in identified areas of need within the sector. Lack of government supervision provides incentive for contractors to cut corners with contract work and/or not perform the contracts properly. GtZ plans to bring in a technical advisor in the upcoming year to work closely with the UWC on institutional strengthening and capacity building. It is not known at this time whether the GtZ Technical Advisor will also be able to assist similar efforts within MWRI, but it is judged that additional assistance for MWRI will be needed. 4.2 LBG’S MONITORING AND EVALUATION SYSTEMS In accordance with the requirements of USAID/Sudan’s ADS Chapter 203, LBG established a monitoring and evaluation system. The LBG Performance Monitoring Plan contains five indicators: two standard Evaluation Report Urban WASH, Juba, Southern Sudan 24 indicators and three custom indicators. Of these indicators, the following three are reported on a monthly and quarterly basis: • Number of people with access to improved drinking water supply; • Number of people with improved public and household sanitation facilities; and • Number of individuals trained in good health and hygiene practices. LBG’s monitoring and evaluation system is adequate for reporting required project outputs and outcomes. However, the following three findings indicate changes that could improve the system: 1. Currently, beneficiaries of point of use water treatment products are listed in the same table as the number of people with improved access to safe water supply. This is incorrect reporting, as increased point of use water treatment does not translate into increased water supply. 2. Currently, beneficiaries of community toilets are listed in the table for number of people with improved household access to sanitation. This is incorrect reporting, as the standard indicator for sanitation does not include users of public toilets. 3. Currently, target beneficiaries for water infrastructure are calculated using SPHERE guidelines for maximum number of users. SPHERE guidelines are not applicable for estimating number of beneficiaries of water infrastructure. In addition to the recommended changes listed above, some relatively minor modifications are recommended to improve accuracy and clarity of reporting. These items have been discussed with the LBG Task Order Manager. Reportedly, several of the changes that were discussed have already been made, which should mean that future reporting of project achievements will have more clarity. LBG established a data quality management system that appears to be working well. Upon request from the evaluation leader, LBG project staff was able to quickly produce documentation of project records that had been randomly chosen. LBG also built the capacity of its subcontractors in data quality assurance and management. LBG performs annual surveys to collect data on 21 additional indicators that are not included in the monthly and quarterly reporting. In the annual surveys, LBG collects data on the percentage of households washing hands after toilet and before meals. Data on use of soap at these critical times is collected but not reported. It is unclear what percentage of households use soap for hand washing at critical times. A baseline survey was conducted in April 2008. Due to the rapidly changing context in Juba, numerous changes in target locations have occurred during project implementation. As a result, the locations assessed in baseline survey do not necessarily correspond with locations where the majority of project work has occurred. Tracking changes over time to data points contained in the baseline survey may not be indicative of changes that have occurred in target communities as a result of the project. Some 150 households were surveyed for the baseline survey. This represents approximately 1% of the estimated population at the time, although there was large uncertainty of the original population size, which made it difficult to perform proper surveying. Due to the small sampling size, data in the baseline survey may not necessarily be representative of the population of Juba. Caution and judgment should be exercised when using baseline data in any future comparisons for measuring project impact. Evaluation Report Urban WASH, Juba, Southern Sudan 25 5. STRATEGIC RECOMMENDATIONS The following recommendations are made to increase the effectiveness and sustainability of USAID/Sudan’s urban WASH programming. The three components of USAID’s Hygiene Improvement Framework were chosen as the lens for strategic recommendations. • Improving Access to Water and Sanitation Infrastructure; • Promoting Hygiene Behavior Changes; and • Strengthening the Enabling Environment (policy framework, institutional capacity). 5.1 URBAN WATER AND SANITATION INFRASTRUCTURE Significant long-term investment in urban water infrastructure is required to improve and expand service delivery to UWC customers. UWC currently has reticulated water systems in four urban centers: Juba, Wau, Malakal, and Renk. It is not known what percentage of the populations of Wau, Malakal, and Renk are served by UWC connections, as such investigation was outside the scope of this evaluation. Currently less than 20% of the Juba population is serviced by a UWC connection, and it is assumed that a similar scenario exists in the other urban centers. Much further work on the four existing UWC systems is needed to service the water needs of urban populations. Additionally, with the growth of other urban areas in Southern Sudan, there is need for urban water services in other urban centers. It is recommended that USAID/Sudan continue funding water infrastructure activities that promote long￾term investment. The following specific items should be prioritized: • Fixing leaking water mains in each of the UWC systems. − For Juba the priority repairs should be on the water main line from the Juba water treatment plant to the newly rehabilitated Konyo Konyo tank. UWC suspects that this section of pipeline contains significant leakages, which could affect water delivery to Kator. LBG has tested water pressure at its new additions in Kator to ensure that adequate pressure exists in the system. However, future UWC network expansion in Kator may be limited by the leaking main line from the water treatment plant. • Rehabilitating the existing water treatment plants, as has been proposed in Wau during the next phase of this project. − The Juba water treatment plant has recently been rehabilitated under MDTF funding and does not require additional work in the short- to medium-term. There is longer-term need to add additional treatment capacity in Juba, as the current treatment plant capacity is insufficient to meet the water needs of all of Juba’s population. • Expanding distribution networks into new urban areas with established tariff cost recovery systems for operations and maintenance. − Following reform of the tariff system and the establishment of a working business model for UWC, there will be need for significant investment in expanding distribution networks, as only a small percentage of the population is currently serviced by a UWC connection. Urban sanitation coverage remains quite low. Significant investment in urban sanitation infrastructure is also warranted. Prior to increasing investments in urban sanitation, however, progress is needed in the following areas: Evaluation Report Urban WASH, Juba, Southern Sudan 26 • An improved policy framework with sanitation responsibilities clearly defined; • Leadership from the appropriate government ministry; and • Consensus on an approach for urban sanitation. It is recommended that USAID/Sudan focus its sanitation programming on improving the policy environment for the expansion of household sanitation coverage. It is recommended that USAID/Sudan advocate for all sanitation responsibilities to be granted to MWRI, which would enable integrated water and sanitation programming to be overseen by the same ministry. Once an improved policy framework with sanitation responsibilities is clearly defined, USAID/Sudan can build the capacity of MWRI to enact sanitation policies, strategies, and enforcement mechanisms. Improving the sanitation policy framework is a process that is likely to require a minimum of one year, and possibly longer. In the absence of significant improvements in the urban sanitation policy framework, USAID/Sudan can focus on inducing demand for household sanitation through increased awareness raising and hygiene promotion activities. Such activities can be incorporated into the PSI multi-channel behavior change hygiene promotion approach. Another way of inducing effective demand for household sanitation is by lowering the cost of latrines to the point where families are willing and/or able to pay for the latrines. It may be possible to further lower costs through bulk purchasing and bulk labor agreements with local masons. It is recommended that USAID/Sudan investigate ways of lowering the cost of household sanitation. Lowering the cost of household sanitation will be helpful not only for inducing demand, but also for formulation of the urban sanitation policy framework. 5.1.1 Hygiene Behavior Changes Hygiene behavior improved somewhat in targeted areas. Further programming is strongly recommended to reinforce hygiene behavior change messaging in areas already targeted and to expand into new target urban areas. It is recommended that social marketing of point of use water treatment products be continued and expanded to new geographic areas. The hygiene behavior change methodology established by PSI should be continued. The following adjustments should be made to improve effectiveness of the existing multi-channel behavior change hygiene promotion approach. • Additional emphasis should be given to the importance of hand washing with soap at critical times. • Additional emphasis should be given to the importance of household sanitation. • Areas where water and sanitation infrastructure are being installed should have specific targeting and reinforcement of hygiene behavior change messaging. 5.2 ENABLING ENVIRONMENT Further institutional strengthening and capacity building of UWC and MWRI are critical for ensuring sustainable investments and service delivery in the WASH sector. In the absence of properly functioning institutions, investments made in the WASH sector are less likely to be sustainable. USAID/Sudan should strongly consider supporting institutional strengthening and capacity building of UWC and MWRI by providing technical assistance in the following areas. Evaluation Report Urban WASH, Juba, Southern Sudan 27 • WASH sector policy dissemination, strategy development and investment planning; • Establishment of a working business model for UWC based on cost recovery for operations and maintenance costs; • Reform of the UWC tariff policy to cover operating costs and to include institutional users; • Strengthening of UWC Finance and Administrative systems: − Establishment of a UWC finance account separate from the GOSS block account; and − Enabling revenue collection from GOSS ministries, schools, and health centers. USAID/Sudan should also strongly consider supporting institutional strengthening and capacity building of MWRI by supporting a Technical Advisor to be placed in MWRI office not only to assist with institutional capacity development, but also to assist with advancing the policy process. MWRI has requested support in these key areas of its responsibility where it currently lacks capacity. In future urban WASH program work, LBG should be specifically directed to support UWC institutional strengthening and capacity building in some or all of the following key areas. • Contracts Management – technical oversight and contracts enforcement mechanisms • Administration and Finance – establishment of formal policies and procedures • Personnel/Human Resources – establishment of formal policies and procedures • Communication – establishment of formal policies and procedures (both external and internal) JICA performed an assessment of UWC capacity in eight key organizational categories. The summary of results from the JICA capacity assessment is contained in Appendix H. The JICA assessment findings could be used as a starting point for any future institutional strengthening and capacity building activities with UWC. Evaluation Report Urban WASH, Juba, Southern Sudan 28 Evaluation Report Urban WASH, Juba, Southern Sudan 29 APPENDIX A. SCOPE OF WORK Management Systems International (MSI) SUPPORT Project with USAID/Sudan1 Evaluation of USAID Support to the Juba Urban Water Corporation (UWC) Implemented by The Louis Berger Group (LBG) (Estimated start date: August 2009) Background Sudan is the largest country in Africa, borders nine countries, and has a population estimated at 40 million on the eve of the first census to be conducted in decades, now scheduled for April 2008. The current population of southern Sudan is estimated at 10-12 million people, with an estimated four million others displaced to northern Sudan or living as refugees outside the country. Southern Sudan has suffered from decades of underdevelopment, war, famine, drought and floods, resulting in the devastation of the South’s economic, political, and social structures. The southern Sudanese people lack access to basic health and education services, as well as the infrastructure needed to build a thriving economy and functioning state. Since the mid-1990s, non-governmental organizations (NGOs), faith-based organizations (FBOs), and international humanitarian relief agencies have become the prime providers of an array of much-needed services. After decades of civil war, parties to Sudan’s north-south civil war signed a Comprehensive Peace Agreement (CPA) in January of 2005. The CPA is comprised of six interlocking agreements regarding: (a) wealth and power sharing, (b) the establishment of the Government of National Unity (GNU) that provides southern Sudanese with representation in the national government, (c) a resolution of conflicts in the oil-rich border regions between northern and southern Sudan (known collectively as the “Three Areas”), (d) the creation of a southern Sudanese government and state governments with appointed leadership until elections in 2009, and (e) the establishment of a six-year interim period after which the citizens of southern Sudan will vote on whether to remain a part of unified Sudan or secede to create an independent state. USAID is working with the GOSS to assist the formation of core government institutions and systems. Since the signing of the CPA, USAID has funded significant amounts of programming in Water, Sanitation and Hygiene (WASH) sector. Typically, WASH funding has been divided between Urban and Rural, with the three major cities of Juba, Wau and Malakal being classified as urban centers. To date, USAID’s Urban WASH program has largely focused on Juba with water distribution system infrastructure expansions, ablution blocks/public latrines, and point-of-use water treatment. Program Description for LBG under Task Order #4 The United States Agency for International Development (USAID)/Sudan created the Sudan Infrastructure Services Project (SISP) funding mechanism to assist the people of Southern Sudan with increased access to infrastructure. Under SISP, the LBG was issued Task Order #4 to develop improved urban access to clean water and improved access to sanitation. The program description of Task Order #4 can be found below in Annex 1. 1 MSI has a 3-year contract to provide Mission-wide support to USAID/Sudan in program and project evaluation and design, MIS management, translation services, facilities management, VIP hosting, and research. An in-country team based in Khartoum and Juba will provide these services, supplemented by short-term technical assistance. USAID/Sudan’s Evaluation Agenda of Juba’s Urban WASH programming USAID/Sudan has requested this evaluation as one of a series of program reviews and assessments to be conducted during FY 2009 of key USAID-funded programs under the Sudan Infrastructure Services Program (ISP) – IQC No. 650-I-00-06-00010-00. The overall purpose of this evaluation is: • To identify successes, lessons learned, and obstacles to progress of this USAID-funded program; • To provide recommendations for any program adjustments that may be necessary to increase the effectiveness and improve the implementation of USAID activities in Juba, Sudan; and • To advise on the adequacy of program input, context, and performance. Purpose and Objectives of this Evaluation LBG’s Juba WASH project activities are taking place within the context of Juba’s ongoing rapid expansion, which is fraught with complexities and challenges. Since the signing of the CPA, Juba has rapidly outgrown its crumbling, war-damaged urban water treatment and distribution systems. Furthermore, Juba’s water systems have suffered due to the lack of a master plan and the poor coordination among the various donor and implementers that are working to improve the water supply in Juba. The timing of this evaluation is opportune for several reasons. First, LBG is in the middle of a 3-year award under Task Order #4, and USAID is seeking to capitalize on lessons learned and successes for its future investments into urban water systems in Sudan. The budget ceiling is $14 million, of which $6.2 million has been obligated. Second, USAID wants to ensure that the program is clearly achieving results within the management capabilities of the Juba UWC and that the necessary management tools are in place to accommodate further improvement in the Juba water systems. Finally, given the endless series of changes and unique challenges to working in Sudan, USAID would like to ensure that future urban WASH programming in Southern Sudan will have the maximum positive impact on the urban health environment. Thus, the objectives of this evaluation are: 2. Identify and, if possible, measure the impact and successes of LBG’s program activities to date vis-à￾vis the program objectives, namely: a. Building the organizational capacity and development of UWC; b. Improving the distribution of water via the new connections made in Kator and Munuki; and c. Increasing the effectiveness of the UWC in providing improved water quality and access to water to customers in Juba town; 3. Assess the sufficiency of data collection with respect to monitoring, program design, program input, and program context; and, 4. Make recommendations, as needed and within the context of current program funding, to: a. Improve the management and organizational capacity of the UWC so that it will be able to improve and sustain its service delivery; and b. Strengthen the collection of program input, context, and performance data to enable robust life-of-program performance monitoring and end-of-program evaluations. The evaluation team will ensure that the evaluation is fully compliant with the terms for Project Evaluations contained in the USAID Automated Directives System (ADS) Series 203 and other relevant regulatory requirements, as may be determined by USAID. Additionally, the team will utilize MSI’s “SUPPORT Evaluation/Special Study Quality Management Guide.” The Guide will be presented to the team members prior to their initial MSI briefing in Washington. Evaluation Report Urban WASH, Juba, Southern Sudan 30 Methodological Approach USAID/Sudan is seeking to increase the rigor of its monitoring and evaluation practices to improve future programming design. This desired approach requires management and impact analysis in order to help better understand key impact constraints and to help facilitate future program design. USAID/Sudan seeks a mixed methodological approach for this evaluation. In the first instance, evaluators should compare program baseline and end state information as collected by LBG in the program performance management plan. This information should be analyzed in conjunction with available program input and context data to determine the impact of activities vis-à-vis stated program objectives. Issues and Questions to Be Addressed The Purposes and Objectives section above describes the key issues that the evaluation team should explore in its field work and address in its analysis and recommendations. The questions below correspond to these issues. The organization of the questions is provided to suggest an outline for structuring the evaluation report. The questions should guide the team’s analysis, but do not need to be addressed sequentially in the report. With the concurrence of USAID/Sudan, the team may decide to add, refine, or delete questions, or modify the proposed structure, if they believe it would highlight key findings or otherwise add value to the analysis. 1. LBG Performance to Date 1.1. Heavy infrastructure (pipelines and urban water system design) • Evaluate LBG’s performance to date in the design and implementation of pipeline extension for the Juba water distribution system. • Evaluate the cost-benefit of LBG heavy infrastructure programming on the overall water and sanitation situation in Juba. 1.2. Light Infrastructure (ablution blocks and emergency river treatment facilities) • Evaluate LBG’s performance to date in the design and implementation of ablution blocks and emergency river treatment facilities for Juba. • Evaluate the cost-benefit of LBG light infrastructure programming on the overall water and sanitation situation in Juba. 1.3. Sanitation & Hygiene (community-focused soft programming) • Evaluate LBG and sub-grantees performance to date in the design and implementation of community-focused soft programming to improve the knowledge/awareness on issues of basic sanitation and hygiene. • Evaluate the cost-benefit of LBG community-focused soft programming on the overall water and sanitation situation in Juba. 1.4. UWC Capacity Building (institution-focused soft programming) • What are the critical institutional constraints on the UWC and government authorities to sustainably and effectively managing the Juba water facility? • Has the LBG program design focused on these institutional constraints and, if so, how successfully have they been addressed? 2. Monitoring and Evaluation • Are there any gaps in data currently being collected that would need be filled in order to successfully fulfill LBG’s scope of work? • Is the current M&E system sufficient for monitoring implementation and incremental impact, USAID/GOSS reporting, and managing program performance? Evaluation Report Urban WASH, Juba, Southern Sudan 31 3. Strategic Recommendations Given the above analysis, what programmatic adjustments should be made to maximize the effectiveness of LGB's work with the UWC in Juba, Sudan? What changes, if any, to the relative level and orientation of LBG’s human and technical resources will be required to implement the program as recommended? 3.1. What programmatic adjustments should be made to maximize the effectiveness of LBG’s work with the UWC? • In what areas has LBG been particularly effective at contributing to improvements in the Juba water and sanitation? • On which areas does more focus need to be placed on to improve the effectiveness of water and sanitation programming in Juba? 3.2. What programmatic adjustments should be made to maximize the sustainability of LBG’s work with the UWC? • What adjustments should be made to address the long-term financial sustainability of the UWC? • What adjustments should be made to address how LBG supports the administrative needs of the UWC? • What adjustments should be considered to improve LBG’s support to the management capacity of the UWC? Required Tasks and Work Plan Tasks (All team members unless otherwise noted) Work Days (6-day weeks) Timeline for Completion Initial Preparation Review advance background documents and SUPPORT Project’s Evaluation and Special Study Guide, travel preparations, and travel days. 4 August 2009 In-Country Evaluation Initial briefings, meetings, field visits, and briefings. 15 August Draft Report Preparation in Sudan. Debriefing, draft report preparation, submit for review 4 Draft due 2 weeks after completion of field work Final due NLT 10 work days after receipt of USAID comments on draft report Travel home 2 Finalize report 3 Total for Evaluation Team Leader 28 Team Composition, Activities and Timing Team Composition The team requirements will be: one expert (team leader) who will work closely with USAID/MSI WASH Advisor. The individual must have extensive experience conducting evaluations. Evaluation Report Urban WASH, Juba, Southern Sudan 32 Note: The CTO and other USAID staff will be encouraged, when practical, to join the evaluation team. In so far as the CTO or other USAID staff join the team, MSI will be responsible for their arrangements (travel, housing in the field, etc.). Lastly, MSI will be responsible for arranging all meetings for the team, in conjunction with the USAID/Sudan Juba office. Activities and Timing The USAID/Sudan Mission requests that the evaluator arrive in Juba for the initial briefings and discussions with USAID’s Health Team Leader and other Mission officers, as well as LBG, GOSS, and WASH sector representatives. Subsequently, the teams will commence field trips and meetings. During the initial meetings in Juba, the evaluation team leader will present in writing and orally the team’s proposed work plan for the entire period of the team’s presence in Sudan as well as thereafter with respect to the submission of the draft and final reports. The work plan will also include a schedule for periodic USAID meetings/progress reports and possible submissions of specific work products, as determined by the two parties. It is envisioned that all team members will be in Sudan the entire duration of the evaluation’s in-country component, i.e., three and a half weeks (six-day work weeks are authorized). Besides travel days, an additional three days are provided for purposes of the team’s report finalization. The team leader will be totally responsible for managing the team members, organizing its work, and ensuring quality control and delivery of the required report as agreed by both parties. Deliverables Deliverables will include the out-briefing and supporting documents and the final report. A detailed written outline and oral debriefing of the conclusions, supporting findings, and recommendations of the evaluation team will be presented to the USAID/Sudan Health Team Leader prior to the team’s departure. Briefings for other USG and/or Sudanese officials will be provided as determined by USAID. The final report, prepared in MS Word, will not exceed 30 pages (excluding executive summary and annexes). Single line spacing and 11 point font are acceptable. The report should include: I. Executive Summary (not to exceed three pages, which can be used as an independent briefing paper) II. Introduction III. Methodology IV. Conclusions and Supporting Findings A. LBG’s Performance to Date i Heavy infrastructure (pipelines and urban water system design) ii Light Infrastructure (ablution blocks and emergency river treatment facilities) iii Sanitation & Hygiene (community-focused soft programming) iv UWC Capacity Building (institution-focused soft programming) B. Monitoring and Evaluation System V. Strategic Recommendations A. Programming Approach Adjustments for Effectiveness B. Programming Approach Adjustments for Sustainability VI. Annexes: This section may include, inter alia, an annex of data sources utilized, key informants interviewed, and a listing of useful tools and products that could be repackaged and produced for wider distribution to other USAID-funded implementing partners in Sudan or USAID-funded political party or legislative programs in other countries. Evaluation Report Urban WASH, Juba, Southern Sudan 33 ANNEX 1: PROGRAM DESCRIPTION FOR LBG UNDER TASK ORDER 4 The United States Agency for International Development USAID/Sudan wishes to assist the people of Southern Sudan to develop improved access to clean water and improved access to sanitation, while empowering local communities to assume responsibility for improved public health management through implementation of appropriate community-based water and sanitation facilities in selected at-risk urban communities. Activities and responsibilities of the Contractor under this Task Order are expected to include the following component activities in Juba. The Contractor will be responsible for: • In close collaboration with non-governmental organizations, JSI, and local authorities, identify specific neighborhoods for program implementation; • Conduct sufficient baseline analysis to allow adequate program monitoring and evaluation activities during and after program implementation, including an evaluation of existing attitudes and practices related to water, sanitation, hygiene and associated health risks and effectiveness of past public and household latrine campaigns, where relevant; • Implement community mobilization activities to establish an appropriate community-based mechanism to implement the program using community driven decision making processes. Effort shall be made to specifically include women and vulnerable groups in this process to assure a balance is struck in stakeholder needs. This activity shall be coordinated with other USAID-funded health related activities as appropriate, such as the work being conducted under the JSI/SHTP; • Setting, design, and construction of effective and affordable household sanitation and hand washing facilities and/or community latrines and ablutions facilities, in close collaboration with the community; • Provision of sub-grants or sub-contracts to NGO’s/CBO’s for community-based promotion of household-level private or shared latrine construction and hygiene improvement in priority cholera￾affected communities, including incentives to stimulate household investment in latrine construction; • Develop a comprehensive hygiene and sanitation promotion strategy including behavior change communications at levels ranging from household to community-wide for the three key behaviors shown to reduce diarrhea and trachoma: hand washing, safe disposal of feces, and treatment/protection of the community’s water supply; • Produce and disseminate promotional materials and messages and spread hygiene behavior change messages; Research and recommend appropriate equipment and treatment processes for installation in selected communities, based upon the following sustainability criteria: − The system shall utilize technology that lends itself to maintenance and operation activities that can be carried out by community members with limited training or education; − The system operation shall not require chemicals, lubricants, or parts that are overly expensive or difficult to obtain in Southern Sudan; − The system design shall optimize operation and maintenance costs against appropriate production rate for the community; − The treatment unit system design life shall be at least five (5) years, and preferably ten (10) years if achievable under the constraints identified above. The treatment equipment shall be housed in a securable building, adequately ventilated and including noise attenuation if operating equipment Evaluation Report Urban WASH, Juba, Southern Sudan 34 is excessively loud at the project perimeter. USAID and LBG will jointly determine appropriate noise limits if this appears to be a problem. • Procurement of water treatment equipment; • Installation of complete community water treatment and delivery systems, including community standpipes and ablutions facilities; • Implement community mobilization activities to form a community-based water system management and sustainability system appropriate to the community in each location where a system is installed. This may be in the form of a neighborhood water committee that assists in the design of the fee collection and funds management system, as well as the location of the project and design of facilities. The objective of this activity is to result in a fully functioning and self-reliant funding, operations, maintenance and management unit in the community, and may include the management and maintenance of sanitation and ablutions facilities as well. • Develop a program to monitor water quality parameters of concern for all water treatment systems provided under this Task Order, including development of necessary procedures and controls to provide data collected to relevant government authorities, as well as to assist the community in maintaining a safe water delivery system to their own community. It is envisioned that this activity will include provision of basic testing equipment and training to the community system operators; and, • Initial operation of the system and training of identified community members to operate and maintain the system. Evaluation Report Urban WASH, Juba, Southern Sudan 35 APPENDIX B. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS No TOR Finding Source Conclusions Recommendations LBG Performance - Heavy Infrastructure 1 1.1 LBG has performed more pipeline installation work than what was initially proposed. Records The work actually performed by LBG has increased to better address the urban water system needs, based upon the conditions encountered during various stages of project implementation. None: contract was established as a ceiling contract. 2 1.1 Several items in original work plan approved Feb 2008 were changed or expanded. Records LBG demonstrated flexibility and adapted project to address changing needs in Juba, in consultation with government officials. None 3 1.1 LBG has given thoughtful consideration to future expansion and development of the UWC system in Kator. Observations, Records Investments made will enable future expansion and development of the UWC system in Kator. Increased likelihood of improved sustainability of system. None 4 1.1 Quality of LBG pipeline construction work appears to be acceptable. Interviews￾UWC, Observations LBG has effectively managed its subcontractors to ensure good quality work. Construction quality appears to have complied with UWC expectations. Continued use of local contractors for cost-effective network expansions and the creation of a competitive construction environment, which will assist the long-term expansion of urban water utilities in South Sudan. 5 1.1 Approximately 41% of LBG subcontractor budget was spent on heavy infrastructure for urban water. Records Proportion of funds that have been spent on heavy infrastructure water projects is reasonable. These projects are long-term investments in urban WASH infrastructure. Continue funding water infrastructure activities that promote long-term investment. LBG Performance - Light Infrastructure 6 1.2 Construction quality of the three community river tanks appears to be acceptable. Some minor improvements can be made in drainage. Observations, Water Systems Sheets LBG has managed its subcontractors to ensure acceptable quality work. None 7 1.2 Construction quality of the four community toilets appears to be acceptable. Some minor improvements can be made in drainage. Observations, Toilets Observation Sheets LBG has managed its subcontractors to ensure acceptable quality work. None Evaluation Report Urban WASH, Juba, Southern Sudan 36 Evaluation Report Urban WASH, Juba, Southern Sudan 37 No TOR Finding Source Conclusions Recommendations 8 1.2 Kator Payam Public Health Officer has realistic business plan for two community tanks and two community toilets. Payam has experience managing community toilets in the past. Interviews￾Kator Payam, LBG Kator Payam appears able to manage the community tanks and toilets. LBG perform periodic monitoring, especially in the initial months following handover, to ensure proper management by payam. 9 1.2 Kator Payam plans to begin managing two community tanks only after formal LBG handover. Interviews￾Kator Payam, LBG Transition may be abrupt and difficult. Payam begin managing community tanks two weeks before handover so that payam and LBG have an overlap period. 10 1.2 Fees collected from users of sanitation or water infrastructure by Kator Payam Public Health Department go into a general payam block account. Interviews￾Kator Payam There is less incentive for the public health department to properly collect fees if the public health department is unable to manage the fees. Investigate working to change the policy to enable public health department to manage fees it collects from community water tanks and community toilets. Capacity building of public health department. 11 1.2 CHF tanker truck chlorinators are also chlorinators at the three community river tank sites. The CHF contract has finished, so chlorinators are no longer receiving payments. Interviews￾CHF Chlorination may not regularly occur at the three community tanks. It is critical that LBG successfully transitions these systems over to payam management, ensuring that user fees cover the costs of continued operations, including regular and consistent chlorination. 12 1.2 Some water vendors were observed collecting overflow water from untreated tank at the Juba Bridge community river tank. Observations No enforcement against collecting untreated water. Some water being sold by jerry can vendors is unsafe. Instruct tank managers to prevent people from obtaining unsafe water at community tank sites. Further awareness-raising on importance of safe water. 13 1.2 Low usage at Atlabara and Gabat community toilets. Observations At current usage levels, it will be difficult to meet operating expenses with revenue collected from users. LBG should work closely with payam officials during initial months of operations to ensure proper accounting of costs. 14 1.2 The Central Equatoria State Rural Water Supply Department has a policy against fees being collected from users. Interviews￾Compass, LBG When boreholes break down, the funds for repair may not be available. USAID/Sudan should not fund any more borehole rehabilitations in CES until the Rural Water Directorate changes its position to be in line with the Southern Sudan Water Policy regarding the collection of user fees for sustainable operation and maintenance. 15 1.2 The original three locations for tanker truck chlorination were expanded to seven locations. Records, Interviews￾LBG LBG demonstrated flexibility and adapted project to address changing needs in Juba, in consultation with None Evaluation Report Urban WASH, Juba, Southern Sudan 38 No TOR Finding Source Conclusions Recommendations government officials. 16 1.2 Approximately 13% of LBG subcontractor budget spent on light infrastructure water projects. Records Proportion of funds that have been spent on light infrastructure water projects is reasonable. These projects are short- to medium-term response activities. Phase out funding for activities with short- to medium-term investment. Increased focus on more sustainable WASH programming. LBG Performance - Sanitation & Hygiene 17 1.3 Responsibility for sanitation is shared between different ministries. Interviews￾MHPPE, MWRI, Records Policies may exist but are not being enacted or enforced. Advocate for all sanitation responsibilities to be granted to a MWRI. capacity building of MWRI to enact sanitation policies, strategies, and enforcement mechanisms. 18 1.3 30% of people surveyed did not have any access to improved sanitation. Many households with access to a latrine share the latrine with multiple households. HH Survey, Observations Many people lack access to improved sanitation facilities. USAID/Sudan should focus its sanitation programming on creating demand and improving the policy environment for the expansion of household sanitation coverage. 19 1.3 150 slabs for household latrines have been constructed at 73 households. Records This represents a small number of households in Kator. Further sanitation awareness￾raising and demand creation to encourage people to construct household latrines. 20 1.3 Household latrine slab construction appears to be of acceptable technical quality. However, no latrine hole covers were observed in use. Toilet Observation Sheets Flies will not be blocked if covers are not in place and used properly, reducing public health benefits. Provision of latrine hole covers. Education on the importance of blocking flies. 21 1.3 Cost of constructing a latrine is higher than most families are currently willing or able to pay. Interviews￾multiple sources Households will not construct latrines until the perceived value of a latrine exceeds the cost. Continued awareness-raising and demand inducement. Investigate ways of lowering cost through bulk purchasing and bulk labor agreements with local masons. 22 1.3 Four local masons in Kator have been trained in construction of low-cost slabs. Interviews￾Compass, LBG Knowledge of constructing low-cost slab is within Kator community. Continue to provide opportunities for these masons to construct low-cost slabs. Look into ways of expanding low-cost slabs to other areas. 23 1.3 Household latrines were constructed in areas that have not yet been surveyed and demarcated. Interviews￾Compass, LBG Possible that some latrines will be demolished. Also people are hesitant to complete construction of latrine superstructure because they fear demolition. Future areas targeted for household latrines should be officially demarcated. It is acknowledged that this may be challenging, given the past history of cholera outbreak prevalence in non-demarcated Evaluation Report Urban WASH, Juba, Southern Sudan 39 No TOR Finding Source Conclusions Recommendations areas. 24 1.3 Many households in Juba are renters. Renters have little incentive to construct household latrines. Interviews￾multiple sources It would be possible to require owners to construct toilets or latrines prior to renting houses. Work with payam and county officials to promote legislation requiring owners to construct latrines for all rental properties. Build capacity of enforcement. 25 1.3 Payam officials do not have guidelines/drawings for standard toilets. Interviews￾former MoH official Payam officials do not necessarily have the technical guidance necessary for promoting household latrines. Work with payam officials and engineers to design standard latrine drawings and bills of quantity. 26 1.3 PSI methodology and materials for hygiene behavior change are well-designed. Observations, Interviews-PSI, PSI has invested in an effective approach for improving hygiene behavior changes. Continued use of the PSI multi channel behavior change approach. 27 1.3 Over 50% of people surveyed had attended a hygiene promotion session. HH Survey PSI has been able to reach people in the communities with hygiene behavior messaging. Hygiene behavior change programming should be continued in current areas and expanded to new areas. 28 1.3 Nearly 70% of people surveyed treat their water with WaterGuard or PUR. HH Survey PSI has been effective in social marketing of water point-of￾use treatment products. POU water treatment social marketing should be continued and expanded to new geographic areas. 29 1.3 Many households have water containers used for handwashing near their latrines. Observations, HH Survey People commonly wash hands with water after using toilets. Use of soap for handwashing is less common. Additional education on the importance of handwashing with soap and water. 30 1.3 Many households have soap. Soap is widely used for washing clothes. Soap is not widely used for handwashing at crucial times. Observations, HH Survey Households already have soap but are not always using it for handwashing at crucial times (before eating and after toilet use). Increased hygiene behavior change messaging, emphasizing importance of handwashing with soap. 31 1.3 Hygiene promotion activities are not specifically aligned with areas where LBG heavy water infrastructure work is occurring. Observations, Interviews-PSI, LBG Community members using new water infrastructure will not necessarily have exposure to improved hygiene behavior trainings. Specifically target areas benefiting from water infrastructure with hygiene behavior change trainings. 32 1.3 LBG has assisted PSI with data quality assurance. Interviews-PSI, LBG LBG has built the capacity of its subcontractors in data quality assurance. LBG should continue building capacity of subcontractors in data quality assurance. 33 1.3 Approximately 46% of LBG subcontractor budget spent on sanitation (11%) and hygiene (35%) projects. Records Proportion of funds that have been spent on sanitation corresponds with current enabling environment. Proportion of funds that have been spent on hygiene represents a significant investment in promoting Continued funding for hygiene behavior change activities. Increased funding for sanitation activities when enabling environment is improved. Evaluation Report Urban WASH, Juba, Southern Sudan 40 No TOR Finding Source Conclusions Recommendations hygiene behavior changes. LBG Performance -UWC Capacity Building 34 1.4 80% of tariffs collected by UWC go into large GOSS pool. No mechanism for UWC to directly use these funds. Interviews￾MWRI, UWC Difficult for UWC to receive funds necessary for ongoing operations and maintenance. Vulnerable to GOSS political forces. USAID/Sudan should consider providing technical assistance supporting sector policy dissemination, strategy development, investment planning, and institutional strengthening. The establishment of a UWC finance account separate from GOSS block account is essential for the long-term operations of a water facility. 35 1.4 UWC does not know the gap between its operating expenses and its tariff revenues, but it is widely thought that revenues from users are insufficient to cover operations and maintenance costs. Interviews￾UWC, Records It will be difficult for UWC to sustain operations in the short term without GOSS subsidy funds. UWC needs support in developing a comprehensive business model for the long￾term sustainability of the water utility. USAID/Sudan should consider providing technical assistance supporting the establishment of a working business model for UWC. Such a business model should be based on cost recovery for operations and maintenance costs. 36 1.4 Less than 20% of the Juba population is serviced by a UWC connection. Interviews￾MWRI, UWC Much further work on the urban water system is needed to service needs of population. Fixing leaking pipes and expanding the network should be prioritized. 37 1.4 UWC Administration does not know with certainty its percent collection of tariffs. UWC has ~2000 registered customers and it is estimated that ~90% households are paying fees to UWC. Interviews￾MWRI, UWC Fees collection data is not fully collected/monitored. Users are willing to pay for water services. Additional administrative assistance is needed to strengthen and improve Admin/Finance and data collection systems. Increasing collection of tariffs from institutional users should be prioritized. 38 1.4 UWC currently consumes ~3000 SDG/day fuel for running generators. When Kator pipeline is opened, estimated fuel cost will be ~4000 SDG/day. Interviews￾UWC UWC will require increased ability to generate and manage revenue. Capacity building to strengthen UWC administration and management. Establishment of UWC finance account separate from GOSS block account. 39 1.4 Government ministries, schools, and health centers do not pay UWC tariffs. Interviews￾UWC UWC does not collect any revenue for water it delivers to institutional users. Enable UWC to collect tariffs from institutional users. Capacity building to strengthen UWC administration and management. Evaluation Report Urban WASH, Juba, Southern Sudan 41 No TOR Finding Source Conclusions Recommendations 40 1.4 Little incentive for maximization of revenue from proposed tanker truck filling program at UWC filling stations. Interviews￾World Bank It is likely that any revenues collected will be less than what could have been collected. Installation of prepaid cards for tanker truck vendors. Management system ensuring that collected revenues are deposited into a UWC account to support operations costs. 41 1.4 Formal UWC administrative procedures either do not exist or are dated to colonial times. Informal systems exist. IDF tool, Interviews at UWC, Records Informal systems are being used. Formal administrative policies and procedures should be established. USAID/Sudan should consider supporting capacity building of administrative systems. 42 1.4 Formal UWC personnel systems either do not exist or are dated to colonial times. Informal systems exist. IDF tool, Interviews at UWC, Records Informal systems are being used. Formal human resources policies and procedures should be established. USAID/Sudan should consider supporting capacity building of human resources systems. 43 1.4 Dissemination of information to the public does not systematically occur. IDF tool, Interviews at UWC, Records Public is ill-informed of activities and actions of the UWC. Formal external communications policies and procedures should be established. USAID/Sudan should consider supporting capacity building of communications systems. 44 1.4 GtZ Technical Advisor who was expected to work closely with the UWC will be delayed in his arrival to Juba. Interviews￾GtZ UWC will be receiving some technical assistance from GtZ. More assistance for MWRI will be needed. USAID/Sudan should consider supporting a Technical Advisor to help strengthen the institutional framework and build the capacity of MWRI. 45 1.4 UWC does not have construction standards. Interviews￾UWC In the absence of construction standards, it is difficult to manage contractors and ensure technical quality. Construction standards should be developed. USAID/Sudan can support technical assistance in development of standards. 46 1.4 UWC has not yet set tariff rate for Kator standpipes. Interviews￾Kator Payam, LBG Kator Payam unable to determine fees that should be collected from users. UWC needs to establish tariff rate for Kator and Munuki standpipes. USAID/Sudan and LBG can support process. 47 1.4 UWC Planning Department was not involved in regular meetings between UWC and LBG. Interviews￾UWC Poor communication between UWC departments. Formal internal communications policies and procedures should be established. USAID/Sudan can support capacity building of communications systems. Evaluation Report Urban WASH, Juba, Southern Sudan 42 No TOR Finding Source Conclusions Recommendations 48 1.4 35% of people surveyed expect that government will take care of problems with their water source. HH Survey People expect government to provide water services and take care of problems that arise. This creates an enabling environment for a workable business model. USAID/Sudan should consider providing technical assistance supporting establishment of a working business model for UWC to expand provision of water services and address any problems that arise in services. 49 1.4 MWRI has requested support in policy setup, regulatory framework, and institutional development. Interviews￾MWRI MWRI currently lacks capacity in these key areas of its responsibility. USAID/Sudan should consider supporting a Technical Advisor to be placed in MWRI office to assist with institutional development and regulatory policy framework of MWRI. 50 1.4 MRWI lacks capacity to manage large numbers of contractors. Interviews￾MWRI Some contractors are not performing their jobs appropriately, and their work is not being supervised by the government. Build capacity of MWRI in contracts and financial management. Support additional training of government field supervisors. 51 1.4 Not all MWRI staff are aware of annual investment in WASH sector from donors and implementers. Interviews￾MWRI MWRI needs to improve information sharing from the Budget Sector Working Group processes. Strengthen capacity of MWRI to form plans for priority areas; strengthen capacity of MWRI to obtain plans from donors and implementers; strengthen capacity of MWRI to coordinate implementors. 52 1.4 MWRI is unable to locate some key regulatory framework documents. Interviews￾MWRI Maintaining records is currently a problem for MWRI. MWRI should establish a system to store important documents and records. USAID/Sudan can consider administrative support. 53 1.4 Strategy development process for WASH sector is stalled/on hold. Interviews￾MWRI, UNICEF There will be further delays with development of an urban water strategy. Begin urban water strategy process with technical support from GtZ. LBG Monitoring & Evaluation 54 2 Beneficiaries of point-of-use water treatment products are listed in the same table as number of people with improved access to safe water supply. Records Incorrect reporting. Point-of￾use water treatment does not equal increased access to improved water supply. LBG needs to revise its reporting on this standard indicator. Point-of-use data should appear in different table than number of people with access to improved water supply. 55 2 Beneficiaries for community toilets are listed in table for people with improved household access to sanitation. Records Incorrect reporting. The standard indicator for sanitation does not include users of public toilets. LBG needs to revise its reporting on this standard indicator. Users of communal toilets should appear in different table than number of people with access to improved sanitation. Evaluation Report Urban WASH, Juba, Southern Sudan 43 No TOR Finding Source Conclusions Recommendations 56 2 Target beneficiaries for water infrastructure are calculated using SPHERE guidelines for maximum number of users. Records SPHERE guidelines are not applicable for estimating number of beneficiaries of water infrastructure. LBG should revise the target beneficiaries based upon calculations of the amount of water being supplied. 57 2 It is difficult to read beneficiary numbers in LBG's indicator reporting table format, as some of the numbers do not align in one horizontal plane. Records Beneficiary numbers may be misread. LBG should revise its beneficiary table format to list all beneficiary numbers on the same line. 58 2 LBG indicator table in reports lists targets for Q1, Q3, and Q4, which equals the yearly target. Q2 data is not listed. Targets are listed, but not actual achieved. Records Table can be improved to be made more clear. LBG should revise the indicator reporting table to include actual achieved listed for each quarter. Also better labeling of table to easily indicate the relevant reporting period. 59 2 Some LBG indicators are reported as monthly snapshots, some are reported as averages over a period. Records, Interviews￾LBG Some confusion exists about the numbers being reported. LBG should have further discussion with USAID/Sudan regarding the ways in which the various indicators are to be reported (snapshot in time vs. averaged over time). 60 2 LBG has established a data quality management system that appears to be working well. Observations LBG is able to document data from project. None 61 2 For Annual Surveys, LBG collects data on percentage of households washing hands after toilet and before meals. Data on use of soap at these critical times is collected but not reported. Records, Interviews￾LBG It is unclear what percentage of households use soap for handwashing at critical times. LBG can compile percentage of households using soap at critical times, in addition to percentage of households washing hands at critical times. Data is already being collected. 62 2 Locations assessed in baseline survey do not necessarily correspond with locations where the majority of project work has occurred. Records, Interviews￾LBG Data contained in the baseline survey may not be indicative of changes that have occurred in communities as a result of the project. Exercise caution and judgment when using baseline data in any future comparisons for measuring project impact. 63 2 150 households surveyed for baseline. This represents approximately 1% of the estimated population at the time, although there was large uncertainty of the original population size, which made it difficult to perform proper surveying. Records, Interviews￾LBG Due to the small sampling size, data in the baseline survey may not necessarily be representative of the population of Juba. Exercise caution and judgment when using baseline data in any future comparisons for measuring project impact. APPENDIX C. DATA EVALUATION SCHEDULE Day Date Activity Participants Monday 17-Aug Team Planning Meeting with MSI and USAID Sam Huston George Wagwa Ami Henson Tuesday 18-Aug Meeting with USAID Meeting with LBG Tony Kolb Kola Fakorede Wednesday 19-Aug Meeting about tanker program Interview - UWC numerous government reps Santorino Tongun, John Nhial Thursday 20-Aug Interview - UWC Interview - Compass Joseph Ebere Kidi Samuel Friday 21-Aug Interview - MWRI Interview - CHF Interview - Kator Public Health Officer Inspect Juba Bridge Community Tank Lawrence Muludyang Fatmah Lemeria, Emmanuel Birokole Iddi Marjan, Atiya Jogotmorgon Saturday 22-Aug Site visits - Atlabara, Jebel, Juba, Gabat community toilets, Jebel Nyoka community tank Sam Huston Kola Fakorede Sunday 23-Aug Compile data Monday 24-Aug Check in with MSI CoP Interview - PSI Interview - Tearfund Oksana Chikina Claire Simmons Tuesday 25-Aug Interview - World Bank Interview - UWC Treatment Plant Interview - MHPPE Gunter Gutknecht Samuel Longa Morris Lomodong, Phillip Ayuel Wednesday 26-Aug Review of LBG Data Quality System Interview - MoH Kola Fakorede Dr. John Rumunu Thursday 27-Aug Interview - Ursatuna Health Center Interview - Carboni Missionaries Interview - USAID Field visit of PSI drama group Meeting with GtZ Maria Teresa Perego Valentino Fabres Mike Andreini Oksana Chikina Manfred Vaneckert Friday 28-Aug Debriefing for USAID Meeting and Interviews - MWRI Bill Hammink, Ami Henson, Sam Huston, Mike Andreini Numerous MWRI directors Saturday 29-Aug Household surveys - Juba, Kator, Munuki 3 PSI staff Evaluation Report Urban WASH, Juba, Southern Sudan 44 Evaluation Report Urban WASH, Juba, Southern Sudan 45 Day Date Activity Participants Sunday 30-Aug Compile data Monday 31-Aug Check in with MSI CoP Begin compiling matrix of findings Site visits - Lalugo Kola Fakorede Tuesday 1-Sep Interview - UNICEF Interview - ACF site visit - Munuki Interview - PSI Sahr Kemoh Paula Tenaglia Oksana Chikina Wednesday 2-Sep Work on matrix of findings Interview - LBG Sam Huston Kola Fakorede Thursday 3-Sep Work on matrix of findings Sam Huston Friday 4-Sep Oral presentation and discussion of findings Interview - GOAL Various key stakeholders Tom Ogello Saturday 5-Sep Draft report Sunday 6-Sep Review data Monday 7-Sep Draft report Tuesday 8-Sep Submit draft report to MSI Wednesday 9-Sep Work on LBG indicators calculation and reporting. Juba lockdown-no movement allowed. Depart Juba-5PM Sam Huston APPENDIX D. LIST OF KEY INFORMANTS INTERVIEWED Name Organization Title Date Sam Huston MSI Water and Sanitation Advisor for USAID 17-Aug Tony Kolb USAID Urban Health Advisor 18-Aug Kola Fakorede LBG Task Order Manager 18-Aug Sanortino Tongun UWC Director of Planning 19-Aug John Nhial UWC Senior Manager for Administration and Finance 19-Aug Joseph Ebere UWC Acting General Manager 20-Aug Kidi Samuel Compass Development Advisor 20-Aug Lawrence Muludyang MWRI Director Programming and Projects 21-Aug Fatmah Lemeria CHF Program Manager 21-Aug Emmanuel Birokole CHF Program Officer Chlorination 21-Aug Iddi Marjan LBG Community Development Coordinator 21-Aug Atiya Jogotmorgon Kator Payam Public Health Officer 21-Aug Oksana Chikina PSI Maternal Child Health Department Manager 24-Aug Claire Simmons Tearfund WASH Technical Advisor 24-Aug Gunter Gutknecht World Bank Water and Sanitation Advisor 25-Aug Samuel Longa UWC Area Manager 25-Aug Morris Lomodong MHPPE Director General for Sanitation 25-Aug Phillip Ayuel MHPPE Director for Sanitation 25-Aug Dr. John Rumunu formerly Ministry of Health former Director General for Preventative Medicine 26-Aug Maria Teresa Perego Ursatuna Health Center Senior Nurse and Logistician 27-Aug Valentino Fabres Comboni Missionaries Brother 27-Aug Mike Andreini USAID Health Development Officer 27-Aug Manfred Vaneckert GtZ Country Director 27-Aug Isaac Liabwel MWRI Undersecretary 28-Aug Alier Oka MWRI Director of Water Resources Management 28-Aug Peter Jalyalh MWRI Technical Advisor Rural Water Supply and Sanitation 28-Aug Gabriel Fonsiana MWRI Director of Irrigation 28-Aug Peter Mahal MWRI Director General Rural Water Supply and Sanitation 28-Aug James Boy MWRI Director General Hydrology and Projects 28-Aug Alex Rubew MWRI Director of Capacity Building and Training 28-Aug Emmanuel Parmenas MWRI Director of Operations and Maintenance 28-Aug James Yok MWRI Director General of Planning and Projects 28-Aug Sahr Kemoh UNICEF WES Specialist 1-Sep Paula Tenaglia Action Against Hunger Country Director 1-Sep Tom Ogello GOAL WASH Coordinator 4-Sep Evaluation Report Urban WASH, Juba, Southern Sudan 46 APPENDIX E. RECORDS REVIEWED GOSS (2007) Water Policy, MWRI, Juba. GOSS (2006) Southern Sudan Household Health Survey, Ministry of Health and Sudan Commission for Census, Statistics, and Evaluation, Juba. JICA (2009) Juba Urban Water Supply and Capacity Development Study in the Southern Sudan, Draft Final Report. Koenig, A. (2009) Country Status Overview on Water Supply and Sanitation – Southern Sudan, Zero draft report, Reform Consult, Nairobi. Kolb, A. and Rainey, R. (2007) Assessment Report: Urban Water, Sanitation, Hygiene (WASH) and Solid Waste Juba, Southern Sudan, USAID, Washington, DC. LBG (2009) Task Order 4: Urban WASH Program FY 2008-2009 Work Plan, Revision 2, Washington, DC. LBG (2009) Sudan Infrastructure Service Project FY 2009 MONTHLY PROGRESS REPORT 1 July – 31 July, 2009, Washington, DC. LBG (2009) Sudan Infrastructure Service Project FY 2009 Q3 PROGRESS REPORT 1 APRIL – 30 JUNE, 2009, Washington, DC. LBG (2009) Sudan Infrastructure Service Project FY 2009 Q2 PROGRESS REPORT 1 JANUARY – 31 MARCH, 2009, Washington, DC. LBG (2009) Revised Performance Monitoring Plan: Transport, Water and Energy Sector Programs, Sudan Infrastructure Services Project, Washington, DC. LBG (2008) Draft Performance Monitoring Plan: Transport, Water and Energy Sector Programs, Sudan Infrastructure Services Project, Washington, DC. LBG (2008) Task Order 4: Water Supply and Sanitation FY 2007-2008 Work Plan, Washington, DC. LBG (2008) Juba Community Water Project Baseline Assessment. UNICEF (2009) Knowledge, Attitudes and Practices (KAP) Survey on Water, Sanitation, Hygiene and Nutrition in Seven States of Southern Sudan, Juba. USAID (2007) Contract Order No. 650-I-00-06-00010-04, Community Water and Sanitation, Washington, DC. Evaluation Report Urban WASH, Juba, Southern Sudan 47 APPENDIX F-1. SURVEY OF HOUSEHOLD WATER SUPPLY, SANITATION AND HYGIENE Mid Term Review of TO4: WASH – Sudan Infrastructure Services Project, Juba, Sudan Male Youth (<12) Juba Female Adult Kator Elderly (>50) Munuki Date: _______________ Record No._______ NO. QUESTIONS AND OBSERVATIONS ANSWER OPTIONS 1. Do you have soap in your household? 1. Yes 2. No 2. When you used soap today or yesterday, what did you use it for? IF FOR WASHING MY OR MY CHILDREN’S HANDS IS MENTIONED, PROBE WHAT WAS THE OCCASION, BUT DO NOT READ THE ANSWERS. (DO NOT READ THE ANSWERS, ASK TO BE SPECIFIC, ENCOURAGE “WHAT ELSE” UNTIL NOTHING FURTHER IS MENTIONED AND CHECK ALL THAT APPLY) 1. Washing Clothes 2. Washing Pots/dishes 3. Washing My Body 4. Washing My Children 5. Washing My Children’s Hands 6. Washing Hands After Defecating 7. Washing Hands After Cleaning Baby 8. Washing Hands Before Preparing Food 9 Washing Hands Before Eating 3. Do the children in your house usually wash their hands before eating? 1. Yes 2. No 4. Do the children in your house usually wash their hands after going to the toilet? 1. Yes 2. No 5. Can you show me where you usually wash your hands and what you use to wash hands? 1. Inside/Near Toilet Facility 2. Inside/Near Kitchen/Cooking Place 3. Elsewhere In Yard 4. Don’t wash hands 6. OBSERVATION ONLY: IS THERE SOAP OR DETERGENT OR LOCALLY USED CLEANSING AGENT? 1. Soap 2. Ash 3. None 7. OBSERVATION ONLY: IS THERE WATER PRESENT? 1. Yes 2. No 8. OBSERVATION ONLY: IS THERE A HANDWASHING DEVICE SUCH AS A TAP, BASIN, BUCKET, SINK, OR TIPPY TAP? 1. Yes 2. No 9. Have you attended any sessions in the community where good hygiene practices were discussed? 1. Yes 2. No 10. What kind of toilet facility does your household use? 1. Flush/Pour-Flush Toilet 2. Ventilated Improved Pit Latrine (VIP) 3. Bush Latrine 4. No Latrine: Field, Bush 11. Do you usually use the latrine? 1. Yes 2. No 12. Is there a baby or infant in your household? 1. Yes 2. No Evaluation Report Urban WASH, Juba, Southern Sudan 48 Evaluation Report Urban WASH, Juba, Southern Sudan 49 NO. QUESTIONS AND OBSERVATIONS ANSWER OPTIONS 13. Where are the baby’s faeces disposed? 1. Dropped in toilet facility 2. Buried in yard 3. Put on trash pile in yard 4. Washed away, water connected to sink or drain 5. Washed away, water discarded outside 14. Do you treat your water in any way to make it safer for drinking? 1. Yes 2. No 15. IF YES, what do you usually do to the water to make it safer to drink? (CHECK ANY THAT APPLY) 1. Add Waterguard 2. Add PUR 3. Let It Stand and Settle/Sedimentation 16. How do you store drinking water? (OBSERVATIONS SHOULD VERIFY ANSWER GIVEN) 1. In Containers (Bucket, Jerry Can, Drum, Etc.) 2. No Water Stored 17. May I see the containers where you store water, please? (OBSERVATION) ARE THE CONTAINERS COVERED? 1. All Are 2. Some Are 3. None Are 18. If there is a problem with the water supply, who would solve it? 1. Outside government people 2. Families in community 3. NGO 4. Don’t know Evaluation Report Urban WASH, Juba, Southern Sudan 50 APPENDIX F-2. WATER SYSTEM OBSERVATION SHEET Assesses quality of construction, reliability of service, ability of users to maintain No Description Score = 2 Score = 1 Score = 0 Result 1 Source Protection No contamination at source, good engineered source protection No contamination at source, limited source protection Contamination at or near source 2 System components protected from harm Most system components hidden and/or protected Some system components hidden and/or protected Several system components are exposed 3 Apron Construction Good quality construction Satisfactory quality construction Poor quality construction 4 Drainage Good drainage at most of the water points (>75%) or if only 1 water point, drainage "good" Good drainage at about half of the water points (~50%) or if only 1 water point, drainage "mediocre" Good drainange at only some of the water points (~25%) or if only 1 water point, drainage "poor" 5 Water Quantity Water point always meets water needs Once or twice a month system does not deliver enough to meet needs Once or twice a week system does not deliver enough to meet needs 6 Water Quality Users feel that water quality is consistently good Users feel that water quality is good most times, with some exceptions Users feel that water quality is consistently not good 7 Predictability of Water Service Women can predict and influence the service hours Women know or can predict when water is available, but they have little influence over service hours Women can not predict hours of service 8 User Fees Payment records kept, and most or all households pay. Payment records kept, but some households do not pay. No payment records, or records not considently kept. 9 Maintenance Users have already performed routine maintenance Users seem able to perform routine maintenance Users do not seem able to perform routine maintenance 10 Spare Parts Spare parts are available in Juba and users know where to get parts Spare parts are available outside Juba and users know where to get parts Users don't know where to get spare parts 11 Management Water users group is active in solving problems and managing water system Water users group exists but is not very active. No water users group TOTAL SCORE APPENDIX F-3. MODIFIED IDF TOOL (BLANK) Start Up (0 Points) Development (1 Point) Expansion (2 Points) Consolidation (3 Points) Sustainability Organizational (4 Points) Features Criteria for Each Progressive Stage Mandates and core functions Ministerial structure under deliberation. Ministry and departmental mandates/structures and core functions defined (by decree or law); initial hiring started. Core functions put into practice; initial hiring completed. How is operational defined? Core functions fully operational; other functions at minimum capacity. All functions operational with critical mass of staff hired; agreed divisional/sectional mandates. Decision- making All decisions made by the Minister with no delegation. What is handed down? Some could define this as delegated. I would think that 0 would come if there are no written procedures in place. Executive decision-making structures defined (written procedures in place), most management decisions made by the Minister and Deputy Ministers. Formal decision-making system operational: management decisions increasingly delegated to department managers. Management decisions increasingly delegated to department managers. Management decisions consistently delegated to appropriate level of the Ministry. Management systems No administrative procedures formalized. Few administrative procedures formalized. Administrative manual in place, although not fully tested or revised. Procedures increasingly formalized. Administrative manual tested and revised, and considered the arbiter of procedures. Human resources—staff Existing staff not fully capable of providing skills required of their positions. Majority of staff participating in training for technical skills. Staff members possess minimum technical skills required of their positions but still lack broader communication skills. Staff members possess complete technical skills required of their positions and majority are participating in training for broader (communication) skills. Staff possesses all skills, including communication, leadership, team building, and management, along with a gender-balanced view of the role of women in government and society. Human resources— systems No formal personnel systems (job descriptions, recruitment, and hiring procedures, etc.) exist. Some, but not all necessary personnel systems exist. Virtually all necessary personnel systems are put into practice (procedures written, recruitment practices in place and operational, etc.). How is ‘institutionalized’ defined? Performance (merit) beginning to be recognized formally. Formal personnel systems are institutionalized, understood by employees, and redress can be pursued. Formal performance appraisal system in place with Evaluation Report Urban WASH, Juba, Southern Sudan 51 Evaluation Report Urban WASH, Juba, Southern Sudan 52 Start Up (0 Points) Development (1 Point) Expansion (2 Points) Consolidation (3 Points) Sustainability Organizational (4 Points) Features Criteria for Each Progressive Stage Little or no recognition of employee performance. provisions for merit-based rewards. Diversity— Women Staff is severely underrepresented by women. Some women are on staff but are not in decision￾making positions (or rarely contribute to decisions). Increased number of women on staff with some participation in decision making. Significant representation of women among staff, and women regularly participate decision making. Composition of staff adequately represents women, and they participate effectively in decision making. Diversity— Regions Regions disproportionately represented. Policy established to increase hiring from regions. Policy implemented for hiring from regions; hiring initiated. Regional representation noticeably improved. Proportional regional representation. Public information Little to no dissemination of information. Dissemination of information occurring but it is not formalized or institutionalized. Formal and regular procedures established for dissemination of public information. Formal and regular procedures for dissemination of public information in use. Civil society and public beginning to use public information from the Ministry/office. All public information disseminated regularly and effectively. Active use of public information from Ministry/office by civil society and public. Public participation in government Little to no interaction with civil society or citizens. Some interaction with civil society and citizens marked by procedures or events that allow for public input and discourse with Ministry, although informal or irregular. Formal systems/procedures exist for obtaining public input. Ministry engages in public debate/discourse with civil society on a regular basis. Citizen and civil society input incorporated into Ministry activities. Systems and procedures for input and discourse are institutionalized. Financial management— budgets No budget for Ministry administration or programs. Basic Ministry budget and financial management system exists. Ministry staff able to develop annual budget. Sufficient number of staff skilled in financial management. Financial management system integrated with government-wide FMIS. Actual Ministry expenditures within 10% of budget. Anti-corruption No anti-corruption systems in place. Anti-corruption systems defined and accepted by Ministry and government officials. Anti-corruption systems established. Staff informed and training underway. Some formal outputs from the systems, such as reports or visible enforcement. Anti-corruption systems fully institutionalized. Public perceives Ministry as non-corrupt. Source: Management Systems International IDF Tool as presented in TIPS Number 15, “Measuring Institutional Capacity”.