Page | 1 USAID - SENEGAL Purchase Order # AID-685-O-12-00110 Mid-term Evaluation of the USAID/PEPAM Water and Sanitation Project (685-A-00-09-00006-00) FINAL REPORT APRIL 17, 2013 Note: The consulting firm, Services de l’Energie en Milieu Sahélien (SEMIS) submitted a report in French to USAID/Senegal. To make the evaluation results available to a wider audience, USAID/Senegal translated the report body into English. In the event of ambiguity or conflict between this English version and the original French language version, the latter will take precedence. SEMIS contact information is: Espace Résidence, Immeuble 14-N°21, Hann Mariste. BP 652 Dakar RP, Sénégal, Tel: (221) 33 832 73 97 Fax: (221) 33 832 61 89. Email: dgsemis@semis.sn, Web: www.semis.sn - RC SN STL 87 B 69. NINEA: 24298942k2. Page | 2 Table of Contents Page 1. Acronyms 03 2. Executive Summary 04 3. Introduction and context of the intervention 09 3.1 Introductory note 09 3.2 Intervention framework 09 4. Evaluation methodology and limitations 10 4.1 Stages of the task 10 4.2 Profile of the evaluators 14 4.3 Study limitations 14 5. Study results 15 5.1 Relevance 15 5.2 Effectiveness 19 5.3 Efficiency 24 5.4 Viability and sustainability 28 5.5 Effects 37 5.6 Project governance 43 5.7 Capacity strengthening 46 5.8 Stakeholder satisfaction 51 5.9 Environmental aspects 53 5.10 Added value by the USAID/PEPAM project 55 5.11 Specific issues per component 58 6. Conclusion and recommendations 67 6.1 Conclusion 67 6.2 Recommendations 68 7. Annexes 7.1 Answers about the comments received on the draft report (See the French version) 7.2 USAID/PEPAM capacity strengthening summary table (See the French version) 7.3 List of people met (See the French version) 7.4 List of documents reviewed (See the French version) 7.5 Evaluation work plan (See the French version) 7.6 Terms of reference 71 Page | 3 ACRONYMS AEMV Adduction d’eau multi-villageoise (Multi-village water supply system) AEP Adduction d’eau potable (Potable water supply system) ARD Agence Régionale de Développement (Regional Development Agency) CFA F Communauté Financière Africaine Franc: The exchange rate used is $1=CFA F486.62147 CLTS Community-led total sanitation CUI Cadre Unifié d’Intervention (Intervention unified framework) DA Direction de l’Assainissement (Sanitation Directorate) DEEC Direction de l’Environnement et des Etablissements Classés (Directorate of the Environment and Classified Natural Resources) DEM Direction de l’Exploitation et de la Maintenance (Operations and Maintenance Directorate) DGPRE Direction de Gestion et Planification des Ressources en Eau (Water Resources Management and Planning Directorate) DH Direction de l’Hydraulique (Hydraulics Directorate) DLM Dispositifs de lave mains (washbasin devices) DLV Ventilated double latrines DREEC Direction Régionale de l’Environnement et des Etablissements Classés (Regional Directorate of the Environment and Classified Natural Resources) FDAL Fin à la défécation à l’air libre (End of open-air defecation) GOS Government of Senegal IDEN Inspection Départementale de l’Education Nationale (Local Area Education Inspection) IEC Information, Education, and Communication MDG Millennium Development Goals (Objectifs de Développement du Millénaire) MeRO Mise en Responsabilité et l’Opérationnalisation (Establishment of Responsibility and Operationalization) MOU Memorandum of Understanding NGO Non-governmental organization (Organisation non gouvernementale) PEPAM Programme d’Eau Potable et d’Assainissement du Millénaire (Millennium potable water and sanitation program) PGES Plan de Gestion Environnementale et Sociale (Social and environmental management plan) PHAST Participatory Hygiene and Sanitation Transformation PLHA Plan Local d’Hydraulique et d’Assainissement (Local Water Supply and Sanitation Plan) PMH Pompe à Motricité Humaine (human motor pump) SARAR Self-esteem, Associative strengths, Resourcefulness, Action-planning and Responsibility SRHYG Service Régional de l’Hygiène (Regional hygiene office) USAID United States Agency for International Development VIP Improved double pit latrine (Latrine améliorée à double fosses ventilée) WADA Water and Development Alliance WASH Water, Sanitation, and Hygiene WSS Water Supply and Sanitation Services WUA Water Users’ Association Page | 4 2. Executive Summary Senegal has set its 2015 Millennium Development Goals (MDGs) for access to safe drinking water and sanitation. On July 21, 2009, the US Agency for International Development (USAID) negotiated the Senegal Millennium Water and Sanitation Program (USAID/PEPAM) to contribute to the achievement of the MDGs for water and sanitation. Specifically, over the July 21, 2009 – June 30, 2014 period, the USAID/PEPAM project aims at improving sustainable access to Water supply and Sanitation Services (WSS) and promoting better hygiene in targeted rural, small town, and peri-urban areas of Senegal. This $21,672,717 worth project is being implemented by a consortium led by Research Triangle Institute International as prime contractor, with the support of TetraTech/Associates in Rural Development Inc., and Relief International/Enterprise Works/VITA as sub-contractors. There are also local non-governmental organizations (NGOs) which collaborate with the consortium and play an important role in the project implementation in the regions of Ziguinchor, Sédhiou, and Kolda, Tambacounda, and Dakar. Eventually, 141,850 people and 93,000 people are expected to benefit from respectively the water and the sanitation interventions. The project implementation approaches include (1) establishment of management committees at each site; (2) partnerships with Local NGOs/CBOs; (3) participatory planning; (4) design and dissemination of tools; (5) capacity strengthening; (6) signature of commitment letters; (7) demand generation; (8) promotion of sustainable and low-cost solutions; (9) promotion of community led total sanitation (CLTS) activities; (10) hygiene promotion campaigns; and (11) implementation of water, sanitation, and hygiene (WASH) in schools as part of the integrated approach. The expected results are the following: 1. Local management of WSS 2. Increased demand for services relating to water supply and sanitation 3. Strengthened local capacities for services relating to water supply and sanitation 4. Water and sanitation infrastructure are constructed / rehabilitated 5. Local management of WASH promotion is improved by using CLTS and WASH in schools. The Economic Growth office of USAID/Senegal commissioned the mid-term evaluation (July 2009 – March 2012) of its USAID/PEPAM project. The consulting firm submitted on April 17, 2013their final evaluation report. This performance evaluation aimed to assess progress to date and identify improvements that will facilitate the attainment of planned results. This independent performance evaluation was conducted by a Senegalese consulting firm “Services de l’Energie en Milieu Sahélien” (SEMIS). A multidisciplinary four-member team performed the contract, using mix-methods that include (a) entry working sessions; (b) document review; (c) visits to 31 localities; (d) interviews with key informant (locally Page | 5 elected and administrative authorities, public technical offices, private service providers, and beneficiary populations); (e) focus group discussions; and (f) household survey. There were over 30 evaluation questions addressing thematic areas, including relevance of the project, efficiency and effectiveness of the implementation, organizational and institutional changes, capacity strengthening, and viability and sustainability of the interventions. The following major findings and conclusions are organized around those thematic areas. Relevance of the project 1. The project aligns with the host country’s water and sanitation-related priorities by consulting and exchanging with the key stakeholders during the planning phase. 2. The sanitation-related proposed solutions perfectly meet the expectations of the beneficiaries, whereas, the proposed potable water-related technological option, although suitable for some areas, is not appropriate for solving potable water access problems in a sustained and effective manner. 50% of the households report that their water-related needs are satisfied, as the water points are not far from their compounds. The arduous nature of the manual pumping of water remains an issue, even though it has slightly decreased with the India pump, unlike with the Erobon pump. The most utilized water technological option (mini borehole equipped with water hand pump) was not that consistent with the expectations of the populations and technical offices. However, the mini-boreholes are conventionally appropriate technology making possible to construct infrastructure of good quality at reasonable cost within the respite. 3. The regional technical public services delivered compliance certificates to the project, which, however, did not always conform to the required water-related technical options, and management committees. The sector policy requires large and medium￾sized infrastructure instead of mini-boreholes. 4. Overall, the project satisfies the needs of the populations who are willing to pay more to get better supplies and services. The households surveyed report that the project implemented its potable water access and sanitation components in a satisfactory manner. The satisfaction rates are 60% and 93% respectively for potable water and sanitation. One reason why households expressed dissatisfaction relates to water needs being partly met. It appeared that water for other purposes than drinking was overlooked because of non-inclusive planning phases. The use of the project-built water source is real and the water quality is assessed positively by the households which consider it to be very good (93%) to good (6%). A very low proportion (1%) of households rated poor the water quality and the reasons put forward are usually related to the bad location of the infrastructure or to the mishandling or conservation spoiling the water smell, taste or color. Implementation of the project 1. At mid-course, 16% of the targeted people (22,500 beneficiaries) gained access to potable water, and 26% of the targeted people (24,500 beneficiaries) gained access to sanitation. Page | 6 2. The targets were not met as evidenced by the 25% completion rate of the infrastructure: 21% (water) and 31% (sanitation). This picture is worrying when it is compared to the 48% disbursement rate of the LOP budget. In sum, the implementation was weakly effective and efficient. 3. The line item relating to sub-contract, consultancy and subsidy, is barely executed, even though it is supposedly the substance and the means to achieve the set project objective. The execution rate is very low by totaling 11%. This low rate resulted from long delays in procurement and execution. 4. Overall, collaboration showed to be difficult because of misunderstanding of counterpart public offices of their roles and responsibilities. There was no strategic oversight and difficulties relating to collaboration with the partner NGO and some public technical line offices at sub-national level were experienced during the implementation. 5. Water and Development Alliance (WADA), Peace Corps/PROSPERE, Red Cross, and other donor interventions are opportunities to synergize. 6. The project has done right, specifically, regarding the development of local water supply and sanitation plans, the promotion of CLTS, the transfer of relevant technologies and practices, the introduction of a monitoring mechanism which integrates environmental control at all implementation stages, the design and dissemination of brochures and other communication tools. Organizational and institutional changes 1. The project led to a decrease in the use of unhealthy water sources by recipient households from 90% to 37%. 2. The open air defecation is ending because the project made it possible for 98% of the households surveyed to gain access to sanitary facilities thanks to subsidies and/or promotion of CLTS. Additionally, more and more households pay a special attention to the fence so that they can use the facilities day and night and therefore avoid resorting to the open air defecation during the day. In the CLTS localities the end of open air defecation stage was reached within the required timeframe. 3. The managing bodies of established water users’ associations are somehow functioning on the basis of meetings held, and meeting summary notes completed and publicly posted. However, regarding the renewal of managing body memberships, only 2 of the 28 managing bodies visited have renewed their memberships. In most instances, the individuals hold the positions they initially filled. This is understandable as the sites are small localities (less than 500 inhabitants) where there are scarce resource-people qualified for managing the water point. 4. The participation of women in water users associations managing bodies is noticeable. Women hold decision-making positions like the vice-president position which, de facto, is reserved for them. However, only 57% of the managing bodies have the vice-president positions filled. To say that even though the women are members of the committees, they do not fully hold this position. 5. The majority of the communities collectively make the decisions relating to the operation of the water point. This reflects a positive impact of the training of management committee members and the information and awareness messages delivered by the extension staff of the partner NGOs. The 30% of the management Page | 7 committees which do not account for the results and management activities align with the 33% of management committees which supply water free of charge and accordingly do not manage the mini-boreholes as agreed under the project framework. 6. 95% of surveyed households report regular hand washing practices. 7. The expenses to treat water-borne diseases have decreased. 8. Constructing mini-boreholes in schools yielded direct and positive effects on school attendance, as evidenced by a decrease in dropouts and by the fact that girls regularly attend schools. 9. 88% of surveyed households report gaining time and using this time to implement their economic activities predominantly in the agricultural sector. Moreover, the girls who were responsible for the water chore prior to the construction of the mini￾boreholes were able to benefit from more time for educational activities. 10. 97% of surveyed households reported that the feces-related illnesses decreased by 93%. Capacity strengthening 1. The project exceeded the targets set regarding the training of the partner NGOs, the private service providers, the relays, and the management committees’ members. 2. Building capacity within local NGOs allowed them to be the very first partners to implement water and sanitation activities locally. 3. Strengthening the capacity of builders and drillers allowed them to complete infrastructure that conform to the technical requirements, and export their technical expertise to Guinea Bissau. 4. Training the relays allowed them to generate demand of sanitary facilities. Viability and Sustainability 1. The established sanitation committees have operated beyond the end of CLTS-related interventions. In the non-CLTS sites they stopped operating after the intervention. 2. The traditional sanitary facilities in the CLTS sites collapse easily during the rainy season. 3. The populations and local NGOs are willing to contribute in cash and/or in kind. 4. The local governments are willing to pay for indigent individuals. 5. The financial viability (coverage of the recurrent costs by the financial surplus by management committees) varies according to the system and the region. The current financial viability of 65% is under serious threat because water is provided free of charge and the flat-rate system makes it difficult to mobilize savings. For 13% of the installed equipment the financial equilibrium is precarious. 6. The construction techniques and the types of materials used suggest that all conditions of technical sustainability of the infrastructure are met. However, the life of expectancy of the infrastructure is jeopardized by threats such as the borehole head is not cemented, the closure device of the emerging pipe is easy-to-open, the frequent breaking of the rope of the Erobon pump, using the manual pump to draw water is laboring, unavailability locally or even nationally of spare parts when needed, the ventilation pipe of the unused septic tank is not closed, and it is required to wait two years before emptying the Sanplat tanks. Page | 8 7. For the Sanplat latrine, the lid flagstone of the pit is a factor jeopardizing the maintenance and reuse of the infrastructure. As it is carried out today, it will be difficult for users to lift it up for emptying the sludge. 8. The hand washing devices may not be user-friendly for some household members (children and elderly cannot manipulate the devices easily). 9. The renewal of cans and other maintenance procedures are not well understood. The evaluation team made 28 recommendations, including the ones listed below. The project management team has to implement all, but one recommendation relating to preliminary studies to be conducted. One may argue that, additionally, USAID/Senegal/EGO is to ensure that steering committee meetings are regularly held. In this respect, the Government of Senegal is also responsible for proper oversight and coordination of the project. Relevance of the project 1. USAID/Senegal/EGO must ensure that all necessary feasibility and baseline studies are conducted to allow proper design and implementation. 2. The project is to also ensure compliance with Senegalese environmental regulations. 3. The project is to fulfill the requirements relating to water infrastructure type, management committee, water supply pricing, and tools. Implementation of the project 1. Shift to medium and large-sized infrastructure and/or focus on peri-urban areas. 2. Clarify the relationships with line technical Ministries and all other actors. 3. The line technical Ministries have to improve their internal communication, specifically, between the central and the sub-national levels. 4. Realign the budget and redefine the priorities and intervention strategy to ensure that the expected number of target populations will be achieved. Capacity strengthening The established management committees are to be trained and coached so that they can understand and properly use the management tools. Viability and Sustainability 1. Awareness raising and attitude change-related activities are to be continued. 2. Ensure that the local governments are early involved in the site selection and the implementation of activities. 3. Document the experiences learned about the mini-boreholes. 4. Partner/synergize with the other actors to make timely available spare parts. 5. Take the appropriate corrective actions to protect the infrastructure, and ease their maintenance. 6. For the Sanplat latrines, partition the pit lid flagstone to ease its opening and emptying. Page | 9 3. Introduction and Context of the Intervention 3.1 Introductory Note The mid-term evaluation is an activity performed in light of the project’s context and procedures. The work was carried out in an impartial manner by taking into account multiple legitimate viewpoints expressed by informants. Additionally, factual data provided by the project staff were used. The evaluation aims to know of the extent to which the objectives set within the results framework were carried-out at mid-course. The question is also about assessing that the expected results will be met by the project end date, and assessing the project’s environmental monitoring in target geographic zones. The evaluation questions listed in the terms of reference, the literature review, and the findings and information gathered from various actors (surveys and discussions) allowed the evaluation team to compare achievements against targets, and analyze qualitative aspects. Thus, in addition to providing a highly accurate picture of the implementation based on specific indicators, the project was analyzed through five performance criteria (relevance, effectiveness, efficiency, effects, and viability/durability) and four cross-cutting themes (project governance, stakeholders satisfaction, capacity strengthening, and project added value). Answers to some specific questions are provided to complement the performance analysis. Finally, recommendations are made to improve the project implementation. 3.2 Intervention Framework Goal Improve sustainable access to water supply and sanitation and promote better hygiene in rural, small-towns, and peri-urban areas of Senegal. Expected results 1. Local management of water supply and sanitation services (WSS) 2. Increased demand for services relating to water supply and sanitation 3. Strengthened local capacities for services relating to water supply and sanitation 4. Water and sanitation infrastructure are constructed/rehabilitated 5. Local management of water, sanitation, and hygiene promotion is improved by using community led total sanitation (CLTS); and water, sanitation, and hygiene (WASH) in schools. Life of project 5 years (July 21, 2009 – June 30, 2014) Intervention zone Regions of Ziguinchor, Sédhiou, Kolda, Tambacounda and Dakar Targeted populations (potable water) 141,850 people Targeted populations (sanitation) 93,000 people Operating method The project management and implementation have been entrusted by USAID to a consortium led by RTI International as prime contractor, with the support of TetraTech/Associates in Rural Development Inc.) and Relief International/Enterprise Works/VITA (RI/EWV) as sub￾contractors. There are 11 NGOs that collaborate with the consortium and play an important role in the project implementation: ADY, AJAC LUKAAL, AJAEDO, PACTE, WESWA, Kabonketoor, ENDA, CARITAS, CASADES, FODDE and Enfance et Paix. Page | 10 4. Evaluation Methodology and Limitations This brief presentation of the methodology presents the major stages of the evaluation. The attached work plan provides further information on site selection, the data collection tools designed, and issues to be investigated. 4.1 Stages of the Task 4.1.1 Stage 1: Kick-off meetings and literature review The kick-off meeting took place on October 12, 2012 in the USAID building. This was a contact-making relationship-building meeting between the evaluators and USAID/PEPAM. Additional meetings with USAID/PEPAM, held on October 16, October 31, November 6, and November 7, 2012, made it possible to select the sites to be visited by the evaluators for the purposes of household surveys and interviews with the project staff. After the kick-off meeting, USAID/PEPAM provided the evaluators with relevant information. As and when required, the evaluators asked for additional documents. USAID/PEPAM did not make all requested documents available. The most important documents needed to inform the evaluation include the cooperative agreement, the annual work plans, activity reports, procurement-related documents, financial supporting documents, site visit reports and minutes, and the work acceptance reports. The literature review made it possible to learn about the socio-economic aspects of the intervention zone and to fully understand the design, implementation and monitoring of the project at different levels. It also made it possible to take stock of the baseline data which guided the identification and definition of problems to be solved. The report and documents prepared during the project implementation provided information on the implementation approaches used and achievements made. The agreements helped to establish the rationale of the intervention: goals, purposes, expected results, activities, financing cost and plan. The institutional actors provided the evaluators with hydraulics and sanitation-related sector policy and regulatory documents. After reviewing those documents it was possible to assess the relevance of the project per the national sector guidance. 4.1.2 Stage 2: Discussions with the actors and field visits 4.1.2.1 Interviews In addition to the project staff, the evaluators met various actors, including: 1. Locally elected and administrative authorities: rural council presidents and governors; 2. Public technical offices: hydraulics directorate, sanitation directorate, regional technical offices for water, sanitation, hygiene, and environment; regional Page | 11 development agencies (ARD); well and borehole surveillance crews; education inspections; health posts; etc. 3. Private providers: partner NGOs, drillers, pump manufacturers, repairer artisans, etc. 4. Beneficiary populations. The interviews made it possible to better understand the project’s progress, its consistency with national strategies, accomplishments, shortfalls, and medium- and long￾term effects. To collect data, the evaluators designed discussion guides according to the actor-specific issues. The evaluators visited sites where infrastructure was constructed / rehabilitated. The site selection was based on various criteria, including geographic distribution, the type of infrastructure, equipment, sanitation facility, etc. The field visits and discussions with regional actors took place over the period December 3-14, 2012. The table below lists the partners met and sites visited. Page | 12 Actors met/site visited Ziguinchor Sédhiou Kolda USAID PEPAM field staff x x Governor x x x Rural Council President x x Regional Development Agency x x x DRH and BPF x x x SRA x x x Regional Hygiene Office x x x DREEC x x Education Inspection x x x Health Post Head nurse x x ADDY x AJAEDO x Kabonketoor x AJAC LUKAAL x PACTE x WESWA x Enfance et Paix x CASADES x x Enda Eau Populaire x FODDE x CARITAS x Mini borehole Enterprise x Pump repairer manufacturer x x x Bouyinor site x Ndiagne site x Diaghour site x Bourofaye Baïnouk site x Boulome site x Bélaye site x Badiana site x Bemet Djimandé x Marakissa HLM site x Bougnery site x Sankouya site x Djiragone Diafar site x Boussoura site x Saré Tening Mara site x Saré Yorohandeng site x Ngaliyel Amadou site x Saré Mamadi site x Sinthiang Diabatou site x 4.1.2.2 Household Surveys A sample of water and sanitation services users were surveyed to assess their involvement in the planning and implementation processes, their opinion on the technical quality of realized infrastructure, the organizational model established to ensure the viability of the water and sanitation services, the effects of the project on living conditions and their satisfaction level. In all, 469 households were surveyed, above the Page | 13 theoretical sample size estimated at 455 households (based on the number of households affected to date). The table below summarizes the sites and the relative number of households surveyed. Region Surveyed site Number of household surveys per site Total per region Region’ share in the total sample of households Ziguinchor BADIANA 10 150 32% BOUINOR 12 BOUROFAYE DIOLA 4 BOUTO 15 BOUYOUYE 2 DIEMBERING 21 MPACK 30 SELETY 19 TAKEME 12 TENGHORY 25 Sédhiou BOUSSOURA 11 158 34% DIAMBANCOUNDA 20 DJIRAGONE DIAFAR 17 KOUDJINY 7 MARAKISSA HLM 32 SANKOUYA 20 SARE DJIMBY 21 SARE TOUMANY MANDJACK 17 TALTO 13 Kolda KANIAKO 27 161 34% MEDINA ALPHA SADOU 10 MEDINA DIATTA SABALY 12 SAM PATHE 12 SAM SABALY 9 SARE DEMARRANG 18 SARE GOLO 7 SARE KONKO 9 SARE LOUNTANG 10 SARE MAMADY 15 SARE SAMBALLE 16 TABANDINTO 16 Total 469 469 100% 4.1.3 Stage 3: Data Processing and Analysis – Draft Report The household survey data were processed, using the Sphinx software. The qualitative data were processed according to the perspective of the project’s performance and crosscutting themes and issues, which were raised in the terms of reference. After the literature review, discussions and field visits, the logical framework matrix was reviewed so as to analyze the intervention in a simple, systematic, and understandable manner. That is a comprehensive view, checking easily the project rationale, its consistency and coherence. This analysis makes it possible to verify the achievement level of the results and to gauge hypotheses that are beyond control but can influence the project’s success. The data were processed from December 17, 2012 through January 15, 2013. While processing the data, clarification and additional information were asked from various actors to ensure results, analyses, and conclusions were accurate. After the data were processed, the preliminary report was drafted. Page | 14 4.1.4 Stage 4: Feedback Request – Final Report On February 19, 2013 a report review meeting took place in the USAID/Senegal building. At that date, the evaluators had received comments from only USAID/Senegal. The hydraulics and sanitation offices sent their comments later, on February 27, 2013. USAID/PEPAM sent its first comments on February 26, 2013, and additional feedback on March 11, 2013, which was received on March 25, 2013. An electronic mail was sent to USAID/Senegal to explain the delay between the second comments and the date the evaluators received them. Some comments were taken into account when drafting the final the report. For the rest, the evaluators provided answers that are appended to the final report (French version). 4.2 Profile of the evaluators A multidisciplinary team made up of the following experts carried out the evaluation:  Team Leader: a water and sanitation engineer, with 12 years of work experience in the implementation and evaluation of water and sanitation programs and projects in Senegal and the western Africa region.  A socio-economist with 8 years of work experience and a good knowledge of rural sociology, organizational dynamics and capacity strengthening-related activities benefiting to potable water users.  An environmental/hydro-geologist with 14 years of work experience; a good knowledge of implementation and monitoring strategy of environmental impact study, and social and environmental management plan (PGES); and an understanding of hydro-geologic issues relating to the intervention zone.  A CLTS and WASH expert, social worker with 10 years of work experience, a good understanding of issues relating to hygiene; sanitation; information, education, and communication (IEC); and monitoring of beneficiaries. 4.3 Limitations of the Study During focus group discussions (primarily in the Ziguinchor zone), there was a feeling that the beneficiaries prepared beforehand and knew exactly what words to use and what answers to provide to the evaluators, which is not a guarantee of objectivity. In such situations, in order to cross-check (triangulate) information, the evaluators asked questions to various actors (members of managing bodies, common users, village chiefs, youth, women, etc.); selected households other than those pre-identified; and ensured that the floor was not monopolized by a single respondent. Some requested information was not made available by USAID/PEPAM for confidentiality reasons, which was not explained. As a result, the evaluators were not able to comment on the enforcement of the recommended financial practices, the bidding processes, the per capita direct investment, and the wealth creation level. Page | 15 5. Results of the Study The project’s results were at first analyzed through five performance criteria (relevance, effectiveness, efficiency, viability and effects), then, through cross-cutting themes (project governance, capacity strengthening, environmental aspects, and project added value). Answers to some specific questions are provided to complement the performance analysis. 5.1 Relevance The relevance assessment is based on the analysis of the project consistency with the needs of the populations and the strategic guidance of the host country and its financial backer. For the specific case of USAID/PEPAM, the analysis was performed from three main perspectives:  The relevance of the project objectives in relation to the needs of the populations;  The project consistency with the strategic guidance of the water and sanitation sector policy relating to rural areas; and  The consistency of the project with the country strategy of the financial backer. 5.1.1 Taking into account the needs of the populations The household surveys and the semi-structured interviews of the various actors made it possible to analyze how the populations’ priority needs are addressed. The surveys show that the main water supply source of the populations in the intervention zone was made up of traditional wells using manual pumping, as 90% of the households surveyed reported using those water supply sources. The water drawn from those wells was often unfit for human consumption (no lids on the wells, outcrop of the water table, non-compliance with the minimal distance required to construct sanitary facilities, etc.), and this contributed to worsening health conditions. It is estimated that 5% of the households surveyed had access to potable water before the project. The project made it possible to decrease the use of the traditional wells, as evidenced by the following household survey results:  The proportion of households reporting the exclusive use of traditional wells decreased from 90% to 37%, a 53 percentage point decrease;  34% of the households surveyed reported exclusively using water sources made available by the project;  27% of the households surveyed reported using both traditional wells and the water sources made available by the project; and  2% of the households surveyed reported using all water sources other than those mentioned above. As a result of the project, the proportion of the targeted households surveyed reporting access to potable water increased from 5% to 60%, a 55 percentage point increase. 50% of the households surveyed reported that the water source made available was primarily Page | 16 used for drinking and cooking. However, the semi-structured interviews of sub-national offices of hydraulics show that the carried out mini-boreholes only partially meet the priority needs of the populations because the mini-boreholes do not entirely resolve the the difficulty in pumping water manually. Regarding sanitation, the proportion of the households surveyed reporting access to sanitary facilities increased from 3% to 80%, a 77 percentage point increase. The facilities made available include improved double pit latrines (VIP), ventilated double latrine (DLV), and Sanplat1 . When the sanitary facilities constructed through the CLTS promotion are considered, then, the access rate increased to 98% (18% of households which were surveyed constructed sanitary facilities following the CLTS-related promotional activities). The semi-structured interviews of sub-national offices of sanitation corroborate this high access rate as they show that the project perfectly meets the expressed needs. Yet, the lack of awareness-raising in the non-CLTS sites and the CLTS infrastructure easily collapsing jeopardize the continuous meeting of the end of open air defecation (FDAL) requirements. In sum, the program objective accords with the water and sanitation-related needs and concerns of the populations. 5.1.2 Consistency with the Sector Policy 5.1.2.1 Alignment with PEPAM The international community is committed to achieving poverty reduction objectives by 2015. To keep up with this commitment, Senegal has been implementing PEPAM since 2005. The framework for PEPAM is the second Rural and Urban Hydraulics and Sanitation Sector Policy document. The PEPAM is a framework which urges all financial and technical partners to report their activities for better synergy and a proper monitoring of access indicators. All contributors must implement activities relating to at least one of the three components required for the PEPAM implementation: potable water infrastructure, sanitary facilities, and unified intervention framework. The project clearly aligns with the PEPAM overall framework by deploying its activities under five components. 5.1.2.2 Planning Process The Government of Senegal (GOS) technical offices of Hydraulics and Sanitation proposed the project intervention sites in the Ziguinchor region. The project resorted to this strategy because there were few, if any, PLHAs and it was urgent to start implementation. The workshop held on November 29, 2009 in the ARD building in Ziguinchor served as a means of validating the site pre-selection made by the Hydraulics and Sanitation technical offices (see answers provided by USAID/PEPAM). The most 1 Sanplat is a model of latrine promoted by the USAID/PEPAM project because it is environmentally friendly and affordable, as it is reproducible in households without grants or subsidies. Page | 17 utilized technological option (mini borehole equipped with water hand pump) was not consistent with the expectations of the populations and technical offices. This slowed down the implementation process during the first year. For the regions of Sédhiou and Kolda, the sites were selected based on the “informed demand” approach with a review of existing planning documents beforehand (PLHAs and regional integrated development plans) and scouting visits. For those regions, the planning process was more participatory with the involvement of almost all actors concerned by the establishment of basic social services (populations, local governments, administrative executives, sub-national technical offices, NGOs, etc.). In sum, the initial planning for Ziguinchor was not totally consistent with the PEPAM guidance in terms of procedures, but the process abided by the fundamental principle of consultation and exchange between actors to lead to action prioritization. In Sédhiou and Kolda, however, the approach complies with the PEPAM water project manual, which refers to PLHAs for identifying priority projects. It should be noted that USAID/PEPAM later carried out PLHAs in the regions of Ziguinchor and Kolda. Those tools were used to identify some investments in the Kolda region, which was not the case in the Ziguinchor region. When designing future interventions, those PLHAs would be an important input and could be taken into account in order to have a better visibility of the potable water supply and sanitation situation and to target priority zones and projects. 5.1.2.3 Infrastructure Building and Capacity Strengthening Water infrastructure type For potable water, the PEPAM water project manual released in 2007 proposes the multi￾village system as the priority choice for achieving the access objectives relating to potable water in rural areas through standpipes and private tapping/connection. In the absence of any multi-village system option, for the localities of less than 1,000 inhabitants, transitional solutions such as manual water drawing and pumping can be proposed provided that:  the locality is not located within a radius of 5 km of multi-village borehole; and  the locality is not located within a 10 km x 10 km area that can gather at least 2,500 people for a multi-village system. Looking closely at the achievements of the USAID/PEPAM project, it is difficult to understand why in some areas the technical option was to build mini-boreholes instead of a multi-village water connection system, which is recommended by the water project manual. For example with better planning it would be possible to group the sites of Belaye, Badiana Eghana, Badiana Eleout, Badiana Gambissara and Badiana CEM in a same system. Water pumping equipment Page | 18 The hand pump type recommended by the manual is the India Mark II, which is more efficient and fairly understood by local craftsmen. On this point, it should be noted that until March 2012, 70% of pumps installed by USAID/PEPAM have been the India Mark II type pumps, 30% of the pumps are Erobon pumps manufactured locally. In general, the project complies with the manual. Potable water-related capacity-strengthening Water users’ associations’ management committees were established to collect flat-rate fees. In some cases, there is free access. Even though the users received training modules to correctly fulfill their roles and responsibilities, it should be noted that the managing bodies established by USAID/PEPAM are not consistent with the 'Harmonization Guide on procedures for establishing water users’ associations and their mandates' and the extension of the principles of the rural borehole reform initiated from 1999 to 2005. Sanitary facility types The sanitation manual proposes a technical package composed of individual and collective facilities for disposal of excreta and wastewater. The technical package consists of:  Improved double pit latrine or pour-flush latrines;  Wash container, showers, washbasin device; and  Public convenience with multiple booths, separate partitions for men and women, including urinals and washbasins. The sanitary facilities built by USAID PEPAM, through sub-grants, comply with the technological package proposed by the PEPAM. It should be noted that the project has gone even further in popularizing two new types of facilities: the improved ventilated double pit latrine and the improved Sanplat latrine. (See viability analysis). Within the framework of the CLTS, the constructed traditional latrines are not consistent with the technological package. In most cases, these facilities do not last long, as they collapse easily during the rainy season. This was observed at Bouyinor, Diaghour, Marakissa HLM and Bougnery. However, the minimum requirements for the facility be declared as appropriate are met: ventilation pipe with mosquito mesh, closed defecation pit, fence and washbasin device. Sanitation-related organizational dynamics In terms of organizational arrangements, committees are established either to record the subsidy requests or lead the CLTS promotional process. In the subsidized sites, only very few facilities are functional after the project end, whereas, in the CLTS sites, the committees continuously ensure compliance with the FDAL requirements. 5.1.3 Consistency of the project with the USAID/Senegal country strategy According to the USAID/Senegal country strategy, helping the country achieve the MDGs of halving the proportion of people without access to potable water and basic sanitation, a major constraint to economic growth will be alleviated. This support will be Page | 19 implemented through an integrated approach to water resources management that could benefit various economic sectors including fisheries and agriculture and help improve biodiversity conservation. The review of the USAID/PEPAM project activities shows some consistency with the MDGs. However, the results framework does not reflect any articulation with the integrated management of water resources as input for economic growth. Notwithstanding this finding, the project has positively contributed to economic growth in the intervention area through making use of the local expertise for the activity implementation (local NGOs, builders and molders, drillers, etc.). Beyond the direct beneficiaries, the use of the services made available positively affects economic growth through improved living conditions of the populations. 5.2 Effectiveness Effectiveness assesses the achievement level of the results. To measure its performance, USAID/PEPAM was committed to reporting data for three USAID standard indicators. The effectiveness analysis will be based primarily on those indicators, and, will subsequently, be completed by an assessment of the set of results defined in the terms of reference. As a reminder, the project is divided into five components, and for each of them, results to be achieved have been defined. The effectiveness analysis is performed against the following items:  Infrastructure completion rate at mid-course;  Distribution of completed infrastructure;  Assessment of the USAID standard indicators; and  Assessment of the results by component. 5.2.1 Infrastructure completion rate at mid-course Description Life of Project Target (*) Accomplishment from 2009 to March 2012 (**) Completion rate by infrastructure type Completion rate by sector Potable water Water supply in peri-urban areas 3 0 0% 21% Multi-village water supply 2 0 0% Deep wells / hand pump 30 0 0% Medium-depth wells / hand pump 84 64 76% Simple wells / hand pump 313 61 19% Rehabilitate the existing wells and add hand pumps 168 11 7% Sanitation Sanitation system in peri￾urban areas 3 0 0% Latrines (Sanplat, DLV, VIP) 31% + washbasin devices (DLMS) 6300 2448 39% Public convenience (schools or markets) 15 1 7% Aggregate completion rate of infrastructure at mid-point 25% * Data originate from "USAID_PEPAM_Plan Dev Infr_AWP 3 2012 final V5_Fr_Oct 19 2011" ** Data originate from "USAID_PEPAM_Rpt_QR2_FY2012_French_May 2_2012_VersFin" At mid-course, the aggregate completion rate of the infrastructure is 25%. It demonstrates low project effectiveness in relation to the resources made available and the nature of work to be carried out. This low performance level requires the readjustment of the infrastructure implementation strategy. It should be noted that local enterprises have the capability to build the project’s planned infrastructure and some proven expertise is available to ensure a proper completion of the works. Thus, the project staff must be assigned to coordination and supervision tasks rather than technical studies or work implementation. The strategy of "making do" is different from project-controlled implementation. Page | 20 5.2.2 Distribution of completed infrastructure Region Potable water Sanitation Tapping infrastructure Pumping equipment Sanplat DLV VIP EP (public Mini convenience) boreholes by local providers Project-controlled borehole construction Rehabilitated wells INDIA EROBON SOLAR Ziguinchor 55 4 2 52 6 1 1014 516 209 1 Kolda 30 4 4 19 15 0 164 190 8 0 Sédhiou 28 4 5 14 17 0 242 104 1 0 All regions 113 12 11 85 38 1 1420 810 21 8 1 weigh 83% 9% 8% 69% 31% 1% 58% 33% 9% 136 124 2448 1 Recap by region Ziguinchor 45% 48% 71% Kolda 28% 27% 15% Sédhiou 27% 25% 14% All regions 100% 100% 100% 5.2.3 Assessment of the USAID standard indicators USAID Standard Indicators Life of project target ** Target for the 2009- March 2012 period Accomplishment from 2009 to March 2012 (**) Accomplishment against LOP target (%) Number of people in target areas with access to improved drinking water supply as a result of USG assistance 141 850 99 650 22 500 16% Number of people in target areas with access to improved sanitation facilities as a result of USG assistance 93 000 34 085 24 580 26% Number of producers’ organizations, water users’ associations, trade and business associations, and Community￾Based Organizations (CBOs) receiving USAID assistance 294 0 535 182% ** Data originate from "USAID_PEPAM_Rpt_QR2_FY2012_French_May 2_2012_VersFin" At mid-course, the LOP target set for indicator 3 was exceeded, whereas, the ones set for the first two standard indicators were not met. With regard to potable water, the situation is problematic. There have been hampering factors, including the reluctance or non-adherence of populations and technical services, the delay in the fund transfer by USAID/PEPAM, and sometimes difficult working conditions in rainy season. At this pace, the results will probably not be achieved unless the current technological option and the intervention strategy are readjusted. In this respect, the option to work in the large and medium-sized water infrastructure is to be fostered and additional resources will have to be found to meet the set target of people. With regard to sanitation, the performance level is acceptable, at mid-course, but, in a longer term, it will be very difficult or even impossible to meet the LOP target. 30,000 people are still to benefit from the project and the budgeted amount (US dollars 450,000) is likely to be insufficient if the project sticks to the planned collective or semi-collective systems, which would make it possible to 800 to 1,200 household connections (approximately 10,000 people). An additional hampering factor will be the upgrading of the downstream facilities. If the technical option and the set target of people are not readjusted, there must some budget realignment to make sufficient resources available. Otherwise, the other option would be to continue with the autonomous system on the outskirts of large urban centers to quickly meet the set target. Page | 21 5.2.4 Assessment of the results per component 5.2.4.1 Result 1: local management of WSS improved Three sub-results were defined to assess the attainment of the result Sub-result 1.1: Strengthened community role in making WSS decision Indicator number Indicator statement LOP target Actual data as of March 2012 Achievement rate 1.1.A Number of partner community WUAs that held at least three public meetings in each quarter to discuss water use issues and decisions 294 103 35% 1.1.B Percentage of WUA management positions in partner communities that are held by women 50% 45% 90% At mid-course, progress towards the sub-result is quite satisfactorily. For indicator 1.1.A, the attainment level is acceptable because it should be taken into account about one year￾inactivity period needed to build the infrastructure and establish the managing bodies. For indicator 1.1.B, the achievement level is very satisfactorily. Overall, the evaluators found that 30% to 40 % of the WUA managing body membership consisted of women. Keep the course for the infrastructure to be built in the future. Sub-result 1.2: Transparency of local WSS governance strengthened 1.2.A Number of partner community WUAs that recorded minutes (procès-verbal) from the three WUA meetings in each quarter 294 66 22% 1.2.B Number of partner community WUAs from which the recorded minutes (procès￾verbal) from the three WUA meetings per quarter were published or otherwise made publicly available 294 60 20% At mid-course, progress towards the sub-result is low. For indicator 1.2.A, the attainment level is very low. By comparing indicator 1.2.A with indicator 1.1.A, it emerges that awareness is to be enhanced on the traceability of the activities of the office. For indicator 1.2.B the finding is the same as the one for indicator 1.2.A. There is a need to review the strategy of strengthening the capacity and accompanying the managing bodies after training provision. Sub-result 1.3: Financial sustainability of local WSS institutions improved 1.3.A Number of WUA members in partner communities trained with program support in financial, technical, and/or administrative management 816 726 89% 1.3.B Number of WUAs in partner communities that have adopted financial practices, such as a fee collection system, advocated by USAID/PEPAM, the GOS, and/or key partners 294 53 18% At mid-course, progress towards the sub-result is acceptable, provided that additional information be provided for the indicator 1.3.B. For indicator, 1.3.A, the attainment level from the quantitative viewpoint is very satisfactorily. However, by comparing indicator 1.3.A with indicators 1.2.A and 1.2.B, it emerges from the qualitative terms that much efforts remain to be made in the information traceability. For indicator 1.3.B, the achievement level is very low. It is quite difficult to understand that there are WUAs which abide by the management principles recommended by the government, whereas the bulk of WUAs do not. The latter supply water free of charge or apply flat-rate systems. Such practices are not consistent with the sector guidance which recommends the volume sales. In some cases, the populations find solutions, which are besides not sustainable for amassing savings in the event of a breakdown of the pumping equipment. The evaluators requested that the list of WUAs contributing to indicator 1.3.B be made available but the information was not provided. 5.2.4.2 Result 2: Local demand for improved water supply and sanitation increased Two sub-results were defined to assess the attainment of the result Sub-result 2.1: Hygiene knowledge and attitudes improved Indicator number Indicator statement LOP target Actual data as of March 2012 Achievement rate 2.1.A Number of people in partner communities who received training in proper hygiene techniques and practices with program support 1536 534 35% 2.1.B Number of household visits to households in partner communities to provide WSS or hygiene IEC with program support 37800 7823 21% 2.1.C Number of schools in target areas visited to provide WSS or hygiene IEC with program support 90 33 37% 2.1.D Number of teachers who have received WSS or hygiene IEC materials and training through program activities 540 183 34% 2.1.E Liters of drinking water potentially treated with program-supported methods for point-of-use application as a result of USG assistance through USAID/PEPAM 6 644 490 417 1 221 620 0.02% At mid-course, progress towards the sub-result is acceptable. For indicator 2.1.A, the attainment level is acceptable. For indicator 2.1.B, the achievement level is low compared to the LOP target. This would demonstrate that the targeted localities are sparsely populated areas and the numbers of residents resulting from the use of the multipliers do not reflect the reality on ground. This finding is confirmed by the field visits, as in several localities it turned out that the actual population is often largely below the data provided. For indicator 2.1.C, the achievement level is acceptable, and at the end the expected result is achievable. For indicator 2.1.D, the achievement level is satisfactory compared to the LOP target. The bases are already made; the stretching can be done quickly. For indicator 2.1.E, the achievement level is very low compared to the LOP target. The relevance of this indicator is questionable because it seems difficult to quantify the water treated by chlorination by the households. Moreover, if the calculation basis is the number of Aquatab tablets sold, then it is not obvious to know neither the destination of the product nor the period of use. It would be more advisable to stop just to tablets sold or made available. Collecting appropriate data for this indicator requires a lot of resources and does not provide almost any added value to the related sub-result. Page | 22 Sub-result 2.2: Stakeholder mobilization for improved WSS at local levels increased Indicator number Indicator statement LOP target Actual data as of March 2012 Achievement rate 2.2.A Number of partner communities where residents have provided money or in-kind contributions for water or sanitation access 294 180 61% 2.2.B Number of village public meetings for water and sanitation IEC conducted in with program support 1872 2542 136% 2.2.C Number of mass media campaigns for water and sanitation IEC reaching targeted areas conducted with program support 14112 3089 22% At mid-course, progress towards the sub-result is satisfactory. For indicator 2.2.A, the attainment level is very significant. Keep the momentum in mobilizing the counterpart resources. For indicator 2.2.B, the LOP target is exceeded. This would demonstrate that a large number of discussions and social mobilizations were carried out. From the quantitative viewpoint, this number is impressive, but it remains the question relating to the delivery quality and the percussion of the delivered messages because in most of the visited sites, the water points and the household sanitary facilities are poorly maintained and in some cases the washbasin devices have disappeared. For indicator 2.2.C, the attainment level is low compared to the LOP target, which seems very high. This indicator must be assessed according to the number of requests received compared with the available offer so as to assess the penetration of the message and its effect on the populations. The relevance of this indicator is questionable. 5.2.4.3 Result 3: Local capacity in improved water and sanitation services strengthened Two sub-results were defined to assess the attainment of the result Sub-result 3.1: Private and small-scale WSS sector strengthened Indicator number Indicator statement LOP target Actual data as of March 2012 Achievement rate 3.1.A Number of producer’s organizations, water users associations, trade and business associations, and Community-Based Organizations (CBOs) receiving USAID assistance through the USAID/PEPAM program 294 535 182% At mid-course, the LOP target is exceeded. This LOP target is to be revised in case private sector providers are to be taken into account. The beneficiaries include builders, drillers, pump manufacturers and repairer artisans. (USAID Standard Indicator). Sub-result 3.2: Local delivery of WSS and products strengthened 3.2.A Number of program-assisted local private maintenance service providers who have at least one maintenance contract established with a WUA 294 42 14% 3.2.B Number of new works contracts established by program-trained or –assisted local providers of WSS or products 294 273 93% At mid-course, progress towards the sub-result is satisfactory subject to additional information. For indicator 3.2.A, the attainment level is quite low. This is understandable because the 2.5 years coincide with the first receipts and the start of operation by users. It would be important that the list of signatory localities of contracts be available for confirmation because on the field, the evaluators did not obtain any signed copy of maintenance contract despite their constant requests, and, additionally, the artisans they met reported that they have not yet signed any contract with the users. For indicator 3.2.B, the attainment level is very satisfactory. It should be noted that almost all contracts have been signed with USAID/PEPAM. It would be relevant to analyze contracts signed with partners other than USAID/PEPAM to assess the impact of the capacity strengthening activities initiated by the project. 5.2.4.4 Result 4: Local capacity to construct or rehabilitate water and sanitation infrastructure strengthened One sub-result was defined to assess the attainment of the result Sub-result 4.1: Local capacity to build appropriate WSS infrastructure strengthened Indicator number Indicator statement LOP target Actual data as of March 2012 Achievement rate 4.1.A Number of hand pumps fabricated locally with program support 80 38 48% 4.2.B Number of household latrines constructed and equipped with a hand-washing device with program support 5700 2458 43% 4.3.C Number of improved toilets provided in institutional settings (schools and health facilities) 9 1 11% At mid-course, progress towards the sub-result is acceptable. For indicator 4.1.A, the attainment level is satisfactory. For indicator 4.2.B, the attainment level is very satisfactory. Strategy to be continued. For indicator 4.1.C, the attainment level is low. This is understandable because an inactivity period was needed to identify and validate the public convenience model to be carried out. To meet the target, the constructions should start before mid-2013 to stay within the time limits. Page | 23 5.2.4.5 Result 5: Improved local management of water, sanitation, and hygiene promotion using CLTS and WASH in schools Three sub-results were defined to assess the attainment of the result Sub-result 5.1: Improved knowledge of best hygiene practices in the target communities Indicator number Indicator statement LOP target Actual data as of March 2012 Achievement rate A Number of villages meeting the end of open air defecation requirements within six months 108 24 22% B.1 Number of people benefiting from sanitary facilities through the CLTS program 22000 5627 26% B.2 Proportion of houses in the target communities having washbasin devices through the CLST program 100% 54% B.3 Number of people gaining access to an improved potable water source through the CLTS program 22000 5292 24% 1.3 Number of CLTS committees trained in hygiene education 108 30 28% 1.4 Number of members of CLTS committees trained in hygiene education Not set 357 1.5 Average proportion of women members of the CLTS committees trained in hygiene education Not set 51% 1.8 Number of community-based IEC/CLTS activities implemented Not set 295 1.9 Number of people sensitized on the end of open air defecation, hygiene promotion, and hand-washing Not set 4717 1.10 Average proportion of women sensitized on the end of open air defecation, hygiene promotion, and hand-washing in the partner communities Not set 41% 1.11 Average proportion of children in the partner communities sensitized on the hygiene promotion, hand-washing, and the management of a healthy surrounding Not set 34% 1.12 Number of liters of potable water treated thanks to the methods recommended by the CLTS program Not set 70600 At mid-course, progress towards the sub-result is satisfactory. For indicator A, the villages met the end of open air defecation requirements 6 to 7 months after the CLTS program was launched. The period between the launching and the celebration is thus consistent. However, to meet the LOP target, it is necessary to launch simultaneously two lots of six villages per region and conform to the six-month time limit per site. The attainment level is fair for indicators B.1 and B.3. For indicator B.2, the attainment level is significant but awareness is to be strengthened so as to reach all households of the villages in which the CLTS was launched. For the other indicators, there were no LOP targets set at the start but the current results are significant in relation to the localities in which the CLTS was launched. Sub-result 5.2: Improved and better managed school-based WASH activities B.4 Number of in-school children in target areas gaining access to improved sanitary facilities (public conveniences) through the CLTS program 0% B.5 Number of in-school children in target areas gaining access to improved potable water supply system through the CLTS program 0% No activity implemented. There is a need to start the process in schools as teachers have already been trained (Indicator 2.1.D) Sub-result 5.3: Community commitment to the management of WSS enhanced 1.1 Number of CLTS committees established 108 35 32% 1.2 Average proportion of women members of the CLTS committees Not set 53% 1.6 Number of CLTS committees having developed their quarterly work plans 108 36 33% 1.7 Number of CLTS committees having implemented their quarterly work plans in a satisfactory manner 108 23 21% At mid-course, progress towards the sub-result is significant. Maintain the agreed upon strategy and launch simultaneously two lots of six villages per region and conform to the six-month time limit per site. Page | 24 5.3 Efficiency Efficiency studies the relationship between the means utilized and their costs, on the one hand, and, on the other hand, the relationship between the means and the investments made. It also studies the relationship between implemented activities and the implementation time limits. In particular, it will be suitable for assessing the following elements:  The procurement procedures;  the execution time lines;  mid-term budget execution; and  costs and achieved results. 5.3.1 The procurement procedures The project constructs by day labor potable water and sanitation infrastructure. It also commissions private providers to execute certain tasks (service contracts) or for the acquisition of supplies and equipment. The evaluators requested of the project implementer the list of contracts but did not obtain the document. The purpose was to analyze the procurement process according to the procedures used. Due to the lack of information, it was not possible to comment on the proportion of contracts in mutual agreement or in calls to competition. Accordingly, the evaluators could not express any opinion on the fulfillment of the quality/cost ratio and the principles of equity, transparency, and free and open competition. However, on the ground the evaluators found that sanitary facilities were constructed by trained local builders. USAID/PEPAM constructed by day labor some mini-boreholes when it came to sites where the manual drilling was not possible due to the unfavorable hydrogeological conditions (deep water table, rocky terrain). The other mini-boreholes were constructed by private service providers trained through the project. As of March 2012, four training sessions involving each two enterprises were organized. Infrastructure was carried out during the training period. The remaining infrastructure was distributed equitably, as far as possible, between enterprises having at least one staff member trained through the project. On the basis of the information provided and relating to the achievements as of March 2012, ten enterprises have carried out infrastructure during the period beside the training period. The following table summarizes the infrastructure carried out by the various service providers. Page | 25 Nº Provider Number of infrastructure built Quota 1 USAID PEPAM construction by day labor 12 9.6% 2 Construction work during training sessions 11 8.8% 2.1 Assukaten Water and Diaminda 3 2.4% 2.2 Ndèye Marie and Mballo 2 1.6% 2.3 Apas and Koutokoyo 3 2.4% 2.4 Sow Kunda and HBM 3 2.4% 3 Construction work by enterprises having at least one staff member trained 102 81.6% 3.1 Assukaten Water 14 11.2% 3.2 Diaminda Service 13 10.4% 3.3 Entreprise SAGNA 15 12% 3.4 Mballo and sons 12 9.6% 3.5 APAS 9 7.2% 3.6 Koutoukoyo 1 0.8% 3.7 CASAPICA 6 4.8% 3.8 KEDEBAYE 6 4.8% 3.9 HBM 13 10.4% 3.10 Omar Sow 13 10.4% Total 125 100.0% 5.3.2 Execution time lines Due to a lack of visibility on previous contracts and their management, it is difficult to comment on their meeting or exceeding the execution deadlines. However, given the very low physical completion rate, it is clear that the procurement processes and/or the construction work execution experienced significant delays. In terms of delay in the execution of the construction work, the case of Djiragone Diafar is noticeable in the sense that the borehole was drilled in November 2011 and as of December 2012 (12 months later) no activity had been completed in order to carry out the work of superstructure and network. The populations who have been looking forward to seeing the execution of this complementary work are becoming totally disheartened. Construction works which generally last less than four months have been spun out when all prerequisites should have been met in year 1 of the project. 5.3.3 The budget execution The overall budget of the project is $ 21 672 717, USAID contributing $20 992 323 (96.9%) and the beneficiaries (populations and NGOs) $680 394 (3.1%). The resources made available by USAID should allow the payment of part of the construction works, the payment of the intellectual service provisions, the acquisition of equipment and logistics, the funding of the managing bodies’ operating costs, and the institutional support. The table below shows at mid-course the execution level of the USAID-funded budget. The data originate from the March 2012 quarterly report. Page | 26 Budget line item Description Total budget ($) Executed as of March 2012 Execution rate as of March 2012 A Personnel 2 540 911 1 555 909 61% B Social benefit 342 238 897 436 262% C Travel 378 853 461 180 122% D Equipment 598 726 212 093 35% E Supply 54 648 595 146 1089% F Sub-contract, consultancy, subsidy 11 917 377 1 275 682 11% G Other direct costs 1 909 087 3 965 859 208% H Total direct costs 8 963 306 8 963 306 51% I Total indirect costs 3 250 484 1 200 665 37% J Project total costs 10 163 971 10 163 971 48% At mid-course the budget execution rate is 48%. This hides a lot of very strong disparities with budget lines that have burst out, namely, the ones relating to benefits, travels, supplies, and other direct costs. On the contrary, the line item relating to sub-contract, consultancy and subsidy, is barely executed, even though it is supposedly the substance and the means to achieve the set project objective. The execution rate is very low by totaling 11%. The table shows that there is an urgent need to define a clear strategy enabling to achieve the expected results. The very low physical execution level of construction works corroborates this finding (see effectiveness analysis). Regarding the contribution of the beneficiaries, it is divided into two categories: the contribution of the populations to gain access to water infrastructure or sanitary facilities and the contribution of the project’s partner local NGOs. The latter contribution is the estimated value of the expenses incurred by the rental and the use of the rolling stock or office equipment. The table below shows the mobilization level of the contribution. Description Total budget ($) Executed as of March 2012 Execution rate as of March 2012 Populations 250 000 155 631.70 62% Local NGOs 209 889 179 428.62 85% Total 459 889 335 060.32 73% The level of efforts made by NGOs is important (an average of nearly $16,000 by organization). This demonstrates an active commitment to the development of their intervention areas. At mid-course, 73% of the contribution has been already mobilized. This is significant and again evidences the importance that the populations attach to the resolution of their access difficulties relating to potable water and sanitation. This would mean that theoretically, only 27% of the counterpart contribution is still to be mobilized when the physical execution level totals 25% (see effectiveness analysis). In other words, if the construction works, as they are planned, are carried out then the contributions will overly exceed the estimated amount. For this purpose, budget realignment would be needed to allocate additional funds to other line items and as top priority to the investment-related line items. 5.3.4 The costs and achieved results It would be important to analyze the direct investment-populations ratio. This exercise can be performed on the basis of the actual payments for the construction works and the support provided by the NGOs. The requested data was not made available by USAID/PEPAM for confidentially reasons. Accordingly, this ratio was not calculated. It should be noted that the ratio would be much more reliable, if it was calculated on the basis of actually reached populations and not estimated populations. Indeed, for the infrastructure already completed at mid-course, the population figures provided by the project were calculated on the basis of PEPAM multipliers: One simple mini-borehole or a rehabilitated well is equivalent to 150 people; one medium-sized or deep mini-borehole equals 300 people; and one household sanitary facility equals 10 people. Page | 27 Using the PEPAM multipliers is debatable and does not inform about the reality of the reached target. The multipliers have a sense at the planning stage and not at the implementation stage when there is a possibility to verify the information on the spot. Thus, the evaluators had lots of difficulties to implement the household surveys because the number of estimated people was largely inferior to the actual number. Accordingly and unless additional information is made available, dividing the investment by the reached inhabitants would not be pertinent. It is worth mentioning that USAID/PEPAM has used the PEPAM multipliers to meet the PEPAM reporting requirements. Moreover, at mid-course, $10 499 031 were already spent. USAID and the beneficiaries contributed $10 163 971 and $335 060, respectively. Only $1 610 742 of the total spending were used to access water and sanitation services, that is to say 15% of the total spending. It emerges from this that the investment share is very low and tremendous efforts are to be made to ensure efficiency. Budget realignment and a redefinition of priorities and the intervention strategy are needed to ensure that the expected number of target populations will be achieved. For illustrative purposes, the table below shows the water and sanitation services-related direct investment figures from various donor-funded projects in the water and sanitation sector. Project Budget ($) Direct investment Direct investment share Target number of people PEPAM BAD 1 45 761 662 38 222 033 84% 223 000 (potable water) 155 000 (sanitation) PEPAM LUX SEN 026 18 211 237 13 412 685 74% 94 000 (potable water) 50 000 (sanitation) PEPAM BA CTB 15 121 334 11 875 552 79% 90 000 (potable water) 40 000 (sanitation) 5.3.5 Contribution to the overall efforts of PEPAM The table below shows the project’s contribution to the PEPAM’s overall efforts to achieve the Millennium Development Goals relating to water and sanitation in rural areas. Indicator PEPAM Objective USAID/PEPAM achievement USAID/PEPAM contribution rate Amount of funding for water and sanitation $563,271,489.03 $10,480,425.37 1.86% Number of people reached by the potable water component 2 300 000 22 500 0.98% Number of household sanitary facilities 315 000 2 448 0.78% Number of public conveniences 3 360 1 0.03% The funding sought after by PEPAM represents an approximated investment of $246.6 per person to be served. As of March 2012, the actual per capita investment reported by USAID/PEPAM is approximately $466.48. This ratio will have to be revised downward before the end of the project by making real efforts in optimization and allocation of resources. It emerges from this that there is a real efficiency problem. Page | 28 5.4 Viability and sustainability The viability analysis makes it possible to assess whether the achieved results are likely to last, or even to increase in the long-run. The sustainability of the infrastructure funded through the project as well as their use and management should be assessed against criteria, including the technical, financial and organizational viability. 5.4.1 Technical viability It assesses the sustainability of the carried out infrastructure and the installed at mid￾course: mini-boreholes, pumping equipment, and sanitary facilities. 5.4.1.1 Mini-boreholes Three types of construction works were made: mini-borer, hand threshing borehole, and Rota Jetting. The project provided trainings to enterprises to provide services, and on the ground the infrastructure visited were consistent with the standard. In general, the construction of the mini-boreholes abides by the conventional techniques for constructing a water borehole. The Materials used to equip the mini-borehole, namely the pipe (full pipes and screens) and filter gravel are suitable for the installation environment. Based on the construction techniques and the types of materials used, all conditions of technical sustainability of the infrastructure are met. However, there are some risks that may have direct effects on the viability of the infrastructure. The evaluators found that after the end of drilling, and before the amenity works of surface and the installation of pumping equipment, borehole head is not cemented and the closure device of the emerging pipe is very "artisanal" and can be removed easily. This could have the following consequences:  the introduction of objects or hazardous products in the infrastructure. Such an introduction can contaminate water table, and  the emerging pipe break can lead to a resumption of the infrastructure or a failure to plumb with as corollary a rapid wear of the pump control rods. Moreover, the viability of a harnessing infrastructure is also linked to its production capacity. At the end of the boring, development is conducted using a plunged pump. This pumping makes it possible to remove the mud contained in the borehole and free the cracks of the linings to ensure a good water intake. Because the flow of the plunged pump is far superior to the flow of the hand pump to be installed, it was deemed unnecessary to conduct flow trials that could increase the cost of the infrastructure. The top gear flow is on the contrary higher. Taking those actions contribute to the sustainability of the infrastructure. Page | 29 5.4.1.2 Pumping equipment At mid-course, all installed pumping equipment, but one solar pump, are hand Erobon or India Mark II pumps.  The viability of the Erobon pump is strongly threatened because of frequent breakings of the rope and the arduous nature of manual pumping. In some visited sites, the rope can be used during only a week and if it breaks down, the tendency is to return to traditional water sources while the objective of the project is to offer a sustainable alternative. For the drawing, the manual pumping is very difficult for women. These two factors make that in the majority of sites a certain discouragement is perceptible at the level of populations and in the short term, there are threats that the equipment be abandoned as well as the water source unless a lasting solution is found.  After a few months of operation the project staff found that the India Mark II pump faced some sustainability issues such as the corrosion of parts. Appropriate Solutions have been found and the corroded equipment has been replaced. The viability of pumping equipment is also highly dependent on the availability of spare parts in the area.  The Erobon pump is manufactured locally. The only problem is the piston that is to be made with imported grain. The project is exploring ways to facilitate the manufacturer artisans’ access to this part. Otherwise, the viability of the pump is threatened.  The spare parts of the India pumps are imported. USAID/PEPAM is implementing a strategy facilitating the supply of users as and when needed. The project is collaborating with two companies (Sygelec hydraulic and the Sara materials enterprise) which ensure the importation of pumps and spare parts. Those companies are also importing pumps and parts for other partners. This approach is to be welcomed because in the absence of attractive market, the replacement of certain parts will be very difficult and will likely threaten the pumping equipment viability. 5.4.1.3 Sanitary facilities The sanitary facilities constructed in the subsidized sites are of good quality and are sustainable.  The VIPs have been well designed and well executed on the ground in compliance with the basic principles: double pits; large enough draining flagstones enabling an easy intervention; well performed ventilation chimney with anti-mosquito mesh; two holes of defecation with one of the foot rests closed; shelter with double hook door.  DLV are also well executed with circular pits. The beneficiary builds the shelter with his/her own means and, in most cases, it is straw-made and is in conformity with the habitat type. To improve the sustainability of the infrastructure and facilitate their subsequent maintenance, the top of the ventilation pipe of the unused pit should be closed to avoid the introduction of objects by children or the entry of water during the rainy season. The lid flagstone of the pit is another factor affecting the maintenance and reuse of the infrastructure. As it is carried Page | 30 out today, it will be difficult for users to lift it up for emptying the sludge. The ideal would be to split it into two parts to facilitate the draining.  The Sanplat built by the project are of very good quality and are different from the known "Sanplat" latrine. The carried out infrastructure consists of a circular pit with a masonry lining and a reinforced concrete flagstone fit with a defecation hole and a ventilation pipe. The beneficiary builds the shelter within his/her means. The lid flagstone of the pit is another factor affecting the maintenance and reuse of the infrastructure. As it is carried out today, it will be difficult for users to lift it up for emptying the sludge. The ideal would be to split it into two parts to facilitate the draining. Moreover, it should be noted that the beneficiaries of Sanplat facilities will face in the short term with a defecation place problem and will be forced to either practice open air defecation or use again the traditional latrines, unless they have adequate resources enabling them to build a similar infrastructure. Indeed, once the facility under use is filled, the beneficiary should wait at least two years before emptying the pit and resuming its use. If there's no alternative, all prerequisites relating to hygiene and sanitation practices will be lost and it is the return to the initial situation. In this case, would it not be more appropriate to popularize only the DLV? In summary, from the construction viewpoint, the carried out infrastructure are sustainable. The only factor that can lead to rapid degradation is the ways the infrastructure are used and maintained. The capacity-strengthening activities aimed at disseminating the principles and practices of hygiene for the proper maintenance of the carried out infrastructure. It should be noted on this point, that in the majority of the visited sites, the infrastructure is pretty well maintained. (See effects). 5.4.2 Financial viability The financial viability analysis relates particularly to the potable water component. Firstly, the financial viability of the systems made at mid-course, then, that of the mini￾potable water supply, and finally, proposals will be made to ensure the financial viability of large-sized infrastructure. 5.4.2.1 Systems made at mid-course At mid-course, the achievements include a borehole equipped with solar pump and mini￾boreholes equipped with hand India or Erobon pump. In the sites, water is not sold on the basis of the quantity. It is either free of charge or a flat-rate pricing. Generally, each adult aged over 15 years participates to the amount of CFA F 100 per month. The tables below present the financial viability analysis by region of the various systems made before a general summary. The data presented are from a cross of excel files provided by USAID/PEPAM (Status of water points made available and Monitoring of village cash desks). Page | 31 Financial viability of equipment in the region of Ziguinchor Pumping equipment type #of installed equipment # of committees having prepared a financial situation Cumulative amount of funds in CFA F as of September 2012 Average funds per committee Average number of months from the operation start to September 2012 Average amount per month and per committee in CFA F Solar 1 1 75,000 75,000 22 3,389 India 52 52 3,115,891 59,921 18 3,369 Erobon 6 6 467,545 77,924 22 3,593 For the solar pump, the average savings after 22 months of operation is CFA F75 000. This represents an average annual savings of about CFA F40 000. The lifetime of a solar pump is estimated at 6 years. The purchasing cost of a pump without panels and other accessories is approximately CFA F1 200 000. The provisions for renewal therefore amount on average to CFA F200 000 per year. It emerges from that the level of savings is largely below the minimum amount to ensure the renewal of equipment. The deficit is bigger when are taking into account other expenses relating to operation and functioning. If the management style is not changed or other alternatives are not found, any failure will translate into a water service outage. The system as it operates now is not financially viable. For the INDIA type pumps, the average savings after 18 months of operation is about CFA F60 000. This represents an average annual savings of about CFA F40 000. This level of savings is satisfactory but still not enough to fully cover the equipment maintenance expenses. Ideally for a Committee, there should be an average annual savings of CFA F60 000 to ensure a proper preventive and curative maintenance. The data analysis shows that: * 30% of the committees have a very good level of savings (> CFA F55 000/year). If this level is maintained it will provide enough provisions to ensure the preventive and curative maintenance. The system is financially viable if the level of savings is maintained. * 40% of the committees have a low savings level (< CFA F25 000/year). For these committees, the system viability is strongly threatened if the level of savings does not evolve. If failures begin to be repetitive, there are chances that operation stops by default of equipment maintenance. There should be an awareness increase of the relevant committees. * 30% of the committees have an intermediate savings level with a precarious balance. For those committees, the preventive maintenance expenses may be affordable during the first three years but if failures start to occur then the system viability will be threatened if the level of savings does not evolve. Awareness is to be strengthened on the importance of the savings to be made. For the Erobon type pumps, the average savings after 22 months of operation is about CFA F78 000. This represents an average annual savings of about CFA F43 000. This savings level is very satisfactory and makes it possible to cover the maintenance expenses of the pump. The issue with the Erobon pump is the frequency of the rope breaking. To avoid resorting to the repairer artisan every time, the attendant at the pump must be trained to minimize maintenance expenses. The savings made will make it possible to thus cope with the curative maintenance and, eventually, it will be possible for the Committee to renew the pump because the investment cost is affordable (approximately CFA F100 000). The in-depth analysis of the savings per committee shows that: * 67% of committees (2/3) have an excellent savings level that allows them to ensure the viability of their system for the moment. These models have to be urged. * 33% of the committees (1/3) have a very low savings level and awareness campaigns should be carried out. If corrective measures are not taken, the system may stop in the short term. Highly threatened financial viability. Awareness is to be increased. Financial viability of equipment in the region of Sédhiou Pumping equipment type #of installed equipment # of committees having prepared a financial situation Cumulative amount of funds in CFA F as of September 2012 Average funds per committee Average number of months from the operation start to September 2012 Average amount per month and per committee in CFA F India 14 14 390,000 27,857 12 2,290 Erobon 17 15 178,250 11,883 14 878 For the INDIA type pumps, the average savings after 12 months of operation is about CFA F28 000. This level of savings is quite low to cover the equipment maintenance expenses. Ideally for a Committee, there should be an average annual savings of CFA F60 000 to ensure a proper preventive and curative maintenance. The data analysis shows that: * 30% of the committees have a very good level of savings (> CFA F55 000/year). If this level is maintained it will provide enough provisions to ensure the preventive and curative maintenance. The system is financially viable if the level of savings is maintained. * 70% of the committees have a low savings level (< CFA F25 000/year). For these committees, the system viability is strongly threatened if the level of savings does not evolve. If failures begin to be repetitive, there are chances that operation stops by default of equipment maintenance. There should be an awareness increase of the relevant committees. For the Erobon type pumps, the average savings after 14 months of operation is about CFA F12 000. This represents an average annual savings of about CFA F10 000. This savings level is low and does not cover the maintenance expenses of the pump. The in-depth analysis of the savings per committee shows that: * 20% of committees (1/5) have a satisfactory savings level that allows them to ensure the viability of their system for the moment. These models have to be urged. * 80% of the committees (4/5) have a very low savings level and awareness campaigns should be carried out. If corrective measures are not taken, the system may stop in the short term. Highly threatened financial viability. Awareness is to be increased. Page | 32 Financial viability of equipment in the region of Kolda Pumping equipment type #of installed equipment # of committees having prepared a financial situation Cumulative amount of funds in CFA F as of September 2012 Average funds per committee Average number of months from the operation start to September 2012 Average amount per month and per committee in CFA F India 19 9 83,500 9,278 13 735 Erobon 15 12 82,000 6,833 15 441 For the INDIA type pumps, the average savings after 13 months of operation is about CFA F9 000. This level of savings is very low to cover the equipment maintenance expenses. Ideally for a Committee, there should be an average annual savings of CFA F60 000 to ensure a proper preventive and curative maintenance. The data analysis shows that no committee is viable yet. There should be an awareness increase in the various sites, otherwise, in the short term all systems will stop. The viability is strongly threatened. For the Erobon type pumps, the average savings after 15 months of operation is about CFA F15 000. This represents an average annual savings of about CFA F5 000. This savings level is very low and does not cover the maintenance expenses of the pump. The in-depth analysis of the savings per committee shows that no system is viable financially. In the short term the pumping equipment is likely to stop. The viability is strongly threatened. Summary table on the financial viability of the installed systems at mid-course Region Total # of systems viable system Intermediate viability Strongly threatened viability Solar pump Ziguinchor 1 0% 0% 100% Sédhiou 0 N/A N/A N/A Kolda 0 N/A N/A N/A ALL 1 0% 0% 100% Hand pump India Mark 2 Ziguinchor 52 30% 30% 40% Sédhiou 14 30% 0% 70% Kolda 19 0% 0% 100% ALL 85 23% 18% 58% Hand pump Erobon Ziguinchor 6 67% 0% 33% Sédhiou 17 20% 0% 80% Kolda 15 0% 0% 100% ALL 38 20% 0% 80% 5.4.2.2 Financial viability of the mini-water supply and sanitation systems The thermic option (generator and plunged electric pump) seems to be selected for the mini-WSS pumping device. These facilities will serve on average 1,200 people per system.  Population size calculation method: The project staff provided the evaluators a Word document entitled "list of sites for large and medium-sized infrastructure. The total population of the 8 sites is 9404 people, that is an average of 1176 people per site. This average was round up to 1200 people.  The data sources of the energy sources: the mentioned thermic option originates from the March 2012 quarterly report (page 20) and information provided by the project staff responsible for the component during the interview on 12/03/2012. The viability and financial profitability analysis shows that the thermic option is not viable and it would be more appropriate to move towards solar to make the system viable and profitable in the long run. Indeed, according to the principles of the management reform of the rural boreholes, the users have to take charge of the depreciation provisions of pumping equipment, standpipes and secondary networks. In addition to these provisions to mobilize, the users must also take charge of the operating and maintenance expenses (fuel, spare parts, driver motivation, indemnities for the sales attendants, and operating costs of the managing bodies). The tables below show the analysis per equipment type: thermic or solar. Page | 33 Generator and plunged electric pump Variables Hypotheses Number of people to be served 1,200 Analysis horizon 5 years Specific consumption unit It is considered a value of 15 l/day/person during the first year. This value is different from the value used in sizing (25 or 35 l/day/person according to the projects). This value was recommended following the monitoring during several years of over 100 WUAs in the center areas of Senegal. It allows making realistic projections to see if the infrastructure is able to mobilize the provisions for renewal but also to ensure a proper operation of the system. For the year 2 and out years, the value will be annually inflated by 5%. Investment cost for renewal (i) For pumping equipment the average cost of the supply and installation of the generator and the plunged pump is estimated at CFA F 7 500 000. Life estimated at 6 years. (ii) For the secondary network to be renewed (PVC 90 and 63), the investment cost is estimated at CFA F 3 500 000. Life is estimated at 20 years. (iii) For the standpipes, it is considered 4 standpipes per network to be installed, that is an estimated investment cost of CFA F1 200 000. Life is estimated at 20 years. Operation and maintenance expenses (i) For the fuel, it is considered a rate of 0.2 l per consumed m3. The current gas oil cost (CFA F792) is rounded up to CFA F800 in year 1. This price will be annually inflated by 5% in year 2 and out years. (ii) For the maintenance of pumping equipment, it is considered for year 1, a rate of 2.5% of the investment cost. This cost will be annually inflated by 5% in year 2 and out years. (iii) For the network maintenance, it is considered a rate of 1.5% of the investment in year 1. This cost will be annually inflated by 5% in year 2 and out years. (iv) For the motivation of the driver, it is considered an amount of CFA F50 000/month. (v) For the water sales attendants, it is considered a rate of CFA F50 per consumed m3. (vi) For the operating expenses of the WUAs, it is considered a rate of CFA F30 per consumed m3. These operating expenses include the costs incurred by the managing body meetings, the photocopies, the travel fees of the managing body members for activities related to WUA, and the communication fees. Financial viability analysis of mini water supply and sanitation systems: results Year 1 Year 2 Year 3 Year 4 Year 5 Annual consumption in m3 : A 6 570 6 899 7 243 7 606 7 986 Provisions for renewal in CFA F: B Pumping equipment Secondary network Standpipes 1 485 000 1 485 000 1 485 000 1 485 000 1 485 000 1 250 000 1 250 000 1 250 000 1 250 000 1 250 000 175 000 175 000 175 000 175 000 175 000 60 000 60 000 60 000 60 000 60 000 Operation expenses in CFA F: C Gas oil Network maintenance Pumping equipment maintenance Driver Water sales attendant Operating expenses of WUA 2 416 800 2 562 828 2 2 721 814 2 894 986 3 083 693 1 051 200 1 158 948 1 277 740 1 408 709 1 553 101 52 500 55 125 57 881 60 775 63 814 187 500 196 875 206 719 217 055 227 907 600 000 600 000 600 000 600 000 600 000 328 500 344 925 362 171 380 280 399 294 197 100 206 955 217 303 228 168 239 576 Water cost (CFA F/m3 : (B+C)/A 594 587 581 576 572 A mini-WSS equipped with thermic system supplies water at a cost varying from CFA F594/m3 to CFA F572/m3 . So to stay viable, the WUAs would be obliged to sell the basin of 25 liters at CFA F15 or CFA F600/m3 . This amount is very high and beyond the amounts charged in rural areas. The prices in rural areas vary between CFA F200/m3 and CFA F400/m3 . If the goal is to stay in the range of prices charged in rural areas, then the WUA will not be able to mobilize the provisions for renewal. In this case, all the revenues from the sale of the water will be used for operating expenses. In the event of failure, the system will stop because resources were not provisioned for its replacement. Under these conditions, the system is not viable. Page | 34 Solar panels and plunged pump Variables Hypotheses Number of people to be served 1,200 Analysis horizon 5 years Specific consumption unit It is considered a value of 15 l/day/person during the first year. This value is different from the value used in sizing (25 or 35 l/day/person according to the projects). This value was recommended following the monitoring during several years of over 100 WUAs in the center areas of Senegal. It makes it possible to s to make realistic projections to see if the infrastructure is able to mobilize the provisions for renewal but also to ensure a proper operation of the system. For the year 2 and out years, the value will be annually inflated by 5%. Investment cost for renewal (i) For the pumping equipment the average cost of the supply and installation of solar panels is estimated at CFA F6 000 000. Depending on the size of populations to be served, a 25 m3/day pump would be largely enough to cover the needs 6 years from now. Considering the unfavorable case with an HMT to 40 m, one would need a generator of about 1, 500 to 1,700 Watt-peak. Life of the panels estimated at 15 years. For the plunged pump, the investment cost is estimated at CFA F1 500 000. Life is estimated at 6 years. (ii) For the secondary network to be renewed (PVC 90 and 63), the investment cost is estimated at CFA F 3 500 000. Life is estimated at 20 years. (iii) For the standpipes, it is considered 4 standpipes per network to be installed, that is an estimated investment cost of CFA F1 200 000. Life is estimated at 20 years. Operation and maintenance expenses (i) For the maintenance of pumping equipment and solar panels, it is considered for year 1, a rate of 2.5% of the investment cost. This cost will be annually inflated by 5% in year 2 and out years. (ii) For the network maintenance, it is considered a rate of 1.5% of the investment in year 1. This cost will be annually inflated by 5% in year 2 and out years. (iii) For the motivation of the driver, it is considered an amount of CFA F50 000/month. (iv) For the water sales attendants, it is considered a rate of CFA F50 per consumed m3. (v) For the operating expenses of the WUAs, it is considered a rate of CFA F30 per consumed m3. These operating expenses include the costs incurred by the managing body meetings, the photocopies, the travel fees of the managing body members for activities related to WUA, and the communication fees. Financial viability analysis of mini water supply and sanitation systems: results Year 1 Year 2 Year 3 Year 4 Year 5 Annual consumption in m3 : A 6 570 6 899 7 243 7 606 7 986 Provisions for renewal in CFA F: B Solar panels Pumping equipment Secondary network Standpipes 885 000 885 000 885 000 885 000 885 000 400 000 400 000 400 000 400 000 400 000 250 000 250 000 250 000 250 000 250 000 175 000 175 000 175 000 175 000 175 000 60 000 60 000 60 000 60 000 60 000 Operation expenses in CFA F: C Network maintenance Pumping equipment maintenance Driver Water sales attendant Operating expenses of WUA 1 365 600 1 403 880 1 444 074 1 486 278 1 530 592 52 500 55 125 57 881 60 775 63 814 187 500 196 875 206 719 217 055 227 907 600 000 600 000 600 000 600 000 600 000 328 500 344 925 362 171 380 280 399 294 197 100 206 955 217 303 228 168 239 576 Water cost (CFA F/m3 : (B+C)/A 343 332 322 312 302 A mini-WSS equipped with solar system supplies water at a cost varying from CFA F343/m3 to CFA F302/m3 . With such costs, the WUA can sell the basin of 25 liters at CFA F10 or CFA F400/m3 . This price is consistent with prices set in rural areas. In these conditions, the system is viable and financially profitable. The revenues from the sale of water make it possible to cover the provisions for renewal and the operating expenses without problems. There would be some surplus which could in the long-run allow lowering the water price to CFA F5 the basin of 25 liters or CFA F200/m3 . For the mini-WSS, as they are not yet carried out, it is more appropriate to opt for the solar system instead of installing thermic equipment. 5.4.2.3 Financial viability of the multi-village water supply systems (AEMV) At mid-course, the project has not yet carried out any AEMV. These systems must be managed by the WUAs and it would be worth assessing their viability for the sustainability of the water supply service. For all purposes, the project should assess if the set prices will make it possible to cover the operating and maintenance expenses, and the provisions for renewal of the elements for which the users are responsible. The lessons learned from the AEMV built by the Luxembourg and Belgian Cooperations include, but are not limited to, the following:  Below 35 m3 per day for a borehole site (which corresponds to the consumption of a population of 1,750 people, on the basis of a consumption of 20 l/day per capita), the water supply cannot be sustainable due to the users solely, even if the water is sold at Page | 35 CFA F10 for a basin of 25 liters. The renewal of equipment will depend on support from the State.  From 35 to 70 m3 per day, or the consumption of a population of 2 000 to 3 000 people, the borehole site is viable. For sustainability of the supply, water is sold at CFA F10 for a basin of 25 liters (CFA F400/m3 ) during a few years before being sold at CFA F5 for a basin of 25 liters (CFA F200/m3 ).  From 70 to 100 m3 per day, or the consumption of a population of more than 3, 000 to 5,000 inhabitants, the borehole site is viable if water is sold at CFA F5 for a basin of 25 liters (CFA F200/m3).  Over 100 m3 per day, or the consumption of a population of over 5,000 people, the site is not only viable but also economically profitable. 5.4.3 Organizational viability 5.4.3.1 Functionality and operation of the committees The diagnosis results of WUAs Show that 88% of the managing bodies of the committees have the various positions filled. Generally the people appointed to the positions of the managing bodies are available, in particular the WUA presidents who are 100% active. 96% of the managing bodies have someone responsible for the pump maintenance, this evidences that the mini-borehole users recognize the importance of operation monitoring. On the contrary, only 57% of the managing bodies have the vice-president positions filled. To say that even though the women are members of the committees, they do not fully hold this position, which is de facto reserved for them. However, there is an average of 5 women in the committee managing bodies. That gives an average rate of 56% female presence, (rate exceeding the 45% provided by the project). This can be explained by the strong involvement of women in social mobilizations and other activities relating to hygiene and sanitation. 84% of the 122 management committees established by the project in the regions of Ziguinchor, Sédhiou, and Kolda regularly hold the 3 quarterly meetings. The survey data show that:  51% of WUAs which held their last meeting in November and December 2012 abide by the principle of 3 quarterly meetings.  38% of the WUAs held their last meetings between June and October 2012.  4% of the WUAs held their last meetings in March 2012.  7% of the WUAs did not hold any meeting in 2012. Holding the 3 meetings by the managing bodies of the WUAs on a regular basis depends on several factors including the discussion of the management results in relation to water consumption charges. The members of the managing bodies must meet once per month but actually there are various practices:  54% of the managing bodies of the WUAs abide by this principle. Page | 36  39% of the managing bodies of the WUAs report meeting twice a month.  4% of the managing bodies of the WUAs report meeting once a week or once every two months. Meeting frequency differences are justified by the management of emergencies or needs related to access to water and sanitation infrastructure. Generally the meeting purposes include the latrine construction-related mobilizations, the encounter objects are mobilizing around the construction of latrines, the recovery of contributions for the pump maintenance. The first managing bodies were established in 2010. Only 2 of the 28 managing bodies visited have renewed their memberships. In most cases, the individuals hold the positions they initially filled. This is understandable as the sites are small localities (less than 500 inhabitants) where there are scarce resource-people qualified for managing the manual water point. 5.4.3.2 Accountability At least 60% of WUAs diagnosed report having held at least a general assembly to inform of the activities and management results. 25% of them specify that they have held more than 4 general assemblies the setting up of the WUAs. In other words the majority of the communities collectively make the decisions relating to the operation of the water point. This principle of community management reflects a positive impact of the training of WUA members and the information and awareness messages delivered by the extension staff of the partner NGO. The 30% of the WUAs which do not account for the results and management activities align with the 33% of WUAs which supply water free of charge and accordingly do not manage the mini-boreholes as agreed under the project framework. 91% of household heads report that their households know of the WUA management results. Actually, the social feature of the water pricing through the flat-rate system, the inclusive management of the water point, the small size of the communities make it easy the flow of management information. 94% of the surveyed household heads having access to the project-funded water point report that they know of the existence of the WUAs. 85% of the surveyed household heads know of the appointment method of the delegates. Such an appointment is made by consensus mostly (68%), election (19%), or co-optation (13%). The co-optation principle is usually applied in small localities. 5.4.3.3 Knowledge and use of the management tools 93% of the managing bodies of the WUAs issue the meeting minutes as well as the general assembly proceedings. This proportion is far above the 54% (meeting minutes) and 49% (general assembly proceedings) reported by USAID/PEPAM as of March 2012. Anyhow, the notebooks designed and provided by the project are very useful by enabling Page | 37 rapid completion and recording in writing of the decisions made during the meetings of the managing bodies. Only 25% of the activity reports or completed management tools are regularly shared with the partner actors. There is a limited follow-up of the WUAs regarding their submissions of mini-borehole operation and management data. The MeRO guide does not include management tools such as the proceedings of the general assemblies, the monthly management reports, and the annual management reports. The issuance of these tools makes it possible to better ensure the organizational dynamics of the established and trained WUAs. The absence of indicators tracking the regular issuance and submission of reports or management outcomes does not allow the evaluators to express any opinion on the organizational viability of the WUAs. Such report-related indicators would definitely complement indicators 1.3.A (Number of WUA members in partner communities trained with program support in financial, technical, and/or administrative management) and 1.3.B (Number of WUAs in partner communities that have adopted financial practices, such as a fee collection system, advocated by USAID/PEPAM, the GOS, and/or key partners). 5.5 Effects At mid-course, the project carried out potable water and sanitation infrastructure. The effect analysis makes it possible to s to assess the activity outcomes on the beneficiary populations. The household survey results served as the basis of the analysis:  Use level of the carried out infrastructure.  Improvement of the living conditions and the wealth creation.  Improvement of the living environment and the effect on health and education. 5.5.1 Use level of the carried out infrastructure 5.5.1.1 Potable water component Before the project intervention, 90% of the households surveyed reported the exclusive use of traditional wells as water source. 96% of the households surveyed reported the use traditional wells, private connection, surface waters, and human motor pump (PMH), equally. Only 3% of the households surveyed reported that they used exclusively potable water sources. After the project intervention, the proportion of households reporting the exclusive use of the traditional wells decreased from 90% to 37%. Following the construction of water points, 34% of the households reported the exclusive use of the mini-boreholes or rehabilitated wells. 27% of households use the project-built water point in combination with other existing sources such as surface waters or the traditional wells. 2% of households reported the use of various potable water sources (PMH or private connection network), which were not built by the project. Page | 38 The USAID PEPAM-built water point is on average 166 meters far from the compound. As a result, only 50% of households surveyed reported that their water needs are fully met. It should be remembered that the scattered habitat (inside a same village) is a significant factor in the location of a water point. The use of the project-built water source is real and the water quality is assessed positively by the households which consider it to be very good (93%) to good (6%). A very low proportion (1%) of households rated poor the water quality and the reasons put forward are usually related to the bad location of the infrastructure or to the mishandling or conservation spoiling the water smell, taste or color. The almost free of charge water supply is usually the reason for using the project-built water point. 36% of the surveyed household heads reported that the WUAs stopped asking for contributions, and the households disbursed on average only CFA F260 to pay their monthly water consumption. Such a payment system satisfies 87% of the households surveyed which thought that the water price was affordable in the light of the low coverage level of the infrastructure, and the supply constraints such as distance, waiting time, and water chore. Another perspective shows that 79% of the households are willing to contribute to an average amount of CFA F16, 677 in order to have a standpipe (69%), a private connection (22%) or another less compelling water source (9%). The households consider the project-built water points to be useful and relevant. Actually, they aspire to an improvement of the service quality. The increase in the water price depends on this improvement with the equipment modernization. 5.5.1.2 Sanitation component The proportion of the households surveyed reporting access to sanitary facilities increased from 3% to 80%, a 77 percentage point increase due to the USAID/PEPAM interventions. When the sanitary facilities constructed through the CLTS promotion are considered, then, the access rate increased to 98%. However, the CLTS latrines do not count in the access rate calculation. USAID/PEPAM made available facilities, including VIP (3%), DLV (34%), Sanplat (45%), and traditional latrines (18%). The households understand well the usefulness of the latrine superstructure. However, due to limited means they cannot opt for VIP type latrines. Approximately 96% of latrines have a fence (66% straw; 10% of bamboo panels; 9% of bricks; and 11% in rammed earth, sheet metal, or linen). The proportion of latrines that do not have any fence is low (approximately 4%). More and more households pay a special attention to the fence so that they can use the facilities day and night and therefore avoid resorting to the open air defecation during the day. USAID/PEPAM subsidized 82% of the households surveyed so that they can gain access to sanitary facilities. 14% of the households surveyed constructed latrines through the CLTS promotion. 4% of the households surveyed constructed latrines by their own funds. For 46% of the households, the "lower cost" is the reason for opting for a latrine type. Page | 39 For 39% of the beneficiaries, the "Good quality of the latrines" is the reason for opting for a latrine type. The rest of the households report other latrine type selection criteria such as the exemption from payment, the advice given by the partner, and the ease of maintenance. Actually, the subsidy greatly facilitated the acceptance of the sanitation program. On average 48% of the households contributed financially to construct their latrines for an average amount of CFA F9, 938, all types of latrines put together. 86% of the beneficiaries considered this contribution amount to be reasonable up to not high. The other beneficiaries reported that they suffered from financial harm during the construction of their latrines because of the irregularity of their revenues primarily earned from the sale of agricultural products. The degree of use of latrine is very satisfactory. 98% of the households reported the use of the latrines built by the project. This high use rate considerably affected the health of the populations while improving their living conditions. 5.5.2 Improvement of living conditions and wealth creation 5.5.2.1 Living conditions evolution The surveyed people considered the effects of the water and sanitation infrastructure to be positive. There was a real need due to the lack of water infrastructure. Even though the households reported that the water needs were covered at only 50% through the USAID/PEPAM-built infrastructure, it was clear that the project interventions allowed 88% of the households surveyed to gain time and use this time to implement their economic activities predominantly in the agricultural sector. Moreover, the girls who were responsible for the water chore prior to the construction of the mini￾boreholes were able to benefit from more time for educational activities. The water hand pumps helped to reduce the water chore. However, the manual drainage is always painful with Erobon pumps and the water drawing is not suitable for certain categories of women who are forced to appeal to the boys for pumping. Thus, some argue that "drawing water from the wells is less difficult that turning an Erobon" pump. Regarding the India pumps, draining water has become less arduous than it used to be. The mini-boreholes located at the school level yielded direct and positive effects as they are a factor favoring the retention of children at school. The availability of potable water attracts students and encourages parents to release their children easily. The education actors surveyed reported that the water points affected a lot the girl retention at school (52% of the average number of students in the recipient schools of Kolda). The project made it possible for girls to use water and sanitation services without being away at length during the recess. As a result, their punctuality and attendance improved, and subsequently their school drop-out rate decreased. Despite the absence of conclusive figures, the environment affect the psychology of children and the availability of potable water and latrines affects the improvement of school performance, according to the Page | 40 Inspector of Kolda Academy and the Inspector responsible for project monitoring in the Ziguinchor education inspection. In general, 63% of the households surveyed reported that their living conditions improved due to the availability of potable water. As evidence, they mentioned the reduction of water borne diseases, the time gained by the women, awareness of hygiene and cleanliness actions, etc. The remaining households expected that the service would improve as all water needs are covered. Concerning sanitation component, the effects on the improvement of living conditions are real as awareness-raising and information campaigns focused on the environment protection in the localities and the effects are visible through the cleanliness of the intervention villages. Additionally, more households gained access to sanitary facilities. The beneficiaries reported generally that a latrine was a true need and “a compound without latrine is a house without a roof or even a house still under construction”. 5.5.2.2 Wealth creation The local builders who carried out the infrastructure evidenced the wealth creation. For instance, the remuneration of the molders and builders is estimated at CFA F42 407 280 from the start to March 2012. (See table below) Remuneration of the molders Latrine type # of bricks # of latrines Total # of bricks Unit price of bricks Estimated amount in CFA F Sanplat 81 1 420 115 020 40 4 600 800 DLV 162 810 131 220 40 5 248 800 VIP 369 218 80 442 40 3 217 680 Total 2 448 326 682 13 067 280 Remuneration of the builder Latrine type # of latrines Latrine unit cost Estimated amount in CFA F Sanplat 1 420 7500 10 650 000 DLV 810 15000 12 150 000 VIP 218 30000 6 540 000 Total 2 448 29 340 000 42 407 280 The injected amount is equivalent to the remuneration of 30 people paid at the local minimum wage for 30 months (2.5 years). The unavailability of data on the contracts made it difficult to measure the level of wealth creation in terms of the increase in the turnover of enterprises and private suppliers recruited under the project. 5.5.3 Improvement of the living environment and the effect on health and education 5.5.3.1 Hygiene and cleanliness of the living environment The social marketing actions for promoting the latrines and the awareness-raising messages on hygiene and sanitation contributed a lot to decrease the open air defecation. Page | 41 The project-built latrines led to the end of open air defecation in almost all beneficiary households. The diseases, related to fecal peril and identified by 97% of the households which benefitted from latrines, decreased by 93% according to the households surveyed. The existence of latrines meeting the standards, the adoption of hygiene and sanitation-related good practices (confirmed by 98% of the households), the participation in the project￾funded social engineering activities (noted by 97% of the households) contributed to the decline of the fecal peril. According to the respondent household heads, latrine maintenance is performed by 98% of the beneficiary households. The survey data on the maintenance frequency show that 80% of the maintenance is performed on a daily basis, 16% of the maintenance performed twice a week, 2% of the maintenance performed once a week, 1% of the maintenance performed once a month, and 1% of the maintenance done after each use. The households that do not express any opinion on the maintenance, because of very irregular maintenance, represent 2% of the beneficiaries. The monthly average expenditure for latrine maintenance is estimated at CFA F1, 353 per household. The partner NGO implemented outreach activities that yielded positive effects. The home visits paid by the WUAs to encourage the proper maintenance of the latrines can corroborate the survey data which inform of the existence of fines when households in the CLTS sites fail to maintain their latrines. The partner NGO initiated "Set Setal" actions which introduced a competition among the localities. The competition is about the cleanliness of the locality environment and the elimination of bush fires which were recurrent in the areas. These actions of regular cleaning of the village have the power to improve the living environment while strengthening the social cohesion. The WUA has played an instrumental role in the monitoring of hygiene and sanitation actions. The aforementioned social organization has the advantage of being able to leave a few attainments to the communities. The satisfaction of the populations with the carried out infrastructure and the infatuation of the communities for these committees are positive assessment criteria for the improvement of the living environment. 5.5.3.2 Hand washing Hand-washing is regularly practiced by 95% of the beneficiary household members, based on the collected survey data. The same households report that in 86% of cases, hand washing is practiced using water and soap. In 6% of the households, the members use water, soap, and ash. In 3% of cases, the household members use only water. The rest of the households have resorted to several practices, depending on the ability to pay soap, bleach, or ash. Page | 42 The hand washing practice had an effect on the reduction of diseases, as recorded by the health services. The practice regularity evidences the positive effects of the messages delivered during outreach and awareness-raising activities relating to the hygiene at the exit of the latrines. The supervision conducted by the WUAs during the home visits is also a factor in promoting adherence to the practice. The survey data corroborate the supervision role. It should be noted nevertheless that the maintenance mechanisms of the "Tipy Taps" are not understood and that the renewal of the cans is a problem. Eventually, the hand washing practice during critical moments may be irregular or even marginal. 5.5.3.3 Decrease in health-related expenses With regard to the effects on household water supply, it should be noted that the availability of safe potable water had positive effects on the health of the households surveyed. The improvement of the health status is clearly visible and even confirmed by the health services notwithstanding the absence of conclusive figures. The households surveyed reported that the water-related health expenses decreased by 74% to 77%. According to the respondent households that benefited directly from investments in small hydraulics, the monthly average amounts decreased from CFA F8, 689 to CFA F2, 287 thanks to the project intervention. The maximum monthly health spending was CFA F65, 000 before project intervention and since the intervention it is CFA F15, 000. This proves that water from traditional wells was regularly contaminated and unfit for human consumption. The availability of potable water has reduced by 57% the cases of water-borne diseases in the targeted regions. The head nurse of the Badion health post (Kolda) confirms that a notorious reduction of diarrheal and malarial diseases is reported in the villages where the project has intervened. 77% of households using the water of the mini-boreholes treat the water and this contributes to the reduction of the diseases. The households treating the water before consumption report that they use methods, including chlorination (51%), filtration (30%) and/or the water purification through the use of the Aquatab product (19%). The ruptures in supply of the Aquatab product make the households resort to chlorine and traditional filtration of the water before consumption. 5.5.3.4 Girls’ school attendance 88% of the households using the project-built water point reported the attendance of girls at the school thanks to the project intervention. However, this means that the girls are still charged to do the household water supply. This rate mitigates the 50% overall satisfaction level relating to the full coverage of the household water needs. The traditional well use for other water needs remains a factor leading to the girls’ school absences or lateness. Subsequently, this increases the risk of school drop-out. Page | 43 5.6 Project governance The project governance analysis refers to the organizational and implementation methods of the activities. Particularly, it is about assessing at mid-course the project strategic oversight, its operational management, and its activity implementation method. 5.6.1 The project strategic oversight The project strategic oversight should be ensured by the coordination and monitoring committee established by the ministerial decree. The committee is made up of 11 members: two representatives from the rural hydraulics and sanitation Minister Cabinet; Directors of the national technical offices (DH-DEM-DGPRE-DA); two representatives from the finance Ministry; the PEPAM Coordinator; a RTI representative; and a USAID representative. The General Secretary of the Ministry in charge of rural hydraulics is designated as the committee chairperson. The PEPAM Coordinator is entrusted with the committee secretariat. The ministerial decree assigned to the committee the following tasks.  Monitor the project implementation;  Track the smooth running of the physical operations (service objective accomplishment) and financial execution on the basis of project progress reports.  Take steps to help resolve problems or conflicts that might arise during the course of the project.  Provide information to the line ministries and donors.  Approve the project annual work plans and their relating budgets. According to the ministerial decree, the committee should meet at least four times a year. At mid-course, it has met only once on July 7, 2011. At mid-course, it is clear that the strategic oversight does not abide by the ministerial decree. The committee has not fulfilled its mandate. The committee has not approved any work plans, progress reports, and budget amendments. Nevertheless, the project keeps on implementing the activities. This would mean that the decisions and recommendations made by the oversight committee do not affect the activities and the results to be achieved. This is a weakness because the committee has the authority to orient, validate, and decide on readjustments needed to accord with the initial objectives and the sector guidance. A functioning committee would help avoid some result shortcomings (effectiveness) and realign some budget line items (efficiency). 5.6.2 The project operational management It refers to the project staff as well as the working relationship with the NGOs and the public sub-national technical offices. Page | 44 5.6.2.1 Project staff The USAID/PEPAM project staff provides the operational management. The project has its headquarters in Dakar and regional offices in Ziguinchor, Kolda and Tambacounda. According to the organizational chart, the staff totals about 40 people, the NGO staff is not counted. For a project with a logic of "make do", this manpower is too high compared to the one of other projects in the sector and with an equivalent funding level. When the manpower is compared to the mid-course outcomes and the organizational chart segmentation, it appears that real efforts should be made regarding the intervention consistency and the task distribution for better project effectiveness. 5.6.2.2 Relationships with the partner NGOs USAID/PEPAM partnered with 11 local NGOs to collect applications, strengthen capacity of the beneficiaries, accompany the beneficiaries, promote CLTS, and supervise the works. The agreements are cost-shared contracts. Some partner NGOs reported satisfactory working relationships. However, others pointed out that, at the outset, the project failed to have a designated person to coordinate the NGO interventions. Then, the project took the following corrective actions:  The project component 1 worked with the program managers and the NGO coordinators.  The project component 2 worked with the NGO extension agents.  The project component 3 worked with the NGO technicians in charge of hydraulic activities.  The project component 4 worked with the NGO technicians in charge of sanitation activities.  The project component 5 worked with the 2 NGOs recruited exclusively for CLTS. 5.6.2.3 Relationships with the sub-national technical offices Hydraulics and Sanitation The project worked closely with the sub-national offices of hydraulics and sanitation. The partnership was very difficult at the start of the project, but following the signing of a memorandum of understanding (MOU) on April 19, 2011 USAID/PEPAM and the DA, DEM and DH, defined a formal framework of mobilization of the regional technical offices. The review of the MOU, supplemented by interviews with the various offices, shows that its implementation was sometimes very problematic. Among others, this would explain the difficulties of mobilization of the technical services mentioned in some cases. It should be noted that beyond funding the travel and field trip-related expenses, USAID/PEPAM strongly supported the regional offices by strengthening capacity and providing computer equipment (photocopier, printers, computers, etc.). There is a need to review certain parts of the MOU. Page | 45 Hygiene and environment USAID/PEPAM has working relations with the sub-national offices in charge of the environment and hygiene. However, there is no MOU signed to formalize those relations. The regional environment office staff deplored the absence of a MOU. Even though a partnership framework was not defined, USAID/PEPAM provided the regional hygiene offices with water quality measuring equipment that makes it possible to test whether the water fits or not to drink. This very positive support reinforces the hygiene office in fulfilling its potable water quality control-related mandate. Education Inspections USAID PEPAM worked well with the regional education inspections by working directly with the offices at the sub-levels (IDEN) through teacher training workshops (see capacity). Additionally, USAID/PEPAM constructed water points in some schools. It was regretted the lack of post-training follow-up to assess the potential effects and the response level of the students regarding the water, hygiene and health themes. In terms of partnership framework, a performance-based contract was signed with each education inspection (Ziguinchor, Sédhiou and Kolda). Regional development agencies USAID/PEPAM did not sign any partnership agreement with the ARD. The ARD interviews show a low involvement in the planning, the investment implementation, and the post-implementation follow-up. The ARD is the "technical arm" of the local governments. It supports them in the investment planning and implementation. In such a capacity, it must be involved in, and actively associated with any intervention for the sake of harmonization and synergy to develop but also for sustaining the investment. According to USAID/PEPAM, an agreement is under consideration for an active involvement of the various ARD. 5.6.3 Activity implementation method The project constructed by day labor its activities; or it commissions consultants for providing intellectual services, or local enterprises and workers for works. In sum, the activities were implemented as listed below. Hydraulic works - Mini-borehole construction by day labor: The project approach is to train the local enterprises in manual technologies and make those enterprises construct the infrastructure. However, when it comes to sites where the manual drilling was not Page | 46 possible due to the unfavorable hydrogeological conditions (deep water table, rocky terrain), USAID/PEPAM constructs by day labor. In order to respond to the populations’ requests, the project pilot-tested a mini driller enabling low cost drilling in areas where the manual drilling would not be possible. This is why some constructions by day labor were done. - The project trained private providers so that they can construct the other mini-boreholes and rehabilitate wells. - The project trained artisans to manufacture locally the Erobon pump. - The project imported the India Mark 2 type pumps and made the trained teams install the pumps. Sanitation works  The project trained local builders who, subsequently, built the household-based latrines.  The project promoted CLTS and, as a result, the populations built improved household-based latrines.  A private provider built a public convenience. Intellectual service provisions  Consultants carried out PLHAs for the three regions.  A consultant conducted the baseline study.  A consulting firm conducted the environmental study.  The project staff and other partners jointly designed practical manuals or outreach tools. Capacity strengthening The capacity strengthening-related activities were implemented by the project staff, the partner NGOs, or external consultants (establishment and capacity strengthening of managing bodies, builder training, driller training, repairer artisan training, teacher training, etc.). 5.7 Capacity strengthening The project implemented awareness-raising activities through social marketing, the participatory hygiene and sanitation transformation / self-esteem, associative strengths, resourcefulness, action-planning and responsibility (PHAST/SARAR) method, and the CLTS approach. These approaches allowed the hygiene promotion (hand pump equipment promotion, sanitary facility promotion, IEC for behavior change). These approaches are very suitable for behavior change communication if they are well used. With regard to the strengthening of competences, the project trained several actors so that they could properly take on their responsibilities: partner NGO staff; public technical offices; managing bodies; teachers; builders; drillers; manufacturer artisans; repairer artisans; etc. Page | 47 The table below summarizes the trainings provided by the project at mid-course. The data are drawn from the documents provided by the project. Nº Training theme # of workshops organized # of participants in the workshops 1 CLTS approach 10 153 2 Orientation and harmonization on the MeRO manual 1 20 3 MeRO approach – financial, technical, organizational, and administrative management 11 699 4 Borehole construction workshop 1 14 5 Rope pump manufacturing 3 15 6 Practical training on performing manual drilling techniques 5 83 7 Manual driller training on the installation of the filter points 1 18 8 Maintenance and repair of hand pumps (Erobon) 1 18 9 Call for tenders and procurement procedures 1 15 10 Corporate management, simplified accounting of an economic unit, Marketing, taxation and organizational management 2 32 11 Applied hydrogeology and hygiene to the manual borehole drilling 2 74 12 Introduction to the Social Marketing approach in the field of water and sanitation 2 37 13 Guidance on the funding application preparation and procedures for setting up the building sites of boreholes and latrines 1 17 14 Guidance on the establishment strategy of the management committees/WUAs 1 13 15 Guidance on the implementation guide of the component 1 and the operational procedures of the USAID/PEPAM project 1 23 16 One water purifier product promotion: AQUATABS 3 35 17 Promotion of hygiene and sanitation in community 5 353 18 Recycling of an Erobon pump manufacturing workshop 1 5 19 Construction techniques of DLV, Sanplat, and VIP latrines 4 192 20 Rope pump manufacturing techniques 2 8 21 Use of the Guide Health Nutrition and environment in elementary schools 6 183 64 2007 Annex 2 to the French version of the evaluation report further details the various trainings the project provided. 5.7.1 The partner NGOs  The NGOs participated in 33 training sessions on the following themes:  Guidance on the implementation guide of the component 1 and the operational procedures of the USAID/PEPAM project  Guidance on the establishment strategy of the management committees/WUAs  Orientation and harmonization on the MeRO manual  Guidance on the funding application preparation and procedures for setting up the building sites of boreholes and latrines  MeRO approach  Performing manual drilling techniques  Installation of the filter points  Maintenance and repair of hand pumps (Erobon)  Call for tenders and procurement procedures  Applied hydrogeology and hygiene to the manual borehole drilling  Introduction to the Social Marketing approach in the field of water and sanitation  One water purifier product promotion: Aquatabs  Construction techniques of DLV, Sanplat, and VIP latrines  CLTS approach Participation in these various formations made it possible them to play their relay role in implementing downstream activities, while contributing upstream in the validation of the project guides and manuals. The training on the MeRO approach allowed them to replicate and accompany the WUAs in the management of water and sanitation Page | 48 infrastructure. Some NGOs acquired new competences in water and sanitation thanks to these trainings. Those NGOs made use of their new skills to assist other partners. For instance, the NGO FODDE (Kolda) became a strategic partner of Oxfam America thanks to the USAID/PEPAM project capacity strengthening efforts. 5.7.2 The beneficiary populations The populations through the established bodies (steering committees, managing bodies of the WUAs, etc.) participated in 27 training sessions on the following themes:  MeRO approach – financial, technical, organizational, and administrative management  CLTS approach  Maintenance and repair of the India and Erobon pumps  Introduction to the Social Marketing approach in the field of water and sanitation  Promotion of Hygiene and sanitation in community The consultants, the project staff, and the partner NGOs strengthened the capacity of the local actors, namely the management committees which must play the relay role of the project and ensure the sustainability of the actions taken. USAID/PEPAM provided the management committees with MeRO management tools that would help them monitor the management activities relating to water and sanitation. The use of these tools is not effective despite the follow-up by NGOs during the implementation period. When the end of open air defecation is declared the CLTS committees are changed into hygiene promotion committees to better follow up the infrastructure built in the village (latrines and water points). The number targeted per training session exceeds the standard of 20 to 25 participants. The plethoric attendance turns the training into information session insofar as it lasts 1-2 days. The beneficiary populations did not use the tools designed for the PHAST/SARAR method. They generally used only the three stacks of assorted cards (latrine management and use of Aquatab tablets) and contamination pathways and barriers. The evaluation team did not meet any committee member who could properly explain the activities implemented, his community level role, and the purpose of the tools made available. Additionally, the tools were not always well stored. 5.7.3 The drilling enterprises At mid-course, USAID/PEPAM trained 11 drilling enterprises or “economic interest groups”. The regional breakdown of those entities is the following: 5 in the Ziguinchor region, 3 in the Sédhiou region, and 3 in the Kolda region. They received theoretical and/or practical trainings on the following themes:  Performing manual drilling Page | 49  Corporate management, simplified accounting of an economic unit, Marketing, taxation and organizational management  Applied hydrogeology and hygiene to the manual borehole drilling  Installation of the filter points  Call for tenders and procurement procedures  Maintenance and repair of the Erobon pumps  Construction of deep boreholes The training allowed the drillers to perform manual drilling according to the required quality standards. Overall, the users hailed the water quality of the boreholes. The drillers knew of the auger drilling technology and the project introduced the manual rotary. The newly transferred technique made it possible to build infrastructure in a shorter time period and this was a positive aspect given the emergency in meeting the water needs of populations. The analysis of the training effect on the turnover increase of the drilling enterprises is difficult to do insofar as the accounts are not accessible, and the management results are not understood. The correct application of the new skills in the financial management and accounting and the procedures for tendering and contracting is not visible. 5.7.4 The pump manufacturer artisans USAID/PEPAM trained 5 Erobon type hand pump repairer manufacturer artisans in mainly the following themes:  Rope pump manufacturing techniques  Maintenance and repair of the hand pumps  Call for tenders and procurement procedures  Corporate management, simplified accounting of an economic unit, Marketing, taxation and organizational management  Retraining in rope pump manufacturing techniques The artisans hailed the high standard of the assembly products and the new manufacturing techniques. The training of the pump manufacturers made it possible to revalue the local know-how. On the contrary, the accounting management results are not available. The maintenance contracts are not effective and not executed. The actors minimized the comparative advantages in terms of professional management all the more because the pump manufacturing and installation costs are considered to be nearly affordable and the logistical support is considered to be low. The framework has not evolved despite the support of the project and the turnovers are low to perform marketing operations in the intervention areas. This, and so, hampers the performance of the manufacturer artisans. Some artisans deplore the content of the messages delivered in the media for the equipment promotion. Page | 50 5.7.5 The builders USAID/PEPAM trained 169 builders mainly on:  DLV, Sanplat and VIP latrine construction Techniques  Call for tenders and procurement procedures  Corporate management, simplified accounting of an economic unit, Marketing, taxation and organizational management The six training sessions organized between May 2010 and March 2012 allowed the trainees to build 2,448 project-supported latrines. The public technical offices which did the technical acceptances considered the latrines to be meeting the requirements. The training outcomes in terms of understanding of the procurement procedures and the accounting management rules are difficult to demonstrate in the light of the weak organization of the builders. 5.7.6 The public technical offices 5.7.6.1 Hydraulics sector Representatives from the regional technical offices of hydraulics participated in 10 training sessions on the following themes:  Workshop on construction of deep boreholes  Applied hydrogeology and hygiene to the manual borehole drilling  Orientation and harmonization on the MeRO manual  Workshop on the rationalization of the technological options in hydrology Their participation helped them to learn about the new manual drilling technologies so that they could supervise the drilling enterprises as well as the technicians of the partner NGOs. However, they did not perform the inspection and the follow up of the drilling enterprises on the ground. 5.7.6.2 Hygiene and sanitation sectors Representatives from the regional technical offices of sanitation participated in 10 training sessions on the following themes:  DLV, Sanplat and VIP latrine construction Techniques  CLTS approach The supervision of the builders during the practical workshops contributed to the compliance with the latrine construction standards, nevertheless there was no work inspection on site. Concerning the CLTS approach, the technical offices contributed to the certification of the end of open air defecation villages. The representatives from the technical offices played the role of resource persons rather than training participants. Page | 51 5.7.6.3 Education sector The education inspections and the IDEN of the targeted regions trained the school actors. The training focused on health, nutrition and environment guide of the elementary school. This allowed 183 teachers to gain skills in hygiene and sanitation in schools. However the cascade trainings for reaching the end target groups, namely the students, were not correctly performed despite the kits provided to 96 teachers. The capacity strengthening activities at the school level were not actually followed-up. The inspectors the evaluators met reported that they did not have any information on the follow-up activities and were still waiting for the project guidelines. This demonstrates the communication flaw relating to the training session organization. Consequently, the training effect is low even though the learning framework has improved in some schools that benefitted from mini-boreholes. The Education Ministry validated the middle school guide in February 2012. The final validation took place in May 2012. 5.8 Stakeholder satisfaction The analysis of the stakeholder satisfaction aims at assessing the opinion level of the major partners at mid-course. It is about the feeling of the USAID/PEPAM external actors with regard to the procedures, the activity coordination, the technologies made available, the capacity strengthening component, the institutional support, etc. 5.8.1 The beneficiary populations The satisfaction level ranges from satisfied to very satisfied for 60% of the households regarding the potable water-related interventions of the project. A small proportion (2%) expressed dissatisfaction because the water needs are not fully met or the water quality problems are not solved. 38% of the households did not want to respond to the question because the project did not have any water-related intervention in their localities or the infrastructure were not yet in use due to delays. A few dissatisfaction reasons mentioned by the populations include:  Unmet latrine needs were still awaiting.  The partner NGOs were entrusted with the outreach and awareness-raising reinforcement actions needed  Potable water needs of the localities not fully met The satisfaction level ranges from satisfied to very satisfied for 93% of the beneficiary households regarding the sanitation intervention and its infrastructure quality. 7% of the households were not satisfied because the latrines in the households were traditional types which were not sustainable (CLTS promotion). Page | 52 In addition to the populations, the local governments made reserves about the project. The reserves include the following:  Lack of actual involvement of local governments in the site choice and the activity implementation process  Weaknesses of the Erobon pump technology;  The large localities were not taken into account  The “productive water” component was not taken into account 5.8.2 The partner NGOs, drillers and artisans The satisfaction rate is estimated at 29% for the NGOs and the private providers (drilling enterprises, artisans, and pump manufacturers). The dissatisfaction reasons mentioned by the NGOs are among others:  Lack of consistency and visibility in the project coordination  Lack of consensus on the project management methods  Lack of stabilization of the monitoring forms  Repetitive tardiness of the subsidy disbursement and very long delay between the approval of funding requests and the funds transfer  Administrative burdens: multiplication and requirement of administrative tools  Bad estimate of the distances to be covered by the NGO field staff; Imbalance between the effectiveness and the implementation cost: the closest intervention village was 50 km far and ENDA perceived only CFA F100, 000 for the monthly fuel costs while there were 4 travels on average per month  Delay in the field activity implementation arising from the delay in transferring funds due to administrative burdens (at least 3 month time lag between the funding request, the fixed obligation grant to be signed by the populations, and the actual transfer)  Blockages at the material supplier level: the NGOs can remain a good month waiting  Blockages at the level of local banks or microfinance institutions: these financial institutions may not have enough financial resources. Example: Of CFA F600, 000 to be cashed during a transaction, only half of the amount is available  Implementation cost underestimated compared with the work volume  Lack of pump maintenance in the absence of any village-based expertise the project should have created (Kolda)  Lack of reinforcement of the follow-up of the established management committees The dissatisfaction reasons mentioned by the drilling enterprises are among others:  Dissatisfaction in relation to the borehole construction cost  Delays in paying the services provided  The hydraulics offices impose acceptance procedural burdens relating to the payment of the enterprise and the pump manufacturer The dissatisfaction reasons mentioned by the pump manufacturer and repairer artisans are among others: Page | 53  The drillers pay only after delivery while the local suppliers require cash payment of the pump raw material  Difficulty of access to the plastic material enabling to manufacture the pump piston  Lack of funding to improve the workshops  Lack of logistical support  The enterprise which staff was not trained in repair techniques of the India pump struggles with difficulty in finding customers due to a stiff competition  Lack of free collaboration of the project  Pumping equipment marketing deficiency 5.8.3 The public technical offices These offices include the sub-national offices of hydraulics, sanitation, hygiene, and environment. The ARD are included too. Their satisfaction rate is estimated at 33%. This demonstrates a high dissatisfaction level. The dissatisfaction reasons mentioned by the public technical offices are among others:  Supposedly, the sub-national offices were not involved in the signing of the memorandum of understanding. It is argued that everything was agreed at the national level and the agreed upon items do not always speak to the regional offices. USAID/PEPAM differs in opinion.  The provision level is low for the field trips and other implementation-related travels  Lack of the activity implementation process follow-up and complete lack of coordination  Fixation on lessening the hydraulic investment cost when best solutions could be considered  Imposed character of the technologies  Procedural burden  Lack of memorandum of understanding with the offices of hygiene and environment, and the ARD 5.9 Environmental aspects Working out visual guides In order to ensure that the project does not affect the environment, USAID/PEPAM prepared a mitigation and environmental risk monitoring plan in the form of visual field guides extracted from the USAID reference document entitled «Environmental guidelines for small-scale activities in Africa ». These guides are meant to quickly identify the important environmental concerns in small scale water supply and sanitation activities (http://www.encapafrica.org/egssaa.htm). This helps to ensure that most of the important environmental deficits detected during the activity design and management are quickly and easily identified for corrective actions. The preparation of these documents as well as the involvement of the sub-national offices (hygiene, hydraulic, sanitation), the local governments (rural communities, municipality) Page | 54 and the civil society (NGOs, media...) reflect a strong enough commitment for taking care of the environment adequately in the project activity implementation. On the basis of the developed tools, it may be arguable that the project took into account the environmental aspects. Environmental monitoring The inspection is done at the construction time and it was on each partner NGO to ensure the compliance with the provisions set forth in the environmental management plan. On the basis of check list forms, ENDA, CARITAS and CASADES regularly made this follow-up in the regions of Kolda and Sédhiou. For the other NGOs, the evaluators did not obtain any verification documents. In all cases, for the implemented activities, monitoring the execution quality needs to be documented so as to be able to assess their implementation. Preservation of the water quality The hygiene office squads performed physicochemical and bacteriological analyses of each water infrastructure after its acceptance. The review of the analysis bulletins shows that the results are satisfactory. This good quality must be preserved. However, certain aspects are to be readjusted, watched over, or improved:  There must be a regular follow-up and a good awareness-raising of the beneficiaries on the water supply, transportation, conservation, and consumption.  Well fence off the water points with solid materials to ensure the sustainable protection of the infrastructure. In this regard, the evaluators found when visiting sites that some infrastructure protection fences had fallen or were in poor condition.  Specify more clearly the tasks and responsibilities of the enterprises for compliance with the environmental criteria. The enterprises certainly benefited from trainings, but they should be further sensitized on the need of correct and regular follow up of the works and they must ensure a proper reclamation of the sites. Effects of the sanitation infrastructure on the environment The sanitation-related effects may be negative if latrines follow up and management measures are not taken. Some latrines are in poor condition:  No fence  Presence of flies entering and leaving the defecation hole or ventilation pipe  Presence of bad smells at the latrine level  No ash  Lid non-proportional to the defecation hole creating passages for flies  No devices to create the hand washing reflex at the exit of latrines Noted deficiency The evaluators noted a significant deficiency, namely the not taking into account of the regulation in force in Senegal, particularly the environmental Code. USAID/PEPAM rather complied with the U.S federal environmental regulations (22 CFR 216) relating to Page | 55 the environmental concerns during the activity implementation. As a result, the sub￾national environment offices criticized USAID/PEPAM and deplored their non￾involvement in time in the project implementation. In December 2011, to comply with the Senegalese regulation, the DEEC authorized USAID/PEPAM to conduct an environmental audit of the infrastructure already completed and a strategic environmental assessment of the infrastructure to be built. These two studies should be done one after the other, recorded in a single document for each of the regions, and submitted to the DREEC for validation. These studies were carried out by a consultant, pre-validated by the DREEC, and are being validated by the DEEC in Dakar. While waiting for this protracted validation, the project implementation continues and is almost ending in its first intervention region. 5.10 Added value by the USAID/PEPAM project The USAID/PEPAM project has added value to the potable water and sanitation sector by making available decision-making tools, namely the PLHAs. By promoting CLTS it also contributed to keep going the reflection in the rural sanitation sub-sector. From the practical view point, USAID/PEPAM proposed appropriate technologies and methods to promote disadvantaged rural populations’ access to potable water and sanitation in a sustainable environment. The following sub-sections deal with these various aspects. 5.10.1 The public technical offices The project carried out 52 unplanned PLHAs. The PLHAs are part of the intervention unified framework (CUI) of the PEPAM instruments. By carrying them out USAID/PEPAM abides by the sector guidance. It should be hailed the fact that PLHAs were carried out for the entirety of the Ziguinchor and Kolda regions (PEPAM BAD II carried out PLHAs for the Sédhiou region). The availability of these documents made it possible to have a comprehensive, clear, and distinct situation of the potable water and sanitation access indicators at the regional level. Moreover, these PLHAs could serve as a reference for all partners intervening in these regions. The PLHAs constitute a significant contribution by USAID/PEPAM to CUI. Beyond the inventory and the stock taken of water and sanitation infrastructure, the PLHAs, if they are well performed by going through all necessary validation stages, make it possible to identify the priority projects and the expectations of potential beneficiaries. Thus, they will serve as input to measure the effects of the current project, but also for the design of future interventions relating to water and sanitation in the concerned regions. The table below summarizes the synthesis elements of the carried out 52 PLHAs. Page | 56 Region Carried out PLHA 2011 population of the rural communities 2011 potable water access rate 2011 household sanitary facility access rate Total By potable water supply Ziguinchor 25 360 965 50% 21% 16% Kolda 27 447 049 46% 15% 32% These results show that there is much to do to fill the potable water and sanitation access gap. The potable water access rate is largely below the national average in 2011 (80.1%). The sanitation access rate in Kolda is quite compliant with the national average in 2011 (34.3%). In Ziguinchor the access gap to fill is huge. 5.10.2 The CLTS promotion This activity, developed in the course of the project, was very relevant. The CLTS approach by targeting the low density villages helps strengthen the awareness-raising activities in the hygiene promotion. Like the PHAST/SARAR method, it puts in situation the beneficiary who understands by him/herself the positive aspects of a sound excreta management and commits him/herself to banishing the open air defecation (fields, house corner, behind the compounds...) for all categories (women, children and adult) of his/her locality. It focuses on the person to be sensitized, to make him/her understand the usefulness and the immediate effect of a sound management of hygiene and sanitation issues. Its effectiveness is that it places the target person in a situation of change. The target person makes his/her decisions and totally takes care of him/herself. This approach is even more relevant that it makes it possible to conceive sanitation not as an infrastructure issue but as an improvement factor of the living conditions through a behavior change following an awareness-raising and outreach process led by endogenous actors. So thanks to this approach, populations were able to construct or rehabilitate on their own funds improved traditional latrines that accord with the minimum rules of hygiene and convenience (defecation hole closing; ventilation pipe with anti-mosquito mesh; height of the flagstone compared with the ground level; infrastructure closing; and existence of a washbasin device). As of March 2012, the beneficiaries have built or rehabilitated 500 improved latrines, that is a 34% execution rate compared with the planned 1,472 infrastructure. In the targeted localities, the populations achieved the FDAL and the hygiene and cleanliness conditions of the village and the households markedly improved. In addition to these findings in all beneficiary sites, there should add a decrease in diarrhea cases, a quasi-absence of flies that used to proliferate in the area, a decrease in cases of snakebites with the clarity of the paths following cleaning operations, etc. The head nurse of Marakissa HLM (Sédhiou Region) provided data that corroborated the opinion expressed by the populations on the health condition improvement. Page | 57 Locality Marakissa HLM 2011 population 238 Diseases Average # of cases per month before intervention Average # of recorded cases per month after FDAL Evolution Chronic diarrhea 5 0 -100% Chronic dermatosis 3 1 -67% Diagnosed malaria cases 6 2 -67% Proportion of the population affected by lack of hygiene￾related diseases 6% 1% Moreover, the CLTS approach is even more relevant that it allows the populations to have at their disposal improved infrastructure at relatively modest costs unlike the subsidy approach which infrastructure costs are quite high and awareness-raising efforts are often defaulting because focused primarily on the marketing and not on the infrastructure hygiene and maintenance principles. Thus, if the CLTS infrastructure were counted then the access would be significantly improved, the only limiting factor being the non-sustainability of the latrines that very often collapse during the rainy season. 5.10.3 The improvement of the sanitation technical package The USAID/PEPAM project enriched the sanitation infrastructure technical package by considerably improving the Sanplat latrine with masonry linings (see sustainability). An innovation is also made in the DLV. They are built with a fixed flagstone unlike the removable flagstone that was "admitted" earlier. Indeed, the moving of a removable flagstone poses hygiene problems with significant health risks incurred by the households because of its handling and the not-yet-dried sludge exposure to open-air. Even if there is a margin for progression to improve this infrastructure, it should be noted that USAID/PEPAM contributed positively to the improvement of the rural sanitation technological package. 5.10.4 The environmental monitoring The environmental monitoring has been at the heart of the project implementation. At any execution stage, the environmental effects were assessed and mitigation actions taken. Even if some development projects in the water and sanitation sector in rural areas have prepared PGES, it should be recognized that the environmental monitoring as it has been integrated into the activity coordination and follow-up mechanisms of USAID/PEPAM is unique. The late involvement of the offices with capacity to deliver the final environmental discharge was the weakness worth mentioning. Notwithstanding this shortcoming, the USAID/PEPAM environmental monitoring model is a reference in the sector. 5.10.5 The introduction of min-borehole construction technology The mini-borehole construction according to the "Rota Jetting" technique is an appropriate technology for carrying out infrastructure in a very reasonable time with highly optimized costs and a better safeness of the water table, if there is a good framing. Page | 58 In sedimentary area and for a certain depth, it is a very interesting alternative to the modern well equipped with PMH. Pending the documentation of this technology to learn more about the harnessing infrastructure performance and limits, it proves that the hand pump-equipped mini-borehole installation cost is much more competitive than a modern well without PMH. If the construction methodology is "validated", then an important step would be crossed to its popularization to resolve the potable water access gap of the small localities isolated and far away from the multi-villages networks. These scattered localities are a real issue for PEPAM because the often prohibitive cost of modern wells and the sometimes very difficult execution conditions hamper their easy access to water. 5.11 Specific issues per component 5.11.1 Component 1: strengthen the participative governance To what extent have PLHAs been developed, validated, and utilized? USAID/PEPAM carried out 52 PLHAs, 25 for the Ziguinchor region and 27 for the Kolda region. Following a call for tenders, 3 consulting firms (GERAD, MSA, and AIDF) were commissioned to carry out the PLHAs in the Kolda region. In the Ziguinchor region the consulting firm MSA was the only contractor. The PLHAs development process was not that participative and in some cases the locally elected officials were informed of the process at that time of the review of the draft documents for validation. This implied that the contractors did not go through, among other stages, the couple of preliminary validations by the populations (inventory data and supply solutions). Of the 52 PLHAs, 51 were validated by the rural communities. In terms of the PLHA use for the project implementation, it should be noted that the PLHAs were not actually used for the choice of sites in the sense that they were developed during the activity implementation period. Indeed, in Ziguinchor, the choice of sites was made with the collaboration of the technical offices which recommended priority sites without scouting visits. In the Sédhiou and Kolda regions, the villages looking for small hydraulic were all pinpointed and selected from the existing planning documents before the start of the project, namely PLD, PAI and existing PLHAs. In the Kolda region, the PLHAs developed thanks to the USAID/PEPAM project were made available after the start of the project in the area. At the project startup, 30 villages were selected. When expanding the site coverage, the PLHAs were used for identifying the sites. The total intervention village numbers were 77 in Kolda and 65 in Sédhiou. The PLHAs that USAID/PEPAM carried out for the Vélingara department were used for choosing the large-sized water infrastructure sites (network extensions, water tower rehabilitation). How well do the local partner organizations understand and effectively implement the trainings by the USAID/PEPAM project? The USAID/PEPAM project provided various trainings to the local partner NGOs. The trainings allowed the partner NGOs to, on the one hand, generate significant applications from the users, and, on the other hand, follow up the investment setting up process Page | 59 (constructions and supervision of the users and managing bodies). Thanks to the trainings, some NGOs were able to acquire and apply new skills in water and sanitation. As a result, these NGOs position themselves as real regional actors in the sector. Examples of such NGOs include Kabonketoor and PACTE (Ziguinchor), Casades and “Enfance et Paix” (Sédhiou), ENDA and Fodde (Kolda). (See the capacity strengthening section for more information on the training modules). How effective are the working relationships between the project staff and the staff of the government agencies at the local and national levels (hydraulics office, upkeep and maintenance office, sanitation office, hygiene office)? At mid-course, it is clear that the project strategic oversight is not consistent with the ministerial decree. The assessment of the working relationship with the national level can be done only on the basis of the functioning of the oversight body in terms of responsibilities and implementation of decisions and recommendations made by the oversight body (responsibility to be taken on). From this perspective, the working relationships are almost non-existent. From another perspective, the working relationships can be assessed against the information flow. In this respect, USAID/PEPAM reported that they have regularly sent the activity reports and the annual work plans to the various relevant directorates. In all cases, there is urgency to make the working relationships fluids and it is the project responsibility to be proactive in the scheduling the oversight body meetings and in implementing the agreed upon actions. At the regional level, USAID/PEPAM signed a memorandum of understanding with the water and sanitation offices so as to frame the interventions and govern the working relationships. It is same with the education inspections. On the contrary, as of March 31, 2012, USAID/PEPAM did not define any collaborative framework with the regional hygiene offices, development agencies, and DREEC. It should be noted that even if the dissatisfaction level is quite high, the project provided to the water, sanitation, and hygiene offices strong support in terms of capacity reinforcement and institutional support. It would be important to hold biannual follow-up and coordination technical committee meetings. This would help involve the regional actors in the project operational management (See project governance and satisfaction of stakeholders sections for more information). 5.11.2 Component 2: increase demand for WSS and hygiene How much demand has been generated thus far, and are activities under USAID/PEPAM sufficient to fulfill demand? The satisfaction level of the demand generated in hydraulics (mini-boreholes and wells) and sanitation (washbasin devices-equipped latrines) is up to 75% compared with the set targets. The table below shows the results as of March 2012. Page | 60 Sub-sector # of actual demand # of infrastructure built Satisfaction rate Hydraulics 180 136 76% Sanitation 3338 2448 73% 75% Source: USAID/PEPAM FY2012 Quarter 2 report dated May 2, 2012 (final French version). The multiple marketing activities (social mobilization, discussions, home visits, community meetings, etc.) helped a lot generate demand for water and sanitation from the target populations. A large number of potential applications are registered and even the non-target surrounding villages plead with the project to support. The populations reported that they understood belatedly because experiences with previous projects made them suspicious of such interventions. There are various and sufficient activities to generate a large number of applications. However, the implementation needs to be readjusted and the target populations better trained, skilled, and empowered. The potential demand generation level is very high (30% of satisfaction compared with the potential demand). Yet, at the time of mobilizing the counterpart funds, the actual demand is less important particularly for the potable water component. In any case, there is need to review the implementation strategy to fill the 25% gap and consequently satisfy the justified applications that were pending. Do the beneficiaries understand the way to use the washbasin devices and Aquatabs, and do they ask for these interventions? All carried out latrines are equipped with washbasin device called "Tipy Tap" by the project. These washbasin devices, made by the beneficiaries, are hand-made and not easily manipulated by some household members, especially the children and the elderly. The hand washing water is very badly collected (in a dirty can) and pours very often on the ground (poor hand washing water management). Most of the washbasin devices are destroyed or kept in the rooms. The reasons are that the children make playthings of the devices. There is a high risk that, in a short-term, hand washing at the exit of the latrine will no longer be a reflex. There is a stock out of the Aquatab tablets. The household survey results showed that the households prefer using the chlorine. Such a result corroborated the statement made by the Ziguinchor regional hygiene squad head. The Aquatab tablet use lasted only one year, even though USAID/ PEPAM had negotiated a permanent distribution with ADEMAS at the local level. This situation makes the population returned to the use of chlorine they prefer. Are the communication tools used by USAID/PEPAM for the hygiene promotion appropriate? Are the community campaigns radio and video broadcast effective? To sensitize the populations on the hygiene issues, the project has used the PHAST/SARAR method, which is based on the person who learns. The project developed tools which were used for implementing the NGO-led activities relating to community mobilization and school awareness-raising. Page | 61 Community mobilization The project designed a manual with visual plates on the three stacks of assorted cards, the contamination pathways and diseases barriers, the sanitation scale and break story which are practical tools of the approach. The manual includes 7 tools.  3 stacks of assorted cards: three themes are tackled. They relate to hand washing at the exit of the latrine, the Aquatab tablet use for the water purification, the water point management and the VIP maintenance.  The contamination pathways and diseases barriers: as its name indicates the tool explains the various disease contamination pathways relating to a lack of hygiene and the barriers to avoid them.  The sanitation scale: the tool highlights the various project technologies so that the beneficiary can make the right choice from the existing technologies.  The break story: this tool describes two different situations, a good vs. a bad one. The purpose is to assess together ways to get to the right situation with the project activities. In parallel to awareness-raising with the tools, there are spots and radio broadcasts at community level. An agreement was signed with the community radio network of the area and the project has recorded 2092 spot broadcasts for the promotion and the hygiene –related behavior change. At the school level The project also designed a manual for use in school. The manual includes three tools, of which two are the three stacks of assorted cards (latrine maintenance and hand washing at the exit of latrines) and a last tool focusing on the school hygiene-related break story. In summary, the communication tools used for the hygiene promotion remain much appropriate. Essentially, the various themes are satisfactory with a few remarks. The open air defecation barrier is limited to the VIP omitting the other technologies proposed by the project. The sump wash-house for waste water management is on the awareness raising tools and is not on the technological package of USAID/PEPAM. In terms of radio communication, the surveys show that only 6% of the households reported hearing the messages. So even if the spots have been translated into local languages, there is a need to review the communication strategy, particularly the broadcast hours. There is also a need to assess if the partner radios normally cover the intervention sites and they have audience among the community. The school-based awareness raising activities have not yet started. Are the events organized on the project sites and at the regional level with the local government technical offices effective? The events organized on the sites with local technical offices focus on awareness-raising actions, infrastructure acceptance visits, and water quality control checks. In terms of awareness-raising, it is about particularly the end of open air defecation declaration ceremonies by the hygiene and sanitation offices. To determine if this event Page | 62 has been effective, the action should have been evaluated at least one year after to assess whether there is no relapse. On the ground, a few months after the celebration of some sites, the FDAL seems to be respected. The acceptance actions relate to water and sanitation infrastructure and are taken by the regional water and sanitation offices, respectively. It was not noted any refusal of acceptance for non-compliance. So the action was effective as long as the objective is to complete the acceptance of the infrastructure and this was done. Concerning water quality, the regional hygiene offices conducted the quality testing on the ground. The office heads found a quasi-absence of coliforms, and this demonstrates that the control action the competent office carried out was effective. 5.11.3 Component 3: Strengthen the capacity of small private service providers and water user associations Are latrine maintenance services available and accessible? The beneficiary provides latrine maintenance services. To that end, the partner NGOs took awareness-raising actions for a routine maintenance and proper use of the sanitation facility. The maintenance is done by using brooms, ash, soap, etc. On the contrary, if by maintenance services, reference is made to private service providers for emptying sludge, then this is not yet available on the sites. What is the effectiveness degree of the training components relating to builders, drillers, WUAs, community relays, corporation management, etc.? The training provided to builders and drillers were very effective and allowed them to carry out infrastructure consistent with the technical requirements (see the technical viability). The training sessions allowed them to claim for other contracts. The most illustrative case is that of Assukatene Water which developed an extensive expertise after training and today it is able to compete and win solicitations in the region and in Guinea Bissau. For the community relays, their training was also effective if it is assessed against the generated applications. On the hygiene principles relating to water and sanitation, and the use and maintenance of sanitation facilities, although training modules addressed them, the situation on the ground is unclear. Awareness-raising is to be reinforced especially in the sites where CLTS was not promoted. For the quality of the training received by WUAs, if some properly use the tools made available, the vast majority of encountered committees do not fully understand them (see the next sub-section on the tool use). (See section on capacity strengthening for additional items). Indicate the quality and degree of the use of manuals and guides developed by the project. Page | 63 The manuals and MeRO guides are acceptable in their design even if they do not totally accord with the harmonized reform tools (social engineering manual - harmonization of procedures and methods of community mobilization, management and maintenance of motorized rural boreholes in Senegal/DRH-DGPRE/March 2007). The project-designed tools do not include some water point management and operation tools. The degree of tool use by established and trained WUAs is low per the low activity level around water points. Indeed a true borehole management does not exist because of the prevailing flat-rate contribution to water consumption. For the awareness-raising activity-related manuals, refer to issue 3 of component 2 "Specific issues by component" component. In addition to the manuals, the project prepared leaflets providing information on the various products (water and sanitation infrastructure). These documents are well done and provide useful information helping decision-making easily. How often is the maintenance made? Specify the community’s capacity to cover maintenance costs. Potable water USAID/PEPAM published a use and maintenance manual very explicit, very clear and very useful for each actor. The manual indicates preventive maintenance actions to be taken locally each week. The preliminary preventive maintenance level observed by site especially depends on the pump attendant. The other level concerns preventive maintenance or repair to be performed by a professional. Although it is not yet operational, the maintenance strategy foresees preventive maintenance visits on a biannual frequency. The repairer artisan will check the pump and advise the beneficiaries on measures to be adopted to ensure a better operation of the equipment. The management committee will be charged of the service provision costs. At mid-course, in terms of maintenance cost recovery capacity, referring to the financial viability analysis, the managing body of the installed solar pump will have difficulties to cover the maintenance costs. The managing bodies of 58% of the India pumps and 80% of the Erobon pumps will also have recovery cost difficulties. For the other systems to be carried out, refer to financial viability analysis. (See financial analysis). Sanitation For the sanitary facilities according to the surveys, latrine maintenance is done by 98% of the beneficiary households. The survey data on the maintenance frequency show that 80% of the maintenance is carried out on a daily basis, 16% of the maintenance carried out twice a week, 2% of the maintenance carried out once a week, 1% of the maintenance carried out once a month, and 1% of the maintenance carried out after each use. The households that do not express any opinion on the maintenance, because of very irregular Page | 64 maintenance, represent 2% of the beneficiaries. This reflects a high level of facility maintenance but remains to be confirmed by the field observation. The monthly average expenditure for latrine maintenance estimated at CFA F1,353 per household is relatively low and affordable by the users. Are members of the maintenance staff trained and available to provide the required services? Did they sign maintenance contracts specifying their roles and responsibilities? In the management Committee, there is a local repairer whose role is to provide small routine maintenance tasks. The project trained those local repairers. Concerning the repairer artisans, to ensure cohesion of interventions and synergy of actions, steps were taken with the other partners, in collaboration with the technical offices in order to better address the pump maintenance issue in a comprehensive and cross-cutting manner for all actors. In the intervention zone, 9 artisans were identified and trained to ensure maintenance for USAID/PEPAM and ICRC. The distribution of repairer artisans was made on the basis of their geographical positions relative to villages to be followed. The signing of the contracts is the last stage of the process underway. 5.11.4 Component 4: Install and rehabilitate the potable water supply and sanitation infrastructure, through a service provision framework Are the water points and wells used? If they are not used, please explain why? At mid-course, the water points built and the rehabilitated wells are used by the beneficiaries. The evaluators did not find any case of non-use in the sites visited. Are the latrines functioning and being used? If they are not being used, explain why? The latrines built are functional and are in use by households. Remarks were made on the facility quality and their design (see technical viability). The few rare households which are not yet using their facilities are waiting for traditional latrines to fill before starting the use of the newly built facility. Has the GOS certified each well and water supply system? An infrastructure certification protocol has not been established by the GOS. An infrastructure is considered to be "certified" by the GOS if the regional hydraulics offices complete the acceptance and declare that the water quality meets the requirements by the regional hygiene office. The water infrastructure listed at mid-course fulfilled these two conditions. Are the infrastructure costs for the communities affordable? The water and sanitation infrastructure costs are generally considered to be affordable by the communities. The households, in their vast majority, are even ready to spend more to improve the water service quality. The populations hailed the efforts made by the project to minimize the costs. Page | 65 On the contrary, in Kolda there are peculiarities insofar as the local governments paid for the populations most of their contributions to the hydraulic infrastructure due to the declared poverty level of the targeted localities. 5.11.5 Component 5: Water and development alliance (WADA), and CLTS To what extent do community members (e.g., school children or women) demonstrate knowledge of the PHAST/SARAR method as behavior change tool for improving water management and sanitation / hygiene behavior? In the CLTS sites, the community members have a good understanding of the hygiene and behavior change principles. This is evidenced by a good level of the compound and street cleanliness, a satisfactory water point and latrine maintenance level. It is also noted in these sites a traceability of the management committee meetings and the action plan implementation. (See the subsection CLTS promotion under the section added value of the USAID/PEPAM project). What benefits and impacts have there been with implementing the WASH program in the school activities? At mid-course, USAID/PEPAM built mini-boreholes in the schools as well as a public convenience. The inspectors the evaluators met reported the following effects:  Highest number of children especially girls are enrolled in schools.  Regular attendance of the children (girls and boys).  Improvement of school performance.  Disease reduction.  Better monitoring of the curricula. Concerning the awareness-raising, only a few teachers were trained on the hygiene and sanitation principles. The stretching had not yet started in the schools. Is USAID/PEPAM on track to meet the FDAL targets? For the first CLTS sites, the villages met the end of open air defecation requirements 6 to 7 months after the CLTS program was launched. The period between the launching and the celebration is thus consistent. For the other sites, the FDAL stage is reached at the end of 6 months. The follow-up visits showed that the FDAL was still respected and there were no open air defecation (DAL) attempts. To meet the FDAL requirements over time, durable sanitary facilities must be available. The facility crumbling during the rainy season jeopardizes the continuous meeting of the FDAL requirements. In the event of a final project evaluation, there should be a close look at this issue so that the experience could be documented or alternate solutions could be found. It is premature at this stage to maintain that the FDAL targets will be met. At least two years after the celebration seems to be a reasonable time lapse before concluding that the FDAL is effective actually. To check compliance with, there should be unexpected spot checks. Page | 66 What is the value and benefit of adding WADA II synergies between WADA and USAID/PEPAM, enlarging the CLTS program and adding WASH in schools? The WADA allows the project to target additional beneficiaries. In the absence of visibility on the financing plan, it is quite difficult to assess the intervention impact on the investment volume in terms of infrastructure construction. Funding increased but the number of infrastructure to be built did not increase accordingly. This would mean that initially planned infrastructure in localities will be rather built in the schools. Under these conditions, the WADA is not well performing as it is neither efficient nor effective. (See effectiveness and efficiency sections). Did the contractual relationship and partnership with the local NGOs produce the expected results and contribute to the results and sustainability of the project? (The project just concluded its partnership with five local NGOs in Ziguinchor, and will soon end partnership with five NGOs in Kolda and Sédhiou.) Contracting with local NGOs was beneficial given the large number of villages to be covered by the project. The local NGOs took multiple communication and social engineering actions that solidified the project structure as well as its components. The NGOs worked on quotas set by the project. As of March 2012, they built 90%, 70%, and 60% of the mini-boreholes compared with the set quotas in Ziguinchor, Sédhiou, and Kolda, respectively. For the sanitary facilities, in Ziguinchor, they built twice the set quota. In Sédhiou and Kolda they built about 70% of the planned facilities. On these bases in terms of achievement, the results expected in relation to the set quotas are being met, provided the partnership is pursued in Sédhiou and Kolda. However if the sustainability of the sanitary facilities is not a problem, concerning most of the hydraulics works carried out at mid-course their durability is strongly threatened. (See viability – project governance sections). What are the value and benefits of the partnership with the Peace Corps PROSPERE project? The Peace Corps volunteers participate in the latrine construction activities in the project intervention areas. The CLTS program in the rural community of Badion (Kolda region) is part of their activity scope. Collaboration may be considered so that they serve as relays in hygiene and sanitation promotional activities with regard to the end of contract with local NGOs. It would be important to also determine how the project could develop latrine subsidy program the Peace Corps intervention sites for a better facility follow-up ensuring regular maintenance and optimal use thanks to awareness-raising activities the volunteers already carried out. Page | 67 What are the value and benefits of the informal partnerships and synergies with other partners such as ACRA, UNICEF, Spanish Bonsaid, JICA, etc.? Synergy was created with the ICRC that built mini-boreholes in the Kolda and Sédhiou regions. ICRC also trained repairer artisans who generally provide the project-funded pump maintenance services. An agreement was established with ICRC for meshing the intervention areas of the two projects, and jointly getting the repairer artisans organized. According to their assignment areas, the repairer artisans of the area intervene regardless of the financial backer (USAID/PEPAM or ICRC). This helps to rationalize the travel times and/or distances of the repairer artisans. It should be also noted that repairer artisans trained by UNICEF are called on for repairs, if need be. The partnership with the Spanish NGO CAPP in Ziguinchor makes it possible to construct mini-boreholes. There too, the lack of visibility on the expected results does not allow the evaluators to express any opinion objectively. 6. Conclusion and recommendations 6.1 Conclusion At mid-course, the USAID/PEPAM project allowed 22,500 people (16% of the targeted objective) to gain access to potable water, and 24,500 people (26% of the targeted objective) to gain access to sanitation. The budget execution rate is 48% and the infrastructure overall physical execution rate is 25%. The direct investment relating to potable water and sanitation services is very low at mid-course (15% of the total amount spent by USAID and beneficiaries, or 8% of the overall budget). Following the analysis of the various performance criteria and cross-cutting themes, the evaluators’ general opinion is that the intervention is still relevant, weakly effective and weakly efficient. The subsidized sanitary facilities are durable, whereas for the potable water component, the supply system sustainability is jeopardized. The project oversight was problematic at the strategic level as well as at the operational level notwithstanding the implementation of a significant capacity strengthening component. In spite of the loose project oversight, the population expressed satisfaction of the project implementation, especially with regard to sanitation. The other various actors expressed rather more or less significant dissatisfaction. Moreover, the project contributed significantly to the water and sanitation sector by making available PLHAs for two regions in their entirety, promoting CLTS, and enriching the sanitation and potable water technical packages. Page | 68 6.2 Recommendations For future interventions in the potable water and sanitation sector, USAID would better go through the identification and conception stages before starting the implementation. These stages consisting of feasibility, baseline, technical, and socio-economic studies make it possible to define the priority needs, list the problems to be solved, and understand the expectations of the various partners before starting the project. 6.2.1 Intervention methods 1. Review the project implementation strategy by focusing on direct investment relating to potable water and sanitation infrastructure. 2. Clearly define and delineate the roles and responsibilities of the USAID/PEPAM field staff and the partner NGO staff. The evaluators noted a lot of overlap and inconsistent messages on the ground. 3. Assign to the various USAID/PEPAM staff the follow-up tasks of specified partner NGOs. This will inform the partner NGOs of their respective interlocutors and backstops in implementing activities. As backstops they will also serve as the interface with the component managers. 4. Reorganize the intervention team in a way that the activities-indicators-results relationship is consistent with the project logical framework. 5. Charge the component managers to monitor their relative indicators. 6. Finalize partnership agreements with ARD, DREEC, and SRHyg. 7. Assess the memorandum of understanding with the hydraulics and sanitation technical offices and think of possibilities to improve the partnership. 8. Make a global budgetary arbitration and provision sufficient resources for the direct investment relating to the potable water and sanitation infrastructure by promoting locally understood technologies consistent with the sector guidance. 9. Hold oversight committee meetings at least twice a year for reviewing the project implementation and providing guidance. 10. Hold operational coordination meetings at least once a quarter with the sub-national technical offices for assessing the implementation on the ground. 6.2.2 Potable water-related technical aspects 1. Review the investment setting approach so as to:  comply with the CUI and promote multi village water supply systems or extensions/densifications on the basis of PLHAs carried out;  equip the mini-boreholes or rehabilitated wells with the India pump instead of the Erobon pump which is characterized by frequent rope breakings and an arduous nature of the manual pumping; and  build mini-boreholes in isolated sites that cannot be served by the multi village water supply systems or network extensions. 2. Prepare and issue a paper documenting the mini-borehole experience and specifying the features of the built infrastructure so that the relevant public offices can possibly endorse the mini-boreholes and that the possibility to motorize the water draining can be considered. 3. Review the mounting method of the India pump cistern. Page | 69 4. Define a common maintenance strategy with the other partners involved in promoting hand pumps in order to define an attractive market for spare part repairers and suppliers. 5. Protect properly the mini-borehole at the end of the drilling by cementing the borehole head and closing compactly the emerging pipe of the pumping chamber. 6. Study the financial viability of the mini-water supply system according to the pumping. 6.2.3 Sanitation-related technical aspects 1. Improve the autonomous sanitary facility construction (washbasin device and Sanplat):  Partition the pit lid flagstone to ease its opening and emptying.  Close the ventilation pipe of the unused septic tank of the DLV to prevent the water ingress or the introduction of foreign bodies. 2. Think about the strategy to be implemented for Sanplat beneficiaries in order to avoid the return to the open air defecation. 3. Continue the CLTS promotion and accompany the sites after the launch for the construction of durable facilities (e.g. support to the bond stone use for masonry linings in partitioning the pit; ventilation pipe with anti-mosquito mesh; iron plate for the defecation hole). This would make it possible to count these facilities as part of the technology package and boost considerably the access rate; otherwise all the efforts made to reach the FDAL stage will be useless as long as they are related to no indicators at the national level to assess the impact on the MDGs. 4. For the CLTS sites, accompany the committees’ action plan implementation by providing a minimum of cleaning equipment (brooms, rakes, wheelbarrows, machete, etc.). 5. For the peri-urban areas, due to the low amount allocated for collective or semi￾collective systems, it is more appropriate to promote autonomous sanitation consisting of excreta management facilities (latrines) and waste water (showers and sump wash-houses). 6. Start the WASH program in the schools and select indicators allowing to track the effect (e.g. gross enrolment rate and girl enrolment rate; absenteeism; repetition rate, etc.). 6.2.4 Organizational aspects 1. Tailor the size of the committee managing bodies for the local realities. The management committees relate to small localities. Despite the adjustments made by the project according to the resource person availability at the locality level, the managing body structuring presents a complex and heavy character compared with the village sizes, the availability or competencies of the voluntary members of the executive boards. During the project implementation in some villages, the managing bodies are functioning and their positions are filled, whereas at the project end the situation may worsen, ranging from poor to low. This should lead to reconsider the necessity to retain some positions. Consequently, there is a need to adapt more the managing bodies to the local realities for more effectiveness. Page | 70 2. Align the MeRO manual on the harmonized tools of the hydraulics subsector and simplify it for the management committees. Some operation and management tools required by the reform were not included in the MeRO manual. It is important for USAID/PEPAM project to align with the harmonized outreach guide and build on the achievements of previous projects for more compliance with the subsector tools. There is a need to simplify in all cases some tools for the management committees. Indeed, for example the procedures to execute an expense are hardly understandable by the managing body members who do not always understand the usefulness and urgency to go through several stages before paying the small service provider who repaired the Erobon pump rope. 3. Translate the management tools into the local languages in order to stress the gender dimension. On average, 45% of the management committee members are women. Because they are more available in the localities than the men and young people who are rather interested in other activities, it is easy to understand that the effectiveness of their actions within the managing bodies depends on their ownership degree of the management tools. Good ownership and use of the tools are only possible if the tools are translated into the local languages. The translation of tools will promote the understanding of the hygiene and sanitation-related activities. 4. Systematize the water payment to allow an effective use of the tools in the MeRO manual. The irregularity of the water consumption fee collection does not allow a better management. The established formal or informal contribution systems are not conducive to the achievement sustainability. Although the methods and principles were adopted by consensus at the village level the irregularities of contribution mobilization reflect on the user side the will to have free water always, awaiting still that a project or program come and solve the water access problems of the locality. 5. Encourage a continuation of the hygiene and sanitation-related household awareness￾raising sessions. This is to create the conditions so that the WUAs can pursue the control and community mobilization actions towards the latrine beneficiaries. This activity requires that the local NGOs which accompanied the management committees during the project implementation period continue to provide support. Most of the trained management committee members positively learned from the process. However, in the medium-term, only a follow-up reinforcement during critical periods can lead to an effective ownership of the management methods, hygiene practices (regularity of hand-washing), and latrine maintenance, especially for the subsidized program. 6. Create regional coordination units of the management committees in view of sustainability. It may be considered at the level of each region a framework comprising the presidents of the mini-boreholes management committees which were established to maintain the momentum, reflect together on facility management methods, and define development methods. Experience exchanges can be possible within this framework of expression or even the partnership combined research. These coordination units can be grouped into a USAID/PEPAM-related WUA and management committee federation once the project withdraws definitively and hands the activities over to the sub-national public offices. Page | 71 Annex 7.6 Terms of reference 1. Purpose and use of the evaluation The purpose of the evaluation is to assess progress to date and identify improvements that will facilitate the attainment of planned results. The evaluation will help to identify strengths and needed adjustments to the intervention approaches, the nature of services, and the efficiency with which the project provides inputs. The evaluation will provide the USAID/PEPAM project and its partners with appropriate information and recommendations on tools, approaches and management for the remainder of the project. 2. Background and Project description Despite outstanding efforts made by the Government of Senegal (GOS), along with the support of development partners and donors, several regions of Senegal still suffer from low access to safe drinking water and a lack of basic sanitation systems. Senegal has set its national Millennium Development Goals (MDGs) for connection to safe and potable drinking water for urban and rural areas at 90 percent (100 percent in urban areas and 82 percent in rural areas) and for sanitation facilities at 67 percent (78 percent and 59 percent in urban and rural areas, respectively). The country is moving swiftly toward these goals. According to the PEPAM Coordination Unit (UCPEPAM) study (December 2010), the current national access rate to potable water in urban areas is 98.5 percent, i.e., 8.5 percentage points gained toward MDGs, compared to the 2004 baseline of 90 percent. During the same period, the rural water coverage rate went from 64 percent in 2005 to 72 percent. Progress for sanitation services is slower. The access rate for urban sanitation grew from 56.7 percent in 2004 to 63.1 percent in December 2010. For access to rural sanitation facilities, 29.6 percent was recorded in 2010, compared to a 26.2 percent baseline rate in 2005. The UC-PEPAM study (2010) revealed that the Casamance region in Southern Senegal, which includes the Ziguinchor, Kolda, and Sédhiou regions, ranks at the bottom of the list in its access to safe drinking water. The Kolda and Sédhiou regions’ access rates are 36.8 percent and 57.9 percent respectively, i.e., less than the national regional average (77.5 percent). The access rate for the Ziguinchor region is 86 percent, substantially higher than the average. Access to sanitation facilities still lags considerably behind that of water supply systems in Casamance. The UC-PEPAM survey report (2010) indicates the following:  In Ziguinchor, the rural population access rate to sanitation infrastructure is 26.2 percent, less than the national average of 29 percent.  In Sédhiou, the sanitation infrastructure inventory revealed that coverage was stable, albeit very low: 8.1 percent in 2009 and 2010.  Kolda demonstrates the same trend as Sédhiou. The rate recorded there was 8.1 percent in 2009 and 2010. USAID/PEPAM is a US $21 million, 5-year project, which started in July 2009 and will end in September 2014. The purpose of the project is to improve sustainable access to water supply and sanitation (WSS) and to promote better hygiene in rural, small towns and peri-urban areas of Senegal. The development hypothesis for the USAID/PEPAM project is such that improved access to water and sanitation and hygiene (WASH) will be achieved if participatory governance is strengthened, demand for WASH services are increased, local WSS business opportunities are created, improved water and sanitation infrastructure are constructed/rehabilitated, and community-led total sanitation (CLTS) strategies are used an promoted. For more details, see the USAID/PEPAM Project Results Framework in Exhibit 1. The Project targets regions within Senegal that currently have low access and service delivery for WSS. The current intervention sites are in the Casamance region - Ziguinchor, Sédhiou, and Page | 72 Kolda. The Project will expand to two additional regions - Tambacounda and peri-urban Dakar - upon recommendations of both USAID/Senegal and the program Oversight and Coordination (Steering) Committee, headed by the Ministry of Habitat, Construction, and Hydraulics (MHCH). USAID/PEPAM now has five key results that contribute to its five components and, in combination, to the purpose of USAID-funded programs, presented in Annex A. The Project has established 37 indicators to track and measure progress on the water and sanitation activities and 18 indicators to track the CLTS, hygiene promotion, and WASH in schools activities. Several of the project monitoring and evaluation (M&E) indicators are appropriately aligned with USAID and GOS–PEPAM indicators. The Project uses PEPAM multipliers and definitions to measure access and count beneficiaries. A key strategy adopted for the implementation of the Project is to ensure that there is active local participation and buy-in with the population’s participation and the involvement of ministerial departments in charge of water and sanitation issues at national and local levels, as well as the collaboration of nongovernmental organizations (NGOs) and community-based organizations (CBOs). This includes the following:  Active participation from civil society/local population  Active and engaged local government  Buy-in and involvement from government technical units and ministries responsible for the provision of water and sanitation services the national and local levels  Capacity building for NGOs and CBOs The project management and implementation has been entrusted by USAID to a consortium led by RTI International as prime contractor, with the support of TetraTech/Associates in Rural Development (ARD, Inc.) and Relief International/Enterprise Works/VITA (RI/EWV) as sub￾contractors. There are 11 partner NGOs that collaborate with the consortium and play an important role in project implementation: ADY, AJAC LUKAAL, AJAEDO, PACTE, WESWA, and Kabonketoor (in Ziguinchor); and ENDA, CARITAS, CASADES, FODDE, and Enfance et Paix (in Sédhiou and Kolda). For CLTS and WADA II activities, Kabonketoor has been contracted to operate in the Ziguinchor region, CASADES in Sédhiou, and ENDA in Kolda. The Water and Development Alliance (WADA) project, jointly funding by USAID and Coca￾Cola, will take place in the region of Casamance, Senegal. This project will leverage USAID and Coca-Cola's unique partnership to build on and support existing USAID programs in Senegal, notably the U.S. Agency for International Development funded Millennium Water and Sanitation Program (USAID/PEPAM) implemented by Research Triangle Institute international (RTI) and its various partners. WADA will provide additional support to both water supply and CLTS components to complement USAID/PEPAM's efforts in these areas. USAID/PEPAM reports on three (3) standard indicators, whose targets and actual data should be evaluated: 1. Number of people in target areas with access to improved drinking water supply as a result of USG assistance. 2. Number of people in target areas with access to improved sanitation facilities as a result of USG assistance. 3. Number of producers’ organizations, water users associations, trade and business associations, and Community-Based Organizations (CBOs) receiving USAID assistance. 3. Information Sources The evaluation team shall familiarize itself with USAID and project documentation. USAID/Senegal will ensure that all relevant documents are available to the team prior to the field work. The documents will include, but are not limited to:  USAID/Senegal Country Development Cooperation Strategy  USAID/PEPAM project description  USAID/PEPAM annual work plans, annual and quarterly reports Page | 73  Performance Management Plan prepared by USAID/PEPAM  Sectoral action plans  Training manuals  Field trip reports  Plan Locaux d’Hydraulique et d’Assainissement/Local Water Supply and Sanitation Plan (PLHAs)  Environmental Monitoring and Mitigation Plan (EMMP)  Branding and Marking plan  One year evaluation conducted by Jay Graham  Environmental Audit  Performance evaluation of the Water and Sanitation Development Grant Program 4. Issues to be investigated The evaluators will investigate to what extent the objectives in the results framework were achieved to date and assess the likelihood that the expected results will be fully achieved by the end of the project as well as determine the project’s environmental impact on the target populations and geographic zones. The evaluation should provide general recommendations and specific conclusions for maintaining momentum and improving implementation of the project, such as modification of activities, staff responsibilities, timelines, or budget allocation. Lessons learned about compliance with USAID Environmental Regulations and Procedures and with Senegalese Environmental Law and Statutes should also be integrated. The evaluation team will answer the following questions:  Has the project been able to complete the planned activities for each component within the stipulated times?  Have the inputs/services provided by the project reached the targeted population at the desired scale?  What formal and informal mechanisms were established in order to involve key stakeholders?  Given progress to date, are the planned results achievable in the present private or public sector organizational environment? Are the targets set realistic?  Are the linkages and relationships developed with governmental agencies adequate? To private sector organizations?  How well have the activities been coordinated and have efficiencies been established?  Are the interventions still relevant after 2.5 years of implementation?  Are the interventions used during the first half of the project conducive to sustainability? What interventions should be adopted to ensure sustainability?  Are the beneficiaries (community members) able to identify that the interventions were financed by USAID and the American people?  For grants and DGP grantee assistance, how effective is support and programming? The following questions to be answered are specific to each component of the project. Component 1: Strengthen participatory governance  To what extent have PLHAs been developed, validated and utilized?  How well do local partner organizations understand and effectively implement training received by USAID/PEPAM?  How effective are the working relationships between the project staff and staff of the government agencies at the local and national level (Direction de l’Hydraulique, Direction de l’Entretien et de la Maintenance, Direction de l’Assainissment, Direction de l’Hygiene)? Component 2: Increase demand for water supply and sanitation (WSS) services and hygiene  How much demand has been generated thus far, and are activities under USAID/PEPAM sufficient to fulfill demand? Page | 74  Do beneficiaries retain the knowledge of how to use handwashing stations and Aquatabs and do they request these interventions?  Are USAID/PEPAM media and communications materials for hygiene promotion appropriate? Are the community radio and video campaigns effective?  Are events at project sites and regions with local government and technical services units effective? Component 3: Strengthen the capacity of small-scale private sector service providers and Water User’s Associations (WUAs)  Are latrine maintenance services available and accessible?  How efficient and effective are training components for builders, drillers, WUAs, community relays, business management etc.? What is the quality and use of manuals and guides developed by the project?  With what frequency do maintenance needs occur, and what is the capacity of the community to cover maintenance costs?  Are maintenance people trained and available to provide service? Have they signed maintenance contracts that specify roles and responsibilities? Component 4: Install and rehabilitate improved drinking water and sanitation infrastructure, using a service delivery framework  Are the water points and wells being used? If they are not being used, why?  Are latrines functioning and being used? If they are not being used, why?  Has the Government of Senegal certified each well and water system?  Are the wells functional and being used?  How affordable is the cost share aspect of infrastructure for communities? Component 5: Water and Development Alliance (WADA) and Community-Led Total Sanitation (CLTS)  To what extent do community members (e.g. school children or women) demonstrate knowledge of the Participatory Hygiene and Sanitation Transformation/ Self-esteem, Associative strengths, Resourcefulness, Action-planning and Responsibility (PHAST/SARAR) method as behavior change tool for improving water management and sanitation/hygiene behavior?  What benefits and impacts have there been with implementing the WASH in schools activities?  Is USAID/PEPAM on track to meet End of open-air defecation (FDAL) targets?  What is the value and benefit of adding WADA II synergies between WADA and USAID/PEPAM, enlarging the CLTS program, and adding WASH in schools?  Did the contractual relationship and partnership with the local NGOs produce the expected results and contribute to the results and sustainability of the project? (the project just concluded its partnership with five local NGOs in Ziguinchor, and will soon end partnerships with five NGOs in Kolda and Sedhiou)?  What are the value and benefits of partnership with Peace Corps PROSPERE project?  What are the value and benefits of the informal partnerships and synergies with other partners such as: ACRA, UNICEF, Spanish Bonsaid, JICA? 5. Methods and Procedures The project period to be reviewed ranges from July 2009 to March 2012. The evaluation team will develop the specific approach to evaluation methodology and analytical framework, a detailed sampling plan, data collection tools (e.g. survey and interview questionnaires, discussion guide for focus groups, etc.) In developing the methodology, USAID expects that the analysis will consider, at minimum, all Section 4 components, topics and issues to be investigated as well as the ENCAP Visual Field Guides for water and sanitation infrastructure. Page | 75 The evaluation team will review documentation provided by USAID and USAID/PEPAM outlined in Section 3 as well as other available information. The team will conduct in-depth interviews with beneficiaries, inspect and analysis of project activities, interviews with staff the project and interviews with technical services (water and sanitation) and local authorities. The evaluation team will meet and interview representatives from USAID/Senegal, USAID/PEPAM, the Government of Senegal (Ministry of Hydraulics), other donors, NGOs, ASUFORs, Water and Sanitation Development partners and other stakeholders in Dakar and in a sample of targeted localities. The evaluation team will visit a purposeful sample of sites. During field visits, the evaluation team will be accompanied by two members of the Ministry of Hydrology, including a consultant or a USAID/Senegal staff. The evaluation team should work in close collaboration with USAID/PEPAM. The team conducting this mid-term evaluation shall review all the relevant documents pertaining to USAID/PEPAM, including those listed in Section 3. The evaluation team shall propose its own methodology, however it is expected that the evaluation will be implemented through document review, key informant interviews, and focus group meetings. USAID/Senegal expects that the analysis will consider at minimum all Section 4 components, topics, and issues to be investigated. 6. Required tasks The tasks in this SOW will be implemented over a period of about 8 weeks (6 –working days per week), starting around October 2012. The schedule below is illustrative:  Familiarization with USAID/PEPAM project and development of evaluation methodology  Interviews and Field visits  Draft evaluation report  One-day debriefing of findings at USAID  Final evaluation report