USAID/Haiti Internal Evaluation of the Community Reconstruction for Economic and Environmental Resiliency Program Final Report December 2003 I. INTRODUCTION This report summarizes the findings of the evaluation of the Community Reconstruction for Economic and Environmental Resiliency (CREER) Program, Agreement No. FCC￾521-2002/1-19-00. The purpose of the evaluation was to enable USAID/Haiti to assess the implementation of activity inputs funded through CREER. The information provided by this evaluation will inform USAID and the U.S. Department of Agriculture (USDA) of a sample of the work achieved under the CREER program. Although the evaluation team observed only a fraction of the activities implemented under CREER, it had a positive impression of the quality of work performed by PADF and its subcontractors under the three major components of the CREER program. The evaluation team noted the overall satisfaction of the beneficiaries regarding work products and CREER project staff. Beneficiaries also spoke to the positive impacts the project activities had or were expected to have on the economic and social conditions in their respective communities. II. BACKGROUND The subject agreement is between the USDA (acting through the Commodity Credit Cooperation) and the Pan American Development Foundation (PADF) for the supply of agricultural commodities in Haiti under the Food for Progress Act of 1985. The objective of CREER is to increase the economic and environmental resiliency in targeted communities in Haiti. The goals of the program include the enhancement of private market development by (1) increasing the standard of living of participating farm families; and (2) increasing commercial farm sales in Haiti. The proceeds from the sales of 17,000 metric tons of wheat (approximately $2.85 million) were used to fund three major program activities: productive rural infrastructure rehabilitation (including the repair of 21.4 kilometers of farm-to-market roads, reconstruction of three irrigation systems, and protection of three major ravines); the rehabilitation/upgrade of seven schools to be used as emergency shelters during natural disasters; and the formation and training of ten local civil protection committees in disaster preparedness. The majority of funds (78%) were allocated for rural infrastructure rehabilitation while 17% and 5% were used for the disaster preparedness and school rehabilitation/upgrade components of CREER. The CREER program was designed to complement USAID’s Hillside Agricultural Program (HAP) by improving infrastructure and protecting the farmers in areas where HAP is supporting agricultural extension and marketing services. The communities 2 chosen for the rural productive infrastructure and disaster preparedness components were located within the geographic areas already targeted by HAP, primarily in the North and Southeast regions of Haiti. The facilities selected for schools upgrade activities were also located in HAP target areas. CREER is a one-year program that was initiated in October 2002 as a follow-on to the Hurricane George Recovery Program (HGRP) and the continuation of disaster preparedness and mitigation of the Program for the Reduction in the Impacts of Disaster Events (PRIDE). Program activities were scheduled to be completed at the end of December 2003. PADF is working with a number of partners to implement CREER— Catholic Relief Services (soil conservation/ravine protection), the Canadian Center for International Studies and Cooperation (irrigation system upgrades), EATT and GNP Ingeniere (road rehabilitation), and several individual sub-contractors (school upgrades). PADF implemented the disaster preparedness training using internal expertise. III. EVALUATION METHODS AND APPROACH A. Evaluation Team Composition and Objectives The evaluation team consisted of Danielle Typinski/PCPS, Marc-Eddy Martin/EGE, Jean W. Camilien Saint-Cyr/EGE, Ernest Paultre/PCPS, and Florence Liautaud/PCPS. The objectives of the evaluation team were to assess the progress of the execution of activity inputs under CREER, make general predictions regarding impact potential where possible, and provide recommendations related to the completion of CREER as well as the implementation of future related PADF activities. B. Limitations of the Scope of the Evaluation Three primary factors limited the scope of the evaluation and impeded a more thorough assessment of activities implemented under CREER: (1) the findings of the evaluation were needed to inform time-sensitive Mission actions related to a proposed PADF project; (2) the broad geographic focus of CREER and difficult traveling conditions limited the team’s ability to access project sites in a timely fashion; and (3) the incomplete status of the program made it difficult to definitively assess the quality of project implementation and nature of impacts. C. Evaluation Scope All project elements—rural infrastructure rehabilitation, school rehabilitation/upgrades, and disaster preparedness—were considered in the evaluation. Due to its time-sensitive nature, the team limited field work to two days (December 2-3, 2003) in the Southeast region of the country. Project sites within the Southeast region were selected based on project type and accessibility. The final list of project activities visited under this evaluation included a total of seven sites—one site each related to irrigation system upgrades, ravine protection, and road rehabilitation, and two sites each for school 3 rehabilitation/upgrades and disaster preparedness. The field work itinerary is provided as Attachment 1. D. Evaluation Methodology The evaluation team utilized rapid appraisal techniques to assess the selected CREER program activities. The primary techniques used included document review, informant interviews, and direct observation. Performance and baseline documentation was limited to the CREER cooperative agreement and amendment, project inventories, cost proposals, and USDA logistics and monetization reports. The interview participants consisted of PADF representatives and community members that were selected for their direct knowledge and expertise regarding the progress and status of activities implemented under CREER. PADF employees Frantz Joseph (CREER Technical Director) and Adrien Joseph (Technical Supervisor for Soil Conservation) accompanied the evaluation team to all project sites visited during the evaluation. Evaluation team members observed activities and recorded what was seen and heard at the project site using the guidance provided in the evaluation scope-of-work (see Attachment 2). IV. RESULTS OF FIELD WORK A. Irrigation System Upgrades—Orangers Irrigation System Project Background. Ample rainfall during the growing season in this area supports irrigated agriculture, but the water flow has traditionally been poorly managed. Prior to the implementation of upgrades under CREER, the Orangers irrigation system was primitive in nature with earthern ditches and makeshift gates. Minor canal rehabilitation of the Orangers irrigation system was initiated by PADF under HGRP. Tasks identified under the CREER cooperative agreement included the installation of 42 gates, four overpasses, and 1,060 meters of concrete masonry. The total approximate cost of the project is US $52,980. The Orangers system serves 344 households1 , with most households possessing an average of one to two carreaux (1.3-2.6 ha). The system comprises five branches that supply water to a total of 175 hectares. Crops of plantains, maize, beans, peas, and tomatoes have traditionally been cultivated on the land irrigated by the system. A users committee of 15 members (all men), democratically elected by the users to serve two￾year terms, manages and monitors the system’s operation. Five “water police” representatives are paid 1,000 gourdes per month to enforce an established set of rules regarding the system’s use. Fees are currently set at 30 gourdes for a one-hour supply of water per week for one year. Water entitlements are determined by the size of the user’s parcel and the type of crop being cultivated. Members of the users’ committee reported that they have a good working relationship with the Ministry of Agriculture. The two entities have collaborated to develop statutes and enforcement mechanisms related to the system’s operation. Local government 1 2001 figures. 4 representatives, justice officials, Ministry of Agriculture staff and the users committee hold regular forums to resolve issues. Three signatories are required to access the users committee’s banking accounts and a representative from the Ministry of Agriculture must approve all expenditures. Site Visit Methodology and Parameters. The evaluation team met with Dessieux Francois, the head of the Orangers Users Committee, and 11 other committee members at the project site. Initial discussion focused on the management, operation, and impacts of the system (Photo 1). The evaluation team addressed issues specific to the construction of the upgrades while surveying stretches of the system. The following questions were provided in the evaluation scope-of-work as guidance to assess work implemented under the project and any discernible project impacts: (1) Have repairs been fully implemented as specified under the cooperative agreement? (2) Do upgrades appear to be structurally sound? Have appropriate construction standards been met? (3) Has training been provided to local user groups to help them manage the new system? (4) Has the number of users increased as a result of the upgrades? (5) Has there been any change in the cropping patterns as a result of the upgrades? Is more land under irrigation? (6) How have farmers contributed to the maintenance of the system? Status of Work Performed Under CREER. Work began in September 2003 and is continuing. Canal masonry is 100% completed and appears to be structurally satisfactory (Photo 2). Additional work remains to be done on the gates and overpasses (Photo 3). Frantz Joseph commented that the gates for the system have been ordered and will be installed upon delivery in December.2 At the time of the evaluation, two overpasses had been constructed and it is anticipated that the remaining two will be built by the end of 2 The gates were not installed at the time of the evaluation visit because the users had asked the contractor to instead extend the secondary canals in order to serve more farmers. The missing gates were later installed at the contractor’s expense. Photo 1: Meeting with members of Orangers Users Committee. 5 December. The users committee has been trained by CECI and the Ministry of Agriculture in the maintenance and management of the system, responsibilities of key positions, budget preparation and financial management. Key Findings. At the time of the evaluation, the majority of the work had been completed for one month. Consequently, it was too early to assess the impacts on-the￾ground that were attributable to CREER inputs. Representatives of the users committee stated that prior to the implementation of upgrades under CREER, much of the water delivered via the system percolated into the soil before reaching crops. The concrete lining of canals has resulted in a more efficient delivery of water with less waste. The users anticipate that the upgrades to the system will increase productivity three to five times per hectare. For example, members of the users committee estimated that the number of bean plots will be increased from 6 to 30 per hectare, plantain trees from 50 to 350, and pea plants from 5 to 50. The users committee also believed that the improved capacity of the system to provide reliable service during the growing season would increase the willingness of users to pay system fees in a timely manner. Members felt that more consistent payments would Photo 2: Example of canal masonry work at the Orangers irrigation system. Photo 3: Example site where gate will be installed in December 2003. 6 result in an increase of funds collected that could be used for the maintenance and upgrade of the system. A culvert enabling the canal to cross the road to Jacmel has already been built from users fees and others are planned. B. Road Rehabilitation—Carrefour Cirouelle-Macary-Berry Road Project Background. The Carrefour Cirouelle-Macary-Berry (CCMB) Road provides access to an area where USAID has been supporting farmer extension activities through HAP. Under HAP, farmers in the vicinity of the CCMB road had improved their agricultural practices and quality of their products but had difficulty transporting crops from the region because of the dilapidated state of the road. CREER focused on improving road conditions and drainage to prevent future erosion and facilitate improved transportation of crops between regions. Tasks that were identified in the cooperative agreement included the rebuilding of the road platform, the installation of additional culverts as needed, construction of transversal and longitudinal drainage canals, the application of concrete pads in steeply curved areas, and the construction of two bridges at the upper end of the road. The original work plan called for the rehabilitation of 18 kilometers, but a devaluation of the Haitian gourde during the course of the project’s implementation resulted in the accumulation of additional funds that were used to rehabilitate an additional 3.4 kilometers. Two subcontractors were used—GNP Ingenierie for the 0-9 kilometer stretch from Carrefour Cirouelle to Macary and EATT for kilometers 10-21 (Macary to Berry). The total approximate cost of the CCMB road project was US $685,000. Site Visit Methodology and Parameters. The evaluation team drove the first nine kilometers of the Carrefour Cirouelle-Macary-Berry (CCMB) Road with PADF representatives and afterwards met with eight members of the Macary Farmer Extension Group, who represented the users of the CCMB Road. Heavy precipitation at the time of the site visit enabled the evaluation team to observe the functioning of the road’s newly installed drainage system. The team was unable to survey the remaining 12 kilometers due to time constraints and the deterioration of weather conditions. The following questions were provided in the evaluation scope-of-work as guidance for assessing the CCMB road rehabilitation project: (1) Have road improvements been implemented as specified in the cooperative agreement? (2) Were appropriate construction standards met? (3) Does the road provide access to USAID farmer extension activities? Status of Work Performed Under CREER. Road construction began in June 2003 and was continuing at the time of the evaluation. Tasks as specified under the work plan were 100% completed, but revisions such as road widening and replacement of surface material were being implemented. The work plan also stipulated the training of users in road maintenance, as well as the provision of tools after the full completion of 7 rehabilitation work. Because revisions were still being implemented at the time of the evaluation, it was too early for the team to assess the training element of the project. Frantz Joseph indicated that road maintenance training would be conducted by the end of December 2003. Key Findings. The subject road is located in mountainous landscape with extremely steep slopes that requires sophisticated drainage measures for protection. The evaluation team had a positive impression of the quality of the road rehabilitation but also observed that: (1) additional transversal culverts might improve drainage for the road and reduce the velocity of the water3 ; and (2) the mostly clay cover material used on kilometers 0-9 left the surface slippery and made traction difficult4 . The rehabilitated road appeared to be well-used with abundant foot and vehicle traffic viewed in both directions (Photo 5 & 6). In addition, the evaluation team observed the selling of products grown in the Berry region in Macary; thus, the road seems to provide market access for farmer extension activities (Photo 7). 3 PADF commented that standards for secondary mountain roads using average rainfall rates were used for the project. However, overhead rate adjustments in the amount of $22,000 will enable PADF to increase the number of drains installed on the road. 4 This problem had been noted by PADF before the time of the evaluation and GNP Ingenierie is currently in the process of re-applying a second material that is sand-based to improve traction. Photo 4: Road expansion work taking place. Photo shows vertical cuts into cliff side at right. . Photo 5: Vehicle and pedestrian traffic using the Carrefour Ciroulle-Macary￾Berry (CCMB) Road. 8 The team also noted the absence of reference to USG funding on CCMB Road project identification signs (Photo 8).5 5 PADF immediately corrected this situation after the evaluation and added appropriate USDA citation to project signs (Photos 9 and 10). Photo 7: Merchant from Berry selling produce in Macary. Photo 8: CMB Road project identification sign Photo 6: Vehicle traffic on the CCMB Road 9 C. Ravine Protection Project Background. PADF identified three ravines in HAP target areas where flooding most threatens farmers. Under CREER, work in the three localities focused on the prevention of catastrophic flash floods by erecting soil conservation structures (rock walls, gully plugs, etc.) and planting large quantities of trees and other perennials for the long-term stabilization of the ravines. The number of barriers installed in each ravine was dependent on the slope and length of the ravines. Site Visit Methodology and Parameters. The evaluation team observed ravine protection devices installed within the proximity of the CCMB Road. Due to poor weather conditions, the team was unable to walk up the slopes of the mountains and observations of the devices had to be derived from the road. The following two questions were provided in the evaluation scope-of-work as guidance for assessing the status and impact of ravine protection measures: (1) Have protection devices been constructed as specified under the cooperative agreement? (2) What is the impact on the low-lying areas as a result of the protection devices? Status of Work Performed Under CREER. Installation of the protection devices was conducted by CRS. Work began in June 2003 and concluded in September 2003. Although PADF was contracted to protect 100,000 meters of ravines under CREER, a total of 120,000 meters of ravines were rehabilitated. Key Findings. Although the evaluation team was limited to making observations from the road, all ravine protection devices appeared to be functional and well-constructed (Photo 9). At the time of the evaluation, insufficient time had passed from project completion to assess the project’s impact on low-lying areas. However, the team noted that sufficient soil retention was occurring on the tiers behind the barriers to enable the growth of plantain and other vegetation, which indicates that the structures were functioning effectively. Photos 9 and 10. Signs at the CCMB project site indicating USDA as the funding source. 10 D. Disaster Preparedness Training—Macary and La Vallee Project Background. Under the Hurricane George Recovery Program in 1999-2000, PADF identified high-risk zones and developed a training and public awareness campaign to help communities prepare for and reduce the impacts of natural disasters. PADF expanded on their work under the HGRP with the development of community action plans under the PRIDE program in 2002. Under CREER, PADF was to work with ten Local Civil Protection Committees (or KPSLs—Komite Proteksyon Sivil Lokal in Creole) and assist them with the establishment of community warning networks, training in damage evaluation, response and first-aid techniques, and the facilitation of disaster simulation exercises with the participation of the Haitian Civil Protection Directorate. Site Visit Methodology and Parameters. The evaluation team and PADF representatives visited with eight members of the Macary KPSL on December 2 and Photo 9: Example of Ravine protection devices constructed along the CCMB Road. Photo 10: Operation of protection devices during a rainstorm. 11 seven members of the La Vallee KPSL and the mayor of La Vallee on December 3 (Photos 11 and 12). The following questions were provided in the evaluation scope-of-work as guidance to assess the disaster preparedness training elements of CREER: (1) Has training taken place per the cooperative agreement? (2) Does documentation exist regarding the implementation of disaster preparedness workshops? (3) Do training participants have a thorough understanding of disaster preparedness principles as a result of the training conducted under CREER? Status of Work Performed Under CREER—Macary. Disaster preparedness training was conducted in July 2003. One simulation was conducted on July 24, 2003. Two stretchers have been bought and were observed on-site; first-aid kits have been ordered and were expected to be delivered in December 2003. Photo 11: Meeting with the members of the Macary KPSL Photo 12: Meeting with the members of the La Vallee KPSL 12 The La Vallee KPSL was last in the series of targeted KPSLs to receive training. The La Vallee simulation took place on August 21, 2003. Stretchers and first-aid kits were ordered, but not yet delivered. Delivery was expected to occur by the end of December 2003. Key Findings. Although written documentation was used during training, there was no such material available in either location. However, extensive questioning by the evaluation team revealed that the members of both KPSLs were knowledgeable of disaster preparedness principles and of their community’s warning systems and action plans. E. School Upgrades—Ecole Nationale de Evelina Levy/Ecole Nationale de Leonie Jastram and Ecole Nationale de Musac Project Backgrounds. PADF chose seven schools in HAP target areas to be upgraded. Schools were selected based on three criteria: (1) school facilities could be used as emergency shelters; (2) proposed repairs must be sufficient to allow the facility to withstand a Category 2 hurricane; and (3) the school must be functioning and perceived by the community as a common asset rather than as a private venture. Priority was given to school buildings within the areas receiving disaster preparedness training. Examples of repair and upgrade work included the replacement and renovation of roofing, the addition of solid doors and windows, and installation of hurricane clips. In addition, per the cooperative agreement, adequate water storage facilities would be added to those schools where such facilities did not exist. The Ecole Nationales de Evelina Levy and de Leonie Jastram are sister primary and secondary schools that are located in the Centre Ville of Jacmel. At the time of the evaluation, the total enrollment for both Evelina Levy and Leonie Jastram was 1,048 students. The approximate cost for the replacement of the roof and addition of doors and windows was US $15,000. Evelina Levy, a late addition to the CREER project, was a supplemental rehabilitation beyond the original contracted number of schools. Citibank was the first donor to provide funding for related repairs to this school. The Ecole Nationale de Musac is a rural primary school with a morning enrollment of 325 students and an afternoon enrollment of 217. The Ecole National de Musac is located approximately four kilometers outside of La Vallee. Parents and school management participated in the selection and prioritization of repairs to be implemented. Approximately US $6,400 was used to paint two buildings, attach hurricane clips to the roofs, and purchase and install one solid door. Site Visit Methodology and Parameters. The evaluation team and PADF representatives met with the Directors of the Ecoles Nationale de Evelina/Leonie Jastram, Madames Bonnard Posy and Sorel Pompee at the project site. Later that same day, the team spoke with Director Antoine Ridore at the Ecole Nationale de Musac (Photo 13). 13 The following questions were provided in the evaluation scope-of-work as guidance for assessing the school upgrade projects: (1) Have the repairs been completed as specified in the project database? (2) Do construction standards appear to have been met? (3) Is the school available to be used as an emergency shelter, per project selection criteria? Status of Work Performed Under CREER. The work at the Ecoles Nationale de Evelina Levy/Leonie Jastram began in June 2003 and was completed in September 2003. At the time of the evaluation, upgrades to the Ecole Nationale de Musac were mostly completed. The painting of the facilities at the Ecole Nationale de Musac (Photo 14) and attachment of hurricane clips to the roof was completed in August 2003. The door was purchased and delivered but was not installed by the subcontractor (Photo 15). In addition, the new rain gutters were partially installed. Photo 14: View of freshly painted building at Ecole Nationale de Musac. Photo 13: Meeting with the director of the Ecole Nationale de Musac. 14 Key Findings—Ecoles Nationale de Evelina Levy/Leonie Jastram. The evaluation team found the school rehabilitation work performed under CREER to be well-designed and implemented (Photos 16 and 17). However, there was confusion among the beneficiaries as to the source of funding for the upgrades. A plaque outside the main doors of the facility attributed the funding for the work to Citigroup; there was no reference to the USG.6 However, PADF had placed a page on its website (www.padf.org) and distributed brochures about the CREER activity that cited USDA as its funding source. 6 PADF commented that the public inauguration of the Citibank project took place before CREER funds were used. An announcement was made during the inauguration that USDA/CREER funds would soon be invested in another section of the combined schools. A second plaque indicating the USG’s role in the rehabilitation was immediately installed after the site visit (Photo 18). Photo 16: View of roof upgrades and beneficiaries at Ecole Nationale de Evelina Levy. Photo 15: View of door frame at Ecole Nationale de Musac where new door will be installed. 15 Key Findings—Ecole Nationale de Musac. Although incomplete, the evaluation team found the work performed at the Ecole Nationale de Musac to be of good quality. One important note is that PADF had received a project completion notice from the subcontractor despite the project’s incomplete status. 7 V. RECOMMENDATIONS Although the evaluation team observed only a fraction of the activities implemented under CREER, it had a positive impression of the work performed by PADF and its subcontractors under the three major components of the CREER program. The evaluation team noted the overall satisfaction of the beneficiaries regarding work products and CREER project staff. Beneficiaries also spoke to the positive impacts the project activities had or were expected to have on the economic and social conditions in their respective communities. Based on the key findings, the evaluation team has also identified two recommendations for future related PADF projects. The recommendation regarding citing of sponsorship 7 PADF accepted the project completion notice without a final inspection visit because a visit had recently been made indicating that there was very little work left to be done—i.e., the installation of a door (which was on location) and some rain gutters. The work was immediately completed after the evaluation and has since been inspected by PADF staff. Photo 17: Close-up view of roof repairs at Ecole Nationale de Evelina Levy. Photo 18: Plaque indicating USDA financing at Ecole Nationale de Evelina Levy. 16 should be implemented for all activities supported under CREER as well as future USG￾funded projects. A. Inclusion of Gender Issues Although several women benefit from the Orangers Irrigation System as users in association with their respective households, no women serve on the users committee and participate in the governance of the system. The evaluation team recommends that gender issues be incorporated into the design of future projects in a manner that promotes the involvement of women at higher levels of decision making and management. B. Citing of Sponsorship Paragraph (g) of the Program Description in the cooperative agreement, “Method of Educating Customers,” states that PADF and its partners will prominently display identification signs at its project sites and in its offices citing USDA sponsorship. The evaluation team observed one such sign at Ecole Nationale de Evelina Levy, which erroneously cited Citibank as the sole funding source, while CCMB identification signs did not indicate a funding source. The team recommends that per the conditions of the CREER cooperative agreement, PADF identify USDA as the source of funding at each project site, as well as modify the identification signs at Ecole Nationale de Evelina Levy and the CCMB Road. PADF should also ensure that the citing of sponsorship is continued as standard practice for all future USG-funded projects.