CORAD DRP PROGRAM Final Evaluation Report Care International World Vision Africare CRS Submitted by: Augustine Wandera May 1, 2006 ACKNOWLEDGEMENT An evaluation effort of this nature could not have been successfully completed without the coordinated efforts of many individuals. The evaluation team wishes to thank all those who facilitated every step of the process - from developing the scope of work to development of survey instruments for the quantitative evaluation survey - providing funding for fieldwork, and assistance in interpreting evaluation findings to facilitate the preparation of this report. AFRICARE, CARE, CRS and World Vision funded the entire evaluation from their DRP resources and we are particularly grateful to USAID/FFP for funding the DRP program. The management of the CORAD Consortium1 members are greatly acknowledged for their cooperation and support during the entire final evaluation process. That this exercise has been successfully completed is profoundly due to their consistence and vision. We owe a special note of gratitude to the CORAD M&E Coordination Unit managed by World Vision for coordinating this evaluation process. We are specifically grateful for the technical guidance and support received from Albert Siminyu (former CORAD M&E Coordinator). Members of the CORAD M&E working group are equally acknowledged for their support during the development of survey instruments for the quantitative component of this evaluation. Their keen analytical insights, thoughtful critiques, and sustained support were invaluable. The fieldwork for this evaluation was greatly facilitated by DRP program managers of the various CORAD members. Marion Almeida (AFRICARE), Finbar Sweeney (CARE), Oscar Maroto (CRS) and Dr. Tom Roberts (WV) are greatly commended for their excellent advance planning and logistical support during the fieldwork phase of this evaluation process. We also want to acknowledge, with special thanks, the entire data collection team – supervisors, enumerators and drivers-for their diligent, accurate, and focused work in the particularly harsh sunny conditions. In particular, we thank supervisors Veronica Smith (AFRICARE), Gibril Bangura (CARE), Dr. Joseph (CRS) and Fred Goba (WV) for their dedication and commitment to collecting good quality data. In a special way we thank Priti Biswas who carried out the qualitative assessment of the DRP, which greatly informed the quantitative data. Special thanks to Myles Harrison (World Vision) for his support and coordination work with all the DRP members, his efforts made working from a distance a lot easily. Finally, our acknowledgement would be incomplete without sincere thanks to DRP beneficiaries who warmly welcomed the evaluation team and generously devoted their valuable time to share their perspectives on DRP activities and benefits. 1 The CORAD Consortium consists of CARE, CRS and WV with AFRICARE as an associate member. ii ACRYNOMS AIDS Acquired Immuno-Deficiency Syndrome ADS USAID Automated Directive System ARI Acute Respiratory Infection CARE CARE (International NGO) CCU Coordination and Compliance Unit CORAD Consortium for Rehabilitation and Development CRS Catholic Relief Services (International NGO) DAP Development Assistance Program DHMT District Health Medical Team DRP Developmental Relief Program EPI Extended Program of Immunization FANTA Food and Nutrition Technical Assistance Project FFAg Food for Agriculture FFP Office of Food for Peace (DCHA) FFW Food for Work FY Fiscal Year GoSL Government of Sierra Leone HA Hectares HIV Human Immuno-Deficiency Virus HQ Head Quarter IDPs Internally Displaced Persons IITA International Institute of Tropical Agriculture IEBF Immediate and Exclusive Breast Feeding IPTT Indicator Performance Tracking Table IR Intermediate Result IVS Inland Valley Swamp LINKS Promoting Linkages for Livelihood Security and Economic Development LOA Life of Activity M&E Monitoring and Evaluation MAFS Ministry of Agriculture and Food Security MCHP Maternal and Child Health Posts MOHS Ministry of Health and Sanitation MT Metric Ton ORS Oral Re-hydration Solution PHU Peripheral Health Unit PL 480 Public Law 480 PPS Probability Propositional to Size SO Strategic Objective SSS Salt Sugar Solution SPSS Statistical Package for Social Scientists STI Sexually Transmitted Infection TBA Traditional Birth Attendant TOT Trainer of Trainer USAID United States Agency for International Development USG United States Government VDC Village Development Committee VGF Vulnerable Group Feeding VHW Village Health Worker WVSL World Vision Sierra Leone WVUS World Vision, USA iii Table of Contents ACKNOWLEDGEMENT ..................................................................................................................................II ACRYNOMS...................................................................................................................................................... III TABLE OF CONTENTS ...................................................................................................................................IV LIST OF TABLES............................................................................................................................................... V EXECUTIVE SUMMARY .................................................................................................................................. 1 A. INTRODUCTION AND BACKGROUND INFORMATION .................................................................... 5 A1. PROBLEM STATEMENT ......................................................................................................................... 5 A2. PROGRAM OBJECTIVES......................................................................................................................... 5 A3. PURPOSE OF EVALUATION.................................................................................................................... 6 A4. SCOPE OF WORK .................................................................................................................................. 6 B. EVALUATION METHODOLOGY.............................................................................................................. 6 B1. THE QUANTITATIVE SURVEY ............................................................................................................... 7 B2. THE QUALITATIVE ASSESSMENT.......................................................................................................... 8 C. FINDINGS....................................................................................................................................................... 9 C1. OBJECTIVE ONE: THE HEALTH OBJECTIVE......................................................................................... 10 IR 1.1 Access to Health Services................................................................................................................. 12 IR 1.2 Improved Health and Nutrition Practices........................................................................................ 14 IR 1.3 Village Level Capacity Building for Health ..................................................................................... 18 C2. OBJECTIVE TWO: THE AGRICULTURE OBJECTIVE .............................................................................. 20 IR 2.1 Farm Productivity ............................................................................................................................ 21 IR 2.2 Reducing Pre and Post Harvest Losses............................................................................................ 24 IR 2.3 Re-establishing Market Linkages..................................................................................................... 26 IR 2.4 Village-Level Capacity Building for Agriculture ............................................................................. 27 IR 2.5 Decision-Making and Governance in Agriculture ........................................................................... 28 IR 2.6 Maintain a Safety Net....................................................................................................................... 29 D. PROGRAM DESIGN................................................................................................................................... 31 E. PROGRAM COORDINATION AND IMPLEMENTATION.................................................................. 32 F. CONCLUSION AND RECOMMENDATIONS.......................................................................................... 35 F1. CONCLUSIONS ............................................................................................................................................ 35 F2. RECOMMENDATIONS .................................................................................................................................. 36 G. APPENDICES................................................................................................................................................ 38 G1. DRP FINAL EVALUATION SCOPE OF WORK ....................................................................................... 38 G2. DRP M&E SYSTEM SETUP................................................................................................................. 39 iv v List of Tables TABLE 1: LIST OF VILLAGES VISITED DURING QUALITATIVE ASSESSMENT............................................................... 9 TABLE 2: INDICATOR ACHIEVEMENT CATEGORIZATION ....................................................................................... 10 TABLE 3: IR 1.1 ACCESS TO HEALTH SERVICES .................................................................................................... 13 TABLE 4: IR 1.2 HOUSEHOLDS ACQUIRE UNDERSTANDING AND SKILLS RELATED TO IMPROVED HEALTH ............. 15 TABLE 5: IR 1.3 VILLAGE LEVEL CAPACITY BUILDING FOR HEALTH...................................................................... 19 TABLE 6: OBJECTIVE TWO: 37,400 RURAL FOOD INSECURE HOUSEHOLDS HAVE IMPROVED SUPPLY AND ACCESS TO FOOD OVER THE THREE YEAR LIFE OF THE PROGRAM ................................................................................... 20 TABLE 7: IR 2.1 FARM PRODUCTIVITY .................................................................................................................. 22 TABLE 8: IR 2.2 REDUCING PRE AND POST HARVEST LOSSES ................................................................................. 24 TABLE 9: IR 2.3 RE-ESTABLISHING MARKET LINKAGES......................................................................................... 26 TABLE 10: IR 2.4 VILLAGE LEVEL CAPACITY BUILDING FOR AGRICULTURE.......................................................... 27 TABLE 11: IR 2.5 DECISION MAKING AND GOVERNANCE IN AGRICULTURE ........................................................... 29 TABLE 12: IR 2.6 MAINTAIN A SAFETY NET .......................................................................................................... 30 EXECUTIVE SUMMARY This report presents the results of the final evaluation of the Developmental Relief Program (DRP), which is being managed by the Consortium for Relief and Development (CORAD) under agreement number FFP-A-00-04-00020-00. This grant was awarded to CORAD by USAID/FFP in March 2004. Its Life of Activity (LOA) was estimated to last 3 years from March 2004 – March 2007. The total amount of funding obligated for this grant is $24,154,809. Of this funding $6,079,381 is 202(e) cash, $5,987,626 is monetization proceeds and $2,731,608 ITSH. The goal of this program was to support Sierra Leone’s recovery from war by on restoring livelihoods for rural households in 30 focal chiefdoms in Bonthe, Kailahun, Koinadugu, Kono and Tonkolili districts.2 To achieve this goal, the DRP program will aim to: improve the health status of 15,800 rural food insecure households (Program Objective One) and; improve supply and access to food for more than 37,000 beneficiaries over its LOA (Program Objective Two). To achieve these objectives, the DRP implemented a series of community￾focused interventions aimed at achieving 4 critical results: (1) strengthening the capacity of existing government Primary Health Care (PHC) systems to improve beneficiary access to PHC services; (2) increasing household knowledge related to improved health and nutrition practices; (3) building the capacity of village-based health/nutrition groups to address health issues and capitalize on opportunities to improve the health status of their community members and; (4) linking strengthened community health/nutrition groups to decision-makers existing at chiefdom and district levels to leverage resources to implement health-related activities in their communities. To increase food supply and access for beneficiaries, the DRP was designed to reestablish livelihoods and do so in ways that will facilitate the transition of target beneficiaries from relief to development. To achieve this result, the program implemented household-focused interventions that focused on six critical areas: (1) maintaining a safety net to address the immediate food and housing needs of target beneficiaries; (2) restoring productivity of inland valley lowland rice farms, tree farms and upland farms; (3) rebuilding storage and processing capacities; (4) reestablishing access to local markets through rehabilitation of farm-to-market roads and market facilities; (5) enhancing the capacity of village-based agricultural groups to address agricultural problems capitalizing on existing opportunities and; (6) linking these strengthened agricultural groups to government institutions existing at chiefdom and district levels to increase their participation decision-making and policy formulation processes. The statement of work called for the evaluation team to focus on the major achievements and impacts of the DRP and identify the major lessons learned. It also required that the evaluation provides programming guidance to CORAD with respect to the priorities of the remaining 12 months of the program, and inform the design process of a follow-on multiyear assistance program (MYAP). This report, therefore, outlines key DRP achievements, discusses their specific impacts and then makes specific recommendations, drawn from the lessons learned from this program. 2 These districts were the most badly affected by the decade-long war and therefore had the highest number of chronic food insecure people during the DRP design stage. 1 A. Major DRP Achievements and Impact A-1: Health Objective Generally, the DRP has contributed towards improving beneficiary access to comprehensive maternal and child care services primarily through: (1) training village health workers; (2) strengthening the capacity of PHUs to provide health services and; (3) organizing village health outreach sessions. During the last two years the DRP has extended training to more than 631 village health workers (VHWs) and more than 1,081 traditional birth attendants (TBAs) who remain the nearest source of health advice and information for participating communities. These VHWs and TBAs received training on comprehensive basic health, sanitation and referral services. Specifically, the training focused on the importance of delivering with a skilled birth attendant and early referral in case of any sign of obstetric distress; malaria prevention and treatment, diarrhea prevention and management (especially among under five); basic community hygiene and sanitation; early referrals (especially for malaria and ARI, and diarrhea among under five). The training of TBAs and VHWs has been effective in delivering basic health and sanitation messages, with a specific focus on promoting safe deliveries and participation of under five in growth monitoring and promotion programs. Final evaluation results indicate that during the past two years of implementing DRP interventions, the proportion of births conducted by a skilled health worker increased by more than 70% (from 20.8 % at baseline to 92%). On this activity, the DRP has already exceeded its LOA target by more than 100%. On promoting exclusive breastfeeding among participating communities, the DRP has had remarkable success. The program has significantly contributed towards the 25% increase in the proportion of children under six months who are exclusively breastfed among target communities. This performance further demonstrates the effectiveness of behavioral change strategies employed by the DRP. Furthermore, partially because of the effectiveness of these trained community-based health workers, the DRP has remarkably contributed towards the increase in household knowledge related to basic health concepts, especially malaria prevention, diarrhea prevention and management as well as cough management. Final evaluation results indicate that during the past 24 months of implementing DRP activities, the proportion of mothers/caregivers who can explain basic health concepts has tremendously increased. For instance, the percentage of respondents who can identify “mosquito bites” as the cause of malaria has increased from 47% to more than 85%. Similarly, during the same period, the prevalence levels of cough and diarrhea have decreased greatly (from 43% to 35% for cough and from 41% to 25% for diarrhea). In addition, over the same period, DRP participants have demonstrated an increased tendency to seek treatment from health facilities, for diarrhea, ARI and malaria occurring among under five. For instance, the percentage of mothers/caregivers who, sought treatment for their child, during cough episode increased by more than 38% (from 42% to 80%). Similarly, the proportion of mothers/caregivers who, sought treatment for their child during malaria episode increased by 20% (from 56% at baseline to 76%). Finally, 76% of respondents whose child had suffered from diarrhea in the last two weeks prior to the survey sought treatment for their child diarrhea compared to 49% at baseline. In addition, the immunization outreach activities organized and supported by the DRP over the last two years have been very successful. These outreach activities have contributed towards a 17% increase in full immunization coverage rates. Final evaluation results indicate 2 that over the last two years of implementing DRP interventions, the proportion of children who are fully immunized by their first birth day has increased from 23% to 39%. This remarkable achievement can be comfortably attributed (at least in part) to the DRP program which in FY 2005 alone assisted GoSL to conduct immunization outreach sessions through which more than 10,000 children were immunized and 7,000 mothers received antenatal services. Lastly, under the health objective, the DRP has initiated the process of building capacity for community-based health/nutrition groups among target communities. This process has primarily been initiated through reestablishing health subcommittees of village development committee (VDCs) through out the DRP target area. In addition, the DRP has conducted training for all VDCs and community-based groups in basic strategic planning and management. By the end of FY 2005, the DRP reported that 35% of all support health/nutrition groups had developed action plans that outlined their needs/priorities and opportunities for meeting these needs. It is also important to note that through this program CORAD members have made significant strides towards reestablishing functional linkages between these community-based groups and decision-makers existing at chiefdom and district headquarters. To achieve this result, the DRP has organized /facilitated an average of three meetings between representatives of these groups and government departments at district and chiefdom levels. During these meetings, community-based groups got the opportunity to share their needs and priorities with decision-makers who in turn sensitized them about government programs and priorities. A-2: Agriculture Objective Under this objective, the DRP has contributed towards increasing beneficiary food security through improved access to and availability of food supply. This achievement has mostly been realized through increased agricultural production and productivity, reduced post￾harvest losses and increased farmer access to markets. To achieve increased food supply, the DRP supported household-focused interventions that directly contributed towards increased agricultural production and productivity. Specifically, the DRP assisted food insecure households to reestablishing production on 9,954 hectares of farmland that was planted to basic food staples (cassava, rice, groundnuts and vegetables). In addition the program extended training to more than 20,000 farmers. These farmers were organized into groups (farmer field schools – FFS averaging 25-30 farmers each) and trained on modern agronomic practices. Information gathered through the DRP internal M&E system indicated that by the end of FY 2005 more than 70% of FFS participants were practicing improved agronomic practices and had registered more than 40% increases in the productivity levels of basic food staples. The most outstanding achievement was registered among FFS’ supported by World Vision whose members (by the end of FY 2005) had doubled their rice productivity by adopting planting improved upland rice seeds (especially the NERICA series). The exciting thing about the DRP program is that increased food production has resulted into plausible improvements in food consumption (both in quantity and quality). For instance, 3 over the last two years DRP communities have registered a 23% increase (from 1.7 to 2.1) in the average daily number of meals consumed per household and 34% increase in dietary diversification (measured by a proxy of the average number of food groups consumed by participating households). The DRP has contributed towards improving access by participating households towards markets through rehabilitation of more than 418 kilometers of farm-to-market roads, five market facilities and 17 weekly markets. Lastly, under the agricultural objective, the DRP has initiated the process of building capacity for community-based agricultural groups among target communities. In addition, the DRP has organized/supported training for all VDCs and community-based groups in basic strategic planning and management. 4 A. INTRODUCTION AND BACKGROUND INFORMATION This report presents results of the final evaluation of the Developmental Relief Program (DRP) implemented by CORAD3 in Sierra Leone. The DRP grant was awarded to CORAD in March 2004 with an estimated Life of Activity (LOA) period of approximately three years. Currently the DRP is beginning its third year of implementation. It is expected in linear terms that the program is about two thirds of life span and therefore that about 67% of its planned activities and/or achievements should have been implemented/ achieved to date. The DRP is funded by USAID/FFP through PL 480 Title II resources to the tune of US $24,154,809. Approximately 25% of this funding is 202(e). The rest comes from Monetization proceeds ($5,987,626) and Internal Transportation, Shipping and Handling￾ITSH ($2,731,608). Additional resources ($409,200) were leveraged from Non-Federal contributions made by CORAD members. A1. Problem Statement4 Sierra Leone has been embroiled in civil war for over a decade. The conflict has claimed at least 20,000 lives and forced almost half of Sierra Leone’s 5 million people to flee their homes. There is light beginning to show at the end of the tunnel, however. The destructive eleven-year civil war has officially ended, and all parts of the country are accessible again. Peaceful elections were held in 2002. All of the approximately 216,000 registered internally displaced persons have been resettled, and 194,500 refugees representing 55 % to 60% of the total known to have left the country have returned. Rural households returning to their communities have been overwhelmed, however, by the situation they now face. Most, especially those returning to the north and the east, have discovered that abandoned farm plots, tree farms, and developed inland valley lowland rice farms have turned to bush. They have found their houses and other structures looted or destroyed and productive assets such as rice mills, water wells, rice stores, oil processing plants, markets and other economic facilities vandalized or destroyed. In addition, government rural support services have continued to function, but have far to go to reach the level of services provided eleven years ago before the war began. Returning households with limited resources have to decide whether to invest their labor in rebuilding their homes, reestablishing their rice fields or brushing their tree farms. Relief programs implemented over the last few years have helped households to begin to recover their livelihoods. Much remains to be done, however, to restore livelihoods to where they were before the war while continuing to provide a safety net for the most vulnerable. A2. Program Objectives The DRP is implemented in 29 focal chiefdoms of Bonthe, Kailahun, Koinadugu, Kono and Tonkolili districts, which by program design and at the start of implementing hosted the highest number of chronically food insecure people in the country. The high vulnerability levels of food insecurity were caused by the decade-long civil conflict that devastated the country’s social and economic systems. The DRP was designed to support Sierra Leone’s recovery from the war by focusing on restoring livelihoods for rural households in 29 chiefdoms in some of the most badly affected districts in the country. Over its three year life, the DRP was designed to reestablish livelihoods - and do so in ways that will facilitate a 4 CORAD FY04 –FY06 DRP Proposal – Problem Analysis and proposed interventions 5 transition from relief to development programs. To achieve this objective, CORAD members are implementing household-focused long-term food security interventions that encompass: access; availability; and utilization, but which retain a targeted food assistance strategy5 . These initiatives, will promote social and economic conditions that enable individuals to gain access to food, either through producing it themselves, or by earning the income to purchase it. To achieve the DRP goal, two program objectives are being implemented. First, the DRP will improve food availability and access by target beneficiaries (Program Objective One). Secondly, the health status of target beneficiaries will be improved, through improved access to maternal and child health (MCH) services and improved knowledge related to basic prevention and management of common illnesses, nutrition and dietary diversification (Program Objective Two). A3. Purpose of Evaluation The purpose of this evaluation is to provide a final, external and independent assessment of program achievements and impacts to date. In addition, the evaluation was commissioned to assess the relevance and sustainability of DRP interventions and outputs and to document lessons learned which will be used to guide the design of the follow-on multiyear assistance program6 . A4. Scope of Work7 The Scope of Work (SOW) called for a final evaluation of Developmental Relief Program (DRP) implemented in Sierra Leone by the Consortium for Relief and Development (CORAD). This evaluation would provide: (1) an independent and external assessment of the relevance, effectiveness, efficiency, impact and sustainability of DRP operations to date and; (2) provide guidance to CORAD on strategic areas of focus for the follow-on Multi-Year Assistance Program (MYAP) that will be designed during January – March 2006. Specifically the SOW required the evaluation team to provide answers to three key questions, and these are; (a) What has the DRP achieved? (b) What, if anything, has the DRP failed to achieve? (c) What are the key “lessons learned” that CORAD and FFP can build upon in the design of their future programs? B. EVALUATION METHODOLOGY This evaluation exercise had two interdependent components. To begin with, a quantitative survey was conducted to establish the quantitative impact created by DRP interventions over a two year period of implementation. Thereafter, a qualitative assessment was commissioned to assess the relevance, effectiveness, impact and sustainability of DRP interventions and document key lessons learned. These lessons learned will be used to improve the quality of management of DRP operations during its last year of implementation as well as inform the design process of a follow-on MYAP. 5 This targeted food assistance strategy is part of the DRP’s safety-net interventions designed to address the immediate food and housing needs of vulnerable households and people in institutions (e.g., IDPs, malnourished children, pregnant and lactating mothers, etc). 6 For this reason, this evaluation was conducted 1 year prior to the end of the DRP 7 The detailed SOW is attached in Appendix G1 6 B1. The Quantitative Survey Quantitative data was collected in January 2006 by replicating a baseline questionnaire that had been administered in May 2004. This multi-indicator survey was conducted to assess the status of key program impact-level indicators for which baseline data exists. A total of 2,640 respondents were selected to participate in this survey using a standard formula to detect change over time footnoted below (see FANTA’s Sampling Guide by Robert Magnani)8 . Additional parameters that went into the calculation included: (1) the probability of detecting a change that is true (alpha, set at 0.95); (2) the power to detect a change if it has really taken place (beta, set at 0.90); (3) the design effect to compensate for the use of a cluster sampling method (D, set at 2); and (4) a “targeting correction factor” of 20%. The latter was based on the assumption that approximately 80% of visited households would qualify to participate in the survey.9 The resulting sample size was increased by 20%10 to account for no attrition and non-response. Using the formula presented in the footnote below11 we concluded that each DRP partner needed a sample of 543. In order to diminish the design effect of the cluster sampling method, (and because of the high projected non-response rate), each CORAD member aimed at sampling a large number clusters. The cluster design factorial was therefore set at 30X 22 (thirty clusters, or localities, with 22 households each), yielding a total of 660 households per CORAD member. The sampling process for this survey had two major stages. First, the Probability Proportional to Size (PPS)12 technique was used to determine the required number of clusters per target district. Thereafter, the PPS was used to determine the required number of clusters to be selected from the selected chiefdoms and sections. A total of 120 clusters with 2,640 households were ultimately sampled. Secondly, a two stage sampling design was used to collect data. Primary clusters were settlements, selected using the PPS. Secondary units constituted of households randomly 8 See Magnani, 2000: Sampling design 9 A minimum criteria was set for households to qualify for the survey- this was that at least 6 person were cared for at the household. During the questionnaire pre-test process, it was estimated that approximately 80% of target households would satisfy this condition. 10 The non-response/attrition correction factor was set at 20% because the survey was undertaken during the harvest time for upland crops and second season inland valley swamps- a very busy time for the targeted farm-families. During this season, the majority of adults in a given household spend their entire day (07:00-05:00GMT) in fields far a way from home. 11 Sample Size formula used to detect change over time: n = D [(Zα + Zβ )2 * (sd1 + sd2 ) / (X2 – X1) 2 ] 2 2 Where: n = required minimum sample size for each survey or comparison group; D = design effect for cluster designs [survey assumed an implicit value of D=2]; X1 = the estimated level of an indicator at the time of the first survey or for control area X2 = the expected level of the indicator either at a future date, so that (X2 - X1), is the size of change that is targeted; [survey used a value of X2=105.2; LINKS targets a 25% increase (over baseline) in the volume of upland rice marketed by target beneficiaries] sd1 and sd2 = expected standard deviations for the indicators for the respective survey rounds Zα = the Z-score corresponding to the degree of confidence desired in order to conclude that a change of the size (X2 - X1)is not due to chance (a-statistical significance level). [survey assumed a value of Zα = 1.65corresponding to a 95% degree of confidence (one-tailed test)] Zβ = the Z-score corresponding to the degree of confidence desired in order to detect with certainty a change of the size (X2 - X1), if such a change has effectively taken place (b-statistical power)[survey used a Zβ=1.282; corresponding to an 90% degree of confidence, that if the desired change (X2 - X1)occurs, it will be detected withy certainty. 12 The PPS means that larger clusters are a given a greater chance of selection than smaller clusters. Before applying the PPS, CORAD partners developed a sampling frame of clusters with measures of size, which were used, select clusters from which respondent households were drawn. 7 Due to the unreliability of census population data for individual households, clusters were selected randomly from a list that included all villages in all sections selected using the PPS technique. Within clusters, households were chosen using a segmentation method. Each selected cluster was divided into smaller segments of approximately equal size, averaging 22 households each. Prior to data collection, the baseline survey questionnaire was revised by the CORAD M&E Coordination Unit (with significant assistance from partner agriculture and health specialists). Key informant interviews and focus group discussions were also conducted, to help triangulate and interpret quantitative survey findings. Due to the nature of the questions involved in the survey, the biggest percentage of survey questions was directed to the household-head and/or spouse(s). Questions regarding household income and on-farm agriculture were directed to the on-farm agriculturist of the household (usually the man). The final demonstration section of the questionnaire (related to household consumption) was directed to the same woman interviewed previously. Evaluation survey data collected was captured using Epi Info data entry module. Data entry screens were designed to support both question validity range checks and questionnaire skips. After data entry, the data was cleaned based on (valid range checks, consistence/logical test and completeness). Data analysis was carried out using SPSS for (frequency distributions and cross tabulations) and Epi Nut Anthropometry for the nutrition assessment. B2. The Qualitative Assessment The qualitative assessment was designed to collect in-depth information regarding the impacts of DRP interventions on the targeted population of poor and vulnerable households in rural Sierra Leone. This assessment focused not only on the targeted outputs of the last two years, but also on additional factors that hinder the progress and well-being of these vulnerable communities. The assessment offered insights into the village-level reality of war-torn poor households, community empowerment dynamics, food insecurity, and local coping mechanisms. The categories of information that were collected included, at the project beneficiary level, the contextual information on the political, social, economic, and natural environment in which households make decisions, the process, reasons, motivation behind participation in the program by different individuals and households, and the impacts of the program on health and food security for participating households. For other stakeholders, the information included the process, strengths, limitations, lessons learnt and future directions for the program. Sample of Villages Qualitative interviewing was carried out in ten project villages within four districts of Tonkolili, Koinadugu, Kailahun and Kono. The sample was designed to cover a range of factors that might influence project success, including the geographic location, ethnic composition, the number of households in the villages, and the relative success in the adoption of project interventions. There were also differences in road access and remoteness among the villages. The ten villages were purposively selected from the number of villages 8 that were surveyed during the quantitative survey. At least two villages were selected from each CORAD partners implementing areas. Table 1: List of villages visited during qualitative assessment Districts Village (implementing partner) Tonkolili Rogbesseh (CARE) Matham (CARE) Koinadugu Helemakona (CARE) Kamaron (CRS) Badala (CRS) Kailahun Manowulo (CRS) Darujanction (Africare) Banda Jume (Africare) Kono Manjama (World Vision) Condama (World Vision) Qualitative Assessment Tools The overall methodology adopted a participatory approach designed to elicit local reflections on the direct and indirect impacts of the project. In doing so, a number of PRA tools were employed including focus group discussions, key informant interviews, and in-depth interviews with different stakeholders. In addition, seasonal calendars, well-being analysis, and transect walks were also applied in each village. The transect walks identified the spatial organization of the village and its relationship to the fields, water bodies, and other resources as well as Farmer Field School (FFS) demonstration plots, community health centres, growth monitoring centers, water wells, reestablished village markets, and other infrastructural features. During the focus group discussions, seasonal calendars were constructed to identify temporal variation in food availability and accessibility, income, and labor demands, disease incidence, and other factors that significantly affected the community. The well-being analysis provided a critical village profile on socio-economic stratification and general characteristics about the background of program beneficiaries. Focus group interviews with women and men were carried out separately and in mixed groups to investigate the level of participation, representation, and intended and unintended impacts of the project. In all the visited villages, the receptiveness of the local population was very high and participation was widespread. Careful analysis of the interview texts and the observational data was oriented towards persistent thematic trends and patterns in response. C. FINDINGS Assessing Progress towards Targets and Objectives In order to assess the progress that the DRP has achieved to date the evaluation team looked at a number of data sources, these included DRP Fiscal Year Results Reports, DRP Indicator 9 Performance Tracking Table (IPTT) results, DRP Evaluation Survey data and Qualitative Assessment Report. Analytical Framework Since this evaluation was carried out before actual end of DRP program, the evaluation team had to come up with some objective way of assessing achievement, while at the same time taking care of fact that the DRP had some more than 12 month to the end. Naturally, a three year program would be expected to have achieved at least two thirds of its LOA targets after two years of program implementation. This thinking formed the basis for developing cut off points that were used to determine the degree of achievement of the Key Program Impact Indicators. Three achievement categories were therefore developed i.e. (1) less or equal to 61% of LOA targets, (2) 62% - 72%13 of LOA targets, and (3) greater or equal to 73% of LOA targets. The achievement categorizations used in the assessment are presented in the table 2 below. Table 2: Indicator Achievement Categorization Percentage categorization Level of achievement less or equal to 61% of LOA target LOA Target missed 62% - 72% of LOA target LOA Target achieved greater or equal to 73% of LOA target LOA Target exceeded C1. Objective One: The Health Objective The goal of this objective is to have 15,800 rural food insecure households achieve improved health status over the three year life of the DRP. Although poor health services affect everyone in rural communities, the DRP focuses mainly on nutritionally vulnerable groups, pregnant and lactating women and young children. This objective is supported by three intermediate results i.e. (1) IR 1.1 Access to Health Services, (2) IR 1.2 Improved Health and Nutrition Practices and (3) IR 1.3 Village Level Capacity Building for Health14. Objective One: Key Program Impact Indicator Results Progress under this objective is measured by three indicators i.e. Percentage of children under five years who are underweight (weight for age), Percentage of infants under six months who are exclusively breastfed and lastly the percentage of safe deliveries (both mother and child healthy) over the last six months. Comparison of these indicators against their LOA targets is presented in table 3 below. Table 3: Objective One: 15,800 rural food insecure households achieve improved health status over the three year Indicator Baseline Final evaluation results LOA target Achieved to date vs LOA target Comments % of children under five years who are underweight (weight for age) 26.3% 26.9% 21.9% LOA target missed. % of infants under six months who are breastfed exclusively 0% 24.8% 20 124% LOA target exceeded 13 This interval is obtained by creating a ±5% interval around 66.7% (67%) since the USAID Automated Directive System (ADS) 203 categories a target as achieved if 95% of the target is achieved. 14 During the design stage, Objective one was designed with four IRs, however during the implementation phase, it was felt that IR 1.3 and IR 1.4 were very similar. Therefore all activities of these two IRs were folded into IR1.3 10 Indicator Baseline Final evaluation results LOA target Achieved to date vs Comments LOA target % of safe deliveries (delivered by a trained birth attendant) over the last six months 20.8% 92% 40.8 225.5% LOA target exceeded Indicator 1: The percentage of children under five years who are underweight Final evaluation survey indicated that children underweight were 26.9%, with a 95% confidence interval of (24.8 - 29.1). Since this confidence interval includes 26.3%, this, suggests that there is no statistical difference between that baseline and the final evaluation survey results. Therefore the prevalence of malnutrition among the under fives seems to be the same as at baseline. Although baseline survey results indicated that the percentage of children under five years who are underweight was 26.3%, operational data from one of CORAD member organization indicates that the baseline was underestimated15. Furthermore, data from numerous other sources, including the WFP and the MOHS, suggest that the incidence of underweight children was actually increasing in most of the country during this same time frame, suggesting that DRP’s activities may well have prevented similar increases in the operational areas. The evaluation team was also informed that un-published results from a recent nutrition survey by the MOHS are indicative of the percentage of children underweight in DRP districts being above 33%16. Underweight, as a measure of malnutrition, has a limitation insofar as it captures past stunting (children with low weight for age due to past acute malnutrition) as well as ongoing malnutrition. Children stunted from the period before the DRP will continue to be registered as underweight even though they are not currently malnourished. This factor is important since DRP has been implemented for two years, but is weighing children up to five years of age. Given the above facts, the evaluation team believes that the baseline for the children underweight was most likely underestimated, and in which case, the DRP has impacted on this result area, first by being able to prevent an increase in malnutrition compared to the other parts of the country and secondly by increasing the number of children under five who are participating in the growth monitoring program in the DRP targeted communities. Indicator 2: The percentage of infants under six months who are exclusively breastfed The percentage of infants under six months who were reported during the evaluation survey as having been exclusively breastfed was 24.7%. This is big raise from the baseline figure of 0%. Compared to the LOA target of 20%, the program exceeded its target by 24%. Given that behavioral change usually takes a while to occur, this is remarkable achievement for the program in such a short time. Indicator 3: The percentage of mothers having safe deliveries Final evaluation survey data in table 3 above indicates that the percentage of mothers who responded as having been delivered by a trained birth attendant was 92%. Of the 92%, 23.7% were assisted by a health worker while 68.3% reported that they were assisted by traditional birth attendants (TBA) during delivery. This implies that about 7 in every 10 mothers were attended to by a TBA. The percentage of mothers attended to by TBAs in the program area is 15 For example in October 2005, under weight was registered at 39.8% in the operational area based on 1,800 children who were weighed. The evaluation team was informed that this figure has been consistent over time. 16 2005 VAM sponsored by WFP. 11 fairly close the national statistics of about 75%. Compared to the LOA target of 40.8%, the program exceeded its LOA target by 125.5%, %. This shows that the training of traditional birth attendants did in fact lead to a higher percentage of women delivering with an attendant trained in appropriate birthing techniques and referral, the first step on the path toward reducing maternal morbidity and mortality in rural Sierra Leone. Conclusion Based on indicators 2 and 3 in table 3 above, the program met the objective of rural food insecure households achieving improved health status over the three year of the project. Based on indicator 1 the program may have been able to prevent an increase in malnutrition compared to the other parts of the country. IR 1.1 Access to Health Services Under IR 1.1, DRP has over the past two years worked toward restoring the functional relationships between rural populations and basic health services by raising awareness about the availability of health services and facilitating, in some cases, the provision of services at the village level through outreach clinics. CORAD consortium members have worked directly with the communities in the Maternal and Child Health Program (MCHP) catchment areas to raise awareness regarding the availability of health services in health facilities. IR 1.1 activities are necessary to restore comprehensive basic health care services that were disrupted by the decade-long civil war. This IR is complemented by repair and construction activities under PHU in IR 2.3. The DRP has registered the following accomplishments in the past two years. IR 1.1 Accomplishments as Reported in Fiscal Year Results Reports17 The DRP trained 45 representatives of community based women associations and community health workers as trainers of trainers (TOT). The training centered on causes and prevention of malnutrition, benefits and recommended standards for immediate and exclusive breast￾feeding; and Ministry of Health and Sanitation (MOHS) recommended maternal and childcare practices. To ensure sustainability, the program partnered with MOHS, and the District Health Management Team (DHMT). MOHS provided the materials that were used by TOT beneficiaries to train others in their respective communities. To address the safety of maternal and infant care, 1,081 Traditional Birth Attendants (TBAs) were trained. The training of the TBAs covered assessment of high-risk symptoms for pregnant and lactating mothers, as well as recommended MOHS nutrition and early childhood development standards. In partnership with 19 Periphery Health Units (PHUs) in target communities, the program customized two MOHS training curricular that were used to deliver a series of TOT courses to community health workers. This intervention was geared to providing participants with effective skills to promote recommended MOHS maternal care, nutrition and early childhood development practices. DRP strengthened the capacity of existing health care systems, 631 village health workers (VHWs) (at least 1 from each DRP village) were trained on comprehensive basic health, sanitation and referral services. The training focused on the importance of delivering with a skilled health birth attendant and early referral in case of any sign of obstetric distress; 17 This is cumulative accomplishment over two fiscal years 12 malaria prevention and treatment; diarrhea prevention and management (especially among under fives); basic community hygiene and sanitation; early referrals (especially for malaria, acute respiratory infections-ARI, and diarrhea especially for under fives) The DRP conducted community clinic outreach sessions through which, more than 10,000 children where immunized and more than 7,000 mothers received ante-natal care services. To enhance the efficiency of DRP operations, outreach sessions were also used to provide basic training on household health and sanitation; nutrition and dietary diversification; food hygiene; as well as prevention and management of common communicable diseases. The DRP has promoted joint supervisory visits of public health facilities and VHWs, as a strategy towards enhancing the capacity of DHMTs to adequately plan and supervise public health care systems. The promotion resulted into 140 joint health supervisory visits being conducted. IR 1.1 Key Program Impact Indicator Results According to the IPTT, impact under IR 1.1 will be assessed by the contribution made by the DRP towards increasing the rate of full immunization for children under 12 months, and prevalence and care seeking for the three most common morbidity symptoms (diarrhea, fever, cough and acute respiratory infections - ARI) among target communities. The four indicators that measure progress under this IR are presented in table 4 below. Table 3: IR 1.1 Access to Health Services Indicator Baseline Final evaluation results LOA target Achieved to date vs LOA target Comments % of children under one year fully immunized before first birthday 22.6% 38.7% 37.6% 103% LOA target exceeded % of target beneficiaries with a cough in the last two weeks who sought consultation from a health facility18 46.4% 80.1% 66.9% 120.5% LOA target exceeded % of target beneficiaries with a fever in the last two weeks who sought consultation from a health facility 51.5% 75.9% 71.5% 106.2% LOA target exceeded Percent of target beneficiaries with diarrhea in the last two weeks who sought consultation from a health facility 49.3% 76.1% 69.3% 109.8% LOA target exceeded Indicator 1: Percentage of children under one year fully immunized before first birthday Final evaluation survey results indicates that 38.7% of the children under 12 months in the households that were interviewed were fully immunized (i.e. had received BCG, DPT, measles, polio and vitamin A). In terms of individual vaccines, 93.7% reported as having received BCG, 66.3% had received Polio 3, DPT3 was reported by 65.0%, measles was at 54.8% and vitamin A at 50.2% as depicted diagrammatically in figure 1. 0 10 20 30 40 50 60 70 80 90 100 % Vaccine Figure 1: Final Evaluation Survey Data Results for Vaccination Survey 93.7 66.3 65 54.8 50.2 38.7 BCG POLIO3 DPT3 Measels Vitamin A Fully immunized 18 Health facility was defined to include (hospitals and clinics) 13 Comparison of final evaluation survey results with the LOA target of children under one year who are fully immunized shows that the program exceeded the LOA target by three percentage points. Indicator 2: Percentage of target beneficiaries with a cough in the last two weeks who sought consultation from a health facility Final evaluation survey results in table 4 above vividly show that this is another area were the program scored highly. The program exceeded the LOA target by 20.5%. Indicator 3: Percentage of target beneficiaries with a fever in the last two weeks who sought consultation from a health facility From the final evaluation survey results in table 4 above the program exceeded the LOA target by 6.2%. Indicator 4: Percentage of target beneficiaries with a diarrhea in the last two weeks who sought consultation from a health facility Final evaluation survey results in table 4 above shows that the program exceeded the LOA target by 9.8%. Conclusion All the four indicators under this intermediate result demonstrate that the program achieved and even exceeded its LOA targets. This is a tremendous accomplishment given the inadequate infrastructure in the target area and low health seeking behavior of the community at the baseline. The program should consolidate its earlier gains of exceeding the target of children under one year who are fully immunized by focusing more on improving vitamin A and measles coverage rates that seem to be below coverage rates of other vaccines. While the program exceeded its LOA targets on immunization, there is clearly room for improvement. The program may wish to focus attention on developing strategies to increase immunization coverage rates19. The behavioral change strategies that DRP employed both in immunization and health seeking by the community have proved to be effective in improving immunization coverage and treatment seeking behavior. The strategies employed by the DRP are worth documenting and replicating in future behavioral change interventions. IR 1.2 Improved Health and Nutrition Practices Under this IR, DRP program specifically, seeks to increase household knowledge related to: immediate and exclusive breastfeeding, effective weaning, rehabilitation of moderately malnourished children, importance of early referrals, diarrhea prevention and management, malaria prevention, HIV & STI prevention, family planning, value of diet diversification, hygiene and water safety, and food processing/food hygiene, safety and storage. To accelerate progress, the program is deliberately promoting monthly growth monitoring for 19 The evaluation team was informed by some CORAD member that government has recognized this as a nationwide problem and has planned two national events to try and address this issue 14 children less than five years. To achieve this result the program has supported both growth monitoring and training of VHWs and TBAs. IR 1.2 Accomplishments as Reported in Fiscal Year Results Reports DRP successfully organized over 30,231 weighings of children in the target chiefdoms. More than 1,400 mothers (with children under 5 years) were trained on infant nutrition and early childhood development. The DRP reestablished health committees for Village Development Committees (VDCs) for all the 375 villages covered by DRP interventions. All VDC members were sensitized about the priority interventions of the DRP (including focused household health and nutrition education). The DRP trained 1,712 VHWs and TBAs as a foundation and sustainability measure for the implementation of community rehabilitation of malnourished children. The DRP supported growth monitoring sessions through which more than 30,200 weighings were conducted. IR 1.2 Key Program Impact Indicator Results The two indicators that measure progress under this intermediate result are the percentage of children under five year participating in growth promotion program and the percentage of beneficiaries who are able to explain health related concepts. The performance of these indicators are presented in that table 5 below Table 4: IR 1.2 Households acquire understanding and skills related to improved health Indicator Baseline Final evaluation results LOA target Achieved to date vs LOA target Comments % of children under 5 years participating in growth promotion programs 33.8% 76.8% 83.8% 91.6% LOA target exceeded % of beneficiaries from targeted beneficiaries able to explain health-related concepts (prevention & mgt of diarrhea & malaria + cough management)20 47% 85.7% 70.5% 121.6% LOA target exceeded Indicator 1: percentage of children under five years participating in growth promotion programs The final evaluation survey results in table 5 above indicate that the percentage of children under five years participating in the growth promotion program was 76.8%. In terms of progress towards achieving the LOA target, this translates into 91.6%, thus indicating that the LOA target was exceeded. However, it should be noted that of the 76.8% who responded as participating in the growth promotion program, 51.4% had no ‘under five’ cards while 25.4% had cards. Although the under five cards are important, the reality is that health worker often charge for them despite official government policy that they should be free. It was thus deemed appropriate to include those people who said they do participate even if they did not have the cards. 20 Assessment of this indicator is based on knowledge of the cause of malaria since the way the Key Program Impact Indicator is worded; there is no direct question in the questionnaire that addresses it. 15 Indicator 2: percentage of beneficiaries from targeted beneficiaries able to explain health￾related concepts (prevention & management of diarrhea, malaria and cough)21 The percentage of beneficiaries who were able to explain the cause of malaria during the final evaluation survey was 85.7%. The achievement of this indicator towards LOA target was at 121.6% suggesting that the program exceeded the LOA target by 21.6%. Incidence of Cough, Diarrhea and Fever All the three diseases except fever registered a decline in incidence between the baseline survey and evaluation survey as indicated in figure 2. Specifically the incidence of cough in the last two weeks before the survey declined from 43% at baseline to 35.7% during the evaluation survey. The incidence of diarrhea in the past two weeks before the survey also declined from 41% and baseline to 24.7% during the evaluation survey. On the other hand, the incidence of fever in the past two weeks was stagnant at 46%. Management for Cough, Diarrhea and Fever In terms of disease management, the final evaluation shows that majority of the households sought treatment between day one and day two after the on set of the disease. This pattern is the same for the three diseases. These results demonstrate the extent to which the program has achieved in terms of improving health practices by the households in the target area. These results further confirm the effectiveness of the strategies/interventions of the DRP. The percentage of care-givers who reported giving children ORS and sugar-salt solution was 66.5 and 33.2 percent respectively, indicating that majority of the care-givers were knowledgeable about the use of ORS during diarrhea episode. The percentage of care-givers who reported using ORS increased from 28% during the baseline to 66.5% during the evaluation, which demonstrated the effectiveness of DRP’s behavioral change strategy. 0 5 10 15 20 25 30 35 40 45 50 % of Children Figure 2: Incidence of Cough, Diarrhea and Fever in Last 2 Weeks Baseline Evaluation Baseline 41 43 46 Evaluation 24.7 35.7 46.4 Diarrhea Cough Malaria 0 10 20 30 40 50 60 % Figure 3: Duration Before Seeking Treatment for Diarrhea, Cough and Malaria Same day Next day After 2 or more days Same day 31.9 28.8 38.3 Next day 54.5 53 50 After 2 or more days 13.6 18.2 11.8 Diarrhea Cough Malaria 21 Assessment of this indicator is based on knowledge of the cause of malaria since the way the Key Program Impact Indicator is worded; there is no direct question in the questionnaire that addresses it. 16 0 10 20 30 40 50 % 60 70 80 Figure 4: Percentage of HHs that Sought Treatment from Health Facility for Cough, Diarrhea and Fever Baseline Evaluation Base ine 49 42 56 Evaluation 76.1 80 75.9 Diarrhea Cough Malaria Treatment for Cough, Diarrhea and Fever Majority of households reported seeking for treatment for all the three diseases from the health facilities. Specifically, the percentage of households that sought for treatment from the health facilities were 80%, 76.1% and 75.9% for cough, diarrhea and fever respectively. These results show significant improvement from the baseline to final evaluation survey across all disease as indicated in figure 4. The primary reasons that the few households which did not use health facilities during the episode of a disease advanced were the cost of medication and distance to the health facilities. 0 10 20 30 40 50 60 % Figure 5: Factors for not visiting health facility during disease episode Distance Cost Doctors/nurses unable Other Distance 47.1 36.4 51.7 Cost 33.3 41.3 31.4 Doctors/nurses unable 2.3 4.3 3.7 Other 17.2 17.9 13.3 Diarrhea Cough Malaria Sources of Treatment for Diarrhea, Cough and Malaria Treatment seeking behavior by households during episodes of any of the three diseases has a similar pattern as depicted in figure 6 below. The clinics are the most frequently used health facilities followed by drug peddlers. There is a significant improvement in treatment seeking behavior compared to the baseline where on average, less than half the households reported using a health facility during any disease episode. This trend demonstrates significant increases in the percentage of care givers seeking treatment during disease episode. 17 0 10 20 30 40 50 60 70 80 % Figure 6: Source of Treatment for Diarrhea, Cough and Fever District Hospital Clinic Drug Shop or pharmacy TBA Traditional healer Blue Flag Volunteer Drug Peddler Other District Hospital 2.8 4.3 5 Clinic 74 9 75.7 70.9 Drug Shop or pharmacy 44.5 5.6 TBA 3.5 1.9 3 2 Traditional healer 0.9 0.6 0 5 Blue Flag Volunteer 1.6 0.3 0.4 Drug Peddler 12 12.6 14.3 O her 0.2 0.1 0.1 Diarrhea Cough Malaria Conclusion The incidence of cough and diarrhea has significantly declined from the baseline, although the incidence of fever has stagnated. There is a remarked improvement in treatment seeking behavior by the households compared to the baseline. More households are seeking for treatment outside the household and specifically in health facilities. Reasons for not attending health facilities during an episode of any of the three focus diseases were reported as distance to the health facility and the cost of the drug. Clinics are the most highly used health facilities; this indicates that the majority of the population is now using government health services, which is a substantial gain especially given the destruction of rural health infrastructure that resulted from the war. IR 1.3 Village Level Capacity Building for Health To achieve this result area, the DRP has provided technical assistance in the form of training to different groups and community based organizations, to enable them continue working together to address nutritional problems and capitalize on opportunities for improving health services and nutrition in the community. Building capacity for these groups and organizations is a critical component of the DRP’s sustainability strategy. The DRP has further strengthened functional relationship between target populations and basic health care service systems by establishing/strengthening Village Development Committees (VDCs) with health subcommittees. 18 IR 1.3 Accomplishments as Reported in Fiscal Year Results Reports The DRP supported to re-establish 375 VDCs – one in each village served. Each of these committees’ membership includes representatives for the youth, women and community health workers. During the sensitization process, participants received orientation and training associated with the principles of good governance, including representation, equity, transparency, and accountability. The DRP continued to strengthen a functional relationship between target populations and basic health care service systems by establishing/strengthening VDCs with health subcommittees. A series of basic business planning trainings were organized for the VDCs and other health related groups. A total of 466 groups (with approximately 7,000 members) benefited from these trainings. Many of these groups have developed action plans to address health-related problems and capitalize on opportunities existing in their communities. IR 1.3 Key Program Impact Indicator Results Progress achieved under this IR is measured by the percentage of health groups which have developed action plans that address health issues and the average number of meetings over the last six months of key representatives. Table 5: IR 1.3 Village level capacity building for health Indicator Baseline Evaluation LOA target Achieved to date vs LOA target Comments % of health groups that have developed action plans that address health issues 0% 35% 50% 70% Target achieved Average number of meetings over the last 6 months of key representatives 0% 3 5 60% Target missed Indicator 1: The percentage of health groups that have developed action plans that address health issues The target for the percentage of health groups that have developed action plans that address health issues was 70% of the LOA target. Therefore in quantitative terms, the LOA target of this indicator was achieved. Indicator 2: The average number of meetings over the last six months of key representatives The LOA target for the average number of meetings over the last six months of key representatives has been attained to a magnitude of 60%. Conclusion Overall, improved village level capacity building for health result area achieved least progress. The major achievement in health capacity building that featured strongly is the presence of trained TBAs in every village. 19 Least progress achieved under this governance IR may be explained in the sense that changes in governance are likely to take a little longer, but also compounded by the recent past of the country. In the words of one of the TBAs, “…before this program, I was anxious about each birth, I had to guess how long the labor was going to last, and what if something goes wrong, now I know how long the pain may last and where to go should there be any problem. I feel so much more confident.” [TBA in Kamaron Village] The program may want to think about better measures of assessing of capacity of committees rather than just developing action plans. Such measures should take care of quality of meetings, capacity for the committees to manage the implementation of action plans since intended results can only be achieved if the plans well implemented. C2. Objective Two: The Agriculture Objective The goal of this objective is to have 37,400 rural food insecure households, have improved supply and access to food over the three year life of the DRP program. This agriculture objective is primarily geared towards agricultural production for home consumption for food insecure households. The program works with local authorities, community leaders, and VDCs to identify vulnerable households, especially women-headed households, who are key participates in the program. The agriculture objective is supported by six intermediate results. These intermediate results target, (1) restoring agricultural production, (2) restoring storage and processing practices and facilities to reduce pre-and post-harvest losses, (3) restoring market infrastructure and market linkages, (4) building village-level capacities for effective planning and decision-making associated with agricultural issues and opportunities, (5) linking these strengthened village￾level organizations and organizations who serve them with the wider governance structure and (6) maintaining a safety net. Each of these intermediate results’ progress is reviewed to establish the level of achievement. Objective Two: Key Program Impact Indicator Results Assessment of this objective is based on three indicators namely, (1) average value of farm production per household of targeted crops, (2) percentage of vulnerable households consuming minimum daily food requirements and (3) percentage change in dietary diversification in household consumption. The performances of the three indicators against their LOA targets are presented in table 7 below. Table 6: Objective Two: 37,400 rural food insecure households have improved supply and access to food over the three year life of the program Indicator Baseline Evaluation Survey results LOA target Achieved to date vs LOA target Comments Average value of farm production per HH of targeted crops (rice, cassava, ground nuts, vegetables, coffee, cacao & palm oil) Leones 277,061 419,986 360,179 116.6% LOA target exceeded % change in dietary diversification of HH consumption (average # of food groups consumed) 3.8 5.1 5 102% LOA target exceeded Average no of eating occasions per HH per day 1.7 2.1 2.2 95.5% LOA target exceeded 20 Indicator 1: Average value of farm production per household of targeted crops (rice, cassava, ground nuts, vegetables, coffee, cacao & palm oil) Leones The value in Leones of farm production per household of targeted crops tremendously improved from 277,061 at the baseline to 419,986 Leones. The key crops that drove this tremendous performance were cocoa, rice, palm oil and coffee. Increased value of farm production at household level would imply more food for consumption, hence improving supply and access to food. The performance of this indicator compared to the LOA targets, indicates that the program performance exceeded the target of contributing to improving supply and access to food for the rural insecure households. Indicator 2: percentage change in dietary diversification of household consumption The percentage change in dietary diversification of household consumption rose above the baseline value by over 1 percentage point to surpass the LOA target. The rise in the index is an indication that households in DRP targeted communities are consuming a wide range of food groups compared to the baseline situation, this also confirms that the households in which these children live have improved supply and assess to food compared to the baseline. Indicator 3: Average number of eating occasions per household per day The average number of eating occasions per household per day greatly improved from the baseline value of 1.7 to 2.1 during the evaluation survey. This means that on average now, the households survive on two eating occasions up from less than two eating occasions at the baseline. This result, demonstrates both, the increase in the eating frequency of the households and the improvement in supply and access to food by the households in the DRP target area. Conclusion The outcomes of the three indicators above clearly exhibit fabulous achievements by the DRP program in contributing towards improving supply and access to food for the rural insecure households in the target area. In summary these results show that both agricultural production and household consumption levels among the CORAD target communities are picking up from the low levels during the baseline. Agricultural production is a function of area under production, yield, farming practices (technologies) and production related losses, this therefore reflects that there has been a transformation in the agricultural production function. IR 2.1 Farm Productivity Through this IR, the DRP is facilitating the restoration of productivity for a wide range of selected crops including food and cash crops that are grown during both the rainy and dry seasons. The DRP uses food for work (FFW) to mobilize labor to clear and rehabilitate overgrown farms. Using FFW to compensate participating individuals and groups confers short-run benefits of increasing food availability, preserving their scarce cash, and protecting precious seed reserves from being consumed. IR 2.1 Accomplishments as Reported in Fiscal Year Results Reports The DRP used over 3,200 mt of food in food-for-work (FFW) activities, to plant selected crops (cassava, groundnuts, maize and sweet potatoes), and tree farms (coffee, cocoa and oil palm). All beneficiary farmers for crops received training in improved agronomic and farm 21 management practices and tree farmers received training on agronomic and modern tree farm management techniques. The DRP supported planting of more than 9,954 hectares to selected food and cash crops. Farmers have been trained in improved agronomic practices which led to planting more than 4,488 hectares to groundnuts, cassava and other food crops. Through training in improved agronomic practices, farmers planted 270 hectares of commercial vegetables. Food for work was used to rehabilitate 1,800 hectares of inland-valley swamps. IR 2.1 Key Program Impact Indicator Results The performance of the six indicators against their LOA targets is presented in table 8 below. Table 7: IR 2.1 Farm productivity Indicator Baseline Evaluation LOA target Achieved to date vs LOA target Comments Area (HA) brought back into production of targeted crops (rice, cassava, ground nuts, vegetables, coffee, cacao, & oil palm) 2,400 9,954 15,968 62% LOA target achieved Area (HA) in ground nuts, cassava and other crops 1,300 4,488 6,500 69% LOA target achieved Tree farms rehabilitated (HA) 1,100 3,398 5,160 66% LOA target achieved Area in vegetables 0 269 960 28% LOA target missed Inland Valley farms rehabilitated 0 1801 3,348 54% LOA target missed Percent of households able to meet their required seed needs 43.75% 93.8% 65.63% 143% LOA Target exceeded Indicator 1: Area brought back into production of targeted crops (rice, cassava, ground nuts, vegetables, coffee, cacao, & oil palm) The area brought back into production of targeted crops (rice, cassava, ground nuts, vegetables, coffee, cacao, & oil palm), shot up by 4 fold from 2,400 hectares at baseline to 9,954 hectares, representing 62% of the LOA target. Increased area under crop production is health for increased farm output and hence food supply and access. The achievements under this result are to a large extent attributable to DRP’s integrated approach of combining FFW, FFAg and training farmers in modern farming techniques. Indicator 2: Area in ground nuts, cassava and other crops Performance of area in ground nuts, cassava and other crops was in line with the performance of area brought back into production of targeted crops. As already observed above in indicator 1 under this IR, the LOA target for this indicator was also achieved. Indicator 3: Area of tree farms rehabilitated The LOA target for tree firms rehabilitated was achieved. This means that tree crop farming especially coffee, cocoa, oil palm and kola nuts which is a significant activity in the target community is on the road to recovery. Tree farming is not only a source of food, but also a source of income for the households in the DRP communities. Final evaluation survey 0 5 10 15 20 25 30 35 40 45 % Figure 7: Percentage of Households Reporting Tree Plantations coffee, cocoa and oil palm Planted Rehab litated Planted 15.9 22.6 38.7 Rehabil tated 35.3 36.5 44.5 Coffee Cocoa Oil Palm 22 data shows that majority of the households (72.4%) responded that they had used food for work to rehabilitate tree plantations or IVS rice farms. This means at least seven out of every 10 households reported receiving FFW assistance for crop rehabilitation. Figure 7 shows that oil palm plantations were the most highly reported as rehabilitated by 44.5% of the households. The rehabilitation of cocoa and coffee trees was reported by 36.5% and 35.3% households respectively. Indicator 4: Area in vegetables The area in vegetables did not reach the planned for in the LOA target. However the implementation team felt that this target had been over ambitious given that very few areas had the tradition or culture of growing vegetables on anything other than a very small scale. As well as the cultural changes needed it was reported that the low adoption of vegetable gardening was mainly attributed to pest damage (particularly in the district of Tonkolili) and difficulty in accessing the local market for vegetables in the community. The DRP recognized the constraint of pest damage and CORAD has made contact with IITA Benin to introduce an environmentally friendly control measure to assist in this area before the end of the DRP. Indicator 5: Inland valley farms rehabilitated Although the LOA target of inland valley farms rehabilitated was missed. From qualitative assessment, it was observed that large areas of inland valley land are now under cultivation and productive. Indicator 6: Percentage of households able to meet their required seed needs Seeds were identified as one of the major constraints to restoring agriculture-based livelihoods. DRP supported over 3,000 households in the community through FFAg. Program interventions had commendable impact on the percentage of households that were able to meet their required seed needs. The LOA target was surpassed; the final evaluation results indicate that about nine out of every 10 households were able to meet their seed needs. Two major activities that contributed immensely to this achievement are direct seed distribution and creation of seed banks. Conclusion In summary based on both quantitative and qualitative assessment, there is unquestionable evidence that this intermediate result was achieved by the program. For example, based on the six indicators, the DRP exceeded LOA targets of four indicators. From qualitative assessment, it was observed that large areas of inland valley land are now under cultivation and productive. In terms of tree production, farmers in some areas have taken loans ranging between 150,000 and 750,000 LN to invest in their coco and coffee farms 23 According to many respondents, inland valley rice production has increased and will continue to do so as more and more people return back to the rural areas and re-establish their rural household economies. As one of the respondents describes the impact of the program, “… this program has made a big difference in our lives; we received tools, seeds, and even food so that we can work on our farm. We take food with us while we are working in our field, cook and eat it there. We also bring some food back for our children. It has been really good to work together…without the program we could not have done this, we would have suffered a lot...’ [Farmer in Daru village] IR 2.2 Reducing Pre and Post Harvest Losses Under this IR, the DRP has implemented activities aimed at reducing post-harvest losses. This objective is primarily pursued through: rehabilitation of community drying and storage facilities destroyed during the war; and training farmers on effective pre-and post-harvest techniques. IR 2.2 Accomplishments as Reported in Fiscal Year Results Reports DRP carried out a village mapping exercise and identified 208 community storages, drying and processing facilities that needed rehabilitation. For each of the identified facilities DRP identified and finalized contracts with beneficiary communities to contribute the required, locally available materials and tools. DRP used food for work to mobilize local labor to rehabilitate 106 community managed drying floors and storage facilities. Success under this result area was slowed primarily because of lack of imported materials for rehabilitation activities. This activity was designed on assumption that recipient communities would be able to contribute all the necessary inputs for rehabilitation (including locally available materials and imported materials e.g., cement). Consequently, DRP did not budget for imported materials which most communities have not been able to afford. The current levels of accomplishments are based on the DRP’s use of 102.8 mts of food used as food- for- work to rehabilitate facilities where target beneficiaries were able to contribute imported and locally available materials. IR 2.2 Key Program Impact Indicator Results The indicator to measure progress under this IR is presented in table 9 below Table 8: IR 2.2 reducing pre and post harvest losses Indicator Baseline Evaluation LOA target Achieved to date vs LOA target Comments Number of drying floors and storage facilities rehabilitated 0 106 462 23% LOA target missed. Indicator 1: Number of drying floors and storage facilities rehabilitated Achievement of results under this intermediate result was hampered by factors external to the program. As already observed above this activity was designed on assumption that recipient communities would be able to contribute all the necessary inputs for rehabilitation (including 24 locally available materials and imported materials e.g., cement). During implementation it became clear that this assumption did not hold. Crop wastage in the last 6 months Final evaluation survey data indicated that 84.9% of the beneficiary households reported suffering from some form of crop wastage in the last six months. However, further analysis showed that in terms of level of loss, 78.3% of the households reported low level losses. Crop wise analysis seems to suggest that more households (88.6%) reported rice wastage as indicated in figure 8. This may be understandable give the rice is one of the main food crops, and therefore it is grown by almost all households. Cassava and cacao come second to rice being reported by 24.2% and 24.0% households respectively. The percentage of households that reported using any measures to reduce crop losses was 54.5%. It is not possible to assess the degree of adoption here, since this information was not collected during the baseline. 0 10 20 30 40 50 60 70 80 90 % Figure 8: Households Reporting Crop Loss Loss 88.6 8.1 24.2 7.4 24 20 16.7 Rice Groundnut Cassava Oil Palm Cacao Coffee Sweet potatoe Qualitative assessment established that rehabilitation of drying floors and storage facilities is one of the less successful activities that caused much frustration among the different staff in the CORAD partnership. The targeted number of drying floors and community storage facilities has not been built or rebuilt due to the unavailability and price of exported building material (such as cement). The drying floors are used seasonally and were not in use at the time of the visit. Villagers report that the drying floors only accommodate 5-8 bags of rice, and they are used primarily by the households located in closer proximity. Rehabilitation of community storage facilities was noted to have not been an effective intervention, primarily because of inconvenient location. In most communities, the location of the facilities was decided by the town chief and elders. In some places, the storage facilities were built by agencies to serve the emergency food distribution operations, thus location is decided for logistical effectiveness rather than community need. In general, although the VDC committees acknowledge the presence of such facilities, they are not located in convenient places for many villagers to use. Moreover, poorer people in the village are reluctant to use these facilities because this will give information to their neighbors about how much food they have in store, which they prefer not to do. Conclusion The DRP scored less success under this intermediate result due to the design assumption of community contributing all the necessary inputs for rehabilitation (including locally available materials and imported materials e.g., cement) which did not hold. The DRP has recently secured approval from FFP to use surplus monetization funds to procure building materials for drying floors so that LOA targets become more achievable in remaining period of the program. The issue of utilization of rehabilitated community facilities depends upon a number of factors which included among others, purpose of construction of the facility, location and size of the facility, and social economic status of the households in the community. Therefore, as DRP undertakes rehabilitation of storage/drying facilitates, 25 these factors need to be brought into perspective to enhance utilization of community facilitates. IR 2.3 Re-establishing Market Linkages Interventions under this IR are aimed at restoring agricultural market systems that were disrupted during the war. Through repairing farm-to-market roads, rehabilitation of market facilities, re-establishing weekly markets and provision of market information, the DRP hopes to attract buyers to beneficiary communities. Expanded market opportunities will then stimulate increased production for household consumption and sale to generate income for other household requirements. IR 2.3 Accomplishments as Reported in Fiscal Year Results Reports Over 1,060 mts of food was distributed as FFW in improving access to markets through improved market roads, the rehabilitation of daily and weekly markets. 418.6 kms of farm-to￾market roads were rehabilitated improving access to health facilities and markets for more than 78,000 people. With this FFW five market facilities and 17 regular markets were re-established. Under this result area, under achievement was primarily due to the inability of target communities to contribute all the necessary locally available and imported materials. IR 2.3 Key Program Impact Indicator Results The performances of the three indicators that measure progress under this result area are presented in table 10 below. Table 9: IR 2.3 Re-establishing market linkages Indicator Baseline Evaluation LOA target Achieved to date vs LOA target Comments Km of farm-to-market roads rehabilitated 0 418 433 97% LOA target exceeded Market facilities rehabilitated 0 5 80 6.3% LOA target missed Number of regular weekly markets reestablished that were previously accessible to participating villages 0 16 51 33 3% LOA target missed Indicator 1: Km of farm to market roads rehabilitated Due to the civil war, farm to market roads had not been maintained and were even impassable in some areas because culverts and bridges have washed out. As way to support increased agricultural production, DRP under took rehabilitation of farm to market roads. The program surpassed its LOA target. Indicator 2: Market facilities rehabilitated Given the increase in agricultural production levels, in the DRP community areas as already noted above, the marketing systems not only provide opportunities for exchange of surplus production, but also stimulate agricultural production. The program missed the LOA target due to external factors. The activities depended heavily on the community providing some inputs and this assumption did not hold. In villages visited during qualitative assessment, 26 market facilities were rebuilt using locally produced materials from palms, trees, and bamboo. These facilities are well-utilized and much appreciated by villagers. Indicator 3: Number of regular weekly markets reestablished that were previously accessible to participating villages Before the civil war, every village had access to one or more weekly markets where agricultural surplus could be sold. The DRP interventions to try and re-establish regular weekly markets has had less impact on the ground as only 33.3% of the LOA target was achieved. Success of re-establishment of regular weekly markets depends on factors like availability of surplus agricultural production and information on these markets among other. Conclusion Overall, DRP’s achievement of this intermediate result have been less successful due to the design assumption of community contributing all the necessary inputs for rehabilitation (including locally available materials and imported materials e.g., cement) which did not hold. It is worth noting that the DRP has recently secured approval from FFP to use surplus monetization funds to procure building materials for market construction so that LOA targets become more achievable in remaining period of the program. IR 2.4 Village-Level Capacity Building for Agriculture The DRP provides technical training and assistance to various groups and community based organizations to enable them continue working together to address agricultural problems and capitalize on opportunities for improving production and marketing. Building capacity for these groups and organizations is a critical component of the DRP’s sustainability strategy. IR 2.4 Key Program Impact Indicator Results The performances of the two indicators that measure progress under this result area are presented in table 11 below. Table 10: IR 2.4 Village level capacity building for agriculture Indicator Baseline Evaluation LOA target Achieved to date vs LOA target Comments Percent of farmer groups who have developed action plans to address agricultural issues 0 77 50 154% LOA target exceeded Value (LE) of resources mobilized by farmer groups for agricultural-related activities 0 2.1m 10m 21% LOA target missed Indicator 1: Percent of farmer groups who have developed action plans to address agricultural issues As a key element of the sustainability strategy of the program, DRP built capacity of various farmer groups to enable them to continue working together to address agricultural problems and capitalize on opportunities for enhancing agricultural productivity in the community. The program performance was excellent, resulting into LOA target being exceeded. The achievements under this result were very evident too during the qualitative assessment. In all the visited areas for example, the FFSs are working well. They have monthly meetings to discuss farm issues; they cooperate in the planting of cassava and rice and other agricultural activities; and they share harvest. Farmer field days have been highly successful 27 according to the organisers and participants, and these activities are highly appreciated and praised by MOA. Many of the FFSs have produced an action plan comprised of a list of tasks they wish to do in the near future such as, clearing lands for cultivation, planting, etc. As one of the farmer puts it, “now we know the benefit of working together in a group, if you have any problem you are not too worried because you know you are not alone…” [FFS member in Badala village] Indicator 2: Value (LE) of resources mobilized by farmer groups for agricultural-related activities As another element of capacity building and sustainability, the program anticipated that farmer groups would mobilize resources that the groups would use to under take agricultural related activities. Although farmer groups have been able to mobilize some resources, the target for this indicator may have been set too high for farmer groups from communities that are just returning to normality with weak resource base. Conclusion Generally DRP achieved a number of successes in this area. FFSs were found to be working well, meeting regularly and had produced an action plan comprised of a list of tasks they wish to do in the near future such as, clearing lands for cultivation, planting, etc. IR 2.5 Decision-Making and Governance in Agriculture The DRP provides technical training and assistance to various groups and community based organizations to enable them continue working together to address agricultural problems and capitalize on opportunities for improving production and marketing. Building capacity for these groups and organizations is a critical component of the DRP’s sustainability strategy. IR 2.5 Accomplishments as Reported in Fiscal Year Results Reports DRP helped in creating agricultural sub-committees for 375 VDCs. VDCs and farmers groups were trained in basic business planning a number of these groups developed annual action plans to address agricultural related problems and capitalize on existing opportunities. The DRP organized an average of three meetings per six months between members of VDCs/village groups and representatives from agricultural-related institutions serving them. Through these meetings, the two stakeholders target beneficiaries had the opportunity to meet representatives from the district agricultural departments and discuss their problems and opportunities. 28 IR 2.5 Key Program Impact Indicator Results The performance indicator for this intermediate result is presented in table 12 below Table 11: IR 2.5 Decision making and governance in agriculture Indicator Baseline Evaluation LOA target Achieved to date vs LOA target Comments Average number of meetings over the last 6 months of key representatives from village-level organizations with leaders of VDCs, Chiefdoms, or government institutions on agricultural related issues 0 3 5 60% LOA target missed Indicator 1: Average number of meetings over the last 6 months of key representatives from village-level organizations with leaders of VDCs, Chiefdoms, or government institutions on agricultural related issues DRP’s interventions under this IR were to develop “functional linkages” with decision￾makers who are responsible for allocating resources and solving problems. The functional linkages were established between the decision-makers residing in the community as part of the Village Development Committee, the traditional community leadership structures, or the formal local government structures. During this period, the DRP has initiated the process of establishing linkages between VDCs and decision-makers at district and chiefdom levels by facilitating an average three meetings in six months between VDC representatives and those of government. Although the DRP has only achieved 60% of LOA target on this activity, there is strong evidence to suggest that if nurtured, these linkages will, in the future, strengthen the capacity of VDCs and other community-based organizations to participate in the planning and implementation of government projects. Opportunities exist for these linkages to be further strengthened and become more functional as the government decentralization process continues to be implemented. IR 2.6 Maintain a Safety Net The DRP provides safety net food and material assistance to vulnerable households, particularly those who return to re-establish their farms and those resettled with few if any assets. Without assistance, these households would remain vulnerable to any spurious local disasters such as fires, wind, and floods, all very common throughout DRP target districts. IR 2.6 Accomplishments as Reported in Fiscal Year Results Reports The DRP members used food to provide vulnerable group feeding (VGF) to 15,091 beneficiaries (mainly polio victims, blind, amputees and returnees). In partnership with other organizations, DRP members used FFW to mobilize labor that rehabilitated 3,392 houses. The houses rehabilitated belong to vulnerable households (mainly female-headed, returnees and amputees). It is estimated that this activity improved shelter for more than 16,000 individuals. In addition, the DRP used of FFW and locally available materials to rehabilitate 115 public infrastructures (clinics, wells and pit latrines). 29 30 IR 2.6 Key Program Impact Indicator Results The three indicators that measure progress under this result area are presented in table 13 below. Table 12: IR 2.6 Maintain a safety net Indicator Baseline Evaluation LOA target Achieved to date vs LOA target Comments Number of houses rebuilt or rehabilitated 0 3,392 4,000 85% LOA target exceeded Number of public structures rehabilitated (wells and pit latrines) 0 115 54 213% LOA target exceeded Number of vulnerable households and people in institutions provided with VGF 0 15,091 13,463 112% LOA target exceeded Indicator 1: Number of houses rebuilt or rehabilitated DRP assisted returning families to re-establish by providing shelter in form house rebuilding or rehabilitation. Through this intervention the program has been able to rebuild or rehabilitate over 3,390 houses, as a result, achieving the LOA target. Indicator 2: Number of public structures rehabilitated (wells and pit latrines) A number of public structures were destroyed during that war; DRP’s intervention in this area was to support the resettlement process by way of provision of structures like wells and latrines. This intervention also provided improved social life for the community in terms of better water and sanitation. DRP did an excellent job by rehabilitating 85 (wells and latrines), exceeding the LOA by over 50 percent. Indicator 3: Number of vulnerable households and people in institutions provided with VGF The program provided safety net food and material assistance to such vulnerable groups. Food assistance targeted for example orphans, the disabled, as well as those hospitalized. DRP reached over 15,000 people with vulnerable group feeding. This achievement exceeded the LOA target by 12%. Conclusion Overall the DRP did a tremendous job in providing a subsistence base for the most vulnerable in the community through food aid. DRP exceeded the LOA targets for this intermediate result. D. PROGRAM DESIGN The CORAD DRP was designed using best practices in program design. This process was facilitated by an external program design specialist who organized consultations with the key stakeholders. These consultations happened at three levels. At the national level, the design specialist consulted with key government ministries (especially Ministry of Health and Ministry of Agriculture, Forestry and Food Security) to gather their perspectives on the food insecurity problems in Sierra Leone. In addition, the design specialist consulted other International NGOs and donors supporting agriculture and health-related programs to identify synergies and gaps that needed to be addressed. Finally, the design specialist gathered secondary data at macro levels that formed the basis for preliminary problem analysis. At the second level, the design specialist collected data from each CORAD member to identify areas of strength and gaps in their agriculture and health that needed to be addressed through the DRP. Finally, the design specialist facilitated the process of collecting data at community level that was used in problem analysis. After collecting primary and secondary data, the design specialist organized a one-week workshop for all key stakeholders (including CORAD members, government and donor representatives) to process the collected data and agree on a preliminary results framework for the DRP. This event was also used by CORAD members to develop a common framework for partnership and agree upon key responsibilities and operational areas. At the end of this dialogue, a draft proposal was prepared and submitted to USAID/FFP. The issues raised by USAID/FFP during the proposal review process were jointly addressed by the four CORAD members. Recommendations for Improvement Because the DRP was designed under a typical post-conflict situation, not enough time was devoted to conduct thorough macro and micro-level analyses. In addition, under this environment, it was practically impossible for target beneficiaries to be involved in program design and planning. After critically reflecting on this process, there are two areas that could be improved during the MYAP design. 1. Involve target beneficiaries in problem analysis and MYAP planning. Because the MYAP is likely to build on the progress achieved by the DRP, CORAD members should consider creating more opportunities for current DRP and new target beneficiaries to be involved in the design processes of MYAP. This might be achieved by conducting thorough community assessments and beneficiary consultations to understand existing gaps and priorities that should be addressed through MYAP. It is also likely that MYAP will target current DRP/LINKS22 communities as well as new villages that have not benefited from the DRP and/or LINKS. For this reason, there is need for CORAD members to establish any contextual differences (needs, priorities and opportunities) that may exist among the different categories of communities targeted. Once established, these contextual differences should be used to guide MYAP strategies to maximize its relevance to each category of communities targeted. 22 The LINKS program was designed to build on the progress achieved through the DRP program. Funded by the Local USAID Mission Development Assistance funding, the LINKS program commenced during the 2nd year of implementing the DRP. Its LOA is estimated to last 3 years. 31 2. It is important for field staff and beneficiaries to be involved in the process of revalidating and refining program strategies. After the DRP was approved, CORAD members created an opportunity for key staff (mostly project managers) and representatives from government to refine the DRP program strategy. At that time (because of the post-conflict operating environment), it was not feasible for beneficiaries to be involved in the program strategy revalidation process. Over the last 2-3 years however, the operating environment has significantly improved, making it technically and operationally feasible for field-level staff and target beneficiaries to be involved in the strategic operations planning process prior to MYAP implementation. The most efficient way to achieve this objective might be to conduct operations planning workshops/meetings at 2 levels: (1) national level; and; (2) mezzo23 level. E. PROGRAM COORDINATION AND IMPLEMENTATION What sets CORAD aside from all other consortia is the excellent level of coordination of the technical and managerial functions of the DRP program. During the DRP planning stage, two program level units were set up to coordinate program implementation and performance management process. These units include: (1) a Compliance and Coordination Unit managed by CARE and; (2) a Monitoring and Evaluation unit managed by World Vision. In addition, the overall management of the DRP was guided by a steering committee. Finally, commodity management and monetization processes were coordinated by CRS. The progress achieved on these technical responsibilities, together with recommendations for improvements are summarized below: The Steering Committee At the beginning of DRP implementation, an eight-member steering committee was formed to oversee its implementation. The steering committee was charged with the responsibility of making strategic decisions on program related to the commodity basket, allocation of resources, contextual monitoring and overall guidance and problem solving for program the program. The steering committee was intended to serve as the primary mechanism for coordination and communication among partners. The committee was composed of the country representatives of each consortium member (or their designate), two representatives from Government, three DRP project management staff, the CCU Manager (CARE), the M&E Coordinator (WV), and the CRS Monetization Manager. The FFP representative was invited as a non-voting member to attend all meetings. Key lessons and recommendations for improvement At the beginning of program implementation, the steering committee met at least monthly and thereafter at least bi-monthly, to review progress of program implementation, identify opportunities and potential problem areas and make recommendations to enhance implementation. In this role, the committee was highly effective. However, its focus was more on operational issues than at strategic decision-making level. Moving into MYAP, the steering committee should be strengthened to provide overall guidance to program management. 23 In the Sierra Leonean context, the mezzo level refers to district and chiefdom levels. 32 The Coordination and Compliance Unit (CCU) The CCU was created to ensure effective coordination of the DRP implementation processes as well as compliance with CARE HQ and USAID/FFP regulations/standards and expectations. In our view, the CCU has been very effective in this role. Specifically, under the able leadership of John Perry (the Coordination and Compliance Manager), the CCU accomplished the following: • Created an effective framework to ensure that information flows into and out from the program including dissemination of reports, guidelines, meeting minutes and program M&E reports was properly managed; • Effectively represented the program, liaising with government, FFP, other agencies and the committee on Food Aid; • Created a framework for sharing experiences and lessons learned across the program; • Effectively coordinated program-level procurement processes and; • Ensured program-level compliance with USAID/FFP regulations and standards. Recommendations for Improvement 1. CORAD members to ensure that CCU feedback is shared with field staff to improve the quality of implementing key DRP activities. 2. Moving into MYAP, CORAD members should explore the possibility of improving communication between their sub-offices and HQ (Freetown) to ensure that CCU reports are shared with their field staff in an expeditious manner, given that the CCU does not have efficient electronic or other means in many cases to copy CORAD member field offices in on completed reports. 3. Opportunities also exist for CCU field trips to be synchronized with the monitoring trips of the M&E Coordination Unit. This will help to reduce the number visitors hosted by field staff as well as improve the effectiveness and efficiency of field monitoring trips. The evaluation team was informed that has begun to happen; we therefore suggest this should be consolidated. Program Monitoring and Evaluation As mentioned above, the M&E processes of the DRP were coordinated by a program-level unit managed by World Vision. This Unit was created to ensure the consortium approach to M&E is standardized to enable meaningful aggregation of results to a program level. In our view, this unit was highly effective and demonstrated the fact that having the M&E functions of this program coordinated by a central unit was not just a good idea, but the most effective way of approaching M&E activities of a consortium-level program. Specifically, the M&E unit has been very effective in: 1. Providing M&E related technical assistance related to developing M&E Frameworks, gathering and analyzing M&E information and other needs identified by the program steering committee; 2. Training and mentoring CORAD partner M&E staff and; 3. Consolidating performance monitoring information prepared by CORAD partners into program-level performance reports submitted to USAID/FFP. 33 The CORAD DRP M&E system24 During the first year of DRP implementation, the M&E Unit coordinated the process of setting up a robust M&E system that allows for consistent and systematic collection and reporting of performance data. The CORAD DRP M&E system is fully operational and generates: • Information used to assess the progress of the program implementation process, as well as progress towards achieving critical results (outputs, and outcomes of completed activities); • Information that facilitates accountability, both functional and strategic accountability; • Information that facilitates evidence-based decision making and; • Lessons learned which are used to improve the quality of program management, impact and sustainability. Long-term lessons learned will also be used to inform the design process of future similar programs. Recommendation for improvement Moving into MYAP, CORAD members should focus on strengthening the existing M&E system with emphasis on standardization and simplification of M&E processes and improving the reliability and integrity of the information generated by the M&E system. The following measures are recommended. (1) Strengthening the CORAD member capacity to collect, analyze and interpret data in a manner that maximizes data quality. This can be achieved through standardizing data collection processes and tools across the board. Lessons learned during the past two years of implementing the DRP should be used to develop practical and user￾friendly data collection tools and guidelines which can be used by all MYAP stakeholders involved in M&E. (2) Maximizing learning opportunities. During the last two years of implementing the DRP emphasis has been placed on setting up and assessing the effectiveness of M&E systems. Very little purposeful learning happened. During MYAP the M&E coordination unit should coordinate the process of intentional learning and sharing of lessons learned as program implementation progresses. This may be achieved through three mechanisms. First, the M&E unit could coordinate the process of developing a framework for technical working group meetings to share lessons learned (by reviewing progress, process and impact, challenges, problems and creative solutions, as well as the effectiveness of different program strategies). Secondly, the M&E unit could develop a framework to make quarterly reports a more “learning and sharing” mechanism rather than a tool for accountability. To achieve this, the M&E unit, the CCU and FFP could be encouraged to provide written feedback to all partners on their progress reports. CORAD members should also explore opportunities to organize quarterly critical reflection events for key stakeholders to assess progress and generate consensus on key lessons and “the way forward”. Thirdly, the M&E Unit could organize/facilitate annual critical reflection events that can be used to assess progress achieved during the previous FY and develop the plan for the next FY. Finally, 24 To establish this system, CORAD partners followed a 5-step process summarized in Annex xx. 34 CORAD members should explore possibilities of organizing cross exchange site visits “to learn more about each other’s progress”. These visits could be organized on a bilateral basis and key lessons shared in quarterly reports and at the steering committee meetings. (3) Strengthening internal data quality control mechanisms. During MYAP, the M&E Coordination Unit should continue to build the capacity of CORAD partner internal data quality control mechanisms. This can be achieved through staff training and organizing semi-annual data quality assessment exercises to assess the strength of partner M&E systems and suggest ways to improve them. F. CONCLUSION AND RECOMMENDATIONS DRP evaluation results highlight a number areas where program strategies have been highly successful (which should be consolidated) as well as gaps that should be addressed during the MYAP. These key conclusions and recommendations are highlighted below. F1. Conclusions The health objective: the DRP household-focused behavioral change strategies have been very effective in improving the status of children and pregnant/lactating mothers. The effects of these strategies are clearly demonstrated through increased full immunization, health seeking behavior and safe delivery rates. Access to health services: The program achieved and even exceeded its LOA targets. This is a tremendous accomplishment given the inadequate infrastructure in the target area and low health seeking behavior of the community at the baseline. Improved health and nutrition practices: There is a remarked improvement in treatment seeking behavior by the households compared to the baseline. More households are seeking for treatment outside the household and specifically in health facilities. Clinics are the most highly used health facilities. Reasons for not attending health facilities during an episode of any of the three focus diseases were reported as distance to the health facility and the cost of the drug. Village level capacity building for health: Overall, this result area achieved least progress. However, the major achievement that featured strongly was the presence of trained TBAs in every village. Least progress achieved under this result may be explained in the sense that changes in governance are likely to take a little longer, but also compounded by the recent past of the country. The agriculture objective: The outcomes of the indicators clearly exhibited achievements by the DRP program in contributing towards improving supply and access to food for the rural insecure households in the target area. In summary both agricultural production and household consumption levels among the CORAD target communities are picking up from the low levels during the baseline. Farm productivity: There is unquestionable evidence that this intermediate result was achieved by the program. Large areas of inland valley land have been rehabilitated and are now under cultivation and productive. In terms of tree production, farmers in some areas have taken loans ranging between 150,000 and 750,000 Le to invest in their coco and coffee farms 35 Reducing pre and post harvest losses: The DRP scored less success under this intermediate result due to the design assumption of community contributing all the necessary inputs for rehabilitation. This assumption did not hold during implementation. The DRP has recently secured approval from FFP to use surplus monetization funds to procure building materials for drying floors. It is expected that a significant percentage of the LOA targets are likely to be achieved in the remaining period of the program. Re-establishing market linkages: The achievement of this intermediate result has been less successful with regards to the rehabilitation of permanent and weekly market structure, but very successful in mobilizing communities to restore key farm to market roads. As mentioned above with the recent approval from FFP it is expected that a significant percentage of the LOA targets are likely to be achieved. Village level capacity building for agriculture: Generally DRP achieved a number of successes in this area, as demonstrated by the number of FFSs working well, meeting regularly and producing action plans to address agricultural issues such as, clearing lands for cultivation, planting, etc. Decision making and governance: Although not especially successful to date, there is strong evidence to suggest that the current gains if nurtured will in the future, strengthen the capacity of VDCs and other community-based organizations, such as FFS, to participate in the planning and implementation of government projects. New opportunities exist for these linkages to be further strengthened and become more functional as the government decentralization process continues to be implemented. Maintain a safety net: Overall the DRP did a tremendous job in providing a subsistence base for the most vulnerable through food aid in the community. F2. Recommendations Health Objective The area that may need special attention during MYAP is improving the nutritional status of children under fives. Whereas there is sufficient evidence to the effect that during the last two years of implementing DRP activities CORAD has contributed towards an increase in the number of children (under five) participating in growth monitoring and promotion activities, Nevertheless, it is important for MYAP to emphasize interventions that will directly contribute towards an improvement in the nutritional/health status of children. Specific emphasis should be placed upon enhancing the sustainability of the activities currently implemented by VHWs and TBAs to ensure continuity after the end of CORAD efforts. To achieve this (sustainability) objective, CORAD should explore possibilities of creating internal incentives for VHWs and TBAs to continue providing health-related information/advice and training in their communities after the end of the DRP’s mandate. In addition CORAD should explore possibilities of ensuring that DRP-supported health workers are integrated into the GoSL health care system, which have the on-going mandate to provide PHU services at community level. 36 Although the program exceeded the LOA target of children under one year who are fully immunized, there are opportunities of improving vitamin A and measles coverage rates that seem to be below coverage rates of other vaccines. Similarly, while the program exceeded its LOA targets on immunization, there is clearly room for improvement. The program may wish to focus attention on developing strategies to increase immunization coverage rates. The behavioral change strategies that DRP employed both in immunization and health seeking by the community have proved to be effective in improving immunization coverage and treatment seeking behavior. These strategies are worth documenting and replicating in future behavioral change interventions. The program may want to think about better measures of assessing of capacity of committees rather than just developing action plans. Such measures should take care of quality of meetings, capacity for the committees to manage the implementation of action plans since intended results can only be achieved if the plans well implemented. Finally, under the health objective, CORAD should explore possibilities of increasing the effectiveness of community-based nutritional/health groups. During the last two years of implementing DRP interventions, CORAD was short for increasing the proportion of health/nutrition groups that develop costed action plans, which is a means to an end. For these groups to positively impact their communities, these action plans need to be implemented and evaluated. Similarly, facilitating meetings between these groups and decision-makers at chiefdom and district levels is not enough. During MYAP CORAD should stress the effectiveness of these meetings to ensure that key recommendations and action points are implemented to positively impact host communities. Agriculture Objective Under the agricultural objective, CORAD should emphasize interventions that will directly lead to a reduction in the post-harvest losses. In terms of infrastructure, this is especially important because of the modest performance and utility on the number of community-based storage facilities rehabilitated during the past two years of implementing DRP activities. Additionally, the submissions received from DRP beneficiaries indicate the need to emphasize household-focused interventions (such as strengthening training on post-harvest management technologies) that will directly contribute towards a reduction in post-harvest losses at household (and then community) levels. The utilization of community facilities depend upon a number of factors which included among others, purpose of construction of the facility, location and size of the facility, and social economic status of the households in the community. Therefore, as CORAD undertakes rehabilitation of storage/drying facilitates, these factors need to be brought into perspective to enhance utilization of community facilitates. The DRP needs to establish a systematic strategy to improve the technical aspects associated with the sustainability of food-for-work activities. This is true, for example, with regard to the use of food to rehabilitate farm-to-market roads and tree crop plantations. DRP evaluation results highlight a number areas where program strategies have been highly successful (which should be consolidated) as well as gaps that should be addressed during the MYAP. These key conclusions and recommendations are highlighted below. 37 G. APPENDICES G1. DRP Final Evaluation Scope of Work 38 G2. DRP M&E System Setup 5-Stage Model Used to Setup the CORAD DRP M&E System Developed a framework for sharing M&E info to facilitate decision￾making and learning Conducted Stakeholder dialogues to refine and generate consensus on the program Results Framework, refine Indicators and finalize targets Program Staff CCU/Steering Committee USAID MoH MAFFS Other Stake￾Holders Developed Common Data Collection tools for output Indicators Developed and Finalized Performance Indicator Reference Sheets Developed a conceptual framework for program evaluation Developed Reporting guidelines /Formats & Analytic Agenda Conducted a multi￾indicator baseline survey Introduced internal data quality control measures/mechanisms Program Performance Reviews & Impact assessment Data collection, analysis, verification, critical reflection & reporting 39 40 Step 1: These steps began with a series of stakeholder dialogues undertaken to refine and generate consensus on the program results framework. During this process, program stakeholders got the opportunity to revalidate the critical intended program results, the cause –and-effect relationships between them, the program development hypothesis, as well as the critical assumptions for program effectiveness. Similarly, stakeholders reviewed, and finalized performance targets for all program indicators25. Step 2: Following the successful finalization of program indicators, the M&E Unit coordinated the process of developing common data collection tools used by all CORAD partners to track output-type indicators. The objective was to ensure standardization of the data collection and analysis procedures used by all CORAD partners in order to enable any meaningful aggregation of results into program-level performance. This process was participatory, and adequate training was provided to all stakeholders involved in program performance monitoring. During this stage, the M&E Unit also finalized Performance Indicator Reference Sheets (PIRS) and developed a conceptual framework for program evaluation. This conceptual framework was developed to guide the implementation of DRP activities in a manner that enhances quality, effectiveness, efficiency, impact and sustainability26. Step 3: At this stage, the M&E unit coordinated the process of developing common reporting guidelines and formats used by all partners to report progress on program performance. These formats are used to compile quarterly reports that: capture progress of the implementation process, articulate immediate outcomes of completed activities, capture short-term lessons learned, discuss problems, challenges and solutions and present planned activities for subsequent reporting periods. In addition, CORAD partners conducted a multi￾indicator baseline survey that generated benchmark values against which the impact created by the program has been assessed. Step 4: One of the most important attributes of a good M&E system is to generate good quality and reliable information that can facilitate evidence-based decision-making and facilitate learning by all stakeholders at the various stages of a program cycle. To maximize data quality, DRP partners identified and established internal quality control systems to minimize the likelihood tampering with performance data and misleading reporting. These data quality control mechanisms include: use of common data collection tools, site visits by project managers to verify project staff reports, site visits by the CCU/M&E Unit to assess performance and verify partner results, and rigorous annual data quality assessments. Step 5: Finally, to maximize learning opportunities from information generated by the M&E system, DRP partners developed a framework for learning. This framework identifies key stakeholders, their information needs and incentives for sharing key M&E information. Currently, information is shared through: quarterly progress reports and updates, staff monthly and quarterly critical reflection events, partner exchange and site visits, national steering committee meetings, as well as annual progress reports. CORAD partners also prepare and share progress updates that may be requested by major stakeholders on ad-hoc basis. 25 A full list of indicators is contained in the IPTT annexed to this report 26 The conceptual framework is summarized in the baseline report