January 2014 This publication was prepared at the request of the United States Agency for International Development. It was prepared independently by Bechir Rassas, Louis Herns Marcelin, Bernard Crenn, and Felipe Tejeda, International Business & Technical Consultants, Inc. (IBTCI), with Interuniversity Institute for Research and Development (INURED) as sub-contractor FINAL EVALUATION REPORT Haiti Title II Multi Year Assistance Programs (MYAP): World Vision COVER PHOTO Credit: World Vision Mothers' Club focus group, La Gonave FINAL EVALUATION OF HAITI TITLE II MULTI YEAR ASSISTANCE PROGRAMS (MYAP) WORLD VISION January 2014 AID-RAN-I-00-09-00016, Order No. AID-521-TO-13-00001 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS ACRONYMS ..............................................................................................................................i EXECUTIVE SUMMARY ....................................................................................................... ii EVALUATION PURPOSE AND QUESTIONS .................................................................... 1 Evaluation Purpose ................................................................................................................................................... 1 Evaluation Questions ................................................................................................................................................ 1 Program Background ................................................................................................................................................ 2 EVALUATION METHODS & LIMITATIONS .................................................................... 7 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ............................................ 9 QUESTION 1: HOW DID THE FOOD SECURITY STATUS OF THE POPULATION CHANGE?.. 9 Findings ....................................................................................................................................................................... 9 Conclusion ................................................................................................................................................................ 15 Recommendations ................................................................................................................................................... 15 QUESTION 2: DID THE PROGRAM ADDRESS THE MOST CRITICAL PROBLEMS OR CONSTRAINTS TO FOOD SECURITY AND RESILIENCY FOR THE MOST VULNERABLE? ....... 16 Findings ..................................................................................................................................................................... 16 Conclusions .............................................................................................................................................................. 21 QUESTION 3: ARE THE CONSTRAINTS FACED BY THE TARGET BENEFICIARIES AS OUTLINED IN THE ORIGINAL DOCUMENT STILL RELEVANT? ........................................................ 22 Findings ..................................................................................................................................................................... 22 Conclusion ................................................................................................................................................................ 24 QUESTION 4: TO WHAT EXTENT DID THE PROGRAM’S THEORY OF CHANGE CONTRIBUTE TO THE MYAPS ACHIEVEMENTS IN TERMS OF PROJECT RESULTS AND OUTCOMES? .......................................................................................................................................................... 24 Findings ..................................................................................................................................................................... 24 Conclusions .............................................................................................................................................................. 29 Recommendations ................................................................................................................................................... 30 QUESTION 5: HAVE WOMEN AND OTHER DISADVANTAGED POPULATION GROUPS WHO PARTICIPATED BEEN DIFFERENTLY AFFECTED (POSITIVELY OR NEGATIVELY) BY THE PROJECT? ................................................................................................................................................................ 30 Findings ..................................................................................................................................................................... 30 Conclusions .............................................................................................................................................................. 32 Recommendations ................................................................................................................................................... 32 QUESTIONS 6-7: IS THERE ADEQUATE EVIDENCE SUGGESTING THAT THE PROJECT OUTCOMES ARE LIKELY TO BE SUSTAINED? WHAT WERE THE MAJOR FACTORS WHICH INFLUENCED THE ACHIEVEMENT OR NON-ACHIEVEMENT OF SUSTAINABILITY OF THE PROJECT? ................................................................................................................................................................ 33 Findings ..................................................................................................................................................................... 33 Conclusions .............................................................................................................................................................. 40 Recommendations ................................................................................................................................................... 40 QUESTIONS 8: TO WHAT EXTENT ARE THE VARIOUS MOTHERS’ CLUBS MODELS IMPLEMENTED BY WVH, CRS AND CRS COST EFFECTIVE? ................................................................ 40 TABLE OF TABLES Table 1: MYAP General Information 3 Table 2: Geographic areas targeted by the World Vision MYAP and program beneficiaries 4 Table 3: World Vision’s MYAP results framework 5 Table 4: Coverage Estimates for All Indicators Measured: World Vision Program 10 Table 5: Key World Vision program activities and outputs in support of PM2A 19 Table 6: Key World Vision program activities and outputs in support of increasing food production and household assets, enhancing market-based livelihoods, and rehabilitating natural resource resiliency 20 Table 7: World Vision’s MYAP Results Framework 25 Table 8: World Vision Program Contributions to MYAP Achievements 27 TABLE OF FIGURES Figure 1: World Vision Program Map 4 Figure 2: Child health and hygiene results of quantitative data from household surveys 11 Figure 3: Child malnutrition and feeding practices 12 Figure 4: World Vision population malnutrition rates compared to other populations 13 Figure 5: Household food security measures over time 14 Figure 6: Uptake of agricultural practices and innovations over the 5-year program 14 Figure 7: Family planning services, birth attendance and health for pregnant women 15 Figure 8. Most common activities addressing food security, referenced in qualitative survey discussions 17 Figure 9: Participation findings from Qualitative Surveys 21 Figure 10: Main impacts reported from food assistance activities 25 Figure 11: Program outcomes and activities deemed likely to persist 34 i ACRONYMS ADP Area Development Program BCC Behavior Change Communication CBO Community Based Organization Col Vols Community Volunteers CS Cooperating Sponsor CHW Community Health Workers CMAM Community Management of Acute Malnutrition CNSA Coordination Nationale de la Sécurité Alimentaire CRS Catholic Relief Services DAP Title II Development Assistance Program EWS Early Warning System FANTA Food and Nutrition Technical Assistance FFA Food for Assets FFP Food for Peace FFW Food for Work GoH Government of Haiti HAS Albert Schweitzer Hospital HDDS Household Dietary Diversity Score FY Fiscal Year IPTT Indicator Performance Tracking Table IYCF Infant and Young Child Feeding MAHFP Months of Adequate Household Food Provisioning MARNDR Ministry of Agriculture, Natural Resources and Rural Development MCHN Maternal and Child Health and Nutrition M&E Monitoring and Evaluation MOH Ministry of Health MUAC Mid-Upper Arm Circumference MUSO Mutuelle de Solidarity (Self Help Group), MYAP Multi-Year Assistance Program OVC Orphans and Vulnerable Children PM2A Preventive Malnutrition under Twos Approach PEPFAR President’s Emergency Plan for AIDS Relief PFA Poid faible pour l’age (low weight for age) PTFA Poid tres faible pour l’age (very low weight for age) PMP Performance Monitoring Plan PNA Preventive Nutrition Approach (see PM2A) SO Strategic Objective USAID United States Agency for International Development W/H Weight for Height Index ii EXECUTIVE SUMMARY BACKGROUND For several decades, the U.S. Government, in collaboration with several international private volunteer organizations (PVOs), has supported a number of food security projects in Haiti using U.S. Public Law 480 Title II resources with a view to improving food security throughout the country. The strategic objectives of the most recent Multi Year Assistance Programs (MYAPs) in Haiti, as stated by USAID in 2007, are: Improved Nutritional and Health Status of Targeted Vulnerable Groups, and Improved Productive and Profitable Livelihoods for Vulnerable Groups. These two strategic objectives are to be achieved through five key interventions: • Improve the nutritional and health practices of targeted vulnerable populations • Improve the quality of and access to health services • Increase food production and household assets • Enhance market-based livelihoods • Rehabilitate natural resource resiliency and local response capabilities Three PVOs -- Agricultural Cooperative Development International and Volunteers in Overseas Cooperative Assistance (ACDI/VOCA), Catholic Relief Service (CRS) and World Vision have been separately implementing Title II MYAPs in Haiti over the past five years in pursuit of the overall program strategic objectives and intermediate results. They did this under one umbrella award which tied them to common goals and similar approaches, though in different geographic areas. Although the goal of the three programs is to reduce food insecurity and increase the resilience of extremely vulnerable rural households, each of these private agencies implemented its own program based on the specific circumstances of the target communities and their unique approach. Each of these Cooperating Sponsors had coastal and mountainous communities in their geographic areas, and these communities each faced different repercussions from the series of four hurricanes (Fay, Gustav, Hanna and Ike) that hit in 2008 and the large earthquake of early 2010. EVALUATION PURPOSE The purpose of this evaluation is to conduct a final performance evaluation of the Haiti MYAP. The evaluation examines the overall performance of the program by investigating three major questions: (1) the extent to which the food security status of the targeted population has changed; (2) the extent to which the MYAP programs have contributed to the resilience of the targeted communities; and (3) the extent to which the various mothers’ clubs models implemented by the Title II cooperating sponsors are cost-effective. The evaluation is expected to help guide and optimize the effectiveness of future Food for Peace programming in Haiti, and compile the best practices and lessons learned. The primary stakeholders for this evaluation include: USAID’s Office of Food for Peace (FFP), World Vision; Catholic Relief Services (CRS), ACDI/VOCA and the Government of Haiti. By virtue of their partnership under iii the Food for Peace Program, the three PVOs/NGOs are also known as “Cooperating Sponsors”, and the acronym CS will refer to these three from here on. EVALUATION QUESTION Relevance Q1 How did the food security status of the targeted population change? Q2. Did the program address the most critical problems or constraints to food security and resiliency for the most vulnerable? Q3. Are the constraints faced by the target beneficiaries as outlined in the original document still relevant? Appropriateness Q4. To what extent did the program’s theory of change contribute to the MYAPs achievements in terms of project results and outcomes? Q5. Have women and other disadvantaged population groups who participated been differently affected (positively or negatively) by the project? Sustainability Q6. Is there adequate evidence suggesting that the project outcomes are likely to be sustained? Q7. What were the major factors which influenced the achievement or non-achievement of sustainability of the project? Cost-effectiveness Q8. To what extent are the various mothers’ clubs models implemented by WVH, CRS and ACDI/VOCA cost effective? METHODOLOGY The evaluation methodology is based on a mixed data collection approach combining quantitative and qualitative data collection as follows: 1) review and analysis of relevant documents; 2) an extensive quantitative survey, with random sampling of households, conducted throughout the MYAP areas to measure progress since 2008 when the MYAP baseline survey was conducted; 3) as part of the randomized survey, measurement of the anthropometric indices of nutrition status of 1912 children; 4) qualitative surveys involving extensive focus-group discussions; and 5) field visits and key informant interviews, including discussions with implementing staff and local authorities. MAJOR FINDINGS The overall World Vision program demonstrates improvement in 11 of the 14 indicators and in all indicators for 2 of the 5 activity areas. The strongest improvements are in family planning and antenatal/post-natal care and agricultural production practices. iv Although children immunization rates have slightly increased (6 percent) since 2008 and are now at 72 percent, measles vaccination rates have decreased by nearly 20 percent and are below the rate for all MYAPs. This is in contrast with hand washing, which improved by over 20 percent, although still below the rate for all MYAPs For child nutrition, two indicators improved significantly (underweight and infant and young child feeding -- IYCF), but the other two (stunting and wasting) remained about the same (stunting) or retreated slightly (wasting). Although the IYCF indicator has increased by 12 percent since 2008, only less than one out three children is being fed according to appropriate feeding practices. For household food security, the indicators suggest that households in the World Vision areas remain very food insecure, having enough food provisions to feed themselves adequately only 30 percent of the time (an average of 3.5 months out of 12 in the last year). The household dietary diversity score (measured by the number of food groups consumed by the household in the last 24 hours) stands at 5 on average. The adoption of improved agricultural practices has improved significantly since 2008. Nearly 9 of every 10 farmers (87 percent) already have adopted three or more recommended agricultural production practices. This figure is reduced to 74 percent when considering four or more practices. The contraceptive prevalence rate for World Vision indicates that over one in two women (58 percent) was using modern family planning methods after 5 years of MYAP activities, a rate 15 percent higher than the contraceptive prevalence rate in 2008. One in two women now receives professional assistance at delivery, an increase of 18 percent from 2008. At over 90 percent, the proportion of women who made at least three prenatal visits is higher than for the MYAP program as a whole. World Vision did address the most critical problems and constraints to food security and resiliency for the most vulnerable. The World Vision program targeted pregnant women, mothers of children under 2-years of age and those acutely malnourished between 24 and 59 months. With its focus on MCHN activities for pregnant and lactating mothers and their under-5 children, World Vision’s approach was consonant with a convergence of research on the most cost-effective way to address nutrition. It was also in line with the county’s national nutrition policy, which features prevention, management of acute malnutrition and nutrition protection in emergency situations. World Vision’s focus on helping vulnerable communities to achieve increased farm and off farm income reflected the country’s poverty reduction strategy that was endorsed by major international development agencies the same year the program started. The constraints faced by the beneficiaries targeted for agricultural and livelihoods assistance as outlined in the original World Vision program document remain considerable. The natural shocks and other events (floods, storms, hurricanes and droughts) that have adversely affected food security for the most vulnerable in the MYAP areas and throughout the country since 2008 are likely to continue. For these reasons, food security in Haiti remains a national concern and the constraints faced by the beneficiaries targeted for nutrition and livelihoods assistance as outlined in the original program document remain as daunting as before. World Vision’s theory of change draws on both in-country factors and standard theory and analysis in the two technical sectors underlying its strategic objectives and intermediate results. However, by v not fully integrating MCHN and agricultural programs in its MYAP areas, World Vision may not have fully capitalized on the synergy between the two interventions. The majority of World Vision’s program activities over the MYAP period benefitted pregnant and lactating women and their under-5 children. However, not all MCHN women beneficiaries were included in World Vision’s livelihoods activities. Two other disadvantaged groups that were specifically targeted were OVCs and PLWAs. The performance indicators used to measure results by World Vision have several shortcomings, including a disproportionately high number of indicators; the predominance of output and process indicators, instead of results indicators to monitor progress; and the absence of indicators to measure capacity building and sustainability. World Vision’s PMP indicators were not sufficiently gender disaggregated. WHV created demand for its program services. However, in its MCHN interventions, World Vision focused on training, a key capacity development ingredient, but did not place sufficient emphasis on other performance gaps with clear milestones for achieving program sustainability goals. In agriculture, the potential for program sustainability would have been enhanced through greater emphasis on market mechanisms to ensure that the supply of agricultural inputs and services are available to program beneficiaries when the program ends. World Vision's sustainability plan is not based on a clearly defined strategy or objective sustainability indicators. Many of World Vision’s MCHN activities are likely to have lasting effects. However, the program focused almost exclusively on training, and no comprehensive capacity building plan was developed to ensure the provision of MCHN services after the end of the program. Many of World Vision’s agricultural, natural resources management and livelihoods activities are likely to have lasting effects. However, ensuring continued supply of agricultural inputs and services after the end of the program would have strengthened program sustainability. MAJOR RECOMMENDATIONS To adequately protect child health, immunization rates in the next set of USAID Title II food aid awards should be brought up to at least 80 percent of children under-5 years old, and hygiene and cleanliness in the household should continue to be promoted to reduce the spread of infectious diseases. Actions to reduce long-term under-nutrition (stunting) are of ongoing importance. Improving child health, household hygiene and household dietary will be essential toward the goal of reducing malnutrition. The specific interventions to be implemented should be based on further operations research, particularly regarding the identification of local foods given to infant and young children that will improve the quality of their diet and fill gaps in micronutrient intake, including deficits of vitamin A, iron, zinc, and folic acid. In particular, whether the expansion of household gardening has any observable impact on micronutrient deficiency diseases should be researched. In the meantime, CSs should give greater attention to mixed methods of nutrient outreach (vertical, clinic-based, integrating agriculture and health extension activities, local fortification of food rations, etc.). The CSs should ramp up messaging about measles immunizations and referral to immunization days or clinics in the mothers group outreach. Future cooperating sponsors should vi be cautious about the possibility of drop off of measles immunization even while other EPI vaccinations appear high for other diseases. Additional research may also be needed to investigate why food insecurity did not improve in many communities even when recommended agricultural practices have been adopted by nearly 90 percent of the population. Investigation may also explore how the linking of home vegetable gardens, fruit trees, and small animal husbandry activities (e.g., chickens and goats) for mothers’ clubs may increase the intake of micronutrients through dietary diversity, thus strengthening the link between adoption of improved agricultural practices and enhanced child nutrition status. The preventive PM2A approach should be applied from the beginning of the activity and throughout the assisted population. More emphasis should be placed on promoting better integration within programs of the agricultural/natural resources management with the MCHN components. In particular, the various agricultural, natural resources management and livelihoods activities should focus on the same target populations as do the MCHN activities. Prioritizing agricultural and livelihoods activities integrated and layered with mothers’ groups would be a crucial step in that direction. Performance indicators of the next MYAP should be anchored in strategic thinking about what should be achieved for program success. For this reason, the overall set of indicators should be streamlined with a view to discarding those that may not be needed. New indicators should be identified to measure results, outcomes and impact, rather than processes and outputs. Women should be better integrated into livelihood activities to better benefit from MCHN services. Better integration of women would provide additional resources or income at the household level to purchase food and pay for medicine and health care services. Each of the CSs could have dovetailed more agriculture, livestock, seeds and tree interventions in the same local populations where the MCHN activities were concentrated, directly recruiting the same women receiving food rations. Gender-sensitive indicators should be developed and used according to FFP guidelines in future Title II programs in Haiti. Those indicators should measure differences in how men and women participate in or benefit from the program. The program should extend assistance to the most disadvantaged groups. These include households with HIV/AIDS and persons infected with tuberculosis (with food aid to improve overall health) extremely vulnerable people (orphans, very sick, homeless, the destitute elderly); and people affected by natural disasters (for example with emergency feeding). Future cooperating sponsors should be required to develop rigorous sustainability plans and monitor their progress against milestones. Experience from other countries demonstrates that, in order to achieve optimal results, the sustainability of Title II programs should be an integral part of program design and should be embedded throughout the cycle from implementation to withdrawal. Community organizations and individuals should also be aware of their post-program roles and responsibilities from the outset. The sustainability plan should at a minimum include: decisions about approach (phase out, gradual phase over); explicit benchmarks for progress and timelines; clear allocation of responsibilities; graduation criteria and progressive phase out of free inputs; a focus on building the capacity of local community and government organizations to progressively vii take up the management and provision of MCHN services; and developing alternative incentive structures (e.g., livelihood programs) to create increased resilience among beneficiaries to self-fund MCHN and other services. To ensure greater sustainability of livelihood activities, stronger emphasis should be placed on private sector participation. 1 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE The purpose of this evaluation is to conduct a final performance evaluation of the World Vision Haiti (World Vision) Multi Year Assistance Programs (MYAP) program conducted under cooperative agreements under USAID. The World Vision MYAP program is part of an overall program that includes two other cooperating sponsors (CSs): ACDI/VOCA and Catholic Relief Services (CRS). This evaluation considers the overall impact of the program by investigating three major questions: (1) the extent to which the food security status of the targeted population has changed; (2) the extent to which the MYAP program has contributed to the resiliency of the targeted communities; and (3) the extent to which the various Mothers’ Clubs models implemented by the Title II cooperating sponsors are cost-effective. Since cost-effectiveness assessments must involve more than one program, the cost-effectiveness question will be investigated only in the overall MYAP program report. The evaluation is expected to help guide and optimize the effectiveness of future Food for Peace programming in Haiti, and compile the best practices and lessons learned. The primary stakeholders for this evaluation include: the Food for Peace program of USAID, World Vision, CRS, ACDI/VOCA, and the Government of Haiti. EVALUATION QUESTIONS Relevance Q1 How did the food security status of the targeted population change? Q2. Did the program address the most critical problems or constraints to food security and resiliency for the most vulnerable? Q3. Are the constraints faced by the target beneficiaries as outlined in the original document still relevant? Appropriateness Q4. To what extent did the program’s theory of change contribute to the MYAPs achievements in terms of project results and outcomes? Q5. Have women and other disadvantaged population groups who participated been differently affected (positively or negatively) by the project? Sustainability Q6. Is there adequate evidence suggesting that the project outcomes are likely to be sustained? Q7. What were the major factors which influenced the achievement or non-achievement of sustainability of the project? 2 Cost-effectiveness Q8. To what extent are the various mothers’ clubs models implemented by World Vision, CRS and World Vision cost effective? PROGRAM BACKGROUND For several decades, the U.S. Government, in collaboration with several international private volunteer organizations (PVOs), has supported a number of food security projects in Haiti using PL-480 Title II resources with a view to improving food security throughout the country. The strategic objectives of the food assistance MYAPs in Haiti as stated by USAID in 2007 are the following: Improved Nutritional and Health Status of Targeted Vulnerable Groups, and Improved Productive and Profitable Livelihoods for Vulnerable Groups. These two strategic objectives were to be achieved through five key areas of intervention: • Improve the nutritional and health practices of targeted vulnerable populations • Improve the quality of and access to health services • Increase food production and household assets • Enhance market-based livelihoods • Rehabilitate natural resource resiliency and local response capabilities Three PVOs -- ACDI/VOCA, CRS and World Vision -- have been separately implementing Title II MYAPs activities in Haiti over the past five years based on the overall program strategic objectives and intermediate results. Although the goal of the three programs is to reduce food insecurity and increased resilience of vulnerable and extremely vulnerable rural households, each PVO (henceforth referred to as cooperating sponsor or CS) implemented its own set of activities, based on the specific circumstances of the target communities and the CS’ unique approach. This is against a background of several decades of U.S. food assistance to Haiti, typically through NGO consortia, the total volume of which peaked at 153,000 metric tons during 2010, a spike composed largely of emergency food aid after the January earthquake that struck Léogâne and Port au Prince. The three CS’s averaged 9,000 metric tons of food distribution per year. Separately, the World Food Program delivered 70,000 metric tons during the 2008-2012 MYAP period, and USDA also supports Food for Education programs. USAID/Haiti contracted this final independent performance evaluation of the Haiti Title II MYAP activities, now that the interventions, conducted during the final phase of implementation. A baseline survey had been conducted in 2008 by the Food and Nutrition Technical Assistance (FANTA) project and a mid-term evaluation was conducted in 2010. This final evaluation examines the performance of MYAP programs in areas targeted by each CS, in achieving the objectives and scopes established by USAID at their inception. The major features of each MYAP program are summarized in the table below. 3 Table 1: MYAP General Information Project Title Multi-Year Assistance Program (MYAP) Award Numbers FFP-A-00-08-00029; FFP-A-00-08-00023; FFP-A-00-08-00024 Award Date 02/19/2008 Funding PL480 Title II Implementing Partners ACDI/VOCA, CRS and WORLD VISIONI AOTR Babette Prévot Amendments to the original three agreements were made in 2011. Final evaluation results for the HAITI Title II MYAP program will be presented in four separate reports, one for each of the three CSs and a fourth report for the overall MYAP program. The present report focuses on the World Vision program. Geographic coverage: World Vision has been operational in Haiti for over 35 years and currently has staff members working through numerous special projects and micro-regional development initiatives in five regions of the country. World Vision Haiti (World Vision) has been present on the Upper Central Plateau and La Gônave for decades. Following its proposal submission in November 2007, World Vision was awarded in February 2008 the USAID Title II funded Multi-Year Assistance Program (MYAP) - SAK PLEN (Full Sack) Resiliency Enhancement Program (SAK REP) and began implementation in sixteen communes in Haiti. World Vision’s MYAP program worked within geographic Departments, which in Haiti are sub￾divided into Communes, then Section Communales, and finally Localities. Localities represent the lowest level for geographic distribution of MYAP services. The World Vision MYAP focused in 6 communes of the Upper Central Plateau in central Haiti, 8 communes in the Lower Central Plateau and the Artibonite valley, and the 2 communes on the Island of La Gônave, to the West of the main island of Hispaniola. These areas were selected by World Vision due to their high rates of food insecurity, and the World Vision intended to reach some 540,369 beneficiaries or about 108,000 households, believed to represent about 57percent of the population in these areas. Maternal Child Health and Nutrition (MCHN) activities in the Lower Central Plateau and Artibonite valley were originally subcontracted to three international and local NGOs, Save the Children, Hospital Albert Schweitzer (HAS), and Management and Resources for Community Health (MARCH)1 , while World Vision implemented the Livelihood activities in 3 (of 8) of these communes. World Vision maintained program-wide responsibility for the management and distribution of all MCHN rations. AGRIDEV, a private enterprise company, received a sub-contract working with farmer groups to link key agricultural commodities to regional and international markets – with a focus on high value vegetable crops under rehabilitated irrigation systems. 1 In August 2010 World Vision terminated its agreement with MARCH, and in March 2011 terminated its agreement with Save the Children in both Upper Plateau Central and Artibonite regions; in 2011 a new partner was added, Hospital Claire Heureuse (HCH), to provide services in the Dessalines Commune also of the Artibonite region. 4 Figure 1: World Vision Program Map Table 2: Geographic areas targeted by the World Vision MYAP and program beneficiaries Communesf Upper Plateau Total Population: 347,000 Target Beneficiaries 330,297 Communes of La Gônave Total Population: 84,250 Target Beneficiaries 80,072 Communes of Lower Plateau & Artibonite Total Population: 605,000 Target Beneficiaries 130,000 1. Hinche 2. Thomonde 3. Boucan Carre 4. Thomassique 5. Cerca￾La￾Source 6. Cerca- Carvajal 7. Anse-à￾Galets 8. Pointe-à￾Raquette 9. Saut d’Eau 10. Mirebalais 11. Lascahobas 12. Savanette 13. Verettes 14. Petite Rivière 15. Dessalines 16. Maissade To achieve its MYAP goal, World Vision supported private sector partners, the Government of Haiti and community-based associations with a view to: (1) enhancing safety nets to promote preventive child nutrition and health practices, promote dietary intake through training and education to build inter-generational knowledge, and give support to active nutritional surveillance; (2) promoting more diverse and sustainable agricultural livelihoods; (3) building community capacity to manage development and respond to shocks by assisting communities to conduct risk and vulnerability assessments and develop action plans; and (4) maintaining a surge capacity for flexible emergency response, in accordance with development relief principles. This was accomplished by assisting individual households, carrying out livelihood activities with agricultural producer groups and savings groups (MUSO), and by working closely with community leaders. World Vision’s health and nutrition focus was carried out through activities at rally posts and food distribution points, using mobile clinics, carrying out home visits, and at Mother’s Clubs and later Mothers’ Care Group (MCG) meetings. 5 The seven major groups targeted for assistance within the World Vision MYAP intervention regions included: 1. Pregnant and lactating women, 2. Infants 6-23 months, 3. Malnourished children 24 – 59 months, 4. Persons living with AIDS (PLWA), 5. Orphans and other vulnerable children (OVC), 6. Farmer associations and lead farmers 7. Community-based organizations (CBO). Results Frameworks and Major Project Activities As illustrated in the program’s results framework below, the goal of World Vision’s MYAP was to reduce insecurity and increased resiliency of vulnerable and extremely vulnerable rural households. The program was structured around two strategic objectives SO1: Improved Nutritional and Health Status of Targeted Vulnerable Groups; and SO2: Improved Productive and Profitable Livelihoods. SO1 was to be achieved through 3 intermediate results focusing on improving nutritional and health practices, improving the quality of and access to health services, and decreasing risks of communicable diseases. SO2 was to be achieved through increasing food production and household assets, enhancing market-based livelihoods, rehabilitating natural resources resiliency and local response capacity, and enhancing program flexibility and community response capacity to acute needs. The following table summarizes the results framework and associated activities for World Vision’s MYAP. A series of intermediate results (IRs) establish the causal link between individual program activities and the goal of the overall program through two strategic objectives. Table 3: World Vision’s MYAP results framework Goal Strategic Objectives Intermediate Results (IR) Goal: Reduce food insecurity and increased resiliency of vulnerable and extremely vulnerable rural households. SO1: Improved Nutritional and Health Status of Targeted Vulnerable Groups IR1.1: Improved Nutritional and Health Practices of Targeted Vulnerable Populations IR1.2: Improved Quality of and Access to Health Services IR1.3 Decreased risks of communicable diseases among targeted communities SO2: Improved Productive and Profitable Livelihoods IR2.1: Increased Food Production and Household Assets IR2.2 Enhanced Market-Based Livelihoods IR2.3 Rehabilitated Natural Resources Resiliency and Local Response Capacity IR2.4 Enhanced program flexibility and community response capacity to acute needs. The critical assumptions and risks that underlay World Vision’s approach to addressing food insecurity as outlined in their 2007 MYAP proposal (World Vision. 2007) included: • Monetization and the capacity of national markets to absorb commodities • Stability of the Haitian currency 6 • Strengthened local partners (government and associations) • Poor transportation infrastructure • Continued GOH support • A stable security situation • A stable economy • Continued availability of USG funding. In light of Haiti’s post-earthquake context, the recent cholera outbreak, the continuous deterioration of economic and environmental conditions, as well as the benefits that could be gained by incorporating several mid-term evaluation recommendations into the program, the original MYAP agreement with World Vision was amended in 2011. The amendment was designed to increase overall program effectiveness and impact, as well as incorporate surge capacity into the design. As a result, World Vision added two new intermediate results to the existing program (IR1.3 and IR2.4) and the inclusion and expansion of other program interventions to respond to the mid-term evaluation recommendations and the changed country context. 7 EVALUATION METHODS & LIMITATIONS The evaluation methodology is based on a mixed data collection approach combining quantitative and qualitative data collection as follows: 1) review and analysis of relevant documents; 2) an extensive quantitative survey, with random sampling of households, conducted throughout the MYAP areas to measure progress since 2008 when the MYAP baseline survey was conducted; 3) as part of the randomized survey, measurement of the anthropometric indices of nutrition status of 1912 children; 4) qualitative surveys involving extensive focus-group discussions; and 5) field visits and key informant interviews, including discussions with implementing staff and local authorities. It is important to note that quantitative surveys were used to compare program effectiveness in reference to data on identical indicators collected in a 2008 baseline survey. To enable valid comparison over time, the survey conducted for this evaluation is based on the same sampling methodology used in the baseline survey. (A detailed methodology for conducting the quantitative and qualitative surveys used in the evaluation is provided as an annex to this report.) Since this evaluation was designed as a performance evaluation, the methodology will be limited to a comparison of the targeted population before the MYAP program was introduced in 2008 (as reflected in the baseline survey) to itself at the end of the MYAP program in 2013 (as reflected in the end-line survey conducted for this evaluation). Without a counterfactual (a control or comparison group), causal inference between the program and the outcomes under consideration, will be limited to “plausible contributions,” not as firm conclusions that program interventions are the only cause of observed results. This is important because since the 2010 earthquake, the number of actors engaged in relief and/or development activities in the World Vision zones of intervention has increased significantly in all sectors, including health, agriculture, natural resources management, and infrastructure. This evaluation is based on a mixed data collection approach using a quantitative and qualitative survey. The qualitative survey included 122 Focus Group participants in World Vision areas. It also used rapid appraisal methods based on key-informant interviews. Although those methods may shed some light on the quantitative results and are otherwise critical in analyzing some of the evaluation questions, they have several limits with regard to their reliability and validity, including: • Informal sampling can lead to imperfectly representative samples • Lack of unambiguous validation procedures to test the answers • Researchers’ inability to go beyond what is reported by informants • Individual biases of the informants • Individual biases of the evaluators Another limitation is that since the evaluation was conducted at the end of the field program (August-September 2013), many MYAP program office staff and most of the field staff were not available for interviews. 8 ORGANIZATION OF THE REPORT The remainder of the report is structured around the evaluation questions listed in this introduction. Since the cost-effectiveness question deals with two Mothers’ Club models from two separate CSs – by comparing the two models and ranking them in terms of their relative cost-effectiveness -- the cost-effectiveness question will be analyzed only in the overall evaluation report. Each subsection is divided into findings, conclusions and recommendations, but conclusions and recommendations appear only as applicable. Findings focus on empirical facts based on data collected during the evaluation. Conclusions synthesize findings and render judgments based on one or more findings. Recommendations are specific actions proposed by the evaluation team based on the findings and conclusions. 9 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS QUESTION 1: HOW DID THE FOOD SECURITY STATUS OF THE POPULATION CHANGE? According to USAID Policy Determination Number 19, “food security exists when all people at all times have both physical and economic access to sufficient food to meet their dietary needs for a productive and healthy life.” The policy determination identifies three food security dimensions as follows: Food availability: sufficient quantities of food from household production, other domestic output, commercial imports or food assistance Food access: adequate resources to obtain appropriate foods for a nutritious diet, which depends on income available to the household, on the distribution of income within the household and on the price of food Food utilization: proper biological use of food, requiring a diet providing sufficient energy and essential nutrients, potable water and adequate sanitation, as well as knowledge within the household of food storage and processing techniques, basic principles of nutrition and proper child care and illness management. To measure the food security status in their areas of intervention, the three CSs conducted a joint baseline survey in 2008 using 14 indicators in five activity areas: child health and household hygiene; child nutrition; household food security; agricultural production practices; and family planning and antenatal/post-natal care. A survey was conducted for this evaluation to measure progress in those areas at the end of the program. FINDINGS The overall World Vision program demonstrates improvement in 11 of the 14 indicators and in all indicators for 2 of the 5 activity areas. Table 4 describes the mean and percentage change relative to the baseline in the food security status of the population in all MYAP areas and in the World Vision MYAP areas. The results indicate that the overall World Vision program demonstrates improvement in 11 of the 14 indicators and in all indicators for 2 of the 5 activity areas. The strongest improvements are in family planning and antenatal/post-natal care and agricultural production practices. No specific differences in trends can be detected when comparing the MCHN/agriculture areas than for the MCHN-only strata. 10 Table 4: Coverage Estimates for All Indicators Measured: World Vision Program Activity Area Indicator Total MYAP World Vision Total MCHN MCHN/Agriculture Mean change Mean change Mean change Mean change Child health and household hygiene % children 12-23 month fully immunized 72.1 15.3% 74 6.4% 72.2 14.4% 75.4 -2.2% % children 12-23 months vaccinated against measles 62.7 -5.3% 59.4 -18.9% 51 -17.9% 66.3 -21.5% % of caregivers of children 0-59 months reporting washing hands with soap at least 2 times in the day preceding the interview 26.7 24.5% 23.3 21.7% 24.1 23.2% 22.6 20.4% Child nutrition % children 0-59 months underweight 9.7 -10% 10.7 -12.3% 10.7 -14.6% 10.7 -4.2% % children 6-59 months stunted 22.8 0.7% 25.3 0.6% 25.4 -0.9% 25.2 5.9% % children 6-59 months wasted 6.8 1.9% 6.3 -0.5% 6.5 -0.9% 5.5 0.8% IYCF indicator 23.2 4.5% 30.2 12.1% 29.6 13.2% 33 6.7% Household Food Security Months of adequate household food provisioning 3.9 -1.7 3.5 -2.4 3.5 -2.7 3.5 -2.1 Household dietary diversity score 5.7 0.2% 5.2 0.2 5.3 0.3 5.1 0.1 Agricultural production practices % farmers using at least 3 sustainable agriculture practices 89.9 14.8% 87 22.8% 89.2 20.1% 85.1 25.6% % farmers using at least 4 sustainable agriculture practices 79.2 25.1% 73.9 30.2% 77.9 33.7% 70.3 27.1% Family Planning and Antenatal/Post Natal Care % currently married/in union women 15-49 y using a modern family planning method 53.1 14.1% 58.3 15.2% 65.5 20.8% 52.1 8.9% % mothers of children <2 years whose last delivery was 45 20.2% 48.2 18.4% 44.8 12.1% 51.7 36% 11 attended by a trained professional % mothers of children <2 years who had at least 3 prenatal care visits by a trained provider during their last pregnancy 83.7 7.2% 91.5 8.2% 87 4.3% 95.4 9.2% Source: Evaluation survey Detailed results by activity areas are as follows: Child health and household hygiene Although children immunization rates have slightly increased (6 percent) since 2008 and are now slightly above the mean immunization rate for the other populations assisted under CRS and ACDI/VOCA food aid (72 percent), they remain well below the 80 percent coverage necessary to attain herd immunity.2 Measles vaccination rates have decreased by nearly 20 percent and are below the rate for all MYAPs. It is difficult to understand how World Vision allowed a decline during this activity in the delivery of the most important vaccine, measles. This is in contrast with hand washing, which improved by over 20 percent, although still below the rate for all MYAPs. Figure 2: Child health and hygiene results of quantitative data from household surveys Child nutrition As for all other MYAP programs, the picture for child nutrition is mixed. While two indicators improved significantly (underweight and infant and young child feeding -- IYCF), the other two 2 Herd or community immunity occurs when the vaccination rate reaches a high enough portion of a population that diseases do not spread and therefore provides protection for even the unvaccinated individuals who have yet to develop immunity. 0% 50% 100% Full imunization Vaccination for measles Hand washing World Vision: Child Health and Household Hygiene 2008 2013 12 (stunting and wasting) remained about the same (stunting) or retreated slightly (wasting). Although the IYCF indicator has increased by 12 percent since 2008, only less than one out three children is being fed according to appropriate feeding practices. Figure 3: Child malnutrition and feeding practices Stunting increased over the program period, which is surprising given the focus on feeding pregnant women, young mothers, and young children. Short term, or acute, malnutrition (weight/ht) did not change significantly over the period of the program, but it was low to begin with, just under 5% of children, thus at roughly the same rate as other middle class nations and arguably at an equilibrium level. That it did not increase significantly over time suggests the success of the preventive approach. The change from 2008 in wasting was neither statistically significant nor noteworthy because in many countries the “equilibrium” minimum level that is often aimed for is 5 percent, which is the lower end of the surveys’ confidence interval and which is roughly the same as many states in the United States. In between the long-term and short-term malnutrition rates is the more generic measure of under￾nutrition, the weight/age (wt/age) measure. Most interestingly, this measure of malnutrition declined remarkably from almost 20 percent to below 10 percent, probably as a result of the food assistance program. This large drop was statistically very significant and represents the largest benefit seen from the MYAP programming. Figure 4: World Vision population malnutrition rates compared to other populations 0% 5% 10% 15% 20% 25% 30% 35% Stunting (HAZ) Underweight (WAZ) Wasting (WHZ) IYCF World Vision: Child Nutrition 2008 2013 13 Analysis of the 2013 household surveys about possible gender associations with malnutrition found none. Malnutrition rates, across age cohorts, were roughly the same for boys and girls. There was a higher rate of weight-for-age malnutrition among infants under six as compared to weight-for-age among children aged 7 – 60 months, which strongly reflects low birth weight as opposed to infant feeding. It suggests that food rations targeted to pregnant women could have been increased with improved health outcomes. Another analysis of the relationship in the survey sample of the association between wasting and stunting found that stunted children were roughly half as likely to be wasted (low weight for height) as well. Put another way, the percentage of wasting was 3.7 percent among those children with long-term malnutrition, i.e. while the percentage of wasting was 7.0 percent among those children who were not stunted. This association was very significant, at the p<.01 level. In other words, in Haiti, low height attainment may, in a perverse way, protect children against the harms and hazards of wasting. Household food security Household-level indicators suggest that households in the World Vision areas remain very food insecure, having enough food provisions to feed themselves adequately only 30 percent of the time (an average of 3.5 months out of 12 in the last year). The household dietary diversity score (measured by the number of food groups consumed by the household in the last 24 hours) stands at 5 on average, lower than for the total MYAP areas and the minimum of 6 groups recommended by UNICEF (2010). Figure 5: Household food security measures over time 0% 5% 10% 15% 20% 25% 30% Stunting (HAZ) Underweight (WAZ) Wasting (WHZ) World Vision: Anthropometric Indicators in Perspective DHS Rural (2012) World Vision(2013) 14 Agricultural production practices Adoption of improved agricultural practices has improved significantly since 2008, but remains below total MYAP performance. Nearly 9 of every 10 farmers (87 percent) already have adopted three or more recommended agricultural production practices. This figure is reduced only to 74 percent when considering four or more practices. Figure 6: Uptake of agricultural practices and innovations over the 5-year program Family planning and antenatal/post-natal care 0 1 2 3 4 5 6 7 Months adequate food provisonning Household Food Security score World Vision: Household Food Security 2008 2013 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% At least 3 sustainable practices At least 4 sustainable practices World Vision: Improved Agricultural Practices 2008 2013 15 The contraceptive prevalence rate indicates that over one in two women (58 percent) was using modern family planning methods after 5 years of MYAP activities, a rate 15 percent higher than the contraceptive prevalence rate in 2008. One in two women now receives professional assistance at delivery, an increase of 18 percent from 2008. The new rate is now higher than the MYAP program as a whole (48 percent vs. 45 percent). At over 90 percent, the proportion of women who made at least three prenatal visits is now also higher than for the MYAP program as a whole. Figure 7: Family planning services, birth attendance and health for pregnant women CONCLUSION The indicators for improved agricultural production practices, family planning and antenatal and prenatal care have shown significant improvement. Other results, especially those for child nutrition and household food security demonstrate that there is ample room for improvement. That chronic under-nutrition (stunting) remains about the same as in 2008 is of a particular concern. RECOMMENDATIONS To adequately protect child health, immunization rates must be brought up to at least 80 percent. Hygiene and cleanliness in the household should continue to be promoted to reduce infectious diseases. Special attention should also be paid to raising measles immunization rates, which have regressed by nearly 20 percent since 2008. Actions to reduce chronic under-nutrition or stunting are of outmost importance. Improving child health and household hygiene and improving the household dietary score as well as its composition will be essential. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Modern family planning Delivery by trained professional At least 3 pre-natal care World Vision: Family Planning and Antenatal/Postnatal Care 2008 2013 16 However, the specific interventions to be implemented should be based on further research, particularly regarding the local foods given to infant and young children to improve the quality of their diet and the micronutrient gaps, including the intake of vitamin A, iron, zinc, and fatty acids. Additional research may be needed to investigate why recommended agricultural practices have been adopted by nearly 90 percent of the population without a corresponding reduction in food insecurity (as reflected in a reduced adequate household food provisioning) or child under-nutrition (as reflected in stunting rates). Particular attention should be placed on linking home vegetable gardens, fruit trees, and small animal husbandry activities (e.g., chickens and goats) for Mothers’ Clubs to increase the intake of micronutrients through dietary divarication. More generally, closer integration of MCHN and agricultural activities should be a primary focus. The very high proportion of mothers who had at least three prenatal care visits by a trained provider during pregnancy is clearly a program success. However, the lower rates of assisted deliveries (less than one in two deliveries) need to be understood so that appropriate corrective actions can be carried out. QUESTION 2: DID THE PROGRAM ADDRESS THE MOST CRITICAL PROBLEMS OR CONSTRAINTS TO FOOD SECURITY AND RESILIENCY FOR THE MOST VULNERABLE? FINDINGS From field visits, interviews with independent stakeholders and review of performance monitoring data, the evaluation found that the most critical constraints to nutrition security among the most vulnerable were addressed by the three CSs. The World Vision program targeted pregnant women, mothers of children under 2-years of age and those acutely malnourished between 24-59 months. The most critical constraints to nutrition security among the most vulnerable were addressed in three intermediate results under SO 1: improved nutritional and health status of targeted vulnerable groups through three intermediate results: improving nutritional and health practices of targeted vulnerable populations; improved quality of and access to health services; and decreased risks of communicable diseases. Targeted groups within World Vision’s MYAP included all pregnant and lactating women, children under 2 and those acutely malnourished between 24-59 months. For World Vision, in three intermediate results under SO 2: improved maternal and child health; increased access to nutritious foods; and improved household sanitation and access to water. These findings are primarily at the output level whereas outcomes are best measured in section three. Targeted groups in all MYAP programs included all pregnant and lactating women, children under 2 and those acutely malnourished between 24-59 months in their respective target communities. 17 Figure 8. Most common activities addressing food security, referenced in qualitative survey discussions A PM2A approach to improving maternal and child health and nutrition In line with the preventing malnutrition under-2 approach (PM2A),3 the major programmatic areas addressed by World Vision are: • Good nutrition during the 1,000 days around a child’s life (the “window of opportunity” from conception to 2 years of age). • Exclusive breastfeeding for 6 months and continued breastfeeding for at least 2 years • Adequate complementary feeding from 6 to 23 months, and malnourished children aged 24-59 months. • Adequate intake of vitamin A, iron and other micronutrients for women and children • Deworming, vaccinations, and diarrhea treatment • Clean water, hand washing and latrines • Family planning, counseling and education Applying PM2A is a preventive4 strategy that reflects widely accepted best practices to improve child nutrition.5 3 PM2A is defined as “a food-assisted approach to reducing the prevalence of child malnutrition by targeting a package of health and nutrition interventions to all pregnant women, mothers of children 0-23 months, and children under 2 in food- insecure program areas, regardless of nutritional status” (USAID/FANTA-2 2010b). 4 USAID/FFP defines prevention as follows: “As in any public health intervention, prevention means population-based coverage. As an analogy, consider the polio vaccine: all children in a population are entitled to, and should get the polio vaccine no matter the socioeconomic status of the household. Similarly, all children in a population with high stunting rates…are at risk of becoming malnourished during the 1000 days between conception and two years of age and thus should be protected from the ravages of nutritional deficiencies” (USAID/FFP 2012a). A study (Menon et al. 2007) conducted in Haiti over a three-year period in communities randomly select to receive a preventive approach of a Title II MCHN program, the prevalence of stunting, underweight and wasting respectively was 4, 6 and 4 percentage points lower after three years of operation compared to communities exposed to the recuperative program approach. The preventive approach was also more cost-effective. 5 There is consensus that the most effective way to positively affect childhood nutrition is to focus efforts on the 1,000 18 According to UNICEF recommendations, children should receive nothing but breast-milk (exclusive breastfeeding) for the first six months of life. Complementary foods should be introduced when a child is six-months old to reduce the risk of malnutrition. UNICEF also recommends that breastfed children age 6–23 months be fed four or more other food groups daily. Non-breastfed children should be fed milk or milk products, in addition to four or more food groups. Infant and Young Child Feeding (IYCF) also recommend that children be fed a minimum number of times per day (UNICEF 2010). The provision of micronutrients and vaccinations is also essential. Applying best practices, both internationally and in Haiti (see, for instance, World Bank 2008; Horton 2008; GOH/MSPP 2012d), World Vision’s interventions included provision of immunization services, vitamin A distribution, oral rehydration salt and zinc, and de-worming to children under 5. Reflecting the same best practices, clean water and improved sanitation were also included in the program. Family planning, counseling and education contribute to maternal and child health and nutrition. In addition to reducing the risk of neo-natal, infant, and under-5 mortality, healthy timing and spacing of pregnancies is associated with improved nutritional status of children. The longer the interval between births, the less likely a child is to be stunted or underweight (USAID/FFP 2010b). Food rations are an integral part of PM2A. Rations help prevent malnutrition by supplementing and improving the quality of the diets for pregnant women, mothers of children 0-5 months of age and children aged 6-23 months. The household ration is intended to supplement the family’s food supply and provide an incentive for program participation. The following table summarizes some of the key program activities and outputs in support of PM2A. day ‘window of opportunity’, the 9 months preceding a child’s birth and the 2 years following a child’s birth. There is consensus that the damage to physical growth, brain development, and human capital formation that occurs during this period is extensive and largely irreversible. Any interventions after this critical period are much less likely to improve nutrition. Starting at birth, improved nutrition yields benefits that cascade through life and even future generations. Under-nutrition affects health and survival through higher mortality and morbidity among neonates, infants, and children, with losses in the future of economic output and increased future spending on health (see, for instance, USAID/FFP 2010b; World Bank 2006a). 19 Table 5: Key World Vision program activities and outputs in support of PM2A • 750,969 targeted beneficiaries reached • 185,857 eligible children involved in USG supported Growth Monitoring and Promotion System • 21,353 postpartum/newborn visits within 3 days of births in MYAP-assisted programs • 61,001 antenatal care visits by skilled providers from USG-assisted facilities • 1,409 services providers trained in maternal/newborn health through MYAP-supported programs • 1,383 service providers trained in child health and nutrition through MYAP-supported health area programs • 117,188 children reached by USG-supported nutrition programs • 56,343 children less than 12 months of age who received DPT3 from MYAP-supported programs • 186,775 children under 5 years of age who received vitamin A from MYAP-supported programs • 20,639 cases of child diarrhea treated in USAID-assisted programs • 8 health facilities rehabilitated • 20,292 receiving at least one post-natal consultations • 49 MSPP personnel trained with USG assistance • 18,481 mothers or caretakers of less than two year-old children trained in exclusive breastfeeding, continued breastfeeding and complementary feeding • 5,390 deliveries attended by skilled birth attendants health personnel • 37,721 people trained in FP/RH • 19,553 counseling visits for FP/RH as a result of MYAP assistance • 61,996 people that have seen or heard a specific FP/RH message • 168 MYAP-assisted service delivery points providing FP counseling or services • 85 communities with hygiene promotion campaigns implemented • 82 communities supplied with oral rehydration and water purification materials • 620 health promoters trained in integrated management of childhood Illness • 128 communities with integrated management of childhood Illness management Source: calculations using data in World Vision program’s IPPT and PMP Management of acute malnutrition World Vision’s MCHN component included an activity to treat severe malnutrition at the household level, via a “Community Management of Acute Malnutrition (CMAM) component. When severely malnourished children 24-59 months of age are identified as malnourished at rally posts, mothers are asked to take them to a health center where they receive a ready-to-use therapeutic food (RUTF), which they can take home to feed the child with every day. They are also eligible to receive, along with children with moderate malnutrition, a more basic monthly dry ration of grains and beans at distribution points.6 Since children with severe acute malnutrition are at high risk of death, it is recommended (see, for instance, FANTA-2 2010b) that PM2A programs, which emphasize frequent, regular contacts with children, include mechanisms to screen for severe acute malnutrition, using mid-upper arm circumference (MUAC) and then to refer cases of malnutrition to appropriate treatment programs. 6 In answering the question of why the MCHN objective states that the objective of preventive programs is the reduction of chronic malnutrition in children under five when program implementers are asked to focus on children under two, USAID/FFP explains that” the broader age range is given in the objective because although implementation activities will focus on under-twos, reductions in the prevalence of malnutrition need to be measured in the under-five reference population in the program area. This is because capturing the impact of a Title II program in terms of reductions in the prevalence of malnutrition takes time, and several cohorts of children under-two need to graduate from the Title II program in order to show impact at a population level”(USAID/FFP 2012a). 20 If such programs are note available, World Vision should advocate for or create a separate but linked program for treating severe acute malnutrition. Through its agriculture and livelihoods activities, the World Vision program addressed the most critical constraints to increased food security as described in the Haiti poverty reduction strategy. Under its SO2: Improved Productive and Profitable Livelihoods, the program aimed at increasing food production and household assets, enhancing market-based livelihoods, and rehabilitating natural resource resiliency. To achieve these results, the program focused on cropping systems and improved productivity; enhanced seed production; improved storage practices; vegetable gardening and household production diversification; multiplication and breeding centers for small animals; agroforestry and household production diversification, promotion of fruit trees on small parcels of vulnerable groups; and promotion of natural resource resiliency. Over the past five years, World Vision’s MYAP provided those services to nearly 30,000 households (see table below). The following table summarizes some of the key program activities and outputs in support of the agricultural productivity, environmental management and market linkages program. Table 6: Key World Vision program activities and outputs in support of increasing food production and household assets, enhancing market-based livelihoods, and rehabilitating natural resource resiliency • 36,682 targeted beneficiaries reached • 21,614 producers using a project-defined minimum number of sustainable agriculture technologies • 29,398 households benefiting directly from USG interventions • 1,535 households with adequate grain and seed storage facilities • 17 technologies or management practices made available for transfer as a result of USG assistance. • 1,675 additional hectares under improved technologies or management practices as a result of MYAP assistance • 28,238 vulnerable households benefiting directly from MYAP assistance. • 27,878 rural households benefiting directly from MYAP assistance • 107 producer organizations, water user associations, trade and business associations, and community-based organizations receiving USG assistance • 24,635 individuals who have received MYAP-supported short-term agricultural sector productivity training • 19,147 beneficiaries adopting a minimum number of technologies • 3,275 households with cash crop market-based production • 576 Number of women’s organizations/associations assisted as a result of MYAP- supported interventions • 16 MSMEs receiving business development services as a result of MYAP assistance • 60 savings groups trained and monitored • 23 entrepreneurs trained in business management • 4,793 people with increased economic benefits derived from sustainable natural resource management and conservation as a result of MYAP assistance • 1,946 people receiving MYAP-supported training in natural resources management and/or biodiversity conservation • 401 people trained in disaster preparedness as a result of USG assistance • 53 assisted communities with disaster early warning and response system in place • 1,069beneficiaries trained in watershed protection Source: calculations using data in World Vision program’s IPPT and PMP The agriculture and livelihoods program was in line with the 2007 Haiti Poverty Reduction Strategy that was endorsed by the World Bank and the International Monetary Fund in 2008. Two major 21 components of the Haiti poverty reduction strategy were to support agriculture and rural development, and to promote private sector development, especially small-scale farmers in vulnerable areas -- two World Vision program objectives. In line with the Haiti poverty reduction strategy, World Vision emphasized sustainable agriculture that benefited small farmers and helped feed local communities. This is all the more important because over 90 percent of the population depends on subsistence agriculture for their livelihoods in the program areas (Agrifeeds 2013). Agriculture in those areas consists mainly of small-scale subsistence farms with an average size of less than 1 ha, where 80 percent of farms fail to produce enough to feed household members and live in a setting where a vicious circle of environmental degradation, little available technology and credit, and weak market infrastructure make the development of sustainable agricultural income difficult (IFAD 2013). IFAD also notes that the overall incidence of poverty in Haiti is 77 per cent. But in rural areas, which are home to 52 per cent of Haiti’s population, 88 per cent of people are poor and 67 per cent are extremely poor – especially women who are heads of households, and fishers who do not have their own boats, rural workers who depend exclusively on wage employment, and landless farmers (sharecroppers). In addition, rural people have a per capita income that is about one third of the income of people living in urban areas. Figure 9: Participation findings from Qualitative Surveys The four bars on the left indicate responses about who are participating, consistent with the program’s objectives and targets, while the two bars on the right point to groups whom the beneficiaries felt ought to be targeted or eligible but were not. CONCLUSIONS World Vision did address the most critical problems and constraints to food security and resiliency for the most vulnerable population groups. With its focus on MCHN activities for pregnant and lactating mothers and their under-5 children, World Vision’s approach was consonant with a convergence of research on the most cost-effective way to address nutrition. It was also in line with 22 the county’s national nutrition policy, which features prevention, management of acute malnutrition and nutrition protection in emergency situations. World Vision’s focus on helping vulnerable communities to achieve increased farm and off farm income reflected the country’s poverty reduction strategy that was endorsed by major international development agencies the same year the program started. World Vision’s agricultural and livelihoods interventions did address the most critical problems facing small-scale and landless farmers in the program areas -- reflecting the country’s poverty reduction strategy that was endorsed by major international development agencies in the same year the MYAP started. QUESTION 3: ARE THE CONSTRAINTS FACED BY THE TARGET BENEFICIARIES AS OUTLINED IN THE ORIGINAL DOCUMENT STILL RELEVANT? FINDINGS The need for further improvement in the food security status of the target population remains high. As detailed in evaluation question 1, key results indicators, especially those related to child nutrition and household food security demonstrate that there is ample room for improvement. The need for further progress in the program area and throughout the country is further illustrated by the Global Hunger Index (GHI) score for Haiti, which the International Food Policy Institute describes as “alarming” (IFPRI 2013). The situation is all the more worrying because the Haiti GHI score has shown only a very minor improvement over the past decade (from 25.7 in 2000 to 23.3 in 2013, for only a 9 percent improvement).7 A series of natural shocks and social and political events have adversely affected food security for the most vulnerable in the MYAP areas and throughout the country since 2008: a food crisis, street demonstrations and a hurricane in 2008; floods and storms in 2008 and 2009; election violence in 2009 and 2010; a devastating earthquake and a cholera epidemic in 2010; and a tropical storm, a hurricane and a food crisis in 2012. These shocks have resulted in de-capitalization of human and physical household assets, increased vulnerability,8 unpredictable increases in food prices; damage to infrastructure; crop damage and other agricultural losses; reduced income opportunities; negative coping strategies such as distress sales of productive livestock (working or breeding animals), eating very little food, selling agricultural or fishing equipment, and overexploiting natural resources; and often severe food insecurity. As 7 The Global Hunger Index (GHI) is a tool designed to comprehensively measure and track hunger globally and by region and country. Calculated each year by the International Food Policy Research Institute (IFPRI), the GHI highlights successes and failures in hunger reduction and provides insights into the drivers of hunger, and food and nutrition insecurity. The index describes hunger as low, moderate, serious, severe, and very severe. 8 The worst shocks reported in a 2011 survey in Haiti (Echeverin 2011) are (in that order) as follows: disease/accident of a household member; death of a household member; cyclone, flood; increase in food prices; animal diseases; irregular rainfall; and crop diseases. 23 reported by WFP, the food consumption score for Haiti9 has decreased to its lowest level in over a decade (WFP 2013a-c). Polling data from across the target populations, as detailed in evaluation question 1, key results indicators, especially those related to child nutrition and household food security demonstrate that there is ample room for improvement. Household survey and qualitative evidence all confirm that food insecurity and poor access to health services remain constraints to the food security progress of the beneficiaries in all the MYAP area of operation. Food security and nutrition remain an important national concern. The government of Haiti has recognized that chronic neglect, malnutrition rates and poverty have resulted in precarious conditions for Haiti’s rural poor. Emergencies are recurrent in Haiti, often jeopardizing potential gains in the fight against malnutrition. Because of chronic food insecurity and the various shocks that occurred in recent years, the Government of Haiti has emphasized the seriousness of the food security situation and designed programs to address it. Implementation of the new National Policy on Nutrition (Aba grangou or “down with hunger” in Creole) and a national restructuring of the local response to household health issues (Kore Fanmi) are underway and are meant to address the root causes of food insecurity for the rural population (GOH 2010; 2011).10 The constraints faced by the beneficiaries targeted for agricultural and livelihoods assistance as outlined in the original program document remains considerable. The continued relevance of the constraints faced by the beneficiaries targeted for agricultural and livelihoods assistance is reflected in the fact that the Government of Haiti considers agriculture in the most vulnerable areas as a strategic sector in the country’s economy and is reflected in its commitment to increase Haiti’s capacity to meet 60–70 percent of its food security needs by 2017 through agricultural development and natural resource management (GOH/MARNDR 2012). Some constraints were new and unanticipated, including displaced and homeless people after the 2010 earthquake: “I was a victim of the earthquake, my home was destroyed, I did not even have clothes when I came here. I came home it was thanks to this project that I participated in a seminar and now I have succeeded and I am working …and receive a monthly salary.” (World Vision focus group discussion participant) Some beneficiaries may argue that their opportunities have been transformed and that they are on the road to graduate out of poverty. In one World Vision area, beneficiaries said “A project in 9 Used by CNSA, the food consumption score for measuring food insecurity in Haiti combines three indicators: the food consumption score, the hunger scale and the dietary diversification score. 10 Aba grangou is a Presidential flagship initiative to lead the new government’s efforts in combating of hunger and malnutrition in Haiti. Launched in 2012, this nationwide program is financed with $30 million from Venezuela's PetroCaribe fund. It aims to halve the number of the hungry population by 2016 and eradicate hunger and malnutrition altogether by 2025. Under Aba grangou, an estimated 2.2 million children are supposed to take part in a school feeding program. The Kore Fanmi (‘family support’ in Creole) Model is an initiative aiming at improving the efficiency of social services delivery. Through the use of Community Agents (Agents Communautaires Polyvalents), this approach aims to build on the experience of collaboration with existing community agents (agents de santé, ColVols), particularly in rural areas, and to implement an integrated and innovative delivery of services in partnership with donors and NGOs, in order to provide health and nutrition services directly to vulnerable families. 24 which 160 people received credit. The credit program was for petit merchants who were very happy to receive about 1,558 Haitian dollars. They used this money to start businesses. You will find people who have little goats, cows, pigs as well. And they can tell you what they did with these things.” CONCLUSION The quantitative survey conducted for this evaluation demonstrates that much remains to be done to improve child nutrition and household food security (see evaluation question 1). The natural shocks and other events (floods, storms, hurricanes and droughts) that have set back food security for the most vulnerable in the MYAP areas and throughout the country since 2008 are likely to continue. The following critical constraints for food security and resiliency for the most vulnerable have significantly decreased after the project: • Insufficient agricultural production (yield based): addressed well through: 1) improved varieties of corn, beans and the demonstration of better agricultural techniques (although were seed producers do not have irrigation, they face problems for seed multiplication); 2) dissemination of new or improved tubers (cassava, sweet potato and yams). • The few irrigated perimeters which the project assisted are now rehabilitated and under good group management with support from the BACs (Bureau Agricole Communal). Although by definition, those with access to irrigated perimeters are not the “most” vulnerable. • Credit constraints: MUSOs help reduce food insecurity and increase resilience by providing a savings mechanism for all and flexible credit for those who need it. In MUSO areas, people no longer use expensive money lender credit. • Food diversification through: 1) vegetable gardens both for home use and for sale; 2) other cash crops: fruit trees, chili peppers, onions; 3) bee-keeping • For these reasons, food security in Haiti remains a national concern and the constraints faced by the beneficiaries targeted for nutrition and livelihoods assistance as outlined in the original program document remain as daunting as before. QUESTION 4: TO WHAT EXTENT DID THE PROGRAM’S THEORY OF CHANGE CONTRIBUTE TO THE MYAPS ACHIEVEMENTS IN TERMS OF PROJECT RESULTS AND OUTCOMES? FINDINGS The main changes observed from field research, key informant interviews and discussions with local communities, including beneficiaries, found an emphasis on vegetable gardens, fruit and vegetable diversity in diets, more income opportunities, and the control over their lives that family planning and credit (through MUSOs) offered women. 25 Figure 10: Main impacts reported from food assistance activities World Vision’s theory of change draws on both in-country factors and standard theory and analysis in the two technical sectors underlying its strategic objectives and intermediate results. A theory of change -- also referred to as development hypothesis or results framework (USAID 2010)11 – is a tool that helps program managers identify clearly defined objectives within a complex development environment and shows, often in graphic representation, how a series of early and intermediate achievements sets the stage for generating longer-term results. It articulates the critical assumptions – also expressed as risks or vulnerabilities (USAID/ADS 201.3.8.3) -- about the process through which change will occur, and specifies the ways in which all the required early and intermediate outcomes related to achieving the next higher order of change will be brought about and documented as it occurs. World Vision’s results framework is summarized in the table below. Table 7: World Vision’s MYAP Results Framework Goal Strategic Objectives Intermediate Results (IR) Goal: Reduce food insecurity and increased resiliency of vulnerable and extremely vulnerable rural households. SO1: Improved Nutritional and Health Status of Targeted Vulnerable Groups IR1.1: Improved Nutritional and Health Practices of Targeted Vulnerable Populations IR1.2: Improved Quality of and Access to Health Services IR1.3 Decreased risks of communicable diseases among targeted communities SO2: Improved IR2.1: Increased Food Production and Household Assets 11 USAID 2010. Performance Monitoring and Evaluation: Building a Results Framework, TIPS Number 13 26 Productive and Profitable Livelihoods IR2.2 Enhanced Market-Based Livelihoods IR2.3 Rehabilitated Natural Resources Resiliency and Local Response Capacity IR2.4 Enhanced program flexibility and community response capacity to acute needs. World Vision’s program has laid out a series of critical assumptions and risk management areas including monetization risk; political stability; security; infrastructural challenges; and institutional constraints. As detailed in the evaluation questions on program relevance, the CS’ program has addressed some of the most critical constraints in agriculture, and the MCHN program has followed scientifically proven best practices and cost-effective interventions, including good nutrition during the 1,000 days around a child’s life; exclusive breastfeeding for 6 months and continued breastfeeding for at least 2 years; adequate intake of vitamin A, iron and other micronutrients for women and children; deworming and vaccinations; clean water and sanitation; and family planning, counseling and education. World Vision integrated agriculture and MCHN in one of its target areas. Experience from other countries (Rodgers et al. 2012) indicates that program components that integrate nutrition and agricultural interventions are likely to yield positive results. Focusing on MCHN as a priority intervention is a major resource allocation advantage. There is a remarkable convergence of scientific knowledge now available showing the high resource gains of MCHN interventions through lower mortality and morbidity, lower susceptibility to obesity and non-communicable diseases, human capital building, productivity gains through higher earning potential, and intergenerational benefits.12 The 2004, 2018 and 2012 Copenhagen Consensus – consisting of a panel of experts, comprising distinguished economists, including several Nobel Laureates -- has consistently ranked nutrition interventions among the most profitable relative to the greatest global challenges, including agriculture, civil conflicts, climate change, communicable diseases, education, governance, migration, and trade reform. However, there remained perceptions among the some participants that unintended consequences were seen where MYAP goals and assumptions were at odds with one another. Some found an incongruity between the one stream of benefits including family planning, with a message to not have children, and then an increase in children because of the targeted food available for young children and pregnant women: “A short time after the food and condom distribution, the only all you see around is very young people carrying children.” Program Contributions to MYAP Achievements As detailed in evaluation question 1, the overall World Vision program demonstrated improvement in 11 of the 14 key indicators and in 2 of the 5 activity areas included in the baseline survey and the survey conducted for this evaluation. In addition, not counting baseline survey indicators and 12 For a comprehensive review of the evidence, see Rassas 2009. 27 indicators for which data were missing partially or for all years, the program met or exceeded its targets for 56 percent of the indicators.13 The following table shows the areas were the World Vision program made valuable contributions to overall MYAP achievements. Table 8: World Vision Program Contributions to MYAP Achievements Livelihoods and natural resources management • 36,682 targeted beneficiaries reached • 21,614 producers using a project-defined minimum number of sustainable agriculture technologies • 29,398 households benefiting directly from USG interventions • 1,535 households with adequate grain and seed storage facilities • 17 technologies or management practices made available for transfer as a result of USG assistance. • 1,675 additional hectares under improved technologies or management practices as a result of MYAP assistance • 28,238 vulnerable households benefiting directly from MYAP assistance. • 27,878 rural households benefiting directly from MYAP assistance • 107 producer organizations, water user associations, trade and business associations, and community￾based organizations receiving USG assistance • 24,635 individuals who have received MYAP-supported short-term agricultural sector productivity training • 19,147 beneficiaries adopting a minimum number of technologies • 3,275 households with cash crop market-based production • 576 Number of women’s organizations/associations assisted as a result of MYAP- supported interventions • 16 MSMEs receiving business development services as a result of MYAP assistance • 60 savings groups trained and monitored • 23 entrepreneurs trained in business management • 4,793 people with increased economic benefits derived from sustainable natural resource management and conservation as a result of MYAP assistance • 1,946people receiving MYAP-supported training in natural resources management and/or biodiversity conservation • 401people trained in disaster preparedness as a result of USG assistance • 53 assisted communities with disaster early warning and response system in place • 1,069beneficiaries trained in watershed protection MCHN • 750,969targeted beneficiaries reached • 185,857eligible children involved in USG supported Growth Monitoring and Promotion System • 21,353 postpartum/newborn visits within 3 days of births in MYAP-assisted programs • 61,001antenatal care visits by skilled providers from USG-assisted facilities • 1,409services providers trained in maternal/newborn health through MYAP-supported programs • 1,383service providers trained in child health and nutrition through MYAP-supported health area programs • 117,188children reached by USG-supported nutrition programs • 56,343children less than 12 months of age who received DPT3 from MYAP-supported programs • 186,775children under 5 years of age who received vitamin A from MYAP-supported programs • 20,639 cases of child diarrhea treated in USAID-assisted programs 13 Calculations using data in the program’s IPTT and PMP sets of indicators. 28 • 8 health facilities rehabilitated • 20,292 receiving at least one post-natal consultations • 49 MSPP personnel trained with USG assistance • 18,481mothers or caretakers of less than two year-old children trained in exclusive breastfeeding, continued breastfeeding and complementary feeding • 5,390 deliveries attended by skilled birth attendants health personnel • 37,721people trained in FP/RH • 19,553 counseling visits for FP/RH as a result of MYAP assistance • 61,996 people that have seen or heard a specific FP/RH message • 168 MYAP-assisted service delivery points providing FP counseling or services • 85 communities with hygiene promotion campaigns implemented • 82 communities supplied with oral rehydration and water purification materials • 620 health promoters trained in integrated management of childhood Illness • 128 communities with integrated management of childhood Illness management Source: calculations using data from World Vision’s IPTT and PMP By not fully integrating MCHN and agricultural programs in its MYAP areas, World Vision may not have fully capitalized on the synergy between the two interventions. By design, integration of agriculture and MCHN in World Vision’s program was applied to only one of its program areas. World Vision operated in areas where it implemented both MCHN and agricultural activities and areas where it implemented only MCHN activities. Focusing exclusively on MCHN in one of the two program target areas may have been suboptimal in terms of program overall benefits and its resource allocation. Integrating MCHN and agriculture throughout the program area would have further enhanced program results. In areas where World Vision implemented MCHN activities in concert with agricultural, natural resources management and livelihoods activities, integration did not prove to be successful because the various agricultural, natural resources management and livelihoods activities focused on target groups that were not always the same as the MCHN groups. In particular, not all mothers of under￾5 children benefited from the program’s agricultural component unless their children were severely malnourished. This approach is at variance with the remarkable convergence of scientific knowledge now available showing the high resource gains of MCHN interventions through lower mortality and morbidity, lower susceptibility to obesity and non-communicable diseases, human capital building, productivity gains through higher earning potential, and intergenerational benefits.14 World Vision’s approach is also at variance With USAID/FFP guidance15 that “PM2A, along with the rest of a Title II program’s MCHN component, should be consistently linked with the program’s agriculture and livelihoods components” (USAID/FFP 2010b). This guidance is corroborated by experience from other countries (Rodgers et al. 2004a; 2012) indicating that program components 14 The 2004, 2018 and 2012 Copenhagen consensus – consisting of a panel of economic experts, comprising the world’s most distinguished economists, including several Nobel Laureates -- has consistently ranked nutrition interventions among the most profitable relative to the greatest global challenges, including agriculture, civil conflicts, climate change, communicable diseases, education, governance, migration, and trade reform. 15 USAID/PM2A 2009 guidance, restated in 2010. 29 should be mutually reinforcing and that the integration of nutrition and agricultural interventions is likely to yield optimal results. World Vision’s performance indicators used to measure results have several shortcomings. Performance indicators measure the results identified in the theory of change. While results identify what the program sets to achieve, indicators define the standard by which those results will be measured. Results are used not only to assist program managers on focusing on the achievement of development results, but they also and most importantly provide objective evidence that results are being achieved. A review of World Vision’s program indicators used in the its performance monitoring plan (PMP) reveals four major shortcomings: At 71, the number of indicators placed a heavy burden on field staff to register, handle and report on the data associated with those indicators – a finding emphasized by both the 2010 mid-term evaluation and the 2012 USAID/Office of the Inspector General Audit report. In addition, 77 percent of those indicators are output or process indicators, rather than results indicators. Excluding the 14 indicators used in the baseline survey, only 3.5 percent of those indicators are results indicators. USAID’s ADS 203.3.3.1 requires that at least one indicator be used for each result, including the highest level objectives of the results chain. A review of the PMP indicates that no indicators were chosen to measure the strategic objectives. The indicators are not sufficiently gender-sensitive (see next evaluation question). No capacity-building or sustainability indicators are included in the PMP (see evaluation question on sustainability). CONCLUSIONS World Vision’s theory of change draws on both in-country factors and standard theory and analysis in the two technical sectors underlying its strategic objectives and intermediate results. However, the lack of integration between program components may have overall benefits. Evidence from other countries (Bolivia, Kenya, Honduras and India) suggests that the program may not have fully capitalized on the synergy between agricultural and natural resources management and MCHN objectives. The performance indicators used to measure results have several shortcomings, including a disproportionately high number of indicators; the predominance of output and process indicators, instead of results indicators to monitor progress; and the absence of indicators to measure capacity building and sustainability. Overall, the program has achieved significant results despite the fact that a large percentage of its targets were not met. However, it is difficult to assess with any degree of accuracy the extent to which the results achieved are attributable to the World Vision program, not least because attribution requires a valid counterfactual. Those results can, nonetheless, plausibly be associated 30 with program interventions -- although this association is further limited by confounding factors due the multiplicity of relief and development projects in the area, especially since the 2010 earthquake. RECOMMENDATIONS The next MYAP should continue to use both in-country factors and best practices as reflected in standard theory and analysis in the two technical sectors underlying its strategic objectives and intermediate results relative to the MCHN and agricultural and natural resources management program . The PM2A approach should be applied from the beginning throughout the program area. The next MYAP should focus on promoting better integration of the agricultural and natural resources management and MCHN components. In particular, the various agricultural, natural resources management and livelihoods activities should focus on the same target groups as the MCHN groups. Prioritizing agricultural and livelihoods activities toward mothers’ groups would be an important step in that direction. Performance indicators of the next MYAP should be anchored in strategic thinking about what must truly be achieved for program success. For this reason, they should be streamlined with a view to discarding what may not be useful. They should also focus on measuring results, rather than processes and output. QUESTION 5: HAVE WOMEN AND OTHER DISADVANTAGED POPULATION GROUPS WHO PARTICIPATED BEEN DIFFERENTLY AFFECTED (POSITIVELY OR NEGATIVELY) BY THE PROJECT? FINDINGS The majority of World Vision’s program activities over the MYAP period benefitted pregnant and lactating women and their under-5 children. Considered two of the most vulnerable and food insecure group, and under the principal of universal targeting, all pregnant and lactating women and their young children in their target zones were the primary targets of World Vision’s MYAP in the MCHN program areas, and thus benefitted the most from the program’s MCHN interventions. Qualitative interviews indicate that many beneficiaries believe that providing food rations to women and children has encouraged pregnancies. Other program participants have denied this claim. FFP has looked at this claim in other countries and concluded that “there is no evidence to indicate that providing preventive nutrition and health services, including food aid rations, to all pregnant females has a pro-natal effect,” and for this reason “would not recommend that any pregnant woman be denied participation in a program activity because she will have/has another child while participating in the activity.” USAID/FFP recommends that “individual development programs address these potential challenges through the educational component that addresses healthy pregnancy spacing, by working with partners to strengthen family planning services as well as by facilitating access to family planning services available in the food aid program area“(USAID/FFP 2012a). 31 One clear consequence of the program, from the Focus Group discussions, was that the status of women changed, they were more aware of their options, more empowered by the MUSOs and mothers’ groups, had their own income, and a sense of independence. “It empowered individuals and now they know they can do more.” “Women have mastered the family planning approach.” “Men were happy to see women as independent.” Although there was agreement in the Focus Groups that women now have the capability to control when to have children, there were also suspicions that women were intentionally having children because of the MYAP’s incentive, by feeding young children, during the life of the MYAP. Two other disadvantaged groups in the World Vision program were specifically targeted: OVCs and PLWAs. World Vision treated all women beneficiaries as vulnerable. Two other disadvantaged groups that received program assistance were orphans and vulnerable children (OVC) and people living with AIDS (PLWA). World Vision’s PMP tracked indicators for the number of persons from vulnerable groups benefiting directly with food rations from program assistance (children, OVC, pregnant and lactating women, and PLWA). The other disadvantaged groups of poor widows/widowers, the blind, and other handicapped people were not singled out for assistance. Not all MCHN women beneficiaries were included in World Vision’s Livelihoods activities. In 2010, the mid-term evaluation found a lack of coordination between the MCHN and the Livelihood components of World Vision’s MYAP, and recommended an “integration of health services and agricultural activities”, in an effort to include mothers of malnourished children into agricultural activities. In some areas World Vision successfully integrated these activities and were able to include women in livelihoods activities (agriculture, livestock, MUSOs). Despite their eligibility and the recommendation of the mid-term evaluation, some mothers were still neither involved in agricultural activities nor in MUSOs. Reasons given for this in the mid-term evaluation in 2010 and in focus group discussions and key informant interviews conducted for this evaluation include: women dropping out of livelihood programs due to new pregnancy or disease; World Vision’s training quota numbers were already reached; and lack of funding for agricultural and credit extension agents to support and train women farmers in all MYAP communes. World Vision’s approach is inconsistent with USAID/FFP, which recommends that a comprehensive Title II program should be “implemented in the same communities as the program’s other activities related to [food] access, availability, and utilization. Ideally, individual households will participate in as many different program interventions as possible. For example, families with a pregnant woman or a child under 2 might participate in a program’s income-generation activity in addition to PM2A” (USAID/FFP 2010b). World Vision’s PMP indicators were not sufficiently gender disaggregated. Gender-based information was not adequately tracked by World Vision. In its 2012 ARR, World Vision reported on a number of indicators that were purported to disaggregate gender information. Under IR1.1: Improved Nutritional and Health Practices of Targeted Vulnerable Populations, World 32 Vision reported on four gender-disaggregated indicators: • Number of services providers trained in maternal/newborn health through USG-supported programs (women/men) • Number of service providers trained in child health and nutrition through USG-supported health area programs (women/men) • Number of people trained in FP/RH (women/men) • Number of counseling visits for FP/RH as a result of USG assistance (women/men) However, only the fourth indicator tracks a direct result on direct program beneficiaries. Under IR 2.2 World Vision tracked the [n]umber of women’s organizations/associations assisted as a result of USG supported interventions. However, without specific data on what was the assistance and an accompanying indicator tracking the results achieved by the organizations as a result of that assistance, there is no evidence that these women’s organizations where able to benefit from the program. For the other indicators (e.g., number of savings groups trained and monitored) the data were not gender￾disaggregated. The significance of this shortcoming has acquired more relevance in the last two years of program implementation when USAID/FFP announced that all Title II programs are required to integrate gender, either as a cross-cutting or strategic objective and that gender must appear in the results framework (USAID/FFP 2011a). The guidance further states that “ gender-sensitive indicators need to show to what extent and in what ways the Title II program has met gender equality objectives in a given year sector and/or achieved results related to gender equality.” CONCLUSIONS World Vision provided assistance to PLWAs and OVCs, but women and young children benefited from World Vision’s program interventions the most. However, World Vision was successful in reaching only a portion of women with its livelihood programs (agriculture, livestock, MUSOs). Since too little gender-disaggregated data were recorded for livelihood activities in the performance monitoring system, it is not possible to estimate the extent fully which women benefited from those activities. RECOMMENDATIONS Women should be better integrated into livelihoods activities to better benefit from MCHN services. Better integration would provide additional resources to purchase food and pay for medicine and health care services. Gender-sensitive indicators should be developed and used according to USAID/FFP guidelines in the next Title II MYAP. Those indicators should measure differences in how men and women have benefited from or have been impacted by the program. 33 Other most disadvantaged groups among the most vulnerable (e.g., the blind, the orphans, and other handicapped people) should be included in program assistance. QUESTIONS 6-7: IS THERE ADEQUATE EVIDENCE SUGGESTING THAT THE PROJECT OUTCOMES ARE LIKELY TO BE SUSTAINED? WHAT WERE THE MAJOR FACTORS WHICH INFLUENCED THE ACHIEVEMENT OR NON-ACHIEVEMENT OF SUSTAINABILITY OF THE PROJECT? FINDINGS Qualitative evidence indicates that many processes, community linkages, outputs and outcomes are likely to be sustainable, though firmer evidence will require a follow up investigation. This finding is based on some consistency of views obtained across hundreds of interviews and Focus Group participants. Specific questions were asked in all the focus groups and key informant interviews about whether any activities, practices, skills or behavior would be likely to persist beyond the life of the project. Those answers which clustered around certain themes, across independent groups, provide some evidence pointing to likely sustainability. The findings here are also based on field visits and observations. The below figure 14 shows key informants’ and focus group participants’ perceptions of the most sustainable MYAP activities. For example, 15 percent of focus group participants perceived MUSOs as the most sustainable activity whereas the food safety was perceived as the least sustainable activity. Comparison of evidence from technical interviews, field observations and PMP data suggest that the MUSOs, vegetable gardens, livestock (goats and cows in particular), and fruit trees have high likelihood of sustainability. In contrast, activities related to natural resource management, reforestation and certain expensive agricultural practices (such as contour bunding, drip kits and metal storage silos) are unlikely to be sustained, based on the reality that they already showed lack of continuity during the five year program cycle. Among the agro-associations, some were strong and likely to sustain, whereas many others, particularly those without business plans, are likely to become dormant. 34 Figure 11: Program outcomes and activities deemed likely to persist Qualitative discussions revealed that much of the knowledge transfer, in gardens and agriculture as well as in health and nutrition would be intrinsically sustainable. Several participants explained “we will remember everything.” What was felt to be not sustainable. There were diverse views about what activities or outcomes were not expected to be sustainable, principally: • Soil conservation • Children who move and therefore fall out of health program coverage • Flood prevention • Ravines and other public works for infrastructure, for which there is no sense of ownership • Pigs distributed which were expensive, fragile and frequently died of diseases. • Concerns about some of the discipline within the MUSOs” “When they set a time to bring money for micro-credit it was not respected” • “There is still a need for a strategy to not just produce with as many seeds as possible but to protect trees” • “In the micro credit (project), we learned how to do marketing, accounting” World Vision's sustainability plan is not based on a clearly defined strategy or objective sustainability indicators. 35 According to USAID, “Sustainability is achieved when host country partners and beneficiaries are empowered to take ownership of development processes, including financing, and maintain project results and impacts beyond the life of the USAID project” (USAID 2012c). In its comments on World Vision’s 2007 proposal, USAID requested that World Vision regularly articulate its approach to sustainability. In its 2011 program amendment, World Vision proposed three sustainability leverage points: household and beneficiary capacity, institutional capacity, and civil society capacity to hold local authorities more accountable for service. 16 However, no detailed and clearly defined strategy was developed to implement these three leverage points. Even though World Vision appropriately identified the need to strengthen human capacity, it concentrated on training as the mechanism of choice and no clear plans (e.g., MOUs, specific partner agreements or action plans) were developed to ensure that GOH ministry staff at the local level (GOH/DSSE and MSSP) takes over the provision of the MCHN services that World Vision had provided and/or funded when the program ends. In interviews with the evaluation team, World Vision staff could not provide any specific timelines or sunset clauses for beneficiary group graduation, other than age-related criteria. No clear expectations for sustainability on the part of World Vision, beneficiaries, and community groups were developed and no specific and measurable mechanisms for guaranteeing the take-up of health service provision by local government health offices were developed. Experience from other countries (Rodgers et al. 2004a; 2004b) indicates that without sustained resources, continued focus on capacity building, and realistic incentives to increase motivation, sustainability can be an elusive goal. The withdrawal of food rations and other free MYAP-provided materials and supplies such as birthing kits, water treatment kits, and family planning products, without considering substitute incentives is likely to jeopardize the sustainability of program activities. For this reason, consideration of alternative incentive structures should be incorporated into program design. World Vision’s sustainability plan does not contain any sustainability indicators. The mid-term evaluation recommended a graduation or program phase-out plans for World Vision, based on a clear graduation strategy from specific localities within communes being worked in. It also suggested that the key criteria for such graduation should be based on objective measures, such as the reduction of malnutrition rates within a specific locality (around a rally post) to levels below a 16 In 2011 program amendment, World Vision states: “SAK REP’s sustainability strategy has been used to inform all program design decisions and thus has been an integral part of interventions from day one. The strategy focuses on building human capacity via three critical sets of leverage points: 1) HH capacity; 2) civil society capacity; and 3) institutional capacity. As visualized in the pyramid diagram, the base of this strategy rests upon SAK REP’s collaborative efforts with partners- FEWSNET, AgriDev and others, to reinforce HH resiliency. Expected outputs include strengthened livelihood skills and reinforced positive preventive behaviors. The second leverage point involves consolidating the ability of local partners, such as ADPs, producer groups and other community-based organizations to form a vibrant and viable civil society. Expected outputs include holding local authorities more accountable for services, greater transparency and empowered communities. Finally, the last leverage point entails working deliberately alongside GOH entities to build their skills sets, professional capacity and service delivery skills. Expected outputs include wider incidence in decision-making and more streamlined processes and procedures.” 36 certain threshold. This evaluation notes that no follow-up actions to implement the recommendation were put in place. A review of program documents reveals the absence of a well-articulated sustainability plan. An examination of the performance monitoring plan also reveals that, with the exception of two indicators on agricultural production practices tracked in the baseline survey,17 no sustainability results and measurable indicators are built into the program. Experience from other countries (Rodgers et al. 2004a; 2012) indicates that without sustained resources, continued focus on capacity building, and realistic incentives to increase motivation, sustainability can be an elusive goal. The withdrawal of food rations and other free MYAP-provided materials and supplies such as birthing kits, water treatment kits, and family planning products, without considering substitute incentives is likely to jeopardize the sustainability of program activities. For this reason, consideration of alternative incentive structures should be incorporated into program design. Many of World Vision’s MCHN activities are likely to have lasting effects. However, the program focused almost exclusively on training, and no comprehensive capacity building plan was developed to ensure the provision of MCHN services after the end of the program. World Vision’s health and nutrition activities were implemented through rally posts, mobile clinics, home visits, food distribution points, and Mothers’ Clubs. World Vision began its program with the Mothers’ Club model that was launched during World Vision’s earlier Development Assistance Program and that was used as the primary mechanism for World Vision’s education and behavior change communication strategy. In 2011, the concept of ‘Lead Mothers network’ was introduced, an adaptation of the MCG model. World Vision adapted the MCG model from ACDI/VOCA’s version of the World Relief and Food for the Hungry models. World Vision’s approach has several advantages: • Rally points where health activities were carried out once a month by the health promoters brought health services and messages to households that have never had such support. These included distribution of oral rehydration salts, immunization, health education, vitamin A supplementation, deworming, and growth monitoring. • Women coming to the program’s mobile clinics benefited from pre- and post-natal services, distribution of iron and foliate supplements, immunizations, voluntary counseling and testing, postpartum and newborn care, growth monitoring and promotion vitamin A and deworming tablets, child health and treatment for malaria, acute respiratory infection and diarrhea, and exclusive breastfeeding. • Training of community volunteers and health promoters has made community-based health practitioners available within every locality. • Training Mothers’ Clubs to help their circle of mothers practice MCHN principles is likely to leave lasting influence. 17 (1) Number of farmers using at least three sustainable agricultural practices, and (2) number of farmers using at least four sustainable agricultural practices 37 • Training mothers, through rally points and Mothers’ Clubs in how to care for their children through vaccination, better hygiene and nutrition is likely to have lasting effects on the community. • Establishing fixed points may lead to government-managed health clinics at or near these locations. • Augmenting MSSP medical staff by hiring nurse assistants and health agents. However, these advantages may have been outweighed by two major shortcomings. First, sustainability hinges on technical assistance ability to build up the skills and capacity of local stakeholders whose involvement will be critical for maintaining development gains after the project ends. Effectiveness and sustainability of an intervention is likely to depend on the behavior of two groups of people – households and service providers (World Bank 2010b; Victora, Habicht, and Bryce 2004; Sjoblom 2012; Di Vinadio 2013). Even women with better knowledge of good child nutrition practices may be limited in their ability to act on this knowledge if they lack access to complementary services such as health care or to markets. In most instances, effectiveness and sustainability hinge on supply-side factors – that is, the ability of providers to deliver services effectively. Second, in addressing how to ensure the supply of MCHN services after MYAP funding ceased, World Vision focused almost exclusively on training as a single method for building the capacity of local organizations and partners such as GOH service providers, Mothers’ Clubs, MCGs and other CBOs. However training is only one piece of the organizational performance puzzle. USAID Human and Institutional Capacity Development (HICD) guidance stresses a series of structured and integrated processes designed to remove significant barriers to the achievement of an institution’s goals and objectives. From this perspective, success of training and other HICD interventions is measured by improvement in overall organizational output and performance, not by the number of individuals trained.18 Through a process of identifying performance gaps and designing performance solutions with clear goals and milestones to meet these gaps, organizations can be helped to achieve sustainability goals. A positive case example where sustainability is likely due to structured collaborative linkages: The staff of the integrated community services department of the Albert Schweitzer Hospital (HAS) have been collaborating with World Vision over the life of the MYAP to provide a number of MCHN services to Mother’s Clubs in 4 Communal Sections in their catchment area. Their staff of four female health agents first helped identify the best mothers in each area by observing their behavior and their children’s health, and picked them to be part of Mothers’ Groups. HAS agents continue to provide advice to Club members on children’s’ nutrition, household hygiene, the health of pregnant and nursing mothers, and children’s health. By working with the Mother’s Clubs the HAS staff have seen behavior change at the community level in many of these areas. As MYAP funding was ending, HAS made a commitment to continue their support to these clubs and will now 18 The USAID HICD manual notes that “training does not have an impact until the knowledge or skills acquired by the trainees have been successfully applied to a specific work situation, which, in turn, results in a measurable improvement in performance. Therefore, successful USAID training should not be measured in terms of the number of individuals trained but rather by the contribution made by trainees to organizational performance improvement.” (USAID/HICD 2010d) 38 begin moving to a Mothers’ Care Group model to reach out to even more mothers in their catchment area, and will soon begin to integrate livelihood activities into their support to these groups. Experience from other countries (Rodgers et al. 2004a; 2004b, 2012) indicates that without sustained resources, continued focus on capacity building, and realistic incentives to increase motivation, sustainability can be an elusive goal. The withdrawal of food rations and other free MYAP-provided materials and supplies such as birthing kits, water treatment kits, and family planning products, without considering substitute incentives is likely to jeopardize the sustainability of program activities. Another case example from Bas Plateau demonstrates the values of integration: A local agricultural association in the Boucan Carré Commune founded in 2006 was struggling to find markets and expand its production, while coping with the food, health and nutrition challenges of rural Haiti. Composed of over 120 members, under the World Vision MYAP the association was chosen in a competitive process to benefit from assistance with agricultural support. Association members benefitted from a full range of program activities: food distribution for mothers and their children, livestock production (goats, chickens, pigs), natural resource management activities, grain silos, bamboo planting, establishing a tree nursery, composting, and institutional strengthening and marketing assistance from a Haitian agricultural development company. The association identified land in the community and donated it to World Vision to serve as a community training center, after which the CS helped with infrastructure, and association members were able to be trained in and practice the activities described above. With the help of the MYAP project, a number of positive benefits have accrued to the community, and association members now are very positive about the sustainability of these activities: mango production has more than quadrupled over the last four years and the association has been assisted in obtaining an organic certification to be able to take advantage of the US market. They boast of more home￾grown vegetables to eat and sell and an active tree nursery run on a semi-commercial scale. Community members are now more aware of the importance of planting trees for soil conservation; women with children are engaging in family gardens (‘jardin lakou’); farmers have more disposable income from mango exports to pay for education fees and health care; there are fewer malnourished kids due to improved access to protein sources (goats and chickens); and exports have increased local jobs in mango transformation. The Bas Plateau association members noted a reduction in malnutrition among children; in part because mothers were taught to use the leaves of Benzolive tree to improve nutrition of kids; many mothers now practice exclusive breast feeding of young children; and young children have fewer diseases due to vaccinations. Finally, as result of their successes, the association was able to convince community members to help rehabilitate 13 kilometers of farm-to-market roads, all done with local labor. Many of World Vision’s agricultural, natural resources management and livelihoods activities are likely to have lasting effects. However, ensuring continued supply of agricultural inputs and services after the end of the program would have strengthened program sustainability. 39 • Many of World Vision’s livelihoods activities have been successful. Several features accounted for that success: • The program has provided technical assistance in seed production technology to support quality seed production and dissemination in the program areas. • World Vision promoted the introduction of improved grain storage in certain areas where farmers were trained to protect their plots with anti-erosive structures, specifically contour canals with tree seedlings planted along the contour. • World Vision promoted gardening and household production diversification through cultivation of a small plot around the household residence that integrates vegetable production, fruit trees, livestock and micro-irrigation. Such an initiative is likely to enhance household food security by improving household nutrition and increasing household income when excess produce is sold. • Multiplication centers and breeding stations were established to promote the restocking of small animals in collaboration with the Ministry of Agriculture, Natural Resources and Rural Development. • Multi-purpose regional and community level nurseries were established to promote household agro-forestry production diversification through access to fruit trees and fast-growing multipurpose trees usually planted on eroded hills and mountainsides. • Training local farmer in the use of the new varieties and cropping methods has resulted in the transfer of valuable knowledge One beneficiary said: “Due to the intervention of World Vision there are many trees, and seeds, the atmosphere has changed.” Other livelihood interventions were perceived as unlikely to take hold because they require relatively large financial resources such as silos and bee-hives or a high level of technical knowledge -- for instance, to keep seed stocks pure, certified and on-going. A review of program documents and extensive interviews with program staff, beneficiaries and partners indicate that no strong links were created between model farmers to service providers. Similarly, no emphasis was placed on entrepreneurship development among community nurseries to become local vendors from whom supplies (e.g., seeds, seedlings) and services (e.g., fruit tree grafting) could be secured on as needed basis. More generally, the program focused on production through donated agricultural inputs, but did not pay sufficient attention to the sustainability of input supply or to the other links of the value chain, particularly and marketing of farm produce through private sector channels. Whereas MCHN activities are unlikely to be sustained without publicly managed arrangements, sustainable livelihood activities require financing models through private sector participation. In the absence of evaluation reports or performance indicators to measure sustainability of private sector organizations supported by the program, it is difficult to assess whether livelihood activities in the areas have been gradually tied to sustainable models. 40 CONCLUSIONS Sustainability is achieved when host country partners and beneficiaries maintain results beyond the life of the program. Many of World Vision’s MCHN, agricultural and livelihoods activities are likely to have lasting effects. However, the World Vision program did not develop a comprehensive capacity development plan to ensure sustainability, nor did it develop sustainability indicators to monitor and evaluate progress, and adjust interventions as needed. WHV created demand for its program services. However, sustainability is not driven by demand alone because it also hinges on supply-side factors – that is, the ability of service providers to deliver their services effectively. In its MCHN interventions, World Vision focused on training, a key capacity development ingredient, but did not place sufficient emphasis on other performance gaps with clear milestones for achieving program sustainability goals. In agriculture, the potential for program sustainability would have been enhanced through greater emphasis on market mechanisms to ensure that the supply of agricultural inputs and services are available to program beneficiaries when the program ends. RECOMMENDATIONS In its future programs, World Vision should develop more rigorous sustainability or “transformational development” plans. Experience from other countries demonstrates that, to achieve optimal results, sustainability of Title II programs should be an integral part of program design and should be embedded throughout implementation to withdrawal. Community organizations and individuals should also be aware of their post-exit roles and responsibilities from the outset. The sustainability plan should at a minimum include: decisions about approach (phase out, gradual phase over); explicit benchmarks for progress and timelines; clear allocation of responsibilities; graduation criteria and progressive withdrawal of free inputs; a focus on building the capacity of local community and government organizations to progressively take up the management and provision of MCHN services; and developing alternative incentive structures (e.g., livelihood enhancement programs) to create increased resiliency among beneficiaries to self-fund MCHN and other services. To ensure greater sustainability of livelihood activities stronger emphasis should be placed on private sector participation early and throughout these forms of food-assisted activities Future cooperating sponsors should be required by USAID to develop rigorous sustainability plans. Experience from other countries demonstrates that, to achieve optimal results, sustainability of Title II programs should be an integral part of program design and should be embedded throughout implementation to withdrawal. In the future, CSs should write a detailed graduation, exit and sustainability strategy for each sector at the inception of FFP supported programs. Reviews against the strategies’ implementation should be taken every 6 months and be reported in their Annual Results Report. Despite the recommendations of the mid-term evaluation, USAID and CSs should rethink whether to pilot new activities late in a program, where rollout occurs largely in the final year. This is intrinsically difficult for communities and capacity building efforts that aim for sustainability. 41 To properly measure sustainability, follow-up research is recommended. The more useful time to derive more solid evidence about resilience and sustainability would be 6 months or a year after program assistance has ended, to observe whether systems actually sustained would be some time after the program had ended. Community organizations and individuals should also be aware of their post-exit roles and responsibilities from the outset. The sustainability plan should at a minimum include: decisions about approach (phase out, gradual phase over); explicit benchmarks for progress and timelines; clear allocation of responsibilities; graduation criteria and progressive withdrawal of free inputs; a focus on building the capacity of local community and government organizations to progressively take up the management and provision of MCHN services; and developing alternative incentive structures (e.g., livelihood enhancement programs) to create increased resiliency among beneficiaries to self-fund MCHN and other services. To ensure greater sustainability of livelihood activities stronger emphasis should be placed on private sector participation. QUESTIONS 8: TO WHAT EXTENT ARE THE VARIOUS MOTHERS’ CLUBS MODELS IMPLEMENTED BY WVH, CRS AND CRS COST EFFECTIVE? Since cost-effectiveness assessments must involve more than one program, the cost-effectiveness question is examined mainly in the overall MYAP program report. World Vision conducted extensive outreach in Haiti with a blended model that combined traditional mothers’ support groups alongside a newer model, the Mothers Care Group Model, tested in other countries. World Vision ran 1950 mothers clubs in its areas, and provided extensive outreach into communities. On a per capita basis, from the household surveys, World Vision had the most impact, i.e. the most positive change in key indicators of the three NGOs examined. However, World Vision also did this at greater cost, and its cost efficiency reduced the efficiency compared to the other CSs. In the future, USAID should build cost-effectiveness hypotheses into the design of new program awards, not only at the time of an independent evaluation. • Implementing partners should be required to maintain a financial dataset that specifically tracks the costs associated with the models and outcomes of interest in operations research. This will correct for the problems introduced by each CS cataloguing their cost data in divergent ways. • Natural or convenience control groups should be identified, if any, for similar tracking between baseline and endline or from the time that a model or approach is introduced.