1 Midterm Evaluation (MTE) for Haiti MYAP Program Haiti MYAP Overview, Methological Approaches, with Major Conclusions, Lessons Learned and Recommendations MYAP Report #1 Prepared by the Mid-Term Evaluation Team: Dr. Richard Swanson, External Consultant & Team Leader, Economic Anthropologist Gregory Charles, Agronomist Colette Vilgrain, M.D. Wesner Antoine, Statistician July 31, 2010 2 LIST OF ACRONYMS ACDI/VOCA Agricultural Cooperative Development International/Volunteers in Overseas Cooperative Assistance ADP Area Development Program (of WVH) or PDZ (Programme de Développement de la Zone) BCC Behavior Change Communication BND Bureau de Nutrition et Développement CBO Community Based Organization Col Vols Community Volunteers CS Cooperating Sponsor CSB Corn Soybean Blend CHW Community Health Workers CMAM Community Management of Acute Malnutrition CNSA Coordination Nationale de la Securite Alimentaire CRS Catholic Relief Services DAP Title II Development Assistance Program EWS Early Warning System FANTA Food and Nutrition Technical Assistance FFA Food for Assets FFP Food for Peace FFW Food for Work GoH Government of Haiti HAS Hospital Albert Schweitzer HDDS Household Dietary Diversity Score FY Fiscal Year IPTT Indicator Performance Tracking Table IYCF Infant and Young Child Feeding LQAS Lot Quality Assurance Sampling MAHFP Months of Adequate Household Food Provisioning MARCH Management & Resources for Community Health MARNDR Ministry of Agriculture, Natural Resources and Rural Development MCHN Maternal and Child Health and Nutrition M&E Monitoring and Evaluation MoH Ministry of Health MSSP Ministry of Health of Haiti (Ministere de la Sante Publique et de la Population) MUAC Mid-Upper Arm Circumference MUSO, MUSOGs Mutuelle de Solidarity (Self-Help Group), Mutual Solidarity (savings & loan) Groups MYAP Multi-Year Assistance Program OVC Orphans and Vulnerable Children PM2A Preventive Malnutrition under Twos Approach PDA Personal Digital Assistant PEPFAR President‟s Emergency Plan for AIDS Relief PFA, PTFA Poid faible pour l‟age, Poid tres faible pour l‟age (low weight for age, very low weight for age) PLWA Persons Living with HIV PMP Performance Monitoring Plan PNA Preventive Nutrition Approach (see PM2A) SAVE Save the Children SO Strategic Objective SYAP Single Year Assistance Program USAID United States Agency for International Development W/H Weight for Height Index WVH World Vision Haiti WVUS World Vision United States 1 carreaux = 1.29 hectares 3 Midterm Evaluation (MTE) for Haiti MYAP Program Table of Contents Acronyms Table of Contents Maps Forward 0.0 Executive Summary 0.1 Overall Impacts of MYAPs 0.2 Major Conclusions & Lessons Learned 0.3 Major Recommendations 1.0 Introduction & Overview of Haiti MYAPs 1.1 Background 1.2 Midterm Evaluation Team (MET) Strategy 1.2.1 MTE Methodology 1.2.2 Quantitative Survey 1.2.3 Qualitative Survey 1.3 USAID Haiti Economic Growth Strategic Framework 1.3.1 USAID’s MYAP Program Hypothesis 1.4 MYAP Cooperating Sponsor’s Program Strategic Frameworks and Mid-Term Objectives 1.4.1 World Vision Haiti 1.4.2 CRS 1.4.3 ACDI/VOCA 2.0 Overall Program Design and Targeting 2.1 Strategies for Common Approaches, Program Integration, and Impact Diffusion 2.2 MYAP Coordination & Technical Leadership 2.3 MYAP M&E System and Targeting 2.3.1 MYAP PMPs & IPTT Indicators 2.3.2 Reporting & Communications 2.3.3 Targeting 2.3.4 Quantitative Survey, Summary of MTE Results 3.0 Overall Program Management and Implementation 3.1 Program Management 3.2 Program Priority Activities and Implementation Process 3.3 Links to Government of Haiti (GoH) 3.4 Staffing and Capacity Development 3.5 MYAP Financing and Commodities 3.6 MYAP Impact on Environment & on Population Effected by Recent Earthquake 4.0 Moving Forward Annexes: Annex 1: Scope of Work for Mid-Term Evaluation Annex 2: Mid-Term Evaluation Logistics Scheduling Table (with External Consultant Haiti Schedule) Annex 3: Protocol for Mid-Term Evaluation Annex 4: Key Documents Consulted Annex 5: List of Key Persons & Organizations Consulted Annex 6 MYAP Indicator Performance Tracking Tables for World Vision Haiti, ACDI/VOCA, & CRS Annex 7: MYAP Mid-Term Quantitative Survey Instrument Annex 8: MYAP Mid-Term Quantitative Data Tables Annex 9: MTE PowerPoint Presentation (June 3, 2010) 4 Maps World Vision Haiti MYAP Geographical Distribution Catholic Relief Service (CRS) MYAP Geographic Distribution 5 ACDI/VOCA MYAP Geographic Distribution Cover Page: Col Vols Leading a month’s session at a World Vision Rally Post 6 Forward The MTE team wishes to express our appreciation for the major efforts made by each of the MYAP Cooperating Sponsors (CS) to support the efforts made in this mid-term evaluation through both the quantitative and qualitative surveys. This was a major effort, involving the time of many people, working under restricted timelines in often very difficult field conditions. Following barely three months after the devastating January 12 earthquake, MYAP staff of all levels were still dealing with not only the personal/family impact of this event, but also the added stress and workloads placed on everyone to provide maximum support to the continuing victims, while trying to continue the momentum of planned programs activities. Port-au-Prince remains in a state of limbo – with destroyed building and rubble everywhere.1 Thousands of households are living under plastic tarps or in tents, as the rainy season has begun. One has a sense that it will take a generation to begin to recover from this tragedy and wonders when some form of reconstruction will begin to offer hope for rebuilding lives. While the four members of the MTE team may have led the field enquiries for this period of internal assessment, the process of necessity was a team approach that involved the input all along the way of many of the senior MYAP leaders, both nationally and regionally within the two major components of the program: MCHN and agriculture. We are grateful for their support, frank and open discussions with us, and hope we have accurately portrayed their programs. Our task was made much easier by the open spirit of enquiry by all three organizations. At this mid-term review, program management was open to self-reflection, self-analysis and assessment of program activities. Were objectives being met, what needed to be modified or changed to achieve even greater results, what was working and what simply has not? This spirit of openness to potential change of direction, if needed, challenged the MTE team to seek to meet these high expectations. The people who made this all possible are listed in Annex 5. Three individuals in particular, however, must be mentioned here. In first place, Mr. Donard Nyirenda, World Vision Haiti‟s head M&E Coordinator was the key person who coordinated all MTE efforts among the three MYAP partners, initially working almost endless hours with each to develop the scope and terms of this evaluation, and then in coordinating overall logistics for each of the MTE consultants, and field enumerators for the quantitative survey. Though World Vision Haiti provided the overall coordination effort, their efforts could not have been successful without the active support and constant involvement of Nael-Jean-Baptiste ACDI/VOCA‟s MYAP overall M&E coordinator and Joseph (Junior) Sander, CRS‟s MYAP overall M&E coordinator. Each of these three worked closely with the MTE team, traveled with us in the field and participated in field interviews and daily logistics. We also are grateful for the full support of each of the MYAP program managers (Lionel Isaac – World Vision Haiti, Jude Marie Banatte – CRS, and Emmet Murphy of ACDI/VOCA). We wish to also specifically thank Vicky Michiner, working with FANTA, on providing virtual support to the team as we were developing the sampling framework and quantitative survey instrument, and Ms. Leslie Osterman, Food for Peace Officer from Washington DC who traveled with the MTE team during part of our time within CRS and ACDI/VOCA regions and provided some insight into FFP perspectives. The MTE team, following the eight or nine days spent within the program regions of each MYAP Cooperating Sponsor provided the field teams and management staff with a preliminary debriefing of our observations on each program. Input given at these sessions was also helpful to the MTE team. Many of the observations made in this report are necessarily subjective, based on observations from short periods of contact. Impressions are gained however through such contact, and sometimes are a reflection of realities that may not always be evident to those busy with daily duties of implementation. 1 ACDI/VOCA notes that ‘over 80% of buildings in Port-au-Prince were damaged or destroyed’. (MYAP 2010 2nd Quarterly Report). The MTE team frequently drove through the city going to rural destinations for this evaluation, and observed sandwiched buildings where nothing yet had been initiated to remove rubble; streets are lined with makeshift shanties and tent communities. 7 Haiti Mid-Term Evaluation of World Vision, CRS, and ACDI/VOCA led MYAPs 0.0 Executive Summary Rather than undertake mid-term reviews of the HAITI MYAPs separately, a decision was made by USAID/Haiti and the Cooperating Sponsors to undertake this as one exercise. This would permit comparison of overall program approaches, accomplishments, lessons learned and to possibly consider programmatic course corrections that could impact all three MYAPs. The review began in April and extended through the end of June, 2010, led by a four-person team made up of an economic anthropologist and team leader, an agriculturalist, medical doctor, and survey/statistics expert. Over the past 2 ½ years of program implementation, all three HAITI MYAPS (World Vision Haiti, CRS, ACDI/VOCA) have directly benefited many hundreds of thousands of rural Haitians through MCHN and agricultural activities – working through sometimes almost unbelievably difficult field circumstances and conditions. Had these organizations not been present in their respective regions of responsibility, pre￾positioned with personnel, food and medical supplies, the negative impacts of the three 2008 hurricanes and the devastating January 12, 2010 earthquake would have been much greater. The need to operationally move from a development assistance mode of implementation to emergency and famine relief and back again has placed challenges upon all three organizations, though they have done so quite effectively, in spite of understandable delays to planned program targeting. 0.1 Overall Outputs and Impacts of MYAPS (to date) All three MYAPS are already achieving significant outputs and impacts that include, but are certainly not limited, to:2  Direct Beneficiaries: 154,449 direct beneficiaries for MCHN and agricultural programs by World Vision Haiti in FY 2009; 15,200 for CRS, and 28,497 for ACDI/VOCA;  Growth Monitoring: WVH: 31,931 children under 5 years of age are being weighed and monitored for their growth each month through 1,104 rally posts, including health/nutrition training to their mothers. In its most recent quarterly report, “the program monitored the growth of 32,049 children, provided DPT3 to 3,641 children less than one year old, provided immunization to 3,063 children of 6-59 months old and distributed first and second vitamin A capsules, respectively, to 6,327 an 4,719 children. Medical attention and treatment were provided to 5,454 children, including the treatment of 1,019 cases of diarrhea.”3  ACDI/VOCA, in its most recent quarterly report records that in spite of the added burdens in providing food to some 153,908 additional people adversely affected by the January 12 earthquake, the MYAP was still able to provide support to some 34,524 people, provided 2,925 children aged 6-23 months and 1,100 pregnant and lactating mothers with food rations, 5,943 children received Vitamin A, provided antenatal care by skilled providers to 3,787 women; within agriculture the program bought 20 MT of improved black beans for planting this season, 52,500 cuttings of manioc were distributed to 5,462 farmers, 67 hectares of short cycle sorghum was planted, 202 farmers in La Vallée and Bainet trained in yam improved 2 More detailed description of specific MYAP impacts, with descriptions of program activities, conclusions, and recommendations is provided in Reports 2, 3, and 4 for World Vision Haiti, Catholic Relief Service, and ACDI/VOCA respectively for this mid-term evaluation. 3 WVHH, January – March 2010 Quarterly Report, April, 2010, p.4. 8 production using minisets,1,064 farmers received training in vegetable gardening, 25,000 orange trees are ready for distribution for grafting as manderines, and much more.4  Food Distribution: 23,506 children under 5 years of age received supplemental food by World Vision Haiti in FY 2009; 29,259 by CRS in FY 2009, and 2,599 by ACDI/VOCA; Each of the MYAP targeted groups received these rations amounting to 4,034 metric tons (for WVH) distributed to date.  254 hectares of watersheds that have already been rehabilitated between FY 2008 and FY 2009 by World Vision Haiti; 592 ha. by CRS, and 691 ha. by ACDI/VOCA;  25,743 pregnant and lactating mothers who have been assisted by the MYAPS, including receipt of food rations, by CRS in FY 08 and FY 09; 521 by ACDI/VOCA;  45,842 primary school children who have been provided supplemental food as well as improved instruction through CRS efforts in FY 2009, with gross attendance rates increasing to 85%.  Besides these already impressive numbers, many thousands of children have been fully vaccinated and yearly received Vitamin A supplements from all three MYAPs.  There have been 50% and 100% increases in agricultural productivity through the introduction of new and improved varieties of black beans, maize, and sweet potato in particular, as well as the promise for more long term impact through thousands of grafted Francisque mango and manderines that have already reached the household plots of thousands of households which will eventually increase their households incomes.  351,250 fruit and forest tree seedlings were produced by World Vision Haiti during the 2nd Quarter of FY 2010 alone, ready for distribution in coming months; similar CRS agro-forestry efforts have trained local area youth in fruit trees (mango) grafting and helped them to establish multiple fruit tree nurseries for local distribution; some 50,000 bitter orange seedlings are almost ready for grafting and distribution within ACDI/VOCA‟s Southeast Department, to be placed on the steep slopes of cooperating farmers.  Increased interest in household production of high-value vegetable cultivars (chili peppers, cabbage, onions, and potatoes, eggplant), already improving nutrition and raising household incomes of thousands of rural Haitian households. Small-scale drip-irrigation for these plots also looks very promising. All three MYAPs have renovated existing irrigation perimeters in their areas, opening up more land for year-round production of high value crops by land poor farmers practicing share-cropping.  Self-help savings and loan mutual solidarity groups (MUSOGs) have been formed by all three MYAPs, and where these have been established and savings accumulated, members of these groups have been able to initiate a number of small economic ventures adding to their household incomes and overall food security. In ACDI/VOCA‟s case, these MUSOGs are seen as an integral part of the safety nets for their EWS program. Strengths of these MYAP programs include:  Dedicated staff working in difficult settings;  Capacity building (training) for local seed producers, mothers for child care and nutrition and household vegetable gardening;  Improved rural household knowledge of MCHN, and better practices of exclusive breastfeeding (WVH,CRS);  Implementation of a full package of MCHN and agricultural services in Haut Plateau (WVH) addressing windows of opportunity and households of malnourished children;  Early Warning Systems (EWS) have become operational in each region; 4 Additional details may be obtained from the ACDI/VOCA January-March 2010 Quarterly Report, June 2, 2010 or from the most recent versions of the IPTT, provided in Annex 6 of this report. 9  Strong linkage with PEPFAR and Save the Children regional projects (Ready to Use Therapeutic Food for acute malnutrition) for addressing the needs of severely malnourished children;  Community based Collaborating Volunteers (Col Vols) as a sustainable approach to promoting rural health (CRS, ACD/VOCA);  Savings and loan groups formed – frequently within Mothers Clubs (CRS)  Increased # and geographic distribution of rally posts (CRS/WVH, ACDI/VOCA) and „Fixed Points‟ (ACDI/VOCA) bringing health services closer to where people live;  World Vision Haiti and CRS MYAPs built upon earlier FFP DAPS, with 5-10 years of prior experience in some communities. 0.2 Major Conclusions & Lessons Learned 1. Program Structuring: World Vision Haiti, CRS, and ACDI/VOCA are highly structured, quality programs achieving most program objectives and targets set for themselves as shown on their Indicator Performance Tracking Tables (IPTTs).5 2. Integrated Approach: There is clearly a need for a more integrated approach – by which we mean specific targeted vulnerable households receiving both MCHN and livelihood/agricultural development support as mutually reinforcing activities Though each MYAP focuses on health and nutrition as one component area, and agricultural and livelihood development as a second component area, each organization approaches these in different ways 3. Targeting of Activities: Households benefiting from assistance in mother child health/nutrition (MCHN) or agriculture/livelihood activities could be better targeted. Initially focusing on less vulnerable households (because they have the land resources, are the respected local leaders), in favorable areas (e.g. irrigated perimeters, La Vallée) as a means of developing techniques and demonstrations that can then more effectively extended to the more vulnerable within communities is a common approach which has been quite effective. Expansion of the use of „Model Farmers‟ and „Model (leader) Mothers (in Mothers Clubs) would also encourage behavioral change communication and entrepreneurial skills within different localities. All three MYAPs have reached the point that greater attention now needs to be given to extending locally tested and proven technologies/cultivars to the more vulnerable households and one means of doing so could be through the members of the MCHN Mothers Clubs. 4. Training: the importance of targeted training becomes very evident through these MYAPs – and the training given is excellent. Yet the strategy of creating „Mothers Clubs‟ uniquely to target specific physiological developmental stages of infants (pregnant women, lactating women, infants under 2 years of age, malnourished children under 5, may take targeting too far. Mothers have children of multiple ages under and over 5 years of age and the MTE team believes that, to be sustainable within the communities, these Mothers Clubs need to serve a larger purpose than simply focusing on a specific infant, at a specific moment in time. The appropriate needs of specific mothers with children at these different physiological levels, linked to the PM2A strategy, can just as effectively be addressed with groupings of mutually 5 IPTTs may be reviewed in Annex 6 of this report. 10 supporting women with children under 5 years of age. In this case, targeting would mean being more observant to the specific configuration of these groups when messages are communicated by the Col Vols – through Leader Mothers – and focusing on the appropriate training materials for specific groups.6 Haiti MYAP CS partners, with the exception of a few Mothers Clubs, are not sufficiently addressing the more general health/nutrition training and socio-economic needs at the most basic household level. 5. Mother/Child Health & Nutrition (MCHN): MCHN activities clearly dominate the MYAP programs of both World Vision Haiti and CRS – and are become increasingly important with ACDI/VOCA as well - with agricultural activities sometimes focused on MCHN beneficiaries households, but in other cases not. 6. Community Management of Acute Malnutrition (CMAM): This is an important initiative within the Haiti MYAP which brings together many of the key elements of addressing infant malnutrition, including PM2A, and provides guidelines of where focus needs to be given for this to be effective. WVH is ideally positioned to support CMAM through their established ADP community structures, which in turn reinforces these activities in other locations without ADPs. 7. Mothers Clubs & Children under 5 years: Haiti MYAP CS approaches to their targeted beneficiaries around locality rally posts through Mothers Clubs could be greatly strengthened.7 Mothers Clubs are mentioned in WVH MYAP documents as the main venue for Behavioral Change Communication (BCC). The MTE team believes that this attention is well-placed, but the membership composition of these clubs for such communication will limit its long term sustainability. Mothers Clubs are not discussed to any great degree within the program documents of the MYAPs reviewed, yet they represent one of the central means for MYAPs to effect behavioral change in a number of areas for both MCHN and economic uplifting. 7. Collaborating Volunteers (Col Vols): Most MYAP Col Vols are male. While women noted that they „got used‟ to male Col Vols giving them training on often very intimate male-female issues, it was also clear that they were most comfortable when hearing about, and discussing, these topics with another woman. Col Vols represent a valuable strategic approach to sustainable community-based health and nutrition knowledge acquisition. 8. Rally Posts: For all three MYAPs, rally posts (Poste de Rassemblement) are key geographic areas within the localities of the Section Communales where mothers congregate, with their children, to receive specific MYAP social services. These services are targeted through some training activities and 6 While the MTE team completely understands that grouping women by the physiological status of the child may be the most efficient means of giving a specific message to a specific mother at a specific time, doing so artificially puts together a cohort of women who may have less incentive to remain together as this specific child grows older. This may be a strategy of ‘controlling the message’ women are receiving in these group meetings, but these women live within a larger community where other influential mother voices are heard. 7 The MTE team believes that membership of Mothers Clubs should focus on mothers with children under 5 years of age – and not on the developmental stages of specific infants. Mothers of children of different ages can help reinforce the MYAP BCC messages within their groups. As the children grow beyond 5, many of these mothers will bear new children. The MYAP can encourage the sustainability of these groups by helping them to benefit from various livelihood development activities. MYAPs are missing an opportunity by not doing this. 11 growth monitoring activities each month. Food rations are often distributed through „points de distribution‟ or literally „distribution (of rations) points‟. 9. Changes in Malnutrition: It is essential to track rates of change in malnutrition, over time, to assess whether or not MYAP activities are having the intended impact. And this must be done at a geographical scale that has some meaning – where a majority of the most vulnerable actually does receive MYAP services. None of the MYAP Cooperating Partners track changes in nutritional rates at the basic locality level where the rally posts are located, though they all have the data to do so. Data such as „low weight for age‟ + very low weight for age‟ (PFA + PTFA), disaggregated by age groups, are aggregated to commune and then regional levels. For example, WVH with 647 rally posts and ACI/VOCA with their 220+ rally posts collects all such data on a monthly basis through local health workers or Col Vols under the supervision of auxiliary nurses and medical doctors. The MYAPs do not yet assess whether, over time, there has actually been an actual decrease in the overall malnutrition rate in specific localities due to program interventions to justify phasing out of some activities and moving on, or reinforcing, other areas of greater need. Adding the tracking of severe malnutrition, with the help of the MUAC measure, can be a useful tool within the CMAM approach to monitor this over time. 10. Approach to Groups and Organizing People for Livelihood Activities: Within all three MYAP Cooperating Sponsors, for agricultural & livelihood activities, programs almost always work with separate groups organized for specific purposes, sometimes uniquely women, to achieve program targets for specific activities (distribution of goats, tree nurseries, chickens, grain silos, irrigation with a focus towards marketable vegetable crops, improved seed, marketing mandarins, etc.). 11. Constraints: Since the beginning of the MYAP 2.5 years ago, the three programs have experienced one natural disaster after another, beginning with 3 hurricanes in 2008, and the January 2010 earthquake and its after-shocks that killed over 230,000 people in the Port-au-Prince region – with over a million people displaced and/or made homeless. MYAPs whose intention was to help move people from famine and relief efforts to more long-term sustainable development activities have found themselves pulled back and forth between these two kinds of efforts, which demand very different logistic , organizational, and delivery systems to the most vulnerable of Haiti. Having themselves lived through incredibly stressful and sometimes dangerous events – with loss of family members in some cases - the impact on MYAP organizational personnel, particularly WVH and Port-au￾Prince based personnel of ACDI/VOCA and CRS has been severe. That they have been able to respond to these sudden surges of need is a testament to the strength and adaptability of these organizations. 12. Program M&E: All three MYAPS have invested heavily in building monitoring and reporting systems from the ground up to regional centers and their national centers. Monitoring of the Indicator Performance Tracking Table (IPTT) data are taken seriously and much effort is given to gather and report on these in a timely manner. Behind each of these indicators are a series of other output indicators which are regularly reported on within quarterly reports. Indeed, as noted elsewhere, one might even say that the IPTT data requirements (i.e. set targets) for the program drive program implementation – seeming to influence, in the case of World Vision Haiti, the very approach 12 used to structure and communicate with Mothers Clubs.8 However, given what the MTE team believes to be far too many indicators being tracked (over 50 on the IPTTs of each PVO – not to speak of other data tracked), one cannot help but wonder who is taking the time to understand the implications of all these data, to analyze them, to influence future implementation. 13. Early Warning System (EWS): EWS tracking efforts have been undertaken by each MYAP CS - though best organized and integrated into Haiti‟s national EWS system, through MARNDR/CNSA, by ACDI/VOCA and CRS. WVH information does not yet appear on the CNSA website, though WVH has been active with EWS in their regions for over a year now and EWS bulletins are prepared. Two early warning bulletins were produced for the months of January and February for example. The CNSA information and bulletins provide quarterly information on rainfall and market price trends, giving recommendations for policy makers and others working within specific Departments on actions that need to be taken. ACDI/VOCA includes in its bulletin a listing of warning levels; it includes a conclusions and recommendations section where alerts are made. Perhaps more importantly, this is all done in conjunction with several partners and GoH agencies within the Department. As such, these bulletins appear to be a potentially very valuable tool for policy makers and implementing programs within Departments. It is not immediately clear from the CNSA (www.cnsahaiti.org) bulletins produced that the EWS recommendations or alerts are actually precipitating some kind of action because of „early warning‟ communicated. 14. Reporting Systems: World Vision Haiti, CRS, and ACDI/VOCA alike have built highly structured reporting systems that appear to be working quite effectively in completing quarterly progress reports and IPTT indicator tables. Quarterly reports for all three MYAPs were initially delayed however since the New Year because of the January earthquake, with FY 2010 2nd Quarter reports (January – March) only becoming available in June or July. What appears to be lacking in these reports is critical analysis of the data reported on, particularly at local levels, to permit MYAPS to assess potentially developing impact within specific areas (not just outputs) so as to permit possible program corrections or even initiating withdraw from some areas or and reallocate resources to reinforce other areas within the different target section communales should successes become apparent. 15. Common Approaches: There is a need for common approaches across MYAPs, including greater standardization in M&E indicators, and even the structuring of the quarterly reports. Though the 3 MYAPS all responded to the same USAID SO and IRs, each developed these in different ways which complicates synthesis of lessons learned across programs. Each MYAP CS has evident strengths and weaknesses, and these strengths could be better adapted across programs. CRS should lead in efforts with Col Vols, Mothers Clubs and savings and loan MUSOG activities; ACDI/VOCA could lead in their focused approach to developing key commodity value-chains and local seed producers, while WVH‟s efforts in developing Community Management of Acute Malnutrition (CMAM) and the linked PM2A efforts, rally posts, mobile clinics, and food distribution sites closer to rural population clusters should be emulated. 8 The justification is that by addressing homogenous groups with focused messages at each developmental stage of the child‟s growth, messages will be more applicable to those hearing them, and be better applied. This may be true, but this approach fails to sufficiently address other issues of equal or greater importance (discussed elsewhere). 13 16. Program Graduation: There is a need for an „End Game‟ and „Graduation of Activities‟ strategy within the MYAPs. Currently „graduation‟ is seen as what will be achieved at the end of the project life – but then we also see MYAPS following earlier FFP DAPS in the same locations. By „graduation of activities we do not mean pulling out of a commune or sectional communales completely, but looking more closely at the variability among the localities of section communales so that the vulnerable households in clusters of localities will have benefited by and adopted key MYAP messages for prevention and livelihood enhancement that would justify moving on to other less well served areas. If the purpose of the MYAP is to strengthen and extend GoH Health services to underserved regions, and maintain these services until such a time that GoH or someone else will be able to take them over, then this would be a different and longer term objective. As an example of this, ACDI/VOCA and CRS health workers (health agents and auxiliary nurses) are under a Haitian Ministry of Health (MSPP) contract, thus supporting the government of Haiti‟s health network, and thereby extending the GoH‟s services to underserved areas. The long term objective is that GoH will be able to take over these services, and the fact that GoH protocols and salary levels for personnel are respected, this remains feasible. The WVH MYAP - though interacting with the MSSP - manages their health centers and health networks. 17. MYAP/SYAP: Insertion of SYAP activities (parallel teams, sometimes with little Haiti experience, field surveys and visits) alongside on-going MYAPs appears to have been disruptive to on￾going Haiti MYAP efforts – whose personnel and logistic resources are already stretched. Given the reality that SYAP activities actually come on line 4+ months after the catastrophic event that led to its initiation – and can no longer be considered „emergency help‟, one might even question the usefulness of some of activities, and whether they should have even be undertaken at all. In its MYAP proposal, WVH promised to „maintain a surge capacity for flexible emergency response‟, and looking back at the hurricane events of 2008 and 2010 earthquake, one would have to say that this objective has been largely fulfilled by all three MYAPs. 0.3 Major Recommendations The MTE team would suggest that, given limited resources and the food insecurity that actually does exist in many regions of Haiti, focus should be returned to original FFP mandates and prioritizations. Because of recent proposed changes in the structuring of USAID funding, with potentially greater funding towards increasing rural agricultural productivity and through market linkages, this might be a good time to return to FFP‟s original core focus addressing the needs of the most vulnerable, including overly high rates of child chronic and acute malnutrition in some regions. We would further suggest that one of the most effective means of addressing this would be through self-selecting mothers club groups (linked to rally posts and health clinics) focused towards preventive health practices and improved nutrition. These efforts would also be integrated, for these same households, to sharply focused livelihood activities like high-value (household) vegetable gardening year around (with drip irrigation if necessary), improved seeds and cropping techniques, and savings and loan group (MUSOG) activities permitting household women to pursue a wide range of economic and commercial opportunities. Doing so holds the greatest promise to actually leave something behind in MYAP regions of action, even if GoH continues to be slow 14 in its own ability to provide the rural health services it would like to. We are NOT suggesting that MYAP efforts in working with model farmers and productive groups for seed multiplication and other agricultural production focused efforts, using a value-chain approach, be abandoned. These are essential strategies to develop the appropriate techniques and cultivars at the local level for use by more vulnerable local households. 9 We are suggesting that a greater focus on the mothers of an expanding number of Mothers Clubs in MYAP localities would be the most effective way to enhance integration of MYAP activities, and have an impact on their household food security and improved household nutrition with available commodities. Common Approaches Based on MTE field observations of the existing Haiti MYAPs, we would recommend the following integrated and coordination approaches for the remaining life of Haiti MYAPs, and future efforts of this kind here. 1. Focus all FFP Title II programs on the MCHN component as the central organizing theme and target beneficiaries centered around hundreds of rally posts (within „tightly targeted geographic areas‟) - probably localities; 2. To receive MYAP services, including food supplements, through a rally post, all mothers with children under 5 years of age should be encouraged to join a mothers club of their choice. Creative use of diversified incentives other than money (seeds, plants, etc.) may help this move forward more quickly; 3. Target the majority of all mothers/households of children with children under 5 years of age within these localities through Mothers Clubs of self-selecting small groups of mothers (with a leader mother), and 100% of registered malnourished children whose mothers should be part of a mothers club, and 4. Involve all targeted Mothers Clubs in at least one, preferably both, of the following two MYAP livelihood activities: (a) establishment of a bio-intensive vegetable & fruit gardens (with drip irrigation for year-round production if necessary) and/or (b) participation in a self-help savings and loan MUSO mutual group. Either or both will lead to both improved household nutrition of all family members and towards other economic livelihood opportunities of choice for the different individuals concerned. Other MYAP livelihood activities should always target these households as beneficiaries. When value-chain linkages to markets are developed for specific agricultural commodities (peppers, grafted mangoes, grafted citrus (mandarins), improved seed, irrigated perimeters, improved goats, etc., these households should be linked in some direct manner. 9 Haiti’s truly vulnerable households are the ones either without or with very little of their own land for cultivation, requiring dependency on share cropping or land renting for crop production. And parcels they do possess are often extremely small in size, and on steep, often rocky, slopes. It would be difficult for a MYAP to realize significant agricultural production enhancements in Haiti without working with less vulnerable farmers on plots of sufficient size and quality to permit increased production. Working on developing irrigated perimeters, even if it means renting and sharecropping, nevertheless provides vulnerable households with more options for food security. 15 Graduation Strategy for MYAP FFP Programs in Haiti 5. Define the only criterion that is really important and achievable within the life of any specific MYAP to trigger graduation as a FFP recipient community in a „tightly targeted geographic area‟ to: Average global malnutrition (age for weight) rates of children 6-59 months = < 10% for twelve full, consecutive months.10 The MTE has suggested tracking rate changes for both MUAC and PFA+PTFA separately as something to consider – MUAC as a screening technique for possible acute malnutrition and PFA+PTFA for more general malnutrition which of course includes chronic malnutrition within specific localities. 6. Define a „tightly targeted geographic area‟ as the Haitian „locality‟ (sub-division of a section communales); Most MYAPS place rally posts at this level. We would suggest that this alone should be the trigger for graduation of a FFP program. If a specific area (locality) does not qualify at this level of malnutrition, then it should not be targeted for FFP program activities either. We do not say that other development activities are not worthwhile – they are – but if food security for the most vulnerable is the core issue as stated in the program goal, and resources are actually limited, then this is the only criteria we should be concerned about achieving in a sustainable way for the targeted population in the „tightly targeted geographic area‟. Indeed, we would go further and state that the target should probably be closer to < 5%, and not < 10%. All current MYAP PVOs should have localities that meet these criteria by this time (and do), and should, even now, be in process of withdrawing direct support of some MYAP activities. What would „withdrawal‟ mean? This could mean ceasing food supplements except for the severely malnourished (if any); it could mean withdrawing agriculture and other livelihood activities, less field presence of MYAP employees in the locality. Note: the level of variability of Growth Monitoring data (PFA, PTFA) is high, particularly when looked at across a number of localities or aggregated. We are not suggesting that the MYAP use such growth monitoring data as a population level indicator – it is only useful at the more restricted geographic level suggested. If community involvement can be engaged in this issue through CMAM initiatives, this will further improve the usefulness of these data as an indicator of improvement at the locality level. 7. Continue to monitor the graduated rally post localities – through the Col Vols who would remain in place and continue their voluntary services through rally posts and support of Mothers Clubs - until the end of the MYAP to verify the levels of malnutrition rates; data would be acquired by monitoring of malnutrition rates (PFA, PTFA) and MUAC measurements from registers kept. Registers would need to continue to be made available each year. 10 Where does this come from? CRS, in their Haiti MYAP proposal, suggested this as a possible trigger for graduation. However, Haiti MYAPs may be able to come up with something even better. For instance ACDI/VOCA noted that, as far as they were aware, the only internationally recognized indicators for acute malnutrition are 10% (state of alert) and 15% (state of emergency). Perhaps an indicator using the MUAC measurement would be preferable. A twelve month period is suggested because anything shorter does not adequately address seasonal fluctuations in weight due to reduced food intake or illness (diarrhea for instance) that are more prevalent in some months over others. 16 8. Document, perhaps through case studies, examples of increasing ownership of local communities in dealing with acute malnutrition within their localities. The emphasis for reinforcing MYAP activities in the Lower Central Plateau should be first to fill the gap of MCHN interventions that have not been realized to date. Addressing severe malnutrition among children 24-59 months could be addressed through the CMAM approach. WVH has already commissioned a report looking at this option, and good recommendations have been put forward. 9. Continue to emphasize in every way possible the strengthening of community structures – especially Village Health Communities (CHV) in support of the CMAM approach. By including the monthly monitoring of malnutrition, including specifically severe malnutrition using MUAC measurements as a proxy for severe malnutrition, impact should in improving child health, reducing mortality as well. The VHCs currently have responsibility for advocacy and identification of health promoter and Col Vols. The MTE team concurs that „their scope of activities should be increased to include identification of malnourished children, development of emergency transport plans for pregnant women, newborns, and children, advocacy for improved access to care through mobile clinics (or fixed points), and additional cadre of health promoters.”11 Training would continue to be provided by the MYAP on leadership, management, and advocacy. Integration 10. Always link agriculture production and livelihood enhancement activities with the MCHN health/nutrition program activities, with a more intentional focus on identifying vulnerable households with children less than 5 years of age or other highly able groups within the community (orphan and widow headed households, households with PLWA and other chronic diseases (blind, TB, crippled). Since children fall out of ‟normal‟ status into malnutrition, and back again, this larger target group would assure greater continuity in impacts received from training received. This increases the likelihood of significant impact on the food security and well being of these households. While undertaking these activities separately with different households may have impact, it will almost certainly be less than when combined. Livelihood Activities (Livestock, Agro-forestry, improved seed support, etc.) 11. Consider prioritization of approach to providing animals (or cross-breeding opportunities) directly to vulnerable household women capable and interested to manage the – within Mothers Clubs - if the MYAP is going to be working with animals. MYAP support for increasing goats, or introducing improved breeds, for instance, must be coupled with different techniques for ending free-ranging animals through improved forage (either home grown, or purchased), as goats are a major contributor to deforestation in all MYAP regions. With such ownership, women should also be able to control the money from future sales of these animals. The whole household will benefit. Always provide training in 11 Personal communication, by email: WVH MYAP responses to review of initial draft of this MTE report, July 2010. 17 keeping books on costs and tracking income for these activities. These data in their turn become the MYAPs opportunity to track impact. Approach to Mothers Clubs 12. Revise Haiti MYAP approaches to Mothers Clubs to focus first, and foremost, on encouraging self-selecting small groups of mothers within a tightly defined geographical area to organize for the care of their household children, with primary focus on mothers whose children are under 5 years of age. Program monitoring will focus on FFP common indicators to assess impact on reducing malnutrition and increasing livelihood food security on the households of these women and their children and households. One should expect the numbers of Mothers Clubs to increase naturally from year to year as well as the number of mothers within existing clubs. Natural growth in the number of Mothers Clubs will happen if mothers see the positive benefits for household children over time by participating mothers, and if they are encouraged to form in this way. Within CRS regions, Mothers Clubs (of 20 cohort mothers), as currently defined, are for the most part stagnant or rigidly non-evolving women‟s groups. Or, because of the focus towards physiological developmental stages within WVH programs, there is little common bonding among mothers within these Mothers Clubs. Mothers Clubs in these MYAP regions are not likely to continue long beyond the life of these programs. They will not become part of the local social fabric – with the possible exception, the MTE team believes of some that have reorganized for other economic pursuits or for group savings and loan purposes. MYAPs should also track the numbers of children < 5 years of age in each Mothers Club, as well as the number of older household children as direct beneficiaries of the program. Activities currently in place in all three MYAP areas could be built upon to move in this new direction. Women currently involved in the „old mother‟s clubs‟ (CRS, from earlier DAPs) might be encouraged, because of the special training they have been receiving for so many years, to reform into new locality based and self-selecting groups of mothers, with some of these women encouraged to become Leader Mothers of the new groups, if the new group should decide to ask them to do so. It must be their decisions however, not one made by the PVO‟s recommendation, or Col Vols recommendations. WVH‟s splintered mothers „clubs‟ could be permitted to reform into locality based small mixed groups (including children of different developmental ages), but also choosing among themselves one mother who would be their group leader (a position that can change over time, as they wish). These are their groups, not a WVH or CRS selected group. It would then be the WVH health agent or CRS Col Vols community health workers job to meet with one or two groups of these Leader Mothers with an intensive monthly training message, as well as other instructions for keeping registers, monitoring of growth cards, doing home visits of their mothers, etc.). Because this may be perceived as a „new approach‟ within areas where earlier arrangements are better known, the MTE team would encourage MYAPs to build in incentives for mothers to form in this way. Besides the obvious advantage of linking up „natural groupings‟ of mothers within small areas (neighbors) for self-support and encouragement, Mothers Clubs could be told that the top 10% of clubs in a particular locality each year will receive public recognition and prizes for those keeping the best records, building up the greatest capital for savings and loan, or most successful vegetable production out 18 of their own household gardens. The „prize‟ might be vouchers to receive 10 grafted mango trees, a marmite of improved bean seed, or grafted citrus trees, an improved buck for breeding that a member of this club would be trained to manage, support to initiating household drip irrigation kits, etc. This would place the „giving‟ of these assets to households linked to some change in behavior sought by the program – and move away from simply free giving of these assets with little or no conditions at all placed on their receipt. All of these livelihood activities would encourage focus to these improve techniques, building the desire of others to participate within the community. 13. Encourage each Mothers Club to nominate a volunteer Leader Mother who will be the point person for their group and receive the targeted training support of the PVO community health worker, who may be the Col Vol, in the case of CRS, or a WVH employee, in the case of WVH or ACDI/VOCA. A Col Vol should not be elected as a Leader Mother. Col Vols training of Leader Mothers (as a group) will also help remove the stigma and possible down-side of largely male community health workers explaining and discussing often very intimate issues with mothers. Size of these Mothers Clubs should be encouraged to not be too large, preferable around 10 mothers, representing households with children of different ages under 5 years of age (though older children may be present too). This permits natural growth of specific Mothers Clubs should friends or neighbors of existing club mothers become pregnant and wish to improve the care of their children. Over time (after 5 years for example), some mothers will have second and third children, but will remain with the original group should they wish to. Continued association can be encouraged by further encouraging Mothers Clubs to promote bio-intensive vegetable gardening for supplemental foods during difficult months of the year, and branching out into any number of potential economic pursuits of individual choice through savings and loan groups or sub-groups within the same Mothers Clubs. 14. Train Leader Mothers to keep Mothers Club registers (a simplified version of that kept currently by PVO community health agents or volunteer Cols Vols). Standard registers should be created by MYAPS for distribution to these Mothers Clubs. Train Leader Mothers to conduct the growth monitoring (weight for age) of the children under 5 years of age for all the mothers in her small group – perhaps using Mid-Upper Arm Circumference (MUAC) tape method. The club‟s register will track each child‟s monthly change in category (yellow, orange, red) on the children‟s growth monitoring cards. These can be confirmed when mothers attend rally posts to receive other more formal social services from Cols Vols and/or other community health workers. MYAP Training 15. Provide each Col Vol with at least one 8 ½ X 11” visual training flip-chart, preferably in color, that covers all of the major themes prioritized for regular training at the rally posts, and within the Mothers Clubs. There should be enough of these so that Leader Mothers can borrow these from their Col Vols for use in her own training of her own group of mothers. This is a MYAP expense that should be given high priority as one of most important training tools that can be provided to localized community groups. Between them, World Vision Haiti, CRS, and ACI/VOCA have identified excellent visual materials. A working group should be set up to create a standardized set of materials for all Col Vols and Mothers Club Leader Mothers to use. 19 The Role of ‘Groups’ & Support for Local Entrepreneurship 16. Consciously avoid the trap of seeing the creation and training of „groups of farmers‟ for various purposes as an end in itself. Economic development activities can and should be successfully launched through group training events (demonstrations), but the objective should always be to use the „group activity‟ as a launching pad towards identifying and then developing individual entrepreneurship – whether this is growing high value vegetables or fruit, raising goats or chickens, building silos, producing high potential seed for sale within communities or regionally. When the groups themselves become the focus for all program assistance, the long term sustainability of these activities may be jeopardized after the departure of the project. When a project continues to provide the seed to farmers, always give the trees to the group, or expect the community groups to give out their seedlings „for free‟, always provides free chickens or goats to a group (even when requesting pass-ons), there is little incentive for individuals or small groups of such individuals to launch out on their own into these potential economic areas of endeavor. They would not be able to compete against the perception of people that an NGO or other donor will eventually give these to them for free – so they wait.12 Encouraging farmers to develop their own labor intensive soil conservation works on their own plots of land – many of which are not even owned by those most vulnerable (but rented or sharecropped) - quickly fail in the face of large food for asset activities in the same regions where such work is paid for with either food or cash by different organizations. 17. Focus greater attention on developing entrepreneurship of individuals actually interested in undertaking some identified business opportunity, and move away from dependence on „groups‟ to promote all program activities (e.g. establishing tree nurseries, multiplying seed, grafting skills, managing goat breeding stock, etc.). Link training provided to some kind of effort manifested by the trainee. ACDI/VOCA expectations to receive back a marmite of improved seed for every marmite given at planting time would be an example of this principal. Group efforts do not have a good track record for continuing beyond the life of the projects that created them. Provide the training and support needed to help these individuals develop their nursery, learn how to obtain the kinds of seeds needed, where the best source of sacks for seedlings can be obtained, and then the skills in nursery establishment and perhaps fruit tree grafting and marketing of products and services as well. To be successful, one also needs people who offer this training to themselves be entrepreneurs – otherwise it is the blind leading the blind or simply theoretical. 18. Move away with encouraging groups to „provide free services‟ to their members – like free trees, even grafted fruit trees, free mating of animals with breeding stock, free grafting of mango tree seedlings, free distribution of chickens, goats (even when a pass-on is required). Dependence on „groups‟ to 12 ACDI/VOCA notes that “During the last two decades, there is a tendency for people to wait for Government or Institutions to assist them, completely free of charge, related to most agricultural interventions. It may take longer than the5 year life of the project to change certain mindsets. We will need to add to the project an outreach component with the objectives of promoting this change of mentality, strengthening the institutional capacity of the numerous CBOs the project assists. Under the best circumstances, farmers are reluctant to pay even 1.5 gourdes for a coffee seedling that is worth 20-25 gourdes”. (July 2010 email communication from Emmet Murphy to Richard Swanson). 20 manage these assets can be a dead end strategy. Identify rather model farmers or entrepreneurs who are willing to put in their own time and effort to develop a business of some kind locally. Then, if the PVO wishes to distribute agro-forestry trees within an area, purchase them from these people, or give vouchers to beneficiaries that will be worth a specific number of trees to be obtained at the nursery. The nursery owner can then redeem this from the PVO. For example, WVH, CRS, and ACDI/VOCA MYAP efforts to promote entrepreneurship by training individuals in top grafting of fruit trees is a good step in this direction; or WVH training individuals to create metal silos for seed grain. Support needs to also be given to these trained individuals in marketing, costing their services for realistic prices. If the sale price for a water sprinkler will cost more than one purchased in the town market, or for a plastic bucket, then this is not a business that will long exist. One must not assume that, given the training to do something, that the entrepreneur will be able to establish themselves successfully. Training some farmers in techniques for animal health care and then believing that these people can now use their skills to provide services to their neighbors for cash could have important impact, but the MYAP needs to track the actual results of these actions to see if they are in fact leading to the results hoped for. Case studies need to be developed on successful ventures, with financial and output data over time demonstrating the impact. Realistic assessments of specific ventures (like creating silos), must be made prior to giving the training; otherwise these efforts will lead nowhere. New creative thinking needs to take place by MYAPs to change perceptions about free things. Perhaps written agreements with farmers receiving free grafted fruit trees might stipulate that a certain quantity of fruit from these trees must be given freely to their CBO group association for distribution to the needy in their community for a five year period for example; or that grafting stock may be taken from their trees for new grafts for a period of five years. As the MYAP may be gone by the time these actions could take place, the local CBO implicated in the activity would need to follow through. Monitoring and Evaluation 19. Seriously consider decentralizing M&E management within all three MYAPs. The highly centralized M&E management approach being used by all three MYAPs places an unrealistic burden on a small group of individuals to track the key MYAP program information data sets. By decentralization, we mean that the operational leaders within each MYAP component should be made responsible for acquiring and tracking the data they are most responsible for, including those included in the IPTT. Those handling commodities, for example, should not simply be sending up information about # of metric tons of food distributed to X number of people, but be looking at the link between those who were supposed to receive these commodities and those that actually received it within different localities. This would help them improve distribution systems when problems arise – as they have – in many households not receiving these food kits. It should not be the MYAP M&E manager or one of his or her assistants to put this together for these managers – an impossible task that is not taking place, in any event. The M&E unit, at the program level, can bring the data together from these different component leaders for quarterly reports needed by MYAP Managers. As it is, data are being tracked to meet the quarterly needs of the IPTTs, without sufficient analysis of data gathered at regional sites on outcomes and impacts at the local level on the most vulnerable – which is what the FFP program is all about. 21 20. Reduce the number of output indicators required in the Indicator Performance Tracking Tables of each Cooperating Sponsor and increase the number of outcome and impact indicators. Tracking all the data required by USAID and FFP is very costly in terms of time spent recording these data through different levels of the program aggregation process. PVOs have not been able to report on some of the indicators at all – so they remain blank from one quarter to the next. In other cases, FFP and USAID both require indicators that are very similar, but different enough to require separate tracking and reporting. One of the versions should be selected, and the other dropped. There are some indicators for which the End of Project (EOP) targets are very small (less than 10) (e.g. „# of new technologies or management practices made available for transfer as a result of UG assistance‟ – with an EOP target of 5. These indicators should be dropped, and only reported on within the narrative portion of the quarterly reports, when appropriate. The MTE reports on individual MYAPs provides some suggestions about changes here. (19) Hire local secondary school students – or other trained people, within the representative MYAP regions, as short term interns to enter sets of data about local rally Posts/posts registers and Mothers Clubs into time series data. These data would then be analyzed by PVO regional health coordinators (in a decentralized M&E approach) to monitor program impacts locally and to identify where the program can begin an exit strategy, or were additional attention is needed. There is not enough analytical work being done on these data, to justify the cost of having obtained it in the first place. All three MYAPs have extremely detailed, excellent data about their program beneficiaries down to the rally post and Mothers Club levels that should be more effectively used. (20) Provide regional health coordinators with GPS instruments to record the latitude and longitude coordinates of:  Rally Posts  All Health Clinics, Dispensaries, „Fixed Points‟, or „Mobil Clinic‟ locations  All Food Distribution Points  The Meeting Points of Mothers Clubs These locations could then be geographically mapped using very simple mapping software – perhaps using Google Maps with their $100/user software, to overlay such points on the area maps of Haiti. This would provide a visual representation of the geographic spread of these GoH health service outreach through MYAP programs in the concerned departments and the location of these health care centers, which is impressive indeed. World Vision Haiti has begun to register these data for some of their Food Distribution Points. ACDI/VOCA already has GPS points for health centers, dispensaries, and fixed points (or food distribution points). 22 Early Warning System (21) Consider adding one or more specific nutrition indicators to the data being regularly tracked within EWS systems. Because of the extensive regional coverage of all three MYAPS through their rally post systems, and the monthly data available for malnourished children coming from these, it would seem that such data could be highlighted for EWS monitoring and reporting everywhere. But the data would need to be reviewed for specific rally posts in specific localities, not as an aggregation of all rally posts within a section communale or commune (such data have little value). Because there are so many rally posts, a number of sentinel posts might be identified and tracked over time (as is done for rain stations or markets). Perhaps each MYAP could classify their rally posts using a system similar to that used by CRS, indicating difficulty of access (access easy, somewhat difficult, or very difficult). Then data reported for such sentinel posts would be disaggregated by these criteria. While malnutrition data do not „warn‟ of future events, they do show trend lines that could point to increasing malnutrition or decreasing malnutrition over time which can serve as a warning of steps that should be taken, or could suggest withdrawal that could take place of some MYAP services provided. Some suggestions about what nutrition indicators might be considered within the EWS, for a selected number of sentinel rally posts, would include (a) average weights for age and/or (b) MUAC. The W/A would be used to analyze trends of underweight children at the locality level and could be useful to track the performance of the MYAP over time. Children referred to fixed points or health center by rally post personnel should be monitored regularly at the rally post – or even more appropriately through the Mothers Club in which this mother and her child are located. The MUAC measurement would help to quickly detect, alert and more quickly lead to treatment of new cases or spikes of acute malnutrition and be a good indicator in early warning system.13 This would be another way to involve the local community in the importance of the CMAM approach, linked the MYAP‟s PM2a efforts. ACDI/VOCA notes that a possible proxy would be the number of new cases of severe malnutrition (kwashikor or marasma) per region. (22) Consider highlighting in future EWS bulletins, after the conclusions and recommendations sections, an accounting of whether past alerts or recommendations made have actually led to some kind of specific action taking place on behalf of concerned communities (i.e. that these data are actually being used as intended) – and who and how this was accomplished. It will also be important for WVH EWS bulletins to begin to appear on the CNSA webpage, along with already existing CRS and ACDI/VOCA bulletins. MYAP & SYAP (23) Care must be given that SYAP funding of activities are complementary to MYAP program activities, and do not undercut years of MYAP efforts. Integrate these as much as possible in terms of activities, allocation of personnel resources, and M&E. Infrastructure repair for irrigation perimeters or 13 ACDI/VOCA notes that the MSPP protocol now calls for ‘weight for height’ rather than ‘weight for age’, as the weight for age protocol has sometimes led to the wrong conclusions about malnutrition levels for an area. This is because the data collected are relevant only for cases reported to the health center or fixed point rather than an indication of rising malnutrition among the entire section communale or entire commune. This might be true for ‘those reporting’ at this aggregated level, since the MTE team has noted that many mothers do not take their children to these referred to locations, after the initial weighing at the Rally Post. The MTE team believes these data are more meaningful when referring to the specific locality in which the rally post is located, and where the original weights were actually taken (not as aggregated data from health clinics or fixed points). 23 the construction of soil conservation structures (gully plugs, terraces) activities with food-for-assets or food-for-cash can completely undermine efforts of PVOs to encourage farmers to develop and maintain such structures upon their own land through their own (voluntary) collective or individual efforts. When food or cash is given away for this, farmers will in the future wait for future events that may bring such „free assistance‟ to their area and remove any incentive to take personal responsibility for actions needed. (24) Initiate food or cash-for-work or assets programs by sub-contracting with local private sector engineering firms or other Haitian private enterprises, but not as contracts with community groups that may have been formed by the PVO, or some earlier PVO in earlier years. It is important to separate to the extent possible, in the minds of the beneficiary communities within a PVOs region of involvement, MYAP activities for community development, and SYAP activities for quick fix cash or food infusion activities to build or repair infrastructure. Repair of primary and secondary irrigation canal structures for example, or the construction of run-off water reservoirs or catchment areas (for vegetable irrigation) are worthy initiatives in the Haute Plateau or on La Gonave – as both will reinforce livelihood and income generation activities supported by the MYAPs. But HOW the work is done is just as important as doing the work itself. SYAP activities should be seen as an opportunity to support local entrepreneurship – who will go out and hire the people needed (or provide the food) for the work to be completed – building local capacity for long term sustainability. The PVO should not be directly involved.14 (25) Amend MYAP agreements to permit MYAP managers to more quickly use SYAP funding when a surge of support is needed – with cost recovery from the SYAP programmed at a later date. The best use of SYAP new funding would be to provide existing MYAPs with the additional resources they need to address need in their regions of activity. MYAPs Collaborating Sponsors have been very effective in quickly moving their regionally placed personnel and stocks of commodities to quickly respond to quick moving and unexpected needs and then to back fill these stocks when they become available. (26) Channel SYAP resources to the leading NGOs within specific regions and avoid asking MYAPs to move into new areas, outside their areas of program efforts, where they do not currently have personnel or other resources in place. This will help to build the capacity of smaller groups, and avoid duplication or overlapping efforts leading to competition and friction among humanitarian groups. (27) Consider purchase of a selected basket of Haiti-grown cultivars for inclusion in FFP delivered food supplements (black beans, manderines, mangoes, sweet potato) to vulnerable populations as one way to not only stimulate the local Haitian agricultural economy but to also provide foods that are already in-country and familiar to Haitians. MYAPs are seeking to increase agricultural production and livelihoods of rural people and this would also be one way to link the MCHN and agricultural components together. Belmont analysis should consider the fact that ANY commodity brought in to Haiti for free delivery to vulnerable Haitians is also removing opportunities from Haitian farmers to sell their own commodities to fill their in-country needs, and therefore depresses their markets. 14 We understand the limitations that are often placed on how SYAP assets can or must be used, or that food is the asset for distribution, not cash. Yet FFP must also consider the long term impacts of these activities on local communities, the unintended consequences, which may in the end do more damage than they help. 24 Partnerships and Working with Government of Haiti (28) Developing close collaborative approaches in working directly with Government of Haiti local and regional agencies in support of health and agriculture initiatives within the MYAPs should always be a first objective of partnering and sustainability. CRS and ACDI/VOCA have been effective in this regard. WVH‟s Community Management of Acute Malnutrition (CMAM) approach is also an excellent way to involve local communities in issues of great importance to households of each community: the well-being of their children. However, it is also important that USAID and FFP be clear with GoH what the intended purpose of FFP Title II programs are, and that they are not intended to be general rural development programs supporting the full range of GoH national priorities within a region. FFP does address some aspects of these priorities, but not all and must remain focused if it is to achieve its own stated goal and objectives. (29) Renew earlier efforts of MYAP partners to meet within each of the major component areas (MCHN, Agriculture, and EWS) to share approaches, lessons learned, and develop common approaches. Such meetings should be planned on a bi-annual basis, for one day, and be held in different geographic locations outside of Port-au-Prince. Each Collaborating Sponsor has what might be considered „best practices‟ taking place and one of these could be the subject of observation and debate at such meetings as well. Topics might include such things as a common approach to Mothers Club savings and loan activities; the best way for a Mothers Club to establish vegetable garden seed nurseries and the use of small scale drip-irrigation systems (and how to resolve the water availability issue) for these gardens; what common format should be used for the registers that Leader Mothers of Mothers Clubs will begin to keep; what common format should be use for tracking the activities of savings and loan groups within Mothers Clubs; how best to prioritize the commodities that should be targeted for value-chain development that have the most direct impact on linking households nutrition with food security , and the linkage to seed and material suppliers and regional and international markets for these commodities. 1.0 Introduction & Overview of Haiti MYAPs Before we begin this review, a couple of observations may be useful to orient ourselves. First of all, how we define and measure „success‟ within a FFP program is important. If by this we mean that a MYAP program has successfully reached the targets it has set for various components of the project, then by this definition, all three Haiti MYAPs could be said to be largely successful. Projects will do everything possible to reach these objectives, even to the extent that the targets become the end themselves, and will drive the program. Quantifiable output targets are no longer seem as the means to an end which should, for example, be the reduction of malnutrition of children under five years of age to a level considered „acceptable‟. 15 Are we trying to impact close to 100% of targeted vulnerable households in some specific area, or is achieving only 50% acceptable, since we want to geographically cover a much larger area with program interventions (and resources won‟t extend this far). Given a gradual scale-up approach, will we ever arrive at our goal, given resources? 15 Since malnutrition in any child is actually not acceptable, defining what might be „acceptable‟ may be difficult, but nevertheless necessary. There will always be some children, because of specific circumstances, will be very difficult to draw out of their malnourished state without much greater intervention over a much longer period of time. 25 Secondly, if a program actually believes that it will not be able to observe or measure impact „already‟ within two or more years into a project, but must wait until the „end of the project‟ to be able to measure this, then the bar is being set too low and project goals and related impact may never actually be achieved. If the „end of the project‟ is also when „impact and goals‟ are to be achieved and measured, then no project will ever take withdrawal or ending strategies seriously. The need for continued interventions will never come to an end as there will always be some „new activity‟, whose addition will be so important to reach our elusive goals. „We‟ve done some agriculture, now let us do some livestock work.‟ Thirdly, if we actually believe that activities increasing agricultural production and income will lead to improved household nutrition, then we only need to consider much of the Western world itself, where household incomes are dramatically better than those of the beneficiaries with whom FFP programs normally target. Doing so, we quickly realize that poor nutrition and eating habits of even those better off can lead to serious health problems for our children – as seen by the overweight epidemic among many children within the United States itself. Close linkage between health and nutrition training and health services needs to be integrated with equally focused agricultural and livelihood activities. Finally, understanding what program goals are must be where all program activities begin. How do we focus sharply, define our target beneficiary population carefully? Will we recognize „success‟ if we see it in front of us? If we have not defined what success will look like, or are careful in targeting, we may never achieve our goals. 1.1 Background The mid-term evaluation for all three of the Food for Peace (FFP) Title II (PL-480) Multi-Year Assistance Programs (MYAP) in Haiti was scheduled for the spring of 2010. Initially with the option to undertake these evaluations separately, the three MYAP cooperating sponsor (CS) partners World Vision Haiti, ACDI-VOCA, and CRS decided that they would do this at the same time, by the same team of consultants, to permit cross-comparison of approaches that could lead to a more integrated program in the future. The projects were required to engage the services of an outside consultant with long term Monitoring and Evaluation (M&E) experience, and experience with Title II programs in particular. Dr. Richard A. Swanson, Economic Anthropologist, with such experience, was contacted in mid-March, 2010 to begin the process for this evaluation. Expectations were that the entire process would be completed by the end of June. In retrospect, given the complexity of each MYAP program, the geographic regions involved, and the two separate surveys that were to be undertaken, the process might have been better served had it begun about a month earlier – given the expected completion date. To meet deadlines, activities had to be compressed, with tight schedules and overlapping surveys. The MTE was to include information from both quantitative and qualitative surveys among beneficiary households within geographical regions in which program implementation has been underway for the past two years (cf.MTE SOW, Annex 1). In the case for both World Vision Haiti and CRS, targeted areas for the MYAP were built upon the foundation of preceding Development Assistance Programs (DAPs) in these areas. Both surveys focused on the direct beneficiaries of implementation over the past two years. For USAID and the PVOs concerned, this MTE was largely to be a qualitative survey effort, with the greatest amount of time and resources given to this effort. The qualitative survey would explore in greater depth the causes and reasons behind trends and issues the CS partners have observed in their own implementation and annual monitoring. The different approaches taken by the different PVOs in 26 addressing the same issues became a focus area for this evaluation, looking closely at the stated objective and desire to „integrate‟ both MCHN and agricultural/livelihood activities. This would include the processes followed for reporting on these efforts and their ultimate impact on the targeted populations. A mid-term review provides an opportunity for possible changes in direction or focus and each CS stated their desire to see this as an opportunity for critical self-examination and assessment, being prepared for possible changes in direction or focus if justified. These MYAPs, initiated in February 2008 for five years16, focused on reducing food insecurity among Haitian households and enabling extremely vulnerable households to respond to shocks, such as that experienced in the three consecutive cyclones of 2009 or the January 12, 2010 magnitude 7 earthquakes that, in the space of about 35 seconds, directly or indirectly affected the entire country and forever changed the lives of thousands of Haitians. The goal of the Title II, Food for Peace (FFP) USAID overall MYAP program is to reduce food insecurity and increase resiliency of vulnerable and extremely vulnerable rural Haitian households by targeting its most vulnerable members: women, children, and youth. CRS perhaps best stated the MYAP programs overall hypothesis in their project document by stating that CRS “expects that by carrying out integrated health, education, and livelihood interventions in tightly targeted geographic areas, food insecurity will be reduced enough over time to warrant exiting a community or area from the Title II program”.17 A principal objective of this MTE is to see if these MYAPs are in the process of proving this hypotheses, and if not, why not. Table 1 below shows that the MYAP is currently being implemented in 35 communes of the following regions of the country, and by World Vision Haiti, CRS, or ACDI-VOCA and their respective sub-contractor partners. The * indicates the area in which the qualitative survey team visits will take place. Within the context of Haiti, MYAP programs work within geographic Departments, which are sub￾divided into Communes, then Section Communales, and finally Localities. Localities represent the lowest level for geographic distribution of MYAP services. Rally Posts are established at this level, though not all Localities in all Section Communales of MYAP programs are covered by rally posts, for lack of resources. Mothers with children under 5 years of age coming to a specific locality Rally Point are formed into one or more Mothers Clubs at this level as well. 1.2 Midterm Evaluation Team (MTE) Strategy The MTE team was led by an economic anthropologist with long M&E experience of USAID programs in different parts of the world. Having himself been the Chief of Party of several long-term USAID funded development programs, including one agricultural development program in Haiti for four years (ADS-II), the realities, challenges and difficulties faced in program implementation are well understood. Experience has also included both mid-term and final evaluations, employing both quantitative and qualitative surveys, of FFP DAPs in Uganda, Zambia, Rwanda, and Malawi. The external consultant was assisted by two professional Haitians with backgrounds in agriculture and health/nutrition, and a fourth team member with a strong quantitative/statistical background to help develop and coordinate the 16 February 2008 – February 2013. 17 KOLE ZEEPOL, CRS MYAP Project Document, FY 220-2012, August 22, 2007, p. 36. 27 quantitative survey portion of the MTE using some 30 enumerators. The external consultant, and the team assisting him, focused on four principal sources of information to complete this mid-term evaluation: (1) Review of existing project documentation (cf. Annex 4), MYAP CS partner Indicator Performance Tracking Table (IPTT) data sets (cf. Annex 6), including the 2008 MYAP quantitative baseline survey; (2) Review and analysis of quantitative Time Series Data from sampled beneficiary populations, and other data sets currently being obtained by MYAP CS partners, and reported in Quarterly Reports or elsewhere; (3) Quantitative Survey to be undertaken by in-country local-hire analyst, Mr. Antoine Wesner, engaged for this purpose. The external consultant, with assistance from Ms. Vicky Michener of FANTA, provided input into the sampling and questions to be asked in the survey, and guidance in the analysis of results. The survey instrument developed for this purpose is provided in Annex 7, and the MYAP MTE quantitative survey data tables are provided in Annex 8. 28 Table 1: Geographic Areas Targeted by Haiti MYAP (World Vision Haiti, CRS, ACDI-VOCA) Department Communes Total Population Target MYAP Beneficiaries Department Communes Total Population Target MYAP Beneficiaries Upper Plateau 6 347,000 330,297 La Gônave 2 84,250 80,072 Hinche* Anse-à- Galets * Thomonde* Pointe-à-Raquette * Boucan Carre* Artibonite (A) & Lower Plateau (LP) 8 605,000 130,000 Thomassique Saut d’Eau (MARCH) Cerca-La-Source Mirebalais (LP) (MARCH) Cerca-Carvajal Lascahobas (LP)* (MARCH) Maissade (SAVE) Savanette (LP) (MARCH) South-East 7 253,427 72,750 Verettes (A)* (HAS) Cote de Fer * Petite Riviere (A)(HAS) Bainet * Dessalines (A) (SAVE) La Vallee South 11 280,175 196,123 Thiotte St Jean du Sud Grand Gosier Ile-a-Vache Belle Anse* Fond des Blanc Anse a Pitre Aquin * Roche-a-Bateau Niippes 1 95,715 67,000 Coteaux Miragoane Port Salut Port-a-Piment Tiburon Les Anglais * Chardonierres * World Vision Haiti ACDI-VOCA CRS 29 (4) Qualitative Survey, led by the external consultant, Dr. Richard Swanson (economic anthropologist), and assisted by two local hire personnel (health/nutrition: Dr. Colette Vilgrain and agricultural specialist: Charles Gregory) during the month of May 2010, among program beneficiaries, stakeholders, and partners. Key prompting questions are included in the protocol (Annex 3) and individuals with whom the MTE team met, interviewed and held focus groups are included with those listed in Annex 5. Together, these four sources of information were used to review the accomplishments of the program for each CS partner and the MYAP as a whole, considering USAID/Haiti‟s own assessment of the program and their suggestions for program change or strengthening. This final report evaluate if the MYAPs are moving towards reaching stated objectives of reducing food insecurity and increasing resiliency among vulnerable communities (and their households). To further guide the process of the MTE, the external consultant developed a detailed logistics table with a proposed scheduling of key events for both surveys (cf. Annex 2). This and a proposed timeline was provided to the Haiti MYAP field teams several weeks prior to the arrival of the external consultant in Haiti so as to begin the planning process and background preparation. A MTE Protocol was also prepared, prior to coming to Haiti, to further guide this processes, providing the field teams with an idea of what the expectations of the MTE team would be. This document was further refined during the process of implementation (cf. Annex 3). One of the first expectations was a request to each MYAP leadership team to prepare for the MTE team‟s arrival a PowerPoint presentation, with hardcopy, focusing on the following 4 specific questions: (1) What are the most important achievements of the project up to this point for each major component of your program? (2) What were the greatest obstacles faced in achieving project objectives for each component? Include impact of recent earthquake here. (3) What, in your opinion, are the most important actions to take in the remaining life of this project for each component of your program? (4) Which beneficiaries should the MTE qualitative evaluation team meet within each component to discuss impact and issues? We will have about 6-7 days for each CS partner in the field, so our field time must be well focused and pre-planned. During the first week of the MTE, an opportunity was given to each MYAP CS to provide the requested information above in writing.18 Following the presentations the CS gave to the MTE team; discussions following that did begin to probe these issues. 1.2.1 Mid-Term Evaluation (MTE) Methodology A Title II mid-term project evaluation, as recommended by Food and Nutrition Technical Assistance (FANTA), should focus towards project impact on the beneficiaries within which program targeted geographical areas. 19 One FANTA technical document states that a mid-term evaluation for a Title II 18 None of the three MYAP teams responded to this request, but presented to the MTE team project descriptive PowerPoint presentations they had developed earlier, in the case of CRS and WVH, at the beginning of the MYAP itself. 19 WVH noted to the MTE team, in its initial email comments to a draft version of the MTE report in July 2010, that “FFP guidelines specifically discourages focus on impacts/outcomes in a mid-term evaluation (my emphasis), citing an internal memo (which the MTE team had not seen before) by CHA/FFP/Director, Jeff Borns, on July 30, 2009, that (mid-term evaluations) (a) 30 program “is oriented toward effects on participant households.”20 We will define „effects‟ as an evaluation to determine the actual results (outcomes, impacts) achieved to date by the project, lessons learned and best practices, so as to permit midterm course changes (if necessary) to the program to fully achieve program objectives. The Scope of Work for this Mid-Term Evaluation (MTE) clearly states that the objectives of the mid-term evaluation are: “…to assess progress toward meeting programs objectives, both individually and as a whole, as well as, to assess the type and level of impact of the earthquake in MYAP areas. In addition to that, it is envisioned that the MTE will help determine the need for adjustments in strategy as well assess the effectiveness of both MCHN and agriculture components and program activities. Specifically, the mid￾term evaluation will pursue the following objectives: The evaluation should asses the project‟s progress in meeting its expected results and the likelihood of attaining its outcomes and impact as defined in the agreed Performance Monitoring Plan (PMP); To explore and explain why certain program activities are not performing as expected; Determine the appropriateness of the overall program-design and strategy for addressing the Intermediate Results. Other elements of program design such as indicators, targets, staffing plan, information system and budget should also be examined. Provide an assessment of progress and quality of MYAP implementation that can serve as a management tool to refine program activities and improve internal management of the program. Emphasis will be placed on the implementation process and its effects at the beneficiary level Provide reliable information regarding how the recent earthquake directly or indirectly impacted the MYAPs To highlight and identify ways to strengthen management and achieve sustainable results; To provide accountability to all stakeholders: USAID, PVOs, beneficiaries and the Government of Haiti”.21 learn from successes and also acknowledge and learn from problems; (b) assess implementation progress and roadblocks rather than focus on outcomes and/or impact; (c) are not required for food aid programs lasting less than 5 years; (d) are not required to be quantitative or population based, participatory qualitative assessments are encouraged; and (e) do not have to be external and/or independent, meaning food air program staff may conduct all aspects of the study.” The MTE team did not focus on impact, as this is only one of many other issues looked at by the MTE team. But we certainly had to be looking for this by the middle of the MYAP life if we were to achieve our own SOW objectives. We do not see any necessary contradiction between the two FFP statements about what needs to be accomplished through a MTE. We attempted during our field interview to determine if MYAP program beneficiaries were receiving the anticipated benefits and if, at least for this sub-group of the area population, the expected outcomes or impacts were being realized. Outcomes are a form of short-term impact, and if these were not associated with project outputs after 2.5 years, then there would be a problem. If we did not do this, then there would be no point to a MTE in the first place – we wouldn’t know what was working well, and what might needed improvement. The MTE team is not sure why some WVH management personnel were so defensive about this issue. 20 USAID FANTA Technical Notes #3, Patricia Barnard, “Title II Evaluation”, April 2002. 21 Annex 1, WV MTE Scope of Work, p.4. 31 Of great importance, after two years and a half years of project implementation should be the question: Are program activities – at least among the targeted beneficiaries – leading towards reaching the program objectives, which includes the anticipated outcomes and impact suggested by the initial project hypothesis above? Do impacts appear sustainable for at least these people and households concerned? These are the central issues this evaluation will seek to explore, while attempting to answer the many specific questions asked within the SOW. Equally important will be to assess the impact of the recent devastating earthquake in Haiti on program implementation. Both the quantitative and qualitative surveys were done in the same regions of the three MYAPs. Information from both the qualitative survey and further analysis of the data sets currently regularly obtained by each CS partner from a sample of rally posts, Health Clinics, and beneficiary households will help fill in details about what is actually taking place within program areas of intervention. At the end of the MTE team‟s efforts, a debriefing was given June 3 to USAID/Haiti and CS MYAP managers, which included a PowerPoint presentation and a first draft of our major recommendations (cf. Annex 9). 1.2.2 Quantitative Survey The quantitative survey focuses on only project beneficiaries of the principal MYAP activities in MCHN and agriculture.22 It is built upon some of the key information acquired through the MYAP‟s baseline survey instrument, and the data points developed there.23 Though FANTA apparently does not always require a quantitative survey for Title I programs at the mid-term review (leaving impact studies until the EOP evaluation), the PVOs all had a desire to have some sort of quantitative measure of the direction of impact or change that might be taking place in their major areas of program intervention, to help inform the qualitative survey; their IPTTs also called for this. Nevertheless, the numbers of questions asked in the qualitative survey are purposely few – for budget reasons - and carefully focused to only the most important data points to inform on possible impact on program beneficiaries. All three PVOs also have extensive data sets that they are tracking for their respective IPTTs and other data sets that the MTE did review and organize into this report, as appropriate, and as time allowed. Greater detail about this survey, including sampling framework, is provided in the MTE protocol of Annex 3. 1.2.3 Qualitative Survey The qualitative surveys in each PVO region expanded beyond looking at the key indicators and the results being reported upon these IPTT data to include probing questions about process, impact, diffusion, sustainability of program activities and the continued vulnerability of targeted populations. Information gleaned through this process represents the bulk of this MTE report. Following each qualitative survey in 22 Some of the PVOs, and WVH in particular wanted to include some control groups (non-beneficiaries) in same regions in which they worked to see if there was any difference between the two populations. Though some control households were included in the quantitative survey, it is not apparent (given the sample size) that such comparisons will be valid. 23 This survey was led by Food and Nutrition Technical Assistance (FANTA), in collaboration with the Haiti CSs for the purpose of establishing baseline values for key identified indicators. 32 each CS region, the MTE team gave the field teams a debriefing of our initial observations, findings, and recommendations. Undertaking a qualitative survey is more of an art than a science. Though the initial probing questions that are asked of different individuals or focus groups are important, it is equally important to understand that one can never predict where these initial questions will lead. As important as the questions being asked during these interview events, the observations and non-verbal communication that takes place around these interview events are often equally important. The qualitative survey team members must themselves have had long years of experience working with small farmers in similar situations because what is often said initially by people being interviewed cannot be taken at face value, or be true at all. Knowing how to ask questions, or rephrase them, to get to the bottom of issues is essential and requires perseverance and patience. The MTE team was frequently told things that we knew, intuitively or by experience, to be untrue, and by further questioning would learn to be quite different from the first responses. For example, farmers saying that they „owned‟ the irrigated plots they were cultivating was usually not true, as most were either share-cropping or renting their plots. Or, farmers saying that there were few months during the year when they were „food secure‟ was usually misleading. In reality, most farmers benefiting from MYAP agriculture and livelihood activities over the past couple years are actually food secure for 10 and 11 months of the year – if not all 12 months – because they only think about what „food is actually in their granaries‟ as a measure of „food security‟. 24 The 2010 MTE quantitative survey among Haiti MYAP beneficiary households received an average response of just under 5 months of adequate household food provisioning – a response that is patently false – but similar to the responses given during the baseline survey in 2007.25 The qualitative survey team found this not to be true at all, and that households, when pressed, were almost embarrassed to admit that they indeed had enough food most of the year. They are not considering that they may have standing crops in the field that can be eaten (i.e. manioc, sweet potato, plantain), or household accumulated animals (from sales of crop surplus) that they will sell, if need be, or they have money put away for a special time of need. One of the strengths of a qualitative survey is that these kinds of issues can be explored in greater depth. 24 By this definition, most Americans would be „food insecure‟, because the actual food in our refrigerators or in our food cupboards would not last much beyond a few weeks or months, at most. Our strategy is to go to the store and get more food, each week, to restock. Rural Haitian farmers‟ strategy is to have a diversity of crops in the ground during their two agricultural seasons each year, as well as to diversify to chickens and goats. If a goat needs to be sold, a mature buck or doe will be sold, some of the money is use to repurchase right away a young animal, and then use the rest of money for food or other household needs. 25 World Vision Haiti noted the „average number of months of adequate household food provisioning‟ as 5.6 months, CRS 5.1 months, ACDI/VOCA 5 months during their baseline in 2007. The same enumerators used in the baseline survey were used during the mid-term survey. This FANTA indicator is not a very meaningful or useful one, and depends greatly on responder‟s perception of why this question is being asked. In Haiti, it is prudent to say one is food insecure, because one does not want to jeopardize possibility of getting „free food‟ from World Vision Haiti, CRS, or ACDI/VOCA. 33 1.3 USAID Haiti Economic Growth, Investing in People & Governance Strategic Framework The FFP MYAP programs within Haiti fit well within three of USAID Haiti‟s strategic program objectives, as illustrated in the CRS slide below: SO #11: Expanded employment and sustainable livelihoods (Economic Growth Program Objective) and SO #12: Increased access to quality social services (Investing in People Program Objective). SO #13: Strengthening civil capacity of populations at risk (Governance Program Objective). 1.4 MYAP Cooperating Sponsor’s Program Strategic Frameworks and Mid-Term Objectives 1.4.1 World Vision Haiti World Vision Haiti‟s Title II program goal was to “reduce food insecurity and increase resiliency of vulnerable and extremely vulnerable rural households in four regions of Haiti”.26 At the outset of the new MYAP program in 2007, replacing and building upon their former Development Assistance Programs (DAPs), World Vision Haiti put forth its development hypothesis on how it would achieve this goal through its stated objectives: The hypothesis, as framed through strategic objectives and intermediate result statements, states that (key concepts bolded): “The underlying chronic causes as well as the acute symptoms of food insecurity in Haiti can be addressed through two synergistic strategic objectives, namely through (SO#1) Improved Nutritional and Health Status of Targeted Vulnerable Groups, and (SO#2) Improved Productive and Profitable Livelihoods for Vulnerable Groups. To accomplish this, World Vision Haiti further developed five intermediate results that were to lead MYAP program cooperating sponsor (CS) partner activities. Each would: (1) Improve the nutritional and health practices of targeted vulnerable populations (SO 1); (2) Improve the quality of and access to health services (SO 1); (3) Increase food production and household assets (SO 2); (4) Enhance market-based livelihoods (SO 2); (5) Rehabilitate natural resource resiliency and local response capabilities (SO 2). Observations: Like CRS and ACDI/VOCA below, World Vision Haiti stresses the importance of what they call their two „synergistic SOs‟ to achieve program objectives. The implied synergy is between health/nutrition activities and agriculture/livelihood activities. The key concepts bolded above provided a major focus for the mid-term evaluation‟s assessment of WVH activities to date. To implement the hypothesis among Haitian households, each CS developed its own unique approach to respond to their common goals and objectives. World Vision Haiti‟s framework highlights the concept of “targeted vulnerable groups” and „food insecurity‟ for FFP activities. 26 World Vision Haiti PowerPoint presentation on proposed MYAP program, slide 4. 34 1.4.2 Catholic Relief Service (CRS) CRS “expects that by carrying out integrated health, education, and livelihood interventions in tightly targeted geographic areas, food insecurity will be reduced enough over time to warrant exiting a community or area from the Title II program”.27 This led to the development of their two strategic objectives (SOs) and 5 Intermediate Results (IRs): S0 #1: Vulnerable communities have increased rural productivity in environmentally sound and economically profitable ways. IR 1.1: Vulnerable communities practice improved soil and water conservation techniques IR 1.2: Vulnerable communities have increased farm and off-farm income SO #2: Vulnerable communities have reinforced their human capital. IR 2.1: Vulnerable households have adopted improved sanitary and health practices IR 2.2: Vulnerable households have improved access to quality basic education IR 2.3: Highly vulnerable individuals and households have improved living conditions Out of the structure above flows the format for the IPTT, and the 61 indicators which fall within each SO and IR. CRS also made the effort to clearly link their activities to the USAID, FFP, GoH, as well as other donors, with one illustration shown below. Presentation MYAP 2008-2012 29 CRS USAID/Haiti Strategy FFP Strategy GOH Strategy SO1: Increased rural productivity IR1.1: Improved soil & Water conservation practices IR1.2: Increased farm and off-farm income SO11:Expanded employment and sustainable livelihoods PC2: Support population at risks PC22: Asset protection PC23:Sustainable management of natural resources PC18: Private sector competitiveness PC11: Strengthening civil society of populations at risk (SO13) SO: Food insecurity in vulnerable populations reduced IR2: Title II impact in the field increased IR2.1 Human capabilities protected and enhanced IR2.2 Livelihoods capacity protected and enhanced IR2.3: Community resiliency protected and enhanced IR2.4: Community capacity to influence factors (decisions) that affect food security increased Agriculture: Improve access to credit – rehabilitation of agricultural infrastructures – Investment in storage and marketing structures and marketable products Environment: Promote environmental of action plan- creation of tree nurseries – environmental management Water: Increase coverage - 27 KOLE ZEEPOL, CRS MYAP Project Document, FY 220-2012, August 22, 2007, p. 36. 35 Presentation MYAP 2008-2012 30 USAID Strategy FFP Strategy SO11: Expanded GOH Strategy employment & sustainable livelihoods PC2: Support populations at risks PC22: Asset increase and protection SO12:Increased access to quality social services PC32: Improve child & health nutrition PC33: Improve maternal health and nutrition PC26:Improve equitable access to quality basic education PC28: Improve quality of workforce through vocational training & technical education (SO13) PC11: Strengthening civil society of populations at risk SO: Food insecurity in vulnerable populations reduced IR2: Title II impact in the field increased IR2.1 Human capabilities protected and enhanced IR2.2 Livelihoods capacity protected and enhanced IR2.3: Community resiliency protected and enhanced IR2.4: Community capacity to influence factors (decisions) that affect food security increased Health & Nutrition: Preventive measures to reduce infant & maternal mortality _ Decentralization of health services Education: Improve quality of public and private schooling – teacher training – School cantines HIV/AIDS: Reduce Risk and vulnerability factors Safety Net: Improve protection for vulnerable populations Emergency: humanitarian aid and post disaster relief CRS SO2: Vulnerable households have reinforced their human capital IR2.1: Adoption of improved sanitary & health practices IR 2.2: Improved access to basic education IR2.3:Improved living conditions Observations: Note the SO 1 focus on communities and SO 2 focus on households and individuals. The CRS framework suggests that most of the communities and households in their targeted areas are perhaps vulnerable and food insecure. The justification CRS provides for their choice of geographically targeted areas also suggests this perspective. Looking at the first map above (page 4), basing a definition of poverty by community access to health services, most CRS targeted regions fall within the „weak‟ to „moderately weak‟ categories.28 Based on the MTE team‟s subjective observations of all three MYAP CS partner regions, the CRS region appears somewhat more favorably endowed than most WVH communes (with the exception of the Low Central Plateau and Artibonite regions) and ACDI/VOCA communes that fall within the „extremely weak‟ and „very weak‟ designations. 28 1. Extremely weak; 2. Very weak; 3. Weak; 4. Moderately weak; and 5 less weak. 36 1.4.3 ACDI/VOCA The strategic framework developed by ACDI/VOCA in their Performance Management Plant for FY 2010 for the Haiti MYAP included the chart below that clearly lays out the overall program goal, with three linked objectives. 29 The same document presented the series of indicators that would be monitored over the life of the project, with baseline values and targets initially set. Performance Indicator Reference Sheets (PIRS) were provided clearly defining each indicator (48 in number), when and how the data would be collected, and by whom. A Performance Management Task Schedule provided further details for the timing of key program activities over the life of the project, with benchmarks. 2.0 Overall Program Design and Targeting Achieving the specified targets and indicator objectives set for a MYAP program does not necessarily mean that program goals of ultimately reducing food insecurity or reducing unacceptably high malnutrition rates among children, etc. will be achieved and sustained into the future. How MCHN and livelihood activities are structured to reach beneficiaries are as important as the message itself, or the services provided - perhaps more important. People purchase what they place value in, and less value on 29 ACI/VOCA, Multi-Year Assistance Program for Haiti, Performance Management Plan, FY 2010, December 2009. Goal: Reduce vulnerability to food insecurity in the Southeast Department of Haiti. Objective 2: Protect vulnerable populations against immediate food insecurity and develop capacity to address long-term nutrition and health needs Objective 1: Increase resiliency against future food insecurity through the protection and enhancement of livelihoods and the development of community capacities Objective 3: Improve the ability of communities to identify and successfully respond to vulnerabilities and impending shocks IR 1.1 Enhanced agricultural productivity, environmental management and market linkages IR 1.2 Improved managerial and financial capacity of community based organizations and local groups IR 1.3 Increased access to financial resources IR 1.4 Increased diversification of livelihood strategies IR 2.1 Improved maternal and child health IR 2.2 Increased access to nutritious foods IR 2.3 Improved household sanitation and access to water IR 3.1 Improved capacity of local organizations and groups to identify and respond to vulnerabilities and potential shocks 37 what is given to them for free.30 Haitians everywhere clearly place a high value in the education of their children – girls and boys – as can be seen in the sacrifices made to pay school fees & uniform costs to send their children to private schools (since there are not enough public schools).31 Between 77% - 93% of MYAP beneficiary households reported that they had school age children and an average of 73% of these are currently paying for these children to go to school! Increasing agricultural production or improving livelihoods through increased incomes does not in itself lead to reduction in malnutrition of the most vulnerable within these communities: the children under 5 years of age and other vulnerable individuals. Who controls these assets and how they are used can also be factors. If a specific area within a MYAP sphere of implementation does not have a significant problem with malnutrition, then should such areas be targeted for FFP activities, no matter how „important‟ the development or livelihood activities are in themselves, or their potential to raise incomes for benefiting farmers? These may be areas or themes that should be taken up by other development funding sources, but perhaps not P.L 480 Title II FFP funds linked to food assistance. Yet, the Haiti MYAPs, as well illustrated through the ACDI/VOCA program in the Southeast Department, are proving some clear lessons about the necessary linkage between working with less vulnerable farmers – often model farmers, or organized groups – to locally develop the cultivars, techniques, local resources to reach out to help the more vulnerable within their communities. What will be important to assess in the remaining years of these MYAPs will be the extent to which MYAP resources become more greatly focused towards reaching out to the truly vulnerable, having largely established the local/regional foundation to do so. Over the past 15 – 20 years, USAID direct funding for agricultural development activities has steadily declined – to the point that the only real source of development funding within many country programs has been through resources made available in FFP DAPs and now MYAPS. This has perhaps led to pressure on FFP to broaden its operating scopes for activities funded through FFP Title II support. The current FFP strategic framework is so broad that almost any kind of development activity, in almost any location, can be justified – and PVOs have been quick to broaden the scopes of their activities as a result. The focus on the most vulnerable regions and households of a region, the focus on reducing malnutrition through integrated health and targeted livelihood activities upon the same beneficiary groups has broadened to become nothing less than general rural development programs. USAID/Haiti and FFP accepted Haiti MYAP proposals with these broad program objectives, and the Cooperating Sponsors are well into implementing different kinds of activities with sometimes quite different approaches. Impact has become more diffuse and generalized – making it extremely difficult to actually show any real impact is taking place on the truly vulnerable, or in reducing the base malnutrition rates of specific localities within section communales of Haiti. Targeting must be consistently intentional, addressing the basic existing social structures of each community – which is the household unit, but also the mothers and their young children within these households. However, not all households or all people working within the „Communitees de Base‟ receiving support will be able to become entrepreneurial leaders in their communities. Yet helping in the 30 Consider the explosion of cell phones among even the most poor throughout rural Haiti, and their creative means of recharging their batteries. The MTE team observed one local entrepreneur who had set up a small solar unit on the roof of his house and charged 10 gourdes (about 25 cents) to community members to recharge their batteries. 31 The issue here is whether they are getting value for their outlay of resources. 38 identification, establishment, and market linkages of such entrepreneurs is essential if most of the MYAP livelihood activities are to become sustainable or are to reach those who are vulnerable within these communities. Greater attention needs to be given to helping such entrepreneurship develop within each targeted community – and care must be taken that the MYAPs welfare approaches do not undercut the ability of these entrepreneurs to find clients for the services or commodities they must sell. Targeting should therefore focus on both the mother clubs (and members of the households of these mothers) as well as some key entrepreneurs within the community or region who will contribute to the overall food security through improved seed multiplication or other MYAP introduced activities (as they have more land and resources to do so, and are most likely to quickly benefit from improved seed, training in improved practices, etc.). Both CRS and World Vision Haiti have been working within their respective regions for many years, in some cases over 30 years with CRS, and for World Vision Haiti approaching 15 years for many of their ADPs. Among the Haiti MYAP Collaborating Sponsors, only CRS addressed the issue of phase-out or „graduation‟ of program activity support in their MYAP proposal, and even proposed some „triggers‟ that would indicate when such action should take place.32 Only one of these appears to the MTE team to be realistic for MYAP programs, and a recommendation is made in support of it below. 2.1 Strategies for Common Approaches, Program Integration, and Impact Diffusion During the first two years of the MYAPs, efforts were made by the Cooperating Sponsors within each of the major program components (MCHN, Agriculture, and EWS) to have national and regional leaders meet quarterly to discuss progress and share about their approaches. This was important as it helped in the development of common approaches on similar areas of intervention. Events following the January 2010 earthquake temporarily halted this exchange, but the MTE team observed this taking place again during April with a meeting on the EWS; plans are in place to restart dialogue on MCHN and Agriculture/Livelihood activities as well. The MTE reports will also stimulate additional discussions in these areas; particularly with regard to the need for improve integration and beneficiary targeting. One of the key common approaches within the MYAPs has been with the „Preventive Malnutrition under Twos‟ (PM2A) approach within the MCHN components. Another area of change within the MYAPs has been an increased realization of the limitations of some of the data being gathered at rally posts (e.g. PFA + PTFA) for determining malnutrition rates within an area, and a move, encouraged by a Haitian Ministry of Health protocol towards MUAC measurements and better tracking of acute and severe malnutrition indicators. For local health service community outreach efforts, the common use of rally posts, Col Cols, and Mothers Clubs indicates that efforts are being made to learn from each other (even though specific CS may use different terms for these, or slightly different approaches ). The creative use of mobile clinics and „fixed points‟ are creative new measures being taken to help bridge the gap between rally posts and health clinics or dispensaries. Within the agricultural/livelihood component of each MYAP, use of model farmers, established CBOs, and newly refurbished small irrigation perimeters as a foundation to build assets for assisting the more 32 The MTE team has reviewed each of these options, and discussed these in the CRS MYAP report #2 of this evaluation. 39 vulnerable households within communities has also been excellent. Multi-purpose and fruit tree nurseries to support local agro-forestry efforts are also to be found throughout all three MYAPs. Integration of program activities (MCHN + Agriculture/Livelihood) for specific vulnerable or most vulnerable households has continued to present a challenge to all three MYAPs. As structured, World Vision Haiti and CRS operate within the MCHN component more as large relief and social welfare distribution agencies than as organizations seeking to help Haitian households and communities to develop the skills and whereabouts to sustainably support and build their own futures.33 Though less pronounced, this tendency spills over to agricultural/livelihood efforts as well – as in the first impulse to „give everything‟ away, or to have people organize „for social service to their communities‟. When asked by the MTE team about this impulse, both WVH and CRS quickly defend their intention to move towards a more „entrepreneurial approach‟ „once people have been trained‟ to do something. But if initiatives do not start on this basis and with this as a clear, verbalized end result, programs rarely ever actually arrive at doing so „before the end of the project‟. As basically humanitarian and initially relief agencies, both have had to struggle to check their „welfare‟ tendencies and give unambiguous support for private sector orientations. All three MYAPs have during the past year initiated efforts to provide more opportunities for beneficiary households to have greater access to a „package of MCHN integrated activities‟. ACDI/VOCA has perhaps had a more intentional and focused private sector approach to its development activities, though even here MCHN and livelihood activities have not yet been sufficiently integrated. To the extent possible, MCHN beneficiaries are targeted for livelihood interventions, and broadly, livelihood interventions are based on the primary livelihoods of the zones worked in. 2.1.1 Graduation Strategies In their MYAP proposal, CRS offered 5 potential criteria for considering what program achievements might justify the „graduation‟ of Title II MYAP activities from a given community. The MTE team has considered these, and offered our own observations at this mid-point of the MYAP program in Haiti. The first criterion offered by CRS was: (1) Average global malnutrition (age for weight) rates of children 6-59 months = < 10% - at the locality geographical level. The MTE team would suggest that, of the five criteria offered (cf. MTE CRS Report 3, p.10), this is the only one that is really important and achievable within the life of any specific MYAP. We would suggest that it could possibly be considered as the trigger for possible graduation of a FFP program. We do not say that other development activities are not worthwhile – they are – but if food security for vulnerable populations is the core issue as stated in the program goal, and resources are actually limited, then this is the criteria we should be concerned about achieving in a sustainable way for the targeted population in the „tightly targeted geographic area‟. As rally posts are located within localities – where such measures are 33 For example, MYAP services provided at Rally Posts are completely free – as most of the inputs such as vaccines, vitamin A, cold chain support, training and counseling are free. When food supplements are given to pregnant women, lactating women, to mothers for children under 5 years of age, for malnourished children, this is also for the most part free. At mobile clinics or health centers, attendees in some locations pay 10-15 gourdes ($2-$3) to the service providers to purchase chlorine, some de￾worming supplies, or perhaps 10% of this for the expenses of the local health committee. 40 most extensively taken – we also think that the locality should be the unit of measure for „graduating‟ from. It would not be meaningful to aggregate all the rally posts data for a section communale or, even worse, a commune – to make such a decision. Indeed, we would go further and state that the target should probably be closer to < 4-6%, and not < 10%. If, after working within a defined locality area for several years34 – with rally posts, Col Vols, Mothers Clubs, etc. – a MYAP program is not able to reduce malnutrition to these levels, then there may be a problem with the approach. Any additional trigger categories may be considered examples of mission creep – worthy in themselves, and probably legitimate topics for other USAID project ventures, but perhaps not for FFP. FFP may wish to reconsider their goals and key priorities for Title II funding which currently include the following very broad FFP strategic framework: (1) Human capabilities protected and enhanced; (2) Livelihood capacities protected and enhanced; (3) Community resiliency protected and enhanced; (4) Community capacity to influence factors (decisions) that affect food security increased; 35 ACDI/VOCA, in considering the above suggestion, noted that, as far as they were aware, the only internationally recognized indicator rates for acute malnutrition are 10% (state of alert) and 15% (state of emergency). The MTE team has suggested that the MUAC measurement, which admittedly is taken as a screening measure for suspicion of malnutrition – to be later confirmed by trained health personnel – might be a useful tool for at least estimating the possible incidence of severe malnutrition. And if compared along with the weight for age measure (PFA+PTFA), this could be useful if NOT aggregated beyond the specific locality level (with rally point data). We understand that the age for weight measure does not tell us what kind of malnutrition we are recording (acute, chronic, or both). 2.2 MYAP Coordination & Technical Leadership The MTE team reviewed the MYAP structures (organograms) of each CS (discussed in greater detail within the specific MYAP reports). Each reflected the different strategic approaches taken by the organization and appeared to function fairly well. CRS and ACDI/VOCA, for example, will not begin field implementation without the direct involvement and reinforcement of GoH regional and local personnel involvement, while World Vision Haiti itself will hire medical staff and operate medical facilities themselves – though with GoH approval and encouragement. All three MYAPs are staffed by very experienced professional staff, whether as doctors, nurses, community health workers, or local agronomists. We were very impressed by CRS and ACDI/VOCA use of local neighborhood volunteers (Col Vols) who represent some of the more sustainable aspects of the entire program effort. Being a smaller organization, ACDI/VOCA seemed the most flexible of the three MYAPs, with greater delegation of authority to field agents, while World Vision Haiti seemed the most bureaucratic and slow￾moving. The MTE team experienced the greatest difficulty in obtaining requested information from WVH – in some cases never receiving this – while both CRS and ACDI/VOCA were more willing or able to quickly move field personnel to gather not-immediately available field information. World Vision 34 In some cases, some CS have worked in the same areas for 12 years now - after 2 DAPS and this MYAP. 35 Food for Peace, Country/Cooperating Sponsor FY 08 – FY XX Multi-Year Assistance Program (MYAP) Proposal, Proposal Application Format, August 1, 2007, page 4. 41 Haiti field personnel appeared more „fearful‟ of sharing information they thought their superiors might not have „given clearance‟ for the MTE team to have. 2.3 MYAP M&E System and Targeting 2.3.1 MYAP PMPs & IPTT Indicators ACDI/VOCA initiated this MYAP with a well designed Performance Management Plan (PMP), including all the important components for a PMP. With this completed, all subsequent quarterly reports submitted to USAID and FFP include a narrative section and IPTT table showing progress achieved towards established targets. ACDI/VOCA has also been attentive to monitoring the usefulness of this management tool, making periodic recommendations for adding, modifying, or dropping indicators that were shown not to be appropriate or needed clarification. As such, program management appears to have been using the PMP as a management tool. World Vision and CRS PMPs were very similar in design with content varying depending on the kind of indicators selected. All three MYAP have IPTTs that are correctly designed, following the strategic framework for each MYAP. Both output and F indicators are present, along with both outcome and impact indicators at the different levels. Indicators are correctly described in the Performance Indicator Reference Sheets (PIRS). Task schedules were prepared over the life of the program to provide benchmarks and timing for key program management tasks, including this mid-term. The quality of the PMPs for all three MYAPs can probably best be attributed to the FANTA training given at the beginning of the programs. As noted below, however, there are simply too many indicators for each MYAP: 58 for WVH, 61 for CRS, and 52 for ACDI/VOCA. All three MYAPs have almost the same set of common indicators below, but these indicators are sometimes worded differently in the different IPTTs of each organization. USAID might have provided guidance in the RFP to use a set of common indicators – with the same wording across all three MYAPs, which would have again facilitated cross-comparisons. Perhaps more bothersome are those USAID and FFP F indicators that are almost the same, but different enough to require separate indicators to be monitored. The data acquired for each are almost identical. Yet M&E MYAP coordinators told the MTE team that USAID and FFP required them to include these, since these were the wordings used in their separate reporting formats. An example: # of direct beneficiaries reached by the NRM program (a FFP indicator) vs. # of people receiving USG supported training in NRM and/or biodiversity conservation (a USAID F indicator) When such cases are multiplied, they add a great deal to the cost in time and resources to report upon, aggregate, and track each quarter. One or other should be selected, and the other dropped. Within a single MYAP, acquisition of data is not standardized across the program. Sub-contractors of WVH - MARCH, HAS, and SAVE - each possessed their own different sets of forms and tables that were completed at local and regional levels, to be sent to their national level headquarters (SAVE in PAP, for example), for translation into the formats requested by WVH for their common MYAP. It would have been more efficient had all partners within WVH‟s MYAP, for example, adopted a common set of 42 documents for their MYAP program, and entered data into the now fairly efficient electronic reporting system developed for WVH‟s MIS. Common FFP indicators: 1. % of children 12-23 months fully immunized by age 12 months 2. % of children 12-23 months vaccinated against measles 3. % of children 6-59 months underweight (PFA + PTFA) 4. % of beneficiary children 6-59 months with weight for age below -2 Z score (wasted) 5. % of beneficiary children 6-59 months with height for age below -2 Z–score (stunted) 6. % of infants & young child feeding (exclusive breastfeeding) 7. % of caregivers of children 0-59 months reporting washing with soap at least 2 of the appropriate times of the day preceding the interview 8. Average number of months of adequate household food provisioning (MAHFP)for beneficiary HHs 9. Household dietary diversity score (HDDS) 10. % of mothers of children < 24 months whose last delivery was attended by a skilled health personnel 11. % of mothers of children < 24 months who had a least 3 prenatal care visits by a trained provider during their last delivery 12. % of farmers using at least 3 sustainable agricultural practices MYAP programs also possess outcome and impact data that are not reported upon in the IPTT. Because of all the required indicators from USAID and FFP, PVOs have not been eager to add anything to an already excessive list of indicators. Yet USAID and FFP are themselves probably more interested in outcomes and impacts than simply the endless output data that the “F” indicators usually represent. Indeed, in their aggregated form, one must wonder of what possible interpretive value such aggregated data could be to USAID or FFP when in receipt of such data from country programs. The USAID mission encouraged use of custom outcome and impact indicators should be given higher priority within the IPTTs, and PVOs encouraged identifying those data that actually mean something with respect to the impact of their programs on program beneficiaries and the goals of the program. The baseline – midterm – and final evaluation „common‟ indicators are interesting in a regional sense – but not well enough focused for this purpose. For example, „the % of underweight children (0-59 months) with weight for age below -2 Z scores‟; ACDI/VOCA started with a baseline value of 23% with an EOP target of 17.5%; WVH‟s IPPT baseline was 23.5% with an EOP target of 21%;36 CRS‟s IPTT baseline was a somewhat more reasonable 14.6% with an EOP target of 11%. However, impact for such an indicator, is best understood at a localized level (not regionally aggregated). If after 10 or 15 years in a locality (as is the case with some WVH and CRS sites), one is still talking about 21% or 11% malnutrition respectively, then one must question the effectiveness of these programs. Of more importance would be the % of localities (rally post rates of malnutrition by month) whose malnutrition rates had dropped to below 6% for a period of six months or more – from whatever it 36 WVH notes that ‘looking at the 2005/2006 DHS, underweight is roughly 30% in Artibonite and Centre. For WVH, bringing a population-based indicator for underweight from 23.5% to 21% could be substantial given the context’ (personal communication). Yet the accuracy of such data is suspect – many of these children may not be malnourished at all, or the data are misleading. As pointed out by other MYAP CS partners to the MTE, much of such high rates are due to the effects of chronic malnutrition which cannot change quickly, added to the fact that the age of the children measured is often inaccurate, leading to such data. Use of MUAC data, going forward, should provide a balance to these weight/age data sets. 43 was at the beginning of the MYAP. ACDI/VOCA itself has about 300 rally posts, with numbers increasing monthly, CRS over 1,000 rally posts, and World Vision Haiti 1,104 rally posts to which thousands of mothers congregate, and around which the Mothers Clubs form. This would be an actionable indicator, one that management could use to start a phase out of activities – and transfer resources to places with higher rates of malnutrition. Just because a specific outcome or impact indicator was not measured at the time of the MYAPs base-line survey does not mean that new indicators of these kind cannot be added later in the life of the project. Not everything needs be measured at „the population level‟. New indicators can be added if the project is able to go back in their own data sets to obtain data for earlier years, as suggested in examples below. Outcomes and impacts on the beneficiaries within the program areas is also important to understand. Other outcome indicators that might be considered would include: (1) the number of operational rally posts; this is an extremely important piece of information, and their geographic location a clear indicator of the reach of program efforts throughout the target regions. (2) The total number of mothers gathering at these monthly rally posts is also important in assessing potential impact of the services and messages being given out. And if Mothers Clubs were to be given their appropriate roles within communities, (3) the evolving numbers of these Mothers Clubs and the geographic location of their monthly meeting place would also be an important indicator of program outreach. The key impact indicator would then be (4) the % of children with PTA or PFTA rates of less than 10%, tracked by averages recorded at the thousands of rally posts. In all three MYAP programs, an important outcome indicator would be (5) the % of women (and their malnourished children) who are instructed at monthly rally posts to go to the regional health clinic for care, and the number who actually do go. The MTE team found that less than 50% of mothers were actually going to the clinic after the Rally Point instructions. An important commodity related outcome indicator would be to track (6) the % of households designated to receive nutritional supplements each month that actually do NOT receive them that month. 2.3.2 Reporting & Communication The last week of every month is spent by program leaders at different levels summarizing the previous three weeks of implementation, and planning for the coming month. In the case of World Vision Haiti within their MCHN activities, the first week of every month focuses on the rally posts throughout their regions, where mothers bring their children for various health care activities, growth monitoring, and receipt of nutritional supplements for targeted needed malnourished children. The second and third weeks of each month focus on the training provided by local health promoters to the various mothers club sub￾groups. Examples of some of the summary data acquisition forms are provided in the MTE PVO reports. These data, and the information obtained through the agricultural programs, flow within each region to the Regional Coordinators for either MCHN or Agriculture, who in their turn eventually provide three months of such data in their quarterly reports to World Vision Haiti‟s central office in Port-au-Prince. Both indicator tables and data and the narrative reports have recently begun to flow electronically into an information management system (MIS) that permits tracking of progress towards intended targets for all targeted indicators. In our meeting with USAID Haiti, they noted that they had not received the October – December, 2009 quarterly reports from any of the three MYAPs – but understood the reason for this as the January 12 44 earthquake and the total upheaval in schedules took place in the midst of trying to quickly retool for relief and emergency support.37 USAID appeared generally satisfied with performance of all three MYAPs, recognizing that ACDI/VOCA started up later, and did not have the kind of pre-existing infrastructure and personnel base that CRS and WVH possessed. Common Formatting: It might have been helpful if USAID and FFP had asked all three organizations to submit their strategic descriptions using USAID/Haiti‟s framework, as given out in the RFPs. Wording in the RFPs might have requested a common integrated approach, with a set of common indicators of the same wording. Since all three MYAPs included major MCHN, agriculture/livelihood, and EWS components (CRS added a fourth - education), it would have been useful if a common reporting format could also have been developed as well. In reviewing reports of all three MYAPs, the MTE team found that the World Vision format for annual and quarterly reports to be the easiest to follow from one period to the next – with each Table 1 referring to the same set of internal indicators monitored from quarter to quarter (example show below)38 . If every MYAP reporting these same data were reporting in this format, program managers and others reviewing the information would be able to synthesize and compare cross programs much easier. This would have facilitated in understanding the reporting, when reviewing all three together as an overall MYAP program. It should be noted that data reported on such tables in these quarterly reports is in addition to that tracked in the IPTTs. One will not find an IPPT indicator on the accumulating # of children 6-24 months old who have received food rations through the MYAP. IPTT indicators aggregate such data in different ways. For example, the number above is included in the IPTT indicator # of children reached by USG-supported feeding programs. ACDI/VOCA and CRS present somewhat comparable information in different formats in tables which summarize activities and achievements of each quarter. Table 1: World Vision Haiti MYAP: MCHN and FFA Beneficiaries Receiving Food Rations Beneficiaries Receiving Food Rations FY 10 Target Quarter 2 Target Quarter 2 Achievement % Achievement Children 6-24 months old 19,500 19,500 18,299 93.8 Malnourished children under 5 years of age 3,160 3,160 1,999 63.3 Pregnant and lactating women 10,125 10,125 10,058 99.3 People Living with HIV/AIDS and OVC 450 450 561 124.7 Direct Food Beneficiaries 33,235 33,235 30,917 93.0 People Benefiting from Food (Direct and Indirect) 158,533 158,533 147,004 92.7 Beneficiaries reached after the earthquake (GFD) - - 279,375 - 37 By the time the MTE review reports had been completed at end of July, the MYAPs had been able to catch up on their quarterly reports, with even the FY 2010 2nd quarterly report (January-March,2010) completed and submitted. 38 WVH MYAP January – March 2010 Quarterly Report, April 2010, p.8. 45 2.3.3 Targeting: Targeting of beneficiaries could be better done. This is true at several levels. At the most basic level, we have Haitian rural household with anywhere from 4-5 members to over 10, most of these with children of various ages. These households are found within specific „localities‟ or small communities at which level all three MYAPS have placed their „rally posts”. In principal, any household with children under 5 years of age can come to these locations to receive some health nutritional training, vaccinations, and growth monitoring. All cases of malnutrition (PFA or PTFA) are referred to the next level – the health clinic and/or „fixed point‟ (ACDI/VOCA) or „mobile clinic‟ for receipt of special services (supplemental food) and added health care. Not all mothers bring their children consistently to the rally posts, nor do all mothers with malnourished children (less than 50% in some cases visited by MTE team) take the next step and go for help at the health center for the food supplements or added care.39 Logistic problems for either the household itself or delivery by the NGO can result in non-delivery/receipt of food supplements by the registered households with malnourished children. ACDI/VOCA, through their BND sub￾contractor in May 2010 planned 1,551 food kits for 24-59 months children (recuperative) and 1,494 were distributed – 96% adhered to the program and only 4% of the mothers did not receive the food for their children. If such levels are maintained, this is very good indeed. MYAPs are not designed to necessarily reach 100% of the malnourished children within a specific locality either – there are pre-set targets for each rally point and sub-region – which added together add up to the targets set by the project in the IPTT, and which also govern the amount of commodities that are planned for delivery within a specific region. The MTE team found that in some MYAP PVO regions, the actual targets might only be 50% of potential need, with the hope that over time the project could „scale-up‟ to meet the entire need. But because of limited resources (commodities and personnel), reaching much less than 100% within a specific locality was the reality. Added together, these different issues mean that it is quite difficult to actually impact the base malnutrition rate of that locality. For the targeting of MYAP activities themselves, in World Vision Haiti MCHN focused areas, agricultural program activities are sometimes included, and in other areas MCHN stood alone without any agricultural program activities. In either case, the actual targeted beneficiaries might or might not be the same households. ACDI/VOCA communes benefiting from MCHN and agriculture/livelihood activities were not necessarily targeting the same vulnerable beneficiaries for these two components. ACDI/VOCA was the only MYAP that had an „agriculture/livelihood only‟ group of beneficiaries (La Vallée and Bainet communes). As explained more fully elsewhere, this was part of their strategy for developing agricultural/ livelihood activities for the other communes and more vulnerable populations of the MYAP through less vulnerable households – while increasing agricultural production and food availability within the entire region.. MCHN activities may eventually be planned in some of the remote areas within the above two communes. CRS has one commune (Aquin) that receives MCHN activities only, while all other communes benefit from both MCHN and agriculture/livelihood activities. And like WVH, specific households within the latter communes do not necessarily have access to an integrated package of MYAP activities. 39 Rations are set at a standard 5 persons/household level. 46 Not all members of a household benefit equally when household income is increased. When women are targeted to receive animals for example, like goats, pigs, or chickens, they are more likely to provide greater care of the animals for the household, involve their children more closely in supervised management of the „family business‟ than is frequently the case when “the household” is given these animals (which usually means the husband). A woman‟s status within the household and community is also enhanced. Cash from the sale of surplus crops or animals by the husband are more likely to end up purchasing additional animals, or a radio, or beer, or other consumer items; woman who sell their produce or animals are more likely to use the money for paying for school fees/uniforms for their children, provide food for their children, purchase medicines and insurance for health services, etc. 2.3.4 Quantitative Survey: Summary of MTE Results Results of the MTE quantitative survey (cf. Annex 8 for all tables of data), when compared to the 2007 population-based baseline, show that all three MYAPS have shown strong achievements. The data below will be incorporated into the respective MYAP IPTTs. For example, WVH had as their LOA target that 82 % children under 2 years of age in their areas of activity would be fully immunized – the survey of program beneficiaries shows 86%. CRS at 87% is equally strong. Both WVH and CRS started their MYAP in areas were the rates were relatively higher to begin with (61% and 59% respectively). On the other hand, ACDI/VOCA‟s efforts began in remote regions with more underserved regions at 21% with achievements among beneficiaries currently showing 35%. The mid-term average rates of malnutrition recorded among MYAP beneficiary households with children < 5 years of age was surprisingly high: 19% - 18% and 21%. A number of reasons could explain for this. Perhaps the most evident reason is that some of the beneficiaries weighed were not living in areas where the MYAP was engaged in MCHN activities – these were areas where agricultural efforts alone were undertaken. Or another reason is that beneficiaries weighed in our sample were from localities will less MCHN follow-up (i.e. households were located far from where health services could be received – so even though weighed at a rally point and found in need of help, the mother may not have been able to take child to a clinic for help). More detailed analysis of these data would clarify this. 47 Table 2: Summary of MTE Common Indicators for the MYAP Program Activity area Indicator WV CRS ACDI-VOCA Label Child Health Child Nutrition % children 12-23 mo fully immunized 167 (86%) (0.82, 0.90) 96 (87%) (0.82, 0.92) 40(35%) (0.28, 0.44) Freq (%) C.I. % children 12-23 mo vaccinated against measles 173 (87%) 98 (87%) 59 (50%) Freq (%) % children 0-59 mo underweight 84 (19%) 64 (18%) 54 (21%) Freq (%) % children 6-59 mo stunted 72 (16%) 54 (15%) 51 (20%) Freq (%) % children 6-59 mo wasted 29 (6%) 17 (5%) 18 (7%) Freq (%) IYCF indicator 1:: mean (sd) 35.2 (0.2) 237 33.9 (0.3) 144 35.4 (0.3) 141 S. size IYCF indicator 2 Household Hygiene % of caregivers of children 0-59 mo reporting washing hands with soap at least 2 times in the day preceding the interview 205 (74%) 206 (86%) 169(65%) Freq (%) Household Food Security Months of adequate household food provisioning (sd) 4.2 (0.2) 273 4.8 (0.1) 235 5.8 (0.1) 228 Av (sd) S. size Household dietary diversity score 5.4 (0.1) 276 7.0 (0.1) 237 5.6(0.1) 459 Family Planning and Reproductive Health % mothers of children <2 years whose last delivery was attended by a trained professional 74 (27%) 163 (68%) 106 (60%) Freq (%) % mothers of children <2 years who had at least 3 prenatal care visits by a trained provider during their last delivery 88 (32%) (28, 37) 273 39 (16%) (13,21) 233 16 (9%) (6, 15) 182 Freq (%) C.I. S. size Household Agriculture % of farmers using at least 3 sustainable agriculture practices 80 (29%) (25, 44) 276 55 (23%) (19,28) 237 93 (36%) (31, 41) Freq (%) C.I. S. size The MAHFP scores above are an example, in the opinion of the MTE team, of misinformation. Our own interviews with some of the same households that may have responded to this survey, noted that households were much better off for food resources since the beginning of MAYP activities. Their harvests were higher because of the new varieties received; more was sold, permitting purchase of a goat or other animal. High value vegetable gardens were in many cases cultivated year around providing additional food and income. In Haiti, with its two rainy seasons each year, there is almost always something in the ground that a household can find to tide them over a period „of less‟ (plantain, manioc, sweet potato, beans – sale of a goat, etc.). Therefore to get a MAHFP value that is actually lower than that of the baseline survey itself is highly suspect.40 People simply do not tell the truth when asked such questions – having made mental calculations that they would more likely get something more from aid￾agency if they said they had need. Most MYAP beneficiaries possessed animals, and in some case multiple ones. The figure below shows that there was not much difference between households that were receiving an „integrated package‟ of project benefits that included MCHN and agriculture, and those that only received one or the other alone. 40 And it is not because the enumerators were not trained on how to ask these questions. Most had been involved in the baseline two years ago as well. 48 We see considerable diversity in the diets of MYAP beneficiaries as well as shown in the table below, though there is considerable variation among certain categories of food. For instance, while only 30% of WVH respondents noted that they had consumed legumes the previous day, 51% of both ACDI/VOCA and CRS beneficiaries noted that they had. Most of all beneficiaries (around 90%) had consumed cereals the previous day. Indicator PARTNERS AREA WV Beneficiaries (N=276) CRS Beneficiaries (N=238) ACDI VOCA Beneficiaries (N=227) 8.3 – Did one of the household members eat of these foods yesterday? (% yes) Cereals 249(90.9%) 196(82.4%) 197(87.2%) Fat 246(90.1%) 226(95.0%) 211(93.8%) Sugar 140(51.1%) 181(76.1%) 178(78.8%) Coffee/tea 119(43.4%) 154(64.7%) 174(77%) Tubers 89(32.2%) 152(64.1%) 116(51.3%) Legumes 82(29.9%) 122(51.3%) 116(51.3%) Fruits 155(56.6%) 192(80.7%) 104(46.2%) Meat 53(19.4%) 72(30.3%) 38(16.9%) Eggs 30(11%) 40(16.8%) 32(14.2%) Fish 24(8.8%) 79(33.3%) 10(4.4%) Hareng 76(27.7%) 67(28.2%) 63(27.9%) Pea 205(74.8%) 165(60.3%) 190(84.1%) Milk 41(15%) 57(24.1%) 41(18.1%) 0 10 20 30 40 50 60 70 80 90 Pigs Goats Poultry Donkeys Horses Cattle 32 49 62 23 11 32 30 40 88 20 10 34 45 61 74 17 17 41 Health Agriculture Both 49 The MTE asked the 741 MYAP beneficiaries about their experiences with the January 12 earthquake. More than half of all households interviewed, in all three regions, noted that they had been impacted in one way or another. About a third of all households received someone from outside their household into their home for a while following this event. Asked what these major impacts were on their households, respondents spoke of the death to a relative, damaged homes, lost jobs, etc. The responses appear to describe more what happened to their relatives in PAP, and not to the beneficiaries themselves, who lived far from the center of these events. When asked what MYAP beneficiaries regarded as the most important benefits of the program, improved varieties of seed, food (supplemental MYAP food), and the improved health care were the major responses, followed by better diets and education for their children. Responses to the question about what were the most important agricultural techniques learned from the MYAPs were disappointing, and probably reflects well that while MCHN was the major component for all three MYAPs, agriculture was the smaller component, and it was not well integrated for all 0 10 20 30 40 50 60 70 80 90 WV (n=153) CRS (n=112) AV (n=173) 0 10 20 30 40 50 60 Nothing Seeds Organic fertilizer Post-harvest TK Tree planting Soil conservation Density control Mulching Food for kids health care Better diete Education Other WV (n=276) CRS (n=238) AV (n=227) 50 beneficiaries. Since the majority of the sample would have been MCHN households, it would stand to reason that the agricultural benefits would be minimal. The highest response was „no benefits‟, followed by agro-forestry, soil conservation, improved seeds, and animal husbandry (chickens and goats). When beneficiaries were asked where the last child under two years of age had been born (almost 100% of surveyed households had children under 5 years of age), the response was as one would expect: about 75% were delivered at home. However further questioning does show an improvement in the quality of the care received at this delivery with about 60% of all households, for all three MYAPs, having present a skilled health personnel – usually the specially trained matrons of the program. 0 10 20 30 40 50 60 Nothing Seeds Mulching Fertilizers Agrosylviculture Soil conservation Density control Post harvest … Integrated Pest … Micro-irrigation Rotation Tree planting Animal husbandry Don’t know Other WV (n=260) CRS (n=238) AV (n=223) 0 10 20 30 40 50 60 70 80 Health Center House Hospital Other 3.3 74.2 22.2 0.4 6 76.5 16.9 0.4 3.4 74.7 20.3 1.7 WV AV CRS 51 The MYAPs have made a major contribution to the vaccination of children in the regions in which they serve, and the survey bore this out. Families have their vaccination cards, and well over 90% have had all their vaccinations. Indicator Sub-groups WV Beneficiaries (N=276) CRS (N=238) ACDI VOCA (N=227) 13.1 – Is there a vaccination card? 194 (98.0%) 112 (99.1%) 110 (94.0%) 13.2 – Did the child receive 3 doses of DTP? 195 (99.7%) 104 (93.7%) 68( 59.1%) 13.3 - Did the child receive 3 doses of Polio? 194 (98.5%) 104 (93.7%) 68 (58.6%) 13.4 - Did the child receive a dose of BCG? 190 (98.4%) 110 (98.2%) 93 (80.2%) 13.5 – Did the child receive a dose against measles? 173 (87.8%) 98 (87.5%) 59 (50.4%) 13.6 - Did the child receive some vitamin A in the past 6 months? 168 (85.3%) 104 (92.0%) 90 (77.6%) In spite of the presence of the MYAPs however, these Haitian households still face challenges, and many remain vulnerable because of the distance they live from health facilities. Households continued to experience the death of a mother or child, and many have chronically ill individuals living within the households (12% -13% of households). Though many other tables could be discussed here, the others may be reviewed in Annex 8 of this document. We wished to show one final table that we found both surprising and somewhat troubling. The enumerators used on this survey were experienced in weight and height measuring for children. The survey found that among MYAP beneficiaries, the levels of malnutrition rate of the children weighed within these households to be unacceptably high for all three MYAPs: 19%- 21% for low height for age, 16% - 20% for low weight for height, and 6% - 7% for low height for weight. Hopefully by the end of this MYAP results will be much better for households that should have potentially been able to benefit by MYAP services over a period of years. 0 5 10 15 20 25 30 35 40 45 50 6.9 17.9 2.2 39.4 24.1 8 1.5 15.8 8.2 0 33.9 37.2 4.9 0 11.4 9.3 0 47.5 27.5 3 1.3 WV (276) AV(227) CRS (238) 52 3.0 Overall Program Management and Implementation Between the MYAPs themselves, there is a bi-monthly coordination meeting, usually held at the WVH PAP central office, bringing together USAID, FFP, MYAP managers, and commodity managers. Here, major program issues and challenges are shared and action items for the coming weeks discussed. This has permitted effective communications among all parties and certainly helped in making rapid responses possible in recent catastrophic earthquake. 3.1 Program Management World Vision Haiti, particularly on La Gônave and in the High Plateau presents itself as a very efficient, well-run operation, closely integrating multiple activities, where possible, with their long-term ADP programs. Regional offices are well-staffed by experienced managers. However, management oversight, collaboration, and the results for all three World Vision Haiti sub-contracting partners in the Lower Central Plateau and Artibonite (MARCH, HAS, and Save the Children) are more ambiguous and problematic. From an agricultural perspective, the latter region, when compared to other WVH regions, is highly favored in terms of rainfall and extensive irrigation systems permitting 12 months of cropping each year in most locations. Food should not be scarce. Efforts to improve performance with MARCH have collapsed – and WVH appears to have moved slowly in resolving this situation. Steps are only now beginning for WVH to take control of the situation. Albert Schweitzer Hospital (HAS) activities focus around health care centers with a clearly recuperative approach to health care. The Hospital and network of dispensaries do not appear to currently have a well structured community outreach program, nor was the program designed to include livelihood development activities. SAVE the Children here appear resource poor and facing difficulties in implementation as well, also with only a MCHN focus, with WVH providing the distribution of food supplements. WVH itself works in agriculture support to 3 communes of the Lower Central Plateau, with beneficiaries that are linked, only accidentally and not intentionally, to MARCH MCHN activities in the 0 5 10 15 20 25 HAZ WAZ WHZ 19 16 6 18 15 5 21 20 7 WV CRS AV 53 same area. Different program activities were undertaken by different sets of individuals – with some areas receiving both MCHN and agriculture, and other areas only receiving agricultural activities. CRS programs based out of Les Cayes, also appeared well-managed, with excellent technical leadership and strong sub-contracting partners. Programs are designed to have a community wide focus in geographic regions where most of the households were classified as „vulnerable‟. Though integration of the different components of the MYAP (MCHN, agriculture, education) was a principal stated objective of the project design document, actual field implementation of different activities are mostly compartmentalized with different target groups. Integration would best be defined as integrating the different MYAP activities within the same geographic areas, though even this was not the case in several communes (where only MCHN activities were undertaken). Central to CRS‟s MCHN approach are its rally posts that are intended to funnel program beneficiaries towards GoH and private health clinics or dispensaries where the actual services are to be received. ACDI/VOCA has put together a strong and creative team in both MCHN and agriculture, based out of their program headquarters in Cyvadier, near Jacmel. ACDI/VOCA has not pursued an integrated MCHN and livelihoods approach – with the two components compartmentalized with different sets of personnel, and generally different target groups. The team estimates about a 20% overlap between their MCHN and agricultural interventions.41 Two communes (La Vallée and Bainet), both favored with above average rainfall, soils, climate, and agricultural diversity, are targeted because of their high agricultural potential. Malnutrition was not considered by one local authority in La Vallée to be a major issue for his area.42 In the other communes of intervention – with significant malnutrition issues – agricultural and MCHN activities are largely focused towards different clients as well. These communes also have highly diverse agro-ecological areas, with both high and low rainfall areas. ACDI/VOCA is in the process of restarting their MCHN component because of the failure of their local sub-contractor partner (MSH) to deliver community based services in a logistically difficult area; this provides an opportunity to also refocus the strategic approach currently being envisioned. Once aware that field performance of this partner was not acceptable, ACDI/VOCA moved quickly and skillfully to correct the situation in a difficult socio-political environment – and are well on track to move forward successfully. 3.2 Program Priority Activities and Implementation Process Program Integration: On the High Plateau around Hinche, WVH MCHN activities pivoted in most areas around long-established ADPs – with the organized committees of specific ADPs managing centrally created nurseries, for example, initially established under the former DAP, as well as a network of community nurseries. Of the three MYAPs, WVH on the High Plateau has made the most progress in initiating a full package of MCHN and agricultural services in households with malnourished children. Yet beneficiaries of agricultural/livelihood activities of the program might or might not be linked to 41 ACDI/VOCA FY 2009 Annual Results report, November4, 2009, p. 2. 42 Both of these communes have extreme variations in agro-ecological zones. The La Vallée towns visited (or passed through) by the MTE team appeared less vulnerable, but the MYAP is also working in numerous isolated areas that are more vulnerable. Many Bainet communities are quite isolated, and according to the ACDI/VOCA team, also vulnerable because of their isolation, steeply cultivated slopes that experience serious erosion through storms, with less access to health care and other services. Wide-scale crop failure was noted in a recent field trip though both La Vallée and Bainet during the last agricultural season. 54 MCHN activities – with their targeted focus on specific children at different developmental phases. Within CRS regions, building on two earlier DAPs, the MYAP‟s approach to farmers and Mothers Clubs maintained approaches initiated in the DAPs, using Col Vols and groups of farmers or „old Mothers Clubs‟ formed in earlier times. World Vision Haiti included efforts to provide small animals to vulnerable household, but this was not done in CRS or ACDI/VOCA areas - yet. ACDI/VOCA has initiated the support of value chains for targeted commodities (black beans, peppers, vegetables, mandarins), but WVH and CRS have been less intentional about this, though they too have initiated promising activities (peppers, Francisque mangoes, high value vegetables). The approach to communicating health/nutrition messages among the same targeted beneficiary groups also differed, the approach to Mothers Clubs varied, as did the location of the services provided. Educational Training for Behavioral Change: Mothers interviewed by the MTE team themselves forcefully stated that it was the educational training received that was the most important asset they gained from the MYAP – and not the food supplements, which was clearly secondary. In any case, these food supplements are usually not sufficient to meet the actual supplemental needs of all the household‟s children (who share what is acquired by the household). Training provided at the household level can lead, and has led, to behavioral changes in how small children are cared for, how household food is prepared in the most nutritious manner, household hygiene, etc – all topics currently being communicated at rally posts and in selected Mothers Clubs. Mothers need the continued encouragement and input of a close circle of fellow mothers to be reminded of and to remain true to knowledge gained, and this can only be achieved by a small group of mothers who live near each other, trust each other, and begin to increasingly care for each other as a support group. This is at the base of the preventive approach to health delivery – helping to reduce the situations that require mothers to have to take their children to distant health clinics or dispensaries in the first place. Communicating to households through self-selecting Mothers Clubs, which may include mothers of children who will drop into and out of malnutrition over time, within specific localities to receive such training support would appear to potentially hold the greatest long term promise to see behavioral change introduced into households throughout the MYAP regions. What may begin as a Mothers Club initially focused on child care and nutrition can evolve into lifelong activities for economic enhancement through targeted livelihood activities. CRS is the only MYAP working within their program areas with elementary age school children through a feeding program which also targets improving the instruction given as well as helping parents to become more involved in their children‟s education. The MTE team did visit one of the schools assisted, but did not meet any parents with whom we might discuss the benefits received from this program. However the rate of regular attendance at the various schools is reported to have risen because of the meals provided, which is in itself a changed behavior which will certainly lead to potentially better educated children. MCHN: The MCHN component of the MYAPs was given the greatest focus for both World Vision Haiti and CRS, with a less well focused and supporting role given to the agriculture and livelihood development component. CRS also added some educational activities in their areas. In the case of 55 ACDI/VOCA, its own focus was initially on the agricultural component, with what turned out to be a weak and inefficient sub-contractor (MSH) focused on MCHN. As a result, by the mid-term evaluation, the agriculture represented the stronger focus, with greatest results. ACDI/VOCA is in the process of itself integrating MCHN activities into its own field programs and has a strong base for expansion of its Mothers Club program. World Vision Haiti distribution of rally posts and what they call „Mothers Clubs‟ on the High Plateau (Hinche, Thomas-Sique, Colladere areas visited) appeared good, with some 30,000 children and their mothers/households impacted here alone. MCHN activities in the Lower Plateau and Artibonite regions, implemented through sub-grantees (MARCH, Save the Children, HAS) appeared much more problematic, particularly in the case of MARCH, which has not been very operational since the beginning of this year. CRS, within its MCHN areas of focus, are indeed reaching large numbers of mothers with their children, at their rally posts, while ACDI/VOCA also has achieve a great deal of penetration into their targeted communities through their own rally posts. Mothers Clubs and Children < 5 Years of Age: World Vision Haiti‟s MYAP, within their MCHN program activities, primarily focuses on children at four infant developmental stages (fetus/pregnant mothers, babies 0-6 months/lactating mothers, infants 6-24 months, and malnourished children 24 months to 5 years of age).43 In order to reach these children, and to focus their behavioral change messages, mothers are organized into sub-groups representing each stage, called by the project “Mothers Clubs”. As a result, World Vision Haiti the numbers of Mothers Clubs reported by World Vision Haiti is artificially high (2,000+). These groupings or names do not have any particular meaning to the mothers themselves, and essentially represent a strategy to organize mothers to receive specific WVH services intended for each sub-group and facilitating monitoring of the program. A child‟s stage clearly changes with the passage of time. Mothers move from one stage to the next as a cohort, and eventually out of the „mothers club‟ entirely as the child exceeds 5 years of age. She may re-enter the process if she is again pregnant, or could be a potential member of more than one mothers club – though in practice she usually will only attend training events for the youngest of their children. If she has already gone through the entire process once, she may not even come to the mothers club training events, but will attend the rally posts for the growth monitoring which could lead to food supplements if her > 5 child is malnourished. The promise of food supplements for some of the mothers‟ club stages is an incentive that brings the women to the training events (which are only about 15 minutes in length). CRS‟s MYAP Mothers Clubs appears not to be a major focus for their efforts – with a life-of-project objective of establishing only 110 Mothers Clubs – in a region where one would expect a couple thousand.44 Women in the existing Mothers Clubs were “invited” by Col Vol community health workers 43 Such a focus is understandable, given World Vision Haiti’s worldwide program strategy, through child sponsorship donations, that funds 15 year long-term community development efforts through their Area Development Programs (ADP’s), of which there are currently 21 in Haiti, with some 52,000 sponsored children. No ADP has yet become sustainable on its own, though the two earliest ones have begun to try to do so – with some continued WVH support. The MYAP is involved in four of WVH’s five regions of effort. 44 This was surprising, as Mothers Clubs appeared to play a much larger role in the earlier CRS DAPS. The loss of interest of mothers may partly be explained because they were hearing the same thing over and over again, and already ‘knew’ the 56 (and CRS) to form into groups of 20 mothers to receive specialized training, and certificates – leading to a system where each Col Vol might have one such Mothers Club, initially established when women coming to a rally post were either pregnant or lactating mothers. The 20 women could come from anyplace within the locality – hours walking distance from each other – with only their „infant developmental stage‟ being the common link between them. These are not „natural groups‟ of mothers who would necessarily want to associate with each other for other reasons (friendships, group savings and loan, economic ventures). ACDI/VOCA inherited an approach to Mothers Clubs through MSH, which was not able to fulfill its outreach mandates within the MYAP and was dropped as a sub-contractor recently. However the approach was a healthier one, in the view of the MTE team, in that mothers from specific localities were free to associate together into a Mothers Club, and include mothers of children of different ages. Some mothers might be pregnant (having an older child), or lactating, or with children less than 5 years of age, as well as older children. However, the approach was not to encourage all mothers coming to rally posts to become members of their own Mothers Clubs, but to have each Col Vol Health Community worker have one Mothers Club. A locality might have 50 -60 mothers bringing their children to the rally posts, but only 15-20 were organized into this locality‟s Mothers Club. In some cases, these Col Vols, themselves members of these communities, even became the Leader of the Mothers Club itself, moving their club into a group savings and loan system as well. Community Based Collaborating Volunteers (Col Vols): Programs (CRS, ACDI/VOCA) that begin with Col Vols appear to have succeeded in developing a group of local leaders willing to volunteer their efforts for community recognition and service – with the option of occasionally being recruited part-time to serve as assistants for local GoH health programs. When programs begin by hiring community health workers, these people rarely will serve later as Col Vols. However experience has demonstrated that a Col Vol, later hired on a short term basis, will return to being a Col Vol again. MYAP Col Vols receive special training, a package of work equipment, and a very small stipend. The MTE team observations of some male Col Vols „in action‟ providing training at rally posts visited suggest that these male Col Vols take a fairly aggressive and commanding attitude towards the women listening to them, a relationship that may not be most conducive to learning some of the lessons being taught. Men are more „available‟ to provide the services of Col Vols – and are freer to travel around an area. But women did note to the MTE team that they felt more comfortable having a woman giving a home visit than would be the case for a non-family man entering the homestead. Giving greater attention to Mothers Clubs with identified Leader Mothers who can provide more of this intimate training and home visits would help to overcome this obstacle. Rally Posts: Services given out at „rally posts‟ impact the larger community. For World Vision Haiti, services are provided by locally WVH hired „community health agents‟, supervised by at least one medically trained person or health auxiliary (also WVH staff member). Here, the targeted children > 5 message. Perhaps they no longer had any reason to remain together as a group, having not been initially formed by their own effort. The introduction of some livelihood activities in the MYAP has helped to renew some interest. 57 years of age, and their mothers, get a package of defined services (vaccinations, deworming of children, Vitamin A supplements, growth monitoring). While all children of these mothers receive growth monitoring, only those under 2 years (6-23 months) receive a preventive ration and those over 2 years but under 59 months who are malnourished (PFA) or severely malnourished (PTFA) receive food supplements.45 A package for food distribution for selected mothers (pregnant or lactating mothers receiving counseling is also given out within the same area at „food distribution points‟ each month (serving the needs of several rally posts). For CRS, rally posts, (postes de resemblements or literally „assembly posts‟), are the critical points within the localities of the communes targeted for assistance where the larger community of mothers and their children, mostly less than 5 years of age, and most of those that come are less than 2 years of age. Rally posts are managed by volunteer community health workers, called Col Vols (Collaborating Volunteers), some of whom are given special training to also provide vaccinations at these posts. A Col Vol, each month, gives a special training message lasting about 15 minutes, then begins the process of weighing and recording all the weights of the children present, targeting those under 2 years of age into his or her rally post register. Those with PFA or PTFA are told to present themselves to the regional health center where they will be provided with care, including some food supplements. Cols Vols will also form one or two groups of 20 mothers into Mothers Clubs, and these mothers are provided additional preventive health training for their children under 5 years of age. ACDI/VOCA uses much the same approach to rally posts as CRS. The only major difference is that because of the lack of health clinics within many of its regions of operation, ACDI/VOCA has worked with specific communities to create „Fixed Points‟ that serve as a kind of community base „health clinic‟ where special medical services can be staged and distributed to mothers from the surrounding rally posts. Since the January earthquake, Save the Children has been present at the health clinics, fixed points, and some „mobile clinics‟ to provide special care for the extremely malnourished children coming from the rally posts. This care will continue to be provided through ACDI/VOCA support after this 8 months service is complete. Addressing Malnutrition: One would assume that a MYAP program, after two years of focused activity and training within a specific locality (and rally post), would have brought the mothers within this specific community to learn the life skills needed for the health and nutrition of their children and households – and some certainly have. Because none of the MYAP PVOs are actually reaching ALL mothers with children under 5 years of age through their different mothers‟ club approaches, it is at the rally posts that all mothers near enough to walk to these places with their children at least have access to growth monitoring and some social services. In the case of CRS, these services are greatly limited – expecting mothers and children to go to health clinics/dispensaries for these. CRS does not provide food supplements at or near rally posts, as does World Vision Haiti, but only will do so through the often distant health clinics. 45 PFA = Pois Faible pour l‟age (d‟enfant), and PTFA = Pois Tres Faible pour l‟age (d‟enfant). 58 0% 5% 10% 15% 20% 25% 30% 35% Jamais Vu Grille Dassa haut Lherisson Ricot Martineau Figure 1 above, using data from a commune in which CRS has worked for over 15 years, illustrates that the base rate of malnutrition is not changing as one might expect in spite of the training given out. The blue series of data above represent the average % of malnourished children (PFA + PTFA) coming from 9 separate rally posts. The green series represents the % of malnourished children from a single rally post (Casse). It is difficult to see any pattern in these data, other than their randomness. Intuitively one might have expected the February-April 2010 data following the recent earthquake (January 12) to have caused a surge of malnutrition – but exactly the opposite is evident with the lowest malnutrition rates ever recorded. Yet the MTE team also reviewed CRS rally post annual data where malnutrition rates registering at the 3% to 4% levels, or lower, over time – coastal areas with higher production potential and closer to health clinics. At a rally post in the Aquin commune called Demarais, served by a Col Vol, and supported by the St. Antoine de Valbourg Health Clinic, of 53 children weighed, ages 1 – 59 months, 3.9% were malnourished (PFA) for February 2010, 3.8% for March, and 2% for April 2010. Closer look at the data also showed that there was one child that was among the two malnourished children at this rally post that remained a problem for all three months. This rally post is a candidate for graduation. They also had two older Mothers Clubs (of 20) among mothers coming to this rally post, though most of mothers did not belong to a Mothers Club. Figure 2: Southeast Department, Cotes de Fer Commune, Communal Section: 6ieme Jamais Vu (6 Rally Post Localities)(ACDI/VOVCA data, January 2009 – May 2010)(PFA+PTFA as a percentage of total children weighed) 0 10 20 30 40 Figure 1: CRS, Commune Les Anglais, 1ere Section Verone Series1 Series3 59 Six (of twelve) rally posts in ACDI/VOCA‟s Southeast Department, Cotes de Fer commune, in the 6th communal section called Jamais Vu, over a seventeen month period given above in Figure 2, also show considerable variation from month to month. What is interesting are the data from the Jamais Vu and Dassa rally posts that show long-term malnutrition rates below 10%, while Grille and Martineau show very high rates of malnutrition. MYAP programs needs to review such data and determine why this is so. The answers may lead to reinforcing some areas and reducing efforts in others. The subjective MTE team‟s observations on this kind of information both in the Jacmel and Les Cayes regions suggests that the presence of health clinics in the locality may explain lower rates, while higher rates are associated with sites at considerable distance from clinics. Another variable may be the presence of Mothers Clubs that help reinforce messages that can help reduce malnutrition by improved child feeding and hygiene practices (with less diarrhea for example). World Vision Haiti data above in Figure 3 were provided as aggregated data from the various rally posts in three Section Communales in the Central Plateau. Aggregate malnutrition rates for Anse a Galets have been below 8% for over a year, while those of the other two section communales both vary between about 13% and 16%. Again, the MYAP needs to determine the cause of this, and plan a programmatic response. A review of the actual data from the specific rally posts at these Central Plateau localities, however, would probably also show wide monthly swings. As with the CRS aggregated data above in Figure 1, aggregation smoothes out the curves, masking the realities of different specific localities. It is for this reason that the MTE team believes that such data need to be analyzed at the locality (rally post) level, and that aggregated data, whether at commune or section communales levels, are not particularly meaningful. Ideally, if the data were to become available for the children less than 5 years of age for all the mothers of a specific mothers club, one would be able to undertake even more accurate analysis. In such a situation, one would actually know the specific children concerned over time – whereas in the more aggregated data from even rally posts, it is not always the same children who are weighed from month to month. This disparity grows as one moves up in aggregation from rally posts to Section Communales and then to the Commune level. 0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% 14.0% 16.0% 18.0% Figure 3: World Vision: 3 Section Communales (Average of their Rally Post Data) (Oct. 2008 - March 2010) Anse à Galets Hinche Verrettes 60 61 MYAP Group Activities: Though one would expect that mothers with malnourished children would always be targeted for inclusion into one or more of the MYAP economic/livelihood development groups, this has not been the reality on the ground for most CRS and ACDI/VOCA groups, and to some extent with World Vision Haiti groups as well – though there have been some attempts by all three MYAPS to refocus in this way in recent months. Groups receiving livelihood activities might include some mothers receiving benefits from the MCHN program at some point in time, but could just as well not have any such women, or their households included. MYAPS have not intentionally targeted „the most vulnerable households‟ of the focus sub-groups listed above to provide an integrated package of activities that would always include MCHN training for household mothers and children under 5 years of age, and for agricultural/livelihood activities to improve households well-being and reduce food insecurity. CRS has only in recent months begun to take a more proactive approach in this regard, with monitoring forms, for Mothers Clubs, for example, specifically noting whether or not members are receiving any kind of livelihood program training inputs. „Development‟ appears more an attempt to organize farmers into groups to do collectively manage different economic activities; however when one looks more closely at how this is done, there does not appear to be an intentional effort or focused commitment by MYAP managers to see that all aspects of the program seek the development of entrepreneurship among selected individuals or households. There are exceptions, of course. AGRIDEV‟s work within the irrigated perimeters (rich valley soils) of some WVH sites to develop regional and international markets for high value vegetable and fruit produce (e.g. peppers, mangoes) would be an example. Most of the CRS agricultural activities appear to be moving in this direction, though the focus of such efforts „through groups‟ may jeopardize their long term viability. Many of these efforts could have a more intentional focus of creating private entrepreneurship – as with the young people learning how to do grafting of fruit trees. Certainly ACDI/VOCA‟s focus on addressing the value chains of targeted livelihood activities and linking specific commodity producers to identified markets would meet this requirement. But all these activities could have benefited from greater attention to linkage with MCHN targeted households. 3.3 Links to Government of Haiti (GoH) Each of the MYAP Cooperating Sponsors has made a serious effort to work in every way possible with weak GoH institutions within their regions of implementation. This working relationship appears strongest with both CRS and ACDI/VOCA. CRS has its focus of channeling all its efforts in MCHN in ways that will help to build up and train Ministry of Health clinics and personnel. The CRS MYAP consciously chose a means of collaboration that, from one perspective, might appear less effective in moving out into the needs of community groups, but that the steps it did take were always taken alongside a Haitian Ministry of Health colleague. For example, mothers with identified needs (pregnancy, malnourished children) at rally posts are always referred to a health clinic for treatments, receipt of services, food supplements, etc.46 Many of these services are provided by World Vision employed health personnel at mobile clinics and rally posts – beyond the reach of the GoH or other private health facilities in their regions. World Vision brings services closer to where the people are, and probably results in more direct benefits to the local populations concerned – but the approach is not designed to strengthen GoH facilities or personnel in the same way as CRS does. Unlike both CRS and ACDI/VOCA, the WVH MYAP operates its own health clinics and dispensaries – 46 These health clinics or dispensaries may be MSSP operated, or run by other private entities, such as the Catholic Church. 62 though with MSSP collaboration.47 ACDI/VOCA approaches in MCHN and agriculture alike have also associated GoH personnel in aspects of MYAP implementation, in training events for seed production. ACDI/VOCA has a strong collaboration with the Ministry of Agriculture (MARNDR & CNSA) and the Ministry of Health (MSSP). For example, MYAP managers attend „Table de Concertation‟ meetings in Jacmel on a regular basis to discuss agricultural issues and share with MARNDR representatives the level of ACDI/VOCA field achievements. All of their health agents and auxiliary nurses are under contract with the MSPP, and salary rates closely follow those of the GoH. GoH, currently without strong enough Agricultural or Health Ministries or financial means to sufficiently reach rural communities, have welcomed the opportunity of working with CRS, ACDI/VOCA, and WVH in rural extension and outreach activities. MYAP funds frequently make it possible for GoH personnel to become involved in field-based activities – providing logistic support for training programs for example. The Early Warning System (EWS) sub-component of all three MYAPs is also a good example of working closely with civil government leaders at Department, Commune, and section communales levels to further program objectives. Efforts to empower local members of the different communities WVH MYAP activities are involved in through Community Management of Acute Malnutrition (CMAM) efforts is another example of such local level collaboration. Local officials are always involved in helping the MYAP to target specific communities within specific regions, even though this may at times reflect political realities more than specific food security priorities. Here, ACDI/VOCA appears to have developed the most interesting working relationship with different government and other NGOs in developing an early warning system to which they all collaborate, and are involved on a regular basis. In PAP, at the WVH central offices, there is a monthly EWS meeting bringing USAID, MYAP managers, GoH representatives from MARNDR/CNSA, occasionally a representatives from FEWSNET and EWS M&E personnel together to review the past month‟s activities and plan for the coming month. Besides these points, the following observations should be made: • In some areas (WVH on La Gônave for example), the MYAP NGOs reinforce public health infrastructures, and provides equipment to health centers. • In other areas, the MYAP NGOs themselves provides health services in remote areas, where there is a lack of resources from the public sector (as for example fixed points, in Southeastern Department), assuming the substantial financial and technical operations in the project • Activities are conducted through public health centers, whenever possible. The GoH supports the MYAP programs through its own health professionals. Health workers are under MOH contract, but receive their salaries from the MYAP project. The MYAP NGOs provide training and ensures community health services through health workers and volunteers. Methods also vary. In some cases, a MYAP (WVH) may assume all the costs of using specific services by the beneficiaries (cases of severely malnourished children), or, in other cases (CRS), the user assumes the cost of using institutional services. • Implementation of many community activities (as for example: „journee sur l‟allaitement maternel‟) are public initiatives that are supported within rural communities by the MYAPs. • Representatives of the Ministry of Health and the Ministry of Agriculture participate in regular meetings 47 The WVH MYAP also has worked through the clinics of a private group in the Lower Central Plateau, the Albert Switzer Hospital and its network of dispensaries. 63 related to the MYAP. Indicators and training sessions are chosen together with those ministries. 3.4 Staffing and Capacity Development The MTE team was impressed by the quality and professional capacities of the World Vision, CRS, and ACDI/VOCA regional and field personnel. There are highly trained people in many different professions in Haiti, though many of these have in recent months fled the country looking for employment opportunities elsewhere. A constant theme encountered in our focus group meetings was how much local farmers appreciated the MYAPs bringing in these agronomists, these fruit tree grafting experts, these health personnel to work with them on improving their livelihoods. Rural Haitians are hard workers, and can greatly improve their household and family well-being when appropriate training is received. MYAPs have been able to identify excellent staff to fill these positions in often difficult to live in areas – always preferring to hire locally when they and find an able person. If any reproach were to be made at all, it would have to be that, because working conditions are so demanding and the task great, that personnel are not provided enough opportunities to themselves grow professionally. For example, there did not appear to be many cross-site visits between MYAPs taking place to observe and learn how others were addressing similar issues. The MTE team was impressed by many of the model farmers and excellent interventions taking place in different places – and others should see what we saw. World Vision, with major programs in many parts of the world, has made serious efforts to promote their senior in-country management teams to positions of greater responsibilities. Just within the small group we interacted with, the WVH MYAP M&E coordinator came from a Malawi program, the SYAP manager had been a DAP manager in Rwanda, and another SYAP leader was a commodities expert from Zimbabwe. These and others like them show a healthy institutional environment for learning and personal growth that should be encouraged. Where the implementation of a MYAP in some sub-region is done through partners, the lack of sufficient funding or a need for restructuration might require partners (as for example World Vision partners HAS and SAVE) to reduce the number of its field health workers. It is important to understand well the underlying reasons for low achievements of a partner – they may be entirely due to lack of the resources needed for staffing and capacity development. Furthermore, the fact that the community health workers (such as Col Vols, agricultural associates) are chosen from community volunteers is a positive element to the sustainability of the MYAP services, because these men and women will continue their lives within these communities after the withdrawal of the project; many will continue to provide these services in the years to come – also voluntarily. 64 3.5 MYAP Financing and Commodities The MTE team did not have the time to explore in any depth the details of MYAP finances or the movement and management of commodities. All three CSs have proven international experience and expertise in the logistics of importing and distributing commodities through FFP programs, and in food and cash work/asset programs involving large groups of people orientated towards specific tasks of soil conservation, rural road repairs and other group efforts. They have worked out a scheme of rotating the responsibilities of managing commoditization of PL 480 food commodities and in the distribution to MYAPs their allotments of food commodities for distribution. One experience did stand out with respect to commodities. The MTE team took every opportunity possible to see the MYAPs in actual „real-life „operational modes – such as holding rally posts or mobile clinics. We observed three approaches to the distribution of food commodities to vulnerable targeted groups. In the case of World Vision, the program makes the effort to bring the food commodities as close as possible to the beneficiaries using „food distribution points‟ that might serve several rally posts. One such distribution point near Hinche is shown here. Trucks bring in and deliver the commodities, which are then distributed by World Vision commodity teams, with the assistance of some health agents. Pre-defined lists are used to check off those who show up for the rations – which would include vegetable oil, lentils, and corn/soybean blend, and people are grouped to distribute these to specific individuals. Our impression was one of organized confusion, with people shouting and shoving, milling around. It looked like an all day affair. ACDI/VOCA, on the other hand, has a sub-contractor (BND) who takes the commodities and repackages them into the right amounts of the various products. These are then transported, on announced days, to distribution points where people on a pre-qualified list can show up to receive their bag of commodities – a much more organized approach. Just pick up the bag and go. CRS, on the other hand, requires those to receive food rations to come down to the sometimes distant GoH or private health clinics where their children are once again weighed, and then they are given coupons to begin a process that leads to a door that will be opened up to measure out the rations for each person. Another time￾consuming process. Clearly ACDI/VOCA‟s method was the most consumer-friendly. The down side is that sometimes these trucks get mired down along the way by unexpected rainfall, as we experienced at Belle Anse one day, unable to deliver or return home that day. 65 3.6 MYAP Impact on Environment & on Population Effected by Recent Earthquake The MYAPs are continuing to have a positive impact on environmental issues within their regions of intervention – all three have been involved in large scale soil conservation works – through the planting of agro-forestry species, as well as the rehabilitation of productive infrastructures (irrigation canals for example) and protecting targeted watersheds. Farmers interviewed at such sites claimed that these efforts had helped them to reclaim land that they were no longer using. Unfortunately, because of the long￾established practice of providing free tree seedlings for these reforestation programs, there has not be established any sustainable system that local people will assure future access to additional seedlings. People have come to assume that some NGO or the GoH will provide them with new future tree seedlings. Reforestation has been going on in Haiti for many decades, using the same techniques used by these MYAPs. And while seedlings are planted, many do not survive long but are eaten up by the free ranging goats on every hillside. Until goats are required to be staked down, or only raised in more intensive systems of feeding, this problem will not be resolved. And an unintended consequence of improving run down irrigation perimeters is that the well-off absentee landlords are given an asset of increased value for which they are able to now charge more in rental fees or take their share of the share cropping arrangements most common in these areas. Because the land users are not the land owners, these irrigated perimeters will once again fall into disrepair, or clog up with silt – no individual sharecropper willing or able to spend their time for these land owners. The MTE team also observed what also happens when hillside plantations of seedlings are established on agriculturally poor land. An entire hillside of small trees, no more than an inch in diameter, had been cut down and stacked in piles to produce charcoal – and poor quality charcoal at that. What is given away is often not valued to the same extent as what is personally owned or purchased. With greater commercial interest being shown in many parts of the country for fruit trees (grafted mangos, citrus, avocados, etc), NGOs and GoH personnel alike need to provide incentives for people to purchase these seedlings themselves, and train individual entrepreneurs to develop both fruit tree and high value vegetable nurseries for sale and resist the temptation to simple give these away as well. The presence of the MYAP programs in different regions of the country – in vulnerable regions – had a very important impact in facilitating the rapid responses that followed both the 1988 cyclones and the January 2010 earthquake. Without the pre-positioned personnel and food resources of these programs – experienced in food relief and emergency efforts - the degree of suffering of the Haitian people would Moving Forw 66 have unquestionably been much greater. Benefits of these programs most certainly also provided some cushion when urban kinsmen fled devastated regions looking for food and shelter. 4.0 Moving Forward Besides responding as much as possible to the various recommendations presented in this report and within the specific MYAP Reports #2, #3, and #4, the MTE suggests that each Haiti MYAP now begin to pay greater attention to developing time series data on key component activities. This is so that, by the time of the MYAP final evaluation, the review team at that time will have the objective data to assess the success of each MYAP in meetings its stated goals and objectives for each component, with special focus on impact. For example, each MYAP should begin to construct time series data for malnutrition (PFA + PTFA and MUAC measures for severe malnutrition) at a number of locality rally posts, one table for posts with „easy access‟ to MCHN services, one table for areas with difficult access. Tables might look something like that shown below for a World Vision/ADRA program in Rwanda (2009). 67 Within the agriculture/livelihood component, MYAPs may wish to monitor a selection of trained seed producer groups and chart the growth, over time from the beginning of the project, the total annual kilograms of improved seed produced each year, and the economic value of this to these entrepreneurs for their future sustainably. Another table might show the growth in household production (kgs.) of tomatoes, eggplant, cabbage, chili peppers) and revenue (in gourdes) for a group of households by commune within Mothers Clubs in household level vegetable gardens. This might be differentiated between those with micro-drip irrigation and those with other means of acquiring water during the year. Whatever activities that should be selected for impact monitoring, the MYAP field team will need to go back over their historical data, recorded at the field level, to put these data sets together, and to them begin to consistently track this into the future, and begin to include this in the quarterly and annual reports. 68 Annexes: Annex 1: Scope of Work for Mid-Term Evaluation Annex 2: Mid-Term Evaluation Logistics Scheduling Table (with External Consultant Haiti Schedule) Annex 3: Protocol for Mid-Term Evaluation Annex 4: Key Documents Consulted Annex 5: List of Key Persons & Organizations Consulted Annex 6 6A: MYAP Indicator Performance Tracking Tables for World Vision Haiti 6B: MYAP Indicator Performance Tracking Tables for ACDI/VOCA 6C: MYAP Indicator Performance Tracking Tables for CRS Annex 7: MYAP Mid-Term Quantitative Survey Instrument Annex 8: MYAP Mid-Term Quantitative Data Tables Annex 9: Mid-Term Evaluation Team PowerPoint Presentation (June 3, 2010)