Catholic Relief Services Burundi Multi-Year Assistance Program (MYAP) Cooperative Agreement #FFP-A-00-08-00080-00 Final Evaluation Report 31 July 2012 (FINAL) MYAP Final Evaluation Report 31 July 2012 i Acknowledgement The members of the evaluation team would like to gratefully acknowledge the support provided by the staff of the MYAP from CRS, IMC, ODEDIM and BADEC in making background documentation available, organizing field visits and being available for meetings and discussions. The team especially appreciated the extraordinary effort that the staff made to ensure that they did not influence the discussions we held with participants. Innumerable program participants and beneficiaries have also provided useful responses to our questions and participated in stimulating discussions. Our gratitude goes out to them as well. While the four of us are responsible for the content of the report, four other members of the team provided valuable support and insights, and we would like to acknowledge their contributions. Mr. Felicien Harerimana from the Burundi Language Center, Ms. Mary Rose Hatungimana from the English Department at the University of Burundi, Mr. Jean Claude Ndikumazamboa independent consultant, and Ms. Danicella Dusabe also an independent consultant all displayed high talent in facilitating communications for us in Kirundi, French and English. All of them made substantial contributions in helping us interpret our observations and discussions with program participants. Hopefully we've listened well, our observations are grounded in reality, and our assessment is accurate. Sincerely, Mr. Ali Aamoum, Commodity Management Specialist Dr. Mirella Mokbel Genequand, Maternal and Child Health and Nutrition Specialist Mr. Bernard Crenn, Agriculture & Livelihoods Specialist Mr. Mike DeVries, Program Design, Monitoring and Evaluation Specialist This report was made public by support from the U.S. Agency for International Development (USAID) under the terms of Cooperative Agreement FFP-A-00-08-00080-00. The authors’ views expressed in this publication do not necessarily reflect the views of USAID or the Government of the United States of America MYAP Final Evaluation Report 31 July 2012 ii The FY 2008-2012 Multi-Year Assistance Program CRS Burundi FINAL EVALUATION I. EXECUTIVE SUMMARY .......................................................................................... 1 II. BACKGROUND ......................................................................................................... 4 A. Overview of the MYAP Strategy .......................................................................................... 4 B. MYAP History and Operating Environment ......................................................................... 5 C. Evaluation Methodology ....................................................................................................... 6 1. Qualitative Component 2. Quantitative Household Survey 3. Evaluation Limitations III. ACTIVITIES AND OUTPUTS PRODUCED UNDER EACH STRATEGIC OBJECTIVE .................................................................................................... 8 A. Outputs Produced Under Strategic Objective 1: Human Capacities/Maternal and Child Health and Nutrition ............................................................................................. 8 1. Cross-Cutting Activities for IR 1.1 and IR 1.3 .............................................................. 9 a. Behavioral Change Communication b. Training of Volunteer Community-Based Health Activists (VCBHA) 2. IR 1.1 Optimal Infant and Young Child Feeding (IYCF) Practices ................................ 11 a. Community-Based Approaches to Encourage Improved IYCF Practices i. Mother Care Groups ii. Positive Deviance/Hearth b. Growth Monitoring i.. Facility-Based Growth Monitoring ii. Community-Based Growth Monitoring c. Community Management of Acute Malnutrition (CMAM) i. Stabilization Centers ii. Outpatient Therapeutic Program (OTP) iii. Supplemental Feeding Program 3. IR 1.2 HIV-Affected Households .................................................................................. 20 a. Appropriate Supplementary Food b. Nutrition Education for PLHWA c. Keyhole Gardens/Vegetable Production 4. IR 1.3 Heath-Seeking Behavior ..................................................................................... 21 5. IR 1.4 Hygiene and Sanitation ....................................................................................... 21 a. Water Point Rehabilitation b. Hygiene and Sanitation Education B. Outputs Produced Under Strategic Objective 2: Livelihoods Capacities .............................. 24 1. IR 2.1 Farm Production ................................................................................................. 25 a. Anti-Erosion Measures b. Crop and Livestock Technologies (Lead Farmers and Farmers) c. Productive Infrastructure Rehabilitation (FFA/CFW) d. DPEA Capacity-Building e. Seed and Livelihoods Fairs f. On-Farm Trials and Multiplication g. Goat Distribution and Cross-Breeding 2. IR 2.2 Revenues ............................................................................................................. 33 a. Agro-Enterprise Capacity Building b. Savings and Internal Lending Communities (SILC) C. Outputs Produced Under Strategic Objective 3: Community Resilience .............................. 36 MYAP Final Evaluation Report 31 July 2012 iii 1. IR 3.1 Early Warning and Response ............................................................................. 36 a. Risk and Vulnerability Assessment b. Community Action Plans 2. IR 3.2 Asset Management/Gender Equity ..................................................................... 38 a. Community Sensitization on Gender Equity in Decision-Making b. Training on Women's Rights IV. PROGRAM IMPACT ASSESSMENT .................................................................. 41 A. Focus of the Final Evaluation .............................................................................................. 41 B. Targeting ............................................................................................................................. 41 C. Outcomes Under SO 1: MCHN .......................................................................................... 44 1. General Comment on Quantitative Tables 2. IR 1.1 Infant and Child Feeding Practices 3. IR 1.2 HIV-Affected Households 4. IR 1.3 Heath-Seeking Behavior 5. IR 1.4 Hygiene and Sanitation 6. Comparison of SO1 Outcomes Between Watershed and the Province D. Outcomes Under SO2: Livelihoods ....................................................................................... 52 1. IR 2.1 Farm Production 2. IR 2.2 Revenues E. Outcomes Under SO 3: Community Resilience ................................................................. 56 1. IR 3.1 Early Warning and Response 2. IR 3.2 Asset Management/Gender Equity F. Collective Impact at the Program Goal Level ....................................................................... 60 V. PROGRAM PROCESS ASSESSMENT ................................................................. 62 A. Program Management B. Partnership C. Knowledge Management D. Program Integration and Complementarity E. Accountability to Beneficiaries F. Resource Management 1. Financial Resources 2. Commodities 3. Human Resources G. Environmental Monitoring and Impact Mitigation VI. GOOD PRACTICES DEVELOPED OR TESTED IN THE MYAP ................... 70 A. Savings and Internal Lending Communities (SILC) Under SO2 B. Gender Intervention Under SO3 C. Mobile Team D. Program Coordination Workshops E. Front-Line Animateurs VII. MAJOR PROGRAM-LEVEL LESSONS LEARNED ........................................ 72 A. The Watershed Development Approach B. Importance of Other MYAP Components for Supporting the Gender Intervention C. Exit Strategies D. Being Participant Driven E. Tracking High Impact Beneficiaries VIII. CONCLUDING REMARKS ................................................................................ 73 MYAP Final Evaluation Report 31 July 2012 iv ANNEXES Annex A: Key Dates in the Life of the MYAP 75 Annex B: Approved Final Evaluation Plan and Scope of Work 76 Annex C: Qualitative Interviews and Site Visits 129 Annex D: SO2 Livelihoods Summary Statistics 145 Annex E: SO3 PAC and Gender Summary Statistics 149 Annex F: New Ideas and Knowledge Brought into the MYAP from Outside 151 Annex G: Recommendations for the Remaining Life of the MYAP 152 Annex H: Commodity Summary Tables 161 TABLES & FIGURES Table 1. MYAP Resource Summary 5 Table 2. Qualitative Survey Baseline and Endline Samples 7 Table 3. Logic Structure for the MCHN Component of the MYAP 8 Table 4. Summary of VCBHA Activities 10 Table 5. Performance Indicators for PD/Hearth 13 Figure 1. CMAM Decision Tree and Referral System 15 Table 6. Daily Ration Scale and Composition (grams per person per day) 16 Table 7. Performance Indicators for Stabilization Centers (2010 & 2011) 17 Table 8. Performance Indicators for Outpatient Therapeutic Program (2010 & 2011) 18 Table 9. Performance Indicators for the Supplementary Feeding Program (2010 & 2011) 18 Table 10. Logic Structure for the Livelihoods Component of the MYAP 24 Table 11. Delivery of Knowledge through Agricultural Technology Transfer 27 Table 12. Logic Structure for the Community Resilience Component of the MYAP 36 Table 13. Reported Beneficiaries Against LOA Targets 42 Table 14. Estimated Beneficiaries for SO1 Through December 2011 43 Table 15. Infant and Young Child Feeding Practices 44 Table 16. Immunization and Growth Monitoring 45 Table 17. Preventing Common Childhood Illnesses 46 Table 18. Women Health Knowledge and Practice 48 Table 19. Hygiene and Sanitation Practices 49 Table 20. Diet Diversity Score 52 Table 21. Agricultural Practices 53 Table 22. Five Most Mentioned Household Expenses 56 Table 23. Household Income Sources 56 Table 24. Household Food Sources 56 Table 25. Average Months of Adequate Food Provisioning Capacity 57 Table 26. Household Survey Impact Indicators for Gender Equity 59 Table 27. Where is Impact Being Generated in the MYAP? 61 Table 28. Nutritional Status of Children Under Five Years of Age 61 Table 29. Cash Expenditure Summary (US$ Through February 2012) 67 MYAP Final Evaluation Report 31 July 2012 v ACRONYMS AND ABBREVIATIONS ACSA Agent Communautaire de Santé Animal AE Agro-Enterprise AIDS Acquired Immune Deficiency Syndrome ANC Antenatal Care ANSS Association Nationale de Soutien aux Séropositifs et Malades du SIDA ARI Acute Respiratory Infection ART Antiretroviral Therapy ASC Agent de Santé Communautaire (VCBHA in English) AT Agent Technical BADEC Bureau d'Appui au Développement et à l'Entraide Communautaire BCC Behavior Change Communication BPS Bureau Provincial de la Santé BUBD Best Used By Date CAFOB Collectif des Associations et ONG Féminines du Burundi CBSD Cassava Brown Streak Disease CCDC Communal Comites de Developpement Communitaire CDC Comites de Developpement Communitaire CDF Centre de Developpement Familial CFW Cash for Work CH Health Center CMAM Community-Based Management of Acute Malnutrition CRS Catholic Relief Services CSB Corn Soya Blend CSLP Cadre Stratégique de Croissance et de Lutte contre la Pauvreté DHS Demographic and Health Survey DPAE Direction Provincial de Agriculture et Elevage EPI Expanded Program on Immunization FARN Foyer d’Apprentissage et de Réhabilitation Nutritionnelle FBu Burundian Franc FFA Food for Assets FFP Food for Peace FFW Food for Work FGD Focus Group Discussion FHI Family Health International FIFO First In First Out FP Family Planning GIZ Gesellschaft für Internationale Zusammenarbeit GFP Gender Focal Points GM Growth Monitoring HIV Human Immuno-Deficiency Virus IYCF Infant and Young Child Feeding IGA Income-Generating Activities IMC International Medical Corps IPTT Indicator Performance Tracking Table MYAP Final Evaluation Report 31 July 2012 vi IR Intermediate Result ISABU Institut des Sciences Agronomiques du Burundi ISAR Institut des Sciences Agronomiques du Rwanda ISTEEBU L'Institut de Statistique et d'Etudes Economiques du Burundi KAP Knowledge, Attitudes and Practice KPC Knowledge, Practice and Coverage LAE Lutte Anti-Erosion LLITN Long-Lasting Insecticide Treated Nets LM Lead Mother M&E Monitoring and Evaluation MCG Mother Care Group MCH Mother-and-Child Health MDG Millenium Development Goal MINAGRIE Ministry of Agriculture and Livestock ML Mère Lumière MML Maize Meal MoH Ministry of Health MOU Memorandum of Understanding MQC Management Quality Coordinator MSF Médecins sans Frontières MTE Mid-Term Evaluation MUAC Mid-Upper Arm Circumference MYAP Multi-Year Assistance Program NAS National Agricultural Strategy NCE No Cost Extension ODEDIM Organisation Diocésaine pour l'Entraide et le Développement Intégral de Muyinga OFSP Orange-Fleshed Sweet Potato ORS Oral Rehydration Solution OTP Outpatient Therapeutic Program (STA in French) PAC Plan d'Action de Communautaire PAPSAD Programme d’assistance pluriannuelle aux ménages vulnérables pour une sécurité alimentaire durable PBF Performance Based Financing PD Positive Deviance PHAST Participatory Hygiene and Sanitation Transformation PLWHA People Living with HIV/AIDS PNAC Programme de Nutrition à Assise Communautaire PNC Post-Natal Care PRONIANUT Programme National Intégré d’Alimentation et de Nutrition PVS Participatory Variety Selection QMP Quality management Protocol RBP+ Réseau Burundais des Personnes Vivant avec le VIH/SIDA RTNB National Radio Station RTUF Ready to Use Therapeutic Food SAM Severe Acute Malnutrition MYAP Final Evaluation Report 31 July 2012 vii SFP Supplementary Feeding Program SILC Savings and Lending Committee SO Strategic Objective SPC Suivi et Promotion de la Croissance SST Service de Stabilisation STA Service Thérapeutique Ambulatoire (OTP in English) SSN Service de Supplémentation Nutritionnelle TPS Techniciens pour la Promotion de Santé UNICEF United Nations Children's Fund VCBHA Volunteer Community-Based Health Activists (ASC in French) WFP World Food Program WHO World Health Organization MYAP Final Evaluation Report 31 July 2012 1 The FY 2008-2012 Multi-Year Assistance Program CRS Burundi FINAL EVALUATION I. EXECUTIVE SUMMARY Catholic Relief Services (CRS) commissioned a Final Evaluation of its Title II Multi-Year Assistance Program (MYAP) being implemented in partnership with the International Medical Corps (IMC), the Bureau d'Appui au Développement et à l'Entraide Communautaire (BADEC), and the Organisation Diocésaine pour l'Entraide et le Développement Intégral de Muyinga (ODEDIM) in three provinces of northern Burundi. The program has three components, (1) a maternal and child health and nutrition component that is focused on pregnant and lactating women and children under the age of five years, (2) a livelihoods component focused on agriculture and natural resource management using a watershed development approach, and (3) a community resilience component which builds local capacities for disaster risk reduction and promotes gender equitable decision-making at the household and community levels. Food resources are used in the MYAP for therapeutic and supplemental feeding as well as in food-for￾work activities. Including a one-year costed extension, the total Life-of-Activity program cost to Food for Peace is estimated at US$ 21,016,4001 with 32,539 MT of commodities for monetization and distribution. The program targeted having impact on 125,000 households throughout the three provinces with the maternal and child health and nutrition component and on 18,000 households in three targeted watersheds within the three provinces with an integrated strategy including all three program components. The program began implementation on 4 August 2008 and is scheduled to be completed by 3 August 2012. The Final Evaluation was conducted by a team of four development professionals2 over the period 7 March through 5 April, 2012, in Burundi. The team reviewed existing secondary sources of information, reviewed available quantitative information from baseline and endline household surveys, and used qualitative survey methods to obtain information to understand the impact achieved by the MYAP. The most significant outputs produced by the program included the following:  1,440 Volunteer Community-Based Health Activists (VCBHA) were trained and equipped by the program to disseminate health knowledge and serve as links between communities and health facilities.  1,886 lead mothers, each working with ten to twelve other mothers in 183 Mother Care Groups, were trained to promote appropriate behavioral change for health seeking behavior and health/hygiene.  120 Mere Lumiere were identified and trained to facilitate Positive Deviance/Hearth sessions for 1,941 pairs of mothers and children.  317 Ministry of Health staff from health centers in the program area were trained to provide Community Managament of Acute Malnutition and Growth Monitoring Services for 1 Total C&F Commodity Value = $17,225,400, Total ITSH Value = $1,720,769 and Total 202e Value = $2,070,231. 2 Mike DeVries, Program Design, Monitoring & Evaluation Specialist; Bernard Crenn, Agriculture & Livelihoods Specialist; Dr. Mirella Mokbel Genequand, Maternal and Child Health and Nutrition Specialist; and Ali Aamoun, Commodity Management Specialist. MYAP Final Evaluation Report 31 July 2012 2 identifying various stages of malnourished children for referral to stabilization centers, outpatient therapeutic programs or supplemental feeding programs.  2,437 severely malnourished children under the age of five years requiring hospital admission received therapeutic food in stabilization centers supported by the MYAP.  17,305 severely malnourished children received therapeutic food through outpatient therapeutic programs supported by the MYAP.  34,461 malnourished children and 5,579 caregivers received supplemental rations through a supplemental feeding program implemented by the MYAP.  An estimated 2300 people living with HIV/AIDS received supplemental food and nutrition education training through the MYAP.  40 water points were rehabilitated to provide potable water for 4,066 households in the watershed collines.  A total of 746 kilometers of contour bunds were constructed through Food for Work (3,866 FFW participants) under the coordination of twenty Anti-Erosion Committees formed and trained by the MYAP.  134 Lead Farmers and around 6,000 farmers were exposed to improved crop and livestock technologies.  Three marais/valleys totaling 237 hectares were rehabilitated with water control structures through Food for Work (4,563 FFW participants).  Seed trials and multiplication plots were established for successful varieties involving twenty-three bean seed varieties, mosaic-resistant cassava varieties, three varieties of orange￾fleshed sweet potatoes, and three varieties of high-altitude rice.  The MYAP distributed 288 Boer bucks imported from Uganda and 5,400 local does from which 1,700 female kids were distributed in a solidarity chain approach, benefitting more than 900 households.  17 formal Agro-Enterprise Associations and 121 informal farmer groups were trained on agro-enterprise operations, financial management and marketing.  278 Savings and Internal Lending Communities (SILC) groups were formed with a total membership of around 6,000 participants covering 4,000 beneficiaries (some participants were members of more than one group).  21 Community Action Plan (PAC) committees were formed and trained to spread messages to mitigate the impact of drought and to plan and mobilize local resources for community development.  158 Community-Based Gender Focal Points and 150 Gender Positive Deviants were identified and trained to disseminate messages on the benefits of gender-balanced household decision-making and to mediate intra-household quarrels. Relative to the outcomes and impact achieved by the program, the following major highlights emerged from the evaluation.  The MYAP's Behavioral Change Communications (BCC) activities were effective in changing breastfeeding practices, particularly with significantly more mothers breastfeeding within one hour after birth, a significant increase in the proportion of children who have completed immunization schedules, and many more mothers given fluids to their children when they have diarrhea.  The household survey highlighted a threefold increase in the percentage of households using an improved toilet, from 6.8% at the baseline to 21.5% at the endline. MYAP Final Evaluation Report 31 July 2012 3  The MYAP has achieved substantial impact on access and supply to potable water with around 20,330 people benefitting from improved water quality and supply from the 40 water points rehabilitated.  Relative to SO1-Level indicators for stunting, wasting and underweight, the MYAP clearly achieved impact on the underweight indicator (an endine estimate of 29.7% against a baseline of 36.6%), but had statistically insignificant impact on the other two indicators.  There is generally good agreement between the household survey results, qualitative interviews and focus group discussions to establish that farm production has increased significantly especially in the rehabilitated marais, and that people are generally eating more, but it cannot be ascertained whether they eat better.  Under SO2, the great majority of participants in the watershed collines now have better quality of life due to project revenue and livelihoods interventions, with poorer segments of the population benefitting more on a relative scale and the better off segments benefitting more on an absolute scale.  On gender equity interventions under SO3, qualitative interviews suggested that as many as 40% to 50% of participating households, or between 4,200 and 6,000 households, have changed toward more gender-balanced household decision-making, resulting in a wide range of individual, household and community benefits. The evaluation team noted that the MYAP is achieving at least some impact with all activities being implemented by the program. In qualitative interviews, program beneficiaries cited the marais development, the goat solidarity chains, the SILC approach, and the gender-balanced household decision-making interventions as those having the greatest impact; and some of these, notably the SILC and gender interventions will continue to be sustained and will likely diffuse to other households after the program ends. The evaluation identified a number of major innovative good practices that produced sustained impact at reasonable cost in the MYAP that merit replication in other similar programs. These included the SILC intervention under SO2, the gender intervention under SO3, the mobile team approach used to ensure technical quality, the program coordination workshops for planning and facilitating program integration, and the use of front-line animateurs as facilitators at the community-level for all program activities. Many lessons specific to an output, approach or a program process are documented in the report. Major lessons identified by the evaluation at the program-level that have been learned from the MYAP experience include benefits and constraints in using the watershed development approach, the importance of other MYAP activities for supporting the gender intervention, the need to develop exit strategies at program onset, the value of being participant-driven, and the need to track high impact beneficiaries participating in multiple activities. The final evaluation found that significant impact has been achieved by the MYAP on around 15,000 households in the watershed collines and on at least 75,000 children and 60,250 caregivers across the three provinces through MCHN capacity building. Some impact, especially the impact achieved from SILC, the goat solidarity chains, and changing gender roles, is very likely to be sustained with not only economic impact, but also social impact. A rough estimate of the cost per direct beneficiary per year for the four year, $21 million program is reasonable, at less than $28. In the judgement of the evaluation team, the CRS Burundi MYAP was a good investment of Title II resources. MYAP Final Evaluation Report 31 July 2012 4 II. BACKGROUND A. Overview of the MYAP Strategy The final goal of the MYAP is to reduce chronic and transitory food insecurity for vulnerable households in the Provinces of Kayanza, Kirundo and Muyinga in northern Burundi. The program has eight intermediate results under three strategic objectives as shown below. Final Goal: By 2012, vulnerable households in the provinces of Kayanza, Kirundo and Muyinga have reduced their chronic and transitory food insecurity. Strategic Objective 1 (Maternal and Child Health and Nutrition): Vulnerable households have enhanced human capacities. Intermediate Result 1.1: Households practice optimal infant and child feeding practices. Outputs: Functional Community-Based Approaches (Positive Deviance/Hearth and Mother Care Groups), Growth Monitoring and Coaching Services, and Community Management of Acute Malnutrition (CMAM) capacities with therapeutic and supplemental feeding. Intermediate Result 1.2: HIV-Affected Households consume sufficient food for optimal nutrition. Outputs: Nutrition Education and Kitchen Gardens, with supplemental rations Intermediate Result 1.3: Households practice good health-seeking behavior. Outputs: Health and Hygiene Education, Community Awareness-Raising, and Capacity Building for Village Health Workers Intermediate Result 1.4: Households use appropriate hygiene and sanitation practices. Outputs: Water Point Rehabilitation and Participatory Hygiene and Sanitation Transformation (PHAST) Education Strategic Objective 2 (Livelihoods): Vulnerable households have enhanced and sustainable livelihoods capacities. Intermediate Result 2.1: Vulnerable households have improved production in environmentally￾sustainable ways. Outputs: Marais Water Management Training, Knowledge on Anti￾Erosion Techniques with demonstrations constructed using Food-for￾Work, Livestock Diversification, Agricultural Technology Transfer, Capacity Building of the Department of Agricultural Extension, and Marais Infrastructure Rehabilitation using Food-for-Work. Intermediate Result 2.2: Vulnerable households have adopted strategies and techniques to diversify and increase revenues. Outputs: Agro-Enterprise Groups and Savings and Internal Lending Communities (SILC) Strategic Objective 3 (Disaster Risk Reduction and Gender): Vulnerable communities have enhanced resiliency. Intermediate Result 3.1: Communities and local government agents have collaboratively developed community-based Early Warning Systems and response action plans. Outputs: Community Risk Assessment Capacities and Community Development Action Planning Capacities Intermediate Result 3.2: Vulnerable households are efficiently managing their assets in an equitable manner. Outputs: Community Awareness-Raising on Gender-Balanced Decision-Making and Women's Rights Activities under SO1 are being implemented across all of the three provinces targeted by the program. Three watersheds encompassing 21 collines were identified within the three provinces, and these are the targeted geographic areas for a more intensive integrated strategy MYAP Final Evaluation Report 31 July 2012 5 including some activities in some locations from SO1 and all activities under SO2 and SO3. The MYAP results framework does not specify target numbers for the program. The proposal, however, indicates that 18,000 households are targeted in the watersheds and 125,000 households are targeted across the three provinces for SO1 activities. Table 1 summarizes the approved resources and expenditures through the completion date of 3 August 2012. Table 1. MYAP Resource Summary B. MYAP History and Operating Environment The CRS Burundi MYAP is the first Multi-Year Assistance Program being implemented in Burundi. Previous recent Title II support was extended primarily for emergency response following the end of the conflict in 2005. The design of the MYAP built on experience from a Consortium Livelihoods Program that was implemented by a consortium of international NGOs in Kirundo with USAID funding from 2006 through 2008, and many elements of the livelihoods component of the MYAP and the basic outline of the gender strategy for SO3 were based on this experience. Over the life of the MYAP, the context in Burundi has evolved from a situation requiring significant rehabilitation in 2008 to a situation now in which development approaches are more appropriate. In 2010, MYAP and CRS Senior Staff recognized the need to revise the MYAP strategy given these changes in the context, and some approaches in the program were subsequently revised. Resource Initial Cooperative Agreement Current Approved Amounts Projections Through March 2012 Distribution Commodities 4,939 MT 6,279 MT 5,817 MT Monetization Commodities 15,100 MT 26,260 MT 26,260 MT Monetization Proceeds and Interest $6,455,072 $8,870,654 $7,198,024 202e $1,463,400 $2,070,231 $2,001,865 ITSH $1,388,500 $1,720,769 $1,720,769 CS Contribution $0 $0 $0 Total Program Cost $15,885,800 $21,016,400 $19,149,533 Direct Beneficiaries Target 18,000 HH 125,000 HH 18,000 HH 125,000 HH 190,000 Direct Beneficiaries Life of Activity 4 August 2008 to 3 August 2011 4 August 2008 to 3 August 2012 Lesson Learned The absence of target numbers in the MYAP Results Framework has caused some confusion for the program as to how many total beneficiaries they are expected to reach with the program over its life, and the program's M&E systems have not been designed to monitor progress against this target. Future MYAP Results Frameworks need to provide some specificity as to the number of people targeted to benefit from the program. MYAP Final Evaluation Report 31 July 2012 6 A significant portion of Kirundo Province was affected by perennial drought from 2008 through 2010 and early 2011, and the MYAP had to modify implementation plans somewhat to provide support for households affected by the drought. National elections were held in Burundi in 2010 and these had two effects on MYAP implementation. Once government staff turnover was completed in the provinces after the election, better relations with the Direction Provincial de Agriculture et Elevage (DPAE) and Bureau Provincial de la Santé (BPS) evolved since the new staff were generally more enthusiastic about workingwith the MYAP. However, on-going political tensions required increased security vigilance to protect staff and other assets. Over the life of the MYAP, the Ministry of Health (MoH) implemented a decentralization process in which more authority was delegated to the provincial level. This slowed progress initially in the MYAP until the staff were trained and began using their new authority. But then, once the decentralization became functional, the MYAP work became easier since decision￾makers in the BPS were closer to the program. The Ministry of Health also changed at least one set of protocols for Community Management of Acute Malnutrition (CMAM); and this, along with staff turnover of government health staff in the provinces, required additional time and resource investments for training. Finally, the appointment of a Food for Peace (FFP) officer to Burundi in 2010 improved awareness of program issues and timely decision-making at higher levels. A one year extension to the program with additional funding was approved in early 2011. Key dates for the MYAP are provided in Annex A. C. Evaluation Methodology The MYAP Final Evaluation is a summative evaluation intended to assess the impact that the program has achieved on the food security of targeted populations. Key questions guiding the overall approach in the evaluation were oriented around: • What has the project done (outputs)? • Who benefited from this? • How did they benefit? • How long will benefits likely be sustained? • What has worked particularly well (good practices)? • What lessons have we learned or what would we do differently next time? Both qualitative and quantitative data were analyzed by the four-person evaluation team. The Evaluation Scope of Work and Terms of Reference approved by FFP are provided in Annex B. 1. Qualitative Component. The evaluation team accompanied by program-independent translators recruited by CRS conducted key informant interviews and focus group discussions with beneficiaries, program participants and implementation staff over a period of twelve days with four days spent in each province. The actual persons and places visited are included in Annex C. Following the field data collection, the team analyzed the information obtained, along with available quantitative information from the household survey, and formulated preliminary observations on outputs, impact, lessons learned and good practices. These were shared with implementation staff and further refined in a validation workshop. Additional information was gathered and analyzed following this event to further clarify observations from the evaluation. MYAP Final Evaluation Report 31 July 2012 7 2. Quantitative Household Survey. CRS contracted L'Institut de Statistique et d'Etudes Economiques du Burundi (ISTEEBU) to implement a household survey to obtain quantitative endline information for comparison against baseline information on key outcome and impact indicators. The survey followed the baseline sampling methodology which used sample frames for gathering data from two populations as indicated in Table 2 below. As described in the previous section, SO1 MCHN activities take place in the entirety of the three provinces and the sample is drawn from the provincial population. SO2 Livelihoods and SO3 DRR and Gender activities only take place in the twenty-one collines designated as watershed collines, and the sample for the household survey to gather information on indicators for these components is only drawn from the population in these collines. Some of the respondents for the provincial￾level SO1 survey reside in the 21 collines, and the final evaluation team took the opportunity to examine differences between these SO1 participants and other SO1 participants outside of the twenty-one collines. The illustrative differences are described in Section IV.C.6 on page 51). Table 2. Quantitative Survey Baseline and Endline Samples Population Questionnaire Baseline Sample Endline Sample Comment Planned Actual Province SO1 Only for both Baseline and Endline 904 Children & 616 mothers 1,050 children & 1,050 mothers 1,246 children & 1,015 mothers “Mothers” and “households” can be used interchangeably. The number of children are those under age 5 in sampled households for whom anthropometric measurements were taken. Targeted Watersheds SO2/SO3 for Baseline & SO1/SO2/SO3 for Endline 605 Households 580 households 577 households An anthropometric survey of children under age 5 was not included in the watershed survey since the focus was on SO2 and SO3. Information from the comparison of baseline and endline household surveys has been incorporated in Sections II and IV. A detailed description of the sampling metholodology that was initially proposed and actually used in the household survey is provided in Annex B. The only difference between the plan and actual implementation of the survey was that ISTEEBU was able to obtain lists of residents from local authorities for each village to serve as a sample frame for selecting households for the survey, rather than having to use the “pen” method as described in the plan. ISTEEBU indicated that there were no other significant changes, challenges or limitations which occurred during data collection for the household survey. The original baseline data was provided to ISTEEBU, and they recalculated baseline estimates using current methods in order to ensure more valid endline-nbaseline comparisons. In some cases, the ISTEEBU baseline estimate differed substantially from the reported baseline estimate. Both estimates are shown in the tables and a more detailed explanation is provided in Section IV.C.1 on page 44. For endline-baseline comparisons of estimates, ISTEEBU used a t-test for determining statistical significance. The variances of estimates were not adjusted to include the design effect. A MYAP Final Evaluation Report 31 July 2012 8 Levenes's test was conducted, however, to confirm equality of variances and use of the proper t￾test statistic for endline-baseline comparisons. The p-value for the 95% confidence interval3 is shown in tables indicating baseline and endline estimates on indicators, beginning with Table 15. In some cases, as indicated in the tables comparing endline to baseline estimates, it was not possible to do the comparison due to the way that the question posed at the baseline did not include several responses that were reported in the endline survey, which prevented a valid comparison. 3. Evaluation Limitations. Baseline data was collected from 9 to 22 December 2008, and endline data was collected from 14 to 21 March 2012. This has implications for some of the interpretation, since the period in December represents a time at which food insecurity is not acute, while March is a period of more significant food insecurity. III. ACTIVITIES AND OUTPUTS PRODUCED UNDER EACH STRATEGIC OBJECTIVE A. Outputs Produced Under Strategic Objective 1: Human Capacities/Maternal & Child Health and Nutrition There are four intermediate results (IR) under SO1 as shown in Table 3. Respective outputs and activities will be presented and discussed in the following sections. While Behavior Change Communication (BCC) is listed as Activity 1.1.1.1 (under output 1 of IR1) and the training and Table 3. Logic Structure for the MCHN Component of the MYAP GOAL: By 2011, vulnerable households in the provinces of Kayanza, Kirundo and Muyinga have reduced their chronic and transitory food insecurity. STRATEGIC OBJECTIVE 1: Vulnerable households have enhanced human capacities. Intermediate Result 1.1: Households practice optimal infant and child feeding practices. Output 1.1.1: Community-based approaches to encourage improved infant and young child feeding practices are established. Output 1.1.2: Pregnant women and children under five have access to routine, comprehensive growth monitoring and coaching Output 1.1.3: Community Management of Acute Malnutrition (CMAM) is provided in target areas. Intermediate Result 1.2: HIV-affected households consume sufficient food for optimal nutrition. Output 1.2.1: HIV-infected individuals on ARTs have access to appropriate supplemental food. Output 1.2.2: HIV-affected households have knowledge of proper diets. Intermediate Result 1.3: Households practice good health-seeking behavior. Output 1.3.1: Vulnerable households have knowledge to prevent common childhood illnesses (diarrhea, malaria, ARI). Output 1.3.2: Vulnerable households respond appropriately to common childhood illnesses (diarrhea, malaria, ARI). Intermediate Result 1.4: Households use appropriate hygiene and sanitation practices. Output 1.4.1: Optimal hygiene and sanitation practices are promoted. Output 1.4.2: Potable water is restored to communities 3 Numerous requests were made to ISTEEBU to provide more specification than probability values for the 95% confidence interval as shown in the tables, but ISTEEBU was unable to provide more specificity without substantial additional investments of time for capacity building and resources. MYAP Final Evaluation Report 31 July 2012 9 equipping of Volunteer Community-Based Health Activists (VCBHA4 ) as Activities 1.3.1.1, 1.3.2.1 and 1.3.2.2 under the first and second outputs of IR3, these can be considered as cross￾cutting strategies needed to achieve the four IRs and are hence discussed first. 1. Cross-Cutting Strategies (IR1.1 and IR1.3). a. Behavior Change Communication (BCC). During the MYAP’s first year, IMC conducted a Knowledge, Attitudes and Practices (KAP) survey and barrier analysis to inform the BCC strategy. BCC themes were chosen accordingly: importance of antenatal and postnatal care (ANC and PNC); Infant and Young Child Feeding (IYCF) practices; food groups; promotion of child growth; prevention of childhood disease and health seeking behavior; hygiene and sanitation; and child development. IMC devised teaching aids in close consultation with the MoH and the Provincial Health Bureau (Bureau Provincial de la Santé or BPS). Dissemination channels were agreed upon and relevant documents produced, such as image boxes and training modules for health staff and volunteers (VCBHA, Lead Mothers in the Mother Care Group approach, and “Mères Lumières” in the PD/Hearth approach). In addition to community sensitization through meetings and individual household visits, the use of mass media was foreseen in the BCC strategy. This was implemented in 2011: IMC signed a contract with the national radio station (RTNB) to broadcast health and nutrition messages, which they developed in collaboration with the “Programme National Intégré d’Alimentation et de Nutrition (PRONIANUT). The Mid-Term Evaluation (MTE) recommended putting more emphasis on “practice” such as promoting best childcare when a child is ill, and preventing malaria through the use of treated mosquito nets. In response to this recommendation: a) IMC revised the BCC communication techniques to include more role-play, educational chats and demonstrations; and b) conducted refresher training of VCBHAs with emphasis on the management of childhood illness. During its field visits to various facility and community￾based activities, the evaluation witnessed twelve BCC sessions run by volunteers or health staff. All VCBHAs, Lead Mothers (LM) and “Mères Lumières” (ML) had their own copy of the respective manual. Except for a couple of cases, the visual aids provided by IMC (plasticized sheets on various topics) were used. In general, BCC sessions were interactive: volunteers (VCBHA, LM and ML) and health staff solicited responses and comments from participants. Volunteers reported that topics were selected at their monthly meetings in consultation with provincial health promoters (“Techniciens pour la Promotion de Santé” – TPS) and community leaders. Discussions with mothers/caregivers suggest that the MYAP BCC strategy was effective in terms of improvements in health-seeking as well as care-providing behavior. Interviewed participants displayed good knowledge and understanding of the messages and reported practicing them, but sometimes were unable to do so because of insufficient/lack of financial resources (such as 4 VCBHA are known in Burundi as “Agent de santé communautaire (ASC)” or “relais communautaire” BCC Materials MYAP Final Evaluation Report 31 July 2012 10 buying soap or building a latrine). KAP survey results will be discussed under Section 3 IR 1.3 Health Seeking Behavior below. The BCC strategy was delivered as intended and relevant adjustments were made in response to MTE recommendations and annual KAP surveys. Training materials for volunteers and health staff are adequate and respond to national/local priorities as they were based on KAP surveys and were developed in close collaboration with PRONIANUT, the MoH department in charge of nutrition. b. Training of Volunteer Community-Based Health Activists (VCBHA). The VCBHA role is to disseminate knowledge among the community, and act as links between their communities and health facilities. Altogether, IMC and BPS trained 1,440 VCBHAs and provided them with kits including backpacks, t-shirts, Mid-Upper Arm Circumference (MUAC) tapes, umbrella and stationary (Activity 1.3.2.2). The t-shirts, backpacks and umbrella include the logo of the MYAP donor (USAID) and consortium NGOs (CRS and IMC). A 2-days refresher training on the management of childhood illnesses, namely the prevention of malaria, diarrhea and Acute Respiratory Infections (ARI) was provided to 761 VCBHAs in 2011 and 659 in 2012 (Activity 1.3.2.1). IMC replenishes tools to VCBHAs for BCC and monthly reporting. On average, each VCBHA conducts one to two group sensitization activities per month with an average of 19 participants per session. Data on VCBHA’s activities from MYAP’s Monitoring and Evaluation (M&E) system are summarized in Table 4. The Mid-Term Evaluation recommended finding a means to encourage VCBHAs through a quarterly award system. In response to this recommendation, IMC developed a revised BCC strategy focusing on household-level improvements, whereby a “Social Contract” sets a mutual commitment between the volunteers (Lead Mothers, VCBHAs) and the target beneficiaries. The volunteer commits him/herself to provide all the necessary skills to the beneficiaries to perform a specific practice and the beneficiary will in turn commit him/herself to adopt and implement the practice within a given period of time. In the last quarter of 2011, 42 VCBHA from the hills where the Star Awards activity took place benefited from kits (a pack of laundry soap and hoe). Table 4. Summary of VCBHA Activities. Activity 2009-2010 2010-2011 Training/Refresher Training 1,400 671 Number of Sensitization Sessions 27,020 17,616 Beneficiaries Reached through Group Sensitization Sessions 512,287 618,517 Number Home Visits 151,651 224,465 During its field visits to health facilities and community-based activities, the evaluation team had the opportunity to observe VCBHAs while conducting BCC sessions or taking anthropometric measurements; and held a focus group discussion with VCBHAs (two women and twelve men) in Muyinga. As also noted in the MTE report, VCBHAs displayed very good practice of anthropometric measurements and adequate communication skills. They reported often working more than the planned two days a week for MYAP-related activities as well as working for other health programs (such as immunization or malaria campaigns) for which they received in-kind or cash motivation. They were generally satisfied with the positive changes they were witnessing in MYAP Final Evaluation Report 31 July 2012 11 their community, such as a decrease in the cases of Severe Acute Malnutrition (SAM) and diarrheal diseases. In their opinion, the main challenge in changing behaviors is latrine construction, which they said was too costly for most vulnerable households. In conclusion, VCBHA training was delivered as intended and relevant adjustments were made in response to MTE recommendations. 2. IR 1.1 Optimal Infant and Young Child Feeding Practices a. Community-Based Approaches to Encourage Improved IYCF Practices i. Mother Care Groups. The Mother Care Group (MCG) approach aims at behavior change using peer-to-peer communication. Mothers whose households demonstrate positive health indicators (such as sanitation and hygiene, healthy children, etc.) are elected as Lead Mothers (LMs) during open community meetings organized by IMC to explain the purpose of the MCG approach and seek approval of local community leaders. LMs are trained by MoH/TPS and IMC health promoters for two days, and are given reporting forms. The plan was to implement the MCG approach in the watershed areas where MYAP SO2 and SO3 activities are implemented by CRS. In 2010, 882 LMs were trained. In 2011, IMC and CRS decided to extend the MCG approach to additional hills of the watershed area: 1004 new LMs were trained (238 in Kayanza, 331 in Kirundo and 435 in Muyinga); 96 new MCGs were added to the previous 87 groups, making a total of 183 MCGs. That same year, 97 Savings and Lending Committees (SILC) groups were established among MCGs; and a monthly star campaign was introduced and implemented jointly by IMC and CRS. The evaluation attended three LM meetings (one in each province). LMs reported having witnessed significant changes in health-seeking behavior in their communities, such as: a) more frequent prenatal consultations and health-assisted deliveries; b) increased and more regular usage of mosquito nets; and c) decrease in cases of malnutrition. They believe that improvements in nutrition are sustainable, as themselves and the mothers/caregivers they are mentoring, have learned to better use locally available foods. However, they reported that extremely vulnerable households may not have the resources to purchase the foods that are being recommended to them to use for their young children, or are unable to afford constructing a latrine. All interviewed LMs have joined SILC; most of them took a loan to undertake Income-Generating Activities (IGA), but some also reported taking a loan to cover urgent medical expenses. Interviewed stakeholders, such as MoH and UNICEF, were aware and supportive of the MCG Lesson Learned The following are key success elements that were applied and ought to be replicated in future BCC activities:  Alignment with national stategies: consultation with MoH/PRONIANUT  Use of KAP survey results to develop/adjust the BCC strategy  Multipronged implementation process (facilty and community-based)  Community consultation in the selection of BCC topics  Building on an existing network of VCBHA, whose role in health and nutrition is already well established and supported in relevant MOH strategies and protocols  Introducing “Star Awards” which is in line with current thinking in Burundi on the integration of VCBHA in the Performance Based Financing (PBF ) MYAP Final Evaluation Report 31 July 2012 12 approach, however no specific reference to it was found by the evaluation in new key documents such as the National Nutrition Policy (validation in progress). In conclusion, MCG was delivered as intended and relevant adjustments were made in response to MTE recommendations. The MTE recommended: a) clarifying/reviewing the purpose of the MCG approach, b) scaling up the number of MCG; and c) conducting action research into a “Father Care Group” that would serve as role models for husbands and fathers. The evaluation noted the following:  Clear and explicit IMC documentation is available on the purpose and implementation strategy of the MCG5 .  The number of MCG increased within and outside the watershed areas  Fathers’ involvement was addressed by IMC under the PD/Hearth approach (discussed in the following section) Although designed as an intervention in its own right, and so far not officially adopted by the MoH, the MCG approach is overall consistent with the VCBHA concept and precepts and this should be highlighted by CRS and IMC in their discussions with the MoH to ensure the sustainability of the MCG approach beyond MYAP. Its integration into national strategies might be, for example by specifying that one of two VCBHAs per “sous-colline” should be a positive deviant mother with small children6 . ii. Positive Deviance/Hearth. Positive Deviance (PD/Hearth), known in Burundi as “Foyers d’Apprentissage et de Réhabilitation Nutritionnelle” (FARN), aims at rehabilitating children with moderate malnutrition at the community-level using locally produced/available foods. It involves identifying mothers of well-nourished children and training them to become “Mères Lumières (ML)” to run BCC and meal preparation sessions for a group of 10 to 12 mothers of moderately malnourished children, for a total of 12 days. PD/Hearth sites are to be established where malnutrition prevalence exceeds 30%. This approach is an integral component of the MoH Community-Based Nutrition Program and is included in the national protocol for the management of malnutrition released in 2010. In 2010-2011, PRONIANUT/UNICEF/Pathfinder organized workshops, with the participation of IMC and other stakeholders, to develop a unified PD/Hearth Manual (validation in progress). During the first year, IMC staff attended a PD/Hearth training held by Pathfinder in order to build capacity of its BCC supervisors in this approach. A training module and a manual were developed, and training of LMs for 2 days by MoH/TPS and IMC/BCC supervisors started in late 2010. The PD/Hearth approach is implemented in Kirundo and Kayanza, as Pathfinder supports 5 See IMC Powerpoint Presentation “Approche care group à Muyinga, Kirundo et Kayanza » 6 The mission has no information about the gender balance among VCBHAs, but there were more males VCBHAs in the sites visited by the mission, and more men participated in the focus group discussion in Muyinga PD/Hearth Feeding Session MYAP Final Evaluation Report 31 July 2012 13 PD/Hearth activities in Muyinga. In 2010, 21 PD/Hearth sessions were completed in Kayanza and 13 in Kirundo; and in 2011: 95 in Kayanza and 26 in Kirundo7 . As summarized in Table 5, MYAP M&E data indicate that the PD/Hearth approach was effective with 87% of children gaining more than 200g after 12 days. A follow-up study among 438 children, one month after the end of the 12-days PD/Hearth session, revealed that 84% of them had maintained their weight gain. Table 5. Performance Indicators for PD/Hearth Indicator (weight gain) 2010-2011 Follow-up Study Number of children Percent Recovery Rate Number of children Percent Recovery Rate Inadequate growth (<200 g) 258 13 71 16 Adequate growth (200-400 g) 905 47 87% 205 47 84% Faster growth (> 400 g) 778 40 162 37 TOTAL 1941 100 538 100 The evaluation visited three PD/Hearth sites (two in Kirundo and one in Kayanza). The site chosen for the sessions (usually the home of the ML) is an occasion for participating mothers to witness good hygiene practices, such as shelves for kitchen utensils. In addition, meal preparation and child feeding give the opportunity for mothers to practice lessons learned on IYCF such as crushing and mixing the food to ensure consumption by the child of a balanced meal containing all recommended food groups, as well as on hygiene such as hand washing with soap (although in one instance this was incorrectly done). As mentioned under the MCG section, in response to the MTE recommendation to increase fathers’ involvement in nutrition activities, IMC chose to invite fathers to PD/Hearth sessions. In one site visited by the evaluation, two fathers were present at the 12th session when children were weighed and resutls on weight gain are shared among participants. IMC provided support, either as food or cash to purchase locally available food items, to complement the foods brought by mothers. Participating mothers reported that contributing to the meals was difficult for most of them due to poverty. In conclusion, PD/Hearth was implemented as planned (there were no specific MTE recommendations on PD/Hearth). 7 In 2011, workshops were organized jointly by PRONIANUT/UNICEF and Pathfinder with all stakeholders (IMC as well as other NGOs implementing the FARN approach) to harmonize and standardize the FARN methodology. IMC plans to resume the expansion of FARN after validation of he modified FARN methodology planned in April 2012. MYAP Final Evaluation Report 31 July 2012 14 b. Growth Monitoring i. Facility-Based Growth Monitoring. Growth Monitoring (GM) aims at supporting preventive care for children under 5 by combining GM with BCC. Under this activity, IMC trains health staff on anthropometric measurements, BCC, counseling mothers, and referral of malnourished children to CMAM programs when required. IMC, in collaboration with PRONIANUT, developed a register and distributed health cards and monitoring tools to health facilities. Health cards include a growth chart, simplified messages on IYCF and home management of childhood illnesses, and a simplified immunization and Vitamin A supplementation schedule. Interviewed health staff reported that the supply of growth charts was however insufficient. In the first year, IMC trained 94 MoH staff, one nurse and one nurse assistant from each health center; and in 2010, 96 nurses and 127 nurse assistants. In addition to these trainings, joint IMC/MoH supervisions provide opportunities for on-site staff coaching. In the 2nd year, health centers conducted 624 growth monitoring sessions for 59,038 chldren registered for growth monitoring sessions, with 662 children transferred to CMAM programs (3 to stabilization centers, 165 to outpatient therapeutic feeding and 494 to SFP). In the 3rd year, health centers conducted 624 growth monitoring sessions for 28,787 children registered for growth monitoring sessions, with 1,172 children transferred to CMAM programs (10 to stabilization centers, 326 to outpatient therapeutic feeding and 836 to Supplementary feeding) An award system was introduced: 4,292 mothers who brought their children to the GM sessions for three consecutive sessions received long￾lasting insecticide treated nets (LLITN) as motivation. The evaluation visited two health centers on GM days. In Kirundo, GM is scheduled on two days a week together with immunization, while in the other two provinces GM is not combined with immunization. The evaluation witnessed a very good practice in one health center in Kirundo, whereby growth charts are filed in boxes (one for each month and one for drop-outs). This system, which is in fact a “due date filing system”, allows adequate follow-up of children: defaulters are easily identified and VCBHAs are asked to conduct home visits to the concerned households. In conclusion, support to GM was delivered as intended (no specific recommendation was given in the MTE report). Lesson Learned PD/Hearth was effective in adressing moderate malnutrition (as shown by IMC M&E data). Its expansion is therefore justified and ought to be completed during the remaining months of MYAP while considering the following:  Continue to encourage fathers’ presence at least in the 1st and last PD/Hearth sessions  Discontinue the financial or in-kind contribution to the meals as this is unlikey to be taken over by MoH with national resources (unless this practice is endorsed in the new national guidelines)  Introduce SILC, in line with the MoH new nutrition policy to “gradually integrate rural development programs aimed at improving food security in the FARN intervention areas”. Growth Monitoring MYAP Final Evaluation Report 31 July 2012 15 As IMC and CRS have been forerunners in the area of GM, they should take the lead in organizing - jointly with PRONIANUT - a workshop involving all nutrition stakeholders to share experiences and harmonize GM (as done for PD/Hearth). ii. Community-Based Growth Monitoring. Lack of a clear national protocol on community-GM has delayed implementation of this activity. However, MUAC measurement techniques are included in the VCBHAs training modules. VCBHAs trained by IMC are therefore able to participate in community-based GM when such a program is launched. The workshop mentioned above would be also an opportunity to discuss/agree upon community-based monitoring. c. Community Management of Acute Malnutrition (CMAM) Under this output, support is provided to inpatient care in stabilization centers (SST), outpatient therapeutic programs (OTP) and SFPs. These three CMAM components are interrelated as shown in Figure 18 . This figure also provides information on admission and discharge criteria. Figure 1. CMAM Decision Tree and Referral System There is one SST in each provincial hospital, while OTP and SFP are run in health centers. Key observations and lessons learned which apply to all three components (such as BCC and 8 Modified version of the flow chart included in the National Protocol Lesson Learned The following are key success elements that were applied and ought to be replicated in future growth monitoring activities:  Expand the use of the due date growth chart filing sytem used in Kirundo  Combine GM with immunization While there are pros and cons as to whether or not to combine GM with immunization, preference should be given to the Kirundo practice – combining GM with immunization – as this is more in line with the national protocol which advocates for integrating nutrition activities into the “Paquet Minimum d’Activités” at health centers and more sensitive to women’s high workload. MYAP Final Evaluation Report 31 July 2012 16 anthropometric measurements), as well as responses to MTE recommendations are discussed jointly for the three components under section iv. Food rations are described in Table 6. The ration for pregnant and nursing women which consists of Corn Soya Blend (CSB), Soy-Fortified Corn Meal (SFCM) yellow peas, and oil, and that of People Living with HIV/AIDS (PLWHA), which consists of CSB, bulgur, yellow peas and oil, are in line with those recommended in the national protocol. The national protocol recommends an additional 25g of sugar for children and 200g of salt for women in SFP and no rations for caregivers accompanying children admitted in SST. Table 6. Daily Ration Scale and Composition (grams per person per day) Distribution Recipient Title II Commodities Non CSB VegOil SFCM Bulgur Title II Yellow Peas SFP Children 120 15 Women 100 15 100 3g salt OTP Children Plumpynut SST Caregivers 50 25 330 100 5g salt PLWHA9 PLWHA 120 25 280 100 15g sugar 750g salt Note: Plumpynut is provided by UNICEF. i. Stabilization Centers. UNICEF provides the commodities required for the rehabilitation of severely malnourished children10. CRS and IMC provide the commodities for the accompanying caregivers and various supplies (such as MUAC tapes, blankets, mosquito nets, weighing scales, height boards, and plastic containers). Technical guidance (training of staff and supervision) and food rations began in Muyinga and Kayanza SST in February 2009. The Kirundo SST was included later in 2009 after withdrawal of Doctors without Borders (MSF). Family food rations were discontinued in January 2011 in compliance with the new national CMAM guidelines. As the default rate increased after the discontinuation of rations to caregivers, CRS and IMC decided to resume ration distribution in March 2011. M&E data are summarized in Table 7. In 2010-2011, performance indicators improved. Food rations to caretakers have most likely contributed to the decrease in the default rate. The death rate remains high, however, exceeding the critical cut-off point defined in the national protocol. According to information provided by the program, the high case fatality rate among children 9 The ration for PLWHA is a family ration consisting of 5 individual rations 10 Plumpynut is a commercial brand of a Ready to Use Therapeutic Food (RUTF). F-75 is a formula diet used during the initial phase of treatment of SAM. F-100 is a formula diet used during the rehabilitation phase after appetite has returned. ReSoMal is a powder used for the preparation of an Oral Rehydration Solution (ORS) Stabilization Center MYAP Final Evaluation Report 31 July 2012 17 admitted to the SST is due to various factors including staff relocation (new appointed staff not adequately trained in the management of severe acute malnutrition with complications as observed in Kirundo by the evaluation), concomitant severe illness such as HIV/AIDS, and late submissions (caregivers bringing their children too late to the SST). Table 7. Performance Indicators for Stabilization Centers (2010 and 2011) Year Total Admissions Recovery Rate Default (Drop￾Out) Rate Non￾Response Death Rate 2 nd 1109 83% 7% 2% 8% 3 rd 1328 88.8% 3.8% 0% 7.3% National Target >75% <15% <15% <5% The three SSTs are well equipped, and staff were generally knowledgeable about CMAM. Implementation was constrained, however, by breakdowns in the supply of F-75 which was hence replaced by diluted F-100. Diluted F-100 is not considered an appropriate alternative to F￾75 because of its higher protein, sodium, and lactose contents11. No recent breakdowns were reported for commodities provided by CRS for accompanying caregivers. Health staff reported that children were mostly referred by health centers or VCBHAs. Re-admissions are occasional. In general, children were successfully discharged to OTPs after about 10 to 14 days, as recommended in the national protocol. In conclusion, the Consortium’s technical and food support have contributed to improvements in SST’s performance. However, improvements in the recovery rate are dependent on several factors that are beyond MYAP’s realm, such as the quality of medical care of concomitant health complications, and the adequate and timely supply of therapeutic foods (F-75 and F-100). ii. Outpatient Therapeutic Program (OTP). The evaluation visited three health centers during OTP sessions. IMC’s main input to OTP is technical and material support (such as the training of health staff, provision of MUAC tapes, and transport of Plumpynut), and supervision. Initially a ration was provided by CRS as an incentive for mothers to bring their malnourished children to the OTP, however this was discontinued in January 2011 in line with the national protocol. Except for one instance, health centers had adequate equipment. UNICEF supplies Plumpynut to OTP for the rehabilitation of children. Health staff reported having faced disruptions in Plumpynut supply. According to UNICEF, with whom the evaluation raised this issue, disruptions are not only due to a supply problem but also to “overuse/misuse” of Plumpynut (unjustified extension of entitlement period). The evaluation cannot confirm this observation. In 2009-2010, support was provided to 45 OTPs (9 in Kayanza, 21 in Kirundo, and 15 in Muyinga). By the end of August 2011, all planned OTPs were functional with minimum IMC supervision (16 in Kayanza, 21 in Kirundo and 15 in Muyinga). MYAP M&E data are summarized in Table 8. In 2011, performance indicators improved. The defaulter rate decreased (from 11% in the 2nd year to 6% in the 3rd year); and the non-response and death rates remained relatively the same and are both below the critical cut-off point defined in the national protocol. 11 http://imtf.org/page/feeding/ MYAP Final Evaluation Report 31 July 2012 18 Table 8. Performance Indicators for Outpatient Therapeutic Program (2010 & 2011) Year Total Admissions Recovery Rate Default (Drop￾Out) Rate Non￾Response Death Rate 2 nd 7968 83% 11% 6% 1% 3 rd 9337 85.6% 5.7% 7.3% 1.2% National Target >75% <15% <15% <5% In conclusion, the MYAP’s support to OTP is mainly technical and material. OTP is expected to remain after MYAP ends as it is an integral component of the national CMAM strategy. OTP continued good performance will depend on the regular and timely supply of Plumpynut by UNICEF and its timely transport to health centers by MoH. The main challenge will be for health staff to adhere very strictly to admission criteria and entitlement duration of OTP in the face of a likely increase in the number of eligible beneficiaries after the SFP stops with the end of the MYAP (as untreated moderately malnourished children could become severely malnourished). iii. Supplementary Feeding Program. SFP is targeted to moderately malnourished children under five years of age and moderately malnourished pregnant and nursing mothers. Children receive a dry weekly ration consisting of a premix of Corn Soya Blend (CSB) and oil for a total of 2 weeks; and mothers receive a ration consisting of maize meal (MML), beans, CSB and vegetable oil. By the end of August 2011, SFP were running in 50 health centers (14 in Kayanza, 21 in Kirundo, 15 in Muyinga). These were able to run SFP services with minimum supervision from IMC. MYAP M&E data are summarized in Table 9. Throughout the life of the MYAP, performance indicators were within national targets and actually improved in 2010- 2011. Table 9. Performance Indicators for the Supplementary Feeding program (2010 & 2011) Year Total Admissions Recovery Rate Default (Drop￾Out) Rate Non￾Response Death Rate 2 nd 15,925 79% 11% 9% 1% 3 rd 18,536 86.1% 5.6% 7.8% 0.46% National Target >75% <15% <15% <5% The evaluation visited four health centers during SF sessions (one in Kirundo, one in Muyinga and two in Kayanza). Beneficiaries, under the supervision of health staff and VCBHAs, participate in the preparation (mixing CSB and oil) and distribution of commodities. Posters showing ration sizes were displayed in all health centers visited by the evaluation. Supplemental Feeding Supplementary Feeding MYAP Final Evaluation Report 31 July 2012 19 iv. Observations. Except for one instance where young children were weighed using an adult beam scale, anthropometric equipment was adequate (hanging spring scale) and regularly calibrated, and measurements were taken in accordance with best practice. Health staff knowledge and practice in nutrition was very good. All interviewed staff had been trained and were well￾versed in the objectives and implementation procedures (such as admission and discharge criteria) of OTP and SFP. However, as mentioned by IMC and confirmed by Provincial Nutrition Focal Points, the high turnover rate among health staff due to frequent relocations is the main challenge facing CMAM and GM implementation. In two health centers, the organization of activities and allocation of tasks among health staff and VCBHAs allowed individual counselling in spite of the high number of beneficiaries. However in other health centers, assessment of the child’s nutritional status was done after all children were weighed and heights measured. It is only after all measurements are taken, that staff review their records and assess whether a child qualifies for supplementary feeding using the World Health Organization (WHO) weight-for-height reference table from the national protocol. This sequencing of activities does not allow immediate feedback to the caretaker on the nutritional status of the child. In most cases, health staff were overwhelmed and did not have time for any individual counseling, not even informing the caregiver whether the child gained weight or not or whether he/she will be enrolled in the OTP or SFP. In all health centers, women/caregivers and the children spend long hours at the health center (from around 7:00 am to about 2 to 3:00 pm). The MTE recommended that IMC start having less “hands-on” role and more of coaching or monitoring of health center staff. During its visits to the various health facilities, the evaluation observed that IMC supervisors were not engaged in implementation. During focus group discussions, IMC staff confirmed being more involved in supervision and coaching. In conclusion, CMAM was delivered, as intended and relevant adjustments) were made in response to changes in national guidance (such as ration size). IMC responded to the MTE recommendation by instructing CMAM supervisors to cease their involvement in implementation of activities and focus on supervision and coaching. High health staff turnover is a major challenge facing CMAM (as well as GM). Lesson Learned The effectiveness of CMAM is contingent on a combination of managerial, technical and social factors:  Early identification and referral of suspected cases  Capacity-building of health care providers at all levels and regular monitoring /on-site training  Continuous availabilityof therapeutic food products (F75, F100. RUTF) and medical kits  Existence of adequate referral arrangements  Links with other programs with preventive initiatives such as promotion of good infant and young child feeding practices Health Center Services MYAP Final Evaluation Report 31 July 2012 20 2. IR 1.2 HIV-Affected Households a. Appropriate Supplementary Food. Supplementary food is provided to moderately malnourished People Living with HIV/AIDS (PLWHA) in Kirundo province. PLWHA receive a monthly individual supplementary ration upon initiation of Antiretroviral Therapy (ART) treatment. In 2011, 550 beneficiaries were enrolled in the program. PLWHA participants benefit from food assistance for twelve months. They are discharged if they meet certain criteria, including meeting a weight threshold. If they do not meet this threshold, they are re-enrolled in the program along with other new articipants thathave been idenfitied. Initially, the MYAP proposed to distribute individual rations in al three provinces, but by the time implementation started, the protocol had changed to a ration for five people, necessitating reducing the geographical coverage for rations to only Kirundo. In consultation with the World Food Program (WFP), HIV/AIDS associations and USAID-Burundi, it was decided that Kayanza and Muyinga would be covered by WFP. The MYAP is collaborating with three associations that manage food distribution, facilitate provision of ART, and provide counseling and day-to-day social and material support. The evaluation had meetings with two of them: the “Association Nationale de soutien aux Séropositifs et Malades du SIDA (ANSS)” and the “Réseau Burundais des Personnes Vivant avec le VIH/SIDA (RBP+)”. Both provide free health care, social care (various types of assistance according to the specific needs of PLWHA members, such as household utensils, clothing or funeral expenses) and psychosocial assistance (counseling on various health issues such as family planning and voluntary testing for family members). No food distribution for HIV/AIDS was planned during the evaluation’s visit in Kirundo (11 to 16 March). However, the evaluation had a focus group discussion with a group of six previous and six current PLWHA food aid beneficiaries. They confirmed having been selected during community meetings in the presence of IMC and representatives from the implementing associations, and were well aware that selection was based on socioeconomic criteria. They also confirmed that their nutritional status was monitored monthly at the time of food distribution. Current beneficiaries reported gaining weight and being able to take their drugs without side effects, while discharged beneficiaries reported having difficulty maintaining their weight. b. Nutrition Education for PLHWA. Information on proper nutrition is provided to PLWHA at ration distribution points. Topics were chosen based on the findings of a dietary diversity survey of PLWHA undertaken in October 2009. Interviewed beneficiaries reported having received information on nutrition (how to prepare balanced meals using the food ration commodities) and health (such as family planning). In conclusion, support to PLWHA was delivered as intended and adjustments were made as required (no recommendation was given in the MTE report regarding the ration or nutrition education). c. Keyhole Gardens/Vegetable Production. In the first few years of the MYAP, the program provided vegetable seeds to 2319 PLWHA in the three provinces and agricultural extension agents were trained to provide assistance. After a CRS-wide training on keyhole gardens in Lesotho in late 2011, the project embarked on a Key Hole Gardens MYAP Final Evaluation Report 31 July 2012 21 promotion and dissemination campaign through the SILC groups. These small but highly productive vegetable gardens are built close to the house for easy maintenance and harvesting. They are usually planted with carrots, onions and a tropical spinach, cost nothing to build and should last about 5 years. The interest and demand is strong with many beneficiaries already collecting and saving the required sticks and ash. Beneficiaries expect that two keyhole gardens per household should suffice. 3. IR 1.3 Heath-Seeking Behavior BCC is an integral component of the preceding immediate results and outputs. The evaluation interviewed all categories of beneficiaries (caregivers of children in GM, SST, OTP and SFP, as well as PLWHA) and volunteers (VCBHA, LM and ML) regarding health-seeking and health￾doing behavior, and findings were presented in the relevant preceding sections. The Consortium has been documenting changes using Knowledge, Practices and Coverage (KPC) surveys in the Provinces of Kayanza, Kirundo and Muyinga. Three such surveys were conducted between 2009 and 2011. The last survey conducted in 2011 revealed that the majority of households recognized are at least two signs of disease (93.3%). Among women whose children had diarrhea within 15 days, 53.2% used ORS as recommended by the strategy of Integrated Management of Childhood Illness (IMCI). Seeking care from a health facility for diarrhea, cough and fever is fairly common and is practiced by more than two thirds of women surveyed. Possession of a mosquito net is relatively high 62.2% of households. In 2011, International Medical Corps identified and selected five households (per month) whose members had considerably improved the hygiene and sanitation conditions of their home environment. Through collaboration with CRS to include performace on agriculture-relaed activities, 60 households were awarded Star Award prizes in the presence of communities and local authorities. Prizes consisted of 20 liter jerrycans, hoes, T-shirt, machetes, cups, cans, boxes of soap and buckets with lids. In conclusion, BCC to improve households' health-seeking knowledge and behaviors was delivered as intended and relevant adjustments were made in response to the KPC surveys. 4. IR 1.4 Hygiene and Sanitation a. Water Point Rehabilitation. The MYAP proposed to rehabilitate existing potable water points (protected springs and pipeline conveyance systems), using Food for Assets/Cash for Work and community participation. Water management committees would be mobilized and trained to be responsible for general operations and basic repairs of water point, organizing systems for keeping water points clean and formulating and monitoring community rules for Lessons Learned Two lessons emerged relative to implementation of activities for PLWHA in the MYAP.  Integration of food assistance into a comprehensive medical and psychosocial package of services results in a multiplier effect of benefits from all components of the package.  Without additional income, such as from a new IGA, to enable households to procure food, the impact of the food provided by the MYAP will likely decline after the program ends. MYAP Final Evaluation Report 31 July 2012 22 using the water point. The program targeted 32 sites over the life of the program and expected 1,350 vulnerable people per site to participate in FFW/CFW. At the time of the evaluation, a total of 40 spring boxes (protected springs) had been rehabilitated or constructed with a focus primarily on improving water quality. An additional five sites are planned to be completed before the end of the program in Kirundo. For each site, a water committee has been formed with between two and six members, and most committees have been trained by IMC in basic operations of the water point, cleanliness around the water point and the PHAST methodology. There has been very little community contribution towards the rehabilitation of the water points. In some locations, the community brought some stone or sand but it was not enough, and the program paid to have more materials brought. Skilled labor was brought and paid for by the MYAP, and unskilled labor was provided by the community. These participants, usually less than ten per site according to those interviewed, were paid cash by the program as Cash for Work. The program did not use Food for Work, but water point rehabilitation typically does not require a large amount of unskilled labor. The technical quality of the rehabilitation is generally good, although the evaluation team could not actually see the filter system installed since it is buried. In five sites visited, good volume, good quality and good continuity (year-round supply) were observed. In one site, the water flow was less than before the rehabilitation but the water quality and continuity were good. In two Expanded Quantity Poor Quality After a Heavy Rain (1 Site) Developing Leaks MYAP Final Evaluation Report 31 July 2012 23 sites, small leakages were evident, and these will likely expand if repairs are not made. In one site visited just after a heavy rain, the water quality was very poor12 . The process used by the water committee for keeping sites clean varied from province to province. In one province, community respondents indicated that “these people were paid by the program (payment of allowances to attend training), therefore they should keep the water point clean”. The Saturday morning community service is when the committee works, and they are not expected to work elsewhere. In another province, the committee organizes users of the water point to clean up the site weekly on a rotational schedule under their supervision. Either approach is working now. All sites visited were clean. The latter approach, however, reflects more dignity for the water committee and is more likely to be sustained. Relative to being able to repair water points when necessary, there is GIZ project that has been working on developing policies for guiding resource mobilization around water points, and until this is done, it has been difficult for the program to promote resource mobilization. Nevertheless, the capacities of the Water Committees built by the MYAP toward being able to repair water points varied from location to location. In one province, the Water Committees interviewed said that “we do not have authority to collect money to buy cement”. In another province, the Water Committee reported that “if we need money, we go with the Chef du Colline to the Commune water coordinator and get authorization (receipt) to collect money”. Part of the rehabilitation of water points by the MYAP involves building or restoring areas for washing clothes in ways to reduce threats of contamination. The program has been following the design recommended by government for these "wash basins". Among all nine sites visited, only once did the evaluation team see someone actually using the wash basins. Some basins appeared to be used, but others clearly had not been used with the most common reason given that people simply refer to use the plastic wash basins set on the ground. 12 Since the final evaluation, the MYAP is in the process of re-constructing this waterpoint to improve the water quality. Lesson Learned Community contributions are generally encouraged to cultivate ownership in water point rehabilitation. Community contributions were not used by the MYAP, however, yet water committees in most areas have a good sense of responsibility for doing simple operations and maintenance. The Commune Administration is perceived by most people in Burundi to have primary responsibility for water points. The MYAP engaged them in identifying sites for rehabilitation and then turned over the rehabilitated sites to them after the work was completed. In interviews during the evaluation, they were aware of the water points that had been rehabilitated but did not fell much responsibility for investing in repairs norhad repairs been initiated for leaking water points. More fully engaging the Commune Adminstration in rehabilitation throughout the process and not just at the beginning and end would likely have cultivated a stronger sense of ownership. Clothes Washing Area MYAP Final Evaluation Report 31 July 2012 24 b. Hygiene and Sanitation Education (PHAST). According to the established Participatory Hygiene and Sanitation Transformation (PHAST) methodology, two members of each water point committee receive training for a total of 12 days13. Trainees are expected to become models regarding the promotion of hygiene in their communities and are responsible for raising awareness about hygiene in the community; maintaining cleanliness around the water sources; and encouraging the presence of a shelf for utensils, a latrine and compost in each household. The evaluation attended two meetings with water committee members and community members. Most interviewees reported having constructed a latrine but mentioned that the cost of building a latrine is prohibitive for most households in their community. They believe that the rehabilitation of the water point together with the BCC sessions they attended have contributed to a decrease in diarrheal diseases among their children B. Outputs Produced Under Strategic Objective 2: Livelihoods Capacities The outputs proposed for each intermediate result under the livelihoods component of the program are shown in Table 10. Table 10. Logic Structure for the Livelihoods Component of the MYAP GOAL: By 2011, vulnerable households in the provinces of Kayanza, Kirundo and Muyinga have reduced their chronic and transitory food insecurity. STRATEGIC OBJECTIVE 2: Vulnerable households have enhanced and sustainable livelihoods capacity (sic). Intermediate Result 2.1: Vulnerable households have improved production in environmentally￾sustainable ways. Output 2.1.1: Vulnerable Households have increased knowledge of appropriate soil conservation techniques. Output 2.1.2: Vulnerable households have increased knowledge of appropriate water management techniques Output 2.1.3: Vulnerable households and farmer groups have increased their knowledge of improved and appropriate agricultural techniques. Output 2.1.4: Vulnerable households and farmer groups have knowledge of crop and livestock diversification. Output 2.1.5: Vulnerable households and farmer groups have improved their knowledge on the utilization of inputs. Output 2.1.6: Vulnerable households and farmer groups have better access to appropriate inputs. Output 2.1.7: Agricultural extension agents have improved their technical capacity to train and support farmers and farmers' groups in natural resource management. Intermediate Result 2.2: Vulnerable households have adopted strategies and techniques to diversify and increase revenues. Output 2.2.1: Vulnerable households have improved access to small scale credit and savings opportunities. Output 2.2.2: Farmers' groups have exploited local market opportunities. Output 2.2.3: Agricultural extension agents are using improved technical skills to train and support local farmers in appropriate agricultural techniques. 13 Nshimirimana J-C (2010) Rapport de la l’Atelier de Formation sur la Méthodologie PHAST – Muyinga-Gashoho. CRS/IMC/USAID Report Constructed Latrine Constructed Latrine MYAP Final Evaluation Report 31 July 2012 25 Building on CRS’s previous experience in food security activities, the MYAP largely achieved its SO2 revised targets and its objective of encouraging environmentally friendly rural economic growth by increasing farmers’ knowledge, access to improved varieties and improving their technical skills, leading to improved production, consumption and sales. Food-for-Assets activities were used to build anti-erosion berms in the hillsides and water management infrastructure in the lowlands (marais). The project addresses well some parts of the National Agricultural Strategy14, particularly for: increasing the technical capacity and the production of subsistence farmers in order to increase food security and reduce rural poverty, reducing imports paid in hard currency with increased production of beans and rice, the promotion of small livestock (for protein, for cash and for increased organic fertilizer), the protection and better utilization of watersheds (agro￾environment), the promotion of small agri-businesses and it purposely does not address the export crops (coffee, tea, cotton) and value chains such as milk, sugar and edible oil products. Furthermore, the activities are farmer based and comprehensive, starting from varietal testing and environmental conservation to keeping seed, accessing finance and selling surplus and re￾investing into productive activities. The project also addresses very well the relevant sections of the Commune level Development Plans. The prime stakeholders for these SO2 activities are: farmers as direct beneficiaries as individuals or households (such as for seed/cuttings and goat distributions, FFW, SILC groups, direct trainings), farmers as indirect beneficiaries as individuals or households (the goat and seed/cuttings solidarity chains, cascade trainings), unpaid technical committees selected among the beneficiaries (LAE, Marais, Livestock), the field agents of the DPAE (Moniteurs Agricoles and ACSA), Associations as informal groupings with a goal to primarily produce for consumption, Agro-Enterprises as formal groupings with the goal of producing to sell (including the tree nurseries). The SO2 activities benefited from the constructive feedback from the MTE in June 2010 and the NCE in August 2011 to make management and implementation changes to achieve better results. The three which stand out are: 1) the M&E team studies of particular implementation barriers among beneficiaries in order to better overcome them, 2) the satisfactory integration of nutrition and food security through some SO1 beneficiaries, particularly the Mother Care Groups (SILC and agricultural products and knowledge), the distribution of vegetable seed to 2,319 PLHIVs and the dissemination of keyhole gardens and 3) a systematic approach and tools for data collection and analysis. 1. IR 2.1 Farm Production. Farm production activities consisted of both genetic improvements to basic stock (vegetal and animal) along with better management techniques, including for soil and water conservation. a. Anti-Erosion Measures. 20 Anti-Erosion committees of 5-7 members (1 per colline) have been formed and trained to encourage and supervise the construction and maintenance of contour berms. A total of 746 km of contour berms have been built (at a rate of 10 meters/person-day), protecting an estimated 1,120 hectares and benefited 3,866 more vulnerable beneficiaries with FFW during the annual few months of chronic food shortage. In April 2012, a tripartite MOU outlining the roles and responsibilities of the local authorities, the DPAE and the LAE 14 Stratégie Agricole Nationale 2008-2015, Ministère de l’Agriculture et de l’Elevage MYAP Final Evaluation Report 31 July 2012 26 committees for the management of the berms was ratified. However, it made no reference to the quantity (percentage of hillside coverage) or quality of the activity (berm profile and grass cover). Contour berming is done traditionally outside of projects but not collectively and not as a paid-for activity. During the project, it is estimated that perhaps another few dozen kilometers of berms have been built independently. It was impossible to assess well both the coverage and quality of the activity as the only indicator was the number of km achieved. An estimate is that about 60-80% of the marais slopes have been bermed and of those 75% are planted at any one time. It is recognized by all that the contour berms have helped to reduce the flooding in the marais, along with the water control measures in section (c) below. Fodder grasses were provided by the beneficiaries and planted on the berms under staff and committee supervision. Farmers now understand and appreciate well the benefit of berms on their land (improved productivity and reduced erosion) even if the berms take away some of the land area used for cropping. However, lower hillside berms were often kept bare on purpose as they are often grazed by roaming cows and goats and farmers had asked for non-edible but useful plant cover, such as vetiver grass, but none was found with the tree nursery suppliers. The survey results indicate that the proportion of farmers practice soil and erosion control measures has increased from 56% to 85% and that those growing fodder grasses has increased from 36% to 62% and in the same ratio for vulnerable and non-vulnerable households. In the longer term, it is not clear how the quality of this activity will be sustained as no benchmark of quality has been established and there needs to be a continuous effort of education and of setting and enforcing best practices, under the joint responsiblities of the committees and local administration. In terms of the quantity of this activity, the barrier analysis reveals that beneficiaries think that proper berm maintenance is achievable if done regularly and is part of the planned community work. The LAE committees have only started to put in place a sustainable system of compensation to encourage them to continue their unpaid work and cost recovery for tools through the production and sale of grasses. There has been no element of hillside erosion damage rehabilitation, such as gullies and few tools are available to do the work though CRS was planning to distribute toolkits soon. The agro-forestry component does not have clear goals for reforestation and erosion management other than the production and distribution of saplings which were bought by CRS only in the first year for distribution to vulnerable household for planting as an anti-erosion measure. The proportion of households having planted trees has increased from 39 to 58% and more Contour Berm Lesson Learned There has been a great synergy in the MYAP between the forced stabling of goats in the improved cross-breeding activities and the required fodder cropping on contour berms. The need for fodder stimuated production on the contour berms whch encouraged construction and use of berms. MYAP Final Evaluation Report 31 July 2012 27 particularly in women-headed households. The saplings were produced by tree nurseries or “associations” who are also part of the agro-enterprise sector (see I.R. 2.2a below) and they have continued to produce and sell saplings in large quantities after the initial support from CRS. b. Crop and Livestock Technologies (Lead Farmers and Farmers). This is a “soft” component that is entirely knowledge based. The project considered using the well proven Farmer Field School approach after the mid-term evaluation had recommended this as an option, but it was decided that the approach could not be effectively implemented in the remaining time in the program. Instead, knowledge, as shown in Table 11, was imparted mostly through trainings by staff, with ongoing dissemination through the DPAE Moniteurs Agricoles (1 per colline) and through model (lead) farmers selected among their peers and with support from the supervisory committees (LAE, Marais and Livestock). Due to the popularity and dynamism of the SILC groups, it was later also put in practice to foster knowledge and dissemination of improved varieties of seeds and cuttings through the SILC groups as they meet once a week and form a strong social network. The general method was to offer an initial training by staff to “lead” beneficiaries, including the Moniteurs Agricoles and the model farmers using flipcharts, some basic handouts and hands-on practice on individual plots. The Moniteurs Agricoles and the Model Farmers then did onward (cascade) trainings to other groups and reported on the trainings. The staff and M&E teams would then assess their performance. In general each of the 134 model farmers trained another 30 beneficiaries and all model farmers received the total package of trainings and products. In some instances, the training was linked to a distribution (either directly or through fairs) of materials (seed, plant cuttings, tools/equipment, goats) but this is covered in sections (e) and (f). Table 11. Delivery of Knowledge in Agricultural Technology Transfer Knowledge Delivery Mechanism Erosion & water runoff control through contour berms Through the LAE committees Reforestation Through the LAE committees Improved water management in valleys - Irrigation gates in valleys and supporting drainage canals Through the Marais committees Improved knowledge and use of seed of improved varieties, seed spacing and field preparation Through the Model Farmers, Moniteurs Agricoles and selected varietal testers and multipliers Crop disease recognition and management Through the Model Farmers, Moniteurs Agricoles and selected varietal testers and multipliers Animal disease recognition and treatment Through the Livestock committees, buck owners and with the support of ACSAs in Kayanza Composting Through the Livestock and LAE committees, Model Farmers Seed and grain storage Through the Model Farmers, Moniteurs Agricoles and selected varietal testers and multipliers From the FG discussions, it is clear that the current impact of trainings is good and farmers highly appreciated the information received, were able to share it and were able to put it in practice. The only barriers to the knowledge dissemination noted were a reticence for farmers to use new practices without having also received new seeds or cuttings and the lack of access to MYAP Final Evaluation Report 31 July 2012 28 mineral fertilizers and pesticides (availability and cost). The process of knowledge dissemination by observation and imitation and by cascade training and coaching is working well. Farmers also indicated not having problems with the storing of the larger harvests, though many expressed a desire to have access to more shared storage (hangars communautaires) 15 . Longer term impact is also good as important synergies linking learning, practice and dissemination have been achieved through the experiential learning with on-farm varietal trials and multiplication in groups or as individuals, the involvement of Model Farmers and DPAE agents and also through the involvement of the SILC groups in sharing knowledge and new varieties. Model Farmers are chosen not only for demonstrating good practice in animal raising, crop production and environmental conservation but also for their social standing and commitment to share their knowledge and disseminate improved seeds and cuttings. They also support committees and local authorities. Most beneficiaries mention they no longer have to buy seed and can keep their own or trade with neighbors. The household survey results could only offer the limited information that the proportion of farmers who had attended trainings increased from 38% to 62% and about 25% more farmers can now delay crop sales until a better time. c. Productive Infrastructure Rehabilitation (FFW/CFW). After studies, surveying and rehabilitation plans made, three marais totaling 237 hectares have been properly (re)habilitated, using a few concrete structures, gabions, wooden poles and unlined channels and benefited 4,563 beneficiaries with FFW. A marais is the lowest lying and flat area collecting all the water and soil from surrounding hills. Thus it is generally very good for agricultural production as it has fertile soils and enough water for year round production but only if the latter is managed. While more vulnerable families were targeted, they were not sufficient in number and other physically able people were also involved. Each marais is divided into a small number of zones (6 to 9), each managed by its own committee, trained by the project to both manage the water distribution and the maintenance with support from the local authorities and users. In the last 6 months of the project, the marais infrastructure and its management have been transferred under a tripartite agreement to the DPAEs, the local authorities and to the committees representing the users. However, four months before the end of the project, only two of three marais had the beginnings of a system to collect user fees to pay for 15 Shared storage construced by previous projects exists in all collines. Water Control Structure with Diversion Channel Lesson Learned The use of group experiential learning techniques ensures that change will be internalized by participants and that they are not doing new activities promoted by the program just to see what additiona benefits they might gain from the program. It also builds capacities of participants to be able to continue trying new ideas and innovating after the program ends. MYAP Final Evaluation Report 31 July 2012 29 maintenance and repairs. In all cases there is also a marais supervisory committee constituted from representatives of each marais zone committee and representatives from the administration. It is clear that the marais are now much more productive and appreciated. Rice is grown in all three rehabiitated marais but to varying extents (80% in Kirundo, 50% in Muyinga and only 3% in Kayanza where it is a new crop at its agro-ecological limit). For example, the price for trading or renting marais plots has generally tripled and most users can now harvest 2 to 3 times/year instead of only once previously and most indicated having tripled their output. If rice is grown between January and June, it is often followed by sweet potato or a combination of corn/beans and then vegetables in the dry season. All now report a much fairer use of water, very little animal damage and no reports of annual flooding (save for the bottom portion of the Muyinga marais next to the natural buffer zone where it is impossible to intervene to drain the water from the upstream proect site). Local authorities assist through imposing fines on stray animals owners, helping to organize work days and resolving plot boundary issues as the land belongs to the state, but first users in the marais have a continuous right to use this common property and this right of usufruct is passed on to descendents or informally traded. Most marais farmers see the infrastructure work primarily as a way to avoid flooding of the marais (they practice only rain fed agriculture, now with two rainfed crops per year) to be managed collectively (or just by someone else, i.e. the committee) and much more less as a means of irrigation as this requires a network of secondary and tertiary channels which require individual but organized maintenance and management and takes away some useable space from the small plots. There were two main issues noted. In two marais, the committees did not feel well supported by the authorities and the users were demotivated to continue working by themselves and canal maintenance was irregular and uneven. The marais is not managed for optimum agricultural production (both for consumption and for market), which also leads to conflicts between rice farmers who require crop submersion and adjacent non-rice farmers, and the rice lands have not been leveled for efficient production. Unfortunately, the baseline survey data was unreliable and a valid endline-baseline comparison was not possible. The barrier analysis reveals that beneficiaries think that proper water channel maintenance is easily achievable if done regularly and should be part of the community work. d. DPAE Capacity-Building. The DPAE is the technical partner at field level and is the intermediary between the national Ministry of Agriculture (MINAGRIE) with the Institut des Sciences Agronomiques du Burundi ( ISABU) for research and the value chain departments to disseminate the product of research to the DPAEs and eventually to the farmers. The DPAEs are chronically underfunded centrally and have a strong responsibility to feed information upward. The directors of the DPAE have a level of autonomy and are primarily interested in forging partnerships which help fund their activities, hence the variety of responses between them, Kirundo marais under rice MYAP Final Evaluation Report 31 July 2012 30 depending largely on the interest of its Director. CRS has an MOU with the MINAGRIE centrally but there is no provision for individual cooperation agreements with each DPAE, particularly if CRS is not funding DPAE administration activities. The DPAEs have multiplication fields in each province but most are under-used for lack of resources and occasionally lack of interest. The DPAEs do not have standard guidelines for marais/watershed development and sustainability, nor do they have the authority to validate the work plans and it’s all done on a case-by-case basis. In practice the DPAEs do not provide a coordination effort amongst NGOs, who tend to do it among themselves informally. Each DPAE has a management layer at the Provincial level with technical head of sections, a layer at the “Zone” level covering several communes (agronomist and veterinarian), a layer at the commune level (agronomist and veterinarian) and at the colline level (Moniteur Agricole and ACSA only in Kayanza). The Directors were all changed in 2010 after the national elections and the relationships had to be re-established. Following on the Poverty Reduction Strategy Paper of 2006 and according to the National Agricultural Strategy 2008-2015 and personal interviews, it is clear that the MINAGRIE is only at the beginning of a long process or (re)organization. While CRS and its partners have kept a line of communication with the DPAEs on a regular basis, only in Kayanza and Kirundo has there been a spirit of cooperation throughout the project. To be more efficient in time and costs, CRS bypassed the internal MINAGRIE processes and established a direct relationship between ISABU for accessing improved varieties and the farmers to test, multiply and disseminate them. The project’s activities of capacity building of the DPAEs and MINAGRIE has been: 1. With ISABU to obtain some varieties externally to test and introduce in Burundi (OFSP, rice, cassava, beans) 2. To improve the technical knowledge of the 20 Moniteurs Agricole and their credibility by involving them on committees (they co-sign all the committee bylaws), helping with cascade trainings and awareness raising, beneficiary identification, with the seed trials and the growing of fodder. 3. With the ACSA, also part of the livestock committees. 4. In the last year, negotiations for the signing of tripartite MOUs for the management of activities linked to Livestock, LAE and Marais. During the negotiations, the roles and responsibilities all the key stakeholders (DPAE, commune level authorities and user committees) are established. 5. A few field visits were organized with the DPAE and local authorities 6. The DPAE was always involved for all animal distributions, especially for zoo-sanitary control. 7. Key consortium staff also have kept informal and personal contacts with DPAE staff, particularly in Kirundo and Kayanza. e. Seed and Livelihoods Fairs. CRS has a long and successful history of using seed and livelihood fairs to improve farmers’ livelihoods with quality products while enabling farmers’ choice to best address their personal needs. To that effect, 11 fairs were implemented until early 2010, during which 11,650 farmers benefitted from such fairs, particularly for improved bean and maize seed and tools. Fairs are particularly used during emergencies and the context in Burundi had moved on sufficiently by 2010 to use more developmental approaches, using on-farm trials and MYAP Final Evaluation Report 31 July 2012 31 multiplication (see section (f) below), and to also speed up the process of introducing and testing new varieties which were not available on the market. During a fair, selected beneficiaries (more vulnerable) use CRS issued coupons of a predetermined value (in this case between $6 and 10) to buy goods (they can also buy more with their own money) from commercial vendors. These vendors have been previously vetted and approved by CRS for the quality of their goods and compliance with the system to redeem the coupons. For rice, selected project farmer multiplicators produced rice seed to sell at fairs. Lastly, there were also distributions to 1,391 PLWHA of market garden seed through the DPAE Moniteurs Agricole with concurrent DPAE training. For rice, selected project farmer multiplicators produced rice seed to sell at fairs. f. On-Farm Trials and Multiplication. Due to the pressing need to introduce new varieties both for productivity increases and resistance to specific pests and the response slowness of the DPAEs which have to use a rigid hierarchical structure for decision-making, the project worked directly with ISABU, the research department of the MINAGRIE to conduct both varietal tests and farmer multiplication. Each product followed a similar approach but adapted to each cultivar: 1. For beans, the MYAP conducted participatory variety selection (PVS) trials of 23 bean seed of improved varieties with 1,300 farmers, many of whom are Lead Mothers in Mother Care Groups, over three seasons. The PVS were conducted using the “mother – baby plot” methodology, in which one “mother” farmer plants four varieties (100 - 400 g per variety) and compares the yield, time to harvest, market value, and taste. Other “baby” farmers, living around the “mother” farmer, receive one variety. ISABU was then approached to supply the seed of the 8 farmer preferred varieties but as it could not fulfill the demand, the multiplication and dissemination were accomplished by the farmers themselves under supervision. 2. For cassava, the MYAP distributed more than a million cuttings tolerant to cassava mosaic disease to 11,084 vulnerable households through a voucher program. Ratoons (regeneration following initial harvest) of secondary fields and tertiary fields are disseminated to vulnerable households in early 2012. During 2011, cassava brownstreak disease (CBSD) was officially detected in provinces bordering Lake Tanganyika, but the MYAP fields were tested negative for CBSD. However, Consortium staff in collaboration with the DPAE and farmer groups conduct the Quality Management Protocol (QMP) prior to a field’s dissemination. This QMP consists of transect walks and visual testing of samples with all stakeholders present. 3. For orange-fleshed sweet potato (OFSP), the MYAP accessed three varieties from Rwanda’s Agricultural Sciences Institute (ISAR) and began multiplying them with three private nurseries as well as providing vitro-plants to ISABU for further multiplication. OFSP is rich in vitamin A and is part of MYAP’s strategy of increasing diet. OFSP is rich in vitamin A and is part of the MYAP’s strategy of increasing diet diversity. The Consortium is now using PVS trials similar to the bean model in the first quarter of 2012 through Mother Care Group participants. At the last count, more than 6,000 beneficiaries had received cuttings. 4. There was an attempt to introduce new varieties of white potato (which can also be a cash crop) to agro-enterprises, but the seed stock was infected by an undetectable bacterial disease MYAP Final Evaluation Report 31 July 2012 32 and the effort was canceled as participants indicated that they could access acceptable seed stock through the DPAEs and commercial vendors and this activity would only reach a small number of beneficiaries overall. 5. For rice, the Consortium monitored farmers’ appreciation and experience with three improved varieties of high-altitude rice. Over 1,500 farmers gained access to these seeds in 2010, and 81 farmers were provided a total of 135 kg of the same varieties in small packets in 2011. The Consortium, with the assistance of Burundi Institute for Agricultural Sciences (ISABU), worked intensively with farmers on proper production techniques, conditioning and marketing. In 2012, the project is focusing on working with farmers on demonstration plots for weeding techniques as well as seeking to hold seasonal calendar discussions to seek solutions for the competition between rice and bean crops for farmers’ time. A total of 154 farmers were involved in the mother/baby plots and 286 groups and individuals in multiplication activities. In the FGD, all farmers wanting new seeds and cuttings indicated that they would soon have access to them at no or low costs. During the FG discussions, beneficiaries indicated that even if they had not yet received new seed or cuttings, they would do so soon at no cost and they knew how and where to get it. g. Goat Distribution & Cross-Breeding. The project distributed 288 Boer bucks imported from Uganda and 5,400 local does. The bucks went to farmers selected for having the means to take care of a high maintenance animal while the does went to more vulnerable families who satisfied a minimum of five criteria: 1) fodder grass grown on contour berms, 2) a stable, 3) a compost heap, 4) mating with a Boer buck and 5) an agreement to pass on the first 2 kids to other families on the solidarity chain list. All participants also received specific trainings on animal management including diseases and treatments. In both cases, there were 2 separate distributions and the second distribution of does was done through fairs so that the farmers could choose the best animal as animals purchased on tenders tended not to be of highest quality. DPAE veterinary staff were present at all goats and ram distributions, thus preventing the introduction of diseases, as had been the case with other distributions. The imported goats have generally been somewhat more fragile, but each consequent breeding improves the resistance of the cross-breeds which generally grow to twice the size in the same amount of time and fetch between 3 and 5 times the price of traditional goats in Kirundo and Muyinga provinces. In Kayanza province the cross-breeds only have a marginal price increase. Boer buck owners do not make much economic profit from breeding services (500Fbu/each) as they are more expensive to maintain, some have to buy fodder and some doe owners do not pay them. But they are proud to have been selected and most have increased the size of their goat herd. Boer Goat Distribution (center) MYAP Final Evaluation Report 31 July 2012 33 Veterinary support from the DPAE is very weak in Kirundo and Muyinga provinces but satisfactory in Kayanza province due to the presence of ACSAs in each colline (funded through an international project). Each sous colline has its Livestock Committee who is responsible for ensuring that the animals are well cared for and that the solidarity chain is working properly with support from the authorities and particularly from the future participants. The Committees are also motivated by acquiring one kid for each 10 distributed through the chain. Members of the solidarity chain are chosen in public in a transparent manner and the chains are now in their 3rd generation and have redistributed more than 1,700 doelings (it takes about 15 months between each generation). When male kids are born, they are exchanged on the market for females to be redistributed to the next solidarity chain recipient and Boer bucks are also exchanged across collines to avoid in-breeding. If recipients do not get to keep the goat at first because they have not complied with the requirements for growing fodder or having a stable, they get a second (and usually satisfactory) chance at the next round. Beneficiaries are satisfied of the trainings they have received, especially for disease recognition and treatment and nearly all beneficiaries de-worm their animals 4 times/year. When a new family has a few goats, they sell between 1 or 2 kids annually on average. But there are other noted benefits from the goat repopulation activities: (1) there is now more goat lending as a social mechanism, (2) there are many imitators who stable their goats and cows and grow fodder for them, (3) there is an increased demand for fodder grown on berms and in fields, (4) children are not required to stay out of school for the purpose of herding goats, (5) access to manure for improved crop productivity, (6) positive linkage to contour berms and fodder, (7) reduced tension between farmers and herders from the disappearance of stray browsing, and (8) the regular income from the sale of kids. However, the activity appears neutral on the consumption of meat within the family (i.e. no one mentioned the killing of goats for eating) or the consumption of goat milk (partly because the Boer breed is for meat and not for milk and partly because different groups do not traditionally drink goat milk). In June 2011, a tripartite MOU outlining the roles and responsibilities of the local authorities, the DPAE and the livestock committees for the management of the solidarity chain was ratified. The household survey results indicate increases in the proportion of farmers who access veterinary services from 26% to 53% and who vaccinate their goats from 6% to 32%. 2. IR 2.2 Revenues a. Agro-Enterprise Capacity Building. In this instance, there is a differentiation between two types of Farmer Associations: associations who represent informal groups of farmers working together primarily for the purpose of producing for consumption in a more efficient way (sharing labor and land primarily) and AEs who are more formalized (at least registered with the Commune and have a bank account) and with a primary purpose of producing to sell. By project’s end, 17 AEs had been actively assisted including 5 tree nurseries and 121 Associations had also been reached. The reason for the low number of AEs supported is that there are very few Associations who have reached that level of performance for vision, technical, financial and management skills. About two thirds of AE and Association members are women. MYAP Final Evaluation Report 31 July 2012 34 All the AEs received intensive trainings on operational and financial management and marketing. Except for the tree nurseries, none do agricultural seed or cuttings multiplication as a business activity but all recognize well the importance of looking for markets before starting production. Support to AEs started first with the setting up of tree nurseries as part of the IR 2.1 anti-erosion activities. All but one were formed from existing AEs and encouraged to diversify their activities. In the first year, CRS bought 80% of the seedlings (agro-forestry and fruit trees) to distribute to beneficiaries to contribute to the anti-erosion activities. In later years, the surviving nurseries found their own markets, with other projects but mostly with schools, churches, commune offices and individuals. For four of these AEs, tree nursery is one of their many products, its main advantage is that of high income for a small land area but the market has to be acquired before starting, unlike many agricultural crops. Some have been able to offer a $35 individual share-out at season’s end (equivalent to the price of a kid). Nurseries acquire their seed stock from Département des Eaux et Forêts in Bujumbura but could not always find seed/cuttings for the specific or improved species wanted by farmers. As an encouragement to perform quality cascade training to other associations the nurseries were rewarded with appropriate plant nursery equipment, but not all received it. Nurseries are busy from May to March and produce very few fruit trees as there is low market demand and they are expensive to produce. The major crops are eucalyptus and grevillea both for construction or fire wood. AEs were also encouraged to assist other associations with their organizational and technical management, but it is difficult to keep them motivated. Some AEs now also do fodder as business and sell every 3 months (particularly in zones with improved breed cows). Most AE members now use more hired labor to work on their personal plots as they prefer to work together on association activities. Most work on the harvest basis of keeping 10-25% for seed, 15-25% for consumption and 50-75% to sell. The money is placed in a bank account and reinvested in productive activities (land, inputs, tools, new projects-usually animals and other crops such as vegetables, etc.). All AEs now use more land, between two and ten times more than before the MYAP and their clients are about 1/3 neighbors and 2/3 neighboring big town buyers. They work together 2 days per week and all agree to the following advantages: (1) better economies of scale with the new (MYAP) technologies when working together providing better yield and higher productivity; (2) risk sharing; (3) they can contract a bank loan for other productive uses; (4) better knowledge management and sharing; and (5) perform similar improvements on their own plots, including the use of mineral fertilizers. Most groups have not yet grown in numbers but are considering letting newcomers in if they buy an equity share, ranging from $20 to $80. AE’s main problems were with having to sell cheap to a buyer or intermediary for lack of transport and/or drying/storage facility and with having difficult access to fertilizers and pesticides, both for reasons of availability and costs. b. Savings and Internal Lending Communities (SILC). CRS has a long and successful history of group based rotating savings and credit associations, known as SILC, whereby 15-20 people agree to first pool a set amount each week and after one month to start distributing the small credits to one/some members to use for up to 3 month at 10% interest until everyone has had a MYAP Final Evaluation Report 31 July 2012 35 chance to borrow and reimburse until 3 months before the end of a maximum cycle length of 12 months. In these last three months, credit is not issued, the reimbursements are all collected and each member gets a dividend proportional to their equity at the share-out. After the pay -out, the group then starts another cycle, sometimes with slightly different rules based on consensus and lessons learned and usually with a higher weekly contribution. All groups have 2 funds, one for productive loans (now with contributions of about $1/week) and one for personal emergencies (with contributions of about $0.2/week). A few groups have had a share-out after 8-9 months so as to coincide with the season for peak cash need (August/September for both schooling). SILC groups are of mixed gender, unlike many women only schemes, and this has been instrumental to foster better gender relations. At the time of the evaluation, 278 SILC groups had been formed, 98 of which were for Mother Care Groups from SO1 and 36 had been formed independently of the project. Women represent about 65% of SILC beneficiaries and 20-30% of members belong to 2 to 4 groups concurrently, primarily to maximize their investment and secondarily to stagger the share-out times. Groups in their second or third cycle have tripled the weekly contribution from the original $0.3 to $1 now and some envisage $2 soon. Coverage of SILC groups among the targeted populations in the 3 provinces ranges from 10% in Muyinga to 70% in Kayanza (small population) based on the potential participation of 2 adults per household. After working on the establishment of the SILC system, the project is now working on measures to make the dissemination self-sustaining and to link other activities to SILC groups due to their strong social cohesion. For example for each colline or small number of collines there will be approved SILC Technical Agents who have the skills, experience and credibility to both support the existing groups and start new groups and some of them already have worked out arrangements to be compensated for their services. They are also trained to promote keyhole gardens (primarily for Mother Care groups, but not only) throughout the SILC groups. Linked to the knowledge transfer, each SILC group is now encouraged to assign members to be knowledge specialists in the different sectors: anti-erosion measures, for each agricultural product, marais, gender and social issues and livestock. It is clear that the SILC groups have had an excellent impact, both economic and social, and they are highly appreciated and very sustainable. All interviewed noted the end of the traditional practice of usury with 100% interest and several mentioned that it gave them the opportunity to diversify activities and do other “life projects”. There are few cases of repayment delays, usually limited to a maximum of 1 month and a few early participants dropped out for not being able to contribute every week and were demotivated to continue. Interestingly none of the SILC beneficiaries have pooled their loans to achieve any group activity with economies of scale. Only two specific issues have been mentioned by beneficiaries. Firstly, interested groups are sometimes discouraged to start as the group has first to spend money to acquire basic equipment (cashbox, locks, calculator, account book) and they need intensive training, though this issue should be resolved once the SILC ATs are effectively in place. Secondly, the large amount of money accumulating before the share-out has caused two problems: 1) if placed in a personal Lesson Learned In addition to the direct benefits from participation in SILC savings and lending, the SILC groups offer a platform and devlivery channel for other benefits. For example, the MYAP disseminated improved varieties of seed and cuttings thrugh the SILC group as part of the crop and livestock technologies intervention. MYAP Final Evaluation Report 31 July 2012 36 bank account, there are cases of an informal tax taken and 2) if kept in the cashbox, it places that person/household at risk of theft. The household survey results indicate an increase in the proportion of the population using a community microcredit scheme from 6% to 65% and a 30% decrease in the use of more usurious systems. C. Outputs Produced Under Strategic Objective 3: Community Resilience Proposed outputs under the Community Resiliency component of the MYAP are shown in the Table 12 and discussed in more depth in the sections which follow. Activities for SO3 were implemented in the 21 collines in the three targeted watersheds by CRS in Kirundo, ODEDIM in Muyinga and BADEC in Kayanza. Table 12. Logic Structure for the Community Resilience Component of the MYAP GOAL: By 2011, vulnerable households in the provinces of Kayanza, Kirundo and Muyinga have reduced their chronic and transitory food insecurity. STRATEGIC OBJECTIVE 3: Vulnerable communities have enhanced resiliency Intermediate Result 3.1: Communities and local government agents have established community-based early warning and response. Output 3.1.1: Communities and local government agents have the skills to assess risk to their food security. Output 2.1.2: Communities have the capacity to develop community action plans. Intermediate Result 3.2: Vulnerable households are efficiently managing their assets in an equitable manner. Output 3.2.1: Women and men understand the importance of mutual decision-making regarding management of household assets. Output 3.2.2: Female and male community opinion leaders and decision-makers have improved knowledge of women's rights. 1. IR 3.1 Early Warning and Response a. Risk and Vulnerability Assessment. In order to build capacities at the community level to be able to cope with food security shocks and stress, the MYAP proposed to train communities, specifically the comités de développement communale (CDC16) and government officials at the colline-level on risk and vulnerability assessments. The program would then provide support for these colline-based institutions to develop community "self-development plans" to respond to food security shocks. A small grant scheme was envisioned to make limited resources available for implementing these plans. In terms of achievements by the MYAP, a PAC (Plan de Action de Communautaire) Committee has been formed in each of the 21 collines targeted under SO3. These committees include the Chef du Colline, an agricultural extension agent from the DPAE, and three persons from each Sous-Colline, including the Chef du Sous Colline and at least one woman. These committees have been trained in early warning, risk assessment, community action planning, women’s rights, and agricultural topics related to farm production to mitigate hunger. Specific messages included planting some crops earlier, planting drought-resistant crops, using available water resources 16 In the original MYAP proposal, the CDC was described as a commune-level institution. In fact, the CDC is a colline-level committee that reports to a Commune-level Comités Communaux de Développement Communautaire (CCDC). MYAP Final Evaluation Report 31 July 2012 37 near water bodies, planting forage for livestock, managing harvest, and seed selection and storage. It is estimated by the program’s monitoring systems that 4,676 persons had been reached directly by the program’s messages on the agricultural topics for mitigating hunger. The original design of the MYAP envisioned building community capacities to be able to mitigate and respond at a community-level to food security shocks. Early warning systems would be established to be able to generate information to provide forewarning to impending events. The context in the MYAP program area and Burundi in general does not have a lot of large-scale risks that require an emergency response from government, NGOs and donors. Drought is the primary risk, but it is relatively localized. In interviews held during the evaluation, others types of shocks mentioned by respondents includes outbreaks of crop pests and diseases including caterpillars in sweet potatoes and rice pests, hail, and wind, but these were all relatively localized, not requiring a major emergency response. It is challenging to build emergency preparation and response capacities when demand for wide-scale emergency response is not large. Unlike other countries, there is no government structure in Burundi with which to integrate community disaster planning, other than DPAE planning to mitigate drought. If a major emergency were to occur, it would be dealt with on an ad hoc basis through the Provincial governor and administration who would assemble information and request assistance from NGOs and donors. Even with the drought that occurred in two communes in the program area from 2008 through 2010, however, a major response was not organized. As a result of this contextual feature, the MYAP strategy for SO3 evolved to focus more on household-level actions for addressing food insecurity and community development led by the colline PAC Committees. The MYAP reports that early warning systems and safety nets have been established in 21 sites. These are not warning systems or safety nets in the traditional sense but reflect, rather, the awareness raising activities for increasing household capacities to mitigate hunger. The program has been involved in a form of early warning. Since April 2009, the MYAP has gathered quantitative data from a sample of participating households on key trigger indicators for monitoring livelihoods stress and reported the information to USAID (discussed in more detail in Section V.C, page 63). The training that the PAC Committees are doing with community members on various ideas for enhancing off-season production are focused on producing existing varieties in different ways and making decisions around harvest and seed to be able to mitigate production constraints. Because the ideas are being pushed by local leaders, people are listening. Good information on the number of persons actually adopting the ideas is unavailable. Qualitative interviews suggested, however, that activities related to sweet potato production, seed PAC Committee School Expansion PAC Committee Bridge Construction MYAP Final Evaluation Report 31 July 2012 38 storage, using manure and making compost have been found to be most useful and are being adopted by some households. At the time of the evaluation, the PAC Committees were much more engaged in doing community development activities, although these were generally limited to activities that did not require significant purchased materials. For classroom construction, the Commune Administration was providing cement and iron sheets for schools, sometimes critically delayed. The small grants program proposed by the MYAP was not implemented based on the assumption that it would cultivate dependency betwwn PAC Committees and the MYAP. b. Community Action Plans. All 21 PAC Committees have developed community action plans and have undertaken a variety of community development projects including building additional classrooms, building housing for teachers, building a school office, widening or maintaining roads and footpaths, building bridges, rehabilitation and protection of water points (not water points rehabilitated by the MYAP), and establishing tree plantations. The PAC Committees include the leadership structure in the colline, making it a potentially empowered committee and a good structure for planning and mobilizing resources. There is a structure already present in the colline, however, the CDC, that has similar functions to the PAC Committee for community development. The CDC exists at the colline level which reports to a CCDC at the commune level. Both were mandated by government decree in 2005 to serve for a period of three years after which elections would be held for posts on the committees. Elections were held in 2009 in some places but not others, and there is a general perception that these committees are relatively dysfunctional. The CDC is authorized to collect money from residents for community development while the PAC Committee is not. Because the PAC Committee contains the colline leadership, however, many committees have been able to collect small amounts of money. In interviews with Commune Administrators or their representatives, it is clear that the administration views the Chef du Colline as their primary planning body for the colline. They know about the CCDC and CDC but admit that these are not very effective. But, they do not really recognize the PAC Committee as anything other than something done for a CRS project. 2. IR 3.2 Asset Management/Gender Equity a. Community Sensitization on Gender Equity in Decision-Making. The strategy in the original MYAP proposal proposed a dual focus on expanding women's participation in decision-making around household assets as well as women's participation in decision-making around community assets. The program intended to identify and train community point persons on gender and to use these persons along with positive deviants to sensitize communities on the potential development benefits that could be realized by greater participation of women in decision-making. The program planned to also facilitate linkages with existing women’s advocacy groups and to provide small grants for women’s associations or other groups such as PLHIV Associations or SILC groups. In terms of actual achievements, a total of 158 Gender Acteur Relais MYAP Final Evaluation Report 31 July 2012 39 Gender Focal Points (Acteur Relais), including 91 men and 67 women, have been identified and trained in the 21 watershed collines on a set of gender modules (difference between gender and sex, equitable distribution of household assets, equal rights between men and women, and benefits of shared household decision-making), women's rights and household conflict mediation. A total of 150 persons have been certified as Positive Deviants, including 88 men and 62 women. These are men and women who have achieved significant change within their households toward more gender-balanced decision-making and shared household workload. The Gender Focal Points and Positive Deviants have organized meetings and events to raise awareness on gender balance. These have included specific meetings on gender balance at the sous-colline level, drama performances containing gender messages and opportunities taken in other meetings to spread gender messages. The key messages for the program are:  Husbands and wives should respect one another for the contributions they make to household life.  Household assets, including the harvest, are to benefit everyone in the household.  Household chores are the responsibility of everyone in the household.  The household can realize benefits by allowing women to join associations.  Making plans and decisions together will make life better for the household. In addition to raising awareness about the benefits of gender balance, Gender Focal Points and Positive Deviants have also provided counseling to households having difficulties. These are typically arguments about using household assets such as finding money to take children or women for medical treatment or to meet school costs. There are also arguments about unilateral selling of household assets and the arguments occasionally become violent enough that women, and sometimes men, are forced to leave the house for a period until tempers cool. The gender team (Gender Focal Points and Positive Deviants) may be invited to mediate in these situations by the husband or the wife, or local leaders. As a result of the efforts of the Gender Focal Points and Positive Deviants toward raising awareness, it is estimated from qualitative interviews that between 70% and 80% of the people living in the 21 collines have heard the gender messages. Relative to encouraging more women's participation in community-level decision-making, management groups formed by the program, especially after the Mid-Term Evaluation, have targeted significant inclusion of women. The nursery associations are composed of 47.9 % women. The Marais Committees have 36.5 % women; the Anti-Erosion Committees have 38.4 % women; and the Livestock Committees have 68.2% women. The Water Point Committees have 39.1 % women, and the PAC Committees has 30.5% women. The selection of persons by the MYAP to be trained as Gender Focal Points was generally highly effective, finding men and women with the right skills and motivation. The training provided by the program on gender topics, women’s rights (provided by a representative from the Association de Femme Jurist) and A Gender Positive Deviant MYAP Final Evaluation Report 31 July 2012 40 conflict mediation, has also effectively empowered them. Positive deviants were rewarded publicly with certificates and are quite proud of their recognition. The extension of gender balance messages by Gender Focal Points and Positive Deviants and the counseling to individual households has been quite effective, especially after the number of Gender Focal Points was expanded from two persons per colline to an average between seven and eight. The task of changing traditionally entrenched gender roles is difficult, and the mutual support provided by a larger team has been effective at reaching the point where benefits become more apparent. Traditional household conflict mediation structures17 such as the chef de colline, chef du sous￾colline, nyumbakumi and abashinganahe, still exist and function with the responsibility to mediate disputes within the community. Their primary responsibility is with disputes between households or groups of households, although they also have responsibility for addressing intra￾household problems, especially when these become disruptive to other households. As a result of the MYAP's intervention, the Gender Focal Points/Positive Deviants are now preferred mediators for intra-household disputes, since they are recognized as effective mediators with good messages on gender balance and they do not charge anything for their services. They have come to be recognized by the local leadership as resource persons in the community on gender balance and conflict mediation, and these leaders are requesting their assistance with households having difficulty. The Gender Focal Points are currently implementing activities with minimal support from the MYAP. They are only submitting monthly reports to the MYAP on activities, and they indicated in interviews that they are not sure who they will submit these reports to after the program ends, except in Kayanza. In Kayanza, another NGO (FHI) is also implementing a gender balance strategy using a similar approach with one person identified as an Acteur Relais in each colline. Usually this person is also a MYAP Acteur Relais and is part of the gender team in the colline. The FHI Acteur Relais reports to the Communal CDF (Centre de Développement Familial) office, and the MYAP Acteur Relais will also do so after the program ends. The MYAP has begun discussions with the CDF about turnover of monitoring responsibilities to them after the program ends, although these are still preliminary discussions. The MYAP has not facilitated development of any linkages with Collectif des Associations et ONG Féminines du Burundi (CAFOB) or other women’s associations, as originally proposed, since there are presently no women’s associations who have a presence (office) in the three provinces, and the program did not have resources to facilitate the establishment of such offices. Special management training sessions for women, as originally proposed, have also not been organized. The program has also recently started working toward forming Associations of Acteur Relais, hoping they will continue to work together and draw support from each other after the program ends. 17 The nyumbakumi is a structure in which a person has been selected from among a group of ten households to organize and provide leadership for the group of ten. The abashinganahe is a traditional community structure in which a household is formally recognized traditionally as a model household. MYAP Final Evaluation Report 31 July 2012 41 The Mid-Term Evaluation (MTE) made a number of recommendations for improving the quality of gender programming in the MYAP. One of these, providing more training to Acteur Relais to be able to provide support to houses in conflict, was focused specifically on activities under IR 3.2., and the program effectively addressed the recommendation. Other MTE recommendations focused on cross-cutting gender issues, specifically engaging women in the design of action research on water harvesting and key hole gardens and establishing a database of participants disaggregated sex. The program responded to both of these recommendations, choosing not to implement the action research but disaggregating the projects's participant database by sex. The MTE also included a gender recommendation as one of the top ten recommendations, to push the "gender envelope". Although the intent of this recommendation is certainly valid, the articulation of it was a bit confusing and the MYAP did not fully internalize it. b. Training on Women's Rights. While the original proposal described building knowledge on women's rights as a separate set of activities, the MYAP included training on women's rights as part of the training provided to Gender Positive Deviants, who then passed on the knowledge as part of their awareness raising activities around changed gender roles. All Gender Positive Deviants received training provided by a representative of the Women's Lawyers Associations, and this knowledge on women's rights was very useful in providing the Gender Positive Deviants information to support their advocacy for changing the way women were viewed in their rural communities. IV. PROGRAM IMPACT ASSESSMENT A. Focus of the Final Evaluation The final evaluation of the MYAP is a summative evaluation intended to assess the impact of the program. The evaluation also analyzed the impact and the processes used in the MYAP to identify good practices and lessons learned relevant to food security programming. The previous section has documented the observations of the evaluation team on program activities and outputs. This section describes the observed outcomes and impact achieved by the program based on qualitative observations and analysis of data from the quantitative household endline survey. B. Targeting One element of determining program impact is to identify who is benefitting from program activities and how well these beneficiaries conform to the targeted impact groups, which have generally been defined in the MYAP as "vulnerable households with chronic and transitory food insecurity". The MYAP proposed to implement SO1 activities targeting the eligible population across all of the three provinces of Kirundo, Muyinga and Kayanza. The MYAP also targeted three watersheds, including Kinyangona in Kayanza, Lake Cohoha in Kirundo and Nyamuswaga Lesson Learned The MYAP did an effective job of identifying Gender Focal Points to be trained to disseminate messages around the benefits of changed gender roles at the household-level. Consulting with existing community-based organizations, they sought men and women with the following qualities:  Keen interest in the type of change being promoted  Courage to be able to confidently spread the messages  Ability to create good rapport with people  Recognized as influential leaders (formal or informal)  Ready to vilunteer time MYAP Final Evaluation Report 31 July 2012 42 in Muyinga, encompassing twenty-one collines; and in fourteen of these watershed collines, the program implemented activities for all three SOs. In the other seven collines, the MYAP implemented SO2 and SO3 activities. Under SO1, beneficiaries were identified mainly through clinical screening criteria of children to determine those with moderate malnutrition eligible for supplemental feeding and those with acute malnutrition eligible for different levels of therapeutic feeding. In SO2, the watershed approach, by the definition used in the MYAP, means that the majority of people living within the targeted watershed areas are eligible for various activities. Table 13 shows the cumulative number of beneficiaries reported in the MYAP's Indicator Performance Tracking Table through the end of FY 2011. The current estimated population in the three provinces is around 2,044,50018 and the Table 13. Reported IPTT Beneficiaries Against LOA Targets MYAP Objective Reported Beneficiaries Through FY 2011 LOA Target SO1 MCHN 330,326 children 94,906 mothers 290,000 children or 125,000 households SO2 Livelihoods 10,808 households 10,900 households SO3 Community Resilience Not reported on the IPTT 18,000 households TOTAL Not reported on the IPTT 125,000 households in the Province and 18,000 households in the watersheds estimated population in the twenty-one watershed collines is around 83,000 (around 16,600 households)19. Beneficiaries shown in the table under SO1 include children under five years of age and caregivers for these children who are participating in different types of MYAP activities. A mother and child can participate in multiple activities in the MYAP. For example, they can be reported as having been screened by village health workers (ASC) and found to be a child with acute malnutrition requiring admission to a health center (SST therapeutic feeding) and then graduating from there to continuing to receive MYAP support as an outpatient (STA therapeutic feeding). Table 14 shows the reported SO1 beneficiaries adjusted by an estimate of this overlap in reporting20 . In interviews held during the evaluation, many chefs at the colline and sous-colline levels in the watershed collines reported that there are few people in the colline who have not participated in the MYAP in one way or another. Many of these people would fit the normal definition of the impact group for the MYAP of vulnerable households with chronic or transitory food insecurity. 18 The estimated has been calculated based on an expansion of the 2008 population census estimate by the annual average population growth rate of 3.462%. 19 Estimates of colline population are not yet available form the 2008 census, so this estimate is based on the relative proportion of the population in the twenty-one collines to the provincial population from the 1998 census. 20 Since the MYAP M&E beneficiary tracking systems do not eliminate the overlap in reporting, the estimate is based on a very rough average estimate of duplication in reporting obtained from the IMC implementation team in each province. MYAP Final Evaluation Report 31 July 2012 43 Some of these would fall outside the definition of the impact group, but the watershed approach requires that they be included as participants in the program. Some of the very poorest households, with limited assets including labor capacities, appear to have been targeted by the program specifically with goat distributions, seed fairs and food-for-work. Table 14. Estimated Beneficiaries for SO1 through December 2011 MYAP Activity Children Under Five Caregivers SST Therapeutic Feeding (Admitted) 2437 --- STA Therapeutic Feeding (Outpatient) 17,305 --- SSN Supplemental Feeding 34,461 5,579 Growth Monitoring 39,212 --- PD/Hearth 1,394 120 Mother Care Groups 32,170 24,622 Health Volunteer Screening 248,161 55,763 TOTAL 375,140 86,084 Adjustment for Overlap 40% 30% Estimated Total Beneficiaries 225,084 60,259 In the watershed collines, the greatest impact of the program is being achieved on those people who are participating in multiple activities. The MYAP’s M&E systems include listings of participants by type of activity, but these have not been collated to show participants who are participating in multiple groups or activities. Focus Group Discussions held during the evaluation suggest that a significant proportion of participants are benefitting from multiple activities. All in all, the final evaluation estimates that 15,000 of the estimated total of 16,600 households in the watershed collines have benefitted from the MYAP and 60% of these, or around 9,000 households, have benefitted substantially through participation in multiple MYAP activities. The evaluation team observed no systematic exclusion of eligible beneficiaries. Respondents interviewed during the evaluation did report a few isolated cases of eligible beneficiaries not being enlisted by community committees or authorities for activities. In one location, for example, people who wanted to do FFW in a marais were asked to contribute something to the committee enlisting people for FFW; and when they unable to pay, they were not listed. Some lead mother participants also reported being excluded from the goat listing process because they were told by the goat committee “you already benefit from the program”. In fact, they may not have been eligible for listing because they are generally not highly vulnerable. In which case, this becomes less of a targeting issue and more of a transparency and communication issue. There were also reports of a goat list being reengineered by local administration. For some activities, self-exclusion likely occurs Lesson Learned The MYAP M&E systems do a pretty good job of tracking participants by activity but do not track the total number of people or households benefitting from the MYAP, making it diffiult to estimate how many people or households have benefitted from the program. The M&E systems should be designed from the start of the program to be able to do this. Other projects that have been able to do so effectively, have completed household profiles to document the names of people in the household and assigned case numbers of identification numbers to be able to track household beneficiaries participating in multiple program activities. MYAP Final Evaluation Report 31 July 2012 44 because of low income, for example, participants who do not have enough cash to purchase shares in SILC and poor households participating in SO1 activities unable to build latrines or purchase soap. It must be emphasized that these reports of exclusion are small, localized incidents; and, by far, people interviewed expressed appreciation for the processes used to identify participants for various activities. C. Outcomes Under SO 1: MCHN 1. General Comment on Quantitative Tables. The tables presented in this section include two columns for the 2008 baseline survey. The first column displays figures as reported in the published baseline survey report, and the second column shows the figures resulting from a re￾analysis done by ISTEEBU. Anthropometric endline data were analyzed using a more recent version of the ENA201121 Software. For a more accurate comparison, the 2008 baseline data were re-analyzed using the same ENA2011 Software. For some indicators, large differences are found between original reported baseline estimates and the ISTEEBU recalculated figures. This is due to the use of different denominator population figures. For example, the original baseline analysis calculated the proportion of women having slept under a mosquito only for those who declared having a mosquito net. ISTEEBU redid the calculation on all the interviewees for both baseline and endline surveys. 2. IR 1.1 Infant and Child Feeding Practices. From the household survey, a higher (but not statistically significant) proportion of women reported practicing breastfeeding in 2012 (77.5% as compared with 74.5% in 2008) as shown in Table 15. Close to two-thirds of women reported Table 15. Infant and Young Child Feeding Practices Indicator Baseline 2008 Endline 2012 (N=1015) Statistical Reported Significance ISTEEBU (N=644) Breastfeeding Percent of Women Who Report Practicing Breastfeeding 77.4% 74.5% 77.5% Not Significant p = 0.0805 Percent of Mothers Breastfeeding within One Hour of Birth 77.5% 77.6% 82.0% Significant p = 0.0155 Percent of Mothers who give Colostrum 94.5% 94.9% 97.2% Significant p = 0.0065 Percent of Mothers Giving Food Other than Milk the First Three days After Birth 86.2% 13.8% 9.8% Significant p = 0.0055 Percent of Children<6 months who received Exclusive Breastfeeding in the last 24 hours Not Available 85.5% 65.0% Baseline data were not suitable to perform a significance test (see page 7) 21http://nutrisurvey.net/ena2011 MYAP Final Evaluation Report 31 July 2012 45 having exclusively breastfed their infant (less than 6 months old child) over the last 24 hours (baseline data were not suitable to perform a statistical significance test.) On the other hand, significantly more mothers reported breastfeeding within one hour after birth and giving colostrum. About two-thirds of interviewed women had their child immunization card (a nonstatistically significant increase from 61.5% in 2008 to 65.8% in the endline survey) as shown in Table 16. The proportion of women reporting practicing breastfeeding in the MYAP project areas remains well below the national average (close to 99%) reported in the DHS 2010 preliminary results. Exclusive breastfeeding is, however, close to the DHS 2010 figure of 69%. Table 16. Immunization and Growth Monitoring Indicator Baseline 2008 Endline 2012 (N=1015) Statistical Reported Significance ISTEEBU (N=644) Immunization Percent children having an immunization card 61.5% 61.5% 65.8% Not significant p = 0.331 Percent of children who completed their immunization schedule during their first year 18.5% 23.2% 48.6% Significant p = 0.000 Vitamin A Percent of children having received a vitamin a supplement in the last 6 months 29.6% 29.6 13.0% Significant p = 0.000 Growth Monitoring Percent of children who participated at least once in growth monitoring 57.1% 57.1% 58.4% Not Signiicant p = 0.303 Percent of children who participated in growth monitoring over the last 3 months 37.5% 37.5% 22.0% Baseline data were not suitable to perform a significance test (see page 7) Overall, the MYAP BCC was effective in changing breastfeeding practices but much remains to be done. There was however a very significant increase in the proportion of children who have completed the immunization schedules during their first year of life, over a twofold increase from 23.2 to 48.6%. In contrast, Vitamin A supplementation has halved, and attendance at growth monitoring sessions was found to be less regular than in 2008. The proportion of women with immunization cards is similar to that reported in the 2010 DHS (61.9%). The very low vitamin A supplementation coverage compared with the UNICEF reported estimate of 73% for 201022, requires investigation. It is likely to be a supply issue that needs to be raised by the program with relevant stakeholders, namely MoH, BPS and UNICEF. 22 http://www.unicef.org/french/infobycountry/burundi_statistics.html MYAP Final Evaluation Report 31 July 2012 46 Despite the observed significant progress, immunization coverage remains low and far from the national target of at least 83%23. Progress in immunization coverage is reliant on the availability of vaccines through the national expanded program on immunization (EPI) and the MYAP’s role is primarily awareness raising and behavior change through BCC. However, more progress was expected in the area of growth monitoring in which MYAP, through IMC, has provided substantial technical (staff training and supervision) and material support (such as growth charts). Participation in growth monitoring is inconclusive, and the decline over the last three months is puzzling. The analysis of the endine data was competed after the evaluation team had completed its field work, so it was not possible to obtain qualitative data to help interpret the unexpected finding from the quantitative data. 3. IR 1.2 HIV-Affected Households. Under IR 1.2, HIV/AIDS-affected individuals on ART were to have access to appropriate supplemental food. Progress was to be measured using the following indicator: percentage of PLWHA households eating the recommended number of food groups. In 2009, only 3.3% of PLWHA households consumed the recommended number of food groups and the target was at at 15%. In 2012, 40% of PLWHA households were found to consume the recommended number of food groups as per the MYAP M&E database. 4. IR 1.3 Heath-Seeking Behavior. As shown in Table 17, mothers’ knowledge regarding childhood illness improved. A larger (although statistically nonsignificant) proportion of women could recognize at least two signs of childhood diseases (85.1% compared to 74.8% in 2008). There is a significant decrease in the proportion of children who suffered from diarrhea over the two weeks preceding the survey. The decrease in diarrhea incidence is probably consequent to improvements in hygiene and sanitation, which are discussed under IR 1.4 below. In contrast, there is a significant increase in the proportion of children who suffered from coughing. It must be noted that the endline survey coincided with the rainy season during which an increase in the number of respiratory infections can occur24 . Table 17. Preventing Common Childhood Illnesses Indicator Baseline 2008 Endline 2012 Statistical Reported ISTEEBU Significance General Percent of Women who recognize at least 2 signs of child illness 74.8% 74.8% (N=644) 85.1% (N=1015) * Percent of Children who had fever over the last two weeks 41.8% 41.8% (N=644) 39.6% (N=1015) Not Significant p = 0.191 Percent of Mothers who consulted health center during their child’s fever 73.2% 64.7% (N=269) 72.9% (N=402) Significant p = 0.01 Diarrhea Percent of children who suffered from diarrhea over the last two weeks 30.7 30.7% (N=644) 26.5% (N=1015) Significant p = 0.03 Percent of mothers who used ORS to treat their child diarrhea 41.1% 41.1% (N=198) 48% (N=269) * Percent of mothers who increased the amount of fluids given to their child during diarrhea 20.4% 18.3% (N=198) 48.3% (N=269) Significant p = 0.000 23 http://www.minisante.bi/index.php?option=com_rokdownloads&view=file&Itemid=152&id=114 24 http://www.who.int/diseasecontrol_emergencies/toolkits/Burundi_profile_ok.pdf MYAP Final Evaluation Report 31 July 2012 47 Percent of mothers who consulted health center during their child’s diarrhea episode 92.8% 41.7% (N=198) 47.6% (N=269) Not Significant p = 0.104 ARI Percent of children who suffered from coughing over the last two weeks 38.4% 38.4% (N=644) 49.3% (N=1015) Significant p = 0.000 Percent of mothers who consulted health center during their child’s cough 89.4% 57.9% (N=247) 47.2% (N=500) Significant p = 0.003 Malaria Percent of households who have at least one mosquito net 25.0% 25.0% (N=644) 68.7% (N=1015) Significant p = 0.000 Percent of women who slept under a mosquito net the night preceding the survey 73.5% 18.4% (N=644) 51.3% (N=1015) * Percent of children who slept under a mosquito net 69.3% 17.3% (N=644) 61.2% (N=1015) * Percent of households having adopted at least one method of malaria prevention 21.1% 25.0% (N=644) 68.0% (N=1015) Significant p = 0.000 * Baseline data were not suitable to perform a significance test (see Page 7) Results concerning health-seeking behavior are mixed. While significantly more women sought health care when their child had a fever, significantly fewer women consulted health centers when their child suffered from coughing. More women consulted health centers when their child had diarrhea, although this increase was not statistically significant. Significant changes are observed, however, regarding home management of diarrhea. Around half of women reported increasing the amount of fluids as opposed to only 18.3% in 2008. Significantly more households possessed at least one mosquito net and used at least one method of malaria prevention. The proportion of children and women who slept under a mosquito net increased to 51.3% for women and to 61.2% for children (significance tests could not be performed on these two indicators). The MYAP’s BCC strategy together with the distribution of mosquito nets, either by IMC as part of the household star award system or by the MoH through the national malaria control program, has produced positive results as far as malaria prevention is concerned. The MYAP aimed at improving women’s knowledge and behavior not only for their children, but also for their own health. As shown in Table 18, the endline survey reported a significant increase in the proportion of women asserting the use of a proven Family Planning (FP) method. Utilization of antenatal health services was already high in 2008 and has been maintained. Improvements are observed in iron supplementation during pregnancy and tetanus vaccination (not a statistically significant increase). A higher proportion of women reported having had assisted deliveries (68.2% as compared with 49% in 2008), and significant improvements are observed in postnatal consultations with 33.1% of women reported having consulted at least once after delivery as compared with 22.5% in 2008. The proportion of women who received iron supplementation either during pregnancy or after delivery increased significantly, suggesting improvements in the quality of care available to women in the MYAP areas. MYAP Final Evaluation Report 31 July 2012 48 According to a recent report on reproductive health in Burundi, insufficient progress has been made over the past two decades on maternal health, and the country is not on track to achieve its Table 18. Women Health Knowledge and Practices Indicator Baseline 2008 Endline 2012 Statistical Reported ISTEEBU Significance Family Planning Percent of women who claim using a FP method 17.4% 17.4% (N=644) 44.1% (N=1015) Significant p = 0.000 Percent of women who use a proven FP Method 14.3% 15.5% (N=99) 37.8% (N=389) Significant p = 0.000 Prenatal Care Percent of women who consulted at least once during their last pregnancy 98.4% 98.4% (N=644) 98.6% (N=1015) Not Significant p = 0.367 Average number of consultations 3.2 3.2 (N=644) 3.2 (N=1015) Not Significant p = 0.236 Percent of women who received iron supplement during their last pregnancy 68.6% 75.8% (N=644) 80.0% (N=1015) Significant p = 0.0455 Percent of women who had tetanus vaccine Not Available 68.6% (N=644) 70.2% (N=1015) Not Significant p = 0.2050 Delivery Percent of women who had assisted delivery 46.4% 49.0% (N=644) 68.2% (N=1015) * Post Natal Care Percent of women who had at least one PNC 22.5% 22.5% (N=644) 33.1% (N=1015) Significant p = 0.000 Percent of women who received iron Supplement after their last pregnancy 11.7% 11.7% (N=644) 18.9% (N=1015) Significant p = 0.000 Percent of women who received Vitamin A supplement after their last Pregnancy 23.5% 23.5% (N=644) 29.1% (N=1015) Significant p = 0.0035 * Baseline data were not suitable to perform a significance test (see page 7) Millennium Development Goals (MDG) 2015 targets25 . Various sources report that the majority of pregnant women receive antenatal care from skilled health personnel but that institutional deliveries and postnatal care are less common in Burundi. Moreover, high-risk pregnancies “too early, too late, too close or too many” are a recognized problem in Burundi, warranting measures to promote Family Planning, which is among the top priorities of the national reproductive health policy26. Overall, the MYAP BCC strategy has been responsive to national reproductive health priorities and has successfully contributed to progress in assisted deliveries and postnatal care. 25 http://siteresources.worldbank.org/INTPRH/Resources/376374-1303736328719/Burundi42211web.pdf 26 http://www.pnsrburundi.org/images/stories/LA-PNSR.pdf MYAP Final Evaluation Report 31 July 2012 49 5. IR 1.4 Hygiene and Sanitation. Overall, the endline survey findings corroborate the evaluation’s field observations in the area of hygiene and sanitation. Statistically significant improvements are seen in some simple but crucial practices as shown in Table 19 such as conserving drinking water in a container with a cover, treating drinking water before use, having a place to wash hands and having soap. Table 19. Hygiene and Sanitation Practices Indicator Baseline 2008 Endline 2012 (N=1015) Statistical Reported Significance ISTEEBU (N=644) Drinking Water Percent of households conserving drinking water in a container with a cover 67.4% 67.7% 77% Significant p = 0.000 Percent of households treating drinking water before use 6% 1.2% 5.4% Significant p = 0.000 Hand Washing Percent of households having a place to wash hands 5% 5.0% 55.5% Significant p = 0.000 Percent of households having soap 71.1% 71.1% 86.3% Significant p = 0.000 Percent of women who wash hands before or after the following activities:  before preparing a meal 8.5% 8.5% 11.1 *  before eating 15.3% - 30.7% *  before feeding a child 5.2% 6.5% 10.3% *  after visiting a toilet 6.8% 7.6% 13.0% *  after cleaning the stools of a child 4.6% 4.5% 7.2% * Latrine Percent of households using an improved toilet (ventilated with a cover, cleanable, and not shared) 15.8% 6.8% 21.5% Significant p = 0.000 Waste Disposal Percent of households disposing of child stool in a hygienic manner (in suitable toilets or latrines) 89.0% 89.4% 82.2% * Percent of households disposing household waste in a pit or by specialized waste collection company 15.8% 15.8% 16.8% Not Significant p = 0.2936 Composite Indicator Percent of mothers adopting food hygiene practices (covered food storage, safe drinking water storage, water treatment before drinking, hand washing before food preparation and eating) 0.0% 0.0% 0.3% * Percent of households having a score of at least 70% on ten hygiene practices 1.9% - 6.0% * * Baseline data were not suitable to perform a significance test (see page 7) MYAP Final Evaluation Report 31 July 2012 50 There is also a significant increase in the proportion of households using an improved toilet (ventilated with a cover, cleanable, and not shared). However, while there is a threefold increase (from 6.8 to 21.5%), much remains to be done in the area of sanitation. As mentioned by people interviewed during the evaluation, whether volunteers or caregivers, the cost of latrine construction is prohibitive. Overall, achievements in hygiene as displayed by a composite indicator - percentage of households having a score of at least 70% on ten hygiene practices – are below the target set at 30%. Relative to waterpoint habitation, program monitoring systems indicate that 4,066 households with 20,330 people are benefitting from the 40 water points rehabilitated. The program took before and after readings on water flow at the same time of the year, and 33 of the 40 sites showed increased flows. Readings for the seven sites not reporting increases showed the same levels of flow before and after; and in 10 of the 40 sites, the water point was non-functional at least part of the year before the rehabilitation. The supply of water has greatly increased in these sites. Observations during the evaluation indicated that leakages in a few sites are reducing flows through pipes, and in one site, respondents interviewed emphatically reported that flow through the pipe was less than before the rehabilitation. Relative to water quality, two water quality assessments over the life of the program indicated consistent good findings on key indicators, including presence of coliform bacteria, turbidity, odor and taste, although in qualitative interviews at least one site showed high levels of turbidity. All in all, the MYAP has achieved substantial impact on access and supply to potable water. The lone indicator in the IPTT related to water supply, however, does not provide baseline information for watershed areas where the MYAP water point interventions were implemented. Baseline data was collected on this indicator as part of the SO1 survey which sampled at the provincial level, while the program intervention occurred in the watershed colines. The sustainability of the impact achieved by the MYAP rests fully on the shoulders of the water committee and its effectiveness in operating and repairing the water points. On operations, all committees at the moment were working effectively. In one province, however, there is tension between the water committee who do all of the cleaning at the water point and the rest of the users of the water point that may result in problems at some point. On maintenance and repairs, where committees have been firmly linked to colline and commune administration, water points are more likely to be sustained. This was done well in one province, satisfactorily in another province, but weakly in the third province. 6. Comparison of SO1 Outcomes Between Watersheds and the Province. Comparison between areas where SO1 activities are stand alone with areas where SO1 activities were implemented with SO2 and SO3 in some watershed collines has yielded mixed results. For some indicators, benefits were, as would be expected, higher in the SO2/SO3 areas, including:  Higher proportion of children having received vitamin A (17% as compared with 13%; p=0.0145)  Higher average number of prenatal consultations (3.4 as compared with 3.2; p=0.000)  Higher proportion of women breastfeeding within one hour after delivery (91% as compared with 82%; p=0.000)  Higher proportion of women giving colostrum (98.9% as compared with 97.2%; p=0.011) MYAP Final Evaluation Report 31 July 2012 51  Higher proportion of women who increased breastfeeding to their child suffering from diarrhea (36.2% as compared with 21.6%; p=0.005)  Higher proportion of women who increased fluids to their child suffering from diarrhea (59.3% as compared with 48.3%; p=0.0017)  Higher proportion of women who sought health care when their child suffered from diarrhea (62.7% as compared with 47.6%; p=0.001)  Higher proportion of women who sought health care when their child suffered from coughing (69.8% as compared with 47.2%; p=0.000)  Higher proportion of women who sought health care when their child suffered from diarrhea (62.7% as compared with 47.6%; p=0.001)  Higher proportion of women who sought health care when their child suffered from fever (87.3% as compared with 72.9%; p=0.000)  Higher proportion of households storing drinking water adequately (93.3% as compared with 77.0%; p=0.000)  Higher proportion of households having a hand washing spot (65.7% as compared with 55.5%; p=0.000)  Higher proportion of households using improved latrines (26.7% as compared with 21.5%; p=0.006)  Higher proportion of households eliminating child faeces in a hygienic manner (91.6% as compared with 82.2%; p=0.000)  Higher proportion of households who practice adequate household garbage disposal (95.3% as compared with 92.2%; p=0.005)  Higher proportion of households storing food adequately (98.7% as compared with 87.7%; p=0.000) For some other indicators, however, results were surprisingly better in the stand alone SO1 areas:  Higher proportion of women reported having adopted a proven FP method (37.8% as compared with 27.7%; p=0.000)  Higher proportion of women reported having taken an iron supplement after their last pregnancy (18.9% as compared with 15.7%; p=0.000)  Higher proportion of women having their child immunization card reported (65.8% as compared with 62.6%; p=0.084)  Higher proportion of children having participated at least once in GM (58.4% as compared with 53.4%; p=0.020)  Higher proportion of children having attended in GM over the last three months (22.0% as compared with 17.1%; p=0.005)  Higher proportion of households having at least one mosquito net (68.7% as compared with 58.7%; p=0.000)  Higher proportion of children who slept under a mosquito (61.2% as compared with 54.0%; p=0.002)  Higher proportion of women who slept under a mosquito net (55.9% as compared with 51.3%; p=0.031) MYAP Final Evaluation Report 31 July 2012 52 D. Outcomes Under SO2: Livelihoods While the current impact of all SO2 activities is high as most activities are of private utility. The main weakness is that of social and financial sustainability over the longer term for those activities with more of a public utility, such as watershed management, as the project did not incorporate exit strategies from the beginning and is attempting to consolidate its achievements only in the last year during the NCE. The IPTT specified one impact indicator for SO2, diet diversity, and data from the household survey is shown in Table 20. The baseline data was collected during an annual period of food security at the end of the calendar year, and the endline data was collected during an annual period of food insecurity in March, so a valid endline-baseline comparison cannot be made. However, it can be noted that the endline difference between non-vulnerable and vulnerable households is only 0.3 (10%) rather than 0.9 (20%) in the baseline, suggesting that vulnerable households eat now relatively better than their peers, even in hard times. . Table 20. Diet Diversity Score*27 Type of Household Baseline 2008 Endline 2012 Statistical N Groups N Groups Significance All Households 845 4.6 577 3.5 Significant p = 0.000 Vulnerable Households** 170 3.8 209 3.3 Significant p = 0.000 Non-Vulnerable Households 675 4.7 368 3.6 Significant p = 0.000 *Food Groups out of 12 Eaten in the Last 24 Hours **Defined as Women-Headed Households, Orphan-Headed Households or recently Repatriated Households There are other general observations concerning SO2, most of which are not covered by the surveys and would need to be confirmed through more detailed studies by the M&E team: 1. There is a good amount of knowledge and product dissemination inside the MYAP collines and a small amount spreading to neighboring collines. 2. While there is no quantitative information on farmers’ adoption of new practices, it is clear that most farmers have adopted many and are still learning by observing the results in others fields. While CRS records indicate that generally 30-50% of women were beneficiaries, the actual results are likely to be higher as often transactions were registered under the name of the head of household. However, women have clearly been more involved with onward knowledge and product dissemination. 3. Although the survey indicates that globally 12% of families (10% of vulnerable ones) have been able to increase their land ownership, many beneficiaries mentioned buying land. The socio-economic impact of this redistribution of land assets is unclear. Only an M&E team study can answer this question. 27 For future evaluations, it would be useful to analyze the statistical significance of the difference between vulnerable and non-vulnerable households for each survey. If the difference is not significant at the endline, but the difference was significant at the baseline, this might suggest that the program did well at targeting vulnerable households and closing the gap between vulnerable and non-vulnerable households. MYAP Final Evaluation Report 31 July 2012 53 4. There is now a larger demand for paid labor as those with larger plots use more at peak times although those with smaller marais plots use less as the soil is now easier to work, but this could not be substantiated from the household survey. 5. The enforced stabling of project goats and the ban on animals in the marais has had a significant positive impact on social relations as there are now few cases of farmer/herder conflicts in a society where the majority of people are both farmers and herders. 6. Most groups feel equipped and motivated to continue by themselves, save for a few committees, mostly marais and anti-erosion who resent the weight of their responsibilities without enough support from the authorities and users. 7. The most vulnerable were prioritized for goat, cassava and sweet potato distributions, the seed and livelihood fairs and the FFW in the marais and for the contour berms. 8. Both women and men were highly appreciative of the positive impact on gender roles and relationships. 1. IR 2.1 Farm Production. There is generally good agreement between the household survey results as shown in Table 21, the interviews and the group discussions, to establish that farm production has increased significantly and that people are generally eating more, but it cannot be ascertained whether they eat better. Some data in the survey could not be used to determine statistical significance (at the 95% confidence level). Table 21. Agricultural Practices (Note: The preferred direction is an increase from baseline to endline, unless otherwise indicated) Indicator Baseline 2008 Endline 2012 Statistical Significance Reported ISTEEBU 1 Percent of Households Using Improved Seed 16% 16% (N=845) 56.5% (N=577) * 2 Percent of Households Growing Vegetables 31% 20% (N=845) 17.5% (N=577) * 3 Percent of Households Using Erosion Control Practices 56% 56% (N=845) 82% (N=577) Significant p = 0.000 4 Percent of Households Planting Trees 52% 52% (N=845) 72% (N=577) Significant p = 0.000 5 Percent of Households Using Compost 86% 55% (N=845) 60% (N=577) Significant p = 0.025 6 Percent of Households Using Green Manure or Cover Crops 0% 14% (N=845) 16% (N=577) Not significant (p=.157) 7 Percent of Households Using Pest and Disease Control Practices 11% 11% (N=845) 18% (N=577) Significant p = 0.000 8 Percent of Households with Animals 60% 60% (N=845) 70% (N=577) Significant p = 0.000 9 Percent of Households with Goats 42% 42% (N=845) 48% (N=577) * 10 Production Sold After Harvest 66% 66% (N=845) 57% (N=577) Significant p = 0.000 11 Production Sold Before Harvest (preferred direction is a decrease) 16% 16% (N=845) 9% (N=577) Significant p = 0.000 12 Percent of Households Storing Harvest 6% 6% (N=845) 14% (N=577) Significant p = 0.000 MYAP Final Evaluation Report 31 July 2012 54 13 Average Number of Animals per Household 14 Cattle .2 .2 (N=845) 0.5 (N=577) * 15 Sheep <.1 <.1 (N=845) 0.4 (N=577) * 16 Goats .9+ .9+ (N=845) 1.6 (N=577) * 17 Pigs .1 .1 (N=845) 0.3 (N=577) * 18 Poultry .6 .6 (N=845) 0.7 (N=577) * 19 Rabbits .2+ .2+ (N=845) 0.3 (N=577) * 20 Other .2 .2 (N=845) 0.3 (N=577) * 21 Percent of Households Giving Supplemental Feed to Any Livestock 32% 32% (N=504) 75% (N=403) * 22 Percent of Households Giving Supplemental Feed to Goats 17% 17% (N=??) 75% (N=280) * 23 Percent of All Households Producing Fodder Crops 49% 49% (N=845) 71% (N=577) Significant p = 0.000 24 Percent of Households Who Have Goats Who Vaccinate Their Animals 13% 13% (N=??) 32% (N=280) * 25 Percent of Households Who Have Goats Who Seek Curative Treatment for Their Animals 74% 74% (N=??) 54% (N=280) * 26 Percent of All Households Cultivating Plots in the Lowlands 62% 62% (N=845) 65% (N=577) Not Significant p=0.156 27 Percent of Vulnerable Households Cultivating Plots in the Lowlands 46% 56% (N=170) 58% (N=209) Not Significant p=0.347 28 Percent of Non-Vulnerable Households Cultivating Plots in the Lowlands 64% 64% (N=675) 69% (N=368) Significant p=0.05 29 Percent of Households Reporting Valleys Undeveloped and Prone to Flooding (preferred direction is a decrease) 27% 44% (N=524) 27% (N=373) Significant p = 0.000 30 Percent Households Reporting Valleys With Water Management Infrastructure 36% 54% (N=524) 74% (N=373) Significant p = 0.000 31 Percent of Households reporting Developed Valleys That are Poorly Maintained (preferred direction is a decrease) 18% 27% (N=524) 32% (N=373) Significant p = 0.000 32 Participation in Agricultural Associations 33% 24% (N=845) 21% (N=577) Not Significant p=0.116 * Baseline data were not suitable to perform a significance test (see Page 7) The household survey indicates ambiguous or contradictory information in some cases where what is measured is more general or not a project specific activity. For example, for Row 2 in Table 21, the project did not have a specific vegetable growing activity. For Row 6, most fields are in constant use, and for Row 9, while the project gave goats to many participants, many others also purchased some goats on their own, as can be corroborated from Rows 14 and 15, MYAP Final Evaluation Report 31 July 2012 55 which are not a project livestock activity but indicate that an increase in livelihood has led to more keeping of larger animals. For Row 25, it is surmised that as more goats are vaccinated, they are less in need of curative treatment. It is also possibe that if people did not need to seek curative treatment because their goat was not sick, this response was a negative response to the question. For Row 27, vulnerable households now use 70% more area in the marais. For Row 31, Rows 29 and 30 denote a positive project impact while Row 31 seems to be a contradiction, but it, in fact, corroborates the fact that while there is now more infrastructure, it is also more difficult to maintain. 2. IR 2.2 Revenues and Livelihoods. The household survey, qualitative interviews and focus groups all indicate that while the great majority of collines dwellers now have a better quality of life due to project activities, the survey corroborates the impression that the originally better-off socio-economic class has benefitted more, as measured in the information on men and non￾vulnerable households. This is not a criticism of the project, but a natural outcome of proportional resource optimization. The poorer segments have benefitted more on a relative scale but the better-off segment has benefitted more on an absolute scale. The survey reveals specific activities where women and vulnerable households have benefitted at least as much as the others: such as in the use of compost and manure, treating diseased plants, applying chemical fertilizer (not a project activity), increasing the use of plots in the marais (+70%), delaying crop sales, accessing SILC, increasing their revenues from crop and animal sales, reducing their food insecurity and training attendance. The very positive impact on livelihoods is partly due to on-farm activities but also due to a synergetic effect of linkages to micro-credit activities which have the double economic benefit of both multiplying positive on-farm effects and softening crises by offering a much less negative impact when a short term (less than three month) coping strategy is needed. Quantitatively, the three indicators of success are: changes in household expenditures (Table 22), changes in household income (Table 23) and changes in the household food sources (Table 24). Table 22 from the survey indicates that fewer households are reporting expenditures on food as a major expense. Qualitative interviews indicated that households are able to produce more food, which is also consistent with the significant increase in the number of households reporting expenses on agricultural inputs as a significant expense. Qualitative interviews also indicated that households are able to use more income for non-food expenses, which is also consistent with the evidence in Table 22 showing a statistically significant increase in households reporting expenses on clothes and social activities. Table 23 is ambiguous as to project impact on household income sources and would require a more detailed study to interpret especially for crop sales and whether the increase in trade is due to more SILC activities. The information shown in the table was made available to the evaluation team after the qualitative field work was completed, so the evaluation team was unable to use the qualitative interviews to obtain information to aid in interpeting the quantitative data. Table 24 reveals the very positive impact of the project on food sources and is consistent with qualitative interviews in which respodents indicated that they were able to produce more food. It should also be noted that the results for tables 22 to 24 indicate good project impact even though the endline survey was taken during a traditional peak time of food insecurity while the baseline was taken during a traditional peak time of food security MYAP Final Evaluation Report 31 July 2012 56 Table 22. Five Household Expenses Most Mentioned Indicator Baseline 2008 Endline 2012 (N=577) Statistical Significance Reported ISTEEBU (N=845) Food 96% 58% 50% Significant p = 0.000 Health Care 84% 27% 24% Not Significant p=0.141 Clothes 77% 5% 9% Significant p = 0.005 Agricultural Inputs 58% 4% 14% Significant p = 0.000 Others (Land Rent, House Construction, Education) 35% 35% N/a N/a Social Activities 7% 0.4% 1.6% Significant p = 0.005 Table 23. Household Income Sources Indicator Baseline 2008 Endline 2012 (N=577) Statistical Significance Reported ISTEEBU (N=845) Crop Sales After Harvest 72% 72% 63% Significant p = 0.005 Crop Sales Before Harvest 5% 5% 16% Significant p = 0.001 Daily Work 56% 56% 50% Significant p = 0.022 Animal Sales 19% 19% 20% Not Significant p=435 Trade 14% 14% 18% Significant p = 0.05 Manufacturing 4% 4% 3% Significant p = 0.05 Salaries 3% 3% 4% Not Significant p=0.26 Rent Out Land 2% 2% 4% Significant p = 0.005 Selling Wood or Charcoal 1% 1% 4% Significant p = 0.000 Selling Food Aid 1% 1% 0% N/a Table 24. Household Food Sources Indicator Baseline 2008 Endline 2012 (N=577) Statistical Significance Reported ISTEEBU (N=845) Purchased Food 97% 38% 23% Significant p = 0.000 Own Production 97% 61% 77% Significant p = 0.000 Other forms of food sources could not be measured reliably E. Outcomes Under SO 3: Community Resilience 1. IR 3.1 Early Warning and Response. The evaluation found some very good initiative being undertaken by the PAC Committees in prioritizing community needs and mobilizing resources. MYAP Final Evaluation Report 31 July 2012 57 The school additions are increasing access to education services, although the number of teachers in schools is a bigger constraint. One PAC Committee is building housing for teachers which may increase the number of teachers. The widened roads and bridges are increasing access to basic services and markets. The only impact indicator in the IPTT for SO3 is a food provisioning indicator which is heavily influenced by SO2 activities and SO1 food distribution activities, making it a relatively poor indicator for capturing SO3 impact. Table 25 shows average months of adequate household food provisioning capacity, comparing baseline information to endline information from the household surveys. In conducting the statistical significance tests for the baseline-endline comparisons, ISTEEBU did not exclude any records from the baseline data to ensure an accurate comparison. The results of the household survey were made available after the evaluation team had already completed field work, so it was not possible to obtain qualitative information to assist in explaining the unusual result shown in the table with vulnerable households having higher food provisioning capacity than non-vulnerable households. Possible explanations could be related to good targeting by the program in ensuring that vulnerable households are prioritized in SO2 activities. They were likely to have been disproportionately represented in any case by food distributions under SO1 and FFW under SO2. Table 25. Average Months of Adequate Food Provisioning Capacity Type of Household Baseline 2008 Endline 2012 Statistical Significance Reported ISTEEBU Months Months Months All Households 6.9 (N=722) 7.1 (N=845) 9.0 (N=577) Significant p = .000 Vulnerable Households* 6.2 (N=92) 7.1 (N=170) 9.2 (N=209) Significant p = .000 Non-Vulnerable Households 7.0 (N=630) 7.1 (N=675) 8.9 (N=368) Significant p = .000 *Defined as Women-Headed Households, Orphan-Headed Households or recently Repatriated Households Sustaining impact achieved by the PAC Committees under SO3 beyond the life of the MYAP will depend on the motivation and institutionalization of the Committee. The motivation appears to be good and the composition of the committee is representative and empowered with local leadership. The institutionalization is threatened, however, since the PAC Committee parallels the CDC at the colline level. The PAC Committee is also not fully recognized by the Commune administration as a body that they can use for their own community development planning and implementation. Lesson Learned To enhance the institutionalization of the PAC Committee, it may have been better to build the capacities of the existing CDC. Starting with an existing structure like the CDC and reforming or revitalizing it takes more time. Results can certainly be achieved more quickly by starting with a new committee. However, the long-term sustainability of the impact achieved by a program is more likely to be achieved if the good capacity building done by a program is fully integrated into formal institutional structures. MYAP Final Evaluation Report 31 July 2012 58 2. IR 3.2 Asset Management/Gender Equity. As indicated in a previous section, 70% to 80% of households in the twenty-one watershed collines have heard the gender messages. Qualitative interviews suggest that as many as 40% to 50% of these have either changed significantly or have started to change toward more gender balanced management of household resources. This represents a significant impact on an estimated 4,200 to 6,000 households. There are a multitude of benefits being realized by these changes. The benefits specifically for women include enhanced dignity associated with the new respect they are getting from husbands for the work that they do for the household and being included in joint decision-making with the husband. They also feel more empowered from being able to join and participate in associations. Many women interviewed also cited that the peace in the household and reduced tension have improved the quality of life. They no longer feel threatened and have to stay outside when quarrels with their husbands escalate to potential violence. Even being able to sleep better at night since there are no quarrels has had benefits in giving women (and men) more energy and better health. The workload for women has also been reduced since men are now helping with household chores and there is more balanced division of labor. Both men and women reported that they have registered their marriages with the local administration. In some cases, husbands and wives have been together for years without being officially registered as married, and now wives are entitled to the benefits of a legal marriage. For husbands, two important benefits emerged from interviews during the evaluation. These were (1) the enhanced dignity that they felt from the respect that they were now receiving from wives and children and (2) the household attaining economic benefits from using household resources more wisely for the benefit of the household and from the shared workload producing better returns on livelihood activities. These economic benefits not only improved the quality of life for the household, but also increased the image for the male household head, recognized now for having more successful livelihoods. When asked whether the gender-balance changes implied that they had less power in the household, nearly all respondents replied that they were still the household head and still had power, but their power came from understanding and respect and not from fear and intimidation as in the past. They also expressed appreciation for the peace in the household, not having to prepare for quarrels when they came home. Other members of the household, or the household as a whole, also benefitted from the gender￾balance changes Children are being put back into school because joint decisions are made to do so and both husband and wife contribute toward meeting school costs. Similarly, household members are getting better health care since joint decisions are being made to use resources for health care and either adult, not just the mother, accompanies children taken for treatment. As mentioned, households are accumulating assets, typically more livestock or land, and with both husbands and wives participating in associations, benefits from this participation have increased. The work on both household chores and livelihoods activities is done more efficiently since everyone contributes and no one is overloaded. Children were reported to be happier, no longer Lesson Learned The impact that has been achieved in the MYAP on changing gender roles has been accomplished through a combination of (1) effective identification of Acteur Relais, (2) good training on different dimensions of gender, human rights and conflict mediation, (3) certification of Positive Deviants willing to give testimonials, and (4) all of this in the context of other MYAP interventions that provide opportunities to practice joint decision-making and changed roles. The absence of any of these four elements would likely have reduced the impact. MYAP Final Evaluation Report 31 July 2012 59 running to hide when father comes home and an argument with mother is expected. Finally, in the past, when these arguments disrupted other families, the household could be fined (usually beer provided to neighbors) by the chef for the disturbances caused. This is no longer happening. Respondents in interviews also described benefits for the community as whole, including neighbors sleeping better and the neighbors and administration no longer having to mediate disputes. The gender-balance messages are also starting to diffuse outside of the twenty-one watershed collines. Around 10% of Acteurs Relais or Positive Deviants could cite specific people living outside the program area to whom they had told of their gender balance experience or even counseled. The baseline and endline surveys obtained information on three indicators to capture the impact of gender-balance activities as shown in Table 26. While the data does provide clear evidence of improved decision-making and participation by women, the evidence is not quite as striking, Table 26. Household Survey Impact Indicators for IR 3.2 Gender Equity Indicator Definition 2008 Baseline 2012 Endline Significance Reported ISTEEBU Percent of households with a score of five out of five on five different types of household decisions. Agricultural investments Food expenses Social expenses Sales of crops or livestock Decisions about credit 46.0% 40.5% (N=845) 47.3% (N=577) Significant p = .005 Women's participation in different types of associations Development Committee Not Reported 2.0% (N=275) 13.0% (N=352) Significant p = .000 Credit & Savings Group .9% (N=275) 29.5% (N=352) Significant p = .000 Water Management Committee .2% (N=275) 4.5% (N=352) Significant p = .000 Farmers or Livestock Producers Association 19.4% (N=275) 21.7% (N=352) Not Significant p = .15 Women's Association 2.9% (N=275) 7.3% (N=352) Significant p = .000 Changes in the participation of women in agricultural meetings Percent of meetings to which women are invited and percent of women participating in meetings once invited Not Reported 21.2% (N=275) 47.8% (N=352) Significant p = .000 Lesson Learned Additional impact indicators for capturing the impact of gender balance-type interventions could include the following:  Changes in the number of cases of internal household conflict heard by a chef du colline or by a tribunal  Changes in school enrollment (although certainly other factors affect this indicator  Changes in marriage registration for mature marriages  Changes in time spent on household chores disaggregated by sex  Changes in feeling respected by others in the household disaggegrated by sex MYAP Final Evaluation Report 31 July 2012 60 especially on household decision-making, as would have been expected from the qualitative interviews. Sustaining the impact achieved by the program depends on whether or not households will revert back to their previous behavior after the program ends. Respondents in qualitative interviews emphatically responded that the benefits they are realizing will ensure that they do not revert. They also said that the Acteur Relais and Positive Deviants will continue working. Acteur Relais and Positive Deviants continuing to spread messages and conduct household counseling depends on their authority and motivation. On the former, the Acteur Relais and Positive Deviants are authorized to conduct their work when husbands or wives from troubled households request their assistance or when local leaders requesting their assistance and often accompany them when they provide counseling support. The withdrawal of the informal authorization provided by the MYAP should not adversely affect their work. On their motivation, all of those Acteur Relais who were interviewed during the evaluation expressed great pride in being recognized as having created positive change in their communities. They obtain a large measure of personal satisfaction for having enabled households to achieve a better quality of life through the awareness- raising and counseling they have provided. F. Collective Impact at the Program Goal Level Table 27 summarizes previous discussions on the various outputs produced by the MYAP and clusters these outputs according to the level of impact being achieved, in the judgement of the evaluation team. Some outputs, as shown in the first column, are producing good impact and the impact is likely to be sustained beyond the life of the program, based on discussions with program beneficiaries and intermediaries. The second column in the table lists those outputs that are producing impact, but the impact will likely diminish over time or the number of beneficiaries is relatively small. The impact for outputs listed in this column is generally less than the impact the program expected or proposed to achieve. The third column indicates outputs that are producing relatively little impact because the program made little progress in implementing activities or the activities implemented produced very little impact on food insecurity. It is worth noting, as indicated in Table 27 with no outputs listed in the last column, that the MYAP is achieving impact in all activities being implemented by the program. In interviews conducted during the evaluation, program beneficiaries cited the marais development, the goat solidarity chains, the SILC approach and the gender-balance interventions as those having the greatest impact; and some of these, notably the SILC and gender interventions will continue to be sustained and will likely diffuse to other households. The goat solidarity chains also have had substantial impact but at a relatively higher cost than either the SILC or gender activities. The marais development is highly visible, but in the absence of better marais management capacity building, these benefits will likely diminish somewhat over time as the infrastructure wears down. Impact that is dependent on food distributions, especially the therapeutic and supplemental feeding under SO1, will begin to diminish in the absence of distributed food. Only in the case of therapeutic feeding for out-patient children, in which UNICEF is expected to continue making ready-to-use-therapeutic food available, will the impact be sustained. MYAP Final Evaluation Report 31 July 2012 61 Table 27. Where is Impact Being Generated in the MYAP? Outputs Producing Good Sustained Impact Outputs Achieving Some Impact but Less than Expected, Small Beneficiary Numbers or Impact Not Likely to be Sustained Outputs Achieving Little or No Impact SO1/IR 1.1: PD/Hearth SO1/IR 1.1: Mother Care Groups SO1/IR 1.1: Community Management of Acute Malnutrition Outpatient Services SO1/IR 1.1: Growth Monitoring Services SO1/IR 1.2: Nutrition Education for PLHIV SO1/IR 1.1: Community Management of Acute Malnutrition Inpatient Services SO1/IR 1.3: Health Education for Caregivers SO1/IR 1.1: Supplemental Feeding Services SO1/IR 1.2: Supplemental Food/Gardens for PLHIV SO1/IR 1.3: Capacity Building of Village Health Workers SO1/IR 1.4: Water Point Rehabilitation SO1/IR 1.4: Participatory Hygiene and Sanitation Transformation (PHAST) SO2/IR2.1: Livestock Management SO2/IR 2.1: Marais Management SO2/IR 2.1: Agricultural Technology Transfer SO2/IR 2.1: Anti-Erosion Capacity Building SO2/IR 2.1: Marais Infrastructure Rehabilitation SO2/IR 2.1: DPAE Capacity Building SO2/IR 2.2: Savings & Lending Internal Communities (SILC) SO2/IR 2.2: Agro-Enterprise SO3/IR 3.2: Gender Equity in Managing Household Assets SO3/IR 3.1: Drought Mitigation Capacity Building SO3/IR 3.3: Knowledge of Women's Rights SO3/IR 3.1: Community Development Action Planning The MYAP’s goal is to reduce chronic and transitory food insecurity in the three provinces. Nutritional status of children under five years is summarized in Table 28. The comparison of Table 28. Nutritional Status of Children Under Five Years of Age Indicator Baseline 2008 Endline 2012 Statistical Reported ISTEEBU Significance Chronic Malnutrition (Stunting, height/age <= -2 SD) 56.0% 31.0% (N=902) 32.7% (N=1242) Not Significant p = 0.202 Acute Malnutrition (Wasting, weight/height <= -2 SD) 6.5% 7.0% (N=902) 5.8% (N=1231) Not Significant p = 0.1265 Underweight (weight for age <= -2 SD) 40.7% 36.6% (N=902) 29.7% (N=1240) Significant p = 0.0005 MYAP Final Evaluation Report 31 July 2012 62 endline to baseline data from the household survey for the three major nutritional status indicators shows a small non-significant decrease in the prevalence of wasting from 7 to 5.8%, a small non-significant increase in chronic malnutrition from 31.0 to 32.7%, and a significant decrease in underweight from 36.6 to 29.7% (p=0.0005). The LOA target for underweight was set at 30% and that of stunting at 50%. However the stunting LOA target is no longer valid since the baseline stunting prevalence figure of 56% was reduced to 31.0% using the newer anthropometric analysis software version. As mentioned, the program has had impact on an estimated 225,000 children under five years of age, 60,000 caregivers for these children, and 15,000 households in the watershed collines. These numbers are relatively close to the proposed targets of 125,000 households at the provincial level and 18,000 households in the watershed collines. In general, the program has had satisfactory impact across all of SO1, good impact across SO2 and good impact across SO3. Although the MYAP has not specified impact indicators at the goal-level, it can be assumed that the program has achieved good impact collectively toward the goal. V. PROGRAM PROCESS ASSESSMENT A. Program Management Program management investigations in final evaluations typically look at management structures, vision and leadership, program planning, problem-solving and decision-making, communications, logistics and procurement to identify good practices and lessons learned. Relative to major observations on management from the evaluation of the MYAP, the senior management team within CRS (Country Representative, Head of Programs and Management Quality Coordinator28) had relatively high staff turnover, especially early in the life of the program. The vacancy in the MQC position at the beginning of the program affected the pace of the startup. Less important but still a factor at start-up, the vacancy in senior program positions meant the program did not have experienced support for operationalizing program concepts committed to in the proposal. Following the Mid-Term Evaluation, program managers made a deliberate effort to facilitate better cross-fertilization between SOs and instituted monthly field coordinating meetings involving all SOs. In addition, full Coordination Meetings usually held twice per year were highly appreciated by staff for enabling them to understand other components of the program and finding ways to facilitate better synergies. Every program that is being implemented by different partners in different locations has the challenge of ensuring technical quality and facilitating cross-fertilization between partners on technical approaches. The MYAP uses a “Mobile Team” composed of CRS staff based in Kirundo Province but who travel extensively to other program areas. Emphasis has been put on ensuring that they are “mobile”, with regular visits to provide technical support to the other provinces, and this model has worked well. The opportunity for CRS to implement directly in Kirundo Province, while somewhat contrary to CRS policy worldwide to work through partners, has provided certain advantages. The direct 28 The Management Quality Coordinator position oversees program support functions in the country office, including financial management, procurement, inventory management, logistics and communications. MYAP Final Evaluation Report 31 July 2012 63 delivery component allows CRS to have direct control over the quality of the program with a “laboratory“ for testing new approaches. The direct delivery also enables CRS to have direct, continuous contact with targeted impact groups to stay abreast of the changing context and the needs and interests of participants. In 2011, the program replaced a CRS Provincial Program Coordinator in Muyinga Province with an ODEDIM Provincial Coordinator. This transition to full partner management was appreciated by implementing staff since they now clearly reported to someone in their organization and not to someone in another organization. In Kayanza Province, the existence of the major commodity warehouse in Ngozi has been the basis for having a CRS Provincial Coordinator. Relative to logistics, only one location indicated that implementation challenges existed because of vehicle/motorcycle shortages early in the program, but these challenges were subsequently resolved. Although the program has relatively few female staff, the motorcycles procured for the program are too large for encouraging female staff to acquire driving skills. Every program has procurement complaints, and these do not to appear to be excessive in the MYAP. Overall, the management functions in the MYAP have been handled very well over the life of the program. Despite an unusual management structure in which the two full-time expatriate senior managers in the MYAP, the Program Coordinator and the Field Coordinator, report to the CRS Head of Programs29, management functions for the MYAP at the program level are being fulfilled well. There is no confusion in the program as to who is responsible for the overall vision and leadership of the program (the MYAP Coordinator). The two senior positions have complementary management styles and communicate well. The lesson learned may be that organizational structure is less important than the individuals occupying positions in the structure. B. Partnership Investigations around partnership look at the different relationships that exist between organizations implementing or providing significant technical support in the program. For the MYAP, these organizations include CRS as the grant recipient and implementing partner in Kirundo for waterpoint rehabilitation, SO2 and SO3 activities. ODEDIM is the implementing partner in Muyinga for these same components, and BADEC is the implementing partner in Kayanza for these components. IMC is the implementing partner in all three provinces for SO1 activities. The BPS is a significant technical and institutional sustainability partner for MCHN in all provinces, and the DPAE is a significant technical and institutional sustainability partner for agricultural activities. The Commune Administration in each of the four communes where the program is being implemented is also a key partner for institutional sustainability for water points, community action planning and gender balance impact. At the beginning of the program, some organizational culture clash occurred between CRS and IMC as they began implementing the program. IMC uses more centralized decision-making, which required more processing time than the CRS systems. The bugs were worked out of the system, however, and the partnership at the time of the final evaluation was quite strong. The 29 The structure is required since the MYAP/PAPSAD Coordinator and the MYAP/PAPSAD Field Coordinator are a married couple. MYAP Final Evaluation Report 31 July 2012 64 delegation of SO1 responsibility entirely to IMC also minimized organizational culture clash. If both CRS and IMC were responsible for SO1, there likely would have been more clashes. The Mid-Term Evaluation prompted better working relations between partners, reducing the isolation between different SOs, and making relationships feel more like real partnerships. Having local partners in the MYAP has been effective, since local organizations have roots in the community; they know the context; working with them also builds local capacities; and the potential for sustaining impact is greater. While there are certainly occasional hiccups in relationships, everyone on both sides of the relationship between CRS and local implementing partners reported that the relationships were good. The Grant Management Unit in CRS has been effective at providing capacity building support for local partners, especially on finance and grant management. With government partners, the relationship between the program and BPS is good, mainly since IMC has experience working with the government health sector. Staff turnover in government positions has raised occasional challenges. With the DPAE, the relationship is less strong. They are aware of the program, but they do not feel strong ownership in the program approaches or structures, except in Kayanza Province. With the Commune Administration, the relationships are strong in each province, but the commitment of the Administration to sustaining program activities could be stronger and varies between provinces. All in all, the NGO partnerships in the MYAP are fully functional and effective. Relative to government partners, the BPS, DPAE and Commune Administrations are all fully informed on program activities, but strong commitments to sustaining program impact after the program ends have not been cultivated. Building this commitment with government begins with fully engaging them in program design and ensuring that the design addresses their priorities and interests. The way that MYAPs are currently designed, within a competitive bidding business model that is fairly prescriptive and highly timebound, however, makes it difficult to fully engage key stakeholders in a good design process. The staff turnover within government also makes it difficult to maintain strong commitments with individuals who may have been part of the original design. C. Knowledge Management Knowledge management generally refers to how knowledge is brought into a program, how it is generated and used within a program and how it is generated and disseminated outside of a program. New ideas and approaches that are brought in, tested and adapted by a program represent “knowledge in”. M&E systems are designed to obtain and use information within the program to make decisions to improve the efficiency or effectiveness of the program, and systems for capturing best practices and lessons learned represent “knowledge out”. Lesson Learned In a consortium program, startup tension can be alleviated by recognizing that organizations have different systems and cultures. Rather than the consortium lead imposing its system, partners should work together from the beginning to develop processing systems for finance and commodities that are compatible with the systems of each of the consortium partners. It is also important to recognize that those organizations with ore centralized decisionmaking are less time efficicent and will require more time to process information and make decisions. MYAP Final Evaluation Report 31 July 2012 65 Relative to knowledge in, the MYAP has benefitted from a wide variety of external training and consultancies bringing new ideas and skills into the program. At least twenty-five specific instances, listed in Annex F, were identified in which the program benefitted from external expertise from consultants, other CRS offices, CRS Technical Advisers or from having sent program staff outside for exposure to new ideas. Relative to Monitoring and Evaluation, the MYAP has good capacity for undertaking monitoring and evaluation activities. Within CRS, there is an M&E team with four positions dedicated to the MYAP. The mobile team based in Kirundo Province provides technical support and monitors technical quality in all three provinces, and provincial management staff spend significant time in the field monitoring activities and providing support. The program also has various data bases documenting participation of beneficiaries in specific activities. Planning provides the basis for good monitoring, and many respondents in interviews during the evaluation expressed appreciation for the way the program plans activities from the frontline to the program level. The program implemented a baseline survey in late 2008 that has provided relevant information for comparison with the endline. The Mid-Term Evaluation implemented in 2010 resulted in useful changes to the MYAP strategy. As a result of the evaluation, deliberate efforts were made to facilitate better interaction within the program across SOs. Revisions were made to the Behavioral Change Communications strategy and the Mother Care Group approach was introduced. The evaluation also spurred the shift that had begun in the program away from rehabilitation toward more developmental approaches and expanded attention on gender across the program. A major M&E tool for MYAPs is the Indicator Performance Tracking Table (IPTT). The IPTT for the MYAP has undergone changes twice over the life of the MYAP, after the baseline and again in 2010 as part of the MYAP strategy revision/cost extension. The MYAP IPTT has relevant types of indicators, although the indicators may not all be placed at the best level and there are some output indicators missing. For example, there are no goal level indicators to capture impact across all SOs. The nutritional status of children under five, diet diversity and food provisioning ability (including purchased food) are often found at this level, but they are located at the Strategic Objective level in the MYAP's IPTT. Some significant outputs that are not monitored in the IPTT include the DPAE capacity building, goat distributions, and the services established with the gender acteur relais. An innovation found in the MYAP is the system that has been developed for monitoring the quality of training being provided by the program. A questionnaire is completed by participants before the training begins (the baseline), and the questionnaire is administered again at the end of training (the endline). The results are compared and will be used to revise the training curriculum, give feedback to the trainers or even change the trainers when evidence suggests that the training is less effective than desired. The M&E Unit has also implemented an extensive number of M&E research investigations including an assessment of the MYAP Fruit Trees Activities in September 2010, a Seed Fair Assessment in 2010, Barrier Analysis on Good Hygiene and Feeding in January 2011, an MCHN Knowledge, Attitudes and Practices Survey in November 2011, and internal investigations by the M&E Team on activities associated with the marais development, the Anti-Erosion Committees, MYAP Final Evaluation Report 31 July 2012 66 the PAC Committees, bean variety preferences, rice activities, market gardens, SILC, seed protection and agricultural technology transfer in 2011 and 2012. MYAPs are also responsible for defining and monitoring trigger indicators for prompting decisions on allocating resources for an emergency. Based in part on the FAO early warning monitoring system indicators, the MYAP identified six sets of trigger indicators and specified response levels for each of these for triggering different types of actions, from “alert”, though “mid-level crisis”, “high-level crisis”, to ”disaster” status. Household level data on these indicators has been collected monthly, collated every two months, and then reported to USAID. This represents a very good effort at operationalizing the trigger indicator concept. Regrettably for testing the system (but certainly good for Burundi), there have been no actions triggered by the system to test its efficacy. The evaluation also observed that communications to USAID from data collection to reporting has been going reasonably well, but communications in reverse from USAID to CRS on trigger indicator reporting in the form of feedback or appreciation has been basically negligible. One final comment related to monitoring and evaluation pertains to the final evaluation itself. In the past with other final evaluations, quantitative data from household surveys was collected and summarized before the qualitative assessment occurred. This capitalized on the opportunity to use qualitative information gathering to facilitate better interpretation of quantitative data. Changes always occur in the context that influence baseline-endline comparisons with new programs by other organizations starting up or multiple factors influencing changes that can be discussed with participants. Based on advice from FFP, however, the household survey for the evaluation of this MYAP was scheduled to occur at the same time as the qualitative information gathering which restricted the opportunity to obtain qualitative data to assist in interpreting quantitative findings. Relative to disseminating knowledge out of the program, the MYAP has held a conference and disseminated the report on the Barrier Analysis conducted for SO1 in 2011. The program also shared experience in Provincial Monthly Health Coordination Meetings organized by the BPS and in monthly coordination meetings with the DPAE. Program reports are also provided to the BPS and DPAE, and staff from these partners participate in various trainings organized by the MYAP. Apart from these activities, however, there is no systematic dissemination of knowledge out from the program in which other interested development agencies, donors or government departments receive knowledge generated by the program. D. Program Integration and Complementarity Evaluations analyze program integration and complementarity at different levels, including integration within the program across strategic objectives, complementarity of the MYAP with other programs and projects being implemented in the same geographic areas by CRS, MYAP partners or other organizations, complementarity of the MYAP with government strategies, and complementarity of the MYAP with other US government investments in Burundi. Within the program, the MYAP has achieved significant progress since the Mid-Term Evaluation in reducing barriers between SOs. The periodic coordination meetings and other initiatives have resulted in common beneficiaries in some areas across SOs, for example, lead mothers also participating in SILC groups. MYAP Final Evaluation Report 31 July 2012 67 On coordinating with other NGOs operating in the same geographic areas, there is not much awareness at the field level of these other programs. In some areas, PAC Committees have become involved in water point protection, not with MYAP water points, but with water points rehabilitated by other organizations recently (World Vision and CARE). In supporting government strategies and protocols, the MYAP is following these closely in SO1; government strategies in SO2 are fairly nebulous so there is not much to follow; and the program is working fairly independently with both components of SO3. E. Accountability to Beneficiaries Just as gender and environmental considerations have over time become part of the standard set of cross-cutting issues, so is now accountability, also as part of quality management30. Although accountability was not mentioned as a cross-cutting issue in the proposal, it was examined by the evaluation team. CRS currently does not have an accountability framework, nor a staff has a remit to look after accountability but it has statements and follows through on the code of conduct, the respect of humanitarian principles, in part by the use of Catholic Social Teaching as a guide. In the field, though not done systematically, CRS has been transparent with its partners and with the beneficiaries, particularly through the sharing of reports and the systematic use of public meetings to present itself, plan activities and promote participatory beneficiary targeting. Although there are no set procedures for feedback and complaints handling, beneficiaries have been able to communicate with the Provincial Coordinators directly and project managers frequently supervise in the field. F. Resource Management 1. Financial Resources. The final evaluation of the MYAP examined the financial status of the program, budgeting processes, financial reporting processes, cash flow, and auditing processes to determine if there were any lessons learned or good practices that could be cited. Table 29 shows expenditures through February 2012, representing 89.6% of the program life. Table 29. Cash Expenditure Summary (US$ through February 2012) Cost Center Monetization Proceeds 202e ITSH Total ODEDIM/BADEC 720,949 --- --- 720,949 IMC 2,074,843 733,649 599,743 3,408,235 CRS Direct Costs 1,460,447 620,624 698,532 2,799,603 CRS Burundi Country Office 825,914 134,274 270,619 1,230,807 Total Direct Costs 5,082,153 1,488,547 1,568,894 8,139,594 NICRA for CRS 420,509 88,915 --- 509,424 NICRA for IMC 449,705 149,628 --- 599,333 Total Costs 5,952,367 1,727,090 1,568,894 9,248,351 Current Approved Budget 8,870,654 2,070,231 1,720,769 12,661,654 Percent Spent 67% 83% 91% 73% 30 See both the Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP, www.alnap.org) and the Humanitarian Accountability Partnership (HAP, www.hapinternational.org) MYAP Final Evaluation Report 31 July 2012 68 The MYAP financial reporting system makes available a monthly Budget Comparison Report for the MYAP Program Manager to review and verify that charges have been made appropriately to the program. Quarterly budget reviews are also held within CRS to monitor spending, correct mistakes and make either financial or programmatic adjustments. Despite these good practices, however, the program is significantly underspent, particularly on monetization funds, with nearly 90% of the program life past and only 67% of the budget spent. This problem of the under￾expenditure emerged only in the final year of the program when the price of wheat increased substantially just before the last monetization sale, increasing the sales proceeds substantially above what had been expected. Also in the last year, CRS requested additional 202e and received approval for the costed extension year. Discussions have been underway with FFP on the under-expenditure, and there are some additional suggestions from the evaluation for spending related to recommendations for the remaining life of the MYAP listed in Annex G. Other than the under-expenditure problem, everything else in the MYAP has functioned well with financial management. There were no major complaints from partners on the budgeting process. Cash flow has never significantly disrupted program activities, and audit findings have been fairly insignificant. On reporting, there were some challenges at the beginning of the program while partners acquired understanding on different organizational reporting processes, but reporting is functioning smoothly now. The Grants Management Unit was also effective at monitoring partners and providing support when required. 2. Commodities. Over its life, the MYAP managed a total 32,247 MT of Title II commodities comprised of 26,260 MT of hard red winter wheat for monetization and 5,987 MT of a basket of commodities for direct distribution in therapeutic feeding, supplemental feeding and food-for￾work. All monetization has been completed, and the weighted average cost recovery for monetization over the life of the MYAP was 78.90%. As mentioned above, the last monetization in late 2011 resulted in a higher than expected amount of cash due to an increase in the price of wheat. By the end of the program, it is projected that 5,987 MT of commodities will have been directly distributed, representing 95.3% of approved commodities, leaving a balance of 292 MT. Commodity losses over the life of the MYAP were negligible for monetization commodities (no major ocean losses) and 1.62% for distributed commodities. For the losses on distributed commodities, a little over 21% occurred in transit from the Dar Es Salam port to Bujumbura. The balance of nearly 80% occurred in-country. Annex H contains tables providing details of commodity quantities (Table H-2), monetization cost recovery (Table H-3) and commodity losses (Table H-4). There are only two potential buyers of wheat in Burundi and CRS used a call for bids process to monetize wheat. Forms were developed to record tender submissions, tender bids, sales agreement, contract terms and conditions, and receipt of payments. The weighted average cost recovery rate is reasonably close to the industry standard of 80%. The evaluation found commodity management in general to be thorough and detailed in the MYAP. CRS Burundi established a series of operational and process manuals to ensure standardization of commodity management in such areas as receiving, damage assessment, reconstitution, and delivery reports for the warehouse operations. A commodity tracking system has been developed to monitor the location and status of each call forward until it arrives at the warehouse in Burundi. Manuals and processes are periodically updated and the staff are trained in US food aid commodity management. The CRS’s manuals, tools and procedures used for MYAP Final Evaluation Report 31 July 2012 69 commodity management in the Burundi program meet USAID commodity management guidelines (Regulation 11) and industry commodity management best practices. Commodities for monetization are sold at the port in Dar Es Saaam. Commodities for distribution are transported from the port in Dar Es Salaam to the main CRS warehouse in Ngozi. Losses over the life of the program between the port and the main warehouse have been fairly minimal, a little over 20 metric tons. The CRS warehouse in Ngozi has ample space to accommodate Title II distribution commodities with a storage capacity of 7,000 MT. An industry best practice of a first in/first out method is used in rotating stocks out of the warehouse. Consistent with Regulation 11, upon receipt of commodities at the port and at the warehouse premises, the CRS representative, the freight forwarder representative and the surveyor conduct a delivery survey. Guards patrol the perimeter and guard the front entrance and the premises around the clock. From the main warehouse, commodities are transported by CRS vehicles using three 20 MT, 10 MT and 3.5MT trucks to distribution points as needed, and an accounting system exists and works properly to prevent and account for losses along the chain. At the distribution points the quantities of food are small and the turnover is rapid to prevent food spoilage and minimize losses. Only one significant incident of commodity loss occurred in FY 2010 with commodities found to be spoiled for unknown reasons and CRS took appropriate measures to dispose of the food in conformity with USAID regulations. CRS also conducted investigations to determine the cause of the spoilage, and the best guess that emerged was that high weather temperatures affected the food fortification components. Commodity losses in general over the life of the MYAP were within an acceptable range. In focus group discussions with food recipients, beneficiaries reported that they have always received the quantities they were entitled to and appreciated the quality of commodities. The evaluation did find one FFW focus group within Kayanza province who reported once receiving compromised corn meal quality. The beneficiaries discovered this once they arrived home with the food and did not report the incident to CRS or the implementing partner. 3. Human Resources. The evaluation examined staff recruitment and retention in the program as well as performance management and staff capacity building. At the beginning of the program up until the middle of 2010, CRS Burundi faced a significant number of transitions in senior international staff positions, including the Country Representative (CR), the Head of Programs (HoP) and the Management Quality Coordinator (MQC) position which oversees program support functions for the country office. The most critical vacancy for program startup was the MQC position, which resulted in some delays in procurement and staff recruitment. As already mentioned in the context discussions, CRS had difficulty early in the life of the program recruiting and retaining international staff at senior positions in the country office. Relative to national staff positions, at the moment there are very few positions that are currently vacant in the MYAP, relatively few positions that have turned over in the last year, and a significant number of staff, estimated at around 40%, who have been in their positions since the beginning of the MYAP. Recruitment and retention of staff has not been a problem for the MYAP. A number of reasons are given for this. There is a relatively large pool of candidates from which to recruit, even in the provinces. Compensation and benefits surveys are conducted by CRS and IMC every two years to ensure that packages remain competitive, and, in general, MYAP Final Evaluation Report 31 July 2012 70 there is an organizational commitment to retaining staff and providing a satisfactory work environment. Local partner compensation and benefits packages are less than those for CRS which sometimes causes tension. The lesser of two evils on this issue is probably tension between two organizations rather than tension within an organization from staff in the same organization getting two different packages. A standard performance management process is in place with CRS and IMC and to some degree with local partners. I. Environmental Monitoring and Impact Mitigation Only a few SO1 activities have had a direct effect on the environment. Principally through the hygiene promotion activities, such as the construction of latrines, which when allied to the confinement of animals for manure production has resulted in a cleaner and healthier environment. Also the rehabilitation of many natural water sources has decreased the opportunity for water contamination. SO2 activities have had the largest project impact on the environment and it is overwhelmingly beneficial, even if not as complete as possible. The impact can be differentiated into direct/primary and indirect/secondary. The foremost primary impact has been the anti-erosion measures on the slopes above the marais which have not only slowed both soil erosion and water run-off but also encouraged the production of fodder grasses. However, neither the quantity nor quality of the berms are satisfactory for maximum effect. The agro-forestry stopped at the production and distribution of seedlings to more vulnerable beneficiaries, accompanied by training but the attempt did not reach the stage of setting clear targets for the quantity or quality of reforestation. Another primary impact is due to the improved water management in the lowlands which has much reduced or eliminated the annual flooding with an attendant benefit of decreasing stagnant waters. The secondary impact is due primarily to many small individual activities encouraged by awareness raising, trainings and imitation with regards to agricultural and livestock activities. Foremost among these is the much reduced straying of animals which allows for contour berms and valley water management structures to last. Another significant impact is that of using the much increased amount of animal manure to fertilize crops. The project encouraged mechanical and biological pest and disease management techniques. While some farmers now use more mineral fertilizers, the amounts used are too small to cause any environmental concerns. And as chemical pesticides are not used, the quality of the water exiting the marais has potentially improved as animals no longer have access to it. Although the project brought in both animal and vegetal varieties from outside, appropriate prevention measures ensured that no new pest or disease was introduced. VI. GOOD PRACTICES DEVELOPED OR TESTED IN THE MYAP Following are good practices identified during the evaluation that either resulted in substantial, irreversible impact or in better use of program resources. A. Savings and Internal Lending Communities (SILC) Under SO2 The Savings and Internal Lending Communities (SILC) intervention is producing multiple, highly appreciated benefits for participants. The most significant benefits are mobilizing capital MYAP Final Evaluation Report 31 July 2012 71 for small-scale investments or coping with livelihoods shocks and providing SILC members with returns on their SILC savings investments. The approach has also provided other social capital benefits, especially for women, by giving them more opportunity for group participation and empowering them to make decisions. This is particularly important in the MYAP context with other interventions focused on facilitating gender-balanced decision-making. The SILC intervention is very likely to be sustained beyond the life of the MYAP, and there is already evidence that it is diffusing to other areas through MYAP community agents working with additional non-MYAP groups. B. Gender Intervention Under SO3 The MYAP strategy to promote more equitable household decision-making and more-balanced gender roles in rural households produced substantial impact. Key elements of the strategy were effective identification of community-based gender focal points (Acteur Relais) and providing them with good training on different dimensions of gender, human rights and conflict mediation. The program also identified men and women in targeted communities who had realized the gender messages and accomplished the desired change in their households. These persons were certified as "positive deviants" and were utilized extensively by the Acteur Relais for giving testimonials on the benefits of the change. The context of other MYAP interventions that provided opportunities to practice joint decision-making and changed roles was also critical to achieving impact. Finally, the context in Burundi seemed ripe for an intervention such as this. Many respondents said “this gender balance came at the right time”, suggesting perhaps that people were ready for change. The only flaw in the program in this set of activities was that the program did not fully engage an institutional partner so that the success could be more widely replicated. C. Mobile Team The mobile team approach used by the MYAP with staff based in the field to have direct contact with participants was effective for ensuring technical quality across the different geographic areas of the program as well as for facilitating the cross-fertilization of ideas and approaches. Other programs have certainly tried similar approaches with Technical Quality Teams or distributing technical responsibilities among different partners. The features that made the Mobile Team work well in Burundi were probably the emphasis on “mobile”, being based close to the front lines, and having good management for coordinating their activities. D. Program Coordination Workshops Program Coordination Workshops involving all implementing and technical partners in the MYAP held usually twice per year are highly appreciated by staff for enabling them to understand other components of the program and finding ways to facilitate more synergies. It is not unusual to find a MYAP with regular annual or semi-annual events in which all components of the program gather together to review the progress over a previous period, discuss problems and highlights and develop plans for the next period. It is worth highlighting again though based on the MYAP experience how effective these can be toward enhancing synergies across SOs, building stronger partnerships and ensuring that the program vision is imparted across the program. Every program should have annual or semi-annual events such as those implemented in the MYAP. MYAP Final Evaluation Report 31 July 2012 72 G. Front-Line Animateurs The MYAP front-line staff in the watershed collines are animateurs through which all SO2 and SO3 program activities are channeled. This approach appears to be very effective at coordinating activities in ways that do not overload participants. It also reduces isolation between different components of the program. These staff have social science and social mobilization skills to complement the technical skills provided by the mobile team. VII. MAJOR PROGRAM-LEVEL LESSONS LEARNED Following are major additional, program-level lessons learned relevant for food security programs. A. The Watershed Development Approach The MYAP is using a watershed approach for programming activities across all SOs, and some strengths and weaknesses in the approach emerged from the experience of the MYAP. The approach is consistent with the physical context in Burundi with livelihoods oriented around a marais and the need to manage both hillsides and lowlands to maximize the benefits from the marais. It is also compatible with the social context, with people living on collines in Burundi. The watershed approach with an intensive integrated strategy can produce clear, visible results to show what is possible to achieve in development. The approach has presented challenges for Commune Administration for planning since they now have some collines that are further developed then others and have to use somewhat different approaches to mobilizing resources. Similarly, for building local level community development planning capacities at the Commune level, working in only some collines in a commune makes it difficult to develop planning capacities at the commune level. Targeting drift31 must be monitored more closely in a watershed approach since local leaders and other power holders will be engaged in the program and could hijack activities to their own personal benefit. B. The Importance of Other MYAP Components for Supporting the Gender Intervention The MYAP achieved impact on changing traditional gender roles in a significant number of households. The question arises on whether the approach being used with Acteur Relais, Positive Deviants and good training would be as effective if it were a stand-alone project, not part of a program with multiple interventions. The opinion of the evaluation team is that the gender approach benefitted substantially from having other MYAP interventions. These established the credibility of the program. The MYAP brought some good benefit to participants through SILC and goat distributions, for example, so participants paid more attention to the gender messages. The other MYAP interventions also expanded resources for households and formed various associations of participants which provided opportunities to practice joint decision making. 31 Targeting drift refers to the case in which the characteristics of actual program participants/beneficiaries have drifted away from the definition of targeted impact groups usually specified when the program is designed. For example, a program may have defined a targeted impact group during the design as being extremely poor. Once the program gets underway, however, frontline staff find it easier to work with better educated people with more resources and are actually pressured by such persons to allow them to participate. The actual beneficiaries do not conform to the targeted impact group; targeting drift has occurred. MYAP Final Evaluation Report 31 July 2012 73 C. Exit Strategies The MYAP is now working hard to find ways to turn over the responsibility for sustaining some elements of program impact to government and communities. Some of these partners, notably the DPAE, Commune Administration and CDF, seem reluctant or are otherwise not yet fully informed. This reinforces the lesson that exit strategies need to be operationalized as part of implementation plans, beginning in year one of a program, and engagement of institutional sustainability partners should begin early, even in the design phase of a program. The current competitive processes used by donors to solicit proposals does not generally allow enough time, however, to cultivate these relationships. D. Being Participant Driven The MYAP has generally had greater impact with interventions in which participants have been given choices, e.g., being able to choose their own goat, rather than being given a goat. Similarly, letting communities identify and enlist eligible participants after clearly understanding the purpose and selection criteria, usually (not always) results in better targeting of beneficiaries. These experiences reinforce the concept that programs that are more participant driven, allowing program participants to make decisions, are generally more effective. E. Tracking High Impact Beneficiaries The MYAP is having greatest impact on those participants who are benefitting from multiple activities, e.g., a household that is participating in the marais, has received FFW, in multiple SILC groups, having a lead mother and a certified gender positive deviant. Most programs, including the MYAP do a pretty good job of tracking participants in each activity. M&E systems also need to be set up from the start to track beneficiaries who are participating in multiple activities. VIII. CONCLUDING REMARKS The MYAP has been implemented in a context that evolved from primarily post-conflict rehabilitation toward a situation in which development approaches are more appropriate....requiring a program to adapt fairly rapidly. In 2010, the program revised the strategy and accelerated implementation, and results here at the end of the program suggest that:  As indicated from evidence described in Section IV, significant impact on food security has been achieved on around 15,000 households in the targeted watersheds.  In addition, at least 75,000 children and 60,250 caregivers have benefitted significantly from MCHN capacity building.  In addition to these there are over 2,300 PLHIV who also benefited from the program.  Some impact, especially that achieved from SILC, goat solidarity chains and changing gender roles, is very likely to be sustained with not only economic impact, but also social impact.  Some impact, e.g., SILC and gender, is also likely to diffuse to surrounding areas outside the MYAP program area. The total program cost (including the C&F value of the food) is around $21 million. The total number of direct beneficiaries in the program is estimated to be around 75,000 children, 60,250 caregivers and 2,300 PLHIV under SO1. In the watershed collines there are an estimated 15,000 households who have benefitted, or around 75,000 people. An estimated 30% of these are MYAP Final Evaluation Report 31 July 2012 74 women and children already included under SO1, leaving a balance of 52,500 direct beneficiaries not included under SO1. A conservative estimate of the total number of direct beneficiaries is around 190,000 persons. The estimated cost per beneficiary per year for a four￾year, $21 million program having direct impact on 190,000 people is less than $28. In addition, the sustainability of the impact and potential replication of parts of the strategy need to be taken into account. In the judgement of the evaluation team, the CRS Burundi MYAP was a good investment of Title II Resources. MYAP Final Evaluation Report DRAFT Annex A 75 Annex A. Key Dates Table A-1. Key Dates for the CRS Burundi MYAP Date Event 1 August 2008 Proposed Start Date 4 August 2008 Official Start Date 3 September 2008 Final Signing of Cooperative Agreement 1 October 2008 MYAP Coordinator Arrives December 2008 & January 2009 Household Baseline Survey 21 January 2009 Arrival of First Commodities for Distribution September 2009 First Monetization Sale May 2010 Mid-Term Evaluation July 2010 Strategy Revision Workshop 11 January 2011 Approval of Cost Extension or the 4th Year 3 August 2012 Official End Date MYAP Final Evaluation Report Annex B CRS Burundi Multi-Year Assistance Program (MYAP) 2008-2012 Final Evaluation Plan USAID\FFP Title II - MYAP Implemented by Cooperating Sponsors CRS and IMC in Burundi (2008-2012) FFP-A-00-08-00080 Revised December 14, 2011 MYAP Final Evaluation Report DRAFT Annex B 77 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Table of Contents 1. Introduction and Background ......................................................................................78 1.1 Objectives of the final evaluation .......................................................................................... 78 1.2 Description of the MYAP ....................................................................................................... 78 1.4 Geographic Coverage ............................................................................................................. 81 2. Team Composition........................................................................................................83 2.1 Roles and responsibilities of consultants: ................................................................................. 83 2.2 MYAP Evaluation Staffing/organization .................................................................................. 84 3. Role and Responsibility of Consortium ......................................................................85 4. Team and Individual Team Member Final Evaluation Plans ......................................85 5. Methodology .................................................................................................................89 5.1 Methodology.............................................................................................................................. 89 5.2 Selection of Indicators .............................................................................................................. 89 5.3 Protocols for the final evaluation .............................................................................................. 89 6. Timeline and Deliverables ............................................................................................93 Activities/Tasks ....................................................................................................................94 Terms of Reference for the Quantitative Assessment Expert and / or firm ..................104 Terms of Reference for the Agriculture Expert (International hire) ...............................108 Terms of Reference for the Nutrition and Health Expert (International Hire)................113 Terms of Reference for the Team Leader/Livelihoods/Community Resilience Expert (International Hire)..............................................................................................................118 Scope of Work for Commodity Expert (Seconded from USAID/FFP).............................123 MYAP Final Evaluation Report DRAFT Annex B 78 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 1. Introduction and Background Catholic Relief Services (CRS), in partnership with International Medical Corps (IMC) is implementing a four -year Multi Year Assistance Program (MYAP) with funding from USAID\FFP (PL480 Title II programs) since 2008. The program aims at reducing chronic and transitory food insecurity of vulnerable populations in the provinces of Kayanza, Kirundo and Muyinga in northern Burundi. The underlying causes of food insecurity in post-conflict Burundi are characterized by slow economic reconstruction, continued population growth and repatriation of refugees and the subsequent increased pressure on land and natural resources, frequent natural shocks made more extreme by global climate change, conflicts over land tenure, tenuous political stability, and latent ethnic tension. 1.1 Objectives of the final evaluation The main objectives of the final evaluation study are to: 1. Assess whether the MYAP outcomes and impacts are achieved in line with the stated goal, objectives and intermediate results; 2. Assess constraints, lessons learned/good practices, opportunities as well as successes in implementation; 3. Determine the relevance and effectiveness of the strategies/approach utilized in the implementation of the program; 4. Assess the sustainability of the program benefit including but not limited to the development of the capacity (knowledge and skills) of stakeholders (community, local government, and partners and Consortium staffs) and quality of the activities (adherence to government and donor technical standards). 5. Compare indicator values at program end data against targets and baseline values. 6. Assess challenges to the project and the impact of these challenges on project performance. 1.2 Description of the MYAP Through this program, partners hope to enhance human capabilities via access to better health and nutrition and potable water, increase livelihood capabilities through environmentally-sustainable agricultural production and off- and on-farm income activities, and improve community resilience and capacity to respond to shocks. The project targets about 340,000 beneficiaries (direct and indirect). In the MYAP project, CRS and IMC are working on the following objective and intermediate results... In addition, the program’s Results Framework and IPTT are attached. MYAP Final Evaluation Report DRAFT Annex B 79 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Goal: Reduce chronic and transitory food insecurity of vulnerable populations in the provinces of Kayanza, Kirundo and Muyinga in northern Burundi. Partners Strategic Objectives (SOs) Intermediate Results (IRs) IMC 1:Vulnerable households have enhanced human capacities 1.1: Households practice optimal infant and child feeding practices. 1.2: HIV-affected households consume sufficient food for optimal nutrition 1.3: Households practice good health-seeking behaviors 1.4: Households use appropriate hygiene and sanitation practices CRS 2: Vulnerable households have enhanced and sustainable livelihoods capacity. 2.1: Vulnerable households have improved production in environmentally-sustainable ways. 2.2: Vulnerable households have adopted strategies and techniques to diversify and increase revenues. CRS 3: Vulnerable communities have enhanced resiliency 3.1: Communities and local government agents have collaboratively developed community-based early warning systems and response action plans 3.2: Vulnerable households are efficiently managing their assets in an equitable manner. In collaboration with the MoH/Burundi (PRONIANUT, Provincial Departments of Health), MINAGRIE (ISABU, Provincial Departments of Agriculture) and the populations of Kayanza, Muyinga and Kirundo 1.3 Achievements as of September 2011: SO1/Health: Health activities are both community and health center based  Health centers: Trained and currently supports 89 nurses, 81 paramedics and 8 medical doctors on community management of acute malnutrition (CMAM); Trained and currently support 96 nurses and 127 paramedics on growth monitoring and counseling; Supporting 52 health centers and 3 hospitals equipped for an adequate implementation of CMAM and growth monitoring services; 20,115 children under five graduated from the CMAM rations and 22,860 children between 0 to 36 months have attended growth monitoring services; Cases of malnutrition have decreased since the MYAP project. Community-based activities:  Community health workers: Trained and currently supports 1,440 community health workers on community screening and referral as well as health and nutrition techniques MYAP Final Evaluation Report DRAFT Annex B 80 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Care Group activities: Trained and currently support 882 lead mothers for the implementation of care groups activities; Identifying and training 1,000 new lead mothers associated with the program extension.  Sensitization activities by community health workers and Care Group Lead Mothers: 27,020 health education sessions held within the health facilities and communities; 512,287 beneficiaries reached through the sensitization campaign and home visits; 1,113 households have improved latrines, 1,073 households have built waste pits/composts and 1,742 have built utensil tables as a result of the hygiene and sanitation messaging; 3,322 people trained on malaria prevention  PD Hearth: 20 positive deviance sites implemented to promote community-based management of acute malnutrition with local foods; Identification in progress for 20 more sites.  Water sources: 40 water sources completed; trained 40 water committees on water source management; trained 90 community members in 40 communities through the Participatory Hygiene And Sanitation Transformation (PHAST) technique  HIV activities: Trained 90 volunteers on appropriate HIV infected people and health education; Provides monthly supplementary rations to 550 vulnerable PLHIV on ARTs in Kirundo; provided vegetable seeds and training to 200 vulnerable households this season and identification is in progress for 1,300 new beneficiaries for Season A 2012 agricultural season. SO2/Agriculture:  Valleys/“marais”: 237 hectares of valley water infrastructure rehabilitated and 19 water management committees elected, trained, and supported.  Anti-erosion: Over 593 km of contour bunds constructed by 3,447 FFW beneficiaries; distribution of 3,500 agro-forestry saplings; 5 tree nurseries supported and trained in tree production, small business management, and agro-enterprise and now autonomous  Technology transfer: Trained 11,791 farmers in composting, use of organic fertilizer, proper seed spacing, planting in rows, crop rotation, animal and crop disease recognition, and below crops  Rice: Multiplied improved varieties in collaboration with ISABU (Burundi Agricultural Institute) with 120 farmers in 2010; over 1,500 farmers gained access to improved varieties of seed in 2011  Cassava: Disease-tolerant cassava planting material provided to 6,373 farmers; 65 individuals and farmer groups are doing tertiary multiplication  Beans: Distribution of beans to 9,350 farmers through seed fairs up to Sept 2010; ongoing participant variety selection (PVS) trials of improved seed varieties with 928 farmers in collaboration with ISABU. MYAP Final Evaluation Report DRAFT Annex B 81 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Sweet potato: Currently multiplying 3 varieties of orange-fleshed sweet potato in collaboration with ISABU and ISAR (Rwanda Agricultural Institute) with 3 associations for PVS trials in September 2011  Goats: Distribution of 5,400 local female goats and 288 imported male Boer goats; 57 goat management committees elected and trained; 1,284 goats re-distributed to 816 beneficiaries through solidarity chains  SILC: Working with 245 SILC groups; working on professionalization of field agents  Agro enterprise: Trained the boards of 122 already existing associations in small business management and agro enterprise; working intensively with 12 agro enterprise associations SO3/Community Resilience:  Gender: Working with more than 150 gender volunteers and over 100 positive deviants through 352 trainings; currently working on reinforcing links between gender volunteers and Care Group Lead Mothers  Community action plans: Working with 21 collines and other sous collines, commune, and provincial government and Ministry of Agriculture officials on emergency action plans; collaboration on provincial-level contingency plans; currently working on monitoring committees to ensure implementation of community action plans 1.4 Geographic Coverage The MYAP covers a total of 2,493 sous-collines in the three provinces of Kayanza, Kirundo, and Muyinga. SO1’s health activities involving health centers and community health workers are taking place in the entirety of the three provinces (yellow zones, including pink insets). SO1’s activities with Care Groups as well as all SO2 and SO3 activities are taking place in the three pink watershed areas MYAP Final Evaluation Report DRAFT Annex B 82 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 composed of 61 sous-collines. The watershed areas are within the three targeted provinces, so in these areas, there is an overlap of all SO1, 2, and 3 activities. 1.5 Key actors: Nutrition sector  IMC – implementing partner for most of SO1  CRS – lead only on water source construction and HIV gardens  Government of Burundi’s Ministry of Health’s PRONIANUT department and Provincial Departments of Health (BPS) Agriculture and animal husbandry sectors  Catholic Relief Services (CRS): Lead implementing partner for SO2 and SO3. Implementing SO2 and SO3 directly in Kirundo province and through partners in Kayanza and Muyinga.  Bureau d’Appui au Développement et à l’Entraide Communautaire de Ngozi (BADEC) : Implementing Partner for SO2 and SO3 in Kayanza province.  Organisation Diocésaine Pour l’Entraide et le Développement Intégral de Muyinga (ODEDIM) : Implementing Partner for SO2 and SO3 in Muyinga province.  Government of Burundi’s Ministry of Agriculture and Provincial Departments of Agriculture and Animal Husbandry (DPAE) Livelihoods and community resilience sectors:  CRS: Lead implementing partner for SO2 and SO3. Implementing SO2 and SO3 directly in Kirundo province and through partners in Kayanza and Muyinga.  Bureau d’Appui au Développement et à l’Entraide Communautaire de Ngozi (BADEC): Implementing Partner for SO2 and SO3 in Kayanza province.  Organisation Diocésaine Pour l’Entraide et le Développement Intégral de Muyinga (ODEDIM): Implementing Partner for SO2 and SO3 in Muyinga province.  Government of Burundi’s Ministry of Agriculture and Provincial Departments of Agriculture and Animal Husbandry (DPAE) Commodity activities 1. CRS: Lead for commodity management, warehousing, transport, and FFW distribution 2. International Medical Corps: Manages commodity distributions in collaboration with health centers and with HIV associations 3. Government of Burundi’s Ministry of Agriculture and Provincial Departments of Agriculture and Animal Husbandry (DPAE) 4. Government of Burundi’s Ministry of Health’s PRONUIT department and Provincial Departments of Health (BPS) 1.6 Major Implementation Challenges The following are summary of challenges faced by Consortium Members and their partners while implementing the MYAP program: Challenges:  Cyclical drought in the September – December season, especially in parts of Kirundo province, which decreases production and increases food insecurity and malnutrition levels.  Community action plans are a relatively new type of activity in Burundi, as most current responses to shocks are on a household level, or involve appeals to NGOs or national MYAP Final Evaluation Report DRAFT Annex B 83 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 ministries. This behavior change is proving more challenging than other activities, but the MYAP has recently conducted a study which has provided some lessons learned, such as dedicating a staff person to the activity, forming monitoring committees, and making training more practical.  Structural challenges on government level with insufficient or poorly remunerated staff. This is especially the case for health activities, where the MYAP supports health centers with overloaded staff and with volunteer community health workers who support many Ministry of Health and NGO activities. This structural challenge hinders full uptake of health messages, especially outside of Care Group areas. 2. Team Composition 2.1 Roles and responsibilities of consultants: a. International consultants The international consultant team is composed of:  One international expert with a strong experience in livelihoods and community resilience issues who will serve as the team leader.  One international Agriculture and Food Security expert  One international Nutrition/health expert  One Commodity expert (to be seconded from USAID) CRS will recruit the international team on behalf of the Consortium. The international consultant with a strong experience in livelihoods and community resilience will serve as the team leader of the evaluation team. He/she will be familiar with MYAP evaluations, possess strong leadership abilities and be knowledgeable on issues related, monitoring and evaluation, program management. The main task of the international consultant will be to lead the team and assure quality of the work throughout the evaluation process. S/he will give due attention during the phases of development of data collection tools, data analysis, report writing and other relevant deliverables by the local consulting firm. In addition, he/she will be responsible for leading sector specialists (international experts) and reviewing and/or finalizing the deliverables of the local consulting firm. b. Local consulting firm The quantitative survey is an integral part of the final evaluation. In collaboration with the Team Leader, the local consulting firm, with proven experience in quantitative analyses, will design quantitative data collection instruments and tools (based on standard KPC – see attached questionnaire), provide refresher training to its enumerators/supervisors, collect quantitative data, analyze data32 and provide the stated deliverables. The local consulting firm will coordinate the quantitative survey which will consist of data collection, processing and analysis in order to compare project achievements against targets and baseline results. The study will replicate the quantitative survey conducted in the baseline study. The consulting firm 32 Data analysis for the quantitative study is the responsibility of local consultant. However, the local consultant is expected to work closely with the international consultant during data analysis. They should agree on the process prior to analysis and report writing. Documentation of the agreement is encouraged. MYAP Final Evaluation Report DRAFT Annex B 84 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 will be supported by its own enumerators and supervisors for the data collection. The firm will ensure that all the tasks are met as per the Terms of Reference by the Consortium’s technical committee and the evaluation Team Leader. Essential prerequisites of the local consultant/firm include substantial experience in: conducting quantitative survey, especially with USAID Food for Peace programs; cluster sample design using statistic parameters; data processing and data analysis using statistics software. S/He will possess bilingual English/French (speaking, reading, and writing) and excellent communication skills; be able to work in a team and willing to travel to remote areas. S/he will have minimum qualification (Master degree in Statistics, Demography, Economy or other relevant field of expertise and proven track record in the field of evaluation. 2.2 MYAP Evaluation Staffing/organization Oversight Committee: Provide additional guidelines and make final review of the FEP and also make strategic and managerial decisions per the recommendation of the Technical Team. Technical Team: The main task of the team is to develop the FEP and lead the process of bid preparations, selection and recruitment of the international and local consultants. The Technical Team is organized by drawing representatives from: CRS/ Burundi, IMC/Burundi; and USAID (observer). The technical team also provides administrative and technical supervision of the consultants’ work. International Consultants: headed by the Team Leader, s/he works closely with the Technical Team. He/she is responsible for the overall evaluation process and output. Specific tasks are described in the attached individual Terms of Reference. Local consultant/firm: S/he is hired to work collaboratively with the Team Leader. The local consultant will be responsible for providing expertise required by the FEP to undertake the field work, design the quantitative survey, train staff, supervise the fieldwork, analyze the data and write the final quantitative report. Specific tasks of the local consultant/firm are provided in the attached Terms of Reference. Data Entry Experts: Data entry is managed by the local consultant/firm. By availing sufficient resources, the local consultant will ensure that data entry and validation is completed within the specified period. Data entry and validation must be finished 13 days after completion of fieldwork. The data entry staff will report to the local consultant. CRS Technical Coordinators: Technical coordination is provided by CRS staff, generally the CRS’ M&E specialist and MYAP Coordinator, who ensure coordination and linkage between field operations and consultants in each evaluation area. The coordinators liaise with the consultants with respect to the activity in the field. The Technical Coordinators communicate with the Team Leader. The best way to represent the relationship between the Technical Coordinators and the Team Leader is one of consultation and facilitation. CRS will serve as the focal point for communicating with the Team Leader on all matters related to general administration and logistics. Field Supervisors: The local consultant will be supported by field supervisors. Supervisors individually review all questionnaire forms on a daily basis, and repeat one or two randomly selected, previously MYAP Final Evaluation Report DRAFT Annex B 85 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 conducted surveys per day. Supervisors are responsible for transferring all completed forms to the local consultant on a weekly basis. Field supervisors will be accountable to the local consultant. Enumerators33: The 10 enumerators will be charged with collecting the data in the field. The number of questionnaires to be administered by one enumerator per day will be determined by the local consultant. Enumerators will preferably be chosen from local areas that have good command of the local language and know the community. The local consultant will be responsible for identifying and hiring these personnel. The enumerators will report to their respective field supervisor. 3. Role and Responsibility of Consortium Lead agency (CRS) - Facilitate coordination throughout the evaluation cycle - Coordinate recruitment of all consultants - Liaise with the Final Evaluation Team Leader and local consulting firm - Organize joint meetings between Consortium members and final evaluation team Consortium members (CMs) Considering the fact that the final evaluation is meant to be independent and external, CRS and other consortium members will have the responsibility of facilitating the evaluation in their respective operational areas. Assign Monitoring and Evaluation officers to coordinate the process, provide on￾time response and facilitate field work throughout the evaluation process. Consortium members may comment on the draft evaluation report and interim reports and correct factual errors but may not ask for changes in the conclusion of the authors. Comments to any outputs and documents produced at different stages of the evaluation if any will be attached in a separate annex. For this reason M & E officers will participate in relevant refresher training sessions but will not be engaged in collecting data. In addition, CRS and other consortium members will provide all relevant documents needed for the consultancy. CRS will also provide support to the evaluation team by making arrangements for hotel reservations, vehicle transport, and facilitate contact with program staff, partners, and local stakeholders. The evaluation team members are expected to provide their own laptop computers, and any software necessary to produce the final evaluation report in the appropriate format. CRS will provide access to printers, photocopiers, and the necessary email and telephone linkages. 4. Team and Individual Team Member Final Evaluation Plans 4.1. Key Evaluation Areas The final evaluation is required to carry out the detailed assessment of the MYAP based on the following essential evaluation areas: - Relevance: Investigate in detail the extent to which the objectives of the MYAP are consistent with the needs of the beneficiaries, the recipients, the host country and donor. 33 Enumerators are not staff member of consortium members, but do have proven experience in related work. MYAP Final Evaluation Report DRAFT Annex B 86 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 - Effectiveness: Look into whether or not the proposed development strategy achieves its long term goals. Verify whether the program meeting targets. Measure the extent to which the goal, objectives and intermediate results are reached. Examine the extent to which strategic objectives are achieved efficiently and in a sustainable manner. - Efficiency: Examine how economic inputs (resources, expertise, time, etc.) are converted into outputs. Conduct a comparison of the value (not necessarily monetary) of the development strategy and the resources allocated to achieve outputs. - Sustainability: Assess the likelihood that the positive effects of the MYAP (such as assets, skills, facilities or improved services) will persist for an extended period after the end of the program (financial assistance). - Impact: Examine the changes in the lives of the population as perceived by them and other partners, plus sustainability-enhancing changes in their environment to which the program has contributed; changes can be positive or negative, intended or unintended. The above mentioned five key evaluation areas are used as a lens that should be taken into account when answering the following key evaluation questions; detailed questions related to technical areas are listed in various terms of reference (see annexes): 4.2. General Final Evaluation Questions Household roaster Questionnaires must start with a household roaster to identify all eligible respondents. (See annex 3; a model of a HH roster) MYAP Design, Implementation and Achievement:  Are planned activities appropriate for the food security problems identified in the selected target areas? Do the framework, assumptions and design match the local food security conditions?  Are the selection criteria appropriate for identifying and reaching target communities and households?  Is the program well integrated in the local government’s strategy and priorities? Are there steps that could be taken to improve the integration as well as food security impacts through greater integration for future programming?  How effective is the program at reaching women? What could have been done to improve women’s participation?  Which interventions are most critical and/or effective in achieving project objectives? What interventions have been more or less successful in meeting the program goal? Why?  Are there any unexpected but important benefits or impacts of the program that should be documented? Are there any negative impacts or unintended consequences of the program that need to be addressed, and how?  To what extent have the major challenges faced by the Consortium members affected MYAP performance? How are these challenges addressed? MYAP Final Evaluation Report DRAFT Annex B 87 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Do food-for-work activities compete with or complement the demands for household labor on other productive activities? Adoption of technologies/practices:  Are beneficiaries adopting desired practices or sustainable agricultural technologies? What new technologies have Consortium Members introduced under the MYAP that are enhancing program objectives?  What is beneficiaries’ primary source of information concerning practices and behaviors? What are other key channels of information? Capacity Strengthening:  Has the program effectively developed the capacity of stakeholders and/or partners? If not, how could the design or implementation be altered to improve capacity strengthening for future programming?  Has the program effectively enabled, or developed the capacity of, beneficiaries? If not, how could the design or implementation be altered to improve capacity strengthening? Sustainability:  Are the outcomes/impact related to adoption of better practices sustainable? (i.e., participants are likely to continue after the project ends?) Which outcomes/impacts are likely or unlikely to be sustainable, and why? What can be done to increase the sustainability for future programming?  Is there a well-developed exit strategy? If so, has the Consortium moved forward to initiate some aspects of that strategy?  Has the program effectively collaborated with local administrative bodies such as regional bureaus, collines and sous-collines offices, local councils, etc? How does the Title II program strengthen or expand the capacity of these entities, and will it be able to maintain the strengthened or expanded capacity? Monitoring and Evaluation/Reporting  As defined and measured, do the performance indicators provide useful and reliable data on progress and impacts?  Are M&E data and anecdotal information shared and utilized with the other stakeholders/government partners working in the area?  Is/are the M&E staff assigned to the MYAP program well capacitated? If not, why and how were the shortcomings addressed?  Is there enough in the budgets allocated for M&E? 4.3. Technical Sector Questions (see annex 2 for detailed questions) Health and Nutrition:  How successful has the program been in protecting the nutrition/health status of specific groups (PLHIV, children under 5, etc.)?  Are health and nutrition BCC materials developed and tested appropriately? Are they tailored to the user, actionable, accurate and linked to growth promotion messages (where growth monitoring is being implemented)? MYAP Final Evaluation Report DRAFT Annex B 88 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Is there change in community practice regarding child care and management due to MYAP program interventions? Livelihood Diversification:  What are the main livelihood sources/enterprises? Which livelihood sources/enterprises are promoted by the MYAP?  Are the technologies/enterprises being promoted well-established and well-suited to the local context and agro-ecological environments?  Are farmers and other community groups able to maintain new productive infrastructure on their own? If not, why and what could be done to address this limitation?  Are Title II program activities linked to relevant government offices and agricultural research centers and has a communication protocol been established? Will these relationships extend beyond the life of the project? Infrastructure (Irrigation, Water)  Does the design and implementation of the infrastructure adhere to local technical minimum standards and guidelines? If not, why?  What are the direct or indirect benefits derived from infrastructure construction or rehabilitation that are not currently being captured?  Are there any unintended negative environmental impacts stemming from infrastructure activities? If so, are there sustainable mitigation measures implemented? Commodities  Are correct procedures and best practices used in receiving, distributing and storing food commodities?  Are commodities being managed appropriately?  Is the food distribution system design efficient?  Are there good procedures to monitor food distributions and in a timely manner?  Are the recommended food rations respected? What are the weaknesses of the distribution scheme and what are the recommendations to address them?  Does the current system protect against “false” beneficiaries from receiving commodities?  Are beneficiaries well-targeted for food rations?  Do beneficiaries understand the quantities of food that they should receive?  To what extent are the natural resource management activities sustainable (i.e. lowland water management and hillside anti-erosion activities)? Does the use of food for work for participation in agricultural production related activities act as an incentive/disincentive to improving productivity, and how? Community-based early warning systems and response action plans:  What is the existing early warning system at local level? Is it effectively linked to the zonal, regional and national system?  How has the program strengthened the systems?  What are the lessons learned for future programming? MYAP Final Evaluation Report DRAFT Annex B 89 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 5. Methodology for the final evaluation 5.1 Methodology Final Evaluation surveys use both quantitative and qualitative methods to determine program achievements against targets and baseline results. Information is gathered on impact indicators using quantitative survey methodology as well as through focus groups discussion and semi-structured interviews to learn the opinions of program participants and Government partners. Review MYAP Program Documents – One of the first activities of the evaluation team will be to conduct a review of MYAP program documents to gain a sound understanding of the nature, the objective, the resources, and the focus of the program. The documents to be reviewed include the MYAP proposal, annual Pipeline and Resource Estimate Proposals (PREP) /Annual Results Reports (ARR), periodic reports, the joint base line survey report, IPTT, and other relevant documents. Review Midterm Review Report - The consultants should review the Midterm Review of the program. Interview Stakeholders - Interview stakeholders involved with MYAP implementation at national, regional, and local levels (collines, sous-collines and watersheds) and also through focus group discussions with community representatives. Field survey (mainly quantitative) - The local consultant, in coordination with the international consultant/Team Leader should use appropriate participatory data collection methodologies and triangulate all information collected. They are also expected to conduct an anthropometric survey. As the field survey is a sample survey, the population in the target areas will be the sampling frame for the survey. To be consistent to the 2008 baseline survey, cluster sampling will be used during this evaluation (refer to annex 1). Qualitative assessment: includes focus group discussions, key informant interviews and observations. The consultants (international and local) will compile check lists for semi-structured interviews to ensure that all necessary information on the evaluation questions outline in this plan are collected. Focus group respondents will be selected from the survey communities, while the key informants will be selected by the consulting staff in consultation with the team leader from the list of potential respondents provided by the Consortium members and local government. 5.2 Selection of Indicators Indicators to be evaluated include the following: - Key program indicators as defined by the MYAP proposal document are listed in Annex 1. In addition to Annex 1, the consultant should refer to the IPTT and other relevant documents (Performance Management Plan and proposal) to select relevant related appropriate indicators (outcomes). - A set of indicators that represent activities specific to each Consortium member. The consultants will review these indicators and come up with specific suggestion if any change is required. - A set of qualitative and quantitative tools is to be compiled by the international and local consultants and/or collected to show the impact of the program and compared against the baseline. 5.3 Protocols for the final evaluation Design and sampling: MYAP Final Evaluation Report DRAFT Annex B 90 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 The sample design for the final evaluation should be consistent with the protocols used for the 2008 MYAP baseline study. Population-based sampling as opposed to beneficiary based sampling is preferred. Taking into account the fact that all three objectives of the MYAP do not have the same geographic coverage and targets, two separate surveys will be conducted with their respective samples: 1) a sample exclusively for SO1 and 2) a sample for both SO2 and SO3. The Sample Size Determination for Single Point-in-time Estimator model will be used (as during the baseline) for collecting outcome/impact data. Two- level cluster sampling will be used (first level sample will include the communities chosen at random from a list of sous-collines and the second level sample represents a selection of households within the community. For SO1, in addition to the Single Point-in-time Estimator method used to determine the sample size for the baseline survey, the final evaluation will also follow R. Magnani sampling guidance to estimate the number of households to visit for data collection. The average household size is estimated at 4.9 persons and the proportion of 6-59 months old children per household is 16.57%. The number of household should be inflated by 10% to compensate for non-response. To ensure that 768 children are reached, approximately 1,040 households will be visited. For SO2 and SO3, the final evaluation will use the same sample size estimation method as during the baseline. By using the single point-in-time estimator to calculate the SO3 final target (75% of months of adequate food provisioning) the corresponding sample size is determined as 576. For the selection of the household to interview, the team will use the following method: from the center of the community, a pen will be thrown to determine the direction towards the community boundary. A path will be followed from the center to the boundary, counting and numbering the houses one side of the road. Upon arrival at the boundary, a random number will be chosen for the first household to visit. For each subsequent household, the next closest household will be chosen. Instrument development: The instruments for the quantitative survey will be consistent with those used in the baseline; however, additional survey tools may be developed if they are needed. Refresher Training: Training will be conducted on the use of all survey tools, until high level of competence is achieved as determined by the Local Consultant lead. Although enumerators are experienced and have worked with the consulting firm on similar surveys, they will be tested on their understanding of the field manual before initiating the survey. Six full days will be devoted for refresher training. Training will focus on the questionnaire forms; the latter will be tested during the training and on the field. Prior to field work, instruments will be fine tuned and the questionnaire translated in the local language. Fieldwork: 10 enumerators and 2 field supervisors will be used to collect data. Each enumerator will be responsible for collecting a set number of complete household surveys per day (to be determined by local consultant). Data validation and entry: All individual questionnaire forms will be double-checked by field supervisors in the field the same day they are collected, and 5% of all surveys will be replicated to ensure the reliability of the information. All the data files will be validated using double entry methods or some equivalent procedure. Consistency and redundancy checks will be built into the data entry system to minimize typing errors. MYAP Final Evaluation Report DRAFT Annex B 91 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Analysis: The analysis of the final evaluation data will first provide a descriptive analysis of all variables. Given that an “adequacy” evaluative model is used, no causal inferences about attribution will be attempted in the analysis. The analysis and final evaluation report will consist essentially of a tabulation of outcome indicators, accompanied by a brief narrative to describe those outcomes relative to program activities and implementation. The International Consultant/Team Leader must calculate and report on confidence intervals for all indicators collected in the quantitative survey. Reporting: Reports (quantitative and qualitative) will be expected at critical junctures of the final evaluation to review accomplishments to date (interim reports will be drafted as sections of the final report, and should be included in the latter to fully document the process). The expected interim reports for the quantitative survey are: - Interim report 1: To be produced before initiation of refresher training. Will include the final selection of indicators, the field manual and the questionnaire forms. - Interim report 2: To be produced at the end of the refresher training. Updates the first inception report with the results of instrument field tests and corresponding adjustments in the field manuals. - Interim report 3: To be produced at the end of field work to list all the problems that emerged in the field, and how they were addressed. If necessary, all changes made during the field phase to the instruments will be explained in this report. - Interim report 4: To be produced at the end of the data entry and cleaning procedures. Includes all the data, with double entry validation tables, frequency distributions for detection of outliers and any other relevant problems encountered during the data collection phase. - Draft and final reports: The content of evaluation report should at least include the following sections: Title page with date, and logos Executive summary Introduction Objective of evaluation Brief description of the program Detail analysis of findings by technical sector Accomplishments and constraints, progress towards objectives/IRs, program quality (management, M&E, commodity, program sustainability, relationship to current and future USAID SOs), integration of MYAP components. Summary of findings by technical sector and regions Cross cutting issues Lessons learned Recommendations by technical sector Annexes MYAP Final Evaluation Report DRAFT Annex B 92 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Composition of consultant team, Tools and Methods, List of sites visited, List of key informants, References, Indicator performance tracking tables (IPTT), List of acronyms etc. As per the FFP requirement, confidence intervals should be calculated and included in the report for all indicators. MYAP Final Evaluation Report DRAFT Annex B 93 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 6. Timeline and Deliverables The process of final evaluation began on October 03, 2011 with the development of the Final Evaluation Plan. As shown in the summary of timetable and deliverables below, the quantitative survey as well as qualitative study are intricate parts of the final evaluation therefore under the supervision of the international consultant/Team Leader. Though tentative dates are shown below, evaluation activities including the quantitative survey will only commence when the final evaluation Team Leader is recruited. Summary of timetable and deliverables are as follows: Task/deliverables Timeline Responsible body Local consultant International consultant Compiled international consultant team’s preliminary work plan, logistics request, and schedule: Due for validation one week prior to arrival in country. 17 Jan. International Consultant (Team Leader) sends document virtually to CRS Compiled team’s interview guide(s) and detailed methodology (description of qualitative methodology in general, and particular how sampling will be done and how data will be analyzed): Due for validation two days following arrival. 20 Jan Team leader to send it virtually after contract is signed Review/update Baseline questionnaires and develop quantitative survey instruments. 23 – 25 Jan. 2012 Local consultant Team Leader arrives in country , Quality assurance by international consultant First meeting of qualitative team, preliminary meetings with consortium members and revision of preliminary work plan, logistics request, and schedule etc. 23 – 25 Jan. Team Leader (Team members arrive) Field manual developed and questionnaire (translated) (soft and hard copy submitted to CRS 23 – 25 Jan 2012 Local consultant Interim report 1(on quantitative survey instruments) submitted 25 Jan., 2012 Same as above International Consultant (Team Leader) approves report Enumerators refresher training/workshop on data collection tools 26 Jan - 01Feb. 2012 Local Consultant Tools for qualitative data collection completed 26 – 27 Jan. Team leader Interim report 2 (on Enumerators training) 02 Feb 2012 Local consultant International Consultant (Team Leader) approves report MYAP Final Evaluation Report DRAFT Annex B 94 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Fieldwork for qualitative study 30 Jan. – 10 Feb. Team leader Test and finalization of data collection tool 2 Feb 2012 Local consultant Data collection 03 Feb. 2012– 18 Feb. 2012 Local consultant Qualitative database 11 Feb. – 15 Feb. Team leader Report outline, highlighting major sections and themes to be covered, specifically addressing all the objectives cited above 16 – 17 Feb. Team leader Interim report 3 (on data collection) 21 –22 Feb. 2012 Local consultant International Consultant (Team Leader) approves report Complete database entry and validation submit soft copy to CRS 23 -27 Feb. 2012 Same as above First draft report submitted to the consortium members 24 Feb. Team leader Interim report 4 submitted (on data entry) 29 Feb., 2012 Local consultant International Consultant (Team Leader) approves report Data processing and descriptive analysis (per agreement) 01 – 06 Mar. Local consultant Draft Quantitative Evaluation reports (soft and hard copy) - First draft 07 - 08 Mar. Same as above - Second draft report submitted to the consortium members 09 Mar. Team leader Review of First draft of quantitative report. 09 - 10- Mar. Same as above Team Leader approves Final quantitative report submitted (soft and hard copy) 12 Mar. Local consultant Final Report to be approved by International Consultant (Team Leader) Final draft 30 Mar. Team leader Date Activities/Tasks Sept.-Oct. , 2011 Technical team discusses the Final Evaluation Plan, sets detailed plan of action and share initial SOWs with CARO and HQ Nov 01-12, 2011 Draft Final Evaluation Plan submitted to and commented by Oversight Committee, including HQ Nov. 12-Dec05, 2011 Oversight Committee forward comments to technical team for revisions and finalization Dec. 07, 2011 Revised document submitted to CARO and HQ. MYAP Final Evaluation Report DRAFT Annex B 95 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Dec 09, 2011 Revised plan for USAID and FANTA for final approval Dec. 09, 2011 Draft Terms of Reference finalized for all consultants Dec. 12, 2011 Technical team and Oversight team decide on the bid process Dec 13-Dec. 30, 2011 The bid process is aired on news paper and web sites Dec. 26, 2012 Refresher training and field tests begin, pilot testing ends. Manuals, instruments updated. Second inception report presented (6 days) Jan. 09, 2012 Selections of Consultants including the local consultant Jan. 13, 2012 Local consultant contract signed Jan. 14, 2012 International Consultants contract signed Jan. 20, 2012 Qualitative team’s preliminary work plan, logistics request, and schedule compiled by Team Leader: Due for validation to CRS one week prior to arrival. (5 days) Jan 20 Compiled team’s interview guide(s) and detailed methodology reviewed and amended (description of qualitative methodology in general, and particular how sampling will be done and how data will be analyzed and resubmitted for validation (1 day) Jan 22, 2012 Arrival of the International experts in county Jan 23, 2012 First meeting of qualitative team, preliminary meetings with consortium members and revision of preliminary work plan, logistics request, and schedule etc. Jan. 23,2012 Local Consultant begins work; instruments/questionnaire (quantitative), sampling, methodology, (3 days) Jan. 25, 2012 Interim report 1 (quantitative) approved by Team Leader (1 day) Jan 25, 2012 Draft field manuals completed, presented/discussed and finalized (2 day) Jan 27 - 30, 2012 Other qualitative tools developed for data collection (2 day) Jan 31, 2012 Fieldwork for qualitative study begins (10 days) Feb 2, 2012 Interim report 2 (quantitative) approved by Team Leader (1 day) Feb 2, 2012 Pilot testing and finalization of data collection tools ends. Feb 3, 2012 Fieldwork for quantitative study begins ( 14 days) Feb 11, 2012 Qualitative database (5 days) Feb 15, 2012 Report outline, highlighting major sections submitted for validation Feb 16, 2012 Restitution workshop for MYAP staff and stake holders to present the initial findings (1 day). Feb 18, 2012 End of quantitative fieldwork. Feb 21, 2012 Interim report 3 presented to Team Leader for approval (2 days) Feb 23, 2012 Data entry begins (4 days) Feb 25, 2012 First draft report to technical committee including the methodology used, pertinent tables and graphs, quantitative and qualitative information, lessons learned and recommendations submitted. Due day of departure. (10 days) Feb 27, 2012 Cleaned raw data submitted to CRS Feb 29, 2012 Validation of interim report 4 by Team Leader (1 day) Mar 1, 2012 Data processing and descriptive analysis (5 days) Mar 2, 2012 Comments on the draft report received by the consultant from the consortium (5 days) Mar. 7-8, 2012 Draft Quantitative Evaluation reports (soft and hard copy) - First draft (2 days) Mar 09-10, 2012 Review of First draft of quantitative report. (2 days) Mar 10, 2012 Consultant submits Second draft report to technical committee (10 day) Mar 12, 2012 Final quantitative report submitted (soft and hard copy, 1 day) MYAP Final Evaluation Report DRAFT Annex B 96 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 7. Illustrative list of Reference documents 1) MYAP 2008-2012 proposal and PREP documents 2) MYAP Baseline report 3) MYAP Mid Term evaluation report 4) MYAP midterm evaluation recommandations implementation document 5) Annual Result reports 6) Organizational chart 7) Maps of sites and distribution of interventions 8) Training materials 9) Field trip reports 10) Project monitoring tools and M&E data collection IPTT (Indicator Performance Tracking Table) 11) PMP (Performance Management plan) 12) DQA (Data Quality Assessment) report 13) USAID/FFP field trip reports 14) Trigger indicators reports 15) Food For Peace documents on MYAPs quantitative assessments requirements 16) Sampling Guide; Robert Magnani; December 1997 Mar 16, 2012 Comments on the second draft report received by the consultant from the consortium (5 days) Mar 30, 2012 Final report is submitted by the consultant incorporating comments from CRS and partners as well as FFP MYAP Final Evaluation Report DRAFT Annex B 97 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Annexes Annex 1: List of Indicators The following are indicators that have to be addressed in this final evaluation. Additional Indicators must be included by reviewing IPTT. a) Baseline Indicators: b) Additional Proposed Indicators34 34 These are not the only list of indicators. The consultant is expected to review Consortium Members’ documents and identify and recommend additional Indicators Indic # Label 1.1.1 % of children 6-59.9 months of age with height for age Z-score < - 2 S.D (Health and Agro/Health; all three provinces) 1.1.3 % of children 6-59.9 months of age with weight for age Z-score < - 2 S.D (Health and Agro/Health; all three provinces) 1.1.4 % of children 6-23.9 months of age with weight for age Z-score < - 2 S.D(Health and Agro/Health; all three provinces) 2.1.1 Household dietary diversity score (Agro/Health; all three provinces) 3.1.1 Average number of months of adequate food provisioning in program(Agro/Health; all three provinces) 1.1.6 % of caregivers demonstrating proper food hygiene behaviors(Health and Agro/Health; all three provinces) 1.1.7 % of caregivers demonstrating environmental hygiene behaviors(Health and Agro/Health; all three provinces) 1.1.10 % of PLHIV eating the recommended # of times per day (Health and Agro/Health; sampling should be only in Kirundo province as this is the only site of activities) 1.1.17 % of children 6-23.9 months who received complementary feeding during last 24hours(Health and Agro/Health; all three provinces) 1.1.18 % of PLHIV eating the recommended # of food groups (Health and Agro/Health; sampling should be only in Kirundo province as this is the only site of activities) 1.1.19 % of caregivers seeking professional treatment/advice for infant and child illness(Health and Agro/Health; all three provinces) 1.1.20 % of vulnerable households that achieve a 70% or more on a household hygiene-and-sanitation￾practices score sheet(Health and Agro/Health; all three provinces) 2.1.10 % of targeted households using at least 3 technical recommendations for soil conservation and water management (Agro/Health; all three provinces) 2.1.11 % of vulnerable households that have fully adopted "new" strategies and techniques to increase and diversify their revenues (Agro/Health; all three provinces) 3.1.12 % of vulnerable households and other households with at least a score of 5 out of 5 on the score MYAP Final Evaluation Report DRAFT Annex B 98 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 In addition to these indicators, agriculture household questionnaire that was used in the baseline to cover the Agro/Health zone in all three provinces will be maintained for the final evaluation. sheet (Agro/Health; all three provinces) MYAP Final Evaluation Report DRAFT Annex B 99 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Annex 2: Sampling MYAP final evaluation sampling methodology The final evaluation will use the same sampling methodology as the baseline study (single point in time estimator), with a slight adjustment. The study will design two samples; a)a sample for the first strategic objective (SO1) activities which are implemented throughout the 3 provinces and b) a sample related to the second and third strategic objectives (SO2 and SO3) activities which are implemented in all 61 communities (sous-collines). For the first strategic objective (SO1), the baseline sample size used for the anthropometric survey of the children aged under 5 was calculated based on a single point in time (conservative estimate) stunting growth rate of 50%. A p value of 5% and the two levels of cluster sampling gave a required sample size of 768 children, or 845 children if a 10% reserve is taken. A total of 904 children were sampled in forty clusters or communities (sous-colline) of twenty-two children each. The single point-in-time estimator used is the follow: - D is the “design effect”. Assume D=2 for FFP/TII programs - pF is the value of the indicator at final evaluation (e.g., proportion stunted is 0.5 at final evaluation is the target) - Error is the maximum tolerable half-width for the confidence interval associated with pF Error = .05 is generally used The use of the formula with a PF value of 50% for determining the final evaluation size gives a sample size of 768 as used for the baseline. The final evaluation will keep 768 as the number of children to measure during the study and consider the recommended minimum requirement of thirty clusters instead of the forty clusters used during the baseline. The number of household to visit in order to reach the 768 children is calculated by using the following information (recommended by the sampling guide of R. Magnani): - The average household size: 4.9 persons - Proportion of 6-59 months old children per household: 16.57% - The number of household is inflated by 10% to compensate for non-response. By using the formula recommended by R. Magnani, the total number of households to visit is 1040. For the first level sample, the communities (sous-collines) will be chosen through the Systematic￾Random Sampling with Probability Proportional to Size (PPS) of communities. The sampling will use list (sample frame) of 2,493 sous-collines. The probability of each sous-colline to include to the 1.96 2 *D* pF (1-pF ) (Error) n = 2 where MYAP Final Evaluation Report DRAFT Annex B 100 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 sample is therefore proportional to it size. For the second level, the selection of the household to interview will use the pen method. See details below for choice of households. The survey on mothers’ knowledge and practices of health, nutrition and hygiene will use the same sample of 1040 households as in the baseline. The final evaluation will take a random sample of thirty communities (sous-collines), spread out over the entire zone of the three provinces. In every selected community (sous-colline), the surveyors will collect the data in 35 households. For the second and third strategic objective (SO2 and SO3), the baseline study calculated the sample size based on the following assumption: CRS hoped that a change of 10% in household response, between the baseline survey and the final survey (e.g. from 50% to 40% or from 50% to 60% households) would be significant at p<0.05 and a power of 80% (see FANTA sampling guide: Mangani, 1997). By using the single point-in-time estimator with a PF value of 0.75% (percentage of months of adequate food provisioning) for the final evaluation, the sample size is 576 households. For the first level sample, the communities (sous collines) will be chosen through systematic-random sampling from a list (sample frame) of the 61 sous-collines where the SO2 and SO3 activities are implemented. Given the limited number of clusters (sous-collines), the study will choose 20 clusters (sous-collines) instead of the required minimum of 30 clusters, and in each sous-colline, 29 households will be visited by the surveyors. The second level sample is the selection of household within the community. In each selected community, the choice of households will be done through the pen method. See details below for choice of households. Choice of households In this study, the household is defined as the group of individuals living under the same roof and eating from the same pot, under the responsibility of one person (head of household). For SO1, if more than one child fulfilled the required criteria living in the same household, they will all be included in this survey. The selection of the household to interview will use the following method: From the center of the community, a pen will be thrown to determine the direction towards the community boundary. A path will be followed from the center to the boundary, counting and numbering the houses one side of the road. Upon arrival at the boundary, a random number will be chosen for the first household to visit. For each subsequent household, the next closest household will be chosen. MYAP Final Evaluation Report DRAFT Annex B 101 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 The two tables below show the sample of sous-collines: Sample for SO1 Province Commune Colline Sous-colline NbHH Pop 2011 inf6_59m 1 KAYANZA BUTAGANZWA NYABIBUYE KAMIGINA 252 1342 222 2 KAYANZA GAHOMBO RUZINGATI MUGONA 175 932 154 3 KAYANZA GATARA NYARURAMBI NYARURAMBI 60 320 53 4 KAYANZA KABARORE RUSAMBI KAYANGE 108 575 95 5 KAYANZA KAYANZA KANYAMIREMBE NYAMUGARI 170 905 150 6 KAYANZA MATONGO MPEMBA MPEMBA 99 527 87 7 KAYANZA MUHANGA JIMBI JIMBI 721 3840 636 8 KAYANZA MURUTA KAVOGA GERERO 216 1150 191 9 KAYANZA RANGO KARAMA KARAMA 120 639 106 10 KIRUNDO BUGABIRA GITWE RUYIVYI 287 1529 253 MYAP Final Evaluation Report DRAFT Annex B 102 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 11 KIRUNDO BUGABIRA RUBUGA GASAGARA 329 1752 290 12 KIRUNDO BUSONI KAGEGE KARARIRE 242 1289 214 13 KIRUNDO BUSONI NYABUGENI NYABUGENI 133 708 117 14 KIRUNDO BWAMBARANGWE BUNYWERA BUKINGA 212 1129 187 15 KIRUNDO GITOBE BUGWANA RUGOMERO 975 5193 861 16 KIRUNDO KIRUNDO BUGERA BUGERA 370 1971 327 17 KIRUNDO KIRUNDO KIYANZA RUNZENZE 263 1401 232 18 KIRUNDO NTEGA GISITWE RUTONDE 118 629 104 19 KIRUNDO NTEGA NYEMERA NAKABAGARA 80 426 71 20 KIRUNDO VUMBI KIZIBA I KATO 91 485 80 21 MUYINGA BUHINYUZA GITARAMUKA GITARAMUKA 308 1641 272 22 MUYINGA BUTIHINDA GATWENZI GATWENZI 97 517 86 23 MUYINGA BUTIHINDA WINGOMA WINGOMA 195 1039 172 24 MUYINGA GASHOHO MURUTA KIGUFI 160 852 141 25 MUYINGA GASORWE KIVUBO KIVUBO 172 916 152 26 MUYINGA GITERANYI KIJUMBURA KIJUMBURA 387 2061 342 27 MUYINGA GITERANYI RUKUNGERE RUKUNGERE 184 980 162 28 MUYINGA MUYINGA KAVUMU KAVUMU 150 799 132 29 MUYINGA MUYINGA MWURIRE MWURIRE 165 879 146 30 MUYINGA MWAKIRO KIYANZA NYAMUGARI 53 282 47 MYAP Final Evaluation Report DRAFT Annex B 103 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Sample for SO2 and SO3 Province Commune Colline Sous-colline NbHH Pop 2011 inf6_59m 1 MUYINGA GASHOHO GISANZE￾RUGERERO BURIMA 297 1582 262 2 MUYINGA GASHOHO BWISHA BWISHA 100 533 88 3 MUYINGA GASHOHO GISHAMBUSHA GATARE 252 1342 222 4 KIRUNDO KIRUNDO KINYANGURUBE GISENYI 271 1443 239 5 KAYANZA GAHOMBO KINYONGA KANDARO 176 1012 168 6 KAYANZA GAHOMBO KINYONGA KIBENGA 144 767 127 7 KIRUNDO KIRUNDO YARANDA KIGOZI 367 1955 324 8 KIRUNDO KIRUNDO KIYANZA KIRUHURA 316 1683 279 9 MUYINGA GASHOHO MURAMA KUNGOMA 345 1838 304 10 KIRUNDO KIRUNDO MURAMBA MIGEREKA I 50 266 44 11 MUYINGA GASHOHO GISHAMBUSHA MIHAMA 166 884 147 12 KIRUNDO KIRUNDO YARANDA MUHERO 264 1406 233 13 MUYINGA GASHOHO MURAMA MURAMA 315 1678 278 14 KAYANZA GATARA KIBAYI MUVUMU 440 2344 388 15 MUYINGA GASHOHO BUSASA NGAHO 147 783 130 16 KAYANZA GAHOMBO MIKONI NYABIKERE 121 644 107 17 KIRUNDO KIRUNDO KAVOMO RUGUNGA 150 799 132 18 MUYINGA GASHOHO GISEBEYI RUMANGA 282 1502 249 19 MUYINGA GASHOHO GISHAMBUSHA RUTONDE 288 1534 254 20 KIRUNDO KIRUNDO KIYANZA RWIRI 139 740 123 MYAP Final Evaluation Report DRAFT Annex B 104 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Annex 3: International and Local Consultants’ ToRs Terms of Reference for the Quantitative Assessment Expert and / or firm Final Evaluation CRS Burundi Multi-Year Assistance Program (MYAP) 2008-2012 I. Introduction/Background information Catholic Relief Services (CRS) and International Medical Corps (IMC) Burundi are implementing a four-year USAID-financed Multi Year Assistance Program (MYAP) from August 2008 to August 2012. The objective of the program is to reduce food insecurity among vulnerable populations in three provinces in northeastern Burundi. The underlying causes of food insecurity in post-conflict Burundi are characterized by slow economic reconstruction, continued population growth and repatriation of refugees and the subsequent increased pressure on land and natural resources, frequent natural shocks made more extreme by global climate change, conflicts over land tenure, tenuous political stability, and latent ethnic tension. The MYAP is pursuing the following strategic objectives: SO1: Vulnerable households have enhanced human capacities. SO2: Vulnerable households have enhanced and sustainable livelihood capacity. SO3: Vulnerable communities have enhanced resiliency. II. Overall Objectives of the final evaluation The main objectives of the final evaluation study are to: 1. Assess whether the MYAP outcomes and impacts are achieved in line with the stated goal, objectives and intermediate results; 2. Assess constraints, lessons learned/good practices, opportunities as well as successes in implementation; 3. Determine the relevance and effectiveness of the strategies/approach utilized in the implementation of the program; 4. Assess the sustainability of the program benefit including but not limited to the development of the capacity (knowledge and skills) of stakeholders (community, local government, and partners and Consortium staffs) and quality of the activities (adherence to government and donor technical standards). 5. Compare indicator values at endline against targets and against their values at baseline. 6. Assess challenges to the project and the impact of these challenges on project performance. III. Specific objectives of the quantitative assessment Though the quantitative assessment will be realized concurrently with the qualitative assessment, it is an integral part of the MYAP Final Evaluation Plan. Data collect in this phase of the process will be versed into the final evaluation report. For this reason, successful implementation of the quantitative assessment requires close coordination with the team of International consultants headed by the Team MYAP Final Evaluation Report DRAFT Annex B 105 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Leader. In addition to the overall objectives listed in the MYAP Final Evaluation Plan (FEP), the local consultant and / or firm will achieve the following specific objectives:  Replicate the baseline quantitative assessment in order to inform on the project achievements compared to the baseline results  Use the quantitative parts of the baseline questionnaire (see baseline report Annex 8) to collect the data for the assessment  Quantitatively assess all the project indicators as stated in the IPTT (Indicators Performance Tracking Table and the PMP (Performance Management Plan)  Provide pertinent quantitative information to the final evaluation team  Assess the relevant quantitative indicators requested by the 2011 Food For Peace SAPQ (Standardized Annual Performance Questionnaire) III. Methodology (See proposed methodology in the Final Evaluation Plan) IV. Illustrative list of Reference documents  MYAP 2008-2012 proposal and PREP documents  MYAP baseline report  MYAP Mid Term evaluation report  Annual Result reports  Organizational chart  Maps of sites and distribution of interventions  Training materials  Field trip reports  IPTT (Indicator Performance Tracking Table)  PMP (Performance Management plan)  DQA (Data Quality Assessment) report  USAID/FFP field trip reports  Trigger indicators reports  Food For Peace documents on MYAPs quantitative assessments requirements  Sampling Guide; Robert Magnani; December 1997 V. Timeline The quantitative assessment will be carried out from January 23rd, 2012 in the three provinces covered by the program. VI. Deliverables (see section 6 of the FEP above) The following items constitute the deliverables of the quantitative assessment and are subject to Final Evaluation International Consultant approval:  Preliminary work plan, logistics request, and schedule. MYAP Final Evaluation Report DRAFT Annex B 106 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Description and clarification of proposed methodology (cf. FEP); review/update of Baseline questionnaires and development of quantitative survey instruments. : Due for validation three days following arrival. This should include the following:  Description of the survey type:  Sample design  Sample universe  Methods to be used:  Planned level of statistical precision and power  Sample size calculation  Sampling frame to be used  Respondent selection procedures  Draft questionnaire  Plan for supervisor and enumerator training and questionnaire pre-testing  Analysis plan, including estimation procedures to be used (weighting and other adjustments), tabulation key for the indicators, any plans to analyze by subgroups, etc.  Draft interim reports and assessment report, including the methodology used, pertinent tables and graphs, Final draft report incorporating comments is submitted to International consultant/Team Leader and CRS. VII. Minimum Qualification required for the expert in quantitative assessment The consultant should have the following background:  Master degree in statistics, Demography, Economy or other relevant degree;  Substantial experience in conducting quantitative survey, especially with USAID Food for Peace programs;  Substantial experience in cluster sample design using statics parameters;  Substantial experience in data processing and data analysis using statistics Software;  Bilingual English and French (speaking, reading, and writing);  Ability to work in a team;  Excellent communication skills;  Willingness to travel to remote areas;  Capable of working under time pressure. VIII. How to Apply: Consultants or firm interested in this assignment should send the following information to CRS Burundi by December 30, 2011  Brief cover letter highlighting relevant experience and skills, as well as confirming availability for January/February 2012 time frame;  Curriculum Vitae;  Written proposal (in English) of at least two pages and not more than 5 pages describing the methodology and actions for completion of the final evaluation;  A writing sample in English from a previous consultancy. A writing sample in French is highly recommended.  Three professional references with phone numbers and/or email addresses; MYAP Final Evaluation Report DRAFT Annex B 107 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  One page budget indicating daily fee and other related consultancy costs. The above materials should be sent by email to: offres@bi.caro.crs.org MYAP Final Evaluation Report DRAFT Annex B 108 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Terms of Reference for the Agriculture Expert (International hire) Final Evaluation CRS Burundi Multi-Year Assistance Program (MYAP) 2008-2012 I. Introduction/Background information Catholic Relief Services (CRS) and International Medical Corps (IMC) Burundi are implementing a four-year USAID-financed Multi Year Assistance Program (MYAP) from August 2008 to August 2012. The objective of the program is to reduce food insecurity among vulnerable populations in three provinces in northeastern Burundi. The underlying causes of food insecurity in post-conflict Burundi are characterized by slow economic reconstruction, continued population growth and repatriation of refugees and the subsequent increased pressure on land and natural resources, frequent natural shocks made more extreme by global climate change, conflicts over land tenure, tenuous political stability, and latent ethnic tension. The MYAP is pursuing the following strategic objectives: SO1: Vulnerable households have enhanced human capacities. SO2: Vulnerable households have enhanced and sustainable livelihood capacity. SO3: Vulnerable communities have enhanced resiliency. II. Objectives of the final evaluation The main objectives of the final evaluation study are to: 1. Assess whether the MYAP outcomes and impacts are achieved in line with the stated goal, objectives and intermediate results; 2. Assess constraints, lessons learned/good practices, opportunities as well as successes in implementation; 3. Determine the relevance and effectiveness of the strategies/approach utilized in the implementation of the program; 4. Assess the sustainability of the program benefit including but not limited to the development of the capacity (knowledge and skills) of stakeholders (community, local government, and partners and Consortium staffs) and quality of the activities (adherence to government and donor technical standards). 5. Compare indicator values at endline against targets and against their values at baseline. 6. Assess challenges to the project and the impact of these challenges on project performance. III. Specific objectives for the Agriculture Expert (Qualitative evaluation) Under the supervision of the Team Leader, this consultant and the rest of the team will analyze and interpret data collected. Specific objectives of the Agriculture/Food security expert are as follows below: .  Assess the achievements in the agriculture and animal husbandry sectors,  Evaluate the relevance, effectiveness and efficiency of the agriculture and animal husbandry interventions. MYAP Final Evaluation Report DRAFT Annex B 109 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Review the results achieved by the agriculture and animal husbandry sectors, particularly at the strategic objective level,  Assess the sustainability of the achievements of the agriculture and animal husbandry sectors. . IV. Composition of the evaluation team  One international Livelihoods/Community Resilience expert who will serve as the Team Leader  One international expert with a strong experience in food security issues such as local crops and agricultural challenges, natural resource management, monitoring and evaluation, program management, familiarity with post-war agricultural development context, familiarity with MYAP evaluations and having a strong leadership quality. One international Nutrition/health expert  One international/national Commodity expert V. Evaluation Questions General Questions:  Did the project do what it aimed to do?  Did the project make a difference in the lives of the intended beneficiaries and other stakeholders?  Was the project’s approach suitable for achieving the set objectives? Design, Implementation and Achievements:  Are agriculture and animal husbandry sector activities appropriate for the food security problems identified in the selected target areas?  What interventions have been more or less successful in meeting targets?  What is the level of completion of key outputs and did this result in achievement of intermediate results and strategic objectives as specified in the approved proposal? Where intermediate results and strategic objectives were not achieved, what factors hindered their achievement?  Which interventions are most critical and/or effective in achieving the agriculture and animal husbandry sector objectives and intermediate results? And why?  What improvements can be made to the design and implementation of such sectors to improve results?  Are the agriculture and animal husbandry sectors well-integrated in the USAID Burundi and Government of Burundi strategies? Are there steps that could have been taken to improve integration as well as food security impacts through greater integration?  What has been the level of coordination/collaboration with Government of Burundi and other actors to leverage government and/or other programs’ resources/interventions for greater impact?  Assess the extent to which recommendations from the mid-term evaluation have been incorporated into the agriculture and animal husbandry sectors.  What has been the level of coordination/collaboration with Government of Burundi and other actors to leverage government and/or other programs’ resources/interventions for greater impact? Program impact on beneficiaries MYAP Final Evaluation Report DRAFT Annex B 110 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  What is the impact of the agriculture and animal husbandry sectors planned interventions on the targeted beneficiaries’ food security?  Are beneficiaries adopting desired practices or behaviors? Are there secondary adopters?  How effective are the agriculture and animal husbandry sectors in targeting and reaching the most vulnerable communities and households? How appropriate were the selection of provinces, communities and individual beneficiaries?  How effective are the agriculture and animal husbandry sectors at reaching women? What could be done to improve women’s participation and leadership in such sectors? Are MYAP agriculture and animal husbandry activities succeeding in transforming gender relations? If not, what can be done to improve these activities?  Are there certain groups within the population with lower rates of adoption and why? Capacity strengthening:  Are training materials appropriate for the participants?  Have the agriculture and animal husbandry sectors effectively enabled, or developed the capacity of beneficiaries? If not how could the design or implementation be altered to improve capacity strengthening?  Have the agriculture and animal husbandry sectors effectively developed the capacity of partners (partners in implementation as well as government partners)? If not, how could the design or implementation be altered to improve capacity strengthening? Sustainability:  Are there any factors that limited community participation and engagement in the program implementation?  Has the program effectively collaborated with local administrative bodies such as provincial agricultural departments, local governments, etc?  Are the outcomes related to adoption of better practices sustainable? Are participants likely to continue after the project ends?  Is there a well-developed exit strategy? If so, has the program moved forward to initiate some aspects of that strategy?  Which outcomes are likely or unlikely to be sustainable, and why?  What can be done to increase the sustainability in future projects? Program monitoring and performance:  Are the agriculture and animal husbandry activities and indicators relevant to meeting the MYAP objectives and targets?  To what extent has the agriculture and animal husbandry sector implementation and management been high-quality, effective, efficient and relevant? What are the weaknesses and the relevant recommendations to solve them for the future?  Does the M&E system defined provide useful and reliable data on the agriculture and animal husbandry sector progress and impacts (through performance indicators)?  Does the technical staff use M&E data and anecdotal information to conduct their work and assess progress? How can they use it more effectively? Specific agriculture and animal husbandry questions  Are the technologies and practices being promoted well-established and well-suited to the local agroecological environments? MYAP Final Evaluation Report DRAFT Annex B 111 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Are farmers able to obtain improved and recommended inputs without program assistance? If not, what would be required in order that they could do so?  What is the agriculture and animal husbandry sectors’ impact on household access to food? What aspects/interventions have had the greatest impact on household food security?  To what extent are the natural resource management activities sustainable (i.e. lowland water management and hillside anti-erosion activities)?  Does the use of food for work for participation in agricultural production related activities at as an incentive/disincentive to improving productivity, and how?  Are farmer and other community groups able to maintain productive infrastructure on their own? If not, why and what could be done in future programming to address this limitation? VI. Methodology (See proposed methodology in the Final Evaluation Plan) VII. Timeline The final evaluation will be carried out from January 23 - March 30, 2012 in the provinces covered by the program. VIII. Deliverables (see FEP) IX. Minimum Qualification required for the expert in Agriculture/Team Leader The consultant should have the following background:  Master’s, Engineer, PhD degree in agronomy, agro economy or other relevant diploma  Five to ten years experience in development project/program management, especially with USAID Food for Peace programs  Substantial experience in program evaluations (quantitative and qualitative methods), especially with USAID Food for Peace programs  Experienced with post-war agricultural development context  Relevant knowledge of the national agriculture policy  Bilingual English and French (speaking, reading, and writing)  Ability to work in a team  Excellent communication skills  Willingness to travel to remote areas  Capable of working under time pressure X. How to Apply: Consultants interested in this assignment should send the following information to CRS Burundi by December 30, 2011  Brief cover letter highlighting relevant experience and skills, as well as confirming availability for January/February 2012 time frame; MYAP Final Evaluation Report DRAFT Annex B 112 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Curriculum Vitae;  Written proposal (in English) of at least two pages and not more than 5 pages describing the methodology and actions for completion of the final evaluation;  A writing sample in English from a previous consultancy. A writing sample in French is highly recommended.  Three professional references with phone numbers and/or email addresses;  One page budget indicating daily fee and other related consultancy costs. The above materials should be sent by email to: offres@bi.caro.crs.org MYAP Final Evaluation Report DRAFT Annex B 113 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Terms of Reference for the Nutrition and Health Expert (International Hire) Final Evaluation CRS Burundi Multi-Year Assistance Program (MYAP) 2008-2012 I. Introduction/Background information Catholic Relief Services (CRS) and International Medical Corps (IMC) Burundi are implementing a four-year USAID-financed Multi Year Assistance Program (MYAP) from August 2008 to August 2012. The objective of the program is to reduce food insecurity among vulnerable populations in three provinces in northeastern Burundi. The underlying causes of food insecurity in post-conflict Burundi are characterized by slow economic reconstruction, continued population growth and repatriation of refugees and the subsequent increased pressure on land and natural resources, frequent natural shocks made more extreme by global climate change, conflicts over land tenure, tenuous political stability, and latent ethnic tension. The MYAP is pursuing the following strategic objectives: SO1: Vulnerable households have enhanced human capacities. SO2: Vulnerable households have enhanced and sustainable livelihood capacity. SO3: Vulnerable communities have enhanced resiliency. II. Overall Objectives of the final evaluation The main objectives of the final evaluation study are: 1. Assess whether the MYAP outcomes and impacts are achieved in line with the stated goal, objectives and intermediate results; 2. Assess constraints, lessons learned/good practices, opportunities as well as successes in implementation; 3. Determine the relevance and effectiveness of the strategies/approach utilized in the implementation of the program; 4. Assess the sustainability of the program benefit including but not limited to the development of the capacity (knowledge and skills) of stakeholders (community, local government, and partners and Consortium staffs) and quality of the activities (adherence to government and donor technical standards). 5. Compare indicator values at end line against targets and against their values at baseline. 6. Assess challenges to the project and the impact of these challenges on project performance. III. Specific objectives for the Nutrition and Health Expert . Quantitative data would have been collected prior in a replication of the baseline methodology (population-based sample survey). Under the supervision of the Team Leader, this consultant and the rest of the team will analyze and interpret data collected. Specific objectives of the Nutrition and Health Expert are as follows:  To assess the achievements in the nutrition and health sector based on the indicators as stated in the approved proposal. In considering these achievements, please consider the relevance, effectiveness and efficiency of the interventions;  To assess the effectiveness of coordination with the government and other actors. MYAP Final Evaluation Report DRAFT Annex B 114 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  To conduct an analysis of the results achieved by the nutrition and health sector, particularly at the strategic objective level,  To conduct a statistical analysis of health and nutrition outcomes and compare to the target populations’ current records or any baseline information;  To document “lessons learned” including strengths and weaknesses, and provide recommendations for future programming as well as other needs in the nutrition and maternal/child health sector that were not addressed IV. Composition of the evaluation team  One international Livelihoods/Community Resilience who will serve as the Team Leader  One international agricultural expert  One international Nutrition/health expert  One international Commodity expert V. Evaluation Questions General Questions:  Did the project do what it aimed to do?  Did the project make a difference in the lives of the intended beneficiaries and other stakeholders?  Was the project’s approach suitable for achieving the set objectives? Design, Implementation and Achievements:  Are nutrition and health sector planned activities appropriate for the food security problems identified in the selected target areas?  What interventions have been more or less successful in meeting targets?  What is the level of effectiveness of the implementation of the exit strategies developed for the nutrition and health sector?  Which interventions are most critical and/or effective in achieving the nutrition and health sector objectives and intermediate results? And why?  What improvements can be made to the design of such a sector to improve results?  What improvements can be made in the implementation of the nutrition and health sector in order to improve results?  Are there opportunities for integrating nutrition and health sector components that could result in greater food security impacts?  What are the factors that hinder/assist the effective integration of components?  Is the nutrition and health sector well-integrated in the USAID Burundi strategy? Are there steps that could have been taken to improve integration as well as food security impacts through greater integration?  Is the nutrition and health sector well-integrated in the local government’s strategy and priorities? Are there steps that could have been taken to improve the integration as well as food security impacts through greater integration?  Assess the extent that recommendations from the mid-term evaluation have been incorporated into the nutrition and health sector?  What is the level of completion of key outputs and did these results in achievement of intermediate results and strategic objectives as specified in the approved proposal? Where MYAP Final Evaluation Report DRAFT Annex B 115 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 intermediate results and strategic objectives were not achieved, what factors hindered their achievement?  What has been the level of coordination/collaboration with Government of Burundi and other actors to leverage government and/or other programs’ resources/interventions for greater impact? Program impact on beneficiaries  What is the impact of the nutrition and health sector planned interventions on the targeted beneficiaries’ food security?  Are beneficiaries adopting desired practices or behaviors? Are there secondary adopters?  How effective is the nutrition and health sector in targeting and reaching the most vulnerable communities and households? How appropriate were the selection of provinces, communities and individual beneficiaries?  How effective is the nutrition and health sector at reaching fathers/men? What could be done in future programming to improve father/men’s participation in such sectors?  Are there certain groups within the population with lower rates of adoption and why? Capacity strengthening:  Are training materials appropriate for the participants?  Has the nutrition and health sector effectively enabled, or developed the capacity of beneficiaries? If not how could the design or implementation be altered in future programming to improve capacity strengthening?  Has the nutrition and health sector effectively developed the capacity of government health partners (health center staff, community health workers)? If not, how could the design or implementation be altered in future programming to improve capacity strengthening? Sustainability:  Are the outcomes related to adoption of better practices sustainable, ie. Are participants likely to continue after the project ends?  Which outcomes are likely or unlikely to be sustainable, and why?  What can be done to increase the sustainability?  Is there a well-developed exit strategy? If so, has the program moved forward to initiate some aspects of that strategy?  Are there any factors that limited community participation and engagement in the program implementation?  Has the program effectively collaborated with local administrative bodies such as provincial health departments, local governments, etc? Program monitoring and performance:  Are the nutrition and health activities and indicators relevant to meeting the MYAP objectives and targets?  To what extent has the nutrition and health sector implementation and management been high￾quality, effective, efficient and relevant? What are the weaknesses and the relevant recommendations to solve them for the future?  Does the M&E system defined provide useful and reliable data on the nutrition and health sector progress and impacts (through performance indicators)? MYAP Final Evaluation Report DRAFT Annex B 116 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Does the technical staff use M&E data and anecdotal information to conduct their work and assess progress? How can they use it more effectively? Specific health and nutrition questions  Does the intervention reflect the nutrition and health problems facing the community?  How successful has the activity been able to leverage government preventative and curative health, water and sanitation and related social services?  What has been the population coverage of the intervention and could it be improved?  Is staff able to advise beneficiaries on safe and appropriate uses of the ration including substitution with local commodities following the end of the program?  Are health and nutrition BCC materials appropriate – tailored to the user, actionable, accurate and linked to growth promotion messages?  What is being done to improve the capabilities of the staff and local partners to respond to community needs and meet the objectives of the program?  How effective has the exit strategy been?  What role if any is there for program "voluntary" health and nutrition promoters receiving incentives, monetary or other, from their communities to continue their work after the program is completed?  Do the health and nutrition volunteers make home visits or do any follow-up? If not, why not and what can be done to increase the likelihood that they do?  Is there a health and nutrition volunteer attrition problem? If so, why do they drop out and what can be done to reduce the rate of attrition? VI. Methodology (See proposed methodology in the Final Evaluation Plan) VII. Timeline The final Evaluation will be carried out from January 23- February 25, 2012 in the provinces covered by the program. A draft version of the report should be submitted prior to departure from Burundi. The final report should be submitted one week following submission of CRS’ feedback on the draft version. VIII. Deliverables The following items constitute the deliverables of the Final Evaluation:  Preliminary work plan, logistics request, and schedule: Due to team leader  Interview Guide(s): Input to be submitted to team leader who will submit for validation following arrival.  Tools for data collection  Report outline, highlighting major sections and themes to be covered, specifically addressing all the objectives cited above  Restitution workshop for MYAP staff and the stake holders to present the initial findings. This will be conducted by the evaluation team  Draft evaluation report, including the methodology used, pertinent tables and graphs, quantitative and qualitative information, lessons learned and recommendations: To be submitted on departure. MYAP Final Evaluation Report DRAFT Annex B 117 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Final draft report incorporating comments from CRS and FFP: Due one week following receipt of comments.  Electronic version of all raw data and tables drawn from the data (on a CD) IX. Minimum Qualification required for the expert in nutrition and health The consultant should have the following background:  Master’s, PhD degree in nutrition or public health, or other relevant diploma  Five to ten years experience in development project/program management, especially USAID Food for Peace programs  Substantial experience in program evaluations (quantitative and qualitative methods), especially USAID Food for Peace programs  Extensive experience conducting and analyzing data from anthropometric surveys.  Relevant knowledge of the national health and nutrition policy  Bilingual English- French  Ability to work in a team  Excellent communication skills  Willingness to travel to remote areas  Capable of working under time pressure  Submission of sample writing of evaluation report X. How to Apply: Consultants interested in this assignment should send the following information to CRS Burundi by December 30, 2011  Brief cover letter highlighting relevant experience and skills, as well as confirming availability for January/February 2012 time frame;  Curriculum Vitae;  Written proposal (in English) of at least two pages and not more than 5 pages describing the methodology and actions for completion of the final evaluation;  A writing sample in English from a previous consultancy. A writing sample in French is highly recommended.  Three professional references with phone numbers and/or email addresses;  One page budget indicating daily fee and other related consultancy costs. The above materials should be sent by email to: offres@bi.caro.crs.org MYAP Final Evaluation Report DRAFT Annex B 118 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Terms of Reference for the Team Leader/Livelihoods/Community Resilience Expert (International Hire) Final Evaluation CRS Burundi Multi-Year Assistance Program (MYAP) 2008-2012 I. Introduction/Background information Catholic Relief Services (CRS) and International Medical Corps (IMC) Burundi are implementing a four-year USAID-financed Multi Year Assistance Program (MYAP) from August 2008 to August 2012. The objective of the program is to reduce food insecurity among vulnerable populations in three provinces in northeastern Burundi. The underlying causes of food insecurity in post-conflict Burundi are characterized by slow economic reconstruction, continued population growth and repatriation of refugees and the subsequent increased pressure on land and natural resources, frequent natural shocks made more extreme by global climate change, conflicts over land tenure, tenuous political stability, and latent ethnic tension. The MYAP is pursuing the following strategic objectives: SO1: Vulnerable households have enhanced human capacities. SO2: Vulnerable households have enhanced and sustainable livelihood capacity. SO3: Vulnerable communities have enhanced resiliency. II. Overall Objectives of the final evaluation The main objectives of the final evaluation study are: 1. Assess whether the MYAP outcomes and impacts are achieved in line with the stated goal, objectives and intermediate results; 2. Assess constraints, lessons learned/good practices, opportunities as well as successes in implementation; 3. Determine the relevance and effectiveness of the strategies/approach utilized in the implementation of the program; 4. Assess the sustainability of the program benefit including but not limited to the development of the capacity (knowledge and skills) of stakeholders (community, local government, and partners and Consortium staffs) and quality of the activities (adherence to government and donor technical standards). 5. Compare indicator values at endline against targets and against their values at baseline. 6. Assess challenges to the project and the impact of these challenges on project performance. III. Specific objectives of the Livelihoods/Community Resilience Expert The international consultant team is a multi-disciplinary team headed by the Livelihoods/Community Resilience expert who serves as the Team Leader. The local consultant, it is an intricate part of the MYAP Final Evaluation Study (see Final Evaluation Plan) therefore realized under the stewardship of the Team Leader . MYAP Final Evaluation Report DRAFT Annex B 119 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Quantitative data would have been collected prior in a replication of the baseline methodology (population-based sample survey). Specific objectives of the Livelihoods/Community Resilience Expert and Final Evaluation Team Leader are as follows:  Provide oversight for the design and execution of the quantitative surveys by making ascertain that quantitative survey methodology is in conformity with overall FEP methodology,  Lead the team and ensure quality of the work throughout the evaluation process.  Give due attention during all phases of development of data collection tools, data analysis, report writing and other relevant deliverables by the local consultant on behalf of the CRS and its partners.  Be responsible for leading sector specialists (international experts).  Be responsible for reviewing and/or finalizing the deliverables by the local consultant as stated in section 6 of the FEP.  Analyze the effectiveness of coordination with the government and other actors,  Measure program outputs and outcomes based on a population-based baseline survey and or evaluation reports as well as with the indicators.  Identify constraints (logistics and operations in general),  Capture qualitative information that is core to CRS Integral Human Development framework to support the quantitative information.  Document lessons learned and provide recommendations for future programming,  Assess the achievements in the livelihoods and community resilience sectors,  Assess the relevance, effectiveness and efficiency of the livelihoods and community resilience interventions,  Assess the results achieved by the livelihoods and community resilience sectors, particularly at the strategic objective level,  Assess the sustainability of the achievements of the livelihoods and community resilience sectors. Within this objective, it is necessary to assess the effectiveness of coordination with the government and other actors,  Synthesize contributions of agriculture, nutrition and commodity experts into one document. IV. Composition of the evaluation team  One international expert with a strong experience in Livelihoods/Community Resilience expert who will serve as the team leader.  One international agricultural and food security expert  One international Nutrition/health expert  One international Commodity expert V. Evaluation Questions General Questions:  Did the project do what it aimed to do?  Did the project make a difference in the lives of the intended beneficiaries and other stakeholders?  Was the project’s approach suitable for achieving the set objectives? Design, Implementation and Achievements: MYAP Final Evaluation Report DRAFT Annex B 120 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Are livelihoods and community resilience sector activities appropriate for the food security problems identified in the selected target areas?  What interventions have been more or less successful in meeting targets?  What is the level of completion of key outputs and did this result in achievement of intermediate results and strategic objectives as specified in the approved proposal? Where intermediate results and strategic objectives were not achieved, what factors hindered their achievement?  Which interventions are most critical and/or effective in achieving the livelihoods and community resilience sector objectives and intermediate results? And why?  What improvements can be made to the design and implementation of such sectors to improve results?  Are the livelihoods and community resilience sectors well-integrated in the USAID Burundi and Government of Burundi strategies? Are there steps that could have been taken to improve integration as well as food security impacts through greater integration?  What has been the level of coordination/collaboration with Government of Burundi and other actors to leverage government and/or other programs’ resources/interventions for greater impact?  Assess the extent that recommendations from the mid-term evaluation have been incorporated into the livelihoods and community resilience sectors.  What has been the level of coordination/collaboration with Government of Burundi and other actors to leverage government and/or other programs’ resources/interventions for greater impact? Program impact on beneficiaries  What is the impact of the livelihoods and community resilience sectors planned interventions on the targeted beneficiaries’ food security?  Are beneficiaries adopting desired practices or behaviors? Are there secondary adopters?  How effective are the livelihoods and community resilience sectors in targeting and reaching the most vulnerable communities and households? How appropriate were the selection of provinces, communities and individual beneficiaries?  How effective are the livelihoods and community resilience sectors at reaching women? What could be done to improve women’s participation and leadership in such sectors? Are MYAP livelihoods and community resilience activities succeeding in transforming gender relations? If not, what can be done to improve these activities?  Are there certain groups within the population with lower rates of adoption and why? Capacity strengthening:  Are training materials appropriate for the participants?  Have the livelihoods and community resilience sectors effectively enabled, or developed the capacity of beneficiaries? If not how could the design or implementation be altered to improve capacity strengthening?  Have the livelihoods and community resilience sectors effectively developed the capacity of partners (partners in implementation as well as government partners)? If not, how could the design or implementation be altered to improve capacity strengthening? Sustainability: MYAP Final Evaluation Report DRAFT Annex B 121 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Are there any factors that limited community participation and engagement in the program implementation?  Has the program effectively collaborated with local administrative bodies such as provincial agricultural departments, local governments, etc?  Are the outcomes related to adoption of better practices sustainable? Are participants likely to continue after the project ends?  Is there a well-developed exit strategy? If so, has the program moved forward to initiate some aspects of that strategy?  Which outcomes are likely or unlikely to be sustainable, and why?  What can be done to increase the sustainability in future projects? Program monitoring and performance:  Are the livelihoods and community resilience activities and indicators relevant to meeting the MYAP objectives and targets?  To what extent has the livelihoods and community resilience sector implementation and management been high-quality, effective, efficient and relevant? What are the weaknesses and the relevant recommendations to solve them for the future?  Does the M&E system defined provide useful and reliable data on the livelihoods and community resilience sector progress and impacts (through performance indicators)?  Does the technical staff use M&E data and anecdotal information to conduct their work and assess progress? How can they use it more effectively? Specific livelihoods and community resilience questions  Are there certain groups within the target population better able to access SILC loans, and why? If there are groups who are unable to access loans, should the program be broadened to include these groups and how could future programs incorporate them?  Have SILC groups contributed directly to household food security, and how?  To what extent do households adopt gender messages on shared decision-making?  How successful has MYAP’s work with agro-enterprise associations? What can be done to improve the design and implementation of agro-enterprise activities?  Has the MYAP succeeded in making gender a cross-cutting theme throughout project activities?  How relevant are community action plans to the context? What can be done to improve the design and implementation of community action plans? VI. Methodology (See proposed methodology in the Final Evaluation Plan) VII. Timeline The final evaluation will be carried out from January 23-March 30, 2012 in the three MYAP provinces. VIII. Deliverables The following items constitute the deliverables of the Final Evaluation:  Preliminary work plan, logistics request, and schedule: Due to team leader MYAP Final Evaluation Report DRAFT Annex B 122 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Interview Guide(s): Input to be submitted to team leader who will submit for validation following arrival.  Tools for data collection  Report outline, highlighting major sections and themes to be covered, specifically addressing all the objectives cited above  Restitution workshop for MYAP staff and the stake holders to present the initial findings. This will be conducted by the evaluation team  Draft evaluation report, including the methodology used, pertinent tables and graphs, quantitative and qualitative information, lessons learned and recommendations: To be submitted on departure.  Final draft report incorporating comments from CRS and FFP: Due one week following receipt of comments.  Electronic version of all raw data and tables drawn from the data (on a CD) IX. Minimum Qualification required for the expert in livelihoods/community resilience The consultant should have the following background:  Master’s, Engineer, PhD degree in agronomy, agro economy, international development, gender and development or other relevant diploma  Five to ten years experience in development project/program management, especially with USAID Food for Peace programs  Substantial experience in program evaluations (quantitative and qualitative methods), especially with USAID Food for Peace programs  Experienced with post-war agricultural development context  Relevant knowledge of the Burundi rural development context  Bilingual English and French (speaking, reading, and writing)  Ability to work in a team  Excellent communication skills  Willingness to travel to remote areas  Capable of working under time pressure X. How to Apply: Consultants interested in this assignment should send the following information to CRS Burundi by December 30, 2011  Brief cover letter highlighting relevant experience and skills, as well as confirming availability for January/February 2012 time frame;  Curriculum Vitae;  Written proposal (in English) of at least two pages and not more than 5 pages describing the methodology and actions for completion of the final evaluation;  A writing sample in English from a previous consultancy. A writing sample in French is highly recommended.  Three professional references with phone numbers and/or email addresses;  One page budget indicating daily fee and other related consultancy costs. The above materials should be sent by email to: offres@bi.caro.crs.org MYAP Final Evaluation Report DRAFT Annex B 123 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Scope of Work for Commodity Expert (Seconded from USAID/FFP) Final Evaluation CRS Burundi Multi-Year Assistance Program (MYAP) 2008-2012 I. Introduction/Background information Catholic Relief Services (CRS) and International Medical Corps (IMC) Burundi are implementing a four-year USAID-financed Multi Year Assistance Program (MYAP) from August 2008 to August 2012. The objective of the program is to reduce food insecurity among vulnerable populations in three provinces in northeastern Burundi. The underlying causes of food insecurity in post-conflict Burundi are characterized by slow economic reconstruction, continued population growth and repatriation of refugees and the subsequent increased pressure on land and natural resources, frequent natural shocks made more extreme by global climate change, conflicts over land tenure, tenuous political stability, and latent ethnic tension. The MYAP is pursuing the following strategic objectives: SO1: Vulnerable households have enhanced human capacities. SO2: Vulnerable households have enhanced and sustainable livelihood capacity. SO3: Vulnerable communities have enhanced resiliency. II. Overall Objectives of the final evaluation The main objectives of the final evaluation study are: 1. Assess whether the MYAP outcomes and impacts are achieved in line with the stated goal, objectives and intermediate results; 2. Assess constraints, lessons learned/good practices, opportunities as well as successes in implementation; 3. Determine the relevance and effectiveness of the strategies/approach utilized in the implementation of the program; 4. Assess the sustainability of the program benefit including but not limited to the development of the capacity (knowledge and skills) of stakeholders (community, local government, and partners and Consortium staffs) and quality of the activities (adherence to government and donor technical standards). 5. Compare indicator values at end line against targets and against their values at baseline. 6. Assess challenges to the project and the impact of these challenges on project performance. III. Specific objectives of the commodity expert Quantitative data will have been collected prior in a replication of the baseline methodology (population-based sample survey). Under the supervision of the Team Leader, this consultant and the rest of the team will analyze and interpret data collected. Specific objectives of the Commodity Expert are as follows:  To assess the achievements in commodity management and distributions, especially considering the relevance, effectiveness and efficiency of the commodity interventions, MYAP Final Evaluation Report DRAFT Annex B 124 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  To assess the efficiency and effectiveness of commodity management and logistics systems as well as compliance with USG regulations  To document “lessons learned” and provide recommendations for future programming IV. Composition of the evaluation team  One international Livelihoods/Community Resilience expert who will serve as the Team Leader  One international Nutrition/health expert  One international agricultural expert  One international Commodity expert with strong experience in commodity management and safety net interventions including identification / selection of vulnerable people, food planning and delivery, monitoring food management and distribution, etc. V. Evaluation Questions General Questions:  Did the project do what it aimed to do?  Did the project make a difference in the lives of the intended beneficiaries and other stakeholders?  Was the project’s approach suitable for achieving the set objectives? Design, Implementation and Achievements:  Are commodity interventions appropriate for the food security problems identified in the selected target areas?  What interventions have been more or less successful in meeting targets?  What is the level of effectiveness of the implementation of the exit strategies developed for commodity activities?  Which commodity interventions are most critical and/or effective in achieving the desired objectives and intermediate results? And why?  What improvements can be made to the design to improve results?  What improvements can be made in the implementation of the commodity activities in order to improve results?  Are there opportunities for better integrating commodity activities that could result in greater food security impacts?  What are the factors that hinder/assist the effective integration of components?  Are the commodity activities well integrated in the USAID Burundi strategy? Are there steps that could have been taken to improve integration as well as food security impacts through greater integration?  Are the commodity activities well integrated in the local government’s strategy and priorities? Are there steps that could have been taken to improve the integration as well as food security impacts through greater integration?  Assess the extent that recommendations from the mid-term evaluation have been incorporated into the commodity activities?  Is the commodity package appropriate to the nutritional needs of the beneficiaries? Program impact on beneficiaries MYAP Final Evaluation Report DRAFT Annex B 125 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  What is the impact of the planned commodity interventions on the targeted beneficiaries’ food security?  Are beneficiaries adopting desired practices or behaviors associated with commodity activities? Are there secondary adopters (i.e. people in the community that did not receive food but adopted the behavior)?  How effective are the commodity interventions in targeting and reaching the most vulnerable communities and households? How appropriate were the selection of provinces, communities and individual beneficiaries?  How effective are the commodity activities at reaching women? What could be done to improve women’s participation in such sectors?  Are there certain groups within the population with lower rates of participation in commodity activities and behavior adoption and why? Capacity strengthening:  Are training materials appropriate for the commodity staff?  Has the commodity management in both CRS and IMC effectively developed the capacity of partners (health center staff, HIV associations)? If not how could the design or implementation be altered to improve capacity strengthening? Sustainability:  Is there a well developed exit strategy? i.e Are other providers of health rations likely to continue providing commodities (WFP, government, etc)? Are people substituting local foods in place of food aid following program graduation, especially for health supplementary feedings? If so, has the program moved forward to initiate some aspects of that strategy?  Are there any factors that limited community participation and engagement in the program implementation?  Has the program effectively collaborated with local administrative bodies such as provincial agricultural departments, local governments, etc? Program monitoring and performance:  Are the commodity activities and indicators relevant to meeting the MYAP objectives and targets?  To what extent has the commodity activity implementation and management been high-quality, effective, efficient and relevant? What are the weaknesses and the relevant recommendations to solve them for the future?  What is the level of achievement of commodity distribution targets? If these levels were not achieved, what factors prevented this?  Does the M&E system defined provide useful and reliable data on the commodity sector progress and impacts?  Does the technical staff use M&E data and anecdotal information to conduct their work and assess progress? How can they use it more effectively? Specific commodity questions  Are correct procedures and best practices used in receiving, distributing and storing food commodities?  Are commodities being managed appropriately?  Is the food distribution system design efficient? MYAP Final Evaluation Report DRAFT Annex B 126 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011  Are there good procedures to monitor food distributions and in a timely manner?  Are the recommended food rations respected? What are the weaknesses of the strategies and what are the recommendations to address them?  Does the current system protect against “false” beneficiaries from receiving commodities?  Are beneficiaries well-targeted for food rations?  Do beneficiaries understand the quantities of food that they should receive?  To what extent are the natural resource management activities sustainable (i.e. lowland water management and hillside anti-erosion activities)? Does the use of food for work for participation in agricultural production related activities at as an incentive/disincentive to improving productivity, and how? VI. Methodology (See proposed methodology in the Final Evaluation Plan) VII. Timeline The final Evaluation will be carried out from January 23-February 25, 2012 in the provinces covered by the program. A draft version of the report should be submitted prior to departure from Burundi. The final report should be submitted one week following submission of CRS’ feedback on the draft version. VIII. Deliverables The following items constitute the deliverables of the Final Evaluation:  Preliminary work plan, logistics request, and schedule: Due to team leader  Interview Guide(s): Input to be submitted to team leader who will submit for validation following arrival.  Tools for data collection  Report outline, highlighting major sections and themes to be covered, specifically addressing all the objectives cited above  Restitution workshop for MYAP staff and the stake holders to present the initial findings. This will be conducted by the evaluation team  Draft evaluation report, including the methodology used, pertinent tables and graphs, quantitative and qualitative information, lessons learned and recommendations: To be submitted on departure.  Final draft report incorporating comments from CRS and FFP: Due one week following receipt of comments.  Electronic version of all raw data and tables drawn from the data (on a CD) IX. Minimum Qualification required for the expert in commodities The consultant should have the following background: - Master’s or PhD degree in finances, microfinance, agro economy, management or other relevant diploma. - Five to ten years experience dealing with commodity management and safety net interventions - Relevant knowledge of the safety net interventions systems and networks in Burundi; MYAP Final Evaluation Report DRAFT Annex B 127 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 - Substantial experience in commodity program evaluations (quantitative and qualitative methods) - Bilingual English- French (the evaluation report will be written in English) - Ability to work in a team - Excellent communication skills - Willingness to travel to remote areas - Capable of working under time pressure; X. How to Apply: Consultants interested in this assignment should send the following information to CRS Burundi by December 30, 2011  Brief cover letter highlighting relevant experience and skills, as well as confirming availability for January/February 2012 time frame;  Curriculum Vitae;  Written proposal (in English) of at least two pages and not more than 5 pages describing the methodology and actions for completion of the final evaluation;  A writing sample in English from a previous consultancy. A writing sample in French is highly recommended.  Three professional references with phone numbers and/or email addresses;  One page budget indicating daily fee and other related consultancy costs. The above materials should be sent by email to: offres@bi.caro.crs.org MYAP Final Evaluation Report DRAFT Annex B 128 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 Annex 4: Module B. Household Roster The first step in carrying out a household survey is to collect information about the composition of the household. This is often referred to as collecting the household roster. The household roster provides information that allows you to identify who in the household is potentially eligible. Ask of the head of household or another responsible adult member of the household. No. Question Response codes Responses ASK OF THE HEAD OF HOUSEHOLD OR ANOTHER RESPONSIBLE ADULT MEMBER OF THE HOUSEHOLD. B1 Hello. My name is __________ and I work for __________. We are conducting a survey about __________. The information we collect will be used for __________. You have been selected by chance for this survey and we would very much appreciate your participation. The survey usually takes about ___ minutes. Your participation is voluntary and you may end the survey at any time or decide not to answer a particular question. Your answers will be kept confidential. Do you agree to participate in the survey? 0 = No >> end module 1 = Yes B2 Do you have any questions for me about the survey before we begin? ANSWER THEIR QUESTIONS. Please tell me the name and sex of each person who lives here, starting with the head of the household. Let me tell you a little bit about what we mean by [‘household.’] For our purposes today, members of a [household] are adults or children that live together and eat from the ‘same pot.’ It should include anyone who has lived in your house for 6 of the last 12 months, but it does not include anyone who lives here but eats separately. ADAPT THE DEFINITION OF ‘HOUSEHOLD’ TO YOUR CONTEXT. LIST THE RESPONDENT, [head of household] ON LINE 1. LIST THE NAMES OF ALL HOUSEHOLD MEMBERS. THEN ASK: Does anyone else live here even if they are not at home now? These may include children in school or household members at work. IF ‘YES,’ COMPLETE THE LISTING. THEN, COLLECT THE REMAINING COLUMNS OF INFORMATION FOR EACH MEMBER, ONE PERSON AT A TIME. ADD A CONTINUATION SHEET IF THERE ARE MORE THAN 10 HOUSEHOLD MEMBERS. MYAP Final Evaluation Report DRAFT Annex B 127 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 o (1) Name (2) Relationship to head of household 0 = Head of household 1 = Spouse 2 = Child 3 = Grandchild 4 = Parent/ grandparent 5 = Sibling (including in-law) 6 = Cousin 7 = Nephew/niece 8 = Aunt/uncle 9 = Other (3) Is [name] male or female? 0=male 1=female (4) ELIGIBLE FOR MODULE C PUT A CHECKMARK IF THIS PERSON WAS IN CHARGE OF THE HOUSEHOLD’S FOOD PREPARATION YESTERDAY (5) Please tell me how old [name] is. How old was [name] on his/her last birthday? RECORD AGE IN COMPLETED YEARS 98=DK, USE ONLY FOR ≥ 50 YEAR OLDS (6) ELIGIBLE FOR MODULE D PUT A CHECKMARK IF THIS IS A CHILD UNDER 6 YEARS OLD (7) FOR EACH CHILD UNDER 6 YEARS OLD: Who is the primary caregiver of [name]? RECORD LINE NUMBER OF PRIMARY CAREGIVER* (8) ELIGIBLE FOR MODULE E PUT A CHECKMARK IF THIS IS A WOMAN AGED 15-49 YEARS (9) ELIGIBLE FOR MODULE F PUT A CHECKMARK IF THIS IS THE HEAD OF HOUSEHOLD (IF HEAD OF HOUSEHOLD IS ABSENT, ANY RESPONSIBLE ADULT CAN BE INTERVIEWED INSTEAD) (10) ELIGIBLE FOR MODULE G PUT A CHECKMARK IF PERSON IS A FARMER** (SEE DEFINITION BELOW) (11) RESULT CODE 1 = Completed 2 = Refused 3 = Respondent still absent after 3 attempts 4 = Other (specify) _____ 1 Other: 2 Other: 3 Other: 4 Other: 5 Other: 6 Other: 7 Other: 8 Other: 9 Other: 10 Other: DEFINITIONS: * The primary caregiver is the person who knows the most about how and what the child is fed. Usually (but not always) this will be the child’s mother. ** Farmers (including herders and fishers) are: 1) men and women who have access to a plot of land (even if very small) over which they make decisions about what will be grown, how it will be grown, and how to dispose of the harvest; AND/OR 2) men and women who have animals and/or aquaculture products over MYAP Final Evaluation Report DRAFT Annex B 128 Catholic Relief Services Burundi award FFP-A-00-08-00080 Final Evaluation Plan revised December 14 2011 which they have decision-making power. Farmers produce food, feed, and fiber, where ‘food’ includes agronomic crops (crops grown in large scale, such as grains), horticulture crops (vegetables, fruit, nuts, berries, and herbs), animal and aquaculture products, as well as natural products (e.g., non-timber forest products, wild fisheries). These farmers may engage in processing and marketing of food, feed, and fiber and may reside in settled communities, mobile pastoralist communities, or refugee/internally displaced person camps. An adult member of the household who does farm work but does not have decision￾making responsibility over the plot OR animals would not be considered a ‘farmer.’ For instance, a woman working on her husband's land who does not control a plot of her own would not be interviewed. Annex 5: Questionnaires Annex 6: Results Framework Annex 7: IPTT Annex 8: Performance Management Plan (PMP) MYAP Final Evaluation Report DRAFT Annex C 129 Annex C. Qualitative Interviews and Site Visits Evaluation Team The qualitative evaluation team was composed of the following persons: Mike DeVries, Livelihoods Specialist (Team Leader) – Responsible for coordinating implementation of the evaluation, including overseeing preparations for the quantitative survey, coordinating the qualitative component of the evaluation and primarily responsible for SO3 and assigned program process investigations, facilitation of meetings and workshops, preparation of debriefings, and completion of the final report. Bernard Cren, Agriculture and Natural Resource Management Specialist – Responsible for investigations under SO2 as well as assigned program process investigations, including drafting sections of the evaluation report on these topics. Mirella Mokbel Genequand, Maternal and Child Health Specialist – Responsible for investigations under SO1 as well as assigned program process investigations, including drafting sections of the evaluation report on these topics. Ali Aamoum, Commodity Management Specialist – Responsible for investigations related to commodity management, including drafting sections of the evaluation report on these topics. Sampling Approach The evaluation team randomly selected sites to visit and people to interview according to the folowing schedule. FIELD VISIT SCHEDULE DATES LOCATION 11 March Travel to Kirundo 12-15 March Kirundo 15 March (pm) Travel to Ngozi 16-17 March Muyinga 18 March Processing Day - No Field Work 19-20 March Muyinga 21-24 March Kayanza 24 (pm) or 25 March Return to Bujumbura Over the course of the four-day field visit to each province in which the MYAP is being implemented, the interviews, focus group discussions and site visits were scheduled as follows.  On arrival in a Province, orientation meeting (one hour) with program implementing staff in the Province. After which, the evaluation team moved on to separate schedules as shown on the following pages. MYAP Final Evaluation Report DRAFT Annex C 130 For SO1 Total Numbe r Visits Kirundo Muyinga Kayanza 12 13 14 15 16 17 19 20 21 22 23 24 Briefing/ debriefin g CRS & IMC x x x x x x STA 4 Rukura migabo 35 Bunyar i Mirwa Maram vya SSN 4 Rushub ije Muyin ga Gashoh o Gikom ero SST 3 Kirund o Hospit al Muyin ga hospita l Kayanz a Hospit al SPC 2 Gitara muka Rukag o and/or Musem a VIH 1 or 2 To be determi ned36 CARE GROUP 4 Kinyag urube or Kavom o 37 Gashoh o Mikoni or Gahom 35 and/or Bunyari on the 14th 36 as no food distribution will take place that week, arrange meeting with HIV association in charge and visit households individually ?? 37 or Yaranda or Mwenya MYAP Final Evaluation Report DRAFT Annex C 131 Muram ba bo FARN 2 Yarand a Mwend o 38 PHAST 3 Muram ba or Kinyan gurube Kavom o or Yarand a Gashoh o ACRONYMS/DEFINITIONS FARN = Foyers d’Apprentissage et de Réhabilitation Nutritionnelle LM = Lead Mothers in Care Groups PHAST = Participatory Hygiene and Sanitation Transformation SST= Service de Stabilisation intégré dans les hôpitaux pour la prise en charge des cas sévères ayant les complications médicales STA = Service Thérapeutique Ambulatoire, intégré dans les Centres de Santé (CDS) pour la prise en charge des cas sévères sans complications médicales SSN = Service de Supplémentation Nutritionnelle, intégré également dans les CDS pour la prise en charge des cas modérés SPC = Suivi et Promotion de la Croissance VCBHA = Volunteer Community-Based Health Activist. RATIONALE FOR/ORGANISATION OF FIELD VISITS Choice of types of activities (STA, FARN, etc.) and their proposed dates has been guided by the monthly planning of activities in each province. For instance for health facility-based activities (SSN & STA), health centers will be visited the day food is being distributed in order to: a) interview as many beneficiaries as possible and b) observe processes (weighing, recording, counseling, nutrition education, cooking demonstrations, food distribution, etc.). SST can be visited any day as beneficiaries are admitted into the hospital, and activities are daily 38 or Ngoma or Nyamwera or Nyamitanga MYAP Final Evaluation Report DRAFT Annex C 132 For Care Groups: it is proposed to join the planned LM meeting; at the end of the meeting go with 1 LM into her house and meet some of her neighbor HHs whom she monitors. FARN: it is proposed to join the planned IMC visit to a new FARN starting in Kirundo and a supervision visit to an existing one in Kayanza PHAST: join planned education session/visit PENDING: Where and how to meet a group of VCBHA? They might be present at the LM meeting (to be verified) HIV (Kirundo): no food distribution planned while team in Kirundo so either:  Meet with members of one HIV association (3 are collaborating with MYAP) and visit a few beneficiary HH individually  Return to Kirundo one day of the following week SO1 FIELD VISITS Tota l No Kirundo Site/Commune Muyinga Site/Commune Kayanza Site/Commune HEALTH FACILTY-BASED ACTIVITIES Ambulatory Therapeutic Feeding (STA) 3 Rukuramigabo/Kirundo Muyange/Bugabira and Gasura/Vumbi - - Supplementary Feeding (SSN) 5 Rushubije/Ntega Muyinga/Muyin ga Gashoho/Gasho ho Rukago/Gahom bo Gikomero/Rang o Therapeutic Feeding (SST) 3 Kirundo Hospital Muyinga Hospital Kayanza Hospital Growth Monitoring 2 Gasura/Vumbi Gitaramuka/Bhinyusa - HIV 2 Kirundo COMMUNITY-BASED ACTIVITIES Mother Care Group 3 Kinyagurube/Kirundo Gashoho/Gashoho Rukago/Gahombo MYAP Final Evaluation Report DRAFT Annex C 133 PD/Hearth 3 Yaranda/Kir undo Rugero/Kirun do Masama/Muhanga PHAST 2 Mwenya/Kirundo Gashoho/Gashoho - OTHER Focus Group Discussions and Individual Interviews HIV/AIDS Beneficiaries (Kirundo) IMC Staff MOH Nutrition Focal Point VCBHAs (Kamararamagambo/Butihinda) IMC Staff MOH Nutrition Focal Point VCBHAs IMC Staff MOH Technical Health Promoter MYAP Final Evaluation Report DRAFT Annex C 134 For SO2 and agriculture activities under SO1 Direct Observations: 1) Marais infrastructure 2) Model farmers (as positive deviants) 3) LAE, contour Bunds with trees & grasses, 4) CARE/PLHIV groups: bean, cassava, etc. 5) Keyhole gardens (SILC, PLHIV, other) 6) Nurseries 7) Rams and goat farmers 8) Fields for production and multiplication 9) Agro-Enterprises List of SO2 benefs in database: Beneficiaire amenagement marais Beneficiaire Chaine de solidarite Beneficiaire Chevres Beneficiaire Distrib direct manioc Beneficiaire Distrib direct P Douce Cultivateur Haricot 2011 A,B,C,12A Cultivateur Pomme de terre Cultivateur Riz FFW entretien manioc FFW LAE Membre associations pepinieres Membre assos agro entrprise Membre comite marais Membre comites Elevage Membre comites LAE Membre groupement SILC Membre SILC et referent technique Multiplicateur manioc Multiplicateur Patate Douce Care group and PLHIV members Sample schedule in each province (try to have meetings in different collines) FGD last no more than 90 minutes Day 1 Day 2 Day 3 Day 4 AM 8-10 Session 1 Briefing with staff FGD1 Col 2 – FGD 5 Col 4 – FGD 8 Col 5 – FGD 11 AM 10- 11:30 Session 2 Col 1 – FGD 2 Col 3 – FGD 6 Col 4 – FGD 9 Col 6 – FGD 12 11:30-13:00 Session 3 Col 1 – FGD 3 Col 3 – FGD 7 Col 5 – FGD 10 Col 6 – FGD 13 PM 13- 14:30 Session 4 Col 2 – FGD 4 D.O., other D.O., other Debriefing with staff PM 14:30- 16 Session 5 Direct Observations, other D.O., other D.O., other Debriefing with staff MYAP Final Evaluation Report DRAFT Annex C 135 FGD Kirundo . goal of 8-16 participants from cross-section of population and area -- gender: ½ women and ½ men . as much as possible choose all different people (not same person in several activities) Per Province Kirundo 1. CRS/partner field staff SO2 2. Marais committees (19) 2-3 members each of committees: Muramba, Yaranda, Kinyangurube, Kavomo, Kiyanza, Mwenya 3. DPAE (ag+vet agents) (21+?) As many as possible 4. LAE committees (20) 2-3 members each of committees: Muramba, Yaranda, Kinyangurube, Kavomo, Kiyanza, Mwenya 5. Model farmers (134 – 1/sous colline?) Aim for 2 each from each colline RWASA Gloriose, NIYONZIMA Isaac, SERUTAGOMA Alphonse, NAMIBURO Pelagie, MUSABWASONI Anastasie, MUHIZIWINTORE Déo MUKANYONGA Frediane, MUTABAZI Elias MUKASONI Audile, MIBURO Michel KANYARUKORI J.Baptiste, NIBOYE Esperance 6. Other beneficiary farmers Not committee, not model, Aim for 2 each from each colline NAHIMANA Venentie, ZIRIKUNANA Leocadie, MUKERABIRORI Pascaline, TOYI Marie, KABARUTA Eugenie, MAPINE Pierre, MANIRAMBONA Venantie, BAYUBAHE Bernard, KAMATAMO Gaudence, NSAVYIMANA Liberathe, NDAYISENGA Egide, NDAYISENGA Euphrasie, CITEGETSE Sophie, KANYARUKORI J.Baptiste 7a. SILC groups (268), 7b. SILC Field agents (TA) 7c. SILC Super Agents (10) Aim for 2 each from each colline KANA Tharcisse, MUKANTWARI Francine, NYABUTIGA Gaspard, NIYONSABA Chantal, HABIYAKARE Ferdinand, NIYOKWIZERA Evelyne, NKEZABAHIZI Gaetan, MACUMI Anastasie, MUKAMANA Jeanine, RUBERINTWARI Jean Bosco, KANDAMA Jeanine, NDAYISHIMIYE Gaspard 8. Care group members (183) 1 each from 12-16 s/c 1 each from 12-16 s/c 9. Agro-Entreprises (12) 2-3 members each of all AE 8-15 people 10. Nurseries (5) 2 members each from 5-7 farmer associations 8-15 people 11. Goat mgmt committees (57) Bizimungu Joseph, Hagenimana Joselyne, Miburo Sylver, Niyonsaba Adèle, MYANDAGARO Barthazard, KAMARIZA Jeanine, Murasandonyi Frédéric, Butoyi Domitille, Nyabenda Berina, Niyonkuru Claver, Sezirahiga Jean Claude, Mutarutwa Virginie 12. Ram owners (288) MUKANTARE Sylvie, NDACAYISABA MYAP Final Evaluation Report DRAFT Annex C 136 Not part of animal committees Aoron, RUKERATABARO Nestor (NYABENDA Muramba ) ), NYABENDA Gilbert, CIZA Gaspard, MUKANDEKEZI Virginie (Ciza Mathieu ), RUSHIMATWARI Salomon, NGABONZIZA Charles, HAKIZIMANA Asman, SEZIRAHIGA Juvenal, KARENZO Pierre, NKURIKIYE Pascal 13. Goat beneficiaries (Chaine de Solidarité), not part of animal committees NKEZABAHIZI Claver, NTARAKA Marie, BITANA Petronie, MVUYEKURE Felicien, MINANI Pascaline, RWASA Zacharie, NDURURUTSE Marie, KANZIZA Marcienne, MADENDE Ernest, NDINZABAHIZI Isidore, NZEYIMANA Matilde, NTAHONTUYE Angele, MACUMII Bernard MYAP Final Evaluation Report DRAFT Annex C 137 FGD Muyinga . goal of 8-16 participants from cross-section of population and sous collines -- gender: ½ women and ½ men . as much as possible choose all different people (not same person in several activities) . Gisebeyi,Kinyami, Gishambusha, Rugerero, Muzingi, Busasa // Kobero, Bunyarukiga, Bonero, Murama, Nyagatovu, Bwisha Per Province Muyinga 1. CRS/partner field staff SO2 2. Marais committees (19) 2-3 members each of committees: Gisebeyi,Kinyami, Gishambusha, Kobero, Bunyarukiga, Bonero 3. DPAE (ag+vet agents) (21+?) As many as possible As many as possible 4. LAE committees (20) 2-3 members each of committees: Gishambusha, Rugerero, Muzingi, Bunyarukiga, Bonero, Murama 5. Model farmers (134 – 1/sous colline?) YAMUREMYE Jacques, NTIRIMENINDA Noella, CISHAHAYO Jean Marie, NZIRORERA Mtherese, KARIMWOMENSHI Oscar, ZIRIKUNAMA Sapiensia, MINANI Diodola, COYITUNGIYE J Paul, KABATESI Mireille, SIKUBWABO Oswald, NZEYIMANA Protais, RWASA Acquilline 6. Other beneficiary farmers Not committee, not model, This will be done through walking in the collines 7a. SILC groups (268), 7b. SILC Field agents Kandenzi Angel, Gaitobwa Agath, Nsaguye Marc, Ndenzako Patricie, Barabona Sylvie, Mpabonimana Anselme, Nibigira Vital, Murekerisoni Violette, Ruvugo Jean, Habimana Francine, Nzeyimana protais, Wakarerwa desiderate, Kamaraba Cesalie, Tuyaga Stany, Nabashengezi Felicite, Ndiracuza Melchior, Kandava Jaqueline, Hakorimari Jean Pierre, Miburo Sarela, Nahimana Patrice, Kankindi Colette, Cishahayo Ezechiel 8. Care group members (183) 1 each from 12-16 s/c 1 each from 12-16 s/c 9. Agro-Entreprises (12) 2-3 members each of all AE 8-15 people 10. Nurseries (5) 2 members each from 5-7 farmer associations 8-15 people 11. Goat mgmt committees (57) Misigaro Domitille, Munezero Guillome, Ntanyungu Spesiose, Ndayingurutse Sylvain, Minani Immaculee, Banyanse Merchior, Nyabenda Marie, Citatira Jean, Kampayano Magyy, Gucenene joseph, Nzoyisaba Christine, Mfasheniryo Mathias 12. Ram owners (288) Not part of animal committees Nyabenda Prosper, Rwasa Ananias, Bitako Charles, Safari Eric, Misago Pierre, Cimpaye Melanie, Nyabenda Leonce, Nduwimana Livella, Mubamba Simon, Kayitesi Mireille, Manirambona Desire MYAP Final Evaluation Report DRAFT Annex C 138 13. Goat beneficiaries (Chaine de Solidarité), not part of animal committees Kankindi Consolate, Vyamungu Issa, Bigirimana Andre, Miburo Agathe, Niyontuntu Jean, Niyonzima Collete, Nibimpa Joseph, Basekari Dorothee, Gahutu Leonard, Cishahayo Emelyne, Nsabimana Charles, Cishahoyo Anesie, Nahingejeje Euphrasie MYAP Final Evaluation Report DRAFT Annex C 139 FGD Kayanza . goal of 8-16 participants from cross-section of population and sous collines -- gender: ½ women and ½ men . as much as possible choose all different people (not same person in several activities) Per Province Kayanza 1. CRS/partner field staff SO2 2. Marais committees (19) 3-4 members each of committees: Kinyonga, Mikoni, Kibayi 3. DPAE (ag+vet agents) (21+?) As many as possible As many as possible 4. LAE committees (20) 3-4 members each of committees: Kinyonga, Mikoni, Kibayi 5. Model farmers (134 – 1/sous colline?) Nahimana Menedore, Misigaro Marcien, Niyibigira Adèle, Bankuwunguka Maxime, Ntakimazi Casimir, Sinzumusi Marie Therese, Barinayera Pascal, Vumiriya Angèle, Nyambuga Pamphile, Ntirampeba Vivine, Nahimana Déo, Girukwishaka Béatrice 6. Other beneficiary farmers Not committee, not model, This will be done through walking in the collines 7a. SILC groups (268), 7b. SILC Field agents NONGABIRE Benigne, NYABENDA Servillien, NSHIMIRIMANA Siyeri, HARERIMANA Dorothee, NTIRUJINAMA Sylvain, NGENDABANYIKWA Cesarie, WABUSA Deo, NDAYIRAGIJE Serapfine, NTIBASHIRINZIGO Anthere, AHISHAKIYE Generose, TUYIKEZE Aline, NDUWAYEZU Evariste 8. Care/PLHIV group members 1 each from 12-16 s/c 1 each from 12-16 s/c 9. Agro-Entreprises (12) 2-3 members each of all AE 8-15 people 10. Nurseries (5) 2 members each from 5-7 farmer associations 8-15 people 11. Goat mgmt committees (57) NKURIKIYE Simeon, Nyandwi Barbine, Nahimana Prudence, Hakizimana Darie, Nyandwi Anatole, Manirakiza Violette, Ntabahungu Charles, Harerimana Odette, Rwajekera Salvator, NTAKIRUTIMANA Elvanie, Hakizimana Damien, Ndabarushimana Emelence 12. Ram owners (288) Not part of animal committees Habarugira Egide, Uwimana Angelle, Namatwi Violette, Niyonkuru Gabriel, Gateretwenimana Felicit, Bwabo Seleverien, Ntarukundo Charlotte, Minani Anselme, Manirambona Jean, Nyabenda Fabiola, Niyibizi Ezechiel, Sindayigaya Agathe 13. Goat beneficiaries (Chaine Nibigira Domitille, Niyonsaba Charles, MYAP Final Evaluation Report DRAFT Annex C 140 de Solidarité), not part of animal committees GAHUNGU Simon, NIYIBIZI Genevieve, Misigaro Vianney, Mvuyekure Cesarie, Nkurunziza Marcien, Nsabayezu Agnes, BARINAYERA Pascal, NAHIMANA Leoncie, Uwimana Rebecca, Yamuremye Janvier, BIGIRIMANA dieudonne, HABONIMANA Angeline List of Key Informant Interviews 1 Sylvain Duhau CRS MYAP Field Coordinator 2 Tom Remington CRS Regional Technical Advisor for Agriculture 3 Jean Baptiste Bigrimana DPAE Kirundo, Chargé du Génie rural 4 Clement ??? DPAE Muyinga, Directeur 5 Marius Bucumi DPAE Kayanza, Directeur 6 Fidèle Garunga ISABU, MINIAGRIE, Directeur du Departement des Etudes du Milieu et des Systèmes de Production List of semi-structured group interviews with technical staff (Marais, Agro-forestry/LAE, Livestock, SILC, Agriculture) 1 5 staff CRS technical staff (Kirundo) 2 4 staff CRS technical staff (mobile team) 3 5 staff ODEDIM technical staff (Muyinga) 3 6 staff (includes 1 M&E) BADEC technical staff (Kayanza) MYAP Final Evaluation Report DRAFT Annex C 141 For SO3, Program Processes and Potable Water under SO1: KIRUNDO PROVINCE MYAP Activity Location Methodology Water Point Rehabilitation Kinyinya 1 Water Point, Gahombo Commune, Mikoni Colline, Nyabikere Sous Colline 1. FGD with the Water Committee (1 1/2 hours) 2. FGD with a group of 10 FFW recipients including 5 men and 5 women (1 1/2 Hours) Gatoke Water Point, Gatara Commune, Kibayi Colline, Kibayi Sous Colline Early Warning and Emergency Action Plans Yaranda Colline 1. FGD with 10 officers and KIyanza Colline members of the PAC (1 1/2 Hours) Kirundo Commune 1. Interview with a knowledgeable representative of Local Administration (1 hour) Kirundo Province 1. Interview with a knowledgeable representative of the Governor's Office (1 hour) Gender Activities Muramba Colline 1. Combined interview with all Gender Focal Points in the commune (1 1/2 Hours) 2. Combined Interview with all Confirmed Gender Positive Deviants in the colline (1 1/2 Hours) 1. FGD with ten lead mothers (1 hour) Kinyangurube Colline KInyanza Colline Program Processes (Management, Partnership, Knowledge Management, Integration, Finance, HR) 1 Interview with the Provincial Coordinator (1 Hour) 2. Interview with Gender Specialist (1 Hour) 3. Interview with Community Action Planning Specialist (1 Hour) 4. Interview with a knowledgeable representative of the DPEA at the Province Level (1 hour) MUYINGA PROVINCE MYAP Activity Location Methodology Water Point Rehabilitation Gisasa Water Point, Gashoho Commune, Bwisha Colline, Mpete Sous Colline 1. FGD with the Water Committee (1 1/2 hours) 2. FGD with a group of 10 FFW recipients including 5 men and 5 women (1 1/2 Hours) Kanyanzogera Water Point, Gashoho Commune, Gisebeyi Colline, Murago Sous Colline Early Warning and Emergency Action Plans Muzingi Colline 1. FGD with 10 officers and Kobero Colline members of the PAC (1 1/2 Hours) Gashoho Commune 1. Interview with a knowledgeable representative of Local Administration (1 hour) MYAP Final Evaluation Report DRAFT Annex C 142 Muyinga Province 1. Interview with a knowledgeable representative of the Governor's Office (1 hour) Gender Activities Nyagatovu Colline 1. Combined interview with all Gender Focal Points in the commune (1 1/2 Hours) 2. Combined Interview with all Confirmed Gender Positive Deviants in the colline (1 1/2 Hours) 3. FGD with ten lead mothers (1 hour) 4. FGD with ten male SO2 participants (1 hour) Busasa Colline Program Processes (Management, Partnership, Knowledge Management, Integration, Finance, HR) 1 Interview with Pascal, Provincial Coordinator (1 Hour) 2. Interview with Abbe Jonathan, Director ODEDIM (1 hour) 3. Interview with Maurice, IMC Program Assistant (1 Hour) 4. Interview with M&E Officer, unless still vacant (1 Hour) 5. Interview with a knowledgeable representative of the DPEA at the Province Level (1 hour) KAYANGA PROVINCE MYAP Activity Location Methodology Water Point Rehabilitation Gahanga Water Point, Gahombo Commune, Kinyonga Colline, Kibenga Sous Colline 1. FGD with the Water Committee (1 1/2 hours) 2. FGD with a group of 10 FFW recipients including 5 men and 5 women (1 1/2 Hours) Kinwera Water Point, Gatara Commune, Kibayi Colline, Kinwera Sous Colline Early Warning and Emergency Action Plans Mikoni Colline 1. FGD with 10 officers and Kinyonga Colline members of the PAC (1 1/2 Hours) Gahombo Commune 1. Interview with a knowledgeable representative of Local Administration (1 hour) Muyinga Province 1. Interview with a knowledgeable representative of the Governor's Office (1 hour) Gender Activities Mikoni Colline 1. Combined interview with all Gender Focal Points in the commune (1 1/2 Hours) 2. Combined Interview with all Confirmed Gender Positive Deviants in the colline (1 1/2 Hours) 3. FGD with ten lead mothers (1 hour) 4. FGD with ten male SO2 Kibayi Colline MYAP Final Evaluation Report DRAFT Annex C 143 participants (1 hour) Program Processes (Management, Partnership, Knowledge Management, Integration, Finance, HR) 1 Interview with the Simon, Provincial Coordinator (1 Hour) 2. Interview with Abbe Jean Baptiste, Director of Badec (1 hour) 3. Interview with Christian, IMC Program Assistant (1 Hour) 4. Interview with Polycarpe, M&E Officer (1 Hour) 5. Interview with a knowledgeable representative of the DPEA at the Province Level (1 hour) MYAP Final Evaluation Report DRAFT Annex C 144 Persons Met for Commodity Management Katherine Overcamp, Program Manager, CRS, Tel.79524529 Melkemu Dereb, CRS Commodity Manager, Tel. 79215654 Sylvain Duhau, head CRS Kirundo Office, Tel. 79521144 Sisay Amanuel , Roving Logistics Coordinator, phone 79238005, IMC Willy Nkurikye, Logistics Coordinator, phone # 79238020, IMC Christian Mayifuil, Site Manager, phone # 79945201, IMC Guy Nziyumvira , Support Statistics, phone# 79928486, IMC Nicodeine Bicamumpaka , chef de poste SST, Kayanza, phone# 79996210 Felecian Mzikiriza, surveillant Medical , Kayanza, Phone# 71510904 Mbomabuca , titulaire de santé Mdikumama Chrestine, Assistant magaziniere, CRS Ngozi, Ndayishimiye Esperance, responsable service de stabilisation, Muyinga, Tel. 79599770 Maurice , IMC Program Assistant, Muyinga, tel. 71147982 Teophile, IMC Program Assistant Kirundo, tel. 79944880 Umizeyimama Leonie, Personnel d’appui, Tel. 79716714 Buimi Jean de Dieu , Titulaire, centre de sante Mukenke, Kirundo, tel. 79338629 Ntibazonkiza Judith , CDS , Travailleuse, Buhoro, Tel.79162538 Macumi Leonardie , Nutritionnel , Centre de sante de Cumba, tel. 79574426 Simbayobewe Emmanuel , aide soignant, cds Bwasare, Muyinga, tel. 79321339 Uwimana Colimie, personnel d’appui, cds Gashoho, Muyinga, tel. 79333288 Nsabimana Elie, personnel d’appui, cds Muyenge, Kirundo, tel 79227561 Butoyi Anesie , personnel d’appui, cds Rukuramgabo, Kirundo, tel. 71455520 Kalissa Ibrahim, Polucon Services LTD, Tel. 71388980 Malumi Claudette, aide soignante, cds Mangondo, Kirundo, Tel.79256752 Sinrinbago Bernard, Infermier, cds Rokago, Kayanza, tel. 79643123 Sinzinkayo Celestin, Infermier, cds Rokago, Kayanza, tel. 79715184 Minani Deo, Travailleur, cds Maramuya, Kayanza, tel. 79172184 Pascal Ntirandekura, Coordinator ODIDEM Muyinga, tel. 79939931 Nocolas Gie, Directeur General, SDV, Tel.76445936 MYAP Final Evaluation Report DRAFT Annex D 145 Annex D. SO2 Livelihoods Summary Statistics SO2 Achievements To Date Kirundo Muyinga Kayanza Total # of HH 5,873 7,421 1,226 14,520 # of collines 6 12 3 21 # of sous collines 22 34 7 63 # of households 5,873 7,421 1,226 14,520 MA (Marais) - hectares 104ha 65ha 68ha 237ha Cost/ha; study+infrastructure+FFW @$1/d+ CFW $1,704 $1,166 # of direct beneficiaries 1,772 1,692 1,099 4,563 # of zones 9 6 8 # of committees 9 6 4 19 Transferred to authorities Nov 2011 provisional Oct 2011 Dues collected 130,000Fbu No 500F/ in June ‘12 ha in rice / # of farmers 80% 50% (850) 3% (55) # of FFW benefs 1031 615 2,560 4,206 LAE (Lutte Anti Erosion) – km FFW 142km 260km 344km 746km LAE – km or # of farmers non-FFW 10%-12% 100+ farmers 10-15% LAE committees 11 6 3 20 % cover marais / % cover total (estimation) ? / ? 60-70% / ? 95%/ 95% # of FFW benefs 1236 995 1635 3,866 LI (Livestock) – rams/mortality 100/9 90/5 98/20 288/34 LI - # of goats CRS 1,800 1,800 1,800 5,400 LI - # of goats redistributed 751 472 461 1,684 LI – committees 22 34 7 63 ATT - Model farmers 50 52 32 134 bean seed (fairs + distribution) 4,089 3,975 3,586 11,650 ATT – cassava cuttings 9,147 975 962 11,084 ATT- OFSP 3,852 975 1,077 5,904 ATT – White potato (failed) 17 18 10 45 ATT- fairs / # benefs 4/3,700 4/3,650 3/3,000 11/10,350 Mother-baby plots 42 36 76 154 Multiplier groups and individuals 110 138 38 286 Vegetable seed to People Living with HIV 483 381 500 1,364 AE - # Agro-Entreprise and total beneficiaries 5 / 195 6 / 69 6 / 166 17 / 430 AE -# tree nurseries and (total beneficiaries) 2 (48) 1 (40) 2 (80) 5 ( 168) AE - associations 42 28 51 121 SILC groups - # CRS/ #IMC / # independent 82 / 26 / 16 41 / 25 / 10 57 / 47 / 10 314:180/98/36 Estimated % belonging to 2 groups 30% 30% 10% ~30% double Estimated % belonging to 3 groups 10% 5% 2% counting Estimated % belonging to 4 groups 2% 0% 0% Estimated coverage of adult population (2/HH) 20% 10% 70% 15% # of members / # of beneficiaries 2,425/1,600 1,247/840 2,337/1,540 6,009/4,000? MYAP Final Evaluation Report DRAFT Annex D 146 # of Technical Agents 9 6 4 19 Keyhole gardens (through SILC) 0 3 40+ 43+ DPAE Moniteurs Agricoles 6 11 3 20 ACSA none none 3 3 Agricultural technology Rationale 1) Proper use of organic fertilizer  Increases production  Improves soil quality  Decreases risk of water pollution/contamination  Encourages maximum use of local inputs  Increases the value of livestock  Decreases need for imported commercial fertilizers 2) Mulching  Increases organic matter in top layer  Significantly decreases erosion from wind and rain  Increases water infiltration rate  Decreases weed pressure  Increases soil nutrients  Decreases soil compaction  Reduces labor required for weeding and tillage 3) Diversification of crops  Reduces the risk of crop failure  Spreads out labor requirements and food availability  Potentially improves diet  Diversifies income potential 4) Agroforestry  Increase long-term productivity  Decrease risk  Decrease wind/water erosion  Diversify income potential  Increase local fuel production  Spread out and/or reduces labor requirements  Improve the local environment 5) Reduced/ conservation tillage  Reduces soil erosion  Reduces labor requirements  Increases organic matter in soil  Improves soil structure  Can reduce weed pressure 6) Proper seed Spacing  Helps maximize yields  May decrease weed pressure  Promotes efficient use of seed 7) Planting in rows  Helps maximize yields  Makes tillage, weeding, mulching, and harvesting easier  Important component of contouring  Promotes efficient use of seed MYAP Final Evaluation Report DRAFT Annex D 147 Agricultural technology Rationale 8) Composting  Improves soil tilth  Increases crop yields  Promotes recycling of organic materials  Reduces waste and pollution  Reduces food available to insects and rodents 9) Proper crop rotation  Promotes healthier soil  Increases overall production  Decreases risk of losses to disease/insects 10) Fallowing  Improves soil structure and fertility  Reduces erosion  Reduces disease pressure 11) Proper seed selection and conservation  Increases production potential  Reduces crop input expenses  Reduces risk 12) Crop disease recognition  Reduces risk  Increases mitigation options 13) Animal disease recognition  Reduces risk  Increases mitigation options 14) Proper livestock feeding and management  Improves productivity  Reduces risk  Decreases veterinary expenses  Maximizes use of local produce 15) Tree/plant production in nurseries  Increases local availability of beneficial trees/plants  Encourages cooperation/collaboration by community members  Encourages agroforestry  Raises awareness/knowledge of the benefits of trees in the community 16) Water management (in lowland inland valleys)  Increases crop production  Increases availability of food during hungry season  Decreases erosion/flooding  Maximizes benefits from available water resources  Reduces risk  Reduces water pollution levels 17) Proper post-harvest handling  Decreases food losses  Improves food quality  Decreases risk  Increases food availability in hungry season  Potentially increases income  Reduces presence of insects and rodents 18) Integrated pest management  Enhances crop and livestock production  Reduces the need for costly chemical pesticides  Reduces environmental pollution 19) Multiple use of water  Increases efficiency  May reduce labor requirements  Can reduce water pollution  Increases production MYAP Final Evaluation Report DRAFT Annex D 148 Agricultural technology Rationale 20) Intercropping  Diversifies production/income  Can increase production  Allows for income while slower growing plants/trees mature  Decreases soil erosion  Decreases insect, weed and disease pressure  Increases land use efficiency 21) Proper pruning  Increases productivity  Improves harvestability 22) Proper fertilizer/ pesticide handling and application  Decreases human health risks  Increases plant productivity and resilience  Decreases pollution potential  Increases soil fertility  Decreases insect and disease pressure  Improves efficiency of land use 23) Proper harvest techniques  Increases income/food available for consumption  Increases quality and value of produce  Can increase following season’s production potential MYAP Final Evaluation Report DRAFT Annex E 149 Annex E: SO3 PAC and Gender Summary Statistics SousCollines Water Action Plan Acteurs Deviants Positifs Demographics Province Commune Colline Interventions Committees Committees Relais Confirmed Others HH Population Kayanza Gahombo Kinyonga All SOs 6 3 10 20 22 326 1490 Mikoni All SOs 6 3 10 15 10 300 1395 Gatara Kibayi All SOs 3 4 10 15 11 600 2672 Muyinga Gashoho Gisebeyi All SOs 3 4 9 4 1 769 4123 Kinyami All SOs 0 3 9 5 2 418 1693 Gishambusha All SOs 2 5 10 7 0 897 2433 Rugerero All SOs 0 4 9 8 0 832 3298 Muzingi All SOs 1 6 10 11 6 1124 3820 Busasa All SOs 0 3 9 5 2 400 1653 Kobero SO2 + SO3 0 3 9 4 3 321 1372 Bunyarukiga SO2 + SO3 0 3 9 2 3 463 1895 Bonero SO2 + SO3 0 5 9 5 2 686 2717 Murama SO2 + SO3 0 3 9 4 1 612 2542 Nyagatovu SO2 + SO3 0 3 9 4 2 361 1389 Bwisha SO2 + SO3 3 4 9 7 1 538 2210 Gashoho 2 Kirundo Kirundo Muramba All SOs 1 5 4 7 13 1036 4219 Yaranda All SOs 0 6 4 0 13 1404 5981 Kinyangurube All SOs 3 5 3 5 1 743 2981 Kavomo All SOs 5 6 2 2 8 920 3815 Kiyanza All SOs 0 6 4 8 0 1273 5027 Mwenya All SOs 3 5 1 1 4 497 1968 Renga 2 MYAP Final Evaluation Report DRAFT Annex E 150 3 4 21 Kayanza 15 10 30 50 43 1226 5557 Muyinga 11 46 110 66 23 7421 29145 Kirundo 14 33 18 23 39 5873 23991 TOTAL 40 89 158 139 105 14520 58693 Sample for Qualitiative Survey MYAP Final Evaluation Report DRAFT Annex F 151 Annex F: New Ideas and Knowledge Brought into the MYAP from Outside The MYAP has benefitted from a wide variety of external training and consultancies bringing new ideas and skills into the program: • Training on commodity management by CRS Rwanda • USAID commodity management workshops • AgroEnterprise Training by an external consultant from CRS Madagascar • Training on Developing Province Level Contingency Plans (Béninois from UNICEF) • Training on Key Hole Gardens by CRS Lesotho • Regional Workshop on Agriculture in Burina Faso • Women’s Rights Training by Kagisye Diomede from the Association de Femme Jurist • Training on Behavioral Change by Food for the Hungry • Training on Fertilization Practices by IFDC Catalyst • Training on Bean Cultivation by the Département des Cultures Légumineuses of l’ISABU (Institut des Sciences Agronomiques du Burundi) • Training of Trainers on SILC by CRS in Senegal • Regional Training of Trainers on SILC in Kigali by CRS and the Association Biraturaba • Staff TDY to Madagascar for exposure to SILC • Training of Trainers on SILC for CRS Burundi Projects: GLCI, MYAP and PM2A • Training by Méthode Niyongendako and Prudence Ndayishimiye for staff and beneficiaries on early warning systems and disaster risk reduction • Training on the Integral Human Development Framework by CRS HQ in Rwanda • Traning on Pro-Pack (Grant Management Software) by CRS Rwanda • FANTA M&E Indicators Workshop using Access Software by Professor Menus Nkurunziza from the University of Burundi, • CRS Technical Adviser Visits • Mary Hennigan (STA Nutrition) • Multiple visits of Tom Remington (STA Agriculture) • Elly Kaganzi (RTA Agriculture/Nutrition) • Steve Walsh (GLCI project – region) • Significant Internal Capacity Building by CRS Staff • SPSS Data Base Management Software, Joseph Ilboudo, M&E Coordinator • Techniques for Mobilizing Focus Groups, Constantin Hien, M&E Coordinator • SharePoint Knowledge Management Software by the IT Department de CRS HQ MYAP Final Evaluation Report DRAFT Annex G 152 Annex G. Recommendations for the Remaining Life of the MYAP SO1 MCHN 1. GENERAL Observations Over the life of MYAP, various nutrition and nutrition-related initiatives (such as PBF) took place (see table below). Overall, MYAP SO1 was conceived/remained well aligned with national strategies, specifically the national protocol for the management of malnutrition. 2009 2010 2011 2012 Establishment of PRONIANUT responsible for Conceptualization, Coordination, Monitoring & Evaluation National Protocole for the Management of Acute Malnutrition Financement Basé sur la Performance (PBF) Workshops to harmonize PD/FARN Draft National Nutrition Policy and Strategy National Food Security & Nutrition Forum National Nutrition Policy and Strategy to be endorsed at highest level Validation of the PD/FARN Manual Draft National Code for the marketing of breast milk substitutes submitted to Cabinet National Guidelines on Infant and Young Child Feeding (IYCF) to be validated in April 2012 Stakeholders discussions to expand CMAM to all provinces In 2011, a national nutrition policy was drafted. Both the ministry of health and agriculture were involved with the support of all nutrition stakeholders in Burundi. This policy should be adopted at the highest level in 2012 (according to information provided by PRONIANUT and confirmed by UNICEF). MYAP Final Evaluation Report DRAFT Annex G 153 Recommendations General  Explore possibilities for MYAP to contribute further to national policies and strategies and to enhance the sustainability of MYAP-supported SO1 activities Specific KEY ACTIONS PROPOSED IN THE NATIONAL NUTRITION POLICY (Draft March 2012) POSSIBLE ACTIONS IN THE REMAINING MONTHS OF MYAP Enhance scaling of the essential components of community-based nutrition, building on the positive deviance/FARN approach Establish more PD/Hearth groups  Priority to be given to the province that has the highest rate of acute malnutrition (based on the 2010 Demographic and Health Survey results) Strengthen the capacity of community health workers and “Maman Lumières” in promoting health and nutrition nutritional screening Conduct a refresher training and provide kits to community health workers and “Maman Lumières”  Favor interprovincial refresher training workshops to enhance sharing of good practices) Gradually integrate rural development programs aimed at improving food security in the FARN intervention areas  Incorporate SILC in any new PD/FARN established PD/FARN 1. Observations According to PRONIANUT, two workshops have been organized (2011/2012) to develop a unified PD/FARN Manual. A validation workshop is planned in April 2012 to finalize/adopt this unified PD/FARN Manual. Recommendations  Explore the possibility for MYAP to contribute to the speedy publication of the new PD/FARN Manual MYAP Final Evaluation Report DRAFT Annex G 154 MOTHER CARE GROUP 1. Observations Both PRONIANUT and UNICEF mentioned to the mission that national authorities supported the MCG, however no reference to this approach is made in either the national protocol or the draft nutrition policy. While quantitative data are available on the immediate and long-term nutritional benefits of PD/FARN (slides 6 and 7 of the stakeholders meeting on Monday 2 April), no such data are available for MCG. Recommendations  Organize a one-day workshop to share MCG approach (objectives and principles) and available findings (such as KAP) with key stakeholders (PRONIANUT, Donors, UN and NGOs) GROWTH MONITORING 1. Observations Constraints Lack of consistency in organization of GM: in one province (Kirundo) GM scheduled on the same day/associated with immunization and in the two others on a separate day Insufficient supply of growth charts BCC (sometimes) conducted without visual aids Good practices Very good growth chart filing system that allows easy follow-up of children (seen in Kirundo Recommendations KEY ACTIONS PROPOSED IN THE NATIONAL NUTRITION POLICY (Draft March 2012) POSSIBLE ACTIONS IN THE REMAINING MONTHS OF MYAP Institutionalize growth monitoring of children from 0 to 24 months in communities and health facilities according to WHO standards  Standardize the scheduling of GM sessions  Expand the use of the Kirundo filing system  Supply growth charts OTHER SUGGESTIONS AND RECOMMENDATIONS FOR THE FUTURE As proposed for SO2, before the project ends, hold a 1 day workshop with the authorities (PRONIANUT/MOH, provincial nutrition focal points, commune chiefs) and relevant nutrition MYAP Final Evaluation Report DRAFT Annex G 155 stakeholders - UNICEF (which provides support to SST and STA) and NGOs working in same province or in other provinces but with the same approach (such as Pathfinder) to share the proven learning, tools and methods available. I also support the following general recommendations proposed for SO2  Exit strategies written, communicated and put in place during the first year  Employ a senior technical staff (T.A.) to supervise the social aspects of all the projects  For food security, environmental protection and nutrition projects, also include an element of dissemination of improved cook-stoves if they are not already used.  For food security projects, ensure that the linkages to environmental protection and nutrition are also included, both in the planning and in the communication with beneficiaries. An element of nutrition education such as the Mother Care Groups or PD/Hearth should also be included. SO2 Liveihoods General: 1. Attempt to make “public interest” activities and service groups financially sustainable (even if it’s on an incentive basis) using cost recovery techniques (all committees and ATs, both for themselves and for their tools). Cost recovery meaning that the running costs (tools, transport and inconvenience – but not time) are compensated. For example committee members (marais, LAE, Goats, SILC ATs) who spend time on “activités communautaires” need to feel rewarded and motivated, so that they are not out of pocket for their time and energy. A good example are the goat committees who get a “free” goat every 10 distributed (a 10% tax effectively). One suggestion for the SILC ATs from now on: they could get paid a small amount from each group for the first 2 share outs from the common pot (small %age or fixed fee) with enough to cover the cost of the hardware so as to enable them to continue with new groups on other hills with no up-front cost to the new groups. Although some have been able to negotiate a deal, it’s better if it’s the same deal in all cases and it is known by all. Marais committees also need to get a built-in systematic reward, perhaps at the time of the user fee collection (%age or fixed fee? – to be jointly agreed). Perhaps also include an element of competition for best practice/best results among the different committees with an annual ceremony and award? Encourage cross visits to best practice sites/groups. Not sure how to compensate the LAE committees, but ideas are: a) “rent” out the tools needed at a small but well known cost (also to help pay for new tools when they break/wear out), b) get a small amount of fodder grass, c) link to a similar payment as the marais committees from the marais user fees based on km of berms constructed/maintained under their supervision? 2. Task the M&E team to perform short studies to better assess a few concerns from the evaluation and for the final report indicators: a. transfer/distribution of land ownership in the marais and in the hills? Who gained and who lost? b. A KAP study of the key TT messages (can also use the 24 TT covered in the PAPSAD survey): MYAP Final Evaluation Report DRAFT Annex G 156 1.Erosion & water runoff control through contour berms 2.Reforestation 3.Improved water management in valleys - Irrigation gates in valleys and supporting drainage canals 4.Improved knowledge and use of seed of improved varieties, seed spacing and field preparation 5.Crop disease recognition and mgmt 6.Animal disease recognition and treatment 7.Composting 8.Seed and grain storage 3. See sectors/activities more as holistic and integrated systems, both on a financial and operational basis, for both hardware and software, for information management and for sustainability, for quantity and quality, for both short-term and long-term effects and impacts. Rather than just achieving numbers written in haste in a project proposal. The ProPack mentions establishing a Detailed Implementation Plan (DIP) once the contract has been approved. An example is the Watershed Management Plan below. 4. Define methods and tools so that the knowledge and products available in MYAP areas can be more easily shared to neighboring collines and at commune level for easy replicability (what are the barriers?). This is done in three complimentary ways: partly self disseminating through evidence based market mechanism (the carrot – I’m willing to pay to receive a direct benefit) and partly through “public policies” such as education, mobilisation, peer recognition (travaux communautaires, special days/events, etc.) and fines (the stick) such as the fines used to decrease the wandering of animals. Environmental Conservation leans more on the education, more education and finally on reprimands and penalties (LAE). Marais management uses both approaches and SILC can largely be self propagating with minimal external support once the replication system is in place. Specific: Watershed (marais and slopes, production and environmental conservation) 1. Make 20 year watershed management plans (operations and economics) with annualised costs with the DPAE, local authorities and committees using external expertise with a simple system of mapping and quality management that they can keep using themselves (i.e. not digital). Consider regular maintenance, repairs and replacement. Include LAE with the construction and maintenance of contour berms to full coverage and the use of the best suited grasses, shrubs and trees (edible and inedible), with a system of incentives, support and penalties. Include agro-forestry (production/supply and environmental/use). If time allows encourage the DPAE/MinAgri to extract from this system a set of general guidelines and good practices for other watershed projects, perhaps a manual? Are there good examples from Rwanda or Burundi to start with? What happened in Busoni? 2. Mobilise users to pay a reasonable amount annually, i.e. at harvest times and an amount equal to a multiple of the day labor rate (built-in indexing) and also perhaps related to plot size for economic justice and better cooperation. MYAP Final Evaluation Report DRAFT Annex G 157 3. Encourage the DPAE/others to perform basic measurements in the watersheds for rainfall and water flows (also to assess if the LAE and water management are effective or not, and to provide data for future interventions and modeling). 4. As part of point 1 above: a) review all the current berms (FFW and not FFW) for a quality assessment and plan remedial work, b) assess the most environmentally at-risk areas (steep slopes, already showing signs of erosion, etc.) and for plan remedial work (minor infrastructure and tree/grass planting). What is the current coverage in terms of quantity and quality and what are the targets (coverage, quality, time scale)? Some ideas: discourage clear cutting and encourage selective harvesting, ensure that no slope is left without vegetation, etc.. Livestock 1. Introduction of ACSAs in the other 2 provinces (already planned). What about the “animal pharmacy communautaire”? 2. Find sources of new/better berm grasses (edible and inedible). Ask ISABU and ISAR? TT 1. (re)train on the use, maintenance and management of shared grain stores, as more are getting used. 2. (re)train model farmers, associations and whoever else relevant about good and safe practices for handling and using fertilisers and pesticides with clear IEC material in Kirundi and with graphics/visuals. 3. Elect Model farmers each year by group consensus (peer recognition) and ensure there is a sufficient number for each sous colline. 4. Complete the linkages between TT activities and SILC. SILC 1. Are there ways to help the people who would like to participate but cannot and use alternate strategies to enable them? Stakeholders 1. Assist them to disseminate the MYAP MonAgris’ new knowledge and expertise to at least their commune colleagues (perhaps have a general meeting presenting the results, methods and tools from the MYAP activities). Make a package of resources that can be used by others. 2. Before the project end, hold a 1 day workshop with the authorities (DPAE and commune chiefs), the other NGOs and projects in the same communes/province to share the proven learning, tools and methods available. MYAP Final Evaluation Report DRAFT Annex G 158 3. In Muyinga, work closely with WV for a smooth transition as they wish to include the MYAP collines in their ADP. Other Suggestions and Recommendations for the future 1. CRS wide: creation and use of standard tools for M&E and data collection/management. There is a unified financial management system, there should be something similar for M&E, especially after so many years of MYAPs. For example to disaggregate all beneficiaries by sex and age groups (<18, adults, and seniors). Establish if the indicator for “beneficiary” is at the household or individual level. Whenever possible, use National ID numbers to identify beneficiaries in databases. 2. SILC: perhaps include an element of micro-project matching grants to the best performing groups, to encourage the next steps, perhaps with linkages to agro-entreprises/associations? 3. Exit strategies written, communicated and put in place during the first year to embed local technical and social ownership and sustainability together and early on. Find out who has the most interest to keep things working and devolve as much responsibility (with concurrent capacity building) as possible. 4. Employ a senior technical staff (T.A.) to supervise the social aspects of all the projects: a) beneficiary motivation (with minimal handouts, gifts), b) social and economic sustainability (now and later), c) all matters of gender, accountability and IEC (trainings, didactic materials,…), d) working with government partners, e) institutional learning (internal and external), f) communication (internal and external), g) etc., etc.. This post is as important as any of the technical support T.As.. 5. For food security, environmental protection and nutrition projects, also include an element of dissemination of improved cook-stoves if they are not already used. 6. For food security projects, ensure that the linkages to environmental protection and nutrition are also included, both in the planning and in the communication with beneficiaries. An element of nutrition education such as the Mother Care Groups or PD/Hearth should also be included. 7. Never consider emergency or post-emergency activities or projects as an end in themselves: always consider that there is a continuum into rehabilitation and development beyond the life of the project (and sometimes even within the life of the project). Sustainability is always a consideration even if not achievable in the short term but no barriers should be erected for its long term either. 8. Follow the CRS PROPACK and implement the DIP and M&E plan with documentation. 9. For geographically based activities (such as watershed rehabilitation and management), obtain and use the best paper and digital maps possible. MYAP Final Evaluation Report DRAFT Annex G 159 SO1 Water Points & SO3 PACS and Gender SO1 Water Points In Muyinga, mediate discussions between the Water Committee and users of the water points to eliminate the perception that the Water Committee is responsible for keeping the water point clean. Combine this with a cross-visit of the Water Committees/ local leaders to Kayanza water point committees to see how they function there. In the few water points that are already showing leaks, facilitate stronger relationships between the Water Committee and the Commune Water Coordinator to be able to mobilize resources and technical support for repairing leaks before they get larger. The Commune will likely want the program to do the work to repair the leaks. This is okay; the MYAP can provide the resources. But, the repairs should be done under the supervision of the Commune using processes that would be used after the program ends whenever a water point repair needs to be undertaken. Perhaps you can find out from the GIZ project what this process is supposed to look like. The MYAP has already planned to do one more round of checks to see how the water points are doing. If it can be organized in the time remaining, this could also include water flow and water quality assessments. On the Gisasa water point in Mpete sous-colline in Muyanga (the water point issuing muddy water water after a heavy rain), the repair of this is going to require technical advice from a water engineer to determine how the spring is being contaminated by surface water whenever a heavy rain comes. There is not enough time to do anything more with the wash basins. It would be interesting, however, to hold some more intensive consultations with users of the water point to get better information on why people are not using the basins and alternative designs for the washing facilities, as a lesson for next time. This is not a high priority, however, given the time available. Similarly, on the PHAST training, there's probably not enough time to replicate or complete the training with those committees in Muyinga that didn't get trained, but if the time and resources can be found, this would be another lower priority activity that could be done. The other hygiene training that is being done through lead mothers is actually reaching people with the messages. SO3 PACs Hold discussions with Commune Administration to discuss how they can use the PAC committees for facilitating Commune community development planning. The program could even facilitate an event to turn-over the PACs to the Commune. The discussions with the Commune should include ideas on how to merge the PACS with the colline-level CDC. It's not clear who is responsible for calling for the next elections of the CDCs, but if it is the Commune that is responsible for this, then the discussions could even include plans for holding these elections. MYAP Final Evaluation Report DRAFT Annex G 160 To continue building capacities of PAC committees to mobilize and manage resources, make available MYAP resources as small grants for community development activities based on proposals received from PAC committees. SO3 Gender Accelerate the cultivation of the relationship with CDF in all provinces and at the central level by holding meetings with them and including them in activities planned through the end of the program. Discussions and field visits should ensure that the CDF senior leadership is fully exposed to the intervention strategy to cultivate their interest in replicating it. Hold a formal turnover event with the Commune Administration to formally turn over Acteur Relais to them. This event should also include some form of recognition and an expression of appreciation for the Acteurs Relais (their own certificate with their picture, perhaps?). Document the intervention strategy used in this component of the program and the experience more fully, hiring outside assistance if necessary to complete this before the program ends,; and hold an open event for donors, NGOS and government to present the key points from the documentation. MYAP Final Evaluation Report DRAFT Annex H 161 Annex H. Commodity Summary Tables Table H-1. Ration Composition and Scale Program Beneficiaries MYAP Commodities & Quantities Recommended by the New GoB Protocol Comments SFP (Quantities are per person and per day for 14 days) Children CSB 120 Oil 15 CSB 120 Oil 15 Sugar 25 Sugar is a new input to the ration Pregnant and Nursing women CSB 100 Oil 15 Peas 50 MML 100 Salt 3 (non￾Title II) CSB 100 Oil 15 Peas 50 MML 100 Salt 3 No change OTP (Quantities are per person and per day for 7days) Children RUTF (Plumpy nut) complemented with Bulgur 330 Lentils 100 CSB 50 Oil 25 RTUF (Plumpy nut) No complementary ration to RUTF recommended SST daily throughout the stay at the hospital (for 7 days) Malnourished patients Therapeutic milk (F75 and F100) Therapeutic milk (F75 and F100) No change Caregivers Bulgur 330 Lentils 100 CSB 50 Oil 25 Salt 5 (non-Title II) No rations recommended for care givers PLWHA PLWHA Bulgur 1400 g/day (i.e. 42 kg/month) Lentils 500gr/day, (i.e. 15 kg/month) CSB 600g/day (i.e. 18 kg/month) Oil 120 g/day (i.e. 3.6754 kg/month) Sugar 67 g/day (i.e. 2 kg/month) Bulgur 42 kg/month Lentils 15 kg/month CSB 18 kg/month Oil 3.6754 kg/month Sugar 2 kg/month No change MYAP Final Evaluation Report DRAFT Annex H 162 Table H–2. LOA Commodity Summary (MT) Distribution Monetization Total FY ’08 (Actual 1,317 4310 5,627 FY ‘09 (Actual) 2,240 7,200 9,440 FY ‘10 (Actual) 1,090 8,000 9,090 FY ‘11 (Actual) 1,340 6,750 8,090 FY12 0 0 0 Total Projected Disbursements through 31 July 2012 5,987 26,260 32,247 Original Proposed LOA Quantities 4,939 15,100 20,039 Revised Projected LOA Quantities 6,279 26,260 32,539 Percentage of LOA Quantities Projected to be Achieved Through July 31, 2012 95.3% 100% 99.1% Table H–3: Monetization Cost Recovery FISCAL YEAR COMMODIT Y QUANTITY ( MT) C&F ($/T) SALE PRICE ($/MT) COST RECOVERY FY08 HRW Wheat 4,310 465.00 425.00 91.40% FY09 HRW Wheat 7,200 399.00 270.00 67. 67% FY10 HRW Wheat 8,000 315.00 265.00 84.13% FY11 HRW Wheat 6,750 532.00 390.00 73.31% LOA TOTAL 26,260 418.43* 324.76* 78.90% *Weighted averages by volume Table H–4. Commodity Loss Summary (MT) Fiscal Year Amount Purchased Ocean Losses Amount Received in Country Transport Losses DSM-BUJ In￾Country Losses Percent Lost Monetized Commodities FY ‘08 4,310 24.64 4,285.35 0 0 .57% FY ‘09 7,200 -46.00 7,246.00 0 0 0% FY ‘10 8,000 39.10 7,960.90 0 0 .48% FY ‘11 6,750 -5.7 6,755.70 0 0 0% Total through FY ‘12 26,260 12.05 26,247.95 0 0 .046% Distributed Commodities FY ‘08 1,208.471 4.444 1,192.787 11.24 1.91 1.46% FY ‘09 2,244.407 4.600 2,234.378 5.429 4.84 0.66% FY ‘10 1,087.125 .400 1,027.929 1.258 61.17 5.78% FY ‘11 1,337.865 0 1,329.011 2.261 7.10 1.78% Total through FY ‘12 5,877.868 9.444 5,784.105 20.188 75.02 1. 62% MYAP Final Evaluation Report DRAFT Annex H 163 Annex H–5: Food Aid Beneficiaries by Group and Gender Implementation Year SO1 Hospital and Health Center-based Distributions (SFP, OTP, SST) FFW Recipients PLWHA Beneficiaries TOTAL M F Total M F Total M F Total Year 1 53,288 70,615 123,903 - - - 599 369 968 124,871 Year 2 70655 271,194 341,849 1,662 5,383 7,045 1,191 4,119 5,310 354,204 Year 3 38,795 204,284 243,079 3,699 1,565 5,264 1,550 4,668 6,218 254,561 Year 4 14,869 123,554 138,423 1,405 1,098 2,503 958 2,543 3,501 144,427 TOTAL 177,607 669,647 847,254 10,722 8,046 14,812 4298 11,699 15,997 878,063