FINALEVALUATION REPORT FOR THE TUBARAMURE PM2A PROGRAM Cooperative Agreement No.AID-FFP-A-09-00004-00 Final Evaluation Report for the Tubaramure PM2A Program Cooperative Agreement No. AID-FFP-A-09-00004-00 Della E. McMillan, Independent Consultant Sidibe Sidikiba, Independent Consultant September 8, 2014 Revised October 22, 2014 Tubaramure Final Evaluation. Table of Contents. October 22, 2014. Final. i Table of Contents Acknowledgements ..................................................................................................................... vii List of Acronyms........................................................................................................................ viii Executive Summary .................................................................................................................... xii Chapter 1. Context and Methodology ......................................................................................... 1 1.0. Overview of the Tubaramure PM2A Program ................................................................. 1 2.0. Early Evolution of the Program ........................................................................................ 4 3.0. Expected Program Outputs, Outcomes, and Impacts ....................................................... 6 4.0. Evaluation Objectives and Methodology .......................................................................... 7 5.0. Organization of the Report ............................................................................................. 13 Chapter 2. Intermediate Result 1: Women and Children Under 5 Access Quality Nutrition and Health Services............................................................................... 15 1.0. Global Strategy ............................................................................................................... 15 2.0. Activities ........................................................................................................................ 16 3.0. Evidence of Results ........................................................................................................ 24 4.0. Factors Which Contributed to and Detracted From Program Relevance, Effectiveness, Efficiency, and Acceptability of Processes, Outputs, and Implementation ............. 39 5.0. Lessons Learned and Best Practice Checklist ................................................................. 45 Chapter 3. Intermediate Result 2: Households Practice Appropriate Health and Nutrition Behaviors ............................................................................................... 48 1.0. Global Strategy ............................................................................................................... 48 2.0. Activities ......................................................................................................................... 49 3.0. Evidence of Results ...................................................................................................... 55 4.0. Factors Which Contributed to and Detracted From Program Relevance, Effectiveness, Efficiency, and Acceptability of Processes, Outputs, and Implementation ............. 65 5.0. Lessons Learned and Best Practice Checklist ................................................................. 75 Chapter 4. Intermediate Result 3: Eligible Women and Children Have Increased Intake of Diverse Food ....................................................................................................... 77 1.0. Global Strategy ............................................................................................................... 77 2.0. Activities ......................................................................................................................... 79 3.0. Evidence of Results ....................................................................................................... 91 4.0. Factors That Contributed to and Detracted from Program Relevance, Effectiveness, Efficiency, and Acceptability of Processes, Outputs, and Implementation ............. 94 5.0. Lessons Learned and Best Practice Checklist ............................................................... 100 Chapter 5. Principal Program Results and Their Likelihood of Being Sustained.............. 102 1.0. Evidence of Results Based on the Program’s Principle Outcome Indicators ............... 103 2.0. Evidence of Results Based on the Program’s Impact Indicators .................................. 109 3.0. Activities Producing Good Outcomes and Their Prospects for Being Sustained Once the Program Ends ................................................................................................... 115 Tubaramure Final Evaluation. Table of Contents. October 22, 2014. Final. ii Annex I.A. List of Specific Questions and Key Questions......................................................124 Annex I.B. SOW for the Team Leader and Evaluation Specialist .......................................128 Annex I.C. SOW for the Primary Health Care and IMCI Specialist ...................................137 Annex II. Tubaramure Program IPTT ...................................................................................144 Annex III. Focus Group Guides ..............................................................................................147 Annex IV. Qualitative Data from Focus Groups ....................................................................176 Annex V. Testimonials...............................................................................................................252 Annex VI. Chapter Annexes .....................................................................................................281 Tubaramure Final Evaluation. Table of Tables. October 22, 2014. Final. iii Table of Tables Table A. Tubaramure Program Lessons Learned and Best Practice Checklist ..................... xix Table 1.1. Framework for the Tubaramure Program .................................................................. 1 Table 1.2. Quantitative Households Survey Sample, Baseline, and Endline............................ 10 Table 1.3. Retroactive Qualitative Classification of Tubaramure Villages Used to Determine the Sample Frame for the Final Qualitative Survey Focus Group Discussions .............................................................................................................. 11 Table 1.4. Number of People Interviewed in the Tubaramure Final Evaluation Focus Groups, July 2014 .................................................................................................... 12 Table 1.5. Number of Key Informant Interviews in the Tubaramure Final Evaluation, July 2014 .................................................................................................................. 13 Table 1.6. Approximate Number of People Who Attended Workshops Associated With the Final Evaluation of the Tubaramure Program .................................................... 13 Table 2.1. Major Outputs Designed to Achieve the Tubaramure Program’s IR1 .................... 15 Table 2.2. Technical and Logistical Support Provided by the Tubaramure Program for the Revision of the Major Protocols that Supported the MoH’s Nutrition Strategy, 2009-2014 ................................................................................................................ 17 Table 2.3. Number of People in Different Target Groups Trained by the Tubaramure Program in Different Protocols, 2009-2014............................................................. 20 Table 2.4. Qualitative Assessment of the Early Impact of PM2A Donations to Health Centers in Cankuzo and Ruyigi Provinces............................................................... 21 Table 2.5. Tubaramure PM2A Program Indicators Used to Track MoH Capacity and Community Demand for Pre and Postnatal Consultations, 2010-2014 ................... 25 Table 2.6. Evolution of Key Indicators Tracking MoH Capacity for IMCI Support at the Clinic and Community Level, 2009-2014 ............................................................... 29 Table 2.7. Tubaramure PM2A Program Indicators Used To Track Clinical and Community￾Based Capacity for Growth Monitoring, 2009-2014 ............................................... 31 Table 2.8. Summary Data Collected on the 12 Pilot Growth Monitoring Sites Created Under the Tubaramure PM2A Program, 2012-2014 ............................................... 33 Table 2.9. The Original Tubaramure MYAP Plan For Treating Children With Moderate and Mild Malnutrition Whose Non-Realization Was Identified as a Critical Constraint to the Achievement of Output 1.4 in the Absence of Community￾Based FARNs........................................................................................................... 38 Table 2.10. Tubaramure PM2A Program Indicators Used to Track Identify, Refer, and Monitor Follow Up on Children with Severe and Acute Malnutrition, 2010-2014 ................................................................................................................ 39 Table 2.11. Evolution in the Number of MoH Staff and Health Centers and Tubaramure Staff Working on IR1 Activities in the Ruyigi and Cankuzo Provinces, 2009-2014 ................................................................................................................ 40 Table 2.12. Priority Issues for Follow Up to Capitalize on the Results of the Tubaramure Program’s IR1 Activities ......................................................................................... 43 Table 2.13. Number of Current MoH Staff Trained Under Tubaramure, 2014 .......................... 44 Table 2.14. Tubaramure Program IR1 Lessons Learned and Best Practice Checklist ............... 45 Tubaramure Final Evaluation. Table of Tables. October 22, 2014. Final. iv Table 3.1. Major Outputs Designed to Achieve the Tubaramure Program’s IR2 .................... 48 Table 3.2. Evolution of the Program’s Complementary Support Given to the Tubaramure￾Facilitated Groupements Under IR2 and IR3 .......................................................... 53 Table 3.3. Percentage of Households Adopting the Main ENA and EHA Practices Promoted by the Tubaramure Program in Ruyigi and Cankuzo Provinces ............. 56 Table 3.4. Number of Tubaramure LMs Who Have Become Community Health Workers, Commune Development Committee Representatives, or Members of the Women’s Forum ...................................................................................................... 64 Table 3.5. IR2 Activities that Affected the Community-Level Outcomes of IR Outputs in the Tubaramure Program ......................................................................................... 66 Table 3.6. Priority Issues for Follow Up to Capitalize on the Results of the Tubaramure Program’s IR2 Activities ......................................................................................... 74 Table 3.7. Tubaramure Program IR2 Lessons Learned and Best Practice Checklist ............... 75 Table 4.1. Major Outputs that were Designed to Achieve the Tubaramure Program’s IR3 ..... 77 Table 4.2. Link Between Mid-Term Evaluation Recommendations and Key Groups of Follow-Up Activities ............................................................................................... 85 Table 4.3. Eligible Women and Children Have Increased Intake of Diversified Foods........... 92 Table 4.4. Percentage of Households in the Final Quantitative Survey Eating Different Numbers of Food Groups......................................................................................... 92 Table 4.5. Percentage of Households Classified as Having Different Levels of Household Food Security in Cankuzo and Ruyigi Provinces in the 2014 SMART Study ........ 94 Table 4.6. Priority Issues for Follow Up to Capitalize on the Results of the Tubaramure Program’s IR3 Activities ......................................................................................... 98 Table 4.7. Tubaramure Program IR3 Lessons Learned and Best Practice Checklist ............. 100 Table 5.1. Women and Children Under 5 Access Quality Nutrition and Health Services ..... 104 Table 5.2. Households Practice Appropriate Health and Nutrition Behaviors ....................... 106 Table 5.3. Percentage of Households Practicing the Main ENA and EHA Practices Promoted by the Tubaramure Program in Ruyigi and Cankuzo Provinces ........... 107 Table 5.4. Eligible Women and Children have Increased Intake of Diversified Foods ......... 108 Table 5.5. Nutritional Status of Children Under 5 Years of Age in the Provinces of Cankuzo and Ruyigi at the Tubaramure Program Baseline (2010) and Endline (2014) ..................................................................................................................... 109 Table 5.6. Percentage of Households Classified as Having Different Levels of Household Food Security in Cankuzo and Ruyigi Provinces in the 2014 SMART Study ...... 111 Table 5.7. Major Outcomes Being Generated in the Tubaramure Program and Options for Sustaining the Impact Once the Program Ends ...................................................... 115 Tubaramure Final Evaluation. Table of Figures. October 22, 2014. Final. v Table of Figures Figure 1.1. Location of Ruyigi and Cankuzo Provinces in Burundi ............................................ 3 Figure 1.2. Evolution of Tubaramure PM2A Beneficiaries Not Yet Graduated from the Program, 2009-2014 .................................................................................................. 5 Figure 2.1. Average Scores for Participants on Tubaramure-Sponsored Pre and Postnatal Training Sessions ..................................................................................................... 19 Figure 2.2. Percentage of Women Having Assisted Births in Health Centers ........................... 28 Figure 2.3. Summary Data Collected on the 12 Pilot GM Sites Created Under the Tubaramure PM2A Program, 2012-2014 ................................................................ 32 Figure 3.1. Year the Current Base of Tubaramure Groupements Were Created in Ruyigi and Cankuzo Provinces................................................................................................... 54 Figure 3.2. Output Indicator 2.4. Percentage of Households Observed Carrying Out Four or More ENA Actions at Time of Household Visit (Observed or Reported) .............. 57 Figure 3.3. Output Indicator 2.5. Percentage of Households Observed Carrying Out Four or More EHA Actions at Time of Household Visit (Observed or Reported) .............. 59 Figure 3.4. Indicator 2.6. Percentage of Mothers With Children Under 2 Who Can State at Least Four of the Six Danger Signs for Childhood Illness and at Least Two of the Four Danger Signs for Pregnant Women ........................................................... 61 Figure 4.1. Evolution of Tubaramure Beneficiaries Not Yet Graduated from the Program, 2009-2014 ................................................................................................................ 78 Figure 4.2. Evolution of Tubaramure-Facilitated SILCs in Ruyigi and Cankuzo Provinces, 2010-Present ............................................................................................................ 87 Figure 4.3. Evolution of Tubaramure-Facilitated Keyhole Gardens in Ruyigi and Cankuzo Provinces, 2010-Present ........................................................................................... 89 Figure 5.1. Percentage of Children Classified as Having Chronic Malnutrition (Stunting, Height/Age <= -2 SD) ............................................................................................ 110 Figure 5.2. Percentage of Children Classified as Having Acute Malnutrition (Wasting, Weight/Height <= -2 SD) ...................................................................................... 112 Figure 5.3. Percentage of Children Classified as Underweight (Weight-for-Age <= -2 SD) .. 113 Figure 5.4. Percentage of Newborns Underweight (< 2500g) ................................................. 114 Tubaramure Final Evaluation. Table of Text Boxes. October 22, 2014. Final. vi Table of Text Boxes Text Box 1.1. Evidence for Tubaramure Population-Based Impacts on Nutrition ...................... 7 Text Box 2.1. Lesson Learned: Critical Importance of Building MoH Capacity for Nutritional Data Collection and Analysis—a Win-Win for Program Planning .................... 22 Text Box 2.2. Lesson Learned: Case Studies of How the Tubaramure Program’s Support to the Quarterly Coordination Meetings Helped Increase Program Efficiency, Impact, and Sustainability ................................................................................... 24 Text Box 2.3. Principal Recommendations of the 2012 Study on Obstacles to the Adoption of Good Practices on Health, Nutrition, and Hygiene for Increasing the Rate of Pre and Postnatal Consultations ...................................................................... 26 Text Box 2.4. MoH Staff Members’ Perceptions of the Impact of the Tubaramure Program on Their Capacity and Service Delivery ............................................................. 30 Text Box 2.5. Testimonials from Mothers About the Impact of the Pilot Community-Based Growth Monitoring Program from the Final Evaluation Pilot Study .................. 34 Text Box 2.6. Sample Testimonials from the MoH About How the Tubaramure Increased Their Capacity to Deliver Quality Clinic-Based Services ................................... 41 Text Box 3.1. Testimonials from the Beneficiary Mothers about the Impact of the Program’s IR2 BCC Activities ............................................................................................. 58 Text Box 3.2. Lesson Learned: Critical Importance of Involving Local Authorities in BCC Campaigns ........................................................................................................... 60 Text Box 3.3. Gender Strategy from the Original Tubaramure MYAP Proposal ...................... 67 Text Box 4.1. The Concept of the Savings and Internal Learning Communities ...................... 82 Text Box 4.2. Mid-Term Evaluation Assessments of the Tubaramure Pipeline and Implemented Commodity System (IR 3, Output 3.1), June 2012 ....................... 83 Text Box 4.3. Testimonials by Tubaramure SILC Members on Their Food Security, Income-Generating Activities, and Quality of Life ............................................. 88 Text Box 4.4. Tubaramure Beneficiaries’ Testimonials on the Impact of Keyhole Garden Plots on Their Household’s Dietary Diversity .................................................... 90 Text Box 4.5. Tubaramure Beneficiary Testimonial on the Impact of the Culinary Demonstrations on Their Household’s Dietary Diversity ................................... 91 Text Box 4.6. PM2A Beneficiary Testimonial on the Impact Joining a Groupement Had on Her Living Standard ............................................................................................ 94 Tubaramure Final Evaluation. Acknowledgements. October 22, 2014. Final. vii Acknowledgements The members of the evaluation team would like to gratefully acknowledge the support provided by the staff of the Tubaramure Preventing Malnutrition in Children Under Two Approach (PM2A) five-year development food assistance program from Catholic Relief Services (CRS) Burundi, International Medical Corps (IMC), Food for the Hungry (FH), and Caritas. The team would particularly like to thank the two provincial coordinators—Guerrier Iryabavyeyi and Edmond Twagirayezu—who helped organize the logistics of getting us to the field. Dr. Evelyn Ngomirakiza accompanied us as a representative of the Ministry of Health (MoH), where she played a critical role in setting up and conducting our interviews with the Governors, the Governors’ staff, various commune officials, and the MoH, as well reviewing and commenting on various drafts of the different chapters. A simple Indicator Performance Tracking Table (IPTT) can never tell all of the tales of a program of this size and complexity. We are deeply grateful to several people who helped us unearth various nuggets of ‘gold’ we would not have discovered otherwise:  Dr. Herve Kaptchouang and Dr. Basile Mukenge Ndumbi from IMC and Dr. Evelyn Ngomirakiza from the MoH helped us document the important role the Tubaramure Program played in revising some of the MoH’s major protocols and training modules for nutrition;  Cyprien Tuyizere, Gerard Biregeya, and Sarah Borgerfrom FH helped us better understand the Tubaramure Program’s extensive investment in formative research and the broader literature to support its Care Group Model; and  Ezéchiel Kabwebwe, Edmond Twagirayezu, and Regine Pacis Nihoreho from CRS helped us document the evolution of the Tubaramure groupements and some of the unanticipated impacts of the Leader Mothers’ training on their wider activities in the local communities. A special thanks is due to Scholastique Ntakirutmana, who handled our hotel reservations, logistics, and contracts with the same seamless efficiency that has made her a CRS/Burundi legend. The team is grateful to Dr. Raphaël Bajay Tchumah, the original Tubaramure Chief of Party; Ruben Johnson, who was Chief of Party during the evaluation; and CRS Country Representative in Burundi and Rwanda Darren Posey. Their frank openness in discussing this highly innovative and experimental program set the right tone for the evaluation. None of this would have been possible without the excellent backup we received from the Tubaramure Monitoring and Evaluation (M&E) officers: Thaddee Niyonzima and Joseph Ilboudo. The team is very grateful to the two translators—Consolata Nahimana and Jean Claude Ndikumazambo—who worked on the interviews that constitute Annexes IV and V. We would also like to recognize the very able team of research assistants that conducted the focus groups: Laurence Sibomana, Kana Ariane, Nijimbere Candide, Mahengamo Pascal, Ndaijimana Andre, and Amerusenge Emery. Lastly, we wish to thank our editor Lynn Hurtak. Della E. McMillan Sidibe Sidikiba Tubaramure Final Evaluation. Acronyms. October 22, 2014. Final. viii List of Acronyms ADS Automative Directive System ASC Agent de Santé Communautaire (Community Health Worker or CHW in English) BCC Behavior change communication BDS Bureau de District Sanitaire (District Health Department) BIF Burundi Francs BPS Bureau Provincial de Santé (Provincial Health Department) CDC Commune Development Committee CDS Centre de santé (health center) CG Care Group CHW Community Health Worker (Agent de Santé Communautaire or ASC in French) C-IMCI Community-Based Integrated Management of Childhood Illness CMAM Community-Based Management of Acute Malnutrition (Prise en Charge Communautaire de la Malnutrition Aigüe or PCMA in French) CPN Prenatal consultation (consultation prénatale) CPoN Postnatal consultation (consultation postnatale) Colline Literally "hill;" an administrative unit Cordaid Catholic Organization for Relief and Development Aid CoSA Comité de Santé (Health Committee) CRS Catholic Relief Services CSB Corn-soy blend DHA District Health Authority DHS Demographic and Health Survey DIP Detailed Implementation Plan DPAE Direction Provinciale de l'Agriculture et de l'Elevage (Provincial Directorate for Agriculture and Livestock) EBF Exclusive breastfeeding EDSB Enquête Démographique et de Santé de Burundi (National Demographic and Health Survey of Burundi) EHA Essential Hygiene Actions ENA Essential Nutrition Actions EPI Expanded Program on Immunization FANTA Food and Nutrition Technical Assistance FAO Food and Agriculture Organization FARN Foyer d'Apprentissage et de Réhabilitation Nutritionnelle (or PD [Positive Deviance]/Hearth) FFP Food for Peace FH Food for the Hungry FLM Federation Luthérienne Mondiale (Lutheran World Federation) FY Fiscal Year Tubaramure Final Evaluation. Acronyms. October 22, 2014. Final. ix GoB Government of Burundi Groupements Economic associations GM Growth monitoring (Suivi Promotion de la Croissance or SPC in French) HDDS Household Dietary Diversity Score HF Health facility HH Household HIV/AIDS Human immunodeficiency virus infection /acquired immunodeficiency syndrome HPT Health Promoter Technician HW Hand washing IBF Immediate breastfeeding IEC Information, Education, Communication IFPRI International Food Policy Research Institute IGA Income-generating activity IMC International Medical Corps IMCI Integrated Management of Childhood Illness (Prise en Charge Intégrée des Maladies de l’Enfance or PCIME in French) IPTT Indicator Performance Tracking Table IR Intermediate result IRA Infection respiratoire aiguë IRC International Red Cross ISTEEBU Institut de Statistiques et d'Etudes Economiques du Burundi (Burundian Institute for Statistics and Economic Studies) ITN Insecticide-treated net ITSH International transport, storage, and handling IYCF Infant and Young Child Feeding Kg Kilogram Km Kilometer LOA Life of Activity LF Leader Father LM Leader Mother M&E Monitoring and evaluation MAE Ministère d'Agriculture et d'Elevage (Ministry of Agriculture and Livestock) MD Doctor of Medicine MEAL Monitoring, evaluation, accountability, and learning MoH Ministry of Health (Ministère de Santé Publique et la Lutte Contre le Sida or MSPLS in French) MS Master of Science MSPLS Ministère de Santé Publique et la Lutte Contre le Sida (Ministry of Health or MoH in English) MT Metric Ton Tubaramure Final Evaluation. Acronyms. October 22, 2014. Final. x MTE Mid-Term Evaluation MUAC Mid-upper arm circumference MYAP Multi-Year Assistance Program N/A Not available or not applicable NGO Non-governmental organization ORS Oral Rehydration Salts PAIOSA Programme d’Appui Institutionnel et Operational au Secteur Agricole PBF Performance-based financing PCMA Prise en Charge Communautaire de la Malnutrition Aigüe (Community-Based Management of Acute Malnutrition or CMAM in English) PCDC Plans Communaux de Développement Communautaire (Commune Community Development Plans) PCIME Prise en Charge Intégrée des Maladies de l’Enfance (Integrated Management of Childhood Illness or IMCI in English) PD/Hearth Positive Deviance/Hearth Model (or FARN in French) PHA Provincial Health Authority PM2A Preventing Malnutrition in Children Under 2 Approach PNDS Plan National de Développement Sanitaire (National Development Plan for Health) PRONIANUT Programme National Intégré d'Alimentation et de Nutrition (National Integrated Program of Food and Nutrition in English) PSN Promotteurs de Santé et Nutrition (Tubaramure Health Promoters of THP in English) PSP Private Service Provider SAM Severe acute malnutrition SD Standard deviation SILC Savings and Internal Lending Community SMART Standardized Monitoring and Assessment Relief and Transitions SOW Scope of work SPC Suivi Promotion de la Croissance (growth monitoring or GM in English) TBD To be determined THP Tubaramure Health Promoter (Promotteurs de Santé et Nutrition or PSN in French) TOPS Technical and Operational Performance Support TOR Terms of reference TPS Public Health Technician Tubaramure Kirundi for, “Let’s Help Them Grow” UNICEF United Nations Children’s Fund USAID United States Agency for International Development WAZ Weight-for-age Z score Tubaramure Final Evaluation. Acronyms. October 22, 2014. Final. xi WFP World Food Program (Programme Alimentaire Mondial or PAM in French) WHO World Health Organization Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xii Executive Summary 1.0. Program Overview Catholic Relief Services (CRS)/Burundi led a consortium of international and national non￾governmental organizations (NGOs)—International Medical Corps (IMC), Food for the Hungry (FH), and Caritas-Burundi—in the implementation of a five-year (2009-2014) United States Agency for International Development (USAID)-financed development program entitled Tubaramure, “Let’s help them grow.” The program used a Preventing Malnutrition in Children Under 2 Approach (PM2A), which recent research in Haiti had identified as possibly being more effective in reducing malnutrition than historically favored remedial methods.1 USAID’s decision to fund a second multi-year assistance program (MYAP) in Burundi was conditional on that program conducting a comparative assessment of the PM2A model for preventing malnutrition that was being co-executed by the International Food Policy Research Institute (IFPRI) and the Food Aid and Nutrition Technical Assistance (FANTA) Project in two countries, Guatemala and Burundi. This unique context accounts for several features of this program, which are critical for understanding the global context of this evaluation:  First and foremost, it explains why the Tubaramure Program focuses exclusively on improving food utilization—i.e. only one of the four dimensions of the traditional Title II food security model for building household food security. Most traditional MYAPs include four strategic objectives: increasing food availability, increasing food access, increasing food utilization, and improving the local communities’ ability to identify and manage risk; 2  Second, it explains why the program is built around the PM2A approach; and  Third, it explains why 60 of the 269 program villages are linked to a special research program conducted by IFPRI with funding from USAID through the FANTA II Project. IFPRI is conducting study of a series of 60 collines (communities)3 to assess the impact and cost effectiveness of the Tubaramure PM2A model on child nutritional status, as well as to evaluate the differential and absolute impact of varying the duration of receiving food rations in two countries, Burundi and Guatemala. The principal objective of the Tubaramure Program was to prevent malnutrition in children under 2 years of age in 268 collines in two highly food-insecure provinces of Burundi, Cankuzo 1 In 2008, International Food Policy and Research Institute (IFPRI)—in collaboration with World Vision-Haiti, Cornell University, and the Food and Nutrition Technical Assistance (FANTA) Project—provided the first programmatic evidence that the preventative approach of blanket-targeting of a food-assisted maternal-and-child health and nutrition program to all children 6-to-24-months old was more effective in reducing the prevalence of stunting, wasting, and being underweight than the traditional recuperative approach based on solely targeting underweight children (weight-for-age Z-score [WAZ] < -2) under 5 years of age. 2 Although the priorities of the Title II Food for Peace (FFP) Program have shifted, the basic structure has remained focused on the classic three-pronged program with the addition of an early warning and response component since 1995. See van Haeften, R.; Anderson, M.A.; Caudill, H.; and Kilmartin, E. 2013. Second Food Aid and Food Security Assessment (FAFSA-2) Summary. Washington, DC: FHI 360/FANTA. [http://www.usaid.gov/sites/default/files/documents/1866/FAFSA-2Summary_Web.pdf). 3 The term colline (hill) is used to refer to the local communities and settlements in Burundi. Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xiii and Ruyigi. To achieve this goal, the program’s activities were organized around the achievement of three intermediate results (IRs):  IR1: Women and children under 5 access quality nutrition and health services;  IR2: Households practice appropriate health and nutrition behaviors; and  IR3: Eligible women and children have increased intake of nutrient-rich diverse foods. 2.0. Evaluation Methodology This final evaluation of the program was conducted by a team of two development professionals in Burundi from July 6-August 7, 2014, in close collaboration with the nutrition focal point for the Burundi Ministry of Health or MoH (Ministère de Santé Publique et la Lutte Contre le Sida or MSPLS in French). The principal objective of the evaluation was to determine:  The relevance, effectiveness, efficiency, and acceptability of processes and outputs;  The factors affecting the implementation and the degree of adherence to the terms of the agreement;  What and why results (outcomes/impacts) have/have not been achieved; and  The sustainability of results. This evaluation was also expected to describe the outcomes and impact (intended and unintended) of the program’s activities; how contextual or program-related factors contributed to greater or less positive outcomes; and factors that promote/threaten the sustainability of positive impacts after the program ends. The team was also asked to address a number of more specific sub-objectives and key questions for each of the IRs. The evaluators reviewed existing secondary sources of information and available quantitative information from baseline to endline household surveys, and used qualitative survey methods in order to better understand the program’s impact. These qualitative methods included the organization of focus group discussions in a representative sample of 20 local collines, as well as key informant interviews with staff, local partners, and local authorities at the provincial, commune, and local levels. 3.0. Evidence of Results The recently completed (May 2014) Institute de Statistiques et d’Etudes Economiques du Burundi (ISTEEBU) study of a stratified random sample of 1,200 households in 40 collines in both provinces found statistically significant reductions (p<0,05) in the rate of acute malnutrition (from 8.4% in 2010 to 4.8% in 2014) and the percentage of children classified as underweight (from 46.7% in 2010 to 22% in 2014). The same study showed a decrease in the number of children classified as stunted from 52.3% in 2010 to 50.5% in 2014, giving a difference of 1.8 points non-statistically significant (p = 0.236). A disaggregated analysis by province shows the Tubaramure Program had a higher impact on stunting in Cankuzo (38.9%) province than Ruyigi (57.2%). The Tubaramure final household survey showed a remarkable 13-point decrease in the prevalence of stunting—from 52.3% in 2010 to 38.9% in 2014—that was statistically significant (p=0.000). Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xiv This significant reduction in the rate of malnutrition (acute malnutrition and underweight) was confirmed using other information gathered during the final evaluation focus group discussions and key informant interviews. Most of the beneficiary focus group discussions stated that the program had a significant impact on children’s health and well being; and the majority of beneficiary women continue to follow the program’s recommendations for child nutrition and improved hygiene even after graduation. In addition to the Tubaramure Program’s direct impact on malnutrition, it has had a huge impact on local capacity at a variety of levels:  The MoH National Nutrition Policies: The MoH staff at all levels are aware of the recently revised (under the Tubaramure Program) protocols to support malnutrition,4 and how these newly revised protocols and the MoH’s training modules for these protocols can be used to build the capacity of the local health districts and Community Health Workers to both treat and prevent malnutrition;  Commune and Provincial-Level Administration: There is an increased awareness of the critical importance of the nutrition activities at the commune and local level that is reflected in the new Commune Community Development Plans (PCDC), even though there is very little formal-level understanding of the actual commune-level data on the activities that were executed in specific villages or how these affected the principal nutrition and sanitation behaviors; and  Gender: Local authorities at all levels of the administration underscored the critical impact of the program on women’s empowerment, which was reflected in: - The much more prominent discussion of sanitation and health issues in the most recent generation of the PCDC; and - The small but important percentage of the 4,930 Leader Mothers (LMs) trained by the program who were elected to district-level community development plans (196), elected to commune-level women’s forums (537), and selected to be MoH Community Health Workers (205). These gender impacts were attributed to the high-quality training the women received during the program, and the program’s emphasis on empowering them as teachers and leaders. 4.0. Key Factors That Affected Program Effectiveness and Efficiency Some of the key factors that effected program implementation, effectiveness, and efficiency were:  The Tubaramure Program’s commitment to a regular model of quarterly meetings with local authorities and Government of Burundi (GoB) staff, and field visits to facilitate joint planning and program integration;  The use of front-line innovators (such as the Tubaramure Health Promoters, who were program employees recruited from and placed in the commune centers) to facilitate all of the community-level program activities in collaboration with representatives of the main 4 These protocols include the MoH-endorsed modules for the Integrated Management of Childhood Illnesses (IMCI); Community Management of Acute Malnutrition (CMAM); Severe and Acute Malnutrition (SAM); Growth Monitoring (GM); and Prenatal and Postnatal Consultation (CPN/CPoN). Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xv ministries that support these activities (such as the volunteer MoH Community Health Workers and the community-based agricultural monitors and veterinary workers);  The program’s emphasis on training female and—after the mid-term—male village leaders (LMs and Leader Fathers [LFs]) to lead and execute the health and nutrition activities;  The early introduction of the Savings and Internal Lending Communities (SILCs) in the second year; and  The program’s development of a new, sophisticated model of commodity tracking at the end of the second year (November 2011) liberated the Tubaramure Heath Promoters from the labor-intensive manual reporting on the program’s PM2A activities. The efficacy of the Tubaramure Program was reduced by:  The MoH staff’s insufficient understanding of the Care Group Model, which was not recognized by any standard MoH protocol;  The program’s staffing structure that inhibited the full range of activities, such as the Tubaramure Heath Promoters working more closely with the MoH/health centers;  The insufficient involvement of men in the program start up in the villages (as outlined in the original MYAP proposal) until just prior to the program’s mid-term evaluation; and  The fact that the original MYAP proposal did not anticipate a wing of activities that would: - Facilitate the LMs developing the types of income-generating activities (IGAs) that would motivate them to continue this volunteer work; - Help the beneficiary households develop the types of agricultural and livestock activities they would need to maintain a more diversified diet after they were no longer eligible for the PM2A rations; or - Provide livelihood benefits whose income would impact the consumption of nutrient-rich food. To address these issues, which were a major focus of discussion during the mid-term evaluation, the Tubaramure Program facilitated:  The creation of 874 groupements, 713 (82%) which are registered, which is a pre￾requisite to working with the Ministry of Agriculture and Livestock (Ministère d'Agriculture et d'Elevage or MAE); 5  Helping organize, train, and equip (with seed, start-up livestock, food processing equipment, and technical assistance) at least two of the new groupements per colline where the program was active;  Training and certifying 48 SILC Private Service Providers (PSPs) in 2013 to help develop and/or sustain a total of 869 program-related SILCs in the area; and  Encouraging the development of 28,117 keyhole gardens in both provinces. Many of these new agro-economic activities (i.e. activities added after mid-term evaluation that were not envisioned in the original program design) have had a huge impact on households’ willingness and capacity to sustain some of the new health behavior and nutrition practices. 5 Approximately 54% of the Tubaramure-facilitated groupements in Cankuzo Province have SILCs and 35% of the Tubaramure-facilitated groupements in Ruyigi. Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xvi 5.0. Critical Challenges to Sustaining the Tubaramure Program’s Results The training needed to sustain the Tubaramure Program IR1 clinic-based activities is likely to be continued because it is built on existing MoH protocols, modules, and trainers. The most critical challenge to maintaining the community-level achievements under IR1 and the new health behaviors that were developed under IR2 will be for the MoH to find better ways to connect the 4,920 Tubaramure-trained LMs with its existing community-based programs. The most critical challenges to sustaining the program’s IR3-level impacts will be for the MAE and the new agriculture and food security programs to continue to train and provide technical assistance to the Tubaramure facilitated-SILCs and groupements; and strengthen the local communities’ access to the improved seed that they need to maintain the keyhole gardens (which are recognized by GoB’s agricultural policy) as a major source of dietary diversity and food intake. 6.0. Examples of Best Practice The evaluators identified a number of examples of programmatic best practices that merit replication and distilled them into a series of best practice and lessons learned checklists for future programs. These best practices include the program’s:  Strong commitment to building MoH capacity through the use of existing training modules, protocols, and staff under IR1, and providing technical and logistical support to the GoB for the revision and/or formulation of key national policies;  Innovative use of the Care Group Model to develop local experts who continue to lobby on behalf of the mainstream MoH programs under IR2;  Use of the PM2A food distribution to attract program participants to the training programs they need to develop new health and sanitation behaviors being promoted under IR1 and IR2;  Innovative Care Group training model, which helped build women’s confidence in their own ability to lead and lobby on behalf of women’s issues like health and hiring women for key community-level positions;  Introduction of a number of two new low-cost innovations— keyhole gardens and Tubaramure facilitated economic groupements—after the mid-term evaluation—that helped build the LMs and beneficiaries to strengthen their livelihoods; and  Early introduction of the low-cost SILC model, which helped provide agricultural loans for livelihood investments as well as build women’s capacity for other types of investments that they needed to ensure a more diversified and nutrient-rich diet. 7.0. Lessons Learned 7.1. For Activities to Build Clinical and Community Capacity to Prevent Malnutrition The final evaluation highlights the critical importance of building new Food for Peace (FFP) programs that conform to the pre-existing national policies and protocols, goals, and priorities of the MoH. One useful contribution of the Tubaramure Program was to highlight the way that a Title II program can provide technical, material, and logistical support to assist with updating Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xvii national policies and protocols (Table A). This type of policy support is much easier when there is a pre-existing MoH protocol that just needs updating, as was the case of the five policy areas that were heavily influenced by Tubaramure under IR1. It is harder when the GoB does not yet have a draft policy or protocol, as was the case of the Care Groups (CGs) in IR2 and PM2A under IR3. 7.2. For Activities to Help Households Practice Appropriate Health and Nutrition Behaviors That Prevent Malnutrition The evaluation provides clear evidence that the Care Group Model can be efficient for galvanizing the types of broad-based behavior change that are needed to sustain the short-term nutritional impacts of a PM2A program. For the Care Group Model to be successful, two top priorities need to be considering how the program’s CG structure will be integrated into the existing health systems from the start of the program; and anticipating the costs and institutional relationships needed to ensure appropriate training of the LMs and MoH staff working with them both during and after the program (Table A). 7.3. For Activities to Help Households Increase Food Intake and Diversity The Tubaramure evaluation shows how the integration of PM2A into a Title II program can accelerate the speed with which the program’s beneficiaries will be willing to develop the new health and nutrition behaviors needed to prevent malnutrition. It also confirms the Tubaramure Mid-Term Evaluation Report’s conclusion that for these PM2A new health and nutrition behaviors to be sustainable, the PM2A activities must be linked to a communication strategy that facilitates high levels of local government buy-in, as well as buy in from the wider population; and more broad-based initiatives to increase food availability and access (Table A). The Tubaramure Program’s commitment to helping the MoH develop a system of regular quarterly coordination meetings helped create high levels of government and interagency buy-in in a format that is likely to be sustained once program funding ends. The program’s behavior change communication (BCC) activities and culinary demonstrations are examples of best practice that helped build the demand for a more diversified diet. Although the program’s support for SILCs and keyhole gardens helped increase the households’ access to a more diversified diet, it is unlikely to be sufficient to build the types of higher-yielding crop and livestock production systems that both areas will need to sustain the program’s positive population-based nutritional impacts. 7.4. For Monitoring, Evaluation, Accountability, and Learning The inclusion of PM2A activities in a Title II program creates a host of monitoring, evaluation accountability, and learning (MEAL) challenges not found in the classic USAID-funded FFP programs. Especially important is managing beneficiary statuses that change regularly based on where they are in the cycle (pregnancy, lactating, child receiving ration, etc.) that has an impact on ration distribution. To address this issue, these programs must develop parallel systems for accountability and reporting for the PM2A distributions, as well as for tracking the routine program outputs, outcomes, and impacts that are reported on in the Indicator Performance Tracking Table (IPTT) required by all USAID FFP programs (Table A). Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xviii 8.0. Summary Comments Was the Tubaramure Program a good investment? That is a question that is being asked by a more in-depth analysis of the cost-benefits by the IFPRI/FANTA program in December 2014. What the evaluation team is able to conclude is the use of PM2A commodities helped jumpstart some of the key capacities that will be needed to reduce both provinces’ chronic malnutrition rates. For these achievements to be sustained, however, will require a host of complementary innovations to reduce the area’s chronic food insecurity. Future programs that combine PM2A programs with other types of food security programming may help identify the magic formula that determines how these can be better linked. Based on the evaluation team’s initial analysis of the most recent update of the program’s quantitative tracking data and the results of their initial focus group discussions and key informant interviews, the current program has met and exceeded the US Government’s expectations for its implementation and impact. The program’s direct link to the IFPRI/FANTA research programs should provide USAID and the GoB with more in-depth analyses of the cost benefits of the PM2A rations versus more conventional recuperative models for treating malnutrition. Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xix Table A. Tubaramure Program Lessons Learned and Best Practice Checklist Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide Government Institutions (MoH and MAE)6 Future Donor￾Funded Activities Future PM2A and Food Security Programs IR1: Women and children under 5 access quality nutrition and health services. Lesson 1. Training programs that contribute to the preparation and revision of existing protocols for the MoH and/or other partners are more likely to be effective, efficient, and sustainable. 1. Build training programs to prevent malnutrition on existing MoH protocols and training modules. X X X 2. Train all MoH staff in the key nutrition treatment and preventions protocols (at the provincial, commune, and community levels) from the start of the program in order to build the roots for sustainability. X X 3. Include a flexible budget for technical and logistical support to update and revise critical MoH protocols and training manuals. X X 4. Build the capacity of the existing MoH trainers in the provincial and district-level health centers to execute training programs for the key protocols being supported. X X 5. Anticipate the need to train and equip Community Health Workers from the start. X X 6. Anticipate the need for a flexible budget line to support formative supervision missions to follow up on basic training of staff in all health centers. X X 7. Anticipate the need for baseline, mid-program, and end-of￾year training programs on all of the key modules in order to accommodate staff turnover. X X 8. Offer certificates to individuals who complete the training to validate their new knowledge to themselves and their managers. X X 9. Strengthen systems for supervising program and staff performance on critical protocols that are underperforming. X Lesson 2. Providing material support to health centers can help motivate the staff and strengthen their capacity to offer higher-quality health and nutrition services. 10. Anticipate the need for a flexible line item of equipment for all local health centers that provide technical support to community-based nutrition and health programs. X X 11. Anticipate the need to update and renew this equipment in the last year of the program. X X 12. Give assistance that is ‘demand driven’ by a health center list of priority needs and MoH norms. X X Lesson 3. The implication and good collaboration with local authorities and NGO partners is essential for the efficient execution and mainstreaming of nutrition interventions into local development plans. 13. Strengthen the existing system and/or create MoH￾managed quarterly meetings that bring together the different health and nutrition actors in the program intervention areas. Maintain Maintain X 14. Strengthen the existing systems for collaborative planning between the MoH Public Health Technicians and Community Yes Yes X 6 This column refers to MoH activities that are critical to maintaining or sustaining the Tubaramure-supported activities once program funding ends. If there is no X in the column, it should be assumed that the MoH is already supporting this activity. Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xx Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide Government Institutions (MoH and MAE)6 Future Donor￾Funded Activities Future PM2A and Food Security Programs Health Workers. 15. Offer training courses to the nurses who manage the local health centers to help them better manage the centers’ nutrition activities and better connect these activities to the other actors (like the MAE), as well as the provincial and commune-level development planning processes that affect their centers. X X X 16. Provide technical and logistical support for local development plans that local governments can use for strategic planning processes as soon as possible in order to ensure that the malnutrition prevention programs are mainstreamed. X 7 X X Lesson 4. Program systems that build on and strengthen the MoH’s existing supervision, monitoring, and evaluation systems help supervisors provide on-the-job training in technical issues that improve service quality and ownership of results. 17. Facilitate joint supervision missions that permit national, provincial, and district-level supervisors to monitor the execution of the activities being supported by the program. Maintain Yes X 18. Harmonize future program’s M&E systems with those of the health centers for the key protocols being supported in order to avoid duplication. X X 19. Strengthen the MoH’s district-level systems for the collection and analysis of nutrition data. X X X 20. Build MoH capacity for more decentralized nutrition data entry and analysis. X 8 X X 21. Support MoH efforts under way in 2014 to add nutrition and community-level sanitation indicators to the list of performance-based financing indicators being tracked by the government to give greater visibility to MoH efforts to promising clinical and community-based new initiatives to prevent malnutrition. X 22.When conducting pilot studies/tests of new initiatives, develop careful tracking systems that allow comparisons of the new initiatives with existing initiatives. X X IR2: Households practice appropriate health and nutrition behaviors. Lesson 1. Care Groups can be a powerful model for promoting the types of broad-based behavior change that are needed to sustain the short-term nutritional impacts of a PM2A program. 1. Consider how the program’s CG structure will be integrated into the existing health systems from the start to ensure its sustainability once program funding ends. X X 2. If the national health system does not recognize the Care X X 7 In the case of Burundi, the timing for this should be no later than the mid-term of the new 2013-2014 Commune Community Development Plans (PCDC)—which should be in 2017—to ensure that the NGOs facilitating these processes and the local administrators are well-versed in the ongoing community-based nutrition and health programs. Early and consistent implication of local administrators in the program’s activities should increase but not guarantee (due to turnover) a more sophisticated and useful analysis of the most critical constraints and planning issues. 8 Train and retrain the provincial and district-level staff in data entry, analysis database creation, and management to avoid labor-intensive hand entry of the MoH and donor forms, which can create costly duplications of effort for M&E and technical staff. Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xxi Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide Government Institutions (MoH and MAE)6 Future Donor￾Funded Activities Future PM2A and Food Security Programs Group Model, consider developing the model as a complement to other activities (like the Hearth Model and community-based growth monitoring [GM]) that are recognized by existing or draft protocols; and continuing to work with the MoH to determine if and how the concept could be better recognized by the existing protocols both during and after the program. 3. Participate in national forums that review various ways that the Care Group Model can strengthen the existing models for community-level BCC. X X X 4. Avoid any sort of direct linkage between the Care Group Model and eligibility for PM2A rations in order to keep the Care Group voluntary and to ensure more broad-based participation. X 5. Complement the Care Groups with mechanisms to reach the wider society with consistent BCC messages that build the wider community’s support and understanding of the messages. X X Lesson 2. Care Groups require careful training and retraining of the implementing staff and beneficiary LMs to be effective BCC agents both during and after the program. 6. Consider adding other staff to deal with non-BCC issues (like IGAs and food distribution) and/or reducing the area of intervention to ensure appropriate backup support. X X Lesson 3. Encourage Care Groups to develop IGAs in order to sustain their BCC activities over time. 7. Anticipate the costs of developing IGAs and/or SILCs to support the LMs developing IGAs from the start of the program. X X 8. Identify the partnerships to develop and sustain the LMs’ IGAs in the initial design, and monitor them so they can be adjusted as the most viable IGAs are identified. X X 9. Budget adequate staff/partner time needed to support these activities so that the IGA activities do not detract from Care Group and BCC trainings. X X Lesson 4. Anticipate the need for a gender-sensitive communication strategy that develops a wide base of community support and understanding for the BCC messages, both during and after a PM2A program. 10. Include a well-thought-out draft gender strategy in all PM2A proposals that anticipates some of the special challenges associated with PM2A that are not found in more conventional food security programs. X 11. Ensure that this strategy complies with the gender strategy of the funding agency (like USAID), as well as donor expectations for gender monitoring and reporting (only recently adopted).9 X 12. Ensure that the program gender strategies are compatible X 9 The USAID gender strategy was formally adopted in March 2012 and updated in 2013 in the USAID Automative Directive System (ADS), Chapter 205 (aka ADS 205) (Integrating Gender Equality and Female Empowerment in USAID’s Program Cycle, New Edition Date: 07/17/2013, Responsible Office: PPL File Name: 205_07171307/17/2013 New Edition), which outlines the expectations for staff integration of the policy into existing and future USAID-funded programs. Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xxii Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide Government Institutions (MoH and MAE)6 Future Donor￾Funded Activities Future PM2A and Food Security Programs with the national gender strategy, and provincial and commune-level coordination structures. 13. Conduct an annual review of each PM2A program’s gender strategy as part of the routine annual review and planning processes. X 14. Include internal indicators for the gender strategy that are tracked as part of the routine internal and donor tracking of the program, even if these are not in the IPTT. X 15. Have a field-based gender specialist in each program intervention zone that can also function as the local capacity building and M&E officer. X 16. Budget basic gender training and retraining of staff and all local government and NGO partner staff the program works with. X IR3: Eligible women and children have increased intake of nutrient-rich diverse foods. Lesson 1. Given the complexity of selecting and distributing rations to the target beneficiaries, programs need strong two-way communication with the local governments. 1. PM2A programs need to anticipate strong public awareness campaign at the beginning of the program to explain the activities and target audiences to local authorities before discussing the activities at the community level to help avoid misunderstandings about why the food is only given to some people and to help control illegal sales. X 2. Strengthen existing coordination groups as a mechanism for staying in touch with local authorities in order to elicit their support in resolving conflicts, avoiding commodity theft, and promoting the program’s community-level activities. X X 3. Produce attractive posters that promote key Essential Nutrition Actions (ENA) and Essential Hygiene Actions (EHA) themes, and distribute them through local authorities to health facilities, food distribution points, and households to help sustain community awareness of critical themes and behaviors. X Lesson 2. Food distribution sites can offer a useful locale for building local government and wider community understanding of the new ENA and EHA themes being promoted by a program. 4. Consider innovative methodologies (e.g. posters, culinary demonstrations, skits) for promoting new ENA and EHA messages at the food distribution sites. X 5. Anticipate the need for water, basic hygiene, and sun and rain protection at PM2A distribution sites. X Lesson 3. Culinary distributions that promote local foods in conjunction with food rations proved a useful tool for on-the-ground nutrition training at the start of a program, which can then be scaled up to promote nutritious local foods. 6. Integrate information on local foods that can complement food rations from the start into all PM2A culinary demonstrations. X 7. Consider ways that cookbooks can help build local authorities, MoH, MAE, and program staff’s understanding of the culinary demonstrations and foods being promoted. X 8. Encourage government agencies (like the district and X Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xxiii Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide Government Institutions (MoH and MAE)6 Future Donor￾Funded Activities Future PM2A and Food Security Programs provincial MoH and the commune and provincial offices of the MAE) to consider ways that cookbooks could be incorporated into and/or support their programs. 9. Consider linking any culinary demonstrations and/or cookbooks developed under the program staff to any new or existing programs to promote community-based GM and Foyer d'Apprentissage et de Réhabilitation Nutritionnelle (FARNs or Positive Deviance [PD]/Hearth). X 10. Network to identify promising high-nutrient foods like soy, soy processing, and amaranth that the local people may not be aware of for pilot testing. X 11. Identify options for high-nutrient weaning foods made with local foods and nutrient-dense foods like soy, and promote them from the start both for children’s health and as potential IGAs for program beneficiaries. X Lesson 4. Keyhole gardens and SILCs are a useful, low-cost model for increasing household diversity, which can be quickly scaled up and sustained with minimum outside support for both PM2A beneficiaries and the wider community. 12. Before introducing keyhole gardens, identify examples of regional or national best practices and options for improved seed in order to make sure that the package is appropriate to the target area. X X 13. Once an appropriate keyhole garden model and package of seeds has been identified, encourage staff to work through the local MAE staff to adapt the program to the local microenvironments in each area. X X 14. Consider ways that the pre-existing technical resources in the area (like the MAE or existing agricultural development programs) can help support the pilot testing and scale up of keyhole gardens to avoid overburdening the program extension agents and strengthening the program’s links to the MAE. X X 15. Identify an experienced SILC expert to train staff on basic principles and models. X X 15. Identify different models for creating new or building the capacity of SILC Private Service Providers (PSPs) in the program area as a way of reducing the burden that SILC activities might place on the program extension agents and/or local MoH staff. X X 16. Anticipate the need to help the most successful SILCs to identify IGAs and access IGA loans from the existing microfinance institutions in the program intervention zone. X X Cross-Cutting MEAL Lesson 1: Programs need to anticipate the need for a well￾designed database that tracks PM2A beneficiaries both prior to, during, and after the distributions end in order to ensure appropriate distribution and follow-up support. X Lesson 2: MEAL programs that build on and strengthening the government’s existing systems for tracking health and development (as the Tubaramure system did for the Burundi X Tubaramure Final Evaluation. Executive Summary. October 22, 2014. Final. xxiv Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide Government Institutions (MoH and MAE)6 Future Donor￾Funded Activities Future PM2A and Food Security Programs MoH) can reduce duplication and strengthen government buy￾in both during and after the program (see IR 1, recommendations 18-20). Lesson 3: Anticipate the need for PM2A programs to have an experienced M&E staff to develop the types of sophisticated MEAL systems that these programs will require from the start of their field operations rollout. X Source: Tubaramure Final Evaluation; July-August, 2014. Revised based on feedback to the first draft, September 25-October 5, 2014. Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 1 Chapter 1 Context and Methodology 1.0. Overview of the Tubaramure PM2A Program The principal objective of the Tubaramure Multi-Year Assistance Program (MYAP) is “malnutrition in children under 2 years of age is prevented” (Table 1.1). The title of the program, Tubaramure, means, “Let’s help them grow” in Kirundi, a Bantu language spoken in parts of Burundi, and was chosen to reflect the central role of the local community in preventing malnutrition. The program’s emphasis on preventing malnutrition was very different from more conventional approaches to nutrition in Sub-Saharan Africa, which focus on treating malnutrition once it has already occurred. The program was designed to support three intermediate results (IRs) deemed critical to preventing malnutrition (Table 1.1). It was also designed to provide a background to a more broad-based, in-depth analysis of the impact and cost effectiveness of Preventing Malnutrition in Children Under 2 Approach (PM2A), an approach that was first identified in 2008 as more effective in reducing malnutrition than the historically favored remedial methods.10 Table 1.1. Framework for the Tubaramure Program Strategic Objective: Malnutrition in children under 2 years of age is prevented. Intermediate Result (IR) Output IR1. Women and children under 5 access quality nutrition and health services (Technical lead: International Medical Corps [IMC]) 1.1. Pregnant and lactating women access pre and postnatal care services. 1.2. Implementation of national Integrated Management of Childhood Illness (IMCI) plan is supported. 1.3. Health facilities supported in providing growth monitoring (GM). 1.4. Severe acute malnutrition (SAM) is detected and referred for treatment. IR2. Households practice appropriate health and nutrition behaviors (Technical Lead: Food for the Hungry [FH]) 2.1. Households (HHs) adopt Essential Nutrition Actions (ENA). 2.2. HHs adopt Essential Hygiene Actions (EHA). 2.3. HHs adopt prevention and management behaviors for maternal and childhood illnesses. IR3. Eligible women and children have increased intake of nutrient-rich diverse foods (Technical Lead: Catholic Relief Services [CRS] in partnership with Caritas) 3.1. Food for Peace (FFP) rations distributed to eligible women and children at community level. 3.2. Mothers and children use FFP rations appropriately. 3.3. HHs use appropriate local foods in addition to FFP ration. Source: CRS; 2009. Tubaramure MYAP Proposal. Bujumbura: CRS. (Revised submission, June 29, 2009). Pg. 7. 10 IFPRI, in collaboration with World Vision-Haiti, Cornell University, and the Food and Nutrition Technical Assistance (FANTA) Project, provided the first programmatic evidence that the preventative approach of blanket￾targeting of a food-assisted maternal-and-child health and nutrition program to all children 6-to-24-months old was more effective in reducing the prevalence of stunting, wasting, and being underweight than the traditional recuperative approach based on solely targeting underweight children (weight-for-age Z-score [WAZ] < -2) under 5 years of age. Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 2 The Tubaramure Program was designed to achieve three IRs.  The first program IR (IR1) was to ensure “women and children (0-59) access quality nutrition and health services” designed to prevent malnutrition, not just treat it. To achieve this, the Tubaramure Program provided technical and logistical support to help the Ministry of Public Health and the Fight Against AIDS (MSPLS or Ministère de Santé Publique et la Lutte Contre le Sida, hereafter referred to as the MoH) revise its existing training modules and protocols dealing with malnutrition and preventing childhood diseases; train staff and health volunteers using the revised modules and protocols; provide certain types of basic equipment (beds, measuring tables, scales, etc.) that the new and existing health facilities needed to support these protocols; and intensify its existing system of supervisory missions to support the newly revised nutrition protocols.  The second IR (IR2) was designed to help “households practice appropriate health and nutrition behaviors” needed to prevent malnutrition and to access the improved health services being developed under IR1. To achieve this, all the mothers who were eligible to receive PM2A rations under IR3 were required to join a beneficiary group that included 10-15 PM2A beneficiaries. Each group elected a Leader Mother (LM) who attended a bi￾monthly Care Group11 training program promoting key nutrition and hygiene behaviors mothers need to maintain child health and nutrition.  The third IR (IR3) was designed to ensure that “eligible women and children have increased intake of diverse foods” by providing a blanket distribution of PM2A food rations during pregnancy and the child’s first two years of life, and promoting the use of appropriate local foods in addition to the Food for Peace (FFP) ration. The program was implemented in a total of 268 collines (literally “hills”) in the 12 communes of Cankuzo and Ruyigi Provinces (Figure 1.1). These border provinces historically have some of the highest malnutrition rates in Burundi. It was executed by a consortium of non-governmental organizations (NGOs)—Catholic Relief Services (CRS), Food for the Hungry (FH), and International Medical Corps (IMC)—that are all leaders in the field of community-based malnutrition and health and food security programming. 11 “A Care Group is a group of 10-15 volunteer, community-based health educators who regularly meet together with NGO program staff for training and supervision. Each of these volunteers then go out at least monthly to do health promotion with a small cohort of mothers of young children. They are different from typical mothers’ groups in that each volunteer is responsible for regularly visiting 10-15 of her neighbors, sharing what she has learned, and facilitating behavior change at the household level. Care Groups create a multiplying effect to equitably reach every beneficiary household with interpersonal behavior change communication (BCC). They also provide the structure for a community health information system that reports on new pregnancies, births, and deaths detected during home visits. The model was created by World Relief in 1995, and pioneered and championed by FH and World Relief since then.” http://www.caregroupinfo.org/blog/ Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 3 Figure 1.1. Location of Ruyigi and Cankuzo Provinces in Burundi Source: Tubaramure Monitoring and Evaluation (M&E) Office. UN Office for the Coordination of Humananitarian Affairs, (OCHA) Bujumbura, Burundi Tel.: (257) 21.80.34, 219157/8 Fax: (257) 21.80.35 Makamba Rutana Bururi Gitega Ruyigi Cankuzo Karuzi Ngozi Bujumbura Mairie Bujumbura Rural Kayanza Muyinga Cibitoke Bubanza airstrips served by WFP internal flight Kirundo Muramvya Mwaro Tubaramure PM2A Area BURUNDI Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 4 Tubaramure was a pioneer program that pilot tested many new methodologies for preventing malnutrition. It was:  The first to pilot test the PM2A concept in Sub-Saharan Africa; and  The first to pilot the concept of Care Groups in relation to PM2A behavior change communication (BCC) strategies worldwide. For these reasons, there was a great deal of national and international interest in the program. The total program cost to FFP to date is US$43,133,600 million, including $9,989,050 million in Section 202(e) cash12 and international transport, storage, and handling (ITSH); $6.9 million in commodities for monetization; and the rest in commodities for distribution. The program, which targeted having an impact on 51,075 mother-child pairs under IR2 and IR3 and 100% of the health centers in both provinces under IR1, began implementation in July 2009 and is scheduled to be completed in November 2014. 2.0. Early Evolution of the Program The consortium carried out a baseline study at the beginning of Tubaramure in March 2010, which set the determination of reference data and targets for performance indicators for the duration of the program. The mid-term evaluation was conducted in May and June of 2012,13 when the program was still at its peak number of beneficiaries and only a small number of women had started to graduate (i.e. were no longer eligible to receive PM2A rations) because their children had reached 2 years of age (Figure 1.2). The general consensus of the mid-term evaluation report was that the program was, “well on its way toward meeting the majority of its objectives and targets and has already achieved visible results in terms of people trained, changes in behavior at the household and community level, improved services at health facilities, and better nutritional status among young children.”14 There was, however, increasing evidence that many of the mothers were dropping out of the Care Groups after they graduated (i.e. once they were no longer required to participate as a condition for getting PM2A rations). There was also a growing concern with developing a more realistic hand-off plan to the MoH. 12 202e: Cash resources made available to FFP partners for enhancing programs. 13 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. (July 2012). 14 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. (July 2012). Pg.1. Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 5 Figure 1.2. Evolution of Tubaramure PM2A Beneficiaries Not Yet Graduated from the Program, 2009-201415 Source: Tubaramure M&E Office; August 2014. Based on the mid-term evaluation report’s analysis and recommendations, the program:  Added a wing of activities designed to help beneficiary mothers and LMs develop income-generating activities (IGAs) to “provide incentives for these groups to remain active;”  Began working in close concert with local partners to develop a concrete, detailed exit plan;  Moved forward with its original plan to pilot test a new system for community-based growth monitoring (GM) to track the mothers and children who had graduated from the ration distribution activities in the 12 collines; and  Strengthened the program’s outreach to and involvement of the beneficiary mother’s husbands. During the same time period, the IFPRI/ Food and Nutrition Technical Assistance (FANTA) project completed its initial analysis of the program’s baseline data on malnutrition, which was presented to the team in a workshop just after the mid-term evaluation. The Tubaramure team followed up on these recommendations, as well as those made by the mid-term evaluation report as part of its routine program reporting. 15 The official total number of direct beneficiaries is listed as 49,650. These figures indicate the number of beneficiaries that have not yet graduated from the PM2A distribution. 8695 16367 13251 3929 354 15346 26459 29278 7359 581 24041 42826 42529 11288 935 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 July 2010 July 2011 July 2012 July 2013 July 2014 Cankuzo Ruyigi Total Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 6 3.0. Expected Program Outputs, Outcomes, and Impacts The Tubaramure Program design was very simple and intended to be highly interactive, with each sub-component affecting the other. Although the initial impetus for belonging to a Care Group was to get rations, it was expected that the mothers would maintain the new behaviors once they saw the impact on their children’s health and nutrition. And it did—based on the results of the program’s final quantitative household surveys (Text Box 1.1), and verified by the focus group discussions and key informant interviews during the final evaluation. The Tubaramure Program expected that the higher-quality services in the health centers would increase the number of local people making use of the local services and raise the provincial￾level rankings of the local health departments. And it did. Although the principal focus of the program was the PM2A beneficiaries, the same program was expected to have a wide variety of spread effects into the general population. It was expected to have a population-based impact that could be tracked through the program’s outcome and impact indicators. And it did. The program was expected to provide a backdrop to a separate, autonomously financed research study by IFPRI and FANTA on 60 communities within the broader program area that would help the international community better understand the strengths and weaknesses of the PM2A model as a development approach that could be compared with similar research in Haiti and Guatemala. This separate research program has been ongoing and is slated to continue its mainstream activities through December 2014 and possibly beyond. Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 7 Text Box 1.1. Evidence for Tubaramure Population-Based Impacts on Nutrition As anticipated in the program document, the Tubaramure Program has had a significant population-based impact on malnutrition. The recently completed (May 2014) Institut de Statistiques et d’Etudes Economiques du Burundi (ISTEEBU) study of a stratified random sample of 1,200 households in 40 collines in both provinces found statistically significant reductions (p<0.05) in the rate of acute malnutrition (from 8.4% in 2010 to 4.8% in 2014) and the percentage of children classified as underweight (from 46.7% in 2010 to 22% in 2014). The same study showed a decrease in the number of children classified as stunted from 52.3% in 2010 to 50.5% in 2014, but this difference was not statistically significant. A disaggregated analysis by province shows that Tubaramure Program had a higher impact on stunting in Cankuzo (38.9%) than Ruyigi (57.2%). The Tubaramure final household survey showed a remarkable 13-point decrease in the prevalence of stunting—from 52.3% in 2010 to 38.9% in 2014—that was statistically significant (p=0.000). This significant reduction in the rate of malnutrition (acute malnutrition and underweight) was confirmed using other information gathered during the final evaluation focus group discussions and key informant interviews. Most of the beneficiary focus group discussions stated:  The program had a significant impact on children’s health and well-being; and  The majority of women continue to follow the program’s recommendations for child nutrition and improved hygiene even after graduation. Every one of the LMs, MoH staff, Community Health Workers, and local authorities in the communities where the team conducted the focus group discussions concluded (Annexes IV and V):  The Tubaramure Program had considerably improved the health and hygiene status of children;  In addition to receiving the PM2A rations, the women had learned new ways to prepare nutritious meals using local foods; and  Most of the beneficiary mothers have continued to use the improved nutrition and hygiene practices (hand washing and latrines) for the new children born after their PM2A graduation. Source: Chapters 2, 3, 4, and 5 of this report. In sum, the Tubaramure Program in Burundi was an experiment that had significance both at the national and international level. This accounts for the relatively unique focus of the evaluation on lessons learned for future programs. 4.0. Evaluation Objectives and Methodology 4.1. Objectives Per FFP requirements, the principal objectives of the final evaluation were to determine:  The relevance, effectiveness, efficiency, and acceptability of processes and outputs;  The factors affecting the implementation and the degree of adherence to the terms of initial agreement;  What and why results (outcomes/impacts) have/have not been achieved; and  The sustainability of the results. The final evaluation was also expected to describe the outcomes and impact (intended and unintended) of the program’s activities; how contextual or program-related factors contributed to Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 8 greater or less positive outcomes; and factors that promote/threaten the sustainability of positive impact after the program ends. Changes of greatest interest were those related to:  Women’s diets;  The use of health services during pregnancy and lactation;  Men’s and women’s knowledge and practices related to hygiene and caring for young children;  Household diets; and  Gender equity in the community and at home. In addition, the team was expected to examine the strengths and weaknesses of the program’s design, implementation, and outputs, considering how well it has:  Been appreciated by the community;  Contributed to positive outcomes;  Avoided unintended negative outcomes; and  Prepared its exit strategy and left sustainable results. 4.2. Relationship to the IFPRI/FANTA Evaluation A separate evaluation of the IFPRI/FANTA research study is scheduled for December 2014. This study aims to better understand and document the impact, cost, and cost effectiveness of PM2A programs, and to generate institutional knowledge for the United States Agency for International Development (USAID)/FFP about how PM2A programs should be designed and implemented to maximize their impact and cost effectiveness. Examples of the questions that the evaluation of the IFPRI/FANTA study—as opposed to the current evaluation—will address include: 16  How much food is needed (individual vs. household ration)?  What types of food/products are best?  Should micronutrient supplements be used in place of individual food rations? and  What is the best timing and duration of exposure to a PM2A program? 4.3. Specific Objectives and Key Questions from the Evaluation Scope of Work In addition to the general objectives of the evaluation, the evaluation outlined seven specific objectives for the evaluation and 11 key questions for IR1; and 11 specific objectives for the evaluation and 25 specific questions for IR2 and IR3. A complete list of the questions is provided in Annex I.A to facilitate a cross-reference to where the questions are addressed in the main text. When questions are addressed, they are also noted in the text’s footnotes. 4.4. Methodology 4.4.1. Principal Data Sources. The approved scope of work (SOW) anticipated that the evaluation would be based on three types of data: 16 Annex I.B, Page 2. Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 9  Quantitative survey: A quantitative, population-based survey that would be comparative with the Tubaramure Program’s baseline survey conducted in March 2010;  Focus group discussions: A series of focus group discussions with Tubaramure beneficiaries and non-program beneficiaries and community leaders in a representative sample of communities; and  Key informant interviews: A series of open-ended interviews with key informants, including program staff associated with CRS, FH, IMC, and Caritas; key MoH partners at the provincial and commune levels; as well as local government authorities at the provincial and commune levels. The same SOW anticipated that the evaluation would be led by a team of two external consultants17 and a senior member of the MoH’s staff tasked with setting up all of the government-level key informant interviews.18 To ensure its independence, the team was expected to recruit and hire its own set of enumerators. 4.4.2. Quantitative Final Survey. The principal quantitative database for the final survey was the program’s final quantitative survey, which was conducted in May 2014 by the same government statistical service that conducted the baseline. This survey conducted a stratified random sample of 1,200 households in 40 collines in Cankuzo and Ruyigi provinces that included 1,196 women and 1,784 children under 5 years. The study was population-based and intended to measure the program’s global impact on malnutrition and health behaviors in both provinces. Sixty-two percent of the women interviewed were beneficiaries; only 160 (fewer than 20%) had been Tubaramure LMs (Table 1.2). 17 Dr. Della E. McMillan (based in Gainesville, Florida) is a livelihoods specialist with an extensive background in gender and capacity building, most recently as the gender consultant for the Millennium Challenge Corporation (MCC) Compact-funded projects in Burkina Faso. She has worked on the design and evaluation of Title II food security programs in Mali, Niger, Burkina Faso, Chad, Guinea, Ethiopia, and Uganda. She is the co-author of two publications in the CRS online monitoring and evaluation publication series. Dr. Sidibe Sidekiba (based in Conakry, Guinea) is a Doctor of Medicine (MD) with a specialization in pediatric nutrition (University of Conakry), with a Master of Science (MS) in Epidemiology (University of Bordeaux) and a MS in health statistics (University of London). He has extensive experience in the design of qualitative and quantitative surveys for Title II programs in Guinea, Rwanda, Mali, Burkina Faso, and Chad. As the senior health advisor of Africare’s Title II program in Guinea, he pilot tested one of the Hearth model programs for the rehabilitation of moderately malnourished children in francophone West Africa. As a MYAP coordinator in Rwanda, Dr. Sidibe pilot tested a series of highly innovative programs for HIV/AIDS-affected households. 18 Dr. Evelyn Ngomirakiza (MD, University of Burundi; MS Nutrition, Makerere University) is director of the Burundi MoH Nutrition Unit, and was an external observer on the team who represented the Government of Burundi. Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 10 Table 1.2. Quantitative Households Survey Sample, Baseline, and Endline Variables Baseline 2010 Endline 2014 Number of collines 20 40 Number of households 1,255 1,200 Number of children under 5 years of age 1,454 1,784 Number of non-direct beneficiary households N/A 450 Number of women interviewed who were LMs N/A 169 Source: Institut de Statistiques et d’Etudes Economiques du Burundi (ISTEEBU); 2014. Rapport Final Enquête Ménage pour l’Évaluation Quantitative Finale du Programme PM2A Tubaramure (Provinces Cankuzo et Ruyigi). Bujumbura: ISTEEBU; August 2014. 4.4.3. Key Informant Interviews. Dr. Della E. McMillan and Dr. Evelyn Ngomirakiza conducted the key informant interviews of program staff and MoH staff. 4.4.4. Community-Level Focus Group Discussions and Key Informant Interviews. The qualitative survey was designed and executed by Health and Nutrition Specialist Dr. Sidibe Sidikiba. The same survey included key informant interviews with 16 staff members associated with the area’s commune-level health centers. The focus group discussions were conducted by six enumerators in Kirundi and then translated from Kirundi to English by two translators. The final choice of enumerators and translators was made by the consultants during the first week in country from a pool of eligible candidates who had worked for Institut de Statistiques et d’Etudes Economiques du Burundi (ISTEEBU) and other programs. Three of the enumerators were from the department of psychology, and three from agricultural fields. All of them had experience with other quantitative and qualitative surveys in Burundi. None of the enumerators—most of them recent graduates of the university— had any previous association with Tubaramure or the implementing partners (CRS, FH, Caritas, and IMC). All of the enumerators received a two-day training during which the questionnaire was translated and pilot tested in one of the survey villages. The six enumerators were divided into two teams. To ensure the quality and consistency of the enumerator’s work, either Dr. Sidibe or one or both of the translators accompanied each team. Each enumerator was responsible for conducting the initial transcription of his interviews in Kirundi, which was then turned into one of the two translators and Dr. Sidibe for a group team review and discussion. The summary of these meetings was typed up daily by the two translators under the direct supervision of Dr. Sidibe (see Annexes IV and V). 4.5. Schedule 4.5.1. Step One: Evaluation Set Up (July 4-12, Bujumbura). During the first week in country, the evaluation consultants focused on key informant interviews; meeting with ISTEEBU staff to finalize the analyses in the quantitative final evaluation; developing a list of focus group guides; selecting the enumerators and translators; and conducting the first round of key informant interviews. A four-part strategy was adopted to ensure that a representative sample of collines was included in the qualitative survey: Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 11  Preparation of a List of Program Activities by Community: With input from the Bujumbura-based Tubaramure staff, the evaluation team developed a sample list of program activities by IR;  Qualitative Ranking of the Activities in Each Colline: Based on this initial list, the two provincial program managers were asked to facilitate a group discussion with their program staff from the Cankuzo and Ruyigi provinces, during which the staff would conduct a qualitative ranking of the principal activities for each IR based on their current level of performance and the prospects of this performance being sustained at its current level after the program ended. To facilitate this group exercise, the teams were asked to rank each activity in each colline on a scale of 1-3, with 1 being the highest and 3 being the lowest. This assessment was then color coded19 and reviewed by all of the senior technical staff in each province;  Qualitative Ranking of the Collines by Their Performance on Key Activities: Based on this number, Dr. Sidibe grouped all of the communities into three clusters (Table 1.3). The first cluster included the villages where the staff’s ranking of the activities indicated that most of the activities were classified as performing well (level 1) with levels of community and technical support that made it likely that they would be sustained at current levels once the program funding ended. The second cluster included villages where most of the activities were ranked at level 2. The third cluster included villages where most of the activities were ranked at level 3; and  Selection of Sample Collines from the Clusters: Based on this initial cluster, the lead consultant for nutrition and health identified a sample of 21 collines from the different clusters—13 from Ruyigi and eight from Cankuzo (Table 1.3). Table 1.3. Retroactive Qualitative Classification of Tubaramure Villages Used to Determine the Sample Frame for the Final Qualitative Survey Focus Group Discussions Colline Clusters Ruyigi/Cankuzo All Collines20 Collines Surveyed Performing well/results most likely to be sustained at current levels even after the program funding ends 63 (26.5%) 6 (28.6%) Average performance/ results likely to be sustained but at lower than current levels 151 (63.5%) 12 (57.1%) Poor performance/results least likely to be sustained 24 (10.0%) 3 (14.3%) Total 238 21 Source: Tubaramure Program Final Evaluation; August 2014. 4.5.2. Step Two: Initial Enumerator Training, Focus Group Discussions, and Key Informant Interviews (July 14-23, 2014, Ruyigi). The second week, the team moved to Ruyigi. During the first two days, the enumerators reviewed the initial translation of the questionnaire. This enabled the team to adjust the formulation of certain questions and to determine the most appropriate composition of the focus groups. The enumerators—under the direct supervision of Dr. Sidibe with assistance from the two translators for supervision and quality control —conducted the interviews in 13 collines. An initial stakeholder debriefing was conducted on July 23 in Ruyigi. 19 Activities that were ranked level 1 were color coded green; those ranked level 2 were color coded yellow; and those ranked level 3 were color coded red. 20 Excludes the 60 collines included in the IFPRI/FANTA survey. Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 12 Working with Dr. Sidibe each evening, the enumerators prepared summary testimonials and analyses by category of informant that were then transcribed into English (See Annex III for a blank copy of the focus group discussion guides in their original format (French); Annex IV for a typed summary of the interviews in English; and Annex V for exemplary testimonials given by different categories of beneficiaries during the focus group discussions in English). 4.5.3. Step Three: Enumerator Training, Focus Group Discussions, Key Informant Interviews, and Partial Stakeholder Debriefing (July 23-29, 2014, Cankuzo). The team started the focus group discussions and key informant interviews in Cankuzo on July 24 and conducted a second stakeholder debriefing on July 29 in Cankuzo. 4.5.4. Step Four: Analysis and Write Up, Follow Up Key Informant Interviews, and Three Stakeholder Debriefings (July 30-August 7, 2014, Bujumbura). On July 30, the team returned to Bujumbura, where the activities focused on finalizing some of the missing quantitative analyses based on the final quantitative survey and conducting three additional stakeholder analyses. 4.5.5. Step Five: Final Write Up and Editing (August 2-24, 2014). Based on input from the stakeholder debriefings and the results of the focus group discussions and quantitative survey, the consultants developed an English version of the draft. Final editing was conducted by the consultants in collaboration with editor Lynn Hurtak. 4.6. Total Number of People Interviewed During Final Evaluation The team interviewed 657 persons in 50 focus groups (Table 1.4); another 113 during key informant interviews in the two provinces; and 19 with program, government, and USAID staff in Bujumbura (Table 1.5). In addition to this, the team conducted five debriefing sessions and one ‘up-briefing’ of the methodology attended by 116 persons (Table 1.6). The debriefing sessions provided useful feedback on the quantitative analyses, lessons learned, and recommendations. Table 1.4. Number of People Interviewed in the Tubaramure Final Evaluation Focus Groups, July 2014 Category of Community-Based Beneficiary Ruyigi Cankuzo # Focus Groups # Persons Encountered # Focus Groups # Persons Encountered Male Female Total Male Female Total Care Group/LMs 6 80 80 4 43 43 Graduated mothers 7 138 138 7 85 85 Saving and Internal Lending Community (SILC) members 5 8 58 66 6 19 40 59 Agro-pastoral groups (groupements) 3 5 24 29 4 14 45 59 Husbands of beneficiary group members/fathers of beneficiary children 4 64 64 4 34 34 Total 25 77 300 377 25 67 213 280 Source: Tubaramure Program Final Evaluation; August 2014. Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 13 Table 1.5. Number of Key Informant Interviews in the Tubaramure Final Evaluation, July 2014 Category of Key Informant Ruyigi Cankuzo National and International Local colline authorities 10 4 MoH personnel: nurses/ medical assistants/medical doctors/Public Health Technicians 21 10 Technical staff of Tubaramure Program, including the Tubaramure Health Promoters 18 12 MoH Community Health Workers 9 6 Other government partners (governor, technical services) 13 10 ISTEEBU 1 MoH national staff 1 CRS national program current and former staff 7 FH national program staff 4 IMC national program staff 2 Caritas program staff 2 USAID/Burundi 2 Total 71 42 19 Source: Tubaramure Program Final Evaluation; August 2014. Table 1.6. Approximate Number of People Who Attended Workshops Associated With the Final Evaluation of the Tubaramure Program Date Location Total Number of Staff and Government Total Number of Consultants, Enumerators, and Translators July 23, 2014 Ruyigi 22 11 July 29, 2014 Cankuzo 13 11 August 1, 2014 Bujumbura 36 2 August 4, 2014 Bujumbura 11 2 August 5, 2014 Bujumbura 34 2 Total 116 NA Methodology: Based on the official meeting records that were summarized by the Tubaramure M&E office. Source: Tubaramure M&E Office; August 2014. 5.0. Organization of the Report Chapters Two-Four provide a brief overview of the major findings of the evaluation as outlined in the approved SOW.21 To facilitate a cross-examination of the major findings between the chapters, each chapter follows the same basic format that examines the following for each of the program’s three IRs:  Original Strategy: The ‘theory of change’ that the activities were designed to address;  Evolution of Activities (Years 1-5): The evolution of the program’s activities related to that particular IR before the mid-term evaluation and any changes that were introduced afterward; 21 As indicated in Section 4.3 of this chapter, in addition to the general objectives of the evaluation, the evaluation outlined seven specific objectives for the evaluation and 11 key questions for IR1; and 11 specific objectives for the evaluation and 25 specific questions for IR2 and IR3. A complete list of the questions is provided in Annex I.A to facilitate a cross-reference to where the questions are addressed in the main text. When questions are addressed, they are also noted in the text’s footnotes. Tubaramure Final Evaluation. Chapter 1. October 22, 2014. Final. 14  Evidence of Results: The early evidence of achievement of the principal program outputs and outcomes for that particular output, and some of the key factors that affected these outputs and outcomes—i.e. the results and effects of the different activities, the effectiveness of the implementation process, and the likelihood that the outputs and outcomes would be sustained; and  Lessons Learned and Recommendations: The major lessons that can be learned from the experience for future programs in Burundi as well as other PM2A programs in other countries. Chapter Five evaluates the overall program impact of the principal program outcomes and the outcomes that are most likely to be sustained once program funding ends. This is followed by six annexes that present:  The cross-cutting questions that the final evaluation team was asked to address, as well as the final approved scopes of work for the team leader/evaluation specialist and the nutrition specialist on the team (Annex I);  The final Indicator Performance Tracking Table (IPTT) for the program (Annex II);  The interview guides used in the qualitative survey (Annex III);  A summary of the major results of the qualitative survey for each guide (Annex IV);  As well as key testimonials (Annex V); and  Additional tables for each of the main chapters (Annex VI). Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 15 Chapter 2 Intermediate Result 1: Women and Children Under 5 Access Quality Nutrition and Health Services 1.0. Global Strategy 1.1. Expected Outputs The activities under the first intermediate result (IR1) of the Tubaramure Program were designed to strengthen the capacity of the Ministry of Health (MoH) facilities and volunteer Community Health Workers to deliver high-quality general health and nutrition services. The program was designed to support MoH’s efforts to improve the quality of health services—including Integrated Management of Childhood Illness (IMCI), growth monitoring (GM), and Community￾Based Management of Acute Malnutrition (CMAM)—and to break the barriers to access.22 The activities were designed to achieve four outputs (Table 2.1). International Medical Corps (IMC) is the technical lead for this IR. Table 2.1. Major Outputs Designed to Achieve the Tubaramure Program’s IR1 IR1. Women and children under 5 access quality nutrition and health services. Output 1.1. Pregnant and lactating women access pre and postnatal care services. Output 1.2. Implementation of national Integrated Management of Childhood Illness (IMCI) plan is supported. Output 1.3. Health facilities supported in providing growth monitoring (GM). Output 1.4. Severe acute malnutrition (SAM) is detected and referred for treatment. Source: CRS; 2009. Tubaramure MYAP Proposal. Bujumbura: CRS. 1.2. The Original Intervention Model To achieve these four outputs, the Tubaramure Multi-Year Assistance Program (MYAP) outlined a three-pronged strategy for IR1 that focused on:  Training: Teaching the MoH’s clinical (i.e. health center-based) and volunteer Community Health Workers and their supervisors (the MoH Public Health Technicians) the basic principles of nutrition and the tools (e.g. pre and postnatal care, growth￾monitoring promotion, integrated management of childhood diseases, and the Community Health Worker integrated manual) they would need to diagnose, treat, and prevent malnutrition;  Material Support: Helping the new and existing centers offer better clinical support for both malnutrition prevention and treatment, as well as general services; and  Monitoring and Supervision: Providing logistical and technical training for: - Supportive supervision by the Tubaramure IR1 technical staff to promote on-the-job training of MoH staff (those who were trained as well as those who were not); - Joint supervision by Tubaramure IR1 and MoH staff for routine tracking, as well as the implementation of the new protocols being revised under IR1; and - Coordination meetings that bring together the different government and non￾governmental organization (NGO) actors involved in community-based health and nutrition activities in both provinces. 22 CRS; 2009. Tubaramure Multi-Year Assistance Program (MYAP) Proposal. Bujumbura: CRS. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 16 2.0. Activities23 24 The Tubaramure program facilitated and supported the training of MoH personnel and Community Health Workers using the MoH’s available protocols, trainers, and training materials. 2.1. Trainings 2.1.1. Technical and Logistical Support to Update and Revision of Existing MoH Modules. All of IR trainings of MoH were designed to complement one of the existing MoH protocols (Annex VI.A.1 and VI.A.2). In 2009-2010, the program worked in close collaboration with the MoH to update most of the protocols and their training modules. Two types of support were provided (Table 2.2):  Simple technical support to revise the protocols in ways that would best take into account the decentralization and new innovations being recommended by the World Health Organization (WHO), United Nations Children’s Fund (UNICEF), or other international donors; and  Logistical support to support various workshops associated with the conception, elaboration, revision, and validation of the protocols. 23 SOW Key Question 1: Are the health component activities being implemented according the Detailed Implementation Plan (DIP)? What are the obstacles and the delays observed? What recommendations can be done for future programs? 24 SOW Specific Objective 1: Assess the overall achievements of the health component of the program (IR1). Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 17 Table 2.2. Technical and Logistical Support Provided by the Tubaramure Program for the Revision of the Major Protocols that Supported the MoH’s Nutrition Strategy, 2009-2014 Major MoH Protocols to Which the Tubaramure Staff Contributed Technical Support and Period Provided Logistical Support for Workshops at Different Level of the MoH Clinical-Level Trainings IMCI or Prise en Charge Intégrée des Maladies de l’Enfance (PCIME) 25 2009-2012 2010-2012 (national and provincial) CMAM or Prise en Charge Communautaire de la Malnutrition Aigüe (PCMA)26 2010 2010-2012 (provincial) GM or Suivi Promotion de la Croissance (SPC)27 Not yet completed 2010-2011 to review the draft (provincial) Prenatal consultation (CPN)28 CPN 2011 2010-2012 (national and provincial) Postnatal consultation (CPoN)29 CPoN 2012 2010-2012 (national and provincial) Community-Level Trainings IMCI/CMAM/GM/CPN/CPoN (community based) 30 2010 (module in Kirundi) 2010 (provincial) The Integrated Manual for Community Health Workers (Manuel Intégrer des Agents de Sante Communautaire) 31 2012 -2013 2012-2013 (provincial) Management Trainings Data management and reporting N/A May 2011 Source: IMC Country Director Basile Ndumbi, IMC Program Manager Hervé Ketsebou, and IMC Site Manager Jean Paul Cubaka; August 5, 2014. In addition to the four core areas—IMCI, CMAM and severe acute malnutrition (SAM), GM, prenatal consultation/postnatal consultation (CPN/CPoN)—the MoH organized complementary trainings at the request of the MoH provincial offices on:  The Manuel Intégré des Agents de Santé Communautaire, popularly known in Kirundi as the Guide Dagadaga, for the 534 existing and 633 new MoH Community Health Workers recruited in 2012-2013; 32 and  The Programme National Intégré d’Alimentation et de Nutrition (PRONIANUT) manual on management and reporting to help guide the provincial-level health officials in 25 Training manual used for clinical-level training of doctors and nurses: MSPLS; 2012. Prise en Charge integrée des maladies de l’enfant (PCIME) Livret des Tableaux. Bujumbura: IMC for Tubaramure. Training manual used for training of the community-level workers (MoH Public Health Technicians and Community Health Workers, and Tubaramure Health Promoters): MSPLS.2010. Livret des tableaux Dagadaga à l’usage de l’agent de santé communautaire. Bujumbura: IMC for Tubaramure. 26 Training manual used: MSPLS; 2010. Protocole national de prise en charge integrée de la malnutrition aigüe. Bujumbura: IMC for Tubaramure. 27 Protocol is in the process of being revised. 28 Training manual used: MSPLS; 2011. Module de formation des prestataires sur la consultation prénatale recentrée. Bujumbura: IMC for Tubaramure. 29 MSPLS. 2012. Module de formation des prestataires en santé sur la consultation postnatale. Bujumbura: IMC for Tubaramure. 30 Training manual used: MSPLS; N.D. IMCI/CMAM/CPN/CPoN community based. Bujumbura: IMC for Tubaramure. 31 Training manual used: MSPLS; 2012. Manuel Intégré des Agents de Santé Communautaire. Bujumbura: IMC for Tubaramure. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 18 backstopping the head doctors (médecins chefs) in the newly created health districts in 2011. 2.1.2. Training of Trainers. The program used the MoH trainers to conduct the trainings both at the district and provincial levels. Prior to each training on each protocol, referred to as ‘activities’ in the health centers, the Tubaramure Program trained at least one trainer on it. Some of these training-of-trainer sessions were organized at the national level; others were organized in the two provinces. Most of the trainers were doctors, though a number of district and provincial-level MoH supervisors were also trained as trainers. Most training-of-trainer sessions were conducted one to two weeks before the training sessions started. The length of training varied according to the theme. 2.1.3. Clinical Trainings. The Tubaramure Program document anticipated two cycles of training over the course of the program: in 2010-2011 and 2012-2013 (Annex VI.A.3). Each cycle included two sessions during which one person from each health center was trained on all four core areas that the program supported—IMCI, CMAM, GM, CPN/CPoN. This planning was respected for all of the protocols except for GM, where the course was offered only in the first session since the MoH protocol was never validated. Based on feedback from the MoH in conjunction with the preparation of the program’s exit strategy, a third cycle of CMAM training33 was offered in 2014. 2.1.4. Community-Level Trainings. All community training followed a cascade model in which the Tubaramure trained the MoH Pubic Health Technicians and local managers for each health center (les Titulaires). The training was conducted by MoH trainers from the PRONIANUT office. Each Public Health Technician was then expected to train the Community Health Workers through a series of community-level trainings. In addition to this cascade training on the basic protocols, each of the Public Health Technicians and Tubaramure Health Promoters were trained on how to use a combined module—which is commonly referred to as Guide Dagadaga in Kirundi (known officially as the IMCI/CMAM/GM/CPN/CPoN module) or the Manuel Intégré des Agents de Santé Communautaire34—that summarized all the community-level nutrition activities being conducted in the communities. Each MoH Public Health Technician was then expected to continue to train and retrain the Community Health Workers that he or she supervised in conjunction with the regular meetings they held with the volunteers at the local health centers. In 2012, the MoH increased the number of Community Health Workers from 534 to 1,167. Once this happened, both MoH provincial offices asked the Tubaramure Program to help train the health workers on the newly validated Dagadaga. 2.1.5. Pre and Posttest and Final Report. All trainings included pre and posttests. All pre and posttest scores were monitored by the monitoring and evaluation (M&E) officer of the Tubaramure technical lead for IR1. In addition, a separate report was generated on each training session by the technical lead for IR1. Based on the raw data on individual pre and posttests 33 One nurse and one support staff member. 34 MoH; 2012. Manuel Intégré des Agents de Santé Communautaire. Bujumbura: IMC for Tubaramure. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 19 for 106 participants in one category of training (CPN and CPoN) that the IR1 leads felt was representative of the others, the evaluation team calculated that the average student score increased from 49% (out of 100) at baseline to 87%; and that 78% of the trainees scored higher than 75% (Figure 2.1). Figure 2.1. Average Scores for Participants on Tubaramure-Sponsored Pre and Postnatal Training Sessions Methodology: Based on the scores recorded for pretest and posttests of 106 trainees from sessions organized in Ruyigi and Cankuzo in September 2010, January 2011, February 2011, and March 2011. Source: Analysis by Sidibe Sidikiba from data provided by IMC; October 15, 2014. 49% 87% 78% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Avg. Pretest Score Avg. Posttest Score % of Staff Posttest Scores ≥ 75% Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 20 2.1.6. Total Number of People Trained. This system of training enabled the MoH to train 1,167 people at every level of the system (Table 2.3). Table 2.3. Number of People in Different Target Groups Trained by the Tubaramure Program in Different Protocols, 2009-2014 Training Area Category of Participants Total Number Trained (2009-2014) IMCI clinical Doctors 9 Nurses 250 Medical assistants 0 CMAM Nurses 229 Doctors 8 Medical assistants 215 GM Doctors 0 Nurses 105 Medical assistants 93 CPN/CPoN Doctors 0 Nurses 236 Medical assistants 0 IMCI/CMAM/GM/CPN/CPoN (community-based) Doctors 6 Nurses 9 MoH Public Health Technicians 23 Tubaramure Health Promoters 35 MoH Community Health Workers 534 Management and reporting Health information systems agents (SIS agents) 9 Supervision Doctors 20 Nurses 20 Integrated Manual for Community Health Workers Nurses 58 MoH Public Health Technicians 26 MoH Community Health Workers 1,167 Source: IMC Program Manager Hervé Ketsebou and IMC Site Manager Jean Paul Cubaka; August 5, 2014. 2.1.7. Link Between Training and the Program’s IR2 Activities. Most of the Tubaramure Health Promoters who were hired before 2013 were trained in all four modules as well as the Care Group Model (see Chapter 3). It was expected that they would then train the Tubaramure Leader Mothers (LMs) as well as the MoH Community Health Workers in the local communities. In June 2014, the program conducted the baseline training of all the MoH Public Health Technicians who supervised the Community Health Workers. The technicians were trained in the IR2 modules as they rolled out. Given the high rates of turnover in MoH staff, the program conducted a single comprehensive training on the five IR2 modules in a joint session with the Tubaramure Health Promoters as part of the program’s closeout plan in July 2014.35 36 35 All of the Tubaramure Health Promoters had extensive baseline training and retraining on the modules earlier in the program. 36 Since the five modules were not validated at the national level by the MoH as official training documents for the MoH Community Health Workers, they were not part of the standard approved (i.e. ‘validated’) set of modules for the MoH public workers during the program. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 21 2.2. Material Support From the beginning, the Tubaramure Program anticipated donating medical and non-medical equipment and supplies to each of the new and established health centers (centres de santé or CDSs). Each center was asked to identify their priority needs within a certain budget limit. Staff encountered during the final evaluation reported that the equipment helped them improve service quality.37 The impact of this material support was especially pronounced in the new centers, many of which were more-or-less non-functional before the support was given (Table 2.4). 2.3. Monitoring and Supervision 2.3.1. Supervision Missions. Two types of supervision missions were planned to strengthen the program’s clinical and community-based programs:  Supportive supervision missions: Each IR1 provincial coordinator was expected to conduct a routine supervision mission in each of the centers twice a month; and  Joint supervision missions: Each quarter, the IR1 provincial coordinator organized joint supervision missions with the district and provincial-level MoH doctors to every health center in their jurisdiction. Several national-level health specialists in the MoH participated in these missions. Table 2.4. Qualitative Assessment of the Early Impact of PM2A Donations to Health Centers in Cankuzo and Ruyigi Provinces Sanitary Infrastructure Total Supported Rendered Functional Due to Support Where Support Helped Considerably Improve Quality of Health Services Cankuzo CDSs in 2009 23 23 3 23 CDSs in 2014 28 28 8 28 Ruyigi CDSs in 2009 27 27 All were already functional 27 CDSs in 2014 33 33 All were already functional 33 Source: IMC Country Director Basile Ndumbi, IMC Program Manager Hervé Ketsebou, and IMC Site Manager Jean Paul Cubaka; August 5, 2014. During the final-evaluation key informant interviews, most staff stated that the interviews were helpful at two levels (Annex VI.A.4):  They enabled the staff and IR1 technical advisors to correct certain errors in the application of the revised protocols that were being taught during the training exercises; and  Local supervisors participating in the supervision missions helped involve them in the execution of the activities being supported under each protocol. 2.3.2. Assistance with the Creation and Analysis of Indicators. To strengthen the reporting on the MoH’s protocols, the program organized a series of trainings on data management and reporting 37 Given the expected wear and tear on the equipment, the program planned to re-equip all of the centers with new material to ensure that all of it was operational when the program ended. This occurred in August 2014. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 22 in 2010. In addition to this training, Tubaramure provided short-term technical support to the provincial MoH offices and health districts for the elaboration of routine reports on nutrition. This assistance was greatly appreciated by some of the staff that the evaluation team interviewed, who stated that it helped them better understand some of the new indicators and reporting forms (Text Box 2.1).38 There is also a great deal of qualitative evidence from the interviews with senior MoH staff at the district, provincial, and national level that this training increased the quality of the data received and the speed with which the MoH was able to report on some of the required indicators for nutrition. Text Box 2.1. Lesson Learned: Critical Importance of Building MoH Capacity for Nutritional Data Collection and Analysis—a Win-Win for Program Planning During the first year, the Tubaramure staff realized that the MoH service providers had very little background in either the collection, analysis, or reporting of MoH’s required indicators for nutrition. Since 2010, the local MoH health centers have been required to report on these indicators. To overcome this situation, the Tubaramure Program trained all of the district and provincial-level health information system agents in the collection, analysis, and reporting of nutritional data. The trainers were staff from the national MoH PRONIANUT office that backstops all of the MoH’s nutrition programs. It was anticipated that the health information systems staff would then train the health center providers with technical backup from the Tubaramure IR1 staff during their routine monthly supportive supervision and quarterly joint coordination missions. These combined efforts helped to improve the quality of the collection and interpretation of the nutrition indicators at the level of the health centers, health districts, and provinces. Since some of the same indicators were being reported by the national government as part of their internal MoH and performance-based financing (PBF)39 indicators, this assistance helped strengthen the provincial and district-level interest in executing and sustaining the Tubaramure activities in the health centers. Source: IMC Program Manager Hervé Ketsebou; August 2014 2.4. Coordination Meetings The program was instrumental in helping the MoH start a number of inter-agency coordination meetings through IR1 support. One that was especially important was a quarterly meeting of all the health sector actors, including all of the ‘vertical’ and NGO40 health and nutrition actors in 38 SOW Key Question 6: How the mid-term evaluation recommendations were taken into account for improving the program implementation? 39 In 2005, the Government of Burundi (GoB) decreed that all health services for children under the age of 5, pregnant women, and women with postnatal complications linked to pregnancy should be free. There were, however, a number of problems linked to delays in reimbursing the health structures (for example, overbilling for medical services, overburdened health personnel with decreased motivation to provide good services, and lack of medicines and equipment), all of which delayed the effective implementation of this government initiative. To deal with the difficulties, the MoH decided to fund a performance-based financing (PBF) system for financing the free health package that linked funding to performance. The PBF has thus become a system for funding health services that is based on results and a contractual relationship between different actors in the GoB’s health services. Health centers are reimbursed based on their performance on a group of pre-determined indicators. This system was pilot tested in three provinces in 2006, then extended to the entire country after April 1, 2010. 40 NGOs in Cankuzo province include: World Vision, Federation Lutherienne Mondaile (FLM) at Cankuzo, and Catholic Organization for Relief and Development Aid (Cordaid); NGOs in Ruyigi province include: International Red Cross (IRC), Action Aid, Food for the Hungry (FH), Catholic Relief Services (CRS), Croix Rouge, Maison Shalom, and Swaa Burundi. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 23 the province, as well as the governor or his representative and the local authorities. These quarterly meetings have met regularly since the second year (Annex VI.A.5). Given the critical importance of these meetings in facilitating the coordination between different health sector actors, the Tubaramure IR1 exit plan started a progressive phase out of its support to these meetings in June 2013, in which one of the other sector actors (like World Vision) would be asked to support the costs of the meeting for every two meetings that Tubaramure supported. In 2012, the MoH, with Tubaramure support, started two additional coordination meetings designed to strengthen the province’s community-level health in nutrition services (Annex VI. A.5):  A quarterly community-level health coordination meeting that brought together the different government and NGO actors (including Tubaramure) working on community￾based health approaches; and  A monthly commune-level meeting during which the MoH Community Health Workers were expected to present their reports to their supervising MoH Public Health Technician. The evaluation team met with governors and most administrators, who cited coordination meetings as an example of a best practice that helped improve the coordination between the different sector actors. The coordination meetings were attended by people at every level of the public health service, and by the most active local and international NGOs that worked on health and nutrition programs. The same meetings helped to clarify different aspects of the Tubaramure Program, including the progressive graduation of the beneficiaries and clarification of various issues—like why some of the test villages in the International Food Policy Research Institute/Food and Nutrition Technical Assistance Project (IFPRI/FANTA) study were not eligible for food and why the district-level ambulances might sometimes be needed to transport a severely malnourished child to a stabilization center (Text Box 2.2). The evaluation team also noted that the quarterly meetings held with the support of the program allowed the various stakeholders in the health and nutrition programs to discuss the health problems and to find possible solutions together (Text Box 2.2). Given the importance of these meetings, the Tubaramure Program exit strategy planned for a withdrawal of its logistical and technical support for the meetings. Since July 2014, one of the other area donors (like World Vision) has supported one of the quarterly meetings for every meeting funded by Tubaramure. Unfortunately, the commune-level meetings stopped in January 2014, since no donor is willing to cover the costs. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 24 Text Box 2.2. Lesson Learned: Case Studies of How the Tubaramure Program’s Support to the Quarterly Coordination Meetings Helped Increase Program Efficiency, Impact, and Sustainability 1. Improved coordination of activities with the program: During one quarterly planning meeting, the health authorities and the participants realized that both World Vision and Tubaramure (IR1) were planning to support community-based monitoring in the same colline (community). Based on these discussions, Tubaramure decided to move their community-based GM to another colline. Given the progressive scale up of community￾based GM programs to sustaining the Tubaramure achievements, this was an example of good coordination in the field that avoided duplication of activities. 2. Better control over fraud: In the meeting discussions, the participants identified a number of cases where the local MoH Public Health Technicians had committed fraud during the 2013 recruitment campaign to increase the number of Community Health Workers. Based on the evidence that was presented during the meetings, the offending Public Health Technicians were transferred and the process for recruiting the Community Health Workers was re-opened in the affected collines. This type of transparent nomination of the Community Health Worker volunteers is critical to sustaining the Tubaramure Program’s activities, especially those of the LMs, who were often applying to be Community Health Workers. 3. Improved access to critical complementary support like ambulances: The transfer of patients across the province for emergency treatment is always a challenge, especially in terms of cost for the operation of the ambulance. For this reason, some emergencies—like those related to transporting malnourished children—were not eligible for this support when the program started. One output of the quarterly meetings was to build the local authorities’ understanding of local needs. In February 2012, a decision was made that each household would contribute to a commune-level fund that would pay the cost of emergency transport for at-risk children as well as other urgent cases in a timely manner. Source: Jean Paul Cubaka; August 2014; and IMC Program Manager Hervé Ketsebou; February 2012. Rapport de la réunion de coordination des partenaires en santé. Bujumbura: IMC for Tubaramure. Pg 6. 3.0. Evidence of Results 3.1. Output 1.1: Pregnant and Lactating Women Access Quality Pre and Postnatal Services41 The Tubaramure IR1 strategy for increasing pregnant and lactating women’s access to quality pre and postnatal services focused on increasing the quality of the prenatal consultation services by:  Improving service quality;  Increasing the demand for women’s understanding of why prenatal consultations are important through the IR2 BCC activities; and  ‘Jumpstarting’ women’s willingness and ability to attend the government’s recommended package of prenatal and postnatal consultations a requirement for remaining eligible for PM2A food rations. 3.1.1. Activities (2010-Mid-Term). By mid-term, about half of the total number of health centers had at least two staff members who had completed the pre and postnatal program in-service training, and that number remained constant even with high levels of staff turnover. One direct impact of the high rates of MoH staff mobility and the limited opportunities for training new staff (since the training was only offered twice during four years) was that the total number of health centers never achieved the program target of 75% trained (Table 2.5). 41 SOW Key Question 3: What interventions have been more or less successful in meeting targets? Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 25 Table 2.5. Tubaramure PM2A Program Indicators Used to Track MoH Capacity and Community Demand for Pre and Postnatal Consultations, 2010-2014 IR1 Key Program Indicators Baseline* 2010 Mid￾Term 2012 Endline 2014* Life of Activity (LOA) Target* Output Indicators 1.5. Percentage of women registered for prenatal service by the sixth month of pregnancy 80.1* N/A 95.1* 95.0 1.6. Percentage of health facilities with two or more staff who completed in-service training in pre/postnatal services 0.0 46.0 57.0 75.0 Outcome Indicators 1.1. Original Indicator: Percentage of women completing the package of four prenatal visits 28.9 64.1** 50.3 98.0 1.1. Mid-Term Revision of Indicator: Percentage of women completing the package of three prenatal visits N/A 94.0** 83.4 98.0 1.2. Original Indicator: Percentage of women completing the package of three postnatal visits 5.0 19.1** 3.1 75.0 1.2. Mid-Term Revision of Indicator: Percentage of women completing the package of two postnatal visits N/A 33.2** 11.1 75.0 *Population-based (i.e. measured in the baseline household survey and final) **For PM2A beneficiaries only. Source: Tubaramure Indicator Performance Tracking Table (IPTT) (Annex II) and Institut de Statistiques et d'Etudes Economiques du Burundi (ISTEEBU); 2014. Enquête Ménage pour l’Evaluation Quantitative Finale du Programme PM2A-Tubaramure (Provinces Cankuzo et Ruyigi). Rapport Final. Bujumbura: ISTEEBU (August 2014). 3.1.2. Mid-Term Evaluation Recommendations. The mid-term evaluation report found that: “Despite increased efforts, only 61% of pregnant women are completing four prenatal visits, and only 12% are completing three postnatal visits. [Based on this information] a suggestion was made by the Tubaramure team to lower the end of program targets. [However, the mid-term evaluation team concluded] it is better to keep the targets high (i.e. at 98% for prenatal consultations and 75% for postnatal consultations) and revert instead to the original indicators of three prenatal and two postnatal visits.”42 In response to one of the mid-term recommendations, the program conducted a barrier analysis in conjunction with its routine performance evaluation to see why the rate of prenatal and postnatal consultations was not increasing.43 This study made seven recommendations to the program for increasing the rate of prenatal and postnatal consultations (Text Box 2.3).44 In response to these recommendations, the program worked with the MoH to intensify the existing efforts to promote pre and postnatal consultations. These efforts, which were often commented on in the key informant interviews during the final evaluation, included: 42 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure (July 2012). Pg. 14. 43 Tubaramure Equipe de Suivi Evaluation ; 2012. Etude sur les Obstacles à l’Adoption des Bonnes Pratiques en Matière de Santé, Nutrition et Hygiène, dans les provinces de Ruyigi et Cankuzo. Rapport Final. Bujumbura: Tubaramure M&E Office (December 2012). Pp. 6-8. 44 The percentage of women who complete the recommended package of prenatal consultations is one of the PBF indicators, so it is one that is monitored very closely by the national MoH. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 26  A concerted effort by the MoH to encourage all the supervising head doctors to become directly involved in helping their clinic and community-level staff to understand the critical importance of pre and postnatal consultations;  An effort by the program to get all of the civil authorities (at the province, commune, and colline levels) involved in promoting pre and postnatal consultation;  An intensified discussion of pre and postnatal consultation in the Care Group training sessions that included specific guidelines for the promotion of pre and postnatal consultation by the LMs in the beneficiary groups they supported; and  On-site promotions by the Tubaramure Health Promoters at the food distribution sites. Text Box 2.3. Principal Recommendations of the 2012 Study on Obstacles to the Adoption of Good Practices on Health, Nutrition, and Hygiene for Increasing the Rate of Pre and Postnatal Consultations Recommendations for increasing the rate of pre and postnatal consultations:  Strengthen the development of more innovative public awareness programs on CPoN. It is important for women to dispel the common belief that they must take off all their clothes during a post-natal consultation.  Replicate the types of beneficiary level public awareness campaigns that are done for CPN for CPoN. The Community Health Workers conduct intensive public awareness campaigns for CPN but not for CPoN. These public awareness programs need to be added to the work program for the Community Health Workers.  Involve the commune-level administrators in order to ensure that these public awareness activities are sustained after the program ends.  Improve the quality of the CPoN services in the MoH health centers. The Health Centers need to be more user friendly. They need to encourage women to participate in the CPoN activities. It is imperative that each CPoN have a strict protocol that clarifies the required number post-natal visits that a woman should have.  Strengthen the husbands general understanding of the CPoN activities so that they understand them better and are involved in supporting them.  Encourage the mothers to take the recommended dosage of Vitamin A and iron supplements. Most important, encourage all women coming in for CPoN activities to use mosquito nets.  Lobby the MoH to make the CPoN activities free. Source: Tubaramure M&E Office; 2012. Etude sur les Obstacles a l’Adoption des Bonnes Pratiques en Matière de Santé Nutrition et Hygiène dans les Provinces de Ruyigi et Cankuzo. Bujumbura: Tubaramure M&E Office. 3.1.3. Results. 3.1.3.1. Percentage of Women Completing Four Prenatal Visits. The national MoH protocol recommends pregnant women complete four prenatal visits. The final quantitative survey showed a 21-point increase in this indicator (population-based result) that was statistically significant (p = 0.000) (Table 2.4). The same study showed the increased the number of mothers with children under 2 years of age who are able to recognize at least two of the four danger signs for pregnant women to 71.3% (population-based result). 45 3.1.3.2. Percentage of Women Completing Three Prenatal Visits. The number of women who completed the recommended number of prenatal visits is even higher if one measures the number 45 Indicator 2.6; Annex II. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 27 of women who completed at least three prenatal visits—which was the recommended revision of this indicator, 83.4% (population-based result) (Table 2.5). This quantitative data from the final household survey concurs with the qualitative information from the focus group discussions, which suggest that (Annex VI.A.6; Annexes IV and V):  The PM2A women graduates are motivated to continue attending the prenatal consultations at the health centers for new pregnancies even after graduation; and  Non-beneficiary mothers (i.e. women who were not eligible to receive PM2A rations during the program) have also increased their attendance at prenatal consultations. 3.1.3.3. Percentage of Women Giving Birth in Health Center. The additional women coming to the health clinics for prenatal consultation increased the percentage of pregnant women choosing to give birth in the health centers (Figure 2.2). This data was confirmed by the health workers’ qualitative assessments as well as the focus group discussions with both beneficiary and non￾beneficiary women (Annexes IV and V). One focus group discussion in the Nyamasenga colline in Ruyigi summarized this,“Women now do pre and postnatal consultations. This was not done before the program. Before, most mothers would deliver their babies at home. Many women suffered from fistulas and ignored why. Now most of the women deliver in health facilities because they are encouraged to during their prenatal consultations. Because of this, there are not as many cases of fistulas, and if there is one, the woman receives timely care. We have this knowledge because of the training that the program developed for us.” 3.1.3.4. Percentage of Women Completing Recommended Number of Postnatal Visits. In contrast, the percentage of women who completed the recommended number of postnatal consultations remained very low, at 11.1% (Table 2.5). One of the most important reasons reported by the MoH and program staff is that postnatal consultation is not one of the top MoH priorities, either in terms of national health strategy or the indicators being tracked. This issue was confirmed in the focus group discussions with both beneficiary and non-beneficiary women (Annexes IV and V).  In Kigamba colline, for example, the women said that none of the health agents they encountered when giving birth emphasized the importance of returning for any sort of postnatal consultation: “After giving birth, none of the delivery agents told us to return for any postnatal consultation; even the LMs didn’t emphasize this;”  Some women said that when they came to the health centers for postnatal care, they were made to wait for hours while the nurses dealt with more urgent curative consultations. Many of the mothers also complained that they only received advice with out any additional medication, food, vitamins, or other support; and  Many of the women interviewed stated that if they felt they were in good health after childbirth and they did not see any need for a postnatal consultation. And if there was no apparent reason to go (in terms of rations or ill health), their husbands would not like the idea of them taking the time off from field and house work to attend the postnatal consultation. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 28 Figure 2.2. Percentage of Women Having Assisted Births in Health Centers Source: ISTEEBU; 2014. Enquête Ménage pour l’Evaluation Quantitative Finale du Programme PM2A￾Tubaramure (Provinces Cankuzo et Ruyigi). Rapport Final. Bujumbura: ISTEEBU (August 2014). Pg. 34. 3.2. Output 1.2: Implementation of National IMCI Plan is Supported46 CRS’s previous MYAP suggested that children were frequently suffering bouts of diarrhea, acute respiratory infections, and malaria, which contributed to the burden of malnutrition. To address this issue, the Tubaramure IR1 strategy focused on:  Increasing the number of MoH staff trained in IMCI;  Training at least 50 LMs together with the Community Health Workers in Community￾Based Integrated Management of Childhood Illness (C-IMCI); and  Reinforcing the potential for the MoH staff and LMs to working together. It was expected that the LMs trained in C-IMCI would refer children to a facility before an illness led to more severe malnutrition requiring in-house hospitalization or rehabilitation. 3.2.1. Activities (2010-Mid-Term). By mid-term, about half the health centers (46%) had at least two staff members trained in the new IMCI protocols, and 100% of the Community Health Workers on a simpler version of the protocols (Table 2.6). Although this represents a 31-point increase over the baseline, it was less than 50% of the original mid-term target for this indicator, which was 100% (Annex II). The program’s below-average achievement on this indicator was attributed to the progressive installation of new health centers, understaffing of the new health centers, and high rates of staff turnover. Although the mid-term evaluation report showed that the program had successfully increased the accuracy with which the health centers were diagnosing and treating children under 5 (Table 2.6), there was concern about the way this indicator was being measured. 46 SOW Key Question 3: What interventions have been more or less successful in meeting targets? 44% 58% 53% 78% 75% 76% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Cankuzo Ruyigi Total Baseline 2010 Endline 2014 Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 29 Table 2.6. Evolution of Key Indicators Tracking MoH Capacity for IMCI Support at the Clinic and Community Level, 2009-2014 IR1 Key Program Indicators Baseline 2010 Mid-Term 2012 Endline 2014 LOA Target Output 1.2. Implementation of national IMCI plan is supported Output Indicators 1.7. Percentage of health facilities with two staff members trained in IMCI protocol through MoH's IMCI office 15.0 46.0 72.0 100.0 1.8. Percentage of collines with two or more Community Health Workers trained in C-IMCI in target areas through MoH's IMCI office 0.0 100.0 100.0 70.0 Outcome Indicator 1.4. Original Indicator: Percentage of health providers (facilities/ CHW) accurately assessing a child using IMCI protocols 0.0 55.0 48.0 80.0 1.4. Mid-term Revision of Indicator: Percentage of nurses accurately diagnosing and treating children under 5 0.0 N/A 100.0 80.0 Source: Tubaramure IPTT (Annex II of this report). 3.2.2. Mid-Term Evaluation Report Recommendations (2012). The mid-term evaluation report recommended changing Indicator 1.4 to,“Measure the nurse’s ability to diagnose and treat children under 5 correctly” by either performing a test (similar to the posttest used during the training modules) at semi-annual intervals or looking at the number of nurses who have correctly completed their five algorithms out of all the nurses in the 50 clinics. The mid-term evaluation report also recommended the program, “See whether the MoH’s IMC office provides training through the nursing schools and whether they might need support from the Tubaramure Program” as a mechanism for increasing the supply of nurses with the necessary skills that the MoH needed to support its newly expanded nutrition programs.47 3.2.3. Results. 3.2.3.1. Percentage of Nurses Accurately Diagnosing and Treating Children Under 5. To measure the impact of the MoH IMCI training programs, the IR2 technical lead developed a tool that it used to measure the accuracy with which the health center-based nurses were able to use the standard ‘algorithms’ (checklists) in the protocol.48 Based on this indicator, all of the nurses interviewed were familiar with and able to use the algorithms by the end of the program, which is a significant over-achievement of the original target of 80% (Table 2.6). This information from the official tracking system corroborates the qualitative data from the key informant interviews with the nurses. Most of them stated that the basic training and supervision services they received under the program increased the quality of the diagnostic and treatment services they were able to offer (Text Box 2.4). The nurse’s chief complaint (from the key informant interviews) was that they did not receive a certificate to recognize their participation in the formal training sessions. 47 Kathy Tilford, Ange Tingbo and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure: Pp. 27 and 20. 48 This tool measured the competencies of a random sample of nurses—both those who had received the Tubaramure-sponsored training and those who had not. When asked whether they felt the measurement tool presented a very accurate assessment of their competencies, both the MoH and IMC staff responded, “yes.” This impression was further validated by the health and nutrition specialist on the final evaluation team. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 30 Text Box 2.4. MoH Staff Members’ Perceptions of the Impact of the Tubaramure Program on Their Capacity and Service Delivery “Program officers through IMCI often came to work with us to support us in the work of prenatal consultation and prevention of malnutrition. We have received material (such as [mid-upper arm circumference] MUAC, food and recipe posters, flip charts, and a scale) through the PM2A program, which we use to monitor the growth of children and the weight of the pregnant women. Before, the nurses had no training on clinical IMCI. Thanks to the training received, we have become very knowledgeable and the quality of service and care have been greatly improved.” Manager for the Nyagutoha Health Center. “Before Tubaramure, I didn’t have enough training in nutrition, especially for identifying malnutrition and treating it. After the training organized by our partners (IMC) and their supervision missions, we have been able to improve our center’s systems for identifying malnourished children and referring the most severe cases to the stabilization centers for treatment. For moderate cases, we try to build the mother’s awareness of the Essential Nutrition Actions (ENA) that we promoting with help from the Tubaramure Program.” Health staff member at the Ruyigi Health Center. Source: Tubaramure Final Evaluation focus group discussions; July 2014. 3.3. Output 1.3: Health Facilities Supported in Providing GM49 50 When the Tubaramure Program started, only about 40% of children under 5 participated in GM activities, and there were no community-based growth monitoring programs. The Tubaramure consortium strategy for growth monitoring focused on:51  Helping the MoH revise and finalize its 2009 draft protocol of facility-based GM;  Supporting the MoH’s “anticipated roll out of community-based growth monitoring;”52 and  Incorporating key messages around the health center-based GM into the LM training under IR2. 3.3.1. Activities (2010-Mid-Term). Since the MoH never completed the projected revision of the draft protocol, the Tubaramure Program’s activities focused on encouraging the active participation of mothers in the center-based GM program by training staff in improved techniques for measurement; upgrading the GM equipment for the centers (Table 2.7); and promoting the concept of GM through the community-based behavior change communication (BCC) programs. Since participation in the center-based growth monitoring was a condition for getting PM2A rations, it is not surprising that the program’s internal performance study showed an immediate and direct impact in percentage of beneficiary children53 0-36 months who attended growth 49 SOW Key Question 3: What interventions have been more or less successful in meeting targets? 50 SOW Specific Objective 6: Evaluate the relevance, the effectiveness, and the performance of the implemented growth monitoring system, with regards to the national protocol. 51 CRS; 2009. Tubaramure MYAP Proposal. Bujumbura: CRS for Tubaramure. Pp 10-11. 52 CRS; 2009. Tubaramure MYAP Proposal. Bujumbura: CRS for Tubaramure. Pp. 10. 53 The 2010, 2011, 2012, and 2013 figures in the IPTT for this indicator track the attendance of the PM2A beneficiary children in GM; they are not population based. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 31 monitoring in a two month period (as recorded on a card): from 16% at baseline to 66% in 2012 at mid-term. The mid-term target was 75% (Annex II, Outcome Indicator 1.3). Table 2.7. Tubaramure PM2A Program Indicators Used To Track Clinical and Community-Based Capacity for Growth Monitoring, 2009-2014 IR1 Key Program Indicators Baseline 2010 Mid-Term 2012 Endline 2014 LOA Target Output 1.3. Health facilities supported in providing GM. 1.9. Number of communes with at least one functioning community-based GM center after Year 3 0 6 12 12 1.10. Percentage of health facilities with upgraded GM equipment 0.0 100.0 56.0 75.0 Source: Tubaramure IPTT (Annex II). 3.3.2. Mid-Term Evaluation Recommendations (2012). One of the key concerns of the mid-term evaluation team was whether or not the beneficiary and non-beneficiary mothers would continue to bring their children in for GM once they were no longer eligible for PM2A rations. This was a legitimate concern since some of the mothers quit attending the beneficiary group meetings and Care Groups after they were no longer eligible to receive PM2A rations. To facilitate the transition and the program’s endline target of 90%, the evaluators made two recommendations, both of which were implemented by the program after the mid-term:54  Combine the health center-based GM with immunization days; and  Execute all 12 pilot community-based GM programs that were envisioned in the original Tubaramure MYAP proposal. Starting in 2012, the Tubaramure Program worked through the MoH Public Health Technicians to develop a pilot program in community-based GM. No additional training was given to the Public Health Technicians before starting the community-based GM because the basic training in GM was part of the training that the Public Health Technicians and Community Health Worker had already received on the new integrated manual. The Tubaramure IR1 specialists did, however, try to supervise the execution of the GM sessions during their monthly supervision visits. 3.3.3. Results. 3.3.3.1. Percentage of Children 0-50 Months Attending GM. One major impact of the Tubaramure Program’s integrated55 program strategy for promoting GM was a very significant population-based increase in the percentage of children attending the health center-based GM system from 16% to 57.3%, even after the women were no longer eligible for PM2A rations.56 54 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure (July 2012). Pg. 29. 55 The final evaluation team has decided to use the term ‘integrated’ to highlight the fact that a series of program supports and activities were focused on achieving the same output. These included making growth monitoring a condition of receiving rations under IR3; promoting growth monitoring through the BCC programs under IR2; and building MoH support for the activity under IR1. 56 This represents an increase of 41.3 percentage points compared to the baseline (2009), giving a very statistically significant increase of over 258% (p = 0.000). Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 32 Although this represents a significant increase, it is still far less than the original Life of Activity (LOA) target of 90%. This is in large part because:  The government never finalized its protocol for community-based GM; and, as a result,  The concept of community-based GM (which Tubaramure supported) was never scaled up to all of the collines with appropriate support from the MoH and other donors. 3.3.3.2. Pilot Program for Community-Based GM. Increased Referrals. The short-term impact of 12 pilot programs for community-based GM was a substantial increase in the number of children referred for either medical or health reasons from the 12 collines that were targeted by the activity. Most of the mothers in the affected collines who were interviewed in the focus groups stated that having community GM increased their willingness and ability to bring their children in regularly (Text Box 2.5.) Unfortunately, there was very little tracking data on the pilot study, other than what is presented in Table 2.8 and Figure 2.3, so it is difficult to document the actual recorded increase in the rate of enrollment that was associated with the community-based programs versus communities where all GM was clinic based. Figure 2.3. Summary Data Collected on the 12 Pilot GM Sites Created Under the Tubaramure PM2A Program, 2012-2014 Source: Table 2.8. 5283 3412 406 8357 4752 886 13640 8164 1292 0 2000 4000 6000 8000 10000 12000 14000 16000 Children Participating Children with Reported Weight Gains Children Referred to the Health Centers Cankuzo Ruyigi Total Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 33 Table 2.8. Summary Data Collected on the 12 Pilot Growth Monitoring Sites Created Under the Tubaramure PM2A Program, 2012-2014 Periods Province Data from the 12 Pilot Community-Based Growth Monitoring Programs Children Participating Children with Reported Weight Gains Children Referred to the Health Centers April-June 2012 Cankuzo 309 279 64 Ruyigi 469 403 13 Total 778 682 77 July-September 2012 Cankuzo 264 177 14 Ruyigi 403 261 39 Total 667 438 53 January-March 2013 Cankuzo 506 353 12 Ruyigi 730 291 140 Total 1236 644 152 April-June 2013 Cankuzo 1128 507 132 Ruyigi 2532 922 352 Total 3660 1429 484 July-September 2013 Cankuzo 818 815 58 Ruyigi 1584 1080 80 Total 2402 1895 138 October-December 2013 Cankuzo 807 447 18 Ruyigi 1077 762 48 Total 1884 1209 66 January 2014 Cankuzo 284 145 6 Ruyigi 271 183 14 Total 555 328 20 February 2014 Cankuzo 209 120 9 Ruyigi 251 187 12 Total 460 307 21 March 2014 Cankuzo 476 197 71 Ruyigi 288 192 14 Total 764 389 85 April 2014 Cankuzo 246 188 7 Ruyigi 346 211 114 Total 592 399 121 May 2014 Cankuzo 236 184 15 Ruyigi 252 146 39 Total 488 330 54 June 2014 Cankuzo 0* 0* 0* Ruyigi 154 114 21 Total 154 114 21 *The evaluation team was not able to explain why this data was a 0. Source: IMC, Burundi; August 20, 2014. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 34 Text Box 2.5. Testimonials from Mothers About the Impact of the Pilot Community-Based Growth Monitoring Program from the Final Evaluation Pilot Study Kirasira Village (Ruyigi) My name is Beatrice NIBIGIRA. I'm 30. I live in Kirasira Village, Butezi Commune in Ruyigi Province, and I am a mother of three daughters. I would like to give a testimony on the growth monitoring of my children. The first child was born before Tubaramure Program. She weighed 2.1 kg at birth; at 1 month, 2.5 kg; after 2 months, 3 kg; three months later, 3.5 kg. After 1 year, the child was 8 kg. I gave up monitoring the weight of the child after 1 year 6 months, because I did not know its significance. After Tubaramure implantation, I was pregnant with the second child who weighed 3.7 kg at birth; 4 kg after a month; 5 kg after 2 months; 6 kg after 3 months; and 7 kg after 4 months. A year after the birth, she had reached 10 kg. When I became a graduated woman, I gave birth to the third child, who weighed 3.6 kg at birth; 4.5 kg after 1 month; and 5.5 kg after 3 months. The child reached 8 kg after 1 year. The child did not increase in weight because of malaria. I continue to monitor my children’s growth because the second child, who is 4 years old, weighs 15 kg. I thank Tubaramure Program for its training on children growth monitoring because the weighing is a real indicator of children growth. Itahe Colline, Ruyigi I’m Gloriose Niyonkuru, aged 42. I live in Itahe Village, in Gisuru Commune of Ruyigi Province. I’m married with five children; four sons and one daughter. However, among the five children, I lost one. Before Tubaramure, children’s growth was not at all good in our community, starting with that of my children. Further to malnutrition, I gave birth to underweight children. Example: My first child was born with 2 kg; they told me that I had given birth to a mouse. The nurse demanded me to breastfeed the child for 6 months without giving him any other food. However, with housework that I had to do, I didn’t follow the advice. I started to feed him at the age of 4 months with foods for adults but his weight didn’t increase. Every time I went for his immunization, I noticed that the increase in weight ranged between 500 g and 1 kg. My second child was born with 2kg 100g. At one year, he had kwashiorkor57 due to malnutrition. I saw his belly swelling, and I thought he was suffering from intestinal worms. Unfortunately, he died at the age of one year and three months. The third child and fourth child were born with the same weight of 2 kg 200. I followed medical advice, but at 5 months, I had given them poor-nutrient foods. They were growing underweight; their weight ranged between 1 kg and 2 kg. My fifth child is a girl born when I was in Tubaramure. The program began when I was pregnant for two months. I was put in the program at 6 months. I followed the training program on good nutrition and health practices. I went for prenatal consultations and postnatal consultations; I ate three or four times a day. I was really fat, and my health had improved. I then gave birth to a child of 4 kg. I breastfed her for 6 months without giving any other food or another drink. I practiced growth monitoring every month. At 6 months, I began to give her porridge and some fruits. She had 7 kg 900 g. At 7 months, I began to give her food for adults; for example bean puree and potato. When I weighed her, she had 8 kg 500 g. I continued to feed varying the diet until now where she has 1 year 6 months; she weighs 13 kg. I appreciate the growth of my child through Tubaramure Program. If I had not participated in this program, my last child would not have had this good growth. 57 Kwashiorkor is a form of severe protein–energy malnutrition characterized by edema, irritability, anorexia, ulcerating dermatoses, and an enlarged liver with fatty infiltrates. Sufficient calorie intake, but with insufficient protein consumption, distinguishes it from marasmus. Kwashiorkor cases occur in areas of famine or poor food supply. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 35 Source: Tubaramure Final Evaluation focus group discussions and key informant interviews; July 2014. National Policy. Although the scale of the 12-colline pilot study was insufficient (only 12 collines out of 268) to have an important population-based impact on the entire province, the pilot study ended up having a more broad-based policy than the one that was intended in the original Tubaramure MYAP proposal. This impact occurred because the preliminary results of the pilot—i.e. the few results that were tracked (Table 2.8), as well as some of the individual case studies from specific collines—were presented at several national health workshops that were organized for other purposes. These presentations ended up convincing some of the senior MoH staff about the importance of moving forward with the community-based growth monitoring protocol that had been stalled for over five years.58 When asked to extrapolate some of the important lessons learned from the pilot studies for this broader policy debate, the Tubaramure staff and the national director of the MoH nutrition unit stated that the pilot studies:  Showed how having an effective network of community-based GM programs could dramatically increase the percentage of children that could be seen; and  Provided some of the first qualitative data to the provincial and national MoH offices about some of the key factors that would need to be addressed for these programs to be successful, including: - Increasing the number of Public Health Technicians and providing them with the means to access the villages for regular supervision missions (i.e. gas money, transportation and per diem); - Building a strong working relationship with the local authorities (both at the commune and colline levels) to promote their understanding of the process;59 - Anticipating the need for linking these programs to community-based programs (like Positive Deviance [PD]/Hearth or Foyer d'Apprentissage et de Réhabilitation Nutritionnelle [FARN]) that could help rehabilitate the large number of moderately malnourished children that a successful community-based GM program would likely identify (see Table 2.8);60 - Providing each colline with the materials that the Tubaramure Program provided them (e.g., scales, measuring tapes) needed to conduct GM, as well as appropriate training modules (like those developed by Tubaramure) for the Community Health Workers who would be in charge of the programs and their Public Health Technician supervisors; and - Creating a user-friendly system for tracking the effectiveness of the community-based GM that the Community Health Workers and their supervisors could use to report the 58 This type of policy impact was anticipated in the proposal as a support to the MoH’s ongoing discussions about the need to add a national protocol for community-based GM. In the original proposal, it was anticipated that Tubaramure would host a one-day workshop to discuss this issue. (See CRS; 2009. Tubaramure MYAP Proposal. Bujumbura: CRS for Tubaramure. Pp 9-12). 59 The Tubaramure IR1 staff stated that this was one of the key factors that distinguished the villages with the most active local participation and benefits from the pilot activity. 60 As discussed earlier in this chapter (Section 3.4), even the health center-based GM programs identified a large number of moderately malnourished children who were not eligible for the hospital-based SAM rehabilitation programs. Although the Burundi MoH health policy recognizes FARNs, they were not supported by the Tubaramure Program due to the program’s technical and research focus on preventing malnutrition, not treating it. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 36 results to the health centers to avoid the kind of under-documentation that occurred during the Tubaramure GM pilot activities.61 61 This under-documentation limited the program’s ability to provide meaningful data-based feedback on the comparative value of community-based GM versus clinic-based GM. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 37 3.4. Output 1.4: Severe Acute Malnutrition is Detected and Referred For Treatment62 63 When the Tubaramure Program started, the United Nations Children’s Fund (UNICEF) supported the MoH for commodities, medicine, and anthropometric equipment to local facilitates to support the scaling up of CMAM with Plumpy’Nut.64 For this reason, the Tubaramure strategy focused on:65  Developing a community-based referral plan for malnutrition;  Using mid-upper arm circumference (MUAC) to identify children with malnutrition through door-to-door campaigns or community-wide events such as fairs or market days;  Mobilizing and training LMs to engage in MUAC screenings with assistance of parents;  Using home visits to follow children after graduation;  Training Community Health Workers and LMs in C-IMCI and counseling skills; and  Training Community Health Workers and LMs on messages to prevent severe acute malnutrition (SAM) and chronic malnutrition. 3.4.1. Activities (2010-Mid-Term). The Tubaramure Program’s multi-pronged approach to building the local community’s capacity to identify and refer malnourished children resulted in a fairly immediate increase in the total number of referrals. Unfortunately, however, at the end of 2010 the World Food Program (WFP) started to close its nutritional supplement series in Cankuzo and Ruyigi because the rate of moderate and acute malnutrition had decreased to the point that that the MoH offices in these provinces were no longer eligible for the WFP rations for treating moderate and acute malnutrition. This shift in the macro-policy context of the SAM activities in both provinces had important implications for the children that the Tubaramure GM activities in the health centers identified as needing rehabilitation help. Specifically, the changes meant:66  Acute malnutrition with complications: Children suffering from acute malnutrition with complications got referred to the specialized centers (linked to a hospital) that benefitted from the UNCEF Plumpy’Nut support; and  Moderate malnutrition without complications: Unless they were in one of the few centers supported by the WFP, the children who were identified as moderately malnourished by the clinic-based GM program got sent home with no treatment other than the advice and counsel of the Community Health Workers and the LMs. Although the WFP reopened its support for moderate malnutrition treatments support in seven 62 SOW Key Question 3: What interventions have been more or less successful in meeting targets? 63 SOW Specific Objective 7: Assess the effectiveness and the performance of the detection and referral of SAM’s cases. 64 Plumpy'Nut is a peanut-based paste in a plastic wrapper for treatment of severe acute malnutrition manufactured by French company Nutriset. 65 CRS; 2009. Tubaramure MYAP Proposal. Bujumbura: CRS. 66 A total of 282,816 children were monitored by the clinical-based growth monitoring between January 2011 and May 2014. Many of them are double and triple counted in the total figure since they attended several growth monitoring sessions. These GM sessions identified 3,848 children as suffering from severe malnutrition. These children were referred to the nutritional rehabilitation centers backstopped UNICEF. Another 4,375 were identified as moderately malnourished and eligible for the WFP rations if the health centers had them. Once the health centers no longer had the rations, these sick children were sent back the village. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 38 health centers in Ruyigi in 2011, this did not cover the entire province. 3.4.2. Mid-Term Evaluation Report Recommendations (2012). One of the key recommendations from the Tubaramure mid-term evaluation report was that children with moderate or mild malnutrition are also identified, registered, and followed up in the community (Table 2.9) as originally envisioned in the Tubaramure PM2A MYAP proposal67 and reiterated as a critical constraint in the Tubaramure Program’s mid-term evaluation report. Table 2.9. The Original Tubaramure MYAP Plan For Treating Children With Moderate and Mild Malnutrition Whose Non-Realization Was Identified as a Critical Constraint to the Achievement of Output 1.4 in the Absence of Community-Based FARNs Presenting With Moderate or Mild Malnutrition, Not Receiving PM2A Supplements Presenting With Moderate or Mild Malnutrition Enrolled in Tubaramure (Mother or Child had Received PM2A Supplements) Children 0 to 24 months with moderate or mild malnutrition at start of Tubaramure (mothers were not pregnant or lactating at program start-up, therefore the children are not enrolled)  Detection  Home visits by Community Health Workers and MLs with specific messages for catch-up feeding  IMCI screening and referral to HF for other services including CMAM Children 0-24 months with moderate or mild malnutrition (mother started in Tubaramure at time of pregnancy)  Detection  Home visits by Community Health Workers and MLs  Use of Title II food - Use of additional foods during illness - Counseling on hygiene - Counseling on home management of illnesses  IMCI screening and referral to other health services Children 25 to 59 months with moderate or mild malnutrition at start of Tubaramure  Detection  Referral for CMAM  Home visits by Community Health Workers and MLs with specific messages for catch-up feeding  IMCI screening and referral to HF for other services Tubaramure graduates 25 to 59 months with mild or moderate malnutrition  Detection  Home visits by Community Health Workers and MLs - Reinforcing learning from Tubaramure - Use of additional foods during illness - Counseling on hygiene and home management of illnesses Source: Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure (July 2012). Pp. 21-22. 3.4.3. Results. Since none of the villages had community-based systems for rehabilitation of moderately malnourished children (other than the LMs’ recipes that were taught and supported under IR2 and IR3):  There were no major shifts in the program’s SAM activities after the mid-term; and  The program performance on the two indicators tracking this continued to be low (Table 2.10).68 Specifically, after a short-term increase during the first two years of the program’s field activities (Table 2.10): - The percentage of children with malnutrition referred from the community who were enrolled in the MoH nutrition services dropped from 92.0% at mid-term (when the majority of the children 0-24 years of age were receiving PM2A rations) to 32%; and 67 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: Tubaramure Program. Pg. 20. 68 Stakeholder review of the first draft of the Tubaramure Program Final Evaluation; September 28, 2014. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 39 - The percentage of children released from the clinic-based program who were visited at home by a Community Health Worker also dropped from 75.0% to 25.2%. Table 2.10. Tubaramure PM2A Program Indicators Used to Track Identify, Refer, and Monitor Follow Up on Children with Severe and Acute Malnutrition, 2010-2014 IR1 Key Program Indicators Baseline 2010 Mid-Term 2012 Endline 2014 LOA Target Output 1.4. SAM is detected and referred for treatment 1.11. Percentage of children with malnutrition referred from the community who are enrolled in nutrition services 72.7 92.0 32.0 85.0 1.12. Percentage of children recovered from acute malnutrition visited at home by the Community Health Workers at least twice a month for three months 7.1 75.0 25.2 90.0 Methodology: The figures in this table are based on health center records for both PM2A and non-PM2A beneficiaries. Source: Tubaramure IPTT (Annex II). 4.0. Factors Which Contributed to and Detracted From Program Relevance, Effectiveness, Efficiency, and Acceptability of Processes, Outputs, and Implementation 4.1. Relevance In 2009, the health system was just at the beginning of a formal decentralization process that was finalized during 2012-2013. The new system, which was formally rolled out at the end of 2012, was designed to increase the quality of the clinical-based health services in Burundi. This included:  Strengthening the MoH’s management and supervision of the health centers;69  Developing a plan for progressively increasing the number of health centers in both provinces; and  Granting legal recognition to the volunteer Community Health Workers. The same decentralization process gave legal recognition to the volunteer Community Health Workers (thus making them a formal part of the health system eligible to train in a more cohesive fashion) and increased their numbers more than three fold (Table 2.11). The Tubaramure Program was highly valuable to this process because it provided a series of training, equipment, and management supports for this system as it transformed. The program’s flexibility—like its willingness to provide some technical and logistical support for the revision of key protocols and to add a training module for the Community Health Workers (once they were trained), the module on data entry and management (to promote better reporting on the performance-based financing [PBF] and MoH indicators), and the module on supervision— helped the MoH improve both service quality and demand by:  Training staff, improving supervision, and providing essential equipment (under IR1);  Providing technical and logistical support for updating and/or developing certain key protocols that are requested and supported by the MoH (under IR1); and 69 Under the new system, the management of the health districts was separated from the management of the hospital. Instead of one doctor managing the entire district, there were now two. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 40  Building the local communities’ demand for these services through the Tubaramure BCC activities (under IR2). Table 2.11. Evolution in the Number of MoH Staff and Health Centers and Tubaramure Staff Working on IR1 Activities in the Ruyigi and Cankuzo Provinces, 2009-2014 MoH Staff and Infrastructure Project Zone Cankuzo Ruyigi 2010 2014 2010 2014 2010 2014 Nurse 221 907 128 451 93 456 Clinical assistant 69 308 43 153 26 155 Public Health Technicians70 24 26 9 13 15 13 Community Health Workers 712 1167 176 407 536 760 Tubaramure Program Staff Doctors 5 43 5 10 0 33 Tubaramure Health Promoters 36 22 15 9 21 13 Tubaramure IMCI IR1 staff 10 12 5 6 5 6 MoH Infrastructure Health centers 50 60 23 28 27 33 Hospitals 6 8 2 2 4 6 Source: IMC, Bujumbura, Burundi; August 2014. 4.2. Effectiveness The program was effective in achieving all of the major output IRs through the training, equipment, and supervision programs that increased the overall quality of health services (Text Box 2.6). The main exception was the IR focused on SAM for reasons beyond the Tubaramure Program’s control. By the program’s mid-term, both provinces were no longer eligible for the WFP rations for children with mild and moderate malnutrition (Section 3.4 of this chapter). This meant that large numbers of children who were identified as moderately or mildly malnourished were sent back to the local communities where there was little community-level support for their treatment other than the PM2A and local food-based treatments that the LMs and Community Health Workers could provide. The key challenge remains as signaled at mid-term and in the original proposal to develop better community and clinical-level systems for ensuring that children with moderate or mild malnutrition are identified, registered, and followed up with in the community. 70 Including the MoH Public Health Technician Coordinator for each province. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 41 Text Box 2.6. Sample Testimonials from the MoH About How the Tubaramure Increased Their Capacity to Deliver Quality Clinic-Based Services “Before the program, I did not have sufficient training in nutrition, especially for screening. Now we have improved our center’s systems for nutritional screening and referral of severe cases to other health facilities. For the mild cases of malnutrition that we identify, we try to train the mothers on essential nutrition actions.” “We see a clear difference between the IMCI-trained agents and those who have not had the training. The health agents who received the IMCI trainings are more efficient and competent in the management of childhood illness compared to those not trained in this approach.” “We now have a functional system for archiving our tracking data through workbooks and worksheets.” “Tubaramure has helped our supervisors to support us in our prenatal consultation work and prevention of malnutrition. The material we received from the program (the MUAC measuring tape, the nutritional posters, the flip charts and scales) has helped us with growth monitoring and monitoring the weight of the women we see during the prenatal consultations.” Source: Final evaluation key informant interviews and focus group discussions; August 2014. Annex IV. The government’s decision to formally recognize the national network of MoH volunteer Community Health Workers at the end of 2012 increased the effectiveness of all the MoH’s community-based programs by increasing the number of MoH volunteer Community Health Workers from 537 in 2009 to 1,167 in 2014. Since the health workers are the official colline￾level agents of the MoH, they will continue to work after the Tubaramure Program ends. The chief factors that decreased the effectiveness of the Tubaramure Program’s IR 1 activities were:  Only a few health centers had rations for treating the children who were identified as having moderate malnutrition (which affected the achievement of Output 1.4); and  The MoH never revised, updated, and validated its GM protocol (which affected the achievement of Output 1.3). It is important to note, however, that even without the protocol, the achievements on growth monitoring were major and appear very likely to be sustained beyond the duration of the program given the increased community-level demand for GM, which is a direct reflection of Tubaramure’s BCC activities under IR2. 4.3. Efficiency The IR1 component of the Tubaramure Program has been very efficient in its use of program resources to build the MoH’s capacity to prevent and treat malnutrition in the two provinces. With very few technical staff in the field, they have supported 61 health centers that serve the entire population of the province and 1,167 Community Health Workers. The program’s efficiency was increased because the activities responded to MoH priorities and needs to execute the four main clinical and community-level protocols that undergird its new nutrition policy. This increased efficiency by increasing the productivity of the MoH’s existing intervention structures, which lowered the unit costs of delivering health services and increased sustainability. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 42 The fact that the national-level MoH required the health districts and provinces to track a number of indicators that were likely to be affected by the successful execution of the program was also very important (Annex VI.A.7). First, it meant that the health districts and the provincial-level staff had a vested interest in the success of the effort because it would reflect well on their national tracking systems that were linked to donor disbursement. Second, it made them more receptive to exploring the Tubaramure-recommended changes, which focused on both malnutrition prevention (a familiar theme) and malnutrition treatment (a more established part of the protocols). 4.4. Sustainability71 72 The IR1 component of the program is very well embedded in the Burundian health system and structure, and there is a strong emphasis on capacity building and systems strengthening. All of this increases the likelihood that these activities will continue once the Tubaramure Program funding stops. 4.4.1. Progress Toward the Execution of the IR1 Exit Strategy. During the second half of the program, IMC designed an exit strategy together with the MoH. This exit strategy included scaling down contributions in fuel and other material support to the MoH. A major strength of the exit strategy was its simple format, which provided a clear illustration of who was responsible for different handover and scale-down activities (Annex VI.A.8). The same exit strategy—which was discussed with the two provincial-level offices of the MoH in June 2013— provided the basis of a revised memorandum of agreement with the MoH that was signed in July 2013. The combination of a clear, simple exit strategy and a memorandum of agreement to co￾execute it is an example of best practice for future programs. 4.4.2. Priority Issues for Follow Up. The MoH is likely to face five priority issues for follow up on the Tubaramure Program’s IR1 activities in the coming year (Table 2.12):  Continuing to monitor the nutrition and health of the women and children who graduated from the PM2A program, as well as new children born after the start of the program, since only a small percentage of the communities have community-based systems for GM and none of them have the PD/Hearth programs for rehabilitating any children that are identified as being moderately malnourished;  Continuing to train new staff in the core protocols and manuals. The high rates of staff turnover in MoH staff have reduced the percentage of local health centers that have at least two staff trained in all of the key protocols with Tubaramure support. For example, as of July 2014, only 18% of the doctors currently on staff have been trained in IMCI and only 12% in CMAM; 26% of the nurses have been trained in IMCI, 34% in CMAM, 20% in growth monitoring and only 21% in CPN/CPoN (Table 2.13). This means that the province will need to rely on its internal resources to train and retrain the staff using its own resources and those of donors that might be interested in supporting the same area. 71 SOW Specific Objective 10: Assess the issues of sustainability both at the institutional, the community, and the households’ levels. 72 SOW Key Question 7: To what extend program activities were financially supported and how will they be sustained at the institution and the community levels, after the program closure? Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 43 The fact that the Tubaramure Program trained with existing protocols and trainers will increase the efficiency (and lower the cost) of this training;  Combating the weak logistical support given to the MoH Public Health Technicians and Community Health Workers, which makes it difficult for them to perform their current slate of duties even without taking into account their projected attempts to better connect the Tubaramure-trained LMs to these activities;  Combating the weak motivation of the MoH Community Health Workers since they receive no direct compensation for their employment; and  Combating the low rates of postnatal consultation in both provinces. Table 2.12. Priority Issues for Follow Up to Capitalize on the Results of the Tubaramure Program’s IR1 Activities Priority Issues for Follow Up For the MoH For Other Donors Working in Health and Nutrition and Food Security in the Two Provinces 1. Continue to monitor the health and nutrition of the PM2A graduates and new children born after the program started. - Strengthen the capacity of the local communities to treat the children that are identified as moderately malnourished -Strengthen the current plans for expanding community-based growth monitoring by linking it to a strategy (like the PD/Hearth model) for community￾based rehabilitation of moderately malnourished children Support ongoing efforts of the MoH to develop community-based GM and community-based programs to rehabilitate moderately malnourished children based on the PD/Hearth model. 2. High rates of staff turnover -Develop a simple system for monitoring the number of staff and volunteers trained in the different protocols for each health center, which can help the MoH better plan future trainings -Anticipate the need for an annual training and retraining courses on the core nutrition modules using the existing base of MoH trainers who got additional training-of￾trainers training under Tubaramure Support ongoing MoH efforts to promote baseline training and retraining of MoH staff on key protocols at the clinical and community level 3. Logistical constraints for community-based programs that include: 1) An insufficient number of MoH Public Health Technicians to supervise the expanded number of Community Health Workers; which is compounded by the 2) insufficient logistical support (transportation, gas) given to the MoH Public Health Technicians and Community Health Workers to conduct their activities -Consider extending the pilot test currently under way in two other provinces to strengthen the MoH Public Health Technicians and Community Health Workers’ ability to support to community￾based nutrition and health programs -Brief new and existing donors on Tubaramure achievements in the provinces that they can strengthen and/or maintain through their existing or new activities in both provinces -Continue to organize the three types of inter-agency coordination meetings developed under Tubaramure -Support MoH efforts to implicate the Public Health Technicians and Community Health Workers in the design and execution and monitoring of all community-level activities -Consider ways to support the MoH’s ongoing efforts to prevent malnutrition -Solicit new technical and logistical support partners for coordination meetings, formative and joint supervision missions 4. Weak motivation of the Community Health Workers since their job is voluntary -Help the MoH Community Health Workers develop income-generating activities (IGAs) by linking them to Savings and Internal Lending Communities (SILCs) and groupements Encourage new NGO partners to support these initiatives Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 44 Priority Issues for Follow Up For the MoH For Other Donors Working in Health and Nutrition and Food Security in the Two Provinces -Encourage the Community Health Workers to organize themselves into associations so that they can be better helped by the government and link to outside development partners who can help them develop income-generating micro programs 5. Low rates of postnatal consultations -Capitalize on the large number of women bringing their children in for vaccinations during the first 45 days of the child’s life to conduct a postnatal consultation for the mother to help mothers avoid multiple trips to town -Continue to involve all MoH agents and local authorities in promoting prenatal and postnatal consultations Encourage new NGO partners to support these initiatives Source: Tubaramure Final Evaluation; August 2014. Table 2.13. Number of Current MoH Staff Trained Under Tubaramure, 2014 Staff Category Total (June 2014) Number of Staff in This Category Who Have Been Trained (%=% of Total Staff) IMCI Clinical CMAM GP CPN/CPoN IMCI/CMAM/ GM/CPN/CPoN (community based) Integrated Community Health Workers Manual Mgt. & Reporting MoH Staff Doctors 34 6 (18%) 4 (12%) N/A N/A 2 (6%) N/A N/A Nurses 413 107 (26%) 142 (34%) 83 (20%) 107 (26%) 6 (1.4%) 51 (12%) N/A Public Health Technicians 34 N/A N/A N/A N/A 27 (79%) 27 N/A Community Health Workers 1,167 N/A N/A N/A N/A Fewer than 54373 (47%) 1,167 N/A Medical assistants 127 0 83 (65%) 68 (54%) N/A N/A N/A N/A Health Information Systems Agents 7 N/A N/A N/A N/A N/A N/A 6 (86%) Tubaramure Program Staff Tubaramure Health Promoters 24 N/A N/A N/A N/A 100% N/A0 N/A Note: N/A means training not relevant for this category of individuals. Source: IMC, June 2014 Census Data, transmitted to final evaluation team August 2014. To address the first issue—which is a national problem, not one that is limited to the two provinces—the MoH is accelerating the finalization of the draft protocol for community-based GM and is encouraging the rapid expansion of the PD/Hearth model in the same communities. World Vision is one of several NGOs that is planning to expand its support for both the PD/Hearth model and GM program in both provinces over the next year. 73 Exact figure is not known. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 45 The second issue can be addressed by the provincial MoH’s getting some of the new donors moving into the area to support community-based GM and rehabilitation programs to also support staff training. To date, none of the MoH Public Health Technicians has ‘official’ motorcycles, and with the demise of Tubaramure Program, most of them will lose access to any special allowance for gas to operate any private motorcycles they might use to circulate. None of the MoH Community Health Workers have official bicycles. These transportation constraints place a real crimp on the abilities of MoH staff, which oversees all of the community-level health and nutrition activities, to do their jobs. The third and fourth issues are under reflection at the national level since they affect every aspect of the MoH’s community outreach. One option is to consider new ways that the MoH can help equip the MoH Public Health Technicians and the volunteer Community Health Workers to get bicycles. Two pilot programs to test a new mechanism for achieving this area already underway. Another option that is also under discussion is to consider various ways that the MoH can help the Community Health Workers to develop income-generating activities (IGAs) by joining Savings and Lending Communities (SILCs) or creating economic associations (groupements). Several options for increasing the rate of postnatal consultation are under consideration including the recommendation putting greater emphasis on vaccination in the months just after birth to encourage the mothers to come for postnatal checkups. 5.0. Lessons Learned and Best Practice Checklist Table 2.14. Tubaramure Program IR1 Lessons Learned and Best Practice Checklist Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide Government Institutions (MoH)74 Future Donor￾Funded Activities Future PM2A and Food Security Programs IR1: Women and children under 5 access quality nutrition and health services. Lesson 1. Training programs that contribute to the preparation and revision of existing protocols for the MoH and/or other partners are more likely to be effective, efficient, and sustainable. 1. Build training programs to prevent malnutrition on existing MoH protocols and training modules. X X X 2. Train all MoH staff in the key nutrition treatment and preventions protocols (at the provincial, commune, and community levels) from the start of the program in order to build the roots for sustainability. X X 3. Include a flexible budget for technical and logistical support to update and revise critical MoH protocols and training manuals. X X 4. Build the capacity of the existing MoH trainers in the provincial and district-level health centers to execute training programs for the key protocols being supported. X X 74 This column refers to MoH activities that are critical to maintaining or sustaining the Tubaramure-supported activities once program funding ends. If there is no X in the column, it should be assumed that the MoH is already supporting this activity. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 46 Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide Government Institutions (MoH)74 Future Donor￾Funded Activities Future PM2A and Food Security Programs 5. Anticipate the need to train and equip Community Health Workers from the start. X X 6. Anticipate the need for a flexible budget line to support formative supervision missions to follow up on basic training of staff in all health centers. X X 7. Anticipate the need for baseline, mid-program, and end-of￾year training programs on all of the key modules in order to accommodate staff turnover. X X 8. Offer certificates to individuals who complete the training to validate their new knowledge to themselves and their managers. X X 9. Strengthen systems for monitoring program and staff performance on critical protocols that are underperforming. Lesson 2. Providing material support to health centers can help motivate the staff and strengthen their capacity to offer higher-quality health and nutrition services. 10. Anticipate the need for a flexible line item of equipment for all local health centers that provide technical support to community-based nutrition and health programs. X X 11. Anticipate the need to update and renew this equipment in the last year of the program. X X 12. Give assistance that is ‘demand driven’ by a health center list of priority needs and MoH norms. X X Lesson 3. The implication and good collaboration with local authorities and NGO partners is essential for the efficient execution and mainstreaming of nutrition interventions into local development plans. 13. Strengthen the existing system and/or create MoH￾managed quarterly meetings that bring together the different health and nutrition actors in the program intervention areas. Maintain Maintain X 14. Strengthen the existing systems for collaborative planning between the MoH Public Health Technicians and Community Health Workers. Yes Yes X 15. Offer training courses to the nurses who manage the local health centers to help them better manage the centers’ nutrition activities and better connect these activities to the other actors (like the MAE), as well as the provincial and commune-level development planning processes that affect their centers. X X X 16. Provide technical and logistical support for local development plans that local governments can use for strategic planning processes as soon as possible in order to ensure that the malnutrition prevention programs are mainstreamed. X 75 X X Lesson 4. Program systems that build on and strengthen the MoH’s existing supervision, monitoring, and evaluation systems help supervisors provide on-the-job training in technical issues that improve service quality and 75 In the case of Burundi, the timing for this should be no later than the mid-term of the new 2013-2014 Commune Community Development Plans (PCDC)—which should be in 2017—to ensure that the NGOs facilitating these processes and the local administrators are well-versed in the ongoing community-based nutrition and health programs. Early and consistent implication of local administrators in the program’s activities should increase but not guarantee (due to turnover) a more sophisticated and useful analysis of the most critical constraints and planning issues. Tubaramure Final Evaluation. Chapter 2: IR1. October 22, 2014. Final. 47 Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide Government Institutions (MoH)74 Future Donor￾Funded Activities Future PM2A and Food Security Programs ownership of results. 17. Facilitate joint supervision missions that permit national, provincial, and district-level supervisors to monitor the execution of the activities being supported by the program. Maintain Yes X 18. Harmonize future program’s M&E systems with those of the health centers for the key protocols being supported in order to avoid duplication. X X 19. Strengthen the MoH’s district-level systems for the collection and analysis of nutrition data. X X X 20. Build MoH capacity for more decentralized nutrition data entry and analysis. X 76 X X 21. Support MoH efforts under way in 2014 to add nutrition and community-level sanitation indicators to the list of performance-based financing indicators being tracked by the government to give greater visibility to MoH efforts to promising clinical and community-based new initiatives to prevent malnutrition. X 22. When conducting pilot studies/tests of new initiatives, develop careful tracking systems that allow comparisons of the new initiatives with existing initiatives. X X Source: Tubaramure Final Evaluation; July-August, 2014. Revised based on feedback to the first draft; September 25-October 5, 2014. 76 Train and retrain the provincial and district-level staff in data entry, analysis database creation, and management to avoid labor-intensive hand entry of the MoH and donor forms, which can create costly duplications of effort for M&E and technical staff. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 48 Chapter 3 Intermediate Result 2: Households Practice Appropriate Health and Nutrition Behaviors 1.0. Global Strategy 1.1. Major Outputs A major strength of the Tubaramure Program was its conceptualization of a broad cross-cutting intermediate result (IR) related to behavior change. The activities under this IR were expected to help the local beneficiaries, households, and communities understand what types of behavior changes will be needed to sustain the nutritional and health improvements associated with the IR1 and IR3 activities. Specifically, the activities were designed to achieve three outputs (Table 3.1). Table 3.1. Major Outputs Designed to Achieve the Tubaramure Program’s IR2 IR2: Households practice appropriate health and nutrition behaviors. Output 2.1. Households (HHs) adopt Essential Nutrition Actions (ENA). Output 2.2. Households adopt Essential Hygiene Actions (EHA). Output 2.3. Households adopt prevention and management behaviors for maternal and childhood illnesses Source: CRS; 2009. Tubaramure Multi-Year Assistance Program (MYAP) Proposal. Bujumbura: CRS. Food for the Hungry (FH) was the technical lead for this IR and implemented the Care Group approach as a primary conduit for behavior change. The Care Group Model is a community￾based strategy for achieving widespread and lasting household and community-level behavior change. Care Groups are comprised of volunteer Leader Mothers (LMs) selected by their neighbors to conduct health promotion with women who are pregnant or have children 0-23 months of age. All beneficiary mothers were expected to participate in a beneficiary group (under the leadership of a LM) as one of the conditions of getting the Preventing Malnutrition in Children Under 2 Approach (PM2A) rations distributed under IR3. 1.2. The Original Intervention Model To achieve these three outputs, the Tubaramure Multi-Year Assistance Program (MYAP) Proposal outlined a three-prong strategy that focused on:  The creation of Care Groups;  Formative research to develop, pilot test, revise, and finalize a group of five behavior change communication (BCC) training models; and  A system of cascade training in which the Tubaramure Health Promoters would use the five training modules to train the LMs, who could use them to train the beneficiary groups. To complement the Care Group Model, the program added a parallel set of activities that supported:  Hygiene and Sanitation Improvements: A series of community-level demonstrations on how to construct some of the key hygiene and sanitation innovations illustrated in the Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 49 BCC training modules (i.e. drying racks, improved latrines, hand washing stations [Tippy Taps], garbage pits, and other low cost innovations); and  Radio Emissions: A series of national and regional-area broadcasts that reinforce the BCC messages. 2.0. Activities 2.1. Program Start Up to Mid-Term Once the initial beneficiary assessment was conducted in 2009, all of the women eligible for PM2A rations were organized into beneficiary groups comprised of 10-12 members and selected a LM to represent them. Once this happened, the LMs became the principal conduit for Tubaramure Program information to and from the beneficiaries for the BCC activities under IR2, as well as the food distribution and culinary demonstrations under IR3. When the program started, there was no national curriculum for the MoH Community Health Workers, so the training material had be adapted based on formative research and a local artist needed to adapt the images based on the Burundi context. One unintended consequence of the program developing its own BCC modules was the inevitable delays that were linked to, “The sheer number of steps involved in developing quality, tested materials; locating a qualified artist for the illustrations; and finding a competent printing house that would respect the contractual requirement.”77 Further delays were created by the inevitable delays involved in scheduling review sessions with the Ministry of Health (MoH) committees charged with oversight. Other delays were created by the heavy workloads of the Tubaramure Health Promoters who worked across all three IRs and had a number of responsibilities in addition to their BCC activities. As a result, “attendance at the second set of monthly meetings (Tubaramure Health Promoters with LMs, and LMs with beneficiaries) is always lower. Sometimes the meeting is not held at all due to scheduling conflicts, competing priorities for the Health Promoters or bad weather. As a result, some Care Groups covered only one lesson per month.” 78 This meant that not all of the beneficiaries were able to hear all of the lessons in the five core modules before they graduated from the PM2A program (i.e., were no longer eligible for food rations since their child had turned 24 months of age) since some trainings did not occur until 2013 (Annex VI.B.1). To address this issue the consortium created an ‘age-specific card’ which is a shorter, separate module synthesizing all the key maternal and child health messages organized according to the stage of pregnancy or the age of the child. Starting in November 2010, this card was distributed to all LMs and many beneficiary mothers during the Tubaramure Health Promoter training sessions. Although the age-specific card provided a brief overview of the other themes, this was considered more in the spirit of information dissemination rather than core BCC training. 77 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pp. 30-31. 78 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pp. 30-31. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 50 2.2. Early Impact of the First PM2A Graduation, Care Group Participation, and Tubaramure Cascade Training Model79 In November 2011, the first group of PM2A beneficiaries (8,000 mother/infant pairs) graduated from the program (i.e., their children turned 24 months of age at the same time). Once their involvement in the beneficiary groups was no longer a condition for getting rations, many LMs and beneficiaries dropped out and/or became more irregular in attendance. The principal reason given for this drop out was lack of motivation (i.e. concrete compensation for her time). To encourage the beneficiary mothers to introduce the new crops and livestock that they would need to continue having a more diversified diet after graduation, the program distributed its first round of vegetable seeds,80 fruit trees, and poultry to the LMs in late 2011, and encouraged beneficiaries to consider forming economic groupements. 81 This assistance catalyzed a small number of the LMs to start organizing their own groupements in the six months before the mid￾term evaluation.82 When some of the initial investigations of beneficiary dropout showed that many women’s husbands were encouraging them to drop out of the program once they were not longer eligible for rations, some Tubaramure Health Promoters encouraged their Care Groups to select Leader Fathers (LFs) who attended trainings and were expected to organize their peers for individual and group BCC activities.83 2.3. Principal Conclusions and Recommendations From the Program Mid-Term The July 2012 mid-term evaluation report concluded that although the program had made significant progress toward the three outputs for IR2, a number of issues would need to be addressed to sustain this performance and reach the end-of-program targets, including:  Determining how to fine-tune messages promoting the need for a variety of foods as important for young children, but also other Infant and Young Child Feeding (IYCF) recommendations such as frequency of small meals, consistency/digestibility of food depending on age, and sufficient quantity of food for age; 79 SOW Key Question 16: How are graduated and current beneficiaries in the community coping with the end of food distribution? 80 One of the most critical constraints on agricultural production in both provinces is the limited access to improved seed. Although a number of seed cooperatives have organized to produce good (though not officially certified) rice seed, there seem to be very limited sources of improved seed for vegetables and almost no national option for the soy bean seed, which is critical to the preparation of the weaning broths that the LMs promote. For this reason, most of the LMs identified improved seed as their top priority for follow-up help on strengthening their production activities. 81 The distribution of these ‘motivations’ was counted as an IR3 support activity in the hopes of encouraging them to promote more diversified diets. 82 Based on the information in the recently created Tubaramure monitoring and evaluation (M&E) database on the groupements, nine of the current groupements were created in 2010; 49 in 2011; and 138 in 2012, both prior to and after the program’s mid-term (Tubaramure M&E Office; August 2014). 83 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 33. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 51  Determining the most effective ways to prevent diarrhea and manage it at home, and promote these behaviors through the Care Group trainings through closer collaboration between FH (the technical lead on IR2) and International Medical Corps (IMC, the technical lead on IR1);  Encouraging a renewed emphasis on, “what can be done at the household and community level to prevent and manage” malaria;84 and  Exploring various ways to strengthen and sustain the Tubaramure Care Group model by: - Reducing the workload of the Tubaramure Health Promoters so that they have more time to focus on the BCC component of their work; - Motivating the LMs to continue attending Care Group meetings after graduating since this was deemed critical to their staying informed and connected to the MoH; - Strengthening the program’s s collaboration with the MoH on the design, execution, and monitoring of the BCC activities, “other than including visits to Care Groups in the program for joint supervisions;” and - Helping men, “especially husbands and fathers, to contribute to behavior change.”85 2.4. Impact of the Mid-Term on IR2 Strategies and Activities86 2.4.1. Barrier-Analysis Studies to Help Reorient the IR2 Strategy. To address these concerns, the program commissioned two barriers-to-behavior-change studies in the six months after the mid￾term evaluation. The first study was designed to:87  Better understand the barriers-to-behavior-change reasons for the dropout rate among beneficiaries and LMs; and  Examine various options for building men’s support for their wives’ continued participation in the Care Groups. This first study made eight recommendations for sustaining the Care Groups and four for building more men’s participation in the program (Annex VI. B.2). The second study focused on examining the barriers to behavior change for five critical behavior changes being targeted by the program—exclusive breastfeeding, postnatal consultation, growth monitoring (GM), construction and use of latrines, and the construction and use of handwashing stations88 (Annex VI. B.3). 84 A FH barrier analysis of the use of insecticide-treated nets (ITNs) in June 2011 provided the basis for this recommendation. 85 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pp. 39-40. 86 SOW Key Question 5: To what extent recommendations from the mid-term evaluation have been incorporated into the nutrition and health sector? 87 Leonie Niyonkuru; 2012. Etude sur la Pérennisation des Care Groups et les Activités Genre dans le Program Tubaramure. Bujumbura: Tubaramure. (November 2012). 88 Tubaramure PM2A Program M&E Team; 2012. Etude sur les Obstacles a l’Adoption des Bonnes Pratiques en Matière de Santé, Nutrition et Hygiène, dans les provinces de Ruyigi et Cankuxo. Rapport Final. Bujumbura: Tubaramure M&E Office. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 52 2.4.2. Stronger Emphasis on Gender and Supporting the Beneficiaries Developing Groupements. Two of the major changes after the mid-term evaluation report were:  The program’s creation of a series of activities targeting men, as well as activities that strengthened men’s participation in the Care Groups; and  A targeted set of activities designed to help the beneficiary groups and LMs develop the types of registered economic groups (groupements) needed to receive any type of economic assistance from the government and/or local or international NGOs. The goal of Tubaramure support to the groupements was to create a mechanism for the LMs to continue to meet with a portion of their beneficiary group by creating new income-generating activities (IGAs) that they managed as a group. Key activities included (Table 3.2):  Informal training by the Tubaramure Health Promoters (under the supervision of the IR2 supervisors in both provinces) on how to create and manage a groupement;  Help with registration; and  The distribution of seed, small numbers of animals, and training in keyhole gardens. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 53 Table 3.2. Evolution of the Program’s Complementary Support Given to the Tubaramure-Facilitated Groupements Under IR2 and IR3 Support Lead Partner Collaboration Number of Groupements 2011 2012 2013 2014 Created 5889 138 572 4 Supported (2/colline) NA NA 508 508/434 Support given to individual LMs FH X  Fruit trees FH X  Roosters FH X Support given to groupements  Organization FH X  Registration FH Commune X X X  Training on and technical support keyhole gardens90 CRS DPAE (informal) X X X Seed distributions CRS DPAE (informal)  Vegetables X X indirect 91  Soy beans92 24 groups 48 groups 434 groups  Maize X X Goats FH DPAE (informal) 24 groups 48 groups93 434 groups Grinders and other equipment to help promote food processing activities CRS 434 Acronyms: DPAE: Direction Provinciale de l'Agriculture et de l'Elevage (Provincial Directorate for Agriculture and Livestock); X: Yes, but no precise figures are available; CRS: Catholic Relief Services. Source: Tubaramure IR3 National Coordinator Regine Pacis Nohoreho, IR3 Coordinator for Tubaramure (Ruyigi) Ezéchiel Kabwebwe, and Tubaramure Provincial Coordinator (Cankuzo) Edmond Twagirayezu; July 2014. The Tubaramure staff executed most of the activities with some assistance from the commune￾level agronomists and veterinarian assistants for keyhole garden training (2012 and 2013) and the selection of the animals for distribution to the groups (2013 and 2014). As of August 2014, the Tubaramure Program has created a total of 874 groupements, 94 713 (82%) of which are registered (i.e. have completed the papers necessary for them to register with the government), which is a prerequisite to working with the Ministry of Agriculture and Livestock (Ministère d'Agriculture et d'Elevage or MAE) (Figure 3.1). 89 Nine of the current groupements were created in 2010; 49 in 2011. 90 The initial training of trainers of Tubaramure supervisors was facilitated by CRS/Lesotho in 2011. 91 Although the program did not distribute vegetable seeds in 2014, several groupements requested assistance with purchasing the seed in Bujumbura, many of them with funds generated by the Savings and Internal Lending Communities (SILCs) (See IR3, Chapter 4). 92 The initial training of trainers and seed for this activity was facilitated by CRS/Rwanda in 2011. 93 The pilot distributions gave one goat to two persons in a more limited number of groups. This meant that the beneficiary groupements each received between 15 and 16 animals. During the second distribution, the program gave two goats per group (including those who had received goats the first time). Since the program supported only two groupements per colline this meant approximately 44 animals per colline. 94 During the month of March 2014, the CRS M&E Office conducted its first comprehensive census of the Tubaramure-facilitated groupements. This census was turned into a database that is regularly updated by the Tubaramure M&E staff (Annex VI.B.4). These figures were based on an analysis of this database. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 54 Figure 3.1. Year the Current Base of Tubaramure Groupements Were Created in Ruyigi and Cankuzo Provinces Source: Tubaramure M&E Office; August 23, 2014. About 58% of the groups (50895 out of the 874) received the total package of assistance (organizational training, technical training, seeds, goat distributions, etc.). Out of this number, the program chose 434 of the assisted groupements (outside the International Food Policy Research Institute/Food and Nutrition Technical Assistance Project [IFPRI/FANTA] research area) that are the focus of the program’s principal livestock development and soy processing activities in 2014 (Table 3.2). This assistance has indeed helped motivate a relatively high percentage of the LMs to continue to attend the Care Group sessions even after graduation. It has also created a new conduit for the LMs to instruct the beneficiary mothers and other members of the community.96 In the first groupement census in March-May 2014:  54% of the groupements in Ruyigi and 67% of the groupements in Cankuzo reported that they routinely incorporated some Essential Nutrition Actions (ENA) instruction by the LMs into their meetings; and  73% of the groupements in Ruyigi and 55% of the groupements in Cankuzo reported including some Essential Hygiene Actions (EHA) instruction into their routine meetings. 95 This figure of 508 groupements is based on the following calculation: 269 collines covered by the program – 15 test villages where no Tubaramure activities were conducted = 254 x 2 groupements by community=508). Since the exact number of groupements targeted by different activities varied, this figure could go up and down. 96 Equipe de Suivi Evaluation CRS-Burundi; 2014. Quelques Résultats du Travail mené sur les Groupements (Associations) de PM2A dans les Provinces de Ruyigi et Cankuzo. PowerPoint Presentation. (March-May 2014). Pp. 4-5. 9 49 138 631 47 0 100 200 300 400 500 600 700 Year 2010 Year 2011 Year 2012 Year 2013 Year 2014 Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 55 The same baseline census estimated that 54% of the groupements in Cankuzo and 35% of the groupements in Ruyigi also have Savings and Internal Lending Communities (SILCs), which increases their chances of being sustained.97 3.0. Evidence of Results 98 99 The program’s output indicators were measured annually and, during the critical start-up period (2011), quarterly by the Tubaramure monitoring and evaluation (M&E) department based on a random survey of 400 beneficiary households. This annual performance survey provided real￾time feedback to the implementing partners and their local government partners on which behaviors were changing most rapidly. The same activities were expected to have a population-based impact through secondary adoptions of the new behaviors. The program’s M&E plan set targets for these final population￾based impacts and measured its progress toward achieving this through the final quantitative survey, in which 37.6% of the households were non-beneficiaries (i.e. did not receive PM2A rations from the Tubaramure Program).100 3.1. Output 2.1. Households Adopt Essential Nutrition Actions The Tubaramure BCC strategy focused on the promotion of six ENAs (Table 3.3). The principal mechanisms for promoting these behaviors were:  Care Group training programs using the five BCC modules;  LMs’ and Tubaramure Health Promoters’ supervision visits to individual homes; and  Tubaramure-sponsored radio programs. 97 Equipe de Suivi Evaluation CRS-Burundi; 2014. Quelques Résultats du Travail mené sur les Groupements (Associations) de PM2A dans les Provinces de Ruyigi et Cankuzo. PowerPoint Presentation. (March-May 2014). Pg. 7. 98 SOW Key Question 6: With regards to the ENA and EHA (for IR1), Integrated Management of Childhood Illness (IMCI) (for IR1), and the food intake and diversity (for IR3), are beneficiaries adopting desired practices or behaviors? Are there some secondary adopters? 99 SOW Key Question 8: To what degree are behavior changes continued by graduated program beneficiaries, for example changes that improve the nutrition of children over the age of two and changes that impact subsequent births? 100 42.1% of the total households in the survey (1,196) were PM2A graduates; 18.1% had a mother that was still receiving PM2A rations; 2.2% had a mother who had abandoned the program; and 37.6% had never been in the program. About a quarter (24.8%) of the women interviewed were LMs. ISTEEBU; 2014. Enquête Ménage pour l’Evaluation Quantitative Finale du Programme PM2A-Tubaramure (Provinces Cankuzo et Ruyigi). Rapport Final. Bujumbura: ISTEEBU (August 2014). Pp. 17, 41. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 56 Table 3.3. Percentage of Households Adopting the Main ENA and EHA Practices Promoted by the Tubaramure Program in Ruyigi and Cankuzo Provinces Actions Baseline Final N=1,255 N=1,200 All Output Indicator 2.4 Percentage of households observed carrying out four or more ENA actions at time of household visit (observed or reported) N/A 69.8101 EBF: Percent of babies < 6 months exclusively breast-fed in last 24 hours 69.4 87.9 Four Food Groups: Consumption of at least four food groups by children 74.2 77.8 IBF: Immediate breastfeeding after birth 88.1 90.2 Iodized Salt: Consumption of Iodized Salt 58.4 GM: Child weighed and measured during the last four months 16.0 57.3 Pregnancy Diet: Increased food consumption by mother during pregnancy N/A 32.0 Output Indicator 2.5 Percentage of households observed carrying out four or more EHA actions at time of household visit (observed or reported) N/A 58.5 HW: Hand washing N/A 36.0 Latrine: Latrines responding to norms 18.4 59.8 Clean Yard N/A 95.8 Pit: Existence of a compost pit N/A 64.2 Platform: Existence of a drying rack for washed dishes 9.0 48.5 Water Purification: Practices water purification 10.6 47.1 ITN: Use of insecticide-treated net 61.3 48.6 Source: Tubaramure Indicator Performance Tracking Table (IPTT) and Institut de Statistiques et d'Etudes Economiques du Burundi (ISTEEBU); 2014. Enquête Ménage pour l’Evaluation Quantitative Finale du Programme PM2A—Tubaramure (Provinces Cankuzo et Ruyigi). Rapport Final. Bujumbura: ISTEEBU (August 2014). Pp. 38- 39. The community-level results of the activities for the PM2A beneficiary households were tracked through a series of internal surveys of 400 randomly chosen beneficiary program households.102 This real-time feedback provided evidence that the beneficiary households were adopting most of the ENA actions.103 The final quantitative survey estimated that 69.8% percent of the households had adopted four or more of the Essential Nutrition Actions (Table 3.3; Figure 3.2). Since this indicator was not measured during the baseline quantitative survey, there is no way of knowing if the impact is statistically significant. 101 The indicator 2.4 on page 19 of the Institut de Statistiques et d'Etudes Economiques du Burundi (ISTEEBU) report is incorrectly labeled as EHA; it is for ENA. The indicator 2.5 on page 19 of the ISTEEBU report is incorrectly labeled ENA; it is EHA. 102 The initial surveys in 2010 were conducted bi-annually. After this time, they were conducted annually, and some of the data used to track the IPTT. 103 Since the Tubaramure Program never developed the community-based systems for growth monitoring that were envisioned in the proposal, there was very limited motivation for the mothers to bring their children to the health center-based growth monitoring programs once they were no longer receiving rations (see Chapter 2, IR 1 discussions of growth monitoring). Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 57 Figure 3.2. Output Indicator 2.4. Percentage of Households Observed Carrying Out Four or More ENA Actions at Time of Household Visit (Observed or Reported) Source: Tubaramure IPTT (Annex II). It is important to emphasize that some of these practices were virtually unknown before the program started. Based on the focus group discussions during the final evaluation (Annexes IV and V), the evaluators concluded that most of the beneficiary mothers are continuing to practice the health behaviors they learned during the Tubaramure Program (Text Box 3.1). 69.8% 90% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Baseline 2010 Endline 2014 LOA Target Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 58 Text Box 3.1. Testimonials from the Beneficiary Mothers about the Impact of the Program’s IR2 BCC Activities “Before Tubaramure, when the women didn’t understand the infants nutritional needs, they gave breast milk as well as supplementary food from an early age. There were a lot of children’s’ illnesses caused by dirty hands like diarrhea. Our children were underweight and often sick. Many births were also underweight (i.e. less than 2.5 kilograms). Since the Tubaramure Program started, most children are fed only breast milk during the first six months before being given broths. Thanks to the training we received and the adoption of improved hygiene practices, the percentage of children with diarrhea has decreased. The children have also gained weight. After graduation (i.e. after no longer being eligible for PM2A rations), I became pregnant and continued to use the good practices for me and after giving birth to the new child who I nursed exclusively during the first six months. Since my new baby is now over six months, I am continuing to look after him using the improved practices by feeding him a nutritious broth made from locally produced food.” “Before the women didn’t know how to prepare the foods that the program recommended. There were also lots of local taboos about certain nutritious foods not being good for you—like the idea that consuming liver would create dental problems in the child after birth or that eggs should only be sold to rich persons. One output of the program has been that women are starting to consume these products. The fact the mothers didn’t know how to prepare balanced meals, increased the rate of malnutrition in our areas. Now the women are better able to follow the improved practices. The distribution of the family posters, which promote locally grown foods and dishes made with these foods, has also helped.” Source: Final Evaluation focus group discussions in Nyagutoha colline; July 2014. The same final evaluation focus group discussions made it clear that the level of understanding and practice of the recommended ENA was much higher for the direct beneficiary mothers (who were taught by a LM) than for non-beneficiary mothers who were not. 3.2. Output 2.2. Households Adopt Essential Hygiene Actions The Tubaramure BCC strategy focused on the achievement of seven EHAs (Table 3.3; Figure 3.3). Although there was widespread agreement that these activities supported the IR1 outputs, these were not activities that were being tracked by the Community Health Workers. The principal mechanisms for promoting these behaviors were the same as those for promoting the ENAs:  LM training programs;  LM and Tubaramure Health Promoter-facilitated home visits; and  Tubaramure-sponsored radio programs. The final quantitative survey estimated that 58.5% percent of the households had adopted four or more of the EHAs (Figure 3.3). Since this indicator was not measured during the baseline survey, there is no way of knowing if the impact is statistically significant. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 59 Figure 3.3. Output Indicator 2.5. Percentage of Households Observed Carrying Out Four or More EHA Actions at Time of Household Visit (Observed or Reported) Source: Tubaramure IPTT (Annex II) Many women who attended the focus group discussions stated that the practice of open-area defecation had considerably decreased in response to the training they received. When asked about which practices were the easiest to adopt, most of the beneficiaries mentioned construction of the drying racks, hand washing, latrine use, and regular bathing. One of the common observations made during the focus group discussions was that, “Before, we all (children and adults) defecated in the open area. After the Tubaramure Program trained us on the importance of using latrines, we constructed latrines in our homes. Most children and adults no longer defecate out of doors.” Key factors that appear to have sustained and even increased these behaviors after the mid-term evaluation were the Tubaramure Program’s:  Commitment to a lower-cost norm for latrines that could be built and sustained with local materials;104 104 Based on evidence from the mid-term focus group discussions and the 2012 barrier-to-behavior-change study, the program shifted to a lower cost mode of latrine construction based on locally available materials and made a Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 60  Efforts to promote greater involvement of the beneficiaries’ and non-beneficiaries’ husbands in the sanitation activities; and  The implication of the local commune administrators, as well as other high-ranking members of the community (including the two governors) in promoting the new sanitation behaviors in speeches and site visits. It is indicative of the successful impact of the program’s efforts to involve local authorities in building a more widespread base of local support (from men, women, and local chiefs) that most of the commune and provincial-level authorities interviewed during the final evaluation were able to describe the various EHA and ENA activities. Many of the same commune authorities and both governors were also able to show ways that they supported these activities (Text Box 3.2). Text Box 3.2. Lesson Learned: Critical Importance of Involving Local Authorities in BCC Campaigns “One of the main impacts of the Tubaramure Program has been to improve sanitation and health practices throughout the province. I was very impressed by the ones I was shown when I participated in the joint supervision site visits with the program. It is an impact that is very much appreciated by the local people and one I support. Since then, I have sought opportunities to include these themes into the speeches I give at various colline and commune-level meetings. This afternoon I am scheduled to give such a speech and I will find a way to incorporate something about how proud I am of that community’s efforts to improve its sanitation and health practices and how important it is to continue to support these.” Source: Key informant interview with Ruyigi Province Governor Cyriaque Nshimirimana; July 2014. In contrast to the ENA actions—which are directly linked to the community-level activities under IR1—the Tubaramure Program’s sanitation activities are, by and large, not currently being tracked by either the Community Health Workers or the health centers as part of the performance-based financing (PBF) indicators that they track for the MoH. Informal discussions with the IR1 technical (IMC) staff during the final evaluation suggest that the MoH is lobbying to add a number of EHA indicators to the PBF indicators, which they consider essential to the successful achievement of its wider curative and preventive mission. conscious effort to increase the involvement of men, and local and provincial-level administrators in building public awareness and support for these efforts. This shift in focus resulted in a sharp population-based impact in latrine use that was confirmed by the final quantitative study. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 61 3.3. Output 2.3. Households Adopt Prevention and Management Behaviors for Maternal and Childhood Diseases105 The final quantitative survey showed a 19-point increase in the percentage of mothers with children under 2 who could state at least four of the six danger signs for childhood illnesses, and at least two of the danger signs for pregnant women (Figure 3.4).106 Although this increase was statistically significant, it was still half the expected population-based target of 45%. Figure 3.4. Indicator 2.6. Percentage of Mothers With Children Under 2 Who Can State at Least Four of the Six Danger Signs for Childhood Illness and at Least Two of the Four Danger Signs for Pregnant Women Source: Tubaramure IPTT (Annex II). The principal reason for this much lower (than the target) population-based impact appears to be the program’s exclusive focus on the target beneficiary households, which limited the impact of the program on the wider population. The final evaluation focus group discussions with beneficiary and non-beneficiary women107 confirmed that substantial progress has been made in identifying and managing several major 105 The original indicator for this output was “Percentage of households with children under 2 with a reserved package of [Oral Rehydration Salts] ORS at time of household visit.” At mid-term, the decision was made to change the indicator for several reasons, including the fact that the ORS packets were not available commercially. Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 38. 106 There was no major difference between Cankuzo and Ruyigi in this indicator—26.7% vs. 23.1%, respectively. 107 In the later years, it appears that some non-beneficiary women joined these groups—especially after graduation when the principal focus of the groups shifted to the SILC and economic groupement activities. 5% 24% 45% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Baseline 2010 Endline 2014 LOA Target Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 62 childhood diseases, but that the level of understanding of the danger signs was much lower for the non-beneficiary women. The same focus group discussions showed an almost universal commitment of the LMs and beneficiary mothers to continuing to use these skills to manage their own children’s illnesses, as well as those of neighbors and family who came to them for consultation. Since this capacity to identify the danger signs is critical to sustaining the MoH’s goals for clinical and Community￾Based Integrated Management of Childhood Illness (C-IMCI)—and to the MoH’s maintaining the substantial increase in these indicators that resulted from the Tubaramure Program—there is a clear need for a joint strategy with the MoH on how to continue to build the capacity of local households to identify, manage, and prevent the province’s most prevalent childhood diseases. Although Tubaramure made a good start, this is an area that needs to continue to be supported. One output of the Tubaramure-sponsored barrier analysis on the use of insecticide-treated nets (ITNs) in June 2011 was a more committed focus on what could be done at the household level to prevent and manage malaria. This activity was strengthened by Caritas’s distribution of bed nets to the LMs to use as demonstration tools during Care Group presentations. By May 2014 (when the final quantitative survey was conducted), 50.7% of the children were sleeping under mosquito nets—45.4% in Ruyigi and 60% in Cankuzo.108 Based on interviews with MoH staff and the focus group discussions, this appears to be an important increase though the evaluation could not document it quantitatively since it was not measured during the baseline survey. 3.4. Evidence of Differential Impacts on Certain Groups109 3.4.1. Beneficiary vs. Non-Beneficiary Women. The focus group discussions indicated that there was a significant difference between the beneficiary and non-beneficiary mothers’ ability to state the six danger signs for the principal childhood illnesses and most of the adoption of the recommended EHA and ENA practices (Annex IV). Unfortunately, only one of the program’s internal IR2 indicators—the percentage of children who slept under mosquito nets the previous night—was disaggregated by beneficiary and non￾beneficiary households, so it is not possible to compare this using the quantitative survey’s results.110 The steep drop in the “percentage of households observed carrying out four or more EHA actions at time of household visit” between 2012 and 2013 (74.3% and 70.1% respectively), when the indicator was measured for beneficiary households only, and 2014 (58.5%), when the indicator was based on the population survey, was attributed to the lower rates of adoption by non-beneficiary households that were not counted in the 2012 and 2013 survey (Annex II). 108 ISTEEBU; 2014. Enquête Ménage pour l’Evaluation Quantitative Finale du Programme PM2A-Tubaramure (Provinces Cankuzo et Ruyigi). Rapport Final. Bujumbura: ISTEEBU (August 2014). Pg. 16. 109 SOW Key Question 7: Are there certain groups within the population with lower rates of adoption and why? 110 That analysis stated, “that a higher percentage was noted for the mothers still in the program and a lower percentage for the children of mothers who had not been in the program.” ISTEEBU; 2014. Enquête Ménage pour l’Evaluation Quantitative Finale du Programme PM2A—Tubaramure (Provinces Cankuzo et Ruyigi). Rapport Final. Bujumbura: ISTEEBU (August 2014). In fact, the figure was 64% for the mothers still in the program, 57.8% for the graduated mothers, and 40.4% for non-beneficiaries, so the disaggregated analysis did not show a big difference (Table A.11.A Pg. 85). Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 63 3.4.2. Vulnerable Groups. Certain vulnerable groups were less able to adopt the recommended ENA and EHA practices than others:  Batwa: The dispersed Batwa111 communities in Cankuzo Province and the three collines with the heaviest concentration of this social group;  Female-Headed Households: Especially those located in the six collines near the Tanzania frontier where a high percentage of the economically active men are migrant workers in Tanzania for the majority of the year.112 Staff reported that once this issue was identified (through Tubaramure Health Promoter feedback to supervisors) the supervisors tried to encourage the colline chiefs to organize group labor to assist these vulnerable women with some of the more labor-intensive innovations like constructing latrines and building compost piles; and  Vulnerable Households Living in Collines with Insufficient Drinking Water: It was difficult for them to adopt many of the ENA and EHA practices. In the short-term, one of the most immediate problems will be for the MoH to work with its new and existing partners moving into both provinces to increase access to potable water. The lack of potable water is a critical constraint in 48.2% of the communities in Ruyigi and 54.9% percent of the communities in Cankuzo.113 Two other groups that staff noted as facing differential constraints in adopting the new BCC messages were:  Landless: Who represent from 7-10% of the population by most MAE estimates; and  HIV/AIDS-affected households. 3.5. Evidence of Impact on Local Institutional, Community, and Household Capacity114 3.5.1. Creation of a Base of Trained LMs. To date, the program has trained 4,920 LMs in both provinces. The program estimates that 60% of the 4,920 LMs are still actively involved with the program, in the sense that they still attend the Care Group meetings that Tubaramure organized during the program’s last quarter. Based on the focus group discussions, most LMs still practice the improved behaviors they were trained in and continue to extend some of these messages to their neighbors (Annexes IV and V). This trained force of volunteer LMs can continue to increase the effectiveness of the mainstream MoH community-based BCC programs. 3.5.2. Increased the Number of Women Organized Into Registered Groupements. The Tubaramure Program has helped created 874 groupements. In contrast to most of the other groupements supported by the two Provincial Directorates for Agriculture and Livestock 111 Staff estimates that the Batwa represent a small percentage of the population—estimated by most staff as between 2-5%. Historically the Batwa have been migratory with few permanent settlements and more limited integration into the national health system. 112 Staff reported that the collines with the highest incidence of migrant labor tended to be along the border with Tanzania. 113 Estimates are based on the initial qualitative classification of the collines that was conducted as a basis for choosing the collines for the interviews. 114 SOW Key Question 13: To what extent the implemented activities under IR2 and 3 have developed and strengthened the institutional, the community, and the households’ capacities? Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 64 (Direction Provinciale de l'Agriculture et de l'Elevage or DPAEs), the majority of the members of these groupements are women. Although many of these groupements are just getting organized, they have increased the capacity of local women to lobby for women farmers and to advocate for the types of services and input they need to build household food security (Annexes IV and V). As the Cankuzo commune agronomist stated during one of the Final Evaluation key informant interviews, “We now know they are there and that they are organized. This makes it easier to help them when new programs come. The trick, however, is to know that they exist and what crops or livestock activities they are interested in. Currently we don’t know where these groupements are and what they are working on except for the few we assisted on keyhole gardens and goat programs.” 3.5.3. Increased Women’s Ability to Lobby for the Health and Nutrition Issues That Concern Them. One direct, but heretofore very poorly documented, impact of the Tubaramure Program has been its impact on women’s empowerment and participation. This is an impact that was noted by both governors and most of the commune authorities that we encountered. Most attributed this empowerment to a combination of factors, but most importantly the IR2 model of teaching and home supervision visits through the LMs. Three indicators of the program’s wider impact on women, which were widely cited by the local authorities, were (Table 3.4):  A small but important number of Tubaramure LMs (196) have been elected to the Commune Development Committees (CDCs) that are charged with executing the recently adopted Commune Community Development Plans (Plans Communaux de Développement Communautaire or PCDC) documents;  537 LMs have been elected to the commune-level women’s forums; and  205 LMs petitioned for and were selected to be MoH Community Health Workers. Table 3.4. Number of Tubaramure LMs Who Have Become Community Health Workers, Commune Development Committee Representatives, or Members of the Women’s Forum Category of Involvement Cankuzo Ruyigi All Tubaramure LMs trained 1,991 2,929 4,920 LMs selected as MoH Community Health Workers 101 104 205 LMs elected to the CDCs 107 89 196 LMs elected to the gender forum 228 308 536 Source: Tubaramure Final Evaluation based on data provided by IR3 Coordinator for Tubaramure (Ruyigi) Ezéchiel Kabwebwe and Tubaramure Provincial Coordinator (Cankuzo) Edmond Twagirayezu; July and August 2014. 3.5.4. Increased Discussion of Community-Based Health and Nutrition Issues in the PCDC Planning Processes. In the past, there has been very little real discussion or analysis of the community-based health and nutrition issues that interested women in the PCDC. One explanation of this, put forth by the chief counselor of Cankuzo Province, is the limited involvement that women have had in these colline and commune-level planning processes. One unintended but obvious impact of the program—which was independently noted by each of the local administrators and province-level authorities that the evaluation team met with—has been to increase women’s willingness and ability (particularly the LMs) to engage in these colline￾level and commune-level planning processes. This increased participation was reflected in the content of the most recent generation of the PCDCs in the three Ruyigi communes where the Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 65 team had access to both the 2014 (second-generation) and 2009 (first-generation) PCDC documents.115 This increased visibility and mainstreaming of some of the Tubaramure achievements into the Ruyigi PCDCs was attributed to three factors:  The increased confidence that the LMs gained from their training and BCC activities;  The qualitative evidence from key informant interviews that many of the LMs have used their Tubaramure contacts (with the Tubaramure Health Promoters and the IR supervisors) to build their credibility with the local administrators of the communes and colline chiefs, who have dominated the PCDC planning process in the past; and  The provincial coordinator and the IR2 and IR3 supervisors for the Tubaramure Program in Ruyigi participated in the initial PCDC planning meetings. Their participation in these meetings helped the commune and colline administrators in Ruyigi to better translate some of the recommendations coming from the colline general assemblies into the technical terms needed for the final PCDC planning documents. 4.0. Factors Which Contributed to and Detracted From Program Relevance, Effectiveness, Efficiency, and Acceptability of Processes, Outputs, and Implementation 4.1. Relevance 4.1.1. Critical Importance of the IR2 Activities for Sustaining the Results Under IR1 and the Program’s Global Impact on Nutrition. The same IR2 activities were highly relevant because:  They supported a series of accepted and approved protocols the MoH adopted and wanted to achieve; and  They built the community capacity for the MoH, which led to some of its most important achievements in the key areas being targeted under IR1 (Table 3.5). 115 All three of the most recent PCDCs for the three communes that the team examined in Ruyigi included activities designed to support more diversified food intake and community-based programs to build and promote EHAs, whereas the previous PCDCs (from 2009) did not. The Tubaramure provincial coordinator for Ruyigi and IR2 and IR3 supervisors for Ruyigi participated in the initial PCDC planning meetings with the local commune administrators and development councilors is a contributing factor since it enabled them to better translate some of the recommendations coming from the colline general assemblies into the technical terms needed for the final document. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 66 Table 3.5. IR2 Activities that Affected the Community-Level Outcomes of IR Outputs in the Tubaramure Program IR1 Outputs and Outcomes IR2 Activities That Contributed to IR1 Outputs and Outcomes Outcome Indicators 1.1, 1.2, 1.3 for IR1116 -Short-term increases in desired health behaviors for prenatal counseling sustained due to mother’s understanding and wider cultural support -Short-term increases in postnatal counseling not sustained because of lower levels of appreciation and or cost/benefits of time needed to access the center level activities -Short-term increase in GM not sustained, but delayed demand created by training reflected in quick up-take in pilot programs for community-based GM in both provinces (see output 1.3 below) Output 1.1. Pregnant and lactating women access pre and postnatal care services IR2 training and LM support increased women’s understanding about the critical importance of prenatal care services (reflected in health center indicators) Output 1.2. Implementation of national IMCI plan is supported IR2 training and support from the LMs has increased the rate of children being referred to the health centers through the Community Health Workers before their illness has become critical (reflected in health center indicators) Output 1.3. Health facilities supported in providing GM -The fact that GM was a required activity to receive rations, combined with IR2 training and support from the Care Groups and LMs, created a short-term increase in the number of children participating in the health center-based GM (reflected in health center indicators) -Although this has not been sustained (is back to baseline levels), the increased community understanding of GM (from the Tubaramure training) is resulting in a very rapid understanding of the new community-based GM initiatives being pilot tested in 12 collines by the Tubaramure Program and eight collines by World Vision in Cankuzo 1.4. Severe acute malnutrition (SAM) is detected and referred for treatment -Training modules have increased the LM, beneficiary, and non-beneficiary mother’s ability to identify severe and acute cases of malnutrition and to refer them to the Community Health Workers for referral to a health center for treatment (reflected in health center indicators) -IMC’s willingness to pay the costs of transporting sick children and someone to accompany the child at the hospital (under IR1) has reinforced the communities’ willingness and ability to refer children to health centers Source: Tubaramure Final Evaluation interviews and literature review; July 2012 The Tubaramure MYAP anticipated that the program would contribute to the MoH’s ongoing consideration of the Care Group Model as a complement to its existing program. The principal reason for this was the Care Group Model was not a top priority for the MoH during this time period. Thus, although the technical lead for IR2 participated in the coalition117 of organizations within Burundi that were implementing Care Groups, little traction was gained. 4.2. Effectiveness The Tubaramure BCC strategy was highly effective in raising knowledge levels and promoting behavior change of the PM2A beneficiaries as demonstrated by the measureable changes in good practices, especially for hygiene, breastfeeding, and the utilization of health services for pregnant 116 Indicator 1.1. % of pregnant women completing at least three prenatal visits; Indicator 1.2. % of mothers completing at least 2 postnatal visits; and Indicator 1.3. % of children 9-36 months attending GM visits at least once in a two-month period. 117 This coalition included Concern, World Relief, CRS, and FH. The members met regularly and lobbied the MoH to recognize the Care Group Model. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 67 women and young children.118 There is qualitative evidence from the focus group discussions that linking PM2A programming rations made these behavior changes occur more quickly than they would have otherwise (Annexes IV and V). The BCC strategy was less effective in achieving its population-based impact. Its effectiveness was decreased by the fact that the BCC programs focused almost exclusively on the PM2A beneficiaries and their families, especially in the first three years. This directly affected the non￾beneficiary mothers’ understanding of the major BCC themes. The health/nutrition specialist on the final evaluation team identified a wide gap between the basic ENA and EHA knowledge of beneficiary mothers and non-beneficiary mothers. The model’s BCC effectiveness was further decreased by the fact that the program did not implement the gender strategy that was outlined in the MYAP proposal (Text Box 3.3). Because of this, there was insufficient implication of the beneficiary’s husbands—and men in general— during the first three years. 119 Based on emerging evidence that a main reason for beneficiary drop out was the lack of involvement of the beneficiaries’ husbands, the program added the concept of the Leader Fathers (LFs) in some Care Groups in 2011. After the mid-term evaluation, the IR2 technical lead worked in close collaboration with the other partners to develop a series of activities that targeted men and encouraged more participation of the beneficiaries’ husbands in the Care Groups. While this was helpful, it was too late to have a major impact on the results.120 Text Box 3.3. Gender Strategy from the Original Tubaramure MYAP Proposal “The program will encourage men to identify and take on day-to-day tasks that promote gender equity in the home and that foster a climate of shared responsibility. The program will rely on community-level strategies for involving men in the program to help break through gender barriers/issues that unfairly burden the women by:  Enlisting the support of husbands and male leaders for women’s initiatives; and  Utilizing men as agents of change. Training modules will focus on:  Practicing responsible fatherhood; and  Creating opportunities for men to learn the skills necessary to provide care and support to women.” Source: CRS; 2009. Tubaramure MYAP Proposal. Bujumbura: CRS. Pp. 20-21. 4.3. Efficiency and Acceptability of Processes and Program Outputs 4.3.1. Link with PM2A Food Distribution. One factor that affected the efficiency and acceptability of the Tubaramure BCC model was the direct linkage between that program and the 118 This is the same observation made during the mid-term (Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 40.) 119 SOW Key Question 15: Are there any factors (barriers/constraints) that limited community participation and engagement in the program implementation? 120 SOW Key Question 12: How effective is the nutrition and health sector at reaching fathers/men? What could be done in future programming to improve father/men’s participation in such sectors? How can this affect sustainability? Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 68 PM2A food distribution.121 All PM2A beneficiaries were required to participate in a beneficiary group with LMs, who became an important link between the program’s food tracking system and the beneficiaries. This link between PM2A and the Care Group diminished the voluntary aspect of the Care Group. Once the original PM2A beneficiaries and LMs graduated—i.e. were no longer required to attend the meetings as a condition for getting their PM2A rations—many of the LMs and beneficiary mothers dropped out because they did not see a concrete motivation (return) for participating. To encourage the PM2A graduates to continue participating in the Care Groups, the program developed a host of motivations for the mothers and beneficiaries in 2011. Since there was no budget for post-distribution motivations and the time was very limited, the program focused on a series of low-cost innovations that they could execute in a very short time period. The same post￾graduation activities created more work for the already overloaded Tubaramure Health Promoters, which further decreased the time they had available for training the LMs and making home visits. 4.3.2. The IR2 BCC Training Modules.122 When the program started, there were no standard training modules for the MoH Community Health Workers. This forced the program to develop its own modules and conduct formative research to ensure that the materials responded to both the local and MoH needs. The fact that the program was unable to build on pre-existing MoH modular materials created a host of execution delays. Another problem was the sheer number of modules, which became an issue once the MoH validated its first integrated module for training the Community Health Workers in August 2012, coinciding with Tubaramure’s mid-term evaluation (See Annex VI.A.2). Most of the MoH staff interviewed during the final evaluation felt that this Integrated Community Health Worker Training Module touched on every one of the themes that was included in Tubaramure’s five BCC training modules. Although the MoH approved the five draft modules for pilot testing, the modules have not yet been formally approved and validated for use by the MoH’s personnel. Although some of the MoH Public Health Technicians that were interviewed during the final evaluation said this did not bother them—i.e. that they would continue to support the LMs using the modules—some of the other technicians said they would not. Key challenges remaining are to develop a solid action plan for final review and validation of the modules with the appropriate MoH units. Given the need to harmonize these modules with the 121 Some of the lessons learned from the Tubaramure experience with adopting Care Groups are discussed in Annex 8 of this recent training module that was developed by Technical and Operational Performance Support (TOPS), Food Security and Nutrition Network Social and Behavioral Change Task Force. 2014. “Care Groups and the Preventing Malnutrition in Children under 2 Approach [PM2A] Annex 8.” In, Care Groups: A Training Manual for Program Design and Implementation. Washington, DC: Technical and Operational Performance Support Program. Annex http://fsnnetwork.org/resource-library/social-and-behavioral-change/care-groups-training-manual-program￾design-and-implement. 122 SOW Key Question 24: PM2A took a modular approach to BCC at the household and community level, and a curriculum was developed for improving health services, particularly at local health clinics. What was the impact of these modules on preventing malnutrition? Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 69 pre-existing (and validated) Integrated Community Health Worker Training Module, the MoH is likely to ask for some of them to be shortened or better harmonized with its integrated manual for the Community Health Workers. Given the long time delays that will be required, this is unlikely to happen before the end of November. If these activities are completed along with some sort of practical module that summarizes the lessons learned from the Tubaramure PM2A Program, it will contribute to the government’s ongoing discussion and review of the Care Group Model that is likely to occur over the next six months.123 4.3.3. High Labor Demands on the Tubaramure Health Promoters. Even if the five BCC training modules had been available at the time they were originally scheduled for, it would have been impossible for the Tubaramure Health Promoters to teach the beneficiary groups correctly given the competing demands on their time. Earlier studies conducted by the Tubaramure IR2 lead (FH) estimated that the maximum number of Care Groups that a Tubaramure Health Promoter could manage successfully would be 11,124 if his or her time was totally focused on their BCC activities (See Annex VI.B.5). By the mid-term, it was already clear that many of the Tubaramure Health Promoters had many more than 11 Care Groups to supervise, and that a host of other activities (food distribution, site visits, culinary demonstrations, and support for groupement and SILC activities) not anticipated in the original MYAP document required time that was time taken away from their cascade training activities. One Tubaramure Health Promoter in Ruyigi had 59 groupements to supervise on top of her other BCC activities under IR2, and one in Cankuzo had 125. Only four of the 24 Tubaramure Health Promoters that were included in the 2014 groupement census had under 20 groupements to support; 12 of the promoters had 21-35; and 8 had over 35.125 The issue of coordinating these different activities was complicated by the fact that the direct supervisors of the Tubaramure Health Promoters who were responsible for the IR2 BCC activities were not employees of the IR2 technical lead but rather CRS who was the technical lead for IR3. This heavy labor burden had a direct and measureable impact on both the time and quality of the Tubaramure Health Promoters’ training using the modules and home visits. There simply were not enough Tubaramure Health Promoters to cover such a large territory adequately with the competing demands placed on them and their decreasing numbers as the program is starting to wind down. One of the health promoters interviewed during the final survey currently covers 13 collines that are quite far from one another where he is responsible for 185 LMs, 26 groupements, and 17 SILCs, as well as supervising all the home visits. This huge workload from new activities decreased their efficacy in monitoring the beneficiaries, especially the LMs on 123 SOW Key Question 18: With regards to the preventative model, what contribution will the PM2A be able to offer to the government? 124 Actually, the recommended ration was 1:9. (see: http://www.caregroupinfo.org/docs/Care_Group_Criteria_November_12_2010.pdf). For budget reasons, the Tubaramure Program revised this figure to 1:11 (source: feedback during the final review of the draft final evaluation document). 125 Equipe de Suivi Evaluation CRS-Burundi; 2014. “Quelques Résultats du Travail mené sur les Groupements (Associations) de PM2A dans les Provinces de Ruyigi et Cankuzo” PowerPoint Presentation. (March-May 2014). Pg. 1. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 70 some of the more complex issues like identification and home-based management of childhood illnesses. 4.3.4. The Tubaramure Health Promoters’ Training and Technical Capacity. The program was careful to provide short basic trainings for new Tubaramure Health Promoters that were hired after the program’s baseline training; this training was usually done on a case-by-case basis by the IR supervisors. In addition to this, the promoters received an on-the-job lesson every two weeks from their supervisor that they were expected to repeat for their Care Groups. One major observation from health and nutrition expert Dr. Sidibe Sidikiba on the final evaluation team is that this amount of training was not enough given the highly technical nature of some of the nutritional training they were expected to give the LMs and technical backup they were supposed to provide the health centers. The Health Promoters performed well considering the constraints under which they were operating. The real issue is that it was unrealistic to expect one single polyvalent (multi-purpose) Tubaramure Health Promoter to provide technical backup to the LMs without strong technical backup and collaboration from the MoH’s community-level personnel (i.e. the MoH Public Health Technicians and the Community Health Workers). Had there been the same number of trained MoH Community Health Workers in the two provinces today (1,167) in 2008, it is clear that the Tubaramure’s IR2 activities would have been designed differently. Unfortunately, when the program was designed there were only 712 MoH Community Health Workers and the statute giving them full legal recognition and training was not finalized until late 2012. Thus, one cannot say that the high workload was the problem, it was simply a contributing factor given other design issues in the program start up. 4.3.5. Lack of Collaboration Between Technical Partners.126 A fifth factor that affected the efficiency and the acceptability of processes and program outputs under IR2 was the fact that the MYAP proposal did not include a clear plan for how the IR1 technical lead (IMC) would backstop and coordinate with the IR2 technical lead (FH) in order to facilitate the links between the IR2 BCC activities and the community-level MoH activities. Had this been done, it might have made it easier to build the MoH support for and understanding of the Care Group Model into the program earlier. The fact that this kind of clarification was not in the MYAP proposal probably can be attributed to the implicit assumption (by all of the partners involved in the initial design) that the MoH was on the cusp of formally recognizing the Care Groups and incorporating them into their core community intervention model. If the MOH had developed a formal protocol for the Care Groups, then the issue of building MoH support for and understanding of the Care Group Model would have never happened. 126 SOW Key Question 2: What are the factors that hinder/assist the effective integration of the program components? SOW Key Question 3: Are there steps that could have been taken to improve integration as well as food security impacts through greater integration? Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 71 4.4. Sustainability127 It is likely that the knowledge levels and certain practices (e.g. improved hygiene and sanitation feeding, optimal breastfeeding, utilization of health services, attendance at prenatal consultations, and dietary diversity) will remain high.128 It is also clear from the focus group discussions with 123 LMs and 223 PM2A beneficiary and non-beneficiary mothers in the collines, that the vast majority of the LMs are still important health and nutrition resource persons. During the last two years of the program, the LMs were the go-to people who connected sick children to Community Health Workers, who then connected them to the health center.129 They were also the go-to people for interfacing with Tubaramure IR1 provincial supervisors to get funds to pay for transporting sick children and supporting the adult who accompanied the sick child during his or her hospitalization. 4.4.1. Progress Toward the Execution of the IR2 Exit Strategy. 4.4.1.1. Current Status. The Tubaramure PM2A Program’s ‘sustainability strategy’ for IR2 was based on two critical assumptions:130  That the Care Groups would remain functional in the Tubaramure Program area; and  That the Care Group model would be taken to scale in the country. If these two assumptions held, the MoH would continue working through the Care Groups and LMs to sustain the BCC messages that were developed under IR2. Although the Government of Burundi (GoB) is still considering a formal Care Group protocol, this has not yet happened. Therefore, once the Tubaramure Program funding ends, the principal vehicle for all MoH communication with the collines will be the MoH Community Health Workers. Once this happens, the LMs will simply support these activities since they are not an official part of the MoH community health system. Whether or not the LMs will continue their BCC and Community Health Worker support functions in the future will depend a great deal on what they perceive as the benefits of investing their time and energy in these volunteer activities. These benefits will in part depend on if and how the MoH agents that supervise the MoH 127 SOW Key Question 22: How has the topic of sustainability of program interventions been dealt with under the program, including a) how sustainability was described/defined at the outset, b) whether there were flaws in the program design that would impact sustainability, c) what mid-course corrections were made, and what were missed, in enhancing program sustainability, and d) what lessons have been learned to improve the sustainability of future PM2A programming in Burundi and elsewhere. SOW Key Question 23: Related to the sustainability topic is how long-lasting are the interventions achieved under IRs 1 and 2? For example, do graduating families continue to apply the lessons they learned to children who a) have graduated, or b) were born after the mothers graduated? 128 The mid-term evaluation made the same observation (Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 40). 129 Many mothers and staff noted that the Tubaramure LMs were not able to write the script needed to refer a child to the health center. Some cited cases where a sick child was referred and denied treatment because they did not have a Community Health Worker-written referral. 130 CRS; 2009. Tubaramure MYAP Proposal. Bujumbura: CRS. Pg. 22. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 72 volunteer Community Health Workers make use of the LMs to support the other donor and non￾donor supported activities that are going to be executed in both provinces in the coming years.131 In July 2014, the Tubaramure IR1 lead (IMC) completed a detailed exit strategy for its activities in collaboration with the MoH that outlined clear responsibilities for handover and bridge activities to ensure effective turnover. Unfortunately, this exit strategy did not identify the how the IR2 activities that supported the IR1 activities would be handed over or sustained. To address this issue, the Tubaramure IR2 lead (FH) staff—working in close collaboration with IMC—organized two five-day training workshops on the five modules that were developed under IR2. This workshop was attended by all of the MoH Public Health Technicians in each province, as well as their immediate supervisors in the provincial offices of the MoH. The Ruyigi training took place over a five-day period from June 16-20, 2014; the Cankuzo training took place over the five-day period from July 7-11, 2014. Although some of the Public Heath Technicians participated in the baseline training of the MoH staff on the Care Group Model in 2009, due to high levels of staff turnover many of them had not. The principal objective of the training was to re-familiarize the MoH Public Health Technicians with the five modules that FH developed for its IR2 activities in the hopes that they will continue to support the LMs’ use of the modules. A secondary objective was to discuss various options for determining how the MoH technicians might build on the LMs’ trainings to support some of their existing and new activities with the volunteer Community Health Workers once the program funding ends in November 2014. Each of the provincial supervisors of the MoH technicians who were interviewed during the final evaluation said they saw the LMs as an incredible resource. To date, however, only a few isolated Public Health Technicians seem to have moved forward with the development of any sort of formal plan about how they might better connect the LMs and the MoH Community Health Workers. To develop these plans, each MoH Public Health Technician who supervises the 1,167 MoH Community Health Workers (who were trained to support the community-level activities component of the four MoH outputs supported under IR1) needs a list of LMs and beneficiaries, as well as a list of the assisted and non-assisted groupements and SILCs by colline. These records exist and can be extremely helpful in connecting the Community Health Workers and Public Health Technicians to the MoH. When the final evaluation team was in the field, all of the program’s records on the LMs were still in two separate locations:  The register that each Tubaramure Health Promoter keeps on the collines that he/she supervises (i.e. one register covers three to four villages); and  The Tubaramure Health Promoter records on the groupements and SILCs, which are also grouped by zone of intervention, which, like the register, regroups multiple villages. An additional source of data on LMs, groupements, and SILCs is the master database that one of the Tubaramure PM2A M&E officers has just completed on each groupement (see Annex VI.B.4), of this chapter for a list of categories in the database). 131 SOW Key Question 16: How are graduated and current beneficiaries in the community coping with the end of food distribution? Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 73 4.4.1.2. Critical Next Steps. During the two exit strategy workshops in Cankuzo and Ruyigi in June and July, the MoH and Tubaramure representatives decided to prepare this informal exit strategy in two steps. Step One: Joint Assessment of the Tubaramure Program’s Colline-Level Impacts (MOH/Tubaramure) (tentative scheduling July-August 30, 2014) Census of Tubaramure Colline Activities Executed Through Care Groups for All 269 Collines in the Program. Each IR2 supervisor for Ruyigi and Cankuzo committed to developing a list of LMs, the trainings they have received, and their current level of activity based on the information recorded in the register. This activity is scheduled for completion in August, after which there is a tentative plan for organizing a provincial-level meeting to discuss the information and to develop a formal handover plan. Joint Tubaramure Health Promoter/MoH Visits to Review the LMs’ Activities. During the same time period, there was a tentative agreement that each Public Health Technician would try to organize joint visits to the collines in order to better understand what they were doing and how they could work better with the volunteer MoH health workers. Although this is a laudable concept, it is difficult to put into practice. Although the Tubaramure Health Promoters (who have the records and know the communities) have motorcycles and a budget for fuel, not all of the MoH Public Health Technicians (who supervise the Community Health Workers) have bicycles, which limits their ability to attend meetings organized at the health centers. Step Two: Provincial Meeting for Handover of the Registers and Lists (tentatively scheduled for September) There was the implicit expectation in the June-July 2014 training sessions that the program would organize some sort of formal handover of the registers and colline-level information on the program’s activities in September 2014. This official hand over did occur for Cankuzo and Ruyigi on October 16, 2014 in Muyinga Province. The Bujumbura official hand over is schedule for October 23, 2014. 132 This is apparently occurring on a case-by-case basis. 4.4.2. Key Challenges the MoH and the DPAE are Likely to Face in Sustaining the Tubaramure Program’s Results Under IR2. The MoH and the DPAE are likely to face priority issues for follow-up on the Tubaramure Program’s IR2 activities in the coming year (Table 3.6). 132 Evariste Habiyambere, Food for the Hungry; October 15, 2014. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 74 Table 3.6. Priority Issues for Follow Up to Capitalize on the Results of the Tubaramure Program’s IR2 Activities Priority Issues for Follow Up For the MoH Over the Next Year For Other Donors Working in Health and Nutrition and Food Security in the Two Provinces Strengthen the LMs’ links to the MoH Community Health Workers and the Public Health Technicians Strengthen MoH Community Health Workers and Public Health Technicians to identify and work with the Tubaramure-trained LMs in ways that strengthen their existing and projected community-based programs Strengthen the Tubaramure￾facilitated groupements’ links to GoB and NGO programs to increase their food security and IGAs -Strengthen DPAE (provincial, commune, and colline-level) capacity -Capitalize on the existing knowledge and organization of the Tubaramure SILCs and groupements Source: Tubaramure Final Evaluation; July 2014. The first priority issue is to determine how to build a sustainable link between the LMs and other PM2A graduates who have benefitted from four years of BCC messages from the training and are clearly (based on the program’s quantitative data and the focus group discussions) practicing many of the malnutrition prevention behaviors designed by the program. This is an issue that directly affects the ability of the MoH to sustain the community-level impacts of the Tubaramure’s IR1 support, because there is a direct connection with the MoH’s mandated activities and strategies. The second priority issue for follow-up is to better link the Tubaramure-facilitated groupements to the MAE agents and donor and NGO-funded programs they need to increase their food security and income-generating activities (IGAs). This second challenge is critical because it is directly linked to the achievement of the program’s global goal of reducing malnutrition because:  It affects food access and availability of the LMs and PM2A graduates directly and, to a lesser extent, the other households in the community; and  It affects the LMs’ willingness and ability to continue carrying out the volunteer activities they were trained to execute through the Care Groups. Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 75 5.0. Lessons Learned and Best Practice Checklist Table 3.7. Tubaramure Program IR2 Lessons Learned and Best Practice Checklist Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide MoH Future Donor￾Funded Activities Future PM2A and Food Security Programs IR2: Households practice appropriate health and nutrition behaviors. Lesson 1. Care Groups can be a powerful model for promoting the types of broad-based behavior change that are needed to sustain the short-term nutritional impacts of a PM2A program. 1. Consider how the program’s CG structure will be integrated into the existing health systems from the start to ensure its sustainability once program funding ends. X X 2. If the national health system does not recognize the Care Group Model, consider developing the model as a complement to other activities (like the Hearth Model and community-based GM) that are recognized by existing or draft protocols; and continuing to work with the MoH to determine if and how the concept could be better recognized by the existing protocols both during and after the program. X X 3. Participate in national forums that review various ways that the Care Group Model can strengthen the existing models for community-level BCC. X X X 4. Avoid any sort of direct linkage between the Care Group Model and eligibility for PM2A rations in order to keep the Care Group voluntary and to ensure more broad based participation. X 5. Complement the Care Groups with mechanisms to reach the wider society with consistent BCC messages that build the wider community’s support and understanding of the messages. X X Lesson 2. Care Groups require careful training and retraining of the implementing staff and beneficiary LMs to be effective BCC agents both during and after the program. 6. Consider adding other staff to deal with non-BCC issues (like IGAs and food distribution) and/or reducing the area of intervention to ensure appropriate backup support. X X Lesson 3. Encourage Care Groups to develop IGAs in order to sustain their BCC activities over time. 7. Anticipate the costs of developing IGAs and/or SILCs to support the LMs developing IGAs from the start of the program. X X 8. Identify the partnerships to develop and sustain the LMs’ IGAs in the initial design, and monitor them so they can be adjusted as the most viable IGAs are identified. X X 9. Budget adequate staff/partner time needed to support these activities so that the IGA activities do not detract from Care Group and BCC trainings. X X Lesson 4. Anticipate the need for a gender-sensitive communication strategy that develops a wide base of community support and understanding for the BCC messages, both during and after a PM2A program. 10. Include a well-thought-out draft gender strategy in all PM2A proposals that anticipates some of the special challenges associated with PM2A that are not found in more conventional food security programs. X Tubaramure Final Evaluation. Chapter 3: IR2. October 22, 2014. Final. 76 Lessons Learned, Best Practices, and Recommendations Ruyigi and Cankuzo Worldwide MoH Future Donor￾Funded Activities Future PM2A and Food Security Programs 11. Ensure that this strategy complies with the gender strategy of the funding agency (like USAID), as well as donor expectations for gender monitoring and reporting (only recently adopted).133 X 12. Ensure that the program gender strategies are compatible with the national gender strategy, and provincial and commune-level coordination structures. X 13. Conduct an annual review of each PM2A program’s gender strategy as part of the routine annual review and planning processes. X 14. Include internal indicators for the gender strategy that are tracked as part of the routine internal and donor tracking of the program, even if these are not in the Indicator Performance Tracking Table (IPTT). X 15. Have a field-based gender specialist in each program intervention zone that can also function as the local capacity building and M&E officer. X 16. Budget basic gender training and retraining of staff and all local government and NGO partner staff the program works with. X Source: Tubaramure Final Evaluation; July-August, 2014. Revised based on feedback to the first draft, September 22-October 5, 2014. 133 The USAID gender strategy was formally adopted in March 2012 and updated in 2013 in the USAID Automative Directive System (ADS), Chapter 205 (aka ADS 205) (Integrating Gender Equality and Female Empowerment in USAID’s Program Cycle, New Edition Date: 07/17/2013, Responsible Office: PPL File Name: 205_07171307/17/2013 New Edition), which outlines the expectations for staff integration of the policy into existing and future USAID-funded programs. Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 77 Chapter 4 Intermediate Result 3: Eligible Women and Children Have Increased Intake of Diverse Food 1.0. Global Strategy The goal of the Tubaramure activities under Intermediate Result 3 (IR3) was to promote the increased food diversity and intake needed to reduce both provinces’ high levels of malnutrition. The IR had three expected outputs, each with a respective set of activities (Table 4.1). Table 4.1. Major Outputs that were Designed to Achieve the Tubaramure Program’s IR3 IR3. Eligible women and children have increased intake of nutrient-rich, diverse foods. Output 3.1. Food For Peace (FFP) rations distributed to eligible women and children at community level. Output 3.2. Mothers and children use FFP rations appropriately. Output 3.3. Households use appropriate local foods in addition to FFP ration. Source: Catholic Relief Services (CRS); 2009. Tubaramure Multi-Year Assistance Program (MYAP) Proposal. Bujumbura: CRS. The short-term impact of the food distribution and culinary demonstrations under Outputs 3.1 and 3.2 was to increase dietary diversity and intake during the program for the beneficiary mothers.134 One of the conditions for getting rations was participation in some of the IR1- supported health services and the IR2 beneficiary group activities. Over the longer term, the program was expected to stimulate local demand for increased food intake and a more diversified diet using local foods through a series of activities under Output 3.3. Although the short-term focus of the activities was on the Preventing Malnutrition in Children Under 2 Approach (PM2A) beneficiaries, the program was expected to have a population-based impact on dietary diversity in both provinces. Since most of the original PM2A beneficiaries have already graduated (Figure 4.1), this evaluation focuses on the activities under Output 3.3 and some of the cross-cutting lessons learned from Outputs 3.1-3.3 for PM2A programs that are hoping to have a population-based impact on household dietary diversity. 134 SOW Key Question 18: With regards to the preventative model, what contribution will the PM2A be able to offer to the government? Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 78 Figure 4.1. Evolution of Tubaramure Beneficiaries Not Yet Graduated from the Program, 2009-2014135 Source: Tubaramure M&E Office; August 2014. 1.1. The Original IR2 Strategy To achieve Output 3.1, the Tubaramure Multi-Year Assistance Program (MYAP) Proposal outlined an efficient system for commodity distribution and targeting. The activities under Output 3.2 focused on:  Working through the Leader Mothers (LMs) to conduct community awareness campaigns to explain the program’s goals, eligibility criteria, duration for PM2A rations, and how these would be reinforced during home visits;  Working through the Ministry of Health’s (MoH’s) commune-level personnel and community leaders to reinforce the rationale for rations; and  Working through the Tubaramure Health Promoters to conduct spot checks of the household ration buckets to monitor the level of commodity utilization and verify awareness of who is targeted in the household. The activities under Output 3.3 focused on increasing household consumption of nutritious food by getting fathers to serve the food to their children rather than selling it. The original MYAP sub-strategy for Output 3.3 focused on:  Developing a strategy for promoting men’s involvement in maternal and child nutrition;  Recipe demonstrations incorporating local foods (including animal products) appropriate for pregnant and lactating women and children under 2; 135 The official total number of direct beneficiaries is listed as 49,650. These figures indicate the number of beneficiaries that have not yet graduated from the PM2A distribution. 8695 16367 13251 3929 354 15346 26459 29278 7359 581 24041 42826 42529 11288 935 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 July 2010 July 2011 July 2012 July 2013 July 2014 Cankuzo Ruyigi Total Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 79  Identification of recipes by Tubaramure and MoH nutrition experts in consultation with LMs;  Promotion of local foods;  Raising farmers’ awareness of increased demand for nutritious foods for pregnant and lactating women and children under 2;  Pilot tests of food preparations that can be used by households once Title II has ended; and  Promotion of seeds and small animals by Catholic Relief Services (CRS), “under its cost share to a small population of the most highly vulnerable beneficiaries with extremely limited access to complementary foods.”136 2.0. Activities137 2.1. Program Start Up to Mid-Term 2.1.1. Output 3.1: Food for Peace (FFP) Rations Distributed to Eligible Women and Children at Community Level. In keeping with the Tubaramure Program focus on preventing malnutrition, the program targeted all pregnant and lactating women and mothers of children under 2 regardless of nutritional status. To ensure that the local population understood the reasons this was being done, the program started with an intensive public awareness program that targeted all of the provincial, communal, and local-level chiefs. The fact that all of the Tubaramure Health Promoters were recruited from the local communes and continued to be based in the commune centers helped facilitate this communication process. The fact that all of the commune and provincial-level authorities were invited to the quarterly coordination meetings was another example of best practice (see Annex VI.A.5). The strong, consistent implication of the local leaders in turn helped minimize commodity theft and illegal sales—often by the women’s husbands—of the commodities. The same consistent pattern of communication with the local authorities helped minimize the inevitable social conflicts that resulted from only a portion of the households in a given colline being eligible for rations, and some collines being excluded from the program because they were the International Food Policy Research Institute/Food and Nutrition Technical Assistance Project (IFPRI/FANTA) test villages. During the first two years of the program, the principal mechanism for executing the activities was through Tubaramure Health Promoters. The health promoters were the direct interface between the warehouses and mobile distribution points and the mothers. They were also responsible for monitoring the mothers’ eligibility and making notes about any shifts in eligibility and rations, such as the child reaching 2 years of age. All of this was done by hand, time consuming, and easily susceptible to human error.138 136 CRS; 2009. Tubaramure MYAP Proposal. Bujumbura: CRS. Pg. 18. 137 SOW Specific Objective 1: Are the health component activities being implemented according the Detailed Implementation Plan (DIP)? What are the obstacles and the delays observed? What recommendation can be done for future program? 138 Most of the Tubaramure Health Workers estimated that this activity consumed the equivalent of about a week of each month during the first two years of food distribution (2010-2011). Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 80 A major turning point was the arrival of a new monitoring and evaluation (M&E) director at the end of the first year. Once he realized how labor intensive the system for documenting the households’ eligibility was, he started working on a new database that would computerize the commodity targeting and reporting. This new system, launched in November 2011 just prior to the graduation of the first 8,000 beneficiaries, had a huge impact on the efficiency of the commodity system by making it possible to have real-time printouts for the warehouse and mobile distribution units, and reducing the amount of time that the Tubaramure Health Promoters needed to compile their reports from about a week to a few hours. This user-friendly—very simple to create and update-database system is an example of a best practice that other PM2A programs should emulate. 2.1.2. Output 3.2: Mothers and Children Use FFP Rations Appropriately. A great deal of effort was expended in the first year of the program explaining the rations, their purpose, who was eligible, what quantities were to be distributed to each category beneficiary, and what the conditions were for continued eligibility. To ensure that the corn-soy blend (CSB) and vegetable oil were correctly prepared, Tubaramure developed and disseminated recipes in both Kirundi and French, and periodically organized cooking demonstrations, sometimes including local foods in season along with the CSB and vegetable oil.139 These culinary demonstrations followed the same cascade training model used under IR2, with the IR3 technical advisor training the Tubaramure Health Workers, who trained the LMs, who then trained the beneficiary groups. To facilitate this process, the IR3 technical lead (CRS) developed a recipe book, which was approved by the MoH for use in the local communities. This cookbook included recipes for some of the nutritious dishes and weaning foods using the FFP rations, as well as foods that could be produced and/or purchased locally. 2.1.3. Output 3.3: Households Use Appropriate Local Foods in Addition to FFP Ration. 2.1.3.1. Culinary Demonstrations. From the start, culinary demonstrations emphasized the use of locally grown nutritious foods for family consumption and children’s weaning broths. Based on the final evaluation focus group discussions, the culinary demonstrations had a major impact on the diets of both beneficiary and non-beneficiary (i.e. non-PM2A) recipients (Annexes IV and V). Many of the women interviewed emphasized that these dietary changes were encouraged by the enthusiastic scale up of the keyhole gardens, as well as increased income from the Savings and Internal Lending Communities (SILCs). One of the most frequently cited constraints to continuing these new dietary practices was the limited household access to improved seed. 2.1.3.2. Savings and Internal Lending Communities. Using its cost share, the CRS SILC specialist for the MYAP program in Kayanza, Kirundo, and Muyinga provinces trained all of the Tubaramure Health Promoters in the design and promotion of SILCs over a 16-month period between August 2010 and November 2011.140 The concept of SILCs was new to both provinces, 139 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 43. 140 The original proposal anticipated the use of the cost-share funds to purchase livestock and seeds. Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 81 but built on the traditional concept of the saving tontine141 (Text Box 4.1), yet took off rapidly with no major problems. By 2011, the Tubaramure Program had created 61 SILCs; by the mid￾term evaluation, there were 439.142 During this time, the principal participants in the SILCS were the LMs and PM2A beneficiaries although some non-beneficiary women were in the groups even from the start. 2.1.3.3. Initial Seed and Livestock Distributions. In early 2012, the program provided gardening seeds, fruit trees, and poultry to four groupements that had been organized by LMs (see Table 3.2 in Chapter 3). The principal justification of these distributions, which were not anticipated in the MYAP proposal, was to motivate these LMs to remain involved in the program after graduation. Although the distributions were appreciated, there were several cases in which these efforts were negatively influenced by the poor, sub-standard quality of the seed, plants, and poultry that was distributed, and poor timing of the distributions (due to the difficulty of accessing certified seed and the premature crop season) that decreased the effectiveness of this activity.143 Most of these agricultural input issues were corrected after the mid-term evaluation when the program was able to conduct longer-term planning for these ‘new’ activities, which were not anticipated in the original MYAP proposal or program budget. 2.1.3.4. Keyhole Gardens. During the same time period, CRS facilitated some of the core Tubaramure staff attending a training workshop at a CRS program in Lesotho and made the first pilot tests in early 2012. The principal justification of this new activity (which was not anticipated in the MYAP proposal) was to help the beneficiary households develop an easy-to￾access source of the vegetables needed to maintain a more diversified diet. By the end of April 2012, there were already 119 gardens—evidence that non-beneficiaries were replicating the gardens—and demand for the Tubaramure Health Providers to help people construct them.144 141 A ‘tontine’ is a traditional form of rotating group savings that is widespread in many parts of Africa. A group is usually comprised of relatives or friends. The group agrees on a specific amount for each member to contribute and a specific day for the contribution, as well as the total number of participants in the scheme. On the agreed date, the members meet and pool the money together and one of them takes the total amount. The chosen member can be selected by lottery or the group can decide to allocate it to someone specific because of urgent matters affecting that person. No member can be selected twice in the same cycle. Before the entry of formal microfinance institutions, microlending existed through these tontines, as well as informal moneylenders and cooperative associations. 142 Tubaramure IR3 National Coordinator Regine Pacis Nohoreho; July 2014, based on reports by Tubaramure IR3 Coordinator (Ruyigi) Ezéchiel Kabwebwe and Tubaramure Provincial Coordinator (Cankuzo) Edmond Twagirayezu. 143 Key informant interviews during the final evaluation and Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 44. 144 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 45. Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 82 Text Box 4.1. The Concept of the Savings and Internal Learning Communities CRS has a long and successful history of group-based rotating savings and credit associations, known as SILCs, where 15-30 people agree to first pool a set amount of money each week and, after one month, start distributing small credits to one/some members to use for up to three months at 10% interest until everyone has had a chance to borrow and reimburse until three months before the end of a maximum cycle length of 12 months. In these last three months, credit is not issued, the reimbursements are all collected, and each member gets a dividend proportional to their equity called a share out. After the payout, the group then starts another cycle, sometimes with slightly different rules based on consensus and lessons learned, and usually with a higher weekly contribution. All groups have two funds, one for productive loans (now with contributions about $1/week) and one for personal emergencies (with contributions of about $0.20/week). A few groups have a share out after eight or nine months to coincide with the season for peak cash need (August/September for school-related expenses). Source: Ali Aamoum, Merella Mokbel Genequand, Bernard Crenn, and Mike Devries; 2012. MYAP Final Evaluation Report. Bujumbura: CRS. July 31, 2012. Pp. 45-35. 2.2. Principal Conclusions and Recommendations From the Program Mid-Term145 146 2.2.1. Output 3.1: FFP Rations Distributed to Eligible Women and Children at Community Level. A thorough review of the Tubaramure Program’s pipeline and the implemented system for ensuring the timely and effective distribution of Title II commodities was conducted during the program’s mid-term evaluation (Text Box 4.2). For this reason, the approved scope of work (SOW) did not include any specific objectives or key questions focused on this element of the program.147 145 SOW Specific Objective 8: How the mid-term evaluation recommendations were taken into account for improving the program implementation? 146 SOW Key Question 5: To what extent recommendations from the mid-term evaluation have been incorporated into the nutrition and health sector? 147 During the first year of distribution (2010), the program had already reached 95% of the eligible women and child pairs, which represented 119% of its original target (see Annex II, Tubaramure Indicator Performance Tracking Table [IPTT]). This evaluation was effusive about the effectiveness and efficiency of the system (Text Box 4.2). The same IR3 food distribution system was lauded by the IFPRI/FANTA staff interviewed during the final evaluation for its effectiveness in getting the right rations to the right individuals in a timely manner. It is critical to underscore the huge complexity of this part of the program since the rations needed to be adjusted for whether the woman was pregnant or lactating, and include a protective ration that had to be adjusted to the size of the family. The principal recommendations for the mid-term were: 1) minor adjustments to the program (recommendations 1-8, 11-12); and 2) working with the community, the husbands, and other volunteers to construct shelters, latrines, and simple handwashing stations using the same principles as those taught to the women and households in the communities (recommendations 9 and 10). Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 83 Text Box 4.2. Mid-Term Evaluation Assessments of the Tubaramure Pipeline and Implemented Commodity System (IR 3, Output 3.1), June 2012 “The commodity management system for the Tubaramure Program has no threshold issues. It is evident that the key players in this component are very knowledgeable about Title II food commodities management principles and regulations. Some of the recommendations and suggestions in this report are already being carried out. With good coaching and closer monitoring of the warehouse managers and the field sites, CRS/Burundi (the technical lead for IR 3) will make its system a stellar model with good practices that others can replicate.” “For Output 3.1, a smoothly functioning commodity supply chain ensures that eligible women and children are receiving the rations as planned. The combined efforts of local authorities, the LMs, and the Tubaramure staff— especially the Tubaramure Health Promoters—ensure that the Title II rations are being used appropriately.” Source: Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pp. 45 and 69. 2.2.2. Output 3.2. Mothers and Children Use FFP Rations Appropriately. The mid-term evaluation report concluded that the program’s extensive investment in explaining the logic behind the rations had paid off in terms of high levels of beneficiary understanding, and that the program’s decision to execute a protective ration was justified and completely necessary.148 2.2.3. Output 3.3. Households Use Appropriate Local Foods in Addition to FFP Ration. Based on the focus group discussions during the mid-term, the mid-term evaluation team concluded that, “everyone seemed to know well…the importance of a balanced diet. Beneficiaries and non￾beneficiaries, men as well as women—all could list local products for the three main food categories; but whether the outcome—increased consumption of local food—will be achieved is another question.”149 150 The original proposal anticipated that, “as part of the formative research,…the project will look at the role of fathers in the feeding of children. Based on the findings the project will develop a strategy for promoting men’s involvement in maternal and children (sic) nutrition.” 151 Another weakness observed at mid-term was the program never executed the formative research that was supposed to look at the role of fathers in feeding their children as a basis for promoting men’s involvement in maternal and child nutrition.152 153 148 The mid-term evaluation team concluded, “It is simply not acceptable or feasible for a mother not to share food with other family members. In this respect, the protection ration definitely contributes to the overall program goal of preventing malnutrition in children under 2.” (Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 44). 149 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 47. 150 SOW Specific Objective 9: To what extent program activities were financially supported, and how will they be sustained at the institution and the community levels after the program closure? 151 CRS; 2009. Tubaramure MYAP Proposal. Bujumbura: CRS. Pg. 17. 152 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 44. The program did, however, develop a module on gender issues that it used to train staff and beneficiaries starting in June 2011 (Source: Feedback on the draft Tubaramure Final Evaluation, September 22, 2014). 153 SOW Key Question 12: How effective is the nutrition and health sector at reaching fathers/men? What could be done in future programming to improve fathers’/men’s participation in such sectors? Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 84 The mid-term evaluation report made six recommendations to strengthen the achievements of the outcome indicators for Output 3.3: 154  Continue to work on a gender action plan that would involve men more fully in promoting maternal and child nutrition;  Determine the feasibility and potential effectiveness of adding activities to increase the availability and access to food in the program;  Continue to explore the possibility of developing a substitute for CSB and vegetable oil;155  Continue to expand keyhole gardens and other agricultural and animal husbandry activities already initiated with the Ministry of Agriculture and Livestock (Ministère d'Agriculture et d'Elevage or MAE);  Examine the current budget to see if cost centers (budget lines) can be adjusted to provide more resources for food production and/or SILC activities; and  Complete the development of a plan to track women and children who have graduated from the ration-distribution component. 2.3. Impact of the Mid-Term Evaluation Recommendations on IR3 Strategies and Activities and Integration with the Program’s IR2 Activities The program’s response to these recommendations focused on (Table 4.2):156  Gender Strategy: Developing a gender action plan to increase men’s involvement in and support for the Tubaramure Program’s activities  Income-Generating Activities (IGAs): Developing a series of activities for helping the LMs develop economic groupements (groups), keyhole gardens, and SILCs that they would need to sustain a more diversified diet. 154 154 Kathy Tilford, Ange Tingbo, and Vera Bensmann; 2012. Mid-Term Evaluation Report for the Tubaramure PM2A Program. Bujumbura: CRS for Tubaramure. Pg. 49 155 Unfortunately, the team did not gather very much information on the follow up to this recommendation about what worked and did not work, under PM2A. 156 Leonie Niyonkuru; 2012. Etude sur la Pérennisation des Care Groups et les Activités Genre dans le Programme Tubaramure. Bujumbura: Tubaramure. (November 2012). Pp. 22-23. Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 85 Table 4.2. Link Between Mid-Term Evaluation Recommendations and Key Groups of Follow-Up Activities Mid-Term Evaluation Recommendations Gender Strategy and Training Income-Generating Activities Organization, Technical Training and Material Support to Develop Groupements Seed and Livestock Distributions to Groupements Expansion of the Number of SILCs Develop Keyhole Gardens Livestock Distributions to Groupements Other Continue to work on a gender action plan that would involve men more fully in promoting maternal and child nutrition X Determine the feasibility and potential effectiveness of adding activities to increase the availability and access to food in the program X X X Continue to explore the possibility of developing a substitute for CSB and vegetable oil157 Activities undertaken but not a focus of evaluation Continue to expand keyhole gardens and other agricultural and animal husbandry activities already initiated with the MAE X Examine the current budget to see if cost centers (budget lines) can be adjusted to provide more resources for food production and/or SILC activities X X X X X X Complete the development of a plan to track women and children who have graduated from the ration-distribution component Not done Source: Tubaramure Final Evaluation in response to a request from the United States Agency for International Development (USAID) reviewers; received October 10, 2014. 157 Unfortunately, the team did not gather very much information on the follow-up to this recommendation about what worked and did not work, under PM2A. Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 86 Based on the recommendations from the mid-term evaluation findings for IR2 and IR3, the program supported two barrier-to-behavior-change studies in 2012, one of which focused on identifying a revised strategy for increasing men’s involvement in the program and sustaining the Care Groups. This same study identified a strategy for helping the beneficiary groups develop the types of registered economic groups (groupements) needed to get more help from the MAE 2.3.1. Rapid Increase in the Number of Groupements Starting in 2012. The Tubaramure Program formulated its first formal groupement strategy in early 2013. By 2014, the program had facilitated the creation of 874 groupements. 158 Given the limited number of resources the program had to help the groupements, it tried to limit its direct assistance (seed, technical assistance, etc.) to two groupements per colline. As of August 2014, 434 of the groupements have been assisted, and an estimated 713 (82%) of all the groupements are registered.159 Most groups are mixed, involving both beneficiary fathers as well as other men in the community. 2.3.2. Provision of Seeds and Livestock to the Groupements. To support the groupements’ activities, the program helped them access seed and small livestock starting in 2012 (Chapter 3, Table 3.2). Certain groupements were also given goats in 2012, 2013, and 2014 (Chapter 3, Table 3.2). By 2014, all of the assisted groupements had received at least one pair of goats using a goat solidarity chain approach.160 The initial beneficiaries in each groupement were expected to then ‘rotate’ a percentage of the offspring from this initial stock of animals to the other members in the groupement. Many of the groupements were also given soybean seeds to pilot test using a model developed by CRS/Rwanda (Chapter 3, Table 3.2). Based on the initial enthusiasm for the concept, the program distributed a full slate of equipment to promote soy production to the 434 assisted groupements in August 2014.161 Although some of the commune and provincial-level veterinarian specialists helped select the animals for distribution, this collaboration was not part of any formally negotiated collaboration with the MoH since this was not anticipated in the original program design.162 2.3.3. Steady Expansion of the SILCs and Training of SILC Private Service Providers. The number of SILCs increased exponentially starting in 2012 (Figure 4.2). 158 For more information on the evolution of the Tubaramure-supported groupement activities, see Chapter 2. 159 Thaddee Niyonzima, Tubaramure M&E Office. August 28, 2014 from the program’s groupement database. 160 Only two of the many groupements created by the LMs in each colline were assisted by the program due to resource constraints. For more information, see Chapter 3. 161 SOW Specific Objective 4: Are provided equipments appropriate, well dispatched, and well used for reaching the expected results? 162 SOW Key Question 4: Is the nutrition and health sector fitting into the local government’s strategy and priorities? What has been the level of coordination/collaboration with the Government of Burundi and other actors? Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 87 Figure 4.2. Evolution of Tubaramure-Facilitated SILCs in Ruyigi and Cankuzo Provinces, 2010-Present Source: Tubaramure IR3 National Coordinator Regine Pacis Nohoreho, based on the field reports of IR3 Coordinator for Tubaramure (Ruyigi) Ezéchiel Kabwebwe and Tubaramure Provincial Coordinator (Cankuzo) Edmond Twagirayezu; July 2014. Updated September 22, 2014. Given the high level of demand for SILCs, the program started working on various measures to make the SILCs more self-sustaining and to reduce the direct involvement of the already overworked Tubaramure Health Promoters (Chapter 3, Section 4.3.3; and Annex VI.B.5). This training was very rigorous: Out of the 96 persons who attended the SILC training-of-trainers workshop over three days, only 48 passed the test to become SILC Private Service Providers (PSPs), with the other 48 recruited as their assistants. Although the focus group discussions and key informant interviews suggest that some of the SILC PSPs are no longer working, many of them continue to train new groups and support existing ones. In theory, the SILC PSPs receive a small honorarium and support for their transportation costs from the SILC groups they support. Two SILC PSPs interviewed said although they had never received a cash payment for their services,163 they had benefitted from ongoing relationship with the SILC groups since they buy seed and other products from their other commercial enterprises.164 The beneficiary and non-beneficiary women interviewed in the final evaluation focus group discussions cited a wide range of benefits accrued by their families from their participation in the SILCs, including loans that they could use to improve their agricultural activities, provide extra food, and pay their families’ school and health fees (Text Box 4.3). 163 The SILC PSPs stated that they sometimes received gasoline for their motorcycles. Both SILC PSPs indicated that this pattern was typical, but the evaluation team was unable to verify this. 164 This was not a face-to-face interview, but based on case studies of specific PSPs that were conducted by the IR2 coordinator at Cankuzo. 29 32 61 175 264 439 288 317 605 425 444 869 0 100 200 300 400 500 600 700 800 900 1000 Cankuzo Ruyigi Total 2011 2012 2013 2014 Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 88 The final quantitative survey estimated 32% of all women in the two provinces are currently involved with SILCs, and 11-13% of the households have a man involved in a SILC.165 Text Box 4.3. Testimonials by Tubaramure SILC Members on Their Food Security, Income-Generating Activities, and Quality of Life I'm Nsavyimana Gaspard; I’m 42 years old. I’m married with six children, three sons and three daughters. Before the Tubaramure Program settled down in our community to teach us SILC activities, I was not saving, and I was the first in our community when it came to poorly managing the family resources. I often drank beer forgetting that I had to save money for the family’s needs. I was unable to pay the school fees for my children. I did not have any livestock at all, not to mention clothes. I walked barefoot. I approached the Tubaramure Health Promoter when I heard about some of the lessons that the program was teaching to the people of our community about SILC activities. I asked them if I could become a member of this group, and they accepted. They explained about all of the activities they do in the SILCs in addition to mutual help. By joining a SILC, I followed their program. I made monthly contributions. I contracted a loan, which was intended to buy seeds and chemical fertilizers for my fields (a loan of BIF 20,000 that I paid back with BIF 22,000). My second loan was for BIF 50,000, to buy school materials for my children. I paid the loan back at BIF 55,000 within two months. After this, I continued to save until we shared dividends. My share of the dividend was used for the purchase of a rooster and a hen. Now I have six chicks. I am waiting for them to grow and lay eggs. I want to sell the eggs to earn more money for my family’s needs. I'm Josephine Ntahondereye, age 50. I spent two years in the SILC group. Before I joined the SILC, I was in another association where I received very few benefits. The members of this association bickered/quarreled all the time, so I joined the SILC group. My husband is paralyzed. He cannot help me at all. I am only looking for what my family needs to survive. When the new school year approaches, the SILC groupement lends me money and I pay for my children’s school fees, their uniforms, and notebooks. Recently, my son was married, and my colleagues gave me the BIF 50,000 that he needed for the dowry. I also asked for a loan from the SILC to buy chemical fertilizers for my field. If I had not had the loan, the harvest would have been bad. Prior to being in a SILC, I had a straw house. After the group shared the dividends, I bought corrugated sheets for my house. We are now safe from the rain. I also bought a goat. I currently have two goats and a pig. Source: Annex V, Testimonials 10 and 12. 2.3.4. Rapid Increase in Keyhole Gardens and Nutrition Posters. In just three years (2012-2014) the program helped facilitate the creation of 28,117 keyhole gardens, especially in Cankuzo (Figure 4.3). The quantitative final survey reported that 44.2% of the households interviewed in the final quantitative survey reported having keyhole gardens—36.8% in Ruyigi and 56.6% in Cankuzo.166 165 Institut de Statistiques et d'Etudes Economiques du Burundi (ISTEEBU); 2014. Enquête Ménage pour l’Evaluation Quantitative Finale du Programme PM2A—Tubaramure (Provinces Cankuzo et Ruyigi). Rapport Final. Bujumbura: ISTEEBU. Pp. 43-44. 166 ISTEEBU; 2014. Enquête Ménage pour l’Evaluation Quantitative Finale du Programme PM2A—Tubaramure (Provinces Cankuzo et Ruyigi). Rapport Final. Bujumbura: ISTEEBU (August 2014). Pg. 40. Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 89 During the same time period, the program distributed a large number of posters promoting some of the local vegetables that could be grown on the keyhole gardens to complement the Essential Nutrition Actions (ENA) and Essential Hygiene Actions (EHA) posters that were developed under IR2. The posters were distributed to all of the local authorities and given to both beneficiary and non-beneficiary families; they were also very prominently displayed at food distribution sites. 43.6% of the households reported that they still displayed one of the posters (50.1% in Cankuzo and 39.7% in Ruyigi). The posters were also prominently displayed in the offices of most of MoH and MAE personnel and local authorities that the team interviewed. This was a low-cost publicity bonanza that future programs might consider replicating. Figure 4.3. Evolution of Tubaramure-Facilitated Keyhole Gardens in Ruyigi and Cankuzo Provinces, 2010- Present Source: Tubaramure IR3 National Coordinator Regine Pacis Nohoreho, based on the routine reports of Tubaramure IR3 Coordinator (Ruyigi) Ezéchiel Kabwebwe and Tubaramure Provincial Coordinator (Cankuzo) Edmond Twagirayezu; July 2014. Updated September 22, 2014. During the final evaluation focus group discussions with members of the groupements and LMs, beneficiaries cited the keyhole gardens as being one of the major program innovations that increased their household dietary diversity (Text Box 4.4; Annexes IV and V). 194 636 820 11380 5979 17359 18932 9185 28117 0 5000 10000 15000 20000 25000 30000 Cankuzo Ruyigi Total 2012 2013 2014 Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 90 Text Box 4.4. Tubaramure Beneficiaries’ Testimonials on the Impact of Keyhole Garden Plots on Their Household’s Dietary Diversity A non-PM2A beneficiary mother who adopted a keyhole garden: One day, I brought one of my sick children to the hospital for tests. The day after, I got the results, and the doctor told me that my child was suffering from malnutrition. I asked for their advice; they suggested I give him vegetables and eggs although I had no money. When the program came in our village, the Tubaramure Health Promoters began to educate us on growing vegetables, both during the dry season and the rainy season, as well as on digging garbage pits. Two months later, I saw a group of 20 people who cultivated vegetables during the dry season at home. I asked them if I could be part of their team, and they accepted. I then set up a vegetable garden around my house during the dry season. After just two months, I was able to make my child meals that contain vegetables, and his weight increased. A PM2A direct-beneficiary mother who developed a keyhole garden: Since developing my garden, I eat vegetables every day, even in the dry season. I also learned something about soy production. I now know about the three food groups, including carbohydrates, lipids, and proteins. In preparing my meals, I care about the three food groups. I use whole meal flours to prepare our porridge. My family’s hygiene has improved since we built a handwashing station. Source: Annex V, Testimonials 7 and 24. 2.3.5. Culinary Demonstrations. Although the program continued to support the culinary distributions, most of them were conducted by the LMs and scheduled as part of the monthly plan developed with the Tubaramure Health Promoters. The demonstrations focused heavily on the use of local food for household consumption and weaning foods (Text Box 4.5). There was also a strong emphasis on the promotion of promising high-nutrient foods like soy and amaranth greens (and the seeds to grow these crops). During the final month (August 2014), the program distributed food-processing equipment to two groupements per colline to encourage the groups to develop IGAs—including restaurants—that could promote some of the new nutrient-rich foods such as soy. One opportunity that was discussed—but not fully promoted—was the idea of producing a high-value weaning food from local crops that the groupements could sell for income. Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 91 Text Box 4.5. Tubaramure Beneficiary Testimonial on the Impact of the Culinary Demonstrations on Their Household’s Dietary Diversity My name is Valerie Kadende. I’m 29. I live in Kirasira Village in Butezi commune in Ruyigi Province. As a housewife, I prepare food for my family. Before the Tubaramure Program, I prepared anything. I could not prepare them a balanced diet; that is to say, one that contains carbohydrates, lipids, and proteins. For instance, I prepared beans and cassava without vegetables, or rice with beans without sauce, and, when I was pregnant, I ate anything because I didn’t know that when pregnant a mother needed a proper diet, such as porridge, vegetables, and fruits. I thought that I cultivated vegetables to sell them and earn money to purchase what I needed. With Tubaramure Program, I learned a lot in the field of cooking, that is to say, the cooking demonstration. We learned how to prepare porridge with flour that we were given with vegetable oil, and how to prepare other local foods. Example: To prepare porridge for the whole family, there is a measure to be followed: Little flour + 3 tablespoons of oil + sugar or banana to replace sugar To prepare porridge for children: Little flour + ¼ tablespoon of oil + sugar or banana to replace sugar + vegetables prepared separately When we don’t have this flour, we use our own flour + peanut + sugar and we mix the two. To prepare other foods, Tubaramure Program taught us how to mix other foods for a balanced diet. Example: Beans + sweet potatoes + vegetables Rice + beans + meat Milk + maize bread Rice + potatoes + meat Note that a lactating woman who drinks milk with maize bread will have her breast milk increased, and a pregnant woman who eats porridge for breakfast, and who eats rice together with potatoes and meat, will have her weight increased, and her baby’s weight will increase at birth as well. I thank Tubaramure Program for this demonstration that it taught us. I would ask the program to train us more in cooking demonstration if there are other recipes that we are ready to buy ingredients. Source: Annex V, Testimonial 21. 3.0. Evidence of Results167 168 3.1. Dietary Diversity169 170 The final quantitative survey showed (Table 4.3):  A highly significant increase in the average household dietary diversity; and  A much smaller, but stable, percentage of children consuming at least four food groups during the 24 hours prior to the survey interview. The disaggregated dietary diversity score was about the same for the two provinces, 8.5 in Cankuzo and 8.0 in Ruyigi. 167 SOW Specific Objective 3: What interventions have been more or less successful in meeting targets? 168 SOW Key Question 8: To what degree are behavior changes continued by graduated program beneficiaries, for example changes that improve the nutrition of children over the age of 2 and changes that impact subsequent births? 169 SOW Key Question 6: With regards to food intake and diversity, are beneficiaries adopting desired practices or behaviors? Are there some secondary adopters? 170 SOW Key Question 10: To what extent the use of local food has sustained a sufficient food intake and balanced diet at the household level after the distribution of Title II commodities? Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 92 Table 4.3. Eligible Women and Children Have Increased Intake of Diversified Foods IR3 Key Program Indicators Baseline 2010 Endline 2014 Statistical Significance LOA Target Indicator 3.1. Average household dietary diversity score 4.6 8.2 Significantly increased p=0.000 9.0 Indicator 3.5. Percentage of children 6-24 months reported as consumed at least four food groups during the last 24 hours 74.2 77.8 Baseline data was not suitable to perform a significant test 90.0 Source: Tubaramure IPTT and Institut de Statistiques et d'Etudes Economiques du Burundi (ISTEEBU); 2014. Enquête Ménage pour l’Evaluation Quantitative Finale du Programme PM2A—Tubaramure (Provinces Cankuzo et Ruyigi). Rapport Final. Bujumbura: ISTEEBU (August 2014). Pp. 20. The same final evaluation survey shows 97% of the households in the survey reported eating four or more food groups, which suggests that the vast majority of the population is eating a more diversified diet than at the program’s baseline (Table 4.4). Since the final survey did not measure the months of adequate household food provisioning (it was measured during the baseline),171 there is no way of knowing if this could be attributed to aggregate increases in food access. Since there was very little external investment in agriculture or livestock in the area (by Tubaramure or any other agency) during the five years of the program—and since most of the PM2A graduates had already graduated by the time of the final survey—the most likely factors causing this shift are:172 173  A major change in both men’s and women’s understanding of and willingness to support household consumption (as opposed to sale) of the products they were already producing;  Easier access to a more diversified group of vegetables from the keyhole gardens for about half the provinces’ households; and  Greater access to cash for purchasing food, especially during lean periods through SILCs or SILC-related investments (like on farm animal fattening), for about 30% of the households. Table 4.4. Percentage of Households in the Final Quantitative Survey Eating Different Numbers of Food Groups Province Number of Food Groups Consumed by the Households Number of 0-3 4-6 7-8 9+ Households Cankuzo 4.0 44.1 35.6 16.3 440 (100%) Ruyigi 2.7 56.3 33.4 7.6 751 (100%) Total 3.2 51.8 34.3 10.8 1,200 (100%) Source: Tubaramure IPTT and ISTEEBU; 2014. Enquête Ménage pour l’Evaluation Quantitative Finale du Programme PM2A—Tubaramure (Provinces Cankuzo et Ruyigi). Rapport Final. Bujumbura: ISTEEBU (August 2014). Pg. 39. 171 This was measured during the baseline. 172 SOW Key Question 1: With regard to the program framework, which interventions have been critical and/or effective in achieving the nutrition and health-sector objectives and intermediate results? And why? 173 SOW Key Question 16: How are graduated and current beneficiaries in the community coping with the end of food distribution? Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 93 3.2. Evidence of Impact on Local Institutional, Community, and Household Capacity174 Six areas where the program has increased the local community’s capacity to promote dietary diversity are:  Training 4,920 LMs who continue to be strong advocates for dietary diversity;  Facilitating the LMs and beneficiary groups organizing into groupements, 713 (82%) of which are registered  Helping to plant the idea of keyhole gardens, which has had a host of secondary adopters;  Building the capacity of two groupements per colline through technical support and training in (Text Box 4.6): - Basic principles of organization; - Some basic technical training in food processing, especially soy milk food processing; and - Promoting small livestock development through the distribution of seed herds to each of the groupements that the program is assisting;  Facilitating the development of 711 SILC groups; and  Creating a strong system of SILC PSPs, who continue to backstop both the existing and new SILCs being developed. One unintended consequence of the SILC activities—which was noted by several of the LMs and beneficiary mothers during the focus group discussions—was to build social cohesion between the different groups living in the village (See Annexes IV and V). 174 SOW Key Question 13: To what extent the implemented activities under IR2 and 3 have developed and strengthened the institutional, the community and the households’ capacities? Tubaramure Final Evaluation. Chapter 4: IR3. October 22, 2014. Final. 94 Text Box 4.6. PM2A Beneficiary Testimonial on the Impact Joining a Groupement Had on Her Living Standard My name is Goreth Nyandwi, I’m 25 years old. I live in Nyamasenga Colline in Nyabitsinda commune, which is in Ruyigi Province. I don’t know how to describe Tubaramure. Honestly, this is a program that helped us a lot in our village. I do not know if you, yourself, have not heard of Tubaramure on the radio. Tubaramure fought against malnutrition for our children. For instance, for me, when I was pregnant for the first time, I learned that I had to go for prenatal consultations at least four times, and after delivery, I had to go for postnatal consultations and for my baby’s immunization. In addition, I had to eat for me and for my baby. The program has supported me by giving me porridge and everything I needed. When my child reached the age of 2 years, the program guided, educated, and encouraged me to continue to raise my baby. It showed me the benefit of joining an association, and that it is easier to help a group than an individual. Tubaramure supported us by giving us two goats and seeds. We worked together, and we purchased another goat. As far as seeds are concerned, if I do not have any, I request for a credit in the association, and I pay it back after harvest, whereas before I could leave some of my land in fallow because of lack of seeds. Before, I was alone, but for the moment, I have many friends. My husband did not help me to work in the fields, but, when he joined our association, he has changed his behavior because, there, they give us advice. We purchase seeds in the association. When one is a member, if for example 1 kilogram costs BIF 1000, a member buys for BIF 500 per kilogram; it is an advantage of the association. At the end of the season, when we harvest, we share the surplus and we keep the rest as seed for the new season. Source: Annex V, Testimonial VI. 4.0. Factors That Contributed to and Detracted from Program Relevance, Effectiveness, Efficiency, and Acceptability of Processes, Outputs, and Implementation 4.1. Relevance The Tubaramure IR3 activities were highly relevant to the overall Tubaramure goal of preventing malnutrition in children under 2. Both provinces had, and continue to have, a high percentage of households classified as having high or limited food security—85% in Cankuzo and 65% in Ruyigi according to the 2014 Institut de Statistiques et d'Etudes Economiques du Burundi Standardized Monitoring and Assessment Relief and Transitions (ISTEEBU SMART) study (Table 4.5). Table 4.5. Percentage of Households Classified as Having Different Levels of Household Food Security in Cankuzo and Ruyigi Provinces in the 2014 SMART Study175 Zone Highly Food Insecure (score <1.5) Limited Food Security (1.5