Midterm Evaluation of Amashiga USAID / FFP Title II Program AID-FFP-A-15-00001 Burundi, 2014-2019 FINAL REPORT Prepared for: Amashiga Consortium Partners (led by Catholic Relief Services) and the United States Agency for International Development Evaluation Team: Brian Sage, Kathy Tilford, Ange Tingbo, Bernard Crenn, Bertin Baharanyi, and Dr. Pierre Kwizera (with remote support from Oxu Solutions Staff) DECEMBER 29, 2017 Cover Photographs: Top: Women waiting for their turn for food rations, nutritional fair site at Nyamirambo, October 2017 (by Ange Tingbo) Bottom: Father and son, MTE focus group discussion Mihama, October 2017 (by Kathy Tilford) Table of Contents Table of Contents............................................................................................................................. i List of Tables ................................................................................................................................... i List of Figures.................................................................................................................................. i List of Annexes................................................................................................................................ i List of Acronyms ............................................................................................................................ii Acknowledgments........................................................................................................................... v Executive Summary....................................................................................................................... vi 1. Introduction ............................................................................................................................. 1 2. Overview of the Programmatic Context and Implementation................................................. 2 2.1 Background on Food Insecurity in Burundi .......................................................................... 2 2.2 Muyinga Province Context.................................................................................................... 3 2.3 Amashiga Program Summary................................................................................................ 3 3. Evaluation’s Scope of Work and Methodology ...................................................................... 7 3.1 Scope of Work and Objectives.............................................................................................. 7 3.2 Team...................................................................................................................................... 8 3.3 Methods................................................................................................................................. 8 3.4 Constraints and Limitations................................................................................................. 13 4. Findings, Conclusions and Recommendations by Sector...................................................... 15 4.1 Purpose 1: Chronic Malnutrition in Children under 5 Years is Reduced................................... 16 4.2 Purpose 2: HH have continuous access to adequate nutritious food in Muyinga ............... 30 4.3 Purpose 3: Decentralized government structures, civil society and private sector strengthen and implement effective and equitable nutrition strategies....................................................... 44 4.4 Crosscutting Sub-Purpose (XSP) HHs and communities adopt gender-equitable decision making practices........................................................................................................................ 57 5. Overall Findings, Conclusions and Recommendations......................................................... 63 5.1 General Findings and Conclusions...................................................................................... 63 5.2 Summary Review of Evaluation Main Objectives and Key Questions............................... 82 5.3 Overall Recommendations .................................................................................................. 88 List of Tables Table 1: Amashiga program goal and purposes and sub-purposes Table 2: Summary of CRS Environmental Documentation Submissions and USAID/BEO Decisions Table 3: Key MTE Questions Table 4: Core Team Members and Key Roles Table 5: Final Sampled Collines during Evaluation Table 6: List of Data Collection Tools and Target Audiences Table 7: P2 Summary of Key Indicators Table 8: Committees Operating in Amashiga’s area of intervention Table 9: P3 Summary Conclusions by Sub-Purpose Table 10: Comparison of X.1 and X.2 IR and Output Language Table 11: Amashiga Inception-To-Date Project Analysis including Variance Table 12: Conclusions Across MTE’s Five Main Questions List of Figures Figure 1: MTE Sampled Collines & Communes Figure 2: Father feeds a tomato to his son, October 2017 Figure 3: Woman participant with her child in Gatanga, October 2017 Figure 4: Rucikiri CoProNut representatives, October 2017 Figure 5: Keyhole garden in Muyinga Province, October 2017 Figure 6: Composting hole with a roof in Muyinga Province, October 2017 Figure 7: Hybrid maize grown in the lowlands. Muyinga Province, October 2017 Figure 8: A contour bund with dried up stalks in Muyinga Province, October 2017 Figure 9: A well-managed community grain store. Muyinga Province, October 2017 Figure 10: A handful of poor quality local bean seed in Muramba, Buhinyusa, October 2017 Figure 11: Too Wide Stacking-Muyinga Warehouse, October 2017 Figure 12: Women & children waiting for food rations under the sun in Nyamirambo, October 2017 List of Annexes Annex A: Amashiga Evaluation Scope of Work for MTE Annex B: Evaluation Plan Annex C: Amashiga Logical Framework Annex D: List of Documents & Information Consulted by Oxu Solutions for Amashiga MTE Annex E.1 & 2: Data Collection Tools in French and English Annex F: MTE Detailed Field Work Calendar Annex G: Field Work Data Collection Achievements Annex H: Oxu Team Summary County Debrief Notes Annex I.1 Amashiga Data Review Annex I.2: Amashiga IPTT Review Annex J.1 Selected Performance Indicators on Use of Health Facilities Annex J.2 Detailed Findings Commodity Management Annex K Project Implementation Gantt ii Amashiga MTE - FINAL List of Acronyms Ab'Ir Abaremeshakiyahob’iterambereridakumira, Kirundi for “Extension agent for development that excludes no one”; volunteers for gender sensitization and counselling AEA Agricultural Extension Agent ANC Antenatal Care ARR Annual Report BXW Banana Xanthomonas Wilt BEO Bureau Environmental Officer CA Cooperative Agreement CC Commune Councils CCDC Communal Community Development Committee CDC Colline Development Committee CF Calls Forward CFW Cash for Work CHW Community Health Worker COP Chief of Party CoProNut Committee for the Promotion of Nutrition CRS Catholic Relief Services CSB+ Corn Soy Blend Plus CSC Community Solidarity Chains DCHA USAID Bureau for Democracy, Conflict and Humanitarian Assistance DCOP Deputy Chief of Party DFAP Development Food Assistance Program DPAE DPSP Provincial Representative of the Ministry of Agriculture and Livestock Direction de la Promotion des Semences et Plants DRR Disaster Risk Reduction EHA Essential Hygiene Actions EMMP Environmental Mitigation and Monitoring Plan ENA Essential Nutrition Actions EWS Early Warning Systems FAM Fertility Awareness Methods (Natural Family Planning) FAN Foyer d’Apprentissage Nutritionnel FARN Foyer d’Apprentissage et de Réhabilitation Nutritionnelle (PD/Hearth) FFA Food-for-Assets FFT Food-for-Training FFP Food for Peace FFW Food-for-Work FGD Focus group discussion FH Faithful House FIFO First-in-first-out FY Fiscal Year GBV Gender Based Violence GoB Government of Burundi iii Amashiga MTE - FINAL HF Health Facility HH Households IBR Initial Beneficiary Registration ICT4D Information and communication technology for development IYCF Infant and Young Child Feeding IEE Initial Environmental Examination IGA Income Generating Activity IHPB Integrated Health Project/Burundi IMC International Medical Corps IMCI Integrated Management of Childhood Illness IMS Individu Multiplicateur de Semence IPM Integrated Pest Management IPTT Indicator Performance Tracking Table IR Immediate Result ISABU Institut des Sciences Agronomiques du Burundi Institute of Agronomic Sciences of Burundi ISFM Integrated Soil Fertility Management IYCF Infant and Young Child Feeding KII Key informant interview LAM Lactation Amenorrhea Method LF Lead Farmer M&E Monitoring and Evaluation MCHN Maternal and Child Health/Nutrition MEAL Monitoring, Evaluation, and Accountability with Learning MFI Microfinance Institution MobiComs Community Mobilizers MoH Ministry of Public Health, Government of Burundi MTE Midterm Evaluation MUAC Mid-upper arm circumference MYAP Multi-Year Assistance Program NGO Nongovernmental Organization NRM Natural Resource Management OBR Office Burundais des Recettes OCHA Office for the Coordination of Humanitarian Affairs (United Nations) ODEDIM Organisation Diocésaine pour l’Entraide et le Développement Intégral de Muyinga Office National de Contrôle et de Certification des Semences OFSP Orange-fleshed sweet-potatoes ONCCS Office National de Contrôle et de Certification des Semences PAI Plan Annuel d'Investissement/Annual Investment Plan P1 Amashiga Purpose 1 : Health and Nutrition P2 Amashiga Purpose 2 : Agriculture and Livelihoods P3 Amashiga Purpose 3: Governance PERSUAP Pesticide Evaluation Report and Safer Use Action Plan/ Rapport d’Evaluation des Pesticides et Plan d’Action pour une Utilisation plus Sécuritaire PD/H Positive Deviance Hearth iv Amashiga MTE - FINAL PHAST Participatory Hygiene and Sanitation Transformation PLW Pregnant and Lactating Women PM2A Preventing Malnutrition in Children under Two Approach PO Producer Organization PRONIANUT National Integrated Program For Food and Nutrition PSP Private Service Provider RBU 2000+ Réseau Burundi 2000 Plus RTNB Burundian National Radio and Television SBCC Social and Behavior Change Communication SDM Standard Day Method SDSR Single Diseased Stem Removal SFBW Soy Fortified Bulgur Wheat SILC Savings and Internal Lending Communities SMG Seed Multiplication Group SOW Scope of Work SMILER Simple Measurement of Indicators for Learning and Evidence-based Reports SSR Specific Service Registration TDY Temporary Duty Yonder; Temporary Assignment TFH The Faithful House curriculum TOC Theory of Change USAID United States Agency for International Development USG United States Government VO Vegetable Oil WASH Water, Sanitation and Hygiene WFP World Food Program XSP Amashiga Cross-Cutting Sub-Purpose: Gender YSP Yellow Split Peas v Amashiga MTE - FINAL Acknowledgments Oxu Solutions would like to acknowledge the many people who made this work possible: - Amashiga project staff and Community Mobilizers who organized the evaluation, managed logistics, and contributed valuable feedback during the evaluation validation workshop for project stakeholders held in October 2017, including but not limited to: Regine Pacis Nihoreho, Leonard Nkunzimana, Jean Bosco Ciza, Alexis Havyarimana, Dr. Billy Kubenga, Salvator Nkurunziza, Simon Nakinaganda, Francois Iradukunda, Longin Nzeyimana, Jean￾Marie Vianney Nsabiyumva, and Joselyne Sindabizera; - Abderahamane Bamba, Amashiga MEAL Coordinator, who was with the MTE team from start to finish deserves special thanks; - Gilbert Ndenzako and Scholastique Ntakirutimana who provided support for letters of invitation, visa, and other administrative support; - Interpreters across visited sites: Désiré Ciza, Jean Paul Musabarakiza, Agathe Minani, Auguste Niyonkuru, Gloriose Muhorakeye; - Yves Nkunzimana, the Amashiga Interim Fleet Manager, and the drivers who supported the MTE Team: Anicet, Dieudonne, Serge, Junior, and Clovis; - Project participants, government administrative and technical representatives, consortium members, and community leaders who agreed to be interviewed or participate in focus groups. Finally, Oxu takes full responsibility for the content of this work. Any errors or misrepresentations in the document are our own unless otherwise referenced. vi Amashiga MTE - FINAL Executive Summary Introduction This report details the findings, conclusions, and recommendations of the midterm evaluation (MTE) of the Burundi Amashiga Development Food Assistance Program (DFAP), funded by the United States Agency for International Development (USAID)/Office of Food for Peace (FFP). Launched in September 2014, Amashiga is a five-year project implemented in Burundi’s north-east Muyinga province. The report has been written for the program consortium led by Catholic Relief Services (CRS); members include International Medical Corps, Réseau Burundi 2000 Plus, Organisation Diocesaine pour l’Entraide et le Développement Intégral de Muyinga, the World Food Program and Bioversity International. The MTE team assessed the achievements, relevance, coherence, coverage, effectiveness, efficiency, outputs and early outcomes of Amashiga’s interventions thus far. Based on this analysis, the report provides prioritized recommendations to maximize results for the remainder of the project’s duration. The goal of Amashiga is sustainable, nationally replicable improvement in child nutrition in Muyinga, Burundi; this goal is supported by three purposes and one crosscutting sub-purpose as follows: Purpose 1 (P1): Chronic Malnutrition in children under 5 years is reduced Sub-Purpose 1.1: Individuals practice appropriate behaviors for optimal growth of children Sub-Purpose 1.2: Health providers deliver high quality, gender- responsive MCHN services to women and children Sub-Purpose 1.3: Communities maintain a positive social and physical environment to support good nutrition for children under-5 and pregnant and lactating women Purpose 2 (P2): Households have continuous access to adequate nutritious food in Muyinga Sub-Purpose 2.1: Households have increased self-supply of diverse food Sub-Purpose 2.2: Households have increased income Purpose 3 (P3): Decentralized government structures, civil society and private sector strengthen and implement effective and equitable nutrition strategies Sub-Purpose 3.1: All communal community development councils (CCDC) and Platforms are prepared to mitigate risks to food security and nutrition Sub-Purpose 3.2: CCDCs ensure gender-responsive, equitable delivery of services to support HH food security and optimal nutrition practices Sub-Purpose 3.3: Practices generated by Amashiga contribute to effective national policy implementation and increased gender-responsiveness of policies (related to reducing chronic malnutrition in under 5 children) Cross-Cutting Sub-Purpose (XSP): Households and communities adopt gender-equitable decision￾making practices The full narrative report for the MTE contains the evaluation team’s identified findings, conclusions, lessons learned, and recommendations for each programmatic sector organized by purpose and sub￾purpose, as well as for cross-cutting activities and themes and for the program as a whole. vii Amashiga MTE - FINAL The MTE team’s overall conclusions by purpose and the cross-cutting sub-purpose are as follows: P1 Many of the P1 activities are significantly behind schedule, especially for Sub-Purposes 1.1 and 1.3; this is due to delays within the project’s control (e.g., slow start for activities such as Talking Books and Community Scoreboards) and to circumstances that to some extent are beyond the project’s control (e.g., no field activities in Year 1 due to insecurity; environment compliance requirements, affecting especially the WASH component; and delays on the MoH side in validating modules for IYCF and FAM). As for Sub-Purpose 1.2, many of the training events for facility-based providers and CHWs have taken place but not according to the original calendar. In spite of these delays, there have been a number of achievements: - The food distributions are going well and over 45,000 participants receive the correct rations on a monthly basis. - The P1 team has been careful to harmonize its health and nutrition approaches with the Government of Burundi’s priorities, protocols, and procedures. - The CoProNuts’ activities and other SBCC initiatives have raised knowledge levels on key Amashiga themes such as nutrition, hygiene and sanitation, optimal child health, and gender equity. - There is a heightened awareness of the importance of husbands and wives working together to make decisions about family resources and family health, especially the health and nutrition of young children. P2 Outside of the Savings and Internal Lending Communities component, overall progress on P2 is significantly behind schedule, cannot meet its original targets and will have difficulty making a sustainable impact: - There are no clear and detailed plans of how to achieve significant and sustainable impact/results by the end of the project. - Due to the many delays and un-kept promises, the project has lost credibility among participants and stakeholders. - There is too much emphasis on the quantity of activities and volume of participants and not enough on the quality of training, implementation, and results. - There is incomplete inter-component integration to ensure that P2 activities add to P1 impact. P3 Purpose 3, as with all other purposes, has experienced significant delays. With the exception of a few activities at the provincial level (e.g. the Provincial DRR Platform) in Year 1 and national-level activities to support the development of a national strategy for the promulgation of the law on fortified flour in Year 2, the majority of activities under the purpose only began in Year 3. Overall, most of the elements are founded on sound conceptual logic. Further, the project undertook some important steps – not reflected in the higher levels of the results framework – that were timely, appropriate and an important foundation to the ultimate logic of the component. These include, most notably, the support to the CCDC (and indirectly to the Communal Councils) to strengthen their planning and accountability processes (how commune funds are allocated and budgeted). Nevertheless, the lack of modifications to the design despite the accumulated delays viii Amashiga MTE - FINAL means that the project seems to be far from achieving the initial logic of the Theory of Change purpose i.e., that commune government and non-government actors are capable (and engage in) actively analyzing food security and nutrition on an ongoing basis so as to take actions to improve the situation and to mitigate risks, so as to continue the role that Amashiga was expected to fill in the short term. XSP This sub-purpose was one of the highlights of the MTE: the project has made meaningful and significant progress in less than two years of implementation. Households have already begun to employ gender-equitable decision-making practices. This can likely be attributed to having a simple and well-executed model. Given the appreciation for the activity, as demonstrated for example by the spontaneous referrals by the colline chiefs of couples with disputes to the Ab’IR, it appears that the approach is well suited to Burundi, with potential for other contexts as well. The evaluation team has drawn conclusions related to the three evaluation objectives as follows: Objective 1: To evaluate the strengths and weaknesses of the implementation of project implementation and the quality of outputs, as regards the adherence to terms agreed by FFP and of their acceptability and perceived value to target communities, identifying factors that appear to enhance or detract from the quality, acceptability, and usefulness of implementation and outputs. The Amashiga Project started with an ambitious but very strong initial Theory of Change and design. Just as the project was starting, Burundi suffered major and widespread violence, affecting access to Muyinga Province, which understandably delayed the project start. On top of this, there seems to have been almost a perfect storm (with good intentions) of new USAID/FFP policies and guidelines that the project was expected to use, particularly with regard to the M&E Plan and environmental compliance. The high level of activity around these requirements masked the consequences of the delays; all of this may have inadvertently reinforced a passive atmosphere among the project team, providing a ready excuse for the delays and inaction. In addition, the turnover in CoPs and in staff at USAID/FFP has further masked the consequences, or has slowed any reaction to, the cumulative delays. The primary job of the MTE is to assess project progress relative to the proposed design and the intended impacts. The MTE team cannot soften the disappointing conclusions presented in this report as this would risk diminishing the urgency for USAID/FFP and CRS to work together quickly and with resolve (which has not happened to date) to agree on a revised implementation plan. The MTE team found very few examples where the project adapted to delays or sought to find alternate ways to implement. Given the relatively short period of time remaining, it is vital that management seize this opportunity to urgently review and plan the remaining quarters in detail. Ultimately, it is the project participants who suffer the consequences of the delays to implementation. The hope of the MTE team is that the Amashiga partners (including USAID/FFP) will work with new resolve to prioritize above all else the delivery of project results to communities in the time remaining. Objective 2: To present evidence of changes (intended and unintended) associated with project interventions and outputs, assess how well the observed changes reflect the Theory of Change (ToC), and identify factors in the implementation or context that impede or promote the observed and intended changes. Overall, levels of activity have been too low and have started too recently to clearly attribute concrete change for or by participants to Amashiga. Notwithstanding this conclusion, the MTE team has noted ix Amashiga MTE - FINAL that participants, community leaders and MoH stakeholders associate a number of positive improvements with the project and gave the following examples: - Improved maternal and child health: children are healthier; birth weights are higher; very few children are in the red zone during nutrition screening; women have easier pregnancies with fewer complications. - More women are going to ANC and delivering at the health center; parents take children to the health center at the first sign of illness rather than waiting; children are up to date on their vaccinations (corroborated by the MTE team in several communities). - Participants are more aware of the importance of hygiene and sanitation to good health and are making simple improvements at the household level: evacuating used water, digging compost pits, constructing drying racks and, to some extent, improving latrines. - There is a good understanding at the community level that production must be increased so that households can transition to having a balanced diet after commodity distribution is phased out. - Commodity distributions are generally well-managed and appreciated by participants and local authorities; the American origin of the food and USG support to the overall project is also widely known. Across all sectors, participants stated that they are more aware of the importance of harmony at the household level, mutual support between husband and wife and joint decision-making. They also described how improved gender relations can have a positive impact on family well-being, especially children’s health and increasing household production and revenue, and gave personal testimonials and other examples of how this knowledge has transformed families. On the other hand, practical application of key messages is less feasible, e.g. having a balanced diet without access to local foods or frequent hand washing when water availability is an issue. Further, there is little understanding among participants, local stakeholders or the Amashiga team as to what the project expects to happen to the project-supported structures and interventions after the current award ends. Objective 3: To recommend adjustments to the ToC, project design, resource allocation, project management, monitoring and evaluation (M&E) plan, or implementation that could improve the likelihood of achieving desired results by the project’s end – based on the evidence collected and conclusions drawn for the evaluation objectives above. In summary, the MTE team recommends that the Amashiga consortium reviews its targets and workplans with a view to cutting targets and certain activities that have either not yet started or cannot have the desired impact in the remaining time period. This requires active involvement from and liaison with USAID-FFP to ensure timely discussions and approval of revisions using three guiding principles: - Emphasis on quality over quantity of activities; - Each activity has a plan by project quarter to the end of the project; each component has an exit- and long-term strategy; and - Communications to local stakeholders, including participants, are a priority. Timely and regular communications should clearly describe the project’s plans and the specific support to be provided. x Amashiga MTE - FINAL Recommendations: The full narrative report presents recommendations for each purpose and for cross￾cutting components (SBCC, Commodity Management, MEAL, Project Design & ToC and Environmental Compliance) in Sections 4 and 5 respectively of the full report. The MTE team’s six overarching recommendations are summarized below: 1. Revise activity planning to determine realistic targets, focusing on quality of outputs and outcomes rather than on quantity of activities - Develop or update strategies for each purpose and key activity area with quarterly milestones until the end of the project; and - Reduce volume and increase quality of activities. 2. Develop a project-wide sustainability plan with specific sustainability measures for all activities - Reduce free inputs; and - Support local actors to conduct activities, rather than having them participate only as attendees, so that they can adopt appropriate responsibilities. 3. Strengthen the integration of the project purposes by identifying points of intersection between sectors. 4. Strengthen the SBCC component to enhance the potential for sustained behavior change and lasting adoption of improved practices across all components. 5. [For both USAID/FFP and Amashiga] Develop a proactive approach to donor relations, clarifying compliance requirements and obtaining approvals. 6. Prepare for the Final Evaluation of Amashiga 1 Amashiga MTE - FINAL 1. Introduction This report details the findings, conclusions, and recommendations of the midterm evaluation (MTE) of the Burundi Amashiga Development Food Assistance Program (DFAP), funded by the United States Agency for International Development (USAID)/Office of Food for Peace (FFP). Launched in September 2014, Amashiga is a five-year program implemented in Burundi’s north-east Muyinga province. The report has been written for the program consortium led by Catholic Relief Services (CRS), and assesses the achievements, relevance, coherence, coverage, effectiveness, efficiency, outputs and early outcomes of Amashiga’s interventions thus far. Based on this analysis, the report provides prioritized recommendations to maximize results for the remainder of the project’s duration. The report comprises five sections. Following this Introduction, Section 2 discusses the Burundi context and summarizes the Amashiga program and its implementation. Section 3 describes the evaluation methodology followed, including key constraints and limitations of the evaluation. Section 4 provides findings, conclusions, and recommendations organized by sector, including: Purpose 1 (P1): Health and Nutrition; Purpose 2 (P2): Agriculture and Livelihoods; and Purpose 3 (P3): Governance; and the cross-cutting Sub-Purpose (XSP): Gender. Section 5 includes observations and recommendations related to Social & Behavior Change Communication; Commodity Management; Financial Management; MEAL; Project Design & Theory of Change; and Environmental Compliance; as well as a summary of the MTE team’s conclusions against the MTE questions and objectives and over-arching recommendations for the project. 2 Amashiga MTE - FINAL 2. Overview of the Programmatic Context and Implementation 2.1 Background on Food Insecurity in Burundi Burundi’s history of food insecurity is linked to high rates of poverty across the country. USAID has indicated that “approximately 80% of the estimated population of 10.88 million lives below the poverty line, which has serious repercussions on the ability of households to meet basic needs.” 1 Poverty in Burundi is particularly high in rural areas, where small-scale subsistence farmers make up 90% of Burundi’s economy but face challenges of land scarcity.2 With generally limited land holdings and low agricultural productivity, Burundian households are ill-equipped to weather the impacts of extreme drought or rains, which “reduce crop yields and induce shocks that asset-thin households can ill afford.”3 Such shocks proved particularly problematic in late 2015 and early 2016 when strong winds, floods and landslides from the El Niño effect, and subsequent irregularly low rainfall, reduced already limited agricultural yields throughout the country.4 Malnutrition is common and chronic stunting is high across Burundi, particularly in rural areas. From 2008-2012, 57.7% of Burundian children showed moderate to severe stunting.5 The Amashiga baseline survey pointed to a mother’s nutritional status as “a significant contributing factor to low birthweight and stunting, [finding a] Women’s Dietary Diversity Score of only 3.7 out of a possible nine food groups per day are being consumed by women of childbearing age.”6 Given this, past projects targeted mothers to improve child nutrition status. However, the Amashiga project identified that often, husbands wield decision-making power for household growing and purchasing choices, and thus “targeting additional nutrition messages to the mother alone will be ineffective.” 7 A full family approach is necessary to address challenges of food insecurity and stunting throughout Burundi’s population of under 5 children. Recent political crises compounded existing environmental contributors to Burundi’s food insecurity and malnutrition challenges. Ignited in April 2015 with the decision of President Nkurunziza to run for a third term in office, widespread political protests, government repression, and human rights violations ensued and led to significant internal displacement and migration. Incidents of gender-based violence (GBV) against women and girls were also on the rise.8 In response, the European Union suspended budgetary aid to Burundi in 2016 and the Burundi government’s ensuing austerity budget included substantial cuts to WASH, health, and education funding.9 By 2017, while the political crisis became less overtly violent, it exacerbated the existing economic downturns and natural disasters that had recently hit the country..10 Recent IPC analyses estimated 18% of Burundi’s population is in humanitarian phases of food insecurity (Stage 3, Crisis and Stage 4, Emergency). While this is an 1 Food Security Country Framework for Burundi FY 2014–FY 2019, USAID Office of Food for Peace, September 2013. 2 The World Bank in Burundi: Overview, http://www.worldbank.org/en/country/burundi/overview. 3 Ibid. 4 Burundi - Situation Report, Food and Agriculture Organization of the United Nations, December 2016, http://www.fao.org/resilience/resources/resources-detail/en/c/456375/. 5 Burundi Statistics, UNICEF, Accessed 10 October 2017, https://www.unicef.org/infobycountry/burundi_statistics.html#114. 6 ICF HH Survey, Muyinga province, Mar-Apr 2015. Preliminary unpublished results. Table A6.1. FFP Baseline Indicators - Amashiga Project 7 CRS Technical Proposal for Amashiga, revised 23 June 2014. 8 Amnesty International Report 2016/2017: Burundi, Amnesty International, 2017. 9 European Union Suspends Aid to Burundi over Political Crisis, New York Times, 14 March 2016, https://www.nytimes.com/2016/03/15/world/africa/european-union-suspends-aid-to-burundi-over-political-crisis.html. 10 Amnesty International Report 2016/17: Burundi, Amnesty International, 2017. 3 Amashiga MTE - FINAL improvement from 2016, projections suggest that the October to December 2017 lean period will see an increase to 27% of Burundi’s population experiencing Stage 3 or Stage 4 food insecurity.11 2.2 Muyinga Province Context Muyinga, a province in northeastern Burundi, is among the most population-dense provinces in the country, exacerbating its vulnerability to food insecurity. Muyinga is afflicted with particularly high stunting rates, at over 50% among under 5 children in 2013 and increasing since.12 Indeed, the Amashiga baseline survey from March - April 2015 indicated that “the nutritional status of children in Muyinga has deteriorated over the past two years and [stunting] is now at 65% of children under five years of age.” 13 Recent environmental and political crises have impacted livelihoods in Muyinga in several ways. Contiguous to the Tanzanian border, Muyinga witnessed the flight of Burundian refugees across its borders from the 2015 political crises onward.14As the conflict has become less overt, the province has also hosted a significant number of Burundian returnees.15 Muyinga experienced higher rates of rainfall in 2017 due to the El Niño effect, and thus was particularly vulnerable to food insecurity challenges. Recent IPC food security analyses identify Muyinga and its neighbors among the more vulnerable provinces in Burundi, experiencing a “Stage 3, Crisis” from April – May 2017, while its immediately neighboring province, Kirundo, experienced areas of “Stage 4, Emergency”.16 While the price of key food commodities in Muyinga (including corn, beans, and rice) decreased considerably from March – April of 2017, 17 Muyinga remains among the provinces of Burundi most negatively impacted by rising food insecurity.18 2.3 Amashiga Program Summary Amashiga is a $49.9 million USAID/FFP-financed DFAP led by CRS that seeks to reduce the rate of stunting through sustainable improvements to child nutrition in Muyinga province, while collecting lessons learned so that project impact can be replicated on a national scale. The five-year Development Food Aid Program is implemented by a CRS-led consortium, which includes IMC (International Medical Corps), RBU 2000+ (Réseau Burundi 2000 Plus), ODEDIM (Organisation Diocesaine pour l’Entraide et le Développement Intégral de Muyinga), WFP (World Food Program), and Bioversity International. The table below summarizes the program’s goal, purposes and sub-purposes. Table 1: Amashiga program goal and purposes and sub-purposes Goal: Sustainable, nationally replicable improvement in child nutrition achieved in Muyinga, Burundi P1: Chronic Malnutrition in children under 5 years is reduced. SP1.1: Individuals practice appropriate behaviors for optimal growth of children SP1.2: Health providers deliver high quality, gender- responsive MCHN services to women and children SP1.3: Communities maintain a positive social and physical environment to support good nutrition for 11 Burundi: Acute Food Insecurity Situation July - September 2017 and Projection for October - December 2017, IPC, http://www.ipcinfo.org/ipcinfo-detail￾forms/ipcinfo-map-detail/en/c/1036960/. 12 FY15 Amashiga Quarterly Report, October – December 2014. 13 ICF HH Survey, Muyinga province, Mar-Apr 2015. Preliminary unpublished results. Table A6.1. FFP Baseline Indicators - Amashiga Project. 14 UNHCR, “Burundians continuing to flee the country one year after crisis began,” 22 April 2016, http://www.unhcr.org/en￾us/news/briefing/2016/4/5719f0119/burundians-continuing-flee-country-year-crisis-began.html 15 Emergency Plan of Action Final Report, IFRC, 30 June 2017. 16 IPC Burundi Results of Situational Analysis of Current Acute Food Insecurity, IPC, April – May 2017. http://www.ipcinfo.org/fileadmin/user_upload/ipcinfo/docs/1_IPC_Burundi_Current_Projected_AcuteFI_2017A_FR.pdf 17 MINAGRIE - Système d’information sur les prix. « Bulletin Mensuel N°4/2017 : Période d’enquête entre le 27 mars et le 23 avril 2017 » . 24 May 2017. 18 OCHA, “Humanitarian bulletin – Burundi,” 1 February 2017, https://www.humanitarianresponse.info/system/files/documents/files/burundi_humanitarian_bulletin_february_2017_en.pdf. 4 Amashiga MTE - FINAL Goal: Sustainable, nationally replicable improvement in child nutrition achieved in Muyinga, Burundi children under-5 and pregnant and lactating women Implemented by: CRS and IMC P2: HH have continuous access to adequate nutritious food in Muyinga SP2.1: HHs have increased self-supply of diverse food SP2.2: HH have increased income Implemented by: CRS, RBU 2000+, Bioversity International and ODEDIM P3: Decentralized government structures, civil society and private sector strengthen and implement effective and equitable nutrition strategies SP3.1: All communal community development councils (CCDC) and Platforms are prepared to mitigate risks to food security and nutrition SP3.2: CCDCs ensure gender-responsive, equitable delivery of services to support HH food security and optimal nutrition practices SP3.3: Practices generated by Amashiga contribute to effective national policy implementation and increased gender-responsiveness of policies (related to reducing chronic malnutrition in under 5 children) Implemented by: CRS, WFP XSP: HHs and communities adopt gender-equitable decision-making practices IRX.1: HHs with behaviors that threaten food security and nutrition are identified and supported to change IRX.2: Couples receive training and support to make HH decisions together IRX.3: Community members' standards for gender equity in decision making in homes and communities increased Implemented by: CRS, ODEDIM In pursuit of Amashiga’s goal and purposes, the program has worked in all seven communes of Muyinga, 230 collines, and 809 sub-collines, targeting 575,738 participants and reaching an estimated 23,057 in FY16. According to the Scope of Work, Amashiga works to improve child nutrition by fostering an enabling environment for the entire community, to ultimately address underlying causes of chronic malnutrition. Key program activities have included social and behavior change communication (SBCC) on preventing stunting and promoting gender equity; capacity building for community health workers (CHWs); nutrition fairs; the establishment of committees for the promotion of nutrition (CoProNuts); training of gender volunteers (Ab’IRs); savings and internal lending communities (SILCs), access to seeds, tools, and training for agricultural workers and entrepreneurs; and capacity building for communal development councils (CCDCs) in creating action plans to mitigate shocks. A combination of political crises, health epidemics and administrative environmental compliance delays have significantly delayed the implementation of the Amashiga program. While Amashiga officially started in September 2014, the 2015 political crisis delayed the rollout of field implementation until October 2015, and widespread implementation was delayed until early 2016.19 From April-September 2015, most Amashiga expat staff temporarily relocated outside of Burundi, key consultants (including the PERSUAP consultant) delayed travel to Muyinga, and national staff were unable to work at full capacity during a period marked by an attempted coup in May, a contested election in July, and the swearing-in of President Nkurunziza for a third term in August.20 While violence was concentrated in 19 As reported in the Amashiga Quarterly Report, April – June; Q3 FY15; Amashiga Annual Results Report FY15; Amashiga Quarterly Report, October - December, Q1 FY16; and Amashiga Quarterly Report, January - March, Q2 FY16. 20 CRS Burundi, “ARR FY15 Narrative Report,” Updated 8 November 2015 & Amnesty International Report 2016/2017: Burundi. 5 Amashiga MTE - FINAL Bujumbura, areas far from the capital such as Muyinga also suffered as Burundians fled the country and agricultural production significantly declined.21 By September 2015, full scale work on Amashiga resumed and dates were set for the start of program implementation in Muyinga province.22 Through the subsequent year, however, allegations of human rights abuses under President Nkurunziza’s government continued. By October 2016, such allegations prompted the Minister of the Interior to ban five leading human rights organizations, and by April 2017 the government created new laws and policies concerning international NGOs reporting and customs procedures. These legal changes significantly delayed the second Amashiga Call Forward in 2016, 23 while adding new layers of complexity for the consortium with requirements on reporting to the government every semester (instead of annually); keeping a foreign currency account at the central bank; and additional requirements.24 Concomitantly, a series of health epidemics – cholera in mid-2016; malaria in early 2016 and early 2017 – delayed implementation of program elements reliant on community health workers and health providers, whose attention was diverted to addressing these crises.25 Furthermore, activities across all purposes, but particularly P2 activities, have experienced delays due to challenges meeting environmental compliance requirements. USAID’s Environmental Procedures known as CFR 216 apply to all projects, programs or activities authorized or approved by USAID and it follows that nearly all projects or programs require some form of environmental documentation26 “not to prevent all environmental impacts associated with development activities” but rather “with a view to ensuring that environmental issues receive adequate consideration in activity design and implementation”. In the approved project proposal for Amashiga, CRS made commitments to: - adhere to USAID’s Global Environmental Management support guidelines27 - contract a consultant to lead an annual study to assess progress on the Environmental Monitoring and Mitigation Plan and develop the required annual Environmental Status Report (ESR) 28 - develop a pesticide compliance plan in line with the Project IEE and the PEA: Commodity Protection by Phosphine Fumigation in USAID Food Programs. 29 Post-award, CRS prepared a number of documents and plans as summarized in Table 2 below. Table 2: Summary of CRS Environmental Documentation Submissions and USAID/BEO Decisions Document CRS Submission USAID/BEO Decision Initial Environmental Examination 1. original 2. revised (including deferral of five outputs) September 201430 April 2015 August 2015: approval with five conditions 21 CRS Burundi, “ARR FY15 Narrative Report,” Updated 8 November 2015 & Amnesty International Report 2016/2017: Burundi. 22 Ibid. 23 CRS Burundi, “ARR FY16 Narrative Report,” Updated 8 November 2016. 24 CRS Burundi, “ARR FY17 Narrative Report Preparatory Document,” Updated 30 April 2017. 25 CRS Burundi, “FY16 Q2 Amashiga Quarterly Report Narrative,” Updated 26 April 2016. 26 USAID Environmental Procedures Training Manual (EPTM) 27 CRS Technical Proposal for Amashiga, revised 23 June 2014, p.35. 28 CRS Technical Proposal for Amashiga, revised 23 June 2014, p. 43. 29 CRS Technical Proposal for Amashiga, revised 23 June 2014, p. 106. 30 Not shared with MTE team 6 Amashiga MTE - FINAL Document CRS Submission USAID/BEO Decision Email response to five conditions from April 2015 approval and request for temporary waiver to proceed with P2 pending preparation and submission of Agriculture PERSUAP October 2016 October 2016: waiver for use of 12 named pesticides (limited in time & scope) Commodities PERSUAP • original • revised March 2016 June 2016 August 2017: approval with one condition Agriculture PERSUAP • local consultant report • international consultant deliverables October 2016 February 2017 June 2017 Approval (with edits1) Initial Environmental Examination 3. Revised integrating PERSUAP, Climate Risk Screening and Adaptation and recommending classification of the five previously-deferred outputs August 2017 November 2017 Approval In accordance with requirements, project outputs were classified in the IEE according to one of the following: - Categorical Exclusion: activities have no adverse effect (i.e. training, technical assistance; not to include any infrastructure rehabilitation) - Negative Determination: no adverse effects expected for activities which are well defined over life of the award:  without conditions (no special mitigation measures needed)  with conditions (mitigation measures specified) - Deferral: elements not well defined; activities will not be implemented until amended IEE is approved. The suspensions to activities attributed by Amashiga team to the delays in obtaining IEE approvals, the evolution (as relevant) of the classifications of relevant outputs in the IEE submissions and implications for the project are described under relevant output findings in Sections 4.1, 4.2 and 4.3. Output wording in the IEE submissions does not reflect the amendments to the project LogFrame. This somewhat complicates understanding by project staff and the MTE team of the process. To simplify presentation for this report, previous wording of relevant outputs is provided in footnotes under the relevant IRs. In addition, the MTE team has noted overall findings, conclusions and recommendations regarding management of the environmental compliance requirementsin Section 5.1 7 Amashiga MTE - FINAL 3. Evaluation’s Scope of Work and Methodology 3.1 Scope of Work and Objectives As described in the evaluation’s Scope of Work (see Annex A: Scope of Work for MTE) and Evaluation Plan (Annex B: Amashiga Midterm Evaluation Plan), the overall goal of the midterm evaluation is to assess the achievements, relevance, coherence, coverage, effectiveness, efficiency, outputs and early outcomes of Amashiga’s interventions thus far, and based on this analysis, provide prioritized recommendations to maximize results for the remainder of the project’s duration. The MTE objectives can be outlined as: 1. To evaluate the strengths and weaknesses of project implementation and the quality of outputs, as regards the adherence to terms agreed by FFP and of their acceptability and perceived value to target communities, identifying factors that appear to enhance or detract from the quality, acceptability, and usefulness of implementation and outputs; 2. To present evidence of changes (intended and unintended) associated with project interventions and outputs, assess how well the observed changes reflect the Theory of Change (TOC), and identify factors in the implementation or context that impede or promote the observed and intended changes; and 3. To recommend adjustments to the TOC, project design, resource allocation, project management, monitoring and evaluation (M&E) plan, or implementation that could improve the likelihood of achieving desired results by the project’s end – based on the evidence collected and conclusions drawn for the evaluation objectives above. Furthermore, the MTE Scope of Work included five key questions to address. Table 3: Key MTE Questions Objective 1 1. How well have the project’s interventions met planned schedules, participant numbers, and outputs? What factors promoted or inhibited adherence to schedules? How were problems and challenges managed? 2. What are the strengths of and challenges to the overall project design, implementation, management, communication and collaboration so far? What factors appear to promote or challenge the project operations or effective collaboration and cooperation among the various stakeholders? 3. In each technical sector, what are the strengths of and challenges to the efficiency of interventions’ implementation and their acceptance in the target communities? How well do implementation processes adhere to underlying principles and project protocols? What factors in the implementation and context are associated with greater/lesser efficiency in producing outputs of higher/lower quality? Which interventions and implementation processes are more or less understood by and acceptable to members of the target communities and why? Objective 2 4. What changes do community members and other stakeholders associate with the project’s interventions? What factors appear to promote and deter the changes? How do the changes correspond to those hypothesized by the project’s TOC? Objective 3 5. Based on the findings from the previous four questions, how could the project be modified to improve its acceptability to targeted communities or the efficiency and effectiveness of its implementation? How should the project’s TOC or results framework be refined or modified? 8 Amashiga MTE - FINAL Within each question, Annex 3 from the SOW (attached as Annex B.2 to this report) also detailed numerous detailed areas of focus and key aspects to consider per question, totaling over 30 “areas of focus” and more than 120 “aspects to consider.” While this list was thus extremely thorough, it was not feasible to address each area of focus and aspect to consider equally. As a result, the MTE team has prioritized these components according to linkages to the results framework; linkages to the evaluation’s overall purpose, three objectives, and five key questions; and feasibility to address the question given the time and resources available. 3.2 Team Oxu employed a team approach to conduct the Amashiga MTE to ensure advanced problem-solving and knowledge creation from combined, diverse expertise and experiences. The core MTE team members cumulatively possess decades of experience in livelihoods, food security, resilience, and gender assessments and programming in Africa, including Burundi. The core team members brought expertise in: agriculture, food security, commodity management, gender-based violence, SBCC, and nutrition/health, as well as conflict management, gender, program design, measurement and learning, and managing and conducting field-based data collection. The roles of the core team members are outlined below. Table 4: Core Team Members and Key Roles Oxu Team Member Sex Role for MTE Brian Sage M Team Lead & Technical Specialist for Gender and Governance Kathleen (Kathy) Tilford F Technical Specialist for SBCC and Nutrition/Public Health Bernard Crenn M Technical Specialist for Agriculture/NRM Ange Tingbo M Technical Specialist for Commodity and Financial Management Pierre Kwizera M Associate Specialist for Public Health Bertin Baharanyi M Associate Specialist for Agriculture/NRM Gloriose Muhorakeye F Interpreter (Commodities) Désiré Ciza M Interpreter (P1) Agathe Minani F Interpreters (P2) Jean Paul Musabarakiza M Interpreter (P2) Auguste Niyonkuru M Interpreter (P3) Basilissa Ndayiziga M Consultant for Governance & Gender 3.3 Methods The evaluation was carried out from August – December 2017. The methodology consisted of three main phases: i) desk review and field work preparation and planning, ii) field data collection and analysis, and iii) report writing. Details on the desk review and field data methodology are included below. Phase 1: Desk Review and Field Work Planning Oxu began with a desk review of internal Amashiga documents and information, as well as select relevant external documents. These documents informed the team’s early understanding of the Amashiga program, and informed refinements for Phase Two qualitative data collection tools. Desk review has continued throughout the evaluation as additional documents or information were provided See Annex D: List of Documents and Information Consulted by Oxu. 9 Amashiga MTE - FINAL Phase 2: Qualitative Data Collection and Analysis The MTE team developed tools and note-taking templates for qualitative data collection in the field. Tools were developed for each sector: P1 Health and Nutrition, P2 Agriculture & Livelihoods, P3 Governance, and the Cross-Cutting Theme of Gender. All tools can be found in Annex E – Data Collection Tools. Throughout field work, Oxu’s qualitative leads carried out data collection with support from local language translators. Qualitative methods included the following: - Meetings and interviews with project staff from all levels: Oxu conducted interviews regarding overall program strategy, management, implementation, and measurement; sectoral approaches and achievements; cross-cutting issues (SBCC, sustainability, and spillover effects in particular); partnerships, including among consortia members and with host-country government agencies; commodity management; and adaptability to insecurity. - Focus Group Discussions (FGDs) were conducted with project participants and community stakeholders. FGDs were conducted with (Committees for the Promotion of Nutrition (CoProNuts); Ab’IRs or Abaremeshakiyaho b’iterambere ridakumira couples and those couples who received their services; members of Communal Community Development Committees and Colline Development Councils; members of Early Warning System (EWS) Committees; Agricultural Extension Agents (AEAs), Private Service Providers (PSPs), and CoProNut representatives). - Key Informant Interviews (KIIs) were conducted with community and institutional stakeholders, including but not limited to: members of NGOs, community based organizations, PSPs, other private enterprises such as radio broadcast companies, seed producers, and others who work in collaboration with or beside Amashiga; Government agents, including community leaders, Community Health Workers, health center staff, national and local government health officials, and AEAs. - Direct observation of program activities, including commodity storage and warehousing units, food distributions, demonstration plots, infrastructure, and other intervention sites. - Collective analysis workshop: Oxu organized a collective analysis workshop at the end of the data collectionwith project staff and implementing partner organizations. The workshop was an opportunity to discuss and receive feedback on preliminary findings, with the aim of improving subsequent days of data collection, increasing understanding of the context and project achievements, and involving project stakeholders in the initial analysis in a participatory manner. These collective analysis exercises also helped validate information gathered through the FGDs, KIIs, and observations. Data collection activities are detailed in Annex G: MTE Qualitative Field work Achievements. In total, the team conducted 156 KIIs, 140 FGDs, and 51 observations. During the field work, data was triangulated throughout the process by speaking with a geographically and programmatically diverse array of stakeholders, project staff, and participants; a review of past project documents; and through direct observation. After field work, each Oxu evaluator conducted a full review of the data collected and the daily summary briefings to confirm and sharpen findings, and the team worked together to examine across the data to discuss salient overall, general, and cross-cutting themes as well as findings related to individual sectors or topic areas. The team presented initial findings to the Amashiga consortium during a validation workshop in Muyinga and to USAID/FFP on 16th and 18th October 2017 respectively. 10 Amashiga MTE - FINAL Sampling & Site Selection The MTE team employed purposive sampling, with random selection for some elements, to select sites for primary data collection. The sampling process to select communities for data collection took into account the following factors: 1. Diversity of geographic areas, considering those closer and further from market and town centers in the target zones and differences in agro-ecological zones; 2. Collines that have a range of level of activities (those with a broad range of activities and those with fewer activities); and 3. Security. The first level of site selection was at the commune level. Given the number of days available in the field, five out of the seven communes in which Amashiga is implemented were sampled. Initially, the team randomly selected the communes of Gashosho, Gasorwe, Butihinda, Giteranyi and Muyinga. However, the Amashiga team pointed out that the commune of Buhinyuza represents a very different agro-ecological zone from the other six communes and has particular socio-cultural characteristics. Following their recommendation, the team added Buhinyuza and dropped Gasorwe in the process. The second level of selection was at the colline level. Per the criteria discussed above, evaluators selected two collines per commune to visit, totaling nine collines: Gisebeyi, Buvumbi, Rwamfu, Tangara, Nonwe, Gakoni, Kara, and Karongwe. In addition, partial sampling based on activities of interest was done in Ruganirwa (Muyinga Commune), Kinyama (Gashoho Commune), Kobera (Butihinda Commune), Gasenyi and Murama (Giteranyi Commune), Muramba (Buhinyuza Commune), and Rugabano (Mwakiro Commune). Table 5: Final Sampled Collines during Evaluation Commune Colline Gashosho Gisebeyi Buvumbi Gasorwe Kizi Gasuru Butihinda Rwamfu Tangara Giteranyi Nonwe Gakoni Muyinga Ruganirwa 11 Amashiga MTE - FINAL Figure 1: MTE Sampled Collines & Communes Map modified from the UN Office for the Coordination of Humanitarian Affairs, Burundi www.ochaburundi.org. Data source: Administrative Boundaries: IGEBU, 2003 Individuals & Group Participant Selection The table below shows the individuals and groups that were sampled during data collection, per each sector and per the tools. Evaluators included a diversity of groups in the process, including project participants, key stakeholders, partner management, commune-level authorities, and others in this process. 12 Amashiga MTE - FINAL Table 6: List of Data Collection Tools and Target Audiences # Purpose Tool and Target Audience P1: Health/ Nutrition and WASH KIIs 1 CRS project staff 2 IMC project staff 3 P1 Community Mobilizers 4 Community Health Workers 5 Health care providers 6 MoH partners FGDs 7 Women participants (receiving MCHN rations) 8 Men participants (wives receiving MCHN rations) 9 CoProNut P2: Agriculture/ Livelihoods KIIs 10 Project staff 11 Key stakeholders 12 FGDs 12.a Farmers 12.b Lead farmers 12.c SILC groups 12.d PSP 12.e SMG 12.f AEA 12.g FFW/ CFW 12.h PO 12.i Non-participants 13 Observations For demonstration/lead farmer fields, FFA/CFW infrastructure, SILC groups, seed multiplication fields, post-harvest technologies, goat farms, youth businesses, ISFM and CSA activities, DRR activities, vet service/supply shop, vocational training centers P3: Governance KIIs 14 Commune Administrator 15 Colline chief 16 P3 Community Mobilizer FGD 17 Early Warning System participants 18 CDC 19 CCDC 20 Non-Participants Commodities & Financial Management KIIs 21 Foods manager 22 Food staff 23 Administrative and financial director - ODEDI 13 Amashiga MTE - FINAL # Purpose Tool and Target Audience 24 Local authorities FGD 25 FFA-FFW-MCH2 Observations 26 Food distributions 27 Food warehousing 28 FFA-FFW achievements Joint and/or XSP KIIs 29 Gender Community Mobilizer 30 SBCC Technical Advisor FGD 31 Ab’IR couples 32 Couples receiving services from Ab’IRs 33 Non-Participants 3.4 Constraints and Limitations The desk review and field data collection generally proceeded well, although the following constraints and limitations were noted: - Due to the timing of the MTE during the dry season, observations of P2 activities were limited. While the MTE team was able to observe some offseason agriculture development efforts in marshland or valley areas, it was not possible to observe harvest or planting processes that are important in the overall purpose; - A number of project documents were not received until during or after data collection. Had these documents been available earlier, they would have facilitated further refinement of interview and focus group questions. The late receipt of these documents also meant that the MTE team spent valuable time during the field data collection period reviewing documents; - The MTE team relied on project staff to assist in selecting some participants for focus group discussions and key informant interviews. While this method increases the risk of selection bias, the MTE team did not see indications that “more successful” participants were selected at higher rates; however, there was some concern, especially during the second half of field data collection, that participants were being coached on what questions to expect. To mitigate this, the MTE team sought to organize its own meetings in the final week of field work; - The sample of eight collines and four communes limits the MTE’s findings. The MTE team opted to fully sample only eight collines and four communes to better understand Amashiga’s activities and dynamics. In this way, the team in effect was able to focus more on the quality of the interventions. To acknowledge the breadth as well as the depth of the project’s scale and to respond to the constraint described in the previous point, the MTE team also chose to partially evaluate several other collines in the province. The MTE team added Ruganirwa colline in Muyinga Province to add a perspective of a peri-urban colline as well as it has a presence of batwa communities. In addition, the MTE team added partial sampling to view particular activities of interest in Kinyama (Gashoho Commune), Kobera (Butihinda Commune), Gasenyi and Murama (Giteranyi Commune), Muramba (Buhinyuza Commune), and Rugabano (Mwakiro Commune). While the MTE team believes the overall approach allowed for more robust findings pertaining to project quality, it could nonetheless be considered a limitation when considering the project at scale; 14 Amashiga MTE - FINAL - Given time constraints, the MTE team did not conduct feedback sessions with communities to discuss findings as originally planned. While such sessions would have been positive from an accountability standpoint, time constraints limited their feasibility. Additionally, the project team seemed to appreciate validation workshops, and thus the priority was given to organizing validation workshops rather than community feedback sessions. Finally, the MTE team worried about the influence community feedback sessions could have on subsequent data collection in other areas, through project staff sharing information with communities where data collection had not yet taken place. - Recruitment options for MTE team members in the field were limited by time and the availability of fewer qualified candidates than expected. Due to the brief time spent in-country for recruitment prior to data collection in the field, there were some limitations to the recruitment of local team members. While additional training and briefing by Oxu team members and other adjustments permitted the MTE team to ensure the quality of the work, these are, nonetheless, important factors for consideration when preparing for future evaluations. Amashiga MTE - FINAL 15 4. Findings, Conclusions and Recommendations by Sector This section presents the sector-level evaluation findings, conclusions and lessons learned, and recommendations for the project’s key programming sectors, as well as findings related to food commodities. Section 4 is organized as follows: 4.1 Purpose 1 (P1): Health and Nutrition 4.2 Purpose 2 (P2): Agriculture and Livelihoods 4.3 Purpose 3 (P3): Governance 4.4 Cross-Cutting Sub-Purpose (XSP): Gender For each of the sector sections, we first provide a summary across all phases of the design related to the sector, present a summary of findings, 31 outline conclusions and lessons learned based on the findings, and finally offer recommendations. Many of the findings presented in this section were presented to the consortium and USAID/FFP during the validation and presentation sessions held in October 2017 and subsequently when reviewing the initial draft of this report. Some were questioned by session participants and this feedback was helpful to the MTE team in refining the findings, conclusions and recommendations presented in this report. Finally, the recommendations are designed to be relevant for the remaining project implementation but also for future programming by consortium members and USAID/FFP. 31 See also Annexes J.1 and J.2 for complementary detailed findings. Amashiga MTE - FINAL 16 4.1 Purpose 1: Chronic Malnutrition in Children under 5 Years is Reduced Introduction to P1 Activities Purpose 1 has three IRs, each focusing on a different group in a concerted effort to reduce chronic malnutrition in children: individuals, especially caregivers for young children; health care providers, especially staff at the health facility level and to a lesser extent, the volunteer Community Health Workers (CHWs); and entire communities, led by the Committees to Promote Nutrition (CoProNuts). The four principal approaches are: - Nutrition fairs: 33 fairs are held each month and include monthly food distributions to maternal and child health/nutrition (MCHN) participants (pregnant women and women with children 0-23 months); nutrition screening of children 6-23 months; and SBCC activities, including nutrition demonstrations on how to use the rations. - Capacity building of providers: Formal training and supportive supervision, primarily for health center staff and CHWs. - CoProNuts: Each sous-colline has an all-volunteer committee whose members include the CHW, an administrative official such as the sous-colline chief, and members of community groups and religious denominations. The CoProNuts’ mandate is to orchestrate community￾wide efforts to reduce malnutrition among young children. - SBCC activities to promote behavior change: Activities include mass sensitization via community meetings and radio and to a lesser extent, small group and one-on-one counseling. Materials developed include nutrition posters and algorithms for health centers and counseling cards for volunteers such as CHWs. The principal P1 project staff at the community level are the 23 Community Mobilizers or MobiComs, supported by two supervisors; a second group, the seven IMC Commune Supervisors, are responsible for supporting community volunteers such as the CHWs, including organizing training events; providing post-training supportive supervision for health center staff; assisting with the fairs; and ensuring quality data collection for their respective communes. Delays for P1 have been numerous and are ongoing. According to project staff, the long IEE approval process delayed WASH activities until August 2017 and delayed nutrition activities such as i) Positive Deviance/Hearth (PD/Hearth), known as FARN or Foyer d’Apprentissage et Réhabilitation Nutritionnelle, and ii) community-level nutrition demonstrations (known as FAN or Foyer d’Apprentissage Nutritionnel). Relevant IEE submissions and conditions are outlined under Findings below. However, nutrition demonstrations using the commodities have been ongoing at the fairs since the second quarter of FY2016, shortly after the fairs began, and it is difficult to understand why community-level nutrition demonstrations have not been held since it is essentially the same type of activity. Project staff indicated that Infant and Young Child Feeding (IYCF) training has not yet started because the Ministry of Public Health (MoH) has not released an approved module; the MoH was also cited as the reason Fertility Awareness Methods (FAM, referring to natural family planning) training for CHWs just started in September 2017. Delays in other major activities include the relatively late start for broadcasting radio messages (March 2017); the production of Talking Books (distribution planned for March 2018); the only activities for the Community Scoreboards have been the development of a training module (dated August 2017) and three informational workshops in August-September 2017. Amashiga MTE - FINAL 17 Sub-Purpose 1.1 Individuals practice appropriate behaviors for optimal growth of children Findings for Sub-Purpose 1.1 Findings for IR1.1.1 and corresponding output(s): IR1.1.1: Caregivers have adequate resources to practice appropriate health and nutrition behaviors Output 1.1.1.1: All eligible MCH beneficiaries receive full rations on time - The only Output listed for this IR is the timely distribution of MCHN rations. Quarterly and annual reports, the Recipient Status Reports, the Commodity Status Reports and 39/41 FGDs with women participants, husbands of participants and CoProNuts all confirm that participants receive the rations on time with few exceptions and that no eligible participants are excluded. The Technical Specialist for Commodity Management on the MTE team corroborated this finding. However, problems occur at the nutrition fairs when participants’ cards do not scan correctly or names are not found in the project database. [21/41 FGDs] In these cases, participants may go home empty-handed and not receive the month’s worth of rations. - The fairs were intended to be gathering places for the whole community with a variety of SBCC activities, including games and contests; vendor stalls; demonstrations on gardens, cooking, food preservation, etc.; and other activities of interest to the general public. However, the main activities have been food distribution and mass SBCC with occasional music and contests. The attendees are primarily the female participants; husbands and other community members (other than CoProNuts) come infrequently. When asked why others do not come to the fairs, respondents stated that they see no point in coming if they are not going to get something, especially since it means giving up a half day or more of their time. [41 FGDs with P1 participants] - A positive finding is increased use of maternal and child health services since the project began. For example, data for Muyinga Province from the National Health Information System shows an increase in pregnant women completing the recommended four antenatal consultations (from 14,129 in 2014 to 21,495 in 2016) and an increase in deliveries at health facilities (from 23,535 in 2014 to 24,516 in 2016). See also Annex J.1 for additional information. - It was also noted during Observations at six fairs that screening of children 6-23 months using the MUAC armband is not systematic and not always done correctly. For other issues with the distribution of MCHN rations, see Section 5.1 Commodities. - The rations provide families with the necessary resources to practice appropriate nutrition for pregnant and lactating women (PLW) and children 6-23 months; those interviewed in all 41 FGDs with P1 participants were able to describe the need to transition from the rations to a balanced diet using locally-available products. However, participants stated that they currently lack the necessary resources to make this transition on a sustained basis. - Among the indicators for IR1.1.1 are existence of a household handwashing station with soap and water and an improved “sanitation facility”, referring to household latrines that incorporate the six requirements for an adequate latrine. Only a handful of handwashing stations, Tippy Taps and improved latrines were observed in households during the MTE. Reasons for not constructing these hygiene and sanitation improvements included lack of materials such as a water container for the Tippy Taps or a slab for a latrine; lack of information on how to build a Tippy Tap; and lack of labor to dig a latrine. - In October 2016, CRS included the provision of nutrition messages and services in its waiver request to USAID/FFP pending submission and approval of an Agriculture PERSUAP. This Amashiga MTE - FINAL 18 Figure 2: Father feeds a tomato from his kitchen garden to one of his sons in Mihama, October 2017 (By K. Tilford) was on the stated understanding that this output32 (among others) fell under a “Negative Determination with conditions” classification in the IEE and that implementation was dependent on an approved Agriculture PERSUAP which had not yet been prepared. Findings for IR1.1.2 and corresponding output(s): IR1.1.2: All community members have knowledge of local threats to and appropriate practices for optimal child growth Output 1.1.2.1: Individuals are trained by Amashiga on improved child care practices Output 1.1.2.2: Couples counseled in Fertility Awareness Methods (FAM) for birth spacing Output 1.1.2.3: Local radio stations broadcast programs on optimal nutrition practices Output 1.1.2.4: Community actors are provided with Talking Books to promote improved optimal health and nutrition - Local threats: In 40/41 total P1 FGDs, community members were not able to articulate local context-specific threats to optimal child growth but rather described more general threats such as low agricultural production, lack of knowledge of appropriate IYCF practices, and a lack of understanding between husband and wife, which results in the couple not making good joint decisions for children’s well-being. The exception was one men’s FGD where the participants cited local threats such as sandy soil making it difficult to grow vegetables and the lack of a year-round fresh-water lake limiting fishing opportunities. - Child care practices: In all 14 FGDs with women participants, the participants provided appropriate responses when queried about standards for optimal child health and nutrition. Women in the two non-participant FGDs were also able to describe appropriate child care practices promoted by Amashiga. Men provided less comprehensive answers but both men and women stressed the importance of a balanced diet; taking children to the health center at the first sign of serious illness; and maintaining harmony at the household level to facilitate joint decision￾making around children’s health. - Counseling in FAM: The project conducted a study in April 2017 (Enquête Sur L’Utilisation des Méthodes Contraceptives Naturelles de Planification Familiale) and selected two natural family planning methods to promote: Standard Day Method (SDM) and the Lactation Amenorrhea Method (LAM). Although several women in the FGDs [7/14 FGDs] stated that modern family planning was included in SBCC talks from CHWs, no participants mentioned receiving counseling in FAM. According to two KIIs with P1 staff, training providers and CHWs in these methods was delayed because the MoH wanted modern methods included in the training. The go-ahead from the MoH was received in September 2017; activities to date have been the 32 Previously worded as follows: O 1.3.4: CoProNut have extended nutrition messages and services at nutrition fairs Amashiga MTE - FINAL 19 development of two training modules, one for facility-based providers and one in LAM for CHWs, and the training of 145 CHWs in LAM in September 2017. - Radio: The project conducted an assessment of available radio stations and preferred times for listening. As of March 2017, radio messages are being broadcast on the most widely￾listened-to radio station, RTNB. However, few people in the 41 FGDs in the four communes (Gashoho, Giteranyi, Butihinda, and Buhinyuza) had heard the Amashiga messages or indeed any messages on similar themes. According to those interviewed, factors accounting for the limited effectiveness of this SBCC channel were a lack of radios and an inability to capture the RTNB station Amashiga. This was corroborated by KIIs with the SBCC Technical Advisor and the P1 Coordinator.33 - Talking Books: To date none of the four Talking Book titles have been distributed. The first title is slated for release in March 2018, quite late in the program compared to the original date for distribution, which was Quarter 2 of FY16. [Document review, KII with SBCC Technical Advisor] Findings for IR1.1.3: (See also Section 4.4 XSP for Gender) IR1.1.3: Couples engage in joint-decision making on child health and nutrition (no outputs) - A consistent finding across the 41 P1 focus groups was the importance of harmony at the household level to ensure that parents work together to make the best decisions for the family’s well-being, especially the health of young children. When queried about the direct and indirect causes of malnutrition in young children, FGD participants in approximately 80% of the 41 FGDs with P1 participants stated that maintaining harmony and parents making decisions together were important factors in promoting optimal child nutrition. - The FGD participants noted that they had heard this message from CoProNuts, Ab’IRs, CHWs, and community leaders, including administrative officials. The multiple pathways they cited demonstrate that this message is transversal and widespread. - In approximately half of the P1 participant FGDs, individuals either provided personal testimonials on how they now work better with their spouse or descriptions of how neighbors and friends resolved issues and transformed decision-making to promote family health and welfare. Conclusions and Lessons Learned for Sub-Purpose 1.1 The major achievements for Sub-Purpose 1.1 are the timely distribution of rations to eligible beneficiaries; increased use of health facilities for antenatal consultations, safer deliveries, and other maternal and child services; and the high levels of knowledge on many of the project’s key messages, especially those promoting optimal nutrition practices, the importance of MCHN services, and hygiene and sanitation behaviors for improved health and nutrition. Although knowledge levels are high, there is little evidence that some of the key messages on nutrition (minimum acceptable diet for children 6-23 months and for women of reproductive age) and WASH (latrine construction and consistent hand washing at all critical moments) are actually being adopted. Reasons given by those interviewed include a lack of resources to move from knowledge to practice: adequate quantities of quality food; availability of water; and assistance with constructing latrines, showers, and other WASH infrastructure.34 33 These FGDs covered 14 sous-collines in four of the seven communes; the situation may be better in other areas. 34 For instance, rations are received in a timely manner but while participants, their husbands and the CoProNuts can all clearly articulate the need to transition to a balanced diet using locally-available products, they currently do not have the necessary Amashiga MTE - FINAL 20 An additional reason that participants are slow to adopt some practices may be that the necessary SBCC reinforcement and support for behavior change is lacking. Certain Outputs that would reinforce the SBCC strategy have not been achieved: a high volume of radio messages reaching the majority of participants; Talking Books; Community Scoreboards; FAM counseling; etc. According to reviews of records in nine health centers, quarterly reports, the P1 FGDs, and KIIs with MoH managers and P1 project staff, women participants are frequenting health centers more often for MCHN services. Statistics show pregnant women completing more Antenatal Care (ANC) visits, including ANC during the first trimester; delivering at a health facility instead of at home; and adhering more consistently to childhood vaccination schedules. These increases in use of MCHN services are likely a result of two things: i) a woman who wants to participate must attend an initial antenatal consultation to confirm her pregnancy and she is strongly encouraged to follow through with three additional consultations and ii) CHWs and project staff check health cards during the fairs and if a woman is not attending consultations or her child is behind on immunizations, she is counseled and may also receive a home visit. The increased use of health center services may decline when distribution ends, especially the number of women seeking ANC during the first trimester of pregnancy. However, the majority of participants in the women participant FGDs and the CoProNut FGDs stated that most of these behaviors are now acquired behaviors, that husbands support them, and that they are confident these behaviors will continue post-project. Heightened gender awareness and changes in behavior at the household level (such as sharing responsibilities more equitably and joint decision-making, including for child health issues) are positive outcomes that can be traced to activities for the cross-cutting Sub-Purpose for Gender. The MTE team noted that interviews with the P1 project staff and quarterly reports showed a tendency for some staff to adopt a passive attitude when activities were blocked instead of demonstrating proactive and adaptive leadership, e.g., developing alternative plans or taking action on non-blocked activities. For example, given the time it takes to develop the text and images for a Talking Book, staff could have started developing the text for the second book instead of waiting until the first one is produced. In addition, even if an activity did not seem practical or logical, staff would refer to the proposal document as a reason to implement the activity as planned. Sub-Purpose 1.2 Health providers deliver high quality, gender-responsive MCHN services to women and children Findings for Sub-Purpose 1.2 Findings for IR1.2.1 and corresponding output(s): IR1.2.1: MCHN service providers know optimal nutrition behaviors and local threats to chronic malnutrition Output 1.2.1.1: CHWs and health center MCHN service providers trained in how to address local threats to under-2 nutrition 111 MCHN providers, 830 CHWs, and 221 Agriculture Monitors were trained in nutrition, including PD/Hearth. [Document review, two KIIs with P1 managers] resources (i.e., availability of and access to varied foodstuffs and increased household revenue) to effect the transition. This conclusion was also reached by the consultant who conducted the positive deviance study in Giteranyi Commune in April 2017: he noted that knowledge levels were high but people were not able to practice optimal child nutrition due to availability and accessibility issues for food and a lack of household income. (Rapport de L’Enquete Deviance Positive – juin 2017) Amashiga MTE - FINAL 21 Using KIIs, evaluators interviewed 27 providers at health centers, including 22 who received at least one training. Of the 27 interviewed, 26 demonstrated knowledge of optimal nutrition behaviors and general (but not local) threats leading to chronic malnutrition in young children. The providers were more likely to cite health problems (such as illness) as a threat to optimal nutrition rather than other general threats such as low agricultural production. Findings for IR1.2.2 and corresponding output(s): IR1.2.2: MCHN service providers implement national protocols that support positive growth of children ≤2 years of age in targeted communities Output 1.2.2.1: MCHN service providers are trained and equipped to implement national nutrition protocols including identifying and addressing chronic malnutrition among ≤2 children & PD Hearth - 62 providers were trained in clinical IMCI, 149 in use of the Mother-Child Health Card, and as noted above, 111 in nutrition and PD/Hearth [Document review, five KIIs with project staff and MoH personnel] - Observations at nine health centers showed that providers have the necessary training, equipment, and clinical protocols to support optimal child development but they do not consistently implement the national protocols, that is, they do not follow all the steps outlined in the protocols. Staff followed all steps in only one of nine centers observed. - Project records and KIIs with MoH and P1 staff showed that some providers were trained in up to three modules whereas other providers in the same health center received no training through Amashiga. The explanation given by project staff in two KIIs was that they have no control over who is trained as the Head Nurse at each facility selects the trainees. Also, approximately half of the 27 providers interviewed noted that debriefing sessions do not routinely take place when colleagues return from training and that sometimes those trained are transferred, resulting in a loss of knowledge. - Counseling cards were not visible at the nine health centers observed. - The twelve mothers and three fathers interviewed via 15 KIIs pronounced themselves satisfied with the services provided at health centers for their children but stated that they had to wait for a long time for these services. - The project’s PD/Hearth strategy document (Stratégie Pour la Mise en Oeuvre des FAN/FARN) proposes that communities provide the food for the 12 days of FARN, not the mothers participating in the sessions. This was confirmed in KIIs with project staff who stated that this mirrors the government’s FARN policy. [Document review, two KIIs with P1 managers] Findings for IR1.2.3 and corresponding output(s): IR1.2.3: MCHN services are delivered in a gender-responsive way Output 1.2.3.1: MCHN service providers trained to provide gender-responsive services - To inform training, the project conducted an assessment of SBCC and gender in health centers. - 45 providers participated in gender training. [Document review] - According to the IPTT, the target is to provide gender training to two providers at each of the 47 health centers. However, only one provider per health center has been trained as of October 2017; the MoH has not made other providers available for this training. - Other than providers stating that they gave priority to pregnant women, none of the 27 providers interviewed were able to define or give examples of gender-responsive services or a gender-responsive facility. [KIIs with 27 providers at health centers] Amashiga MTE - FINAL 22 Figure 3: Woman participant with her child in Gatanga, October 2017 (By K. Tilford) Conclusions and Lessons Learned for Sub-Purpose 1.2 Targets for training providers and CHWs have been achieved for nutrition, including PD/Hearth, and use of the Mother-Child Health Card; targets have been partially achieved for clinical IMCI and gender but no targets have been reached for IYCF or FAM training. However, even when targets are achieved, they only represent quantitative Outputs and not Outcomes. At the Outcome level, observations at health centers and KIIs with health providers and project staff showed that even with supportive supervision, providers at facilities are not uniformly delivering high￾quality, gender-responsive MCHN services. Providers and project staff gave several reasons why the training and supportive supervision have not had the desired result including i) a lack of staff at health facilities means that nurses have a heavy case load and not enough time to follow the protocols for each patient and ii) the project’s Commune Supervisors (responsible for supportive supervision) are too few to effectively carry out all their responsibilities. The lesson learned for Amashiga and future projects is that unless providers have the time, motivation and a reasonable case load, it will be difficult for them to apply any training received. The training serves mainly as an investment in capacity reinforcement and may not necessarily result in improved services. The other conclusion for Sub-Purpose 1.2 concerns the PD/Hearth strategy: as it is currently described in the Amashiga strategy document, the FARN component of PD/Hearth will not be sustainable, will be difficult to implement in a food-insecure setting, and is unlikely to promote a strong sense of self-efficacy among poor mothers who want to learn how to use their modest means to improve IYCF. Reasons for this conclusion are: no PD strategies have been identified for dissemination; as noted in a June 2017 study commissioned by the project, food prices in the proposed pilot commune (Giteranyi) are high and availability of diverse food items is limited (Rapport de l’Enquete ‘Deviance Positive’ dans la Commune de Giteranyi); the community is expected to provide all the food for the 12 days of demonstrations, meaning participants do not have the opportunity to see how their limited resources could be used to prepare adequate complementary feeding dishes; and unpaid CoProNuts and CHWs will be asked to devote considerable time and effort to the activity whereas a group of Lead Mothers from previous projects (Mamans Lumières) will be paid to carry out the nutrition demonstrations, possibly creating friction between the Lead Mothers and the volunteers. Sub-Purpose 1.3: Communities maintain a positive social and physical environment to support good nutrition for children under-5 and pregnant and lactating women Findings for Sub-Purpose 1.3 This Sub-Purpose focuses on improving the physical environment through WASH activities and promoting a positive social environment for good nutrition by engaging the whole community, the CoProNuts’ mandate. Amashiga MTE - FINAL 23 Findings for IR1.3.1: IR1.3.1: Neighbors and family support caretakers' optimal health and nutrition behaviors (no outputs) - There were no activities planned for IR1.3.1 in the DIP submissions for FY15, FY16 and FY17 nor were any activities mentioned during the KIIs and FGDs. The two indicators did not lend themselves to being measured during the MTE. .35 Findings for IR1.3.2: IR1.3.2: Community maintains a clean environment with adequate sanitation and water (no outputs) - Other than SBCC messages and some demonstrations for compost pits, dish drying racks, and latrines, most of the WASH activities have not yet started due to environmental compliance concerns. At the time of its IEE submission in April 2015, CRS considered that the output36 had not been sufficiently defined (more than six months after the start of the project) to enable a thorough analysis of potential environmental impact. CRS therefore asked for a classification of “deferral” until such time as the details could be confirmed. The organization committed to submitting a request for modification to USAID/FFP/BEO by 31 March 2016 at the latest. In July 2016, the Amashiga Chief of Party (COP) instructed team members that this activity (among others) could not be implemented prior to receiving FFP approval. In October 2016, CRS provided clarification by email to USAID/FFP that this output consisted mainly of identifying lessons and emerging practices in promoting WASH; mapping out efforts in WASH promotion and conducting formative research on WASH behaviors; establishing and training new water associations; and training masons in latrine construction. CRS did not, however, formally request an IEE amendment and reclassification of the output, before its August 2017 submission at which time its revised IEE included a detailed explanation of the activities and proposed mitigation actions for the output. The BEO approved this submission in November 2017. - As of August 2017, project staff decided to move forward with WASH activities: draft manuals have been developed for WASH and for Participatory Hygiene and Sanitation Transformation (PHAST) and the project plans to train P1 MobiComs, the MoH Health Promotion Technicians, and all Water Associations in WASH-related topics. - Although a PHAST module was developed for training P1 MobiComs, it is not planned that the MobiComs will actually implement PHAST in all sous-collines. - The WASH Project Officer has worked with the commune-level water authorities to update the list of the 3,300+ Water Associations that are responsible for water points in the province. According to the KII with the Provincial Coordinator for Health Promotion, many of these associations have not been functional in years. - Compost pits and drying racks are the most widely-adopted WASH improvements promoted by the project according to FGDs with women participants, their husbands, and the 14 CoProNuts interviewed; this was confirmed during observations in communities during the field work. - According to participants in the P1 FGDs, latrine construction is the most difficult WASH intervention to adopt as it requires building materials, space, and physical labor. 35 IR1.3.1: Neighbors and family support caretakers' optimal health and nutrition behaviors 1.3.1.1.: % of mothers of under-2s who received support from neighbor or family member outside the HH while pregnant and 1.3.1.2.: % of parents that received something from a neighbor or family member outside the HH to support their child <2 years of age 36 Previously worded as follows: 1.2.3: HH have increased use of hygiene behaviors, potable water and sanitation infrastructure. Amashiga MTE - FINAL 24 - During the KIIs with the WASH Project Officer and eight P1 MobiComs, they acknowledged that adding WASH activities to the MobiComs’ responsibilities (e.g. working with Water Associations and starting more intensive community-level work to promote hygiene and sanitation) will seriously stretch their ability to do quality work, especially as i) the MobiComs spend a week or more each month organizing the fairs and ii) community-level nutrition activities such as demonstrations are also planned in Year 4. - The project does not intend to devote any financial resources to building or improving water and sanitation facilities. [KII with WASH Project Officer] Findings for IR1.3.3 and corresponding output(s): IR1.3.3: Functioning CoProNuts engage their entire community to promote optimal growth for children under 2 years of age (no outputs) According to the project proposal, the CoProNuts were established to be a permanent, sustainable structure at the sous-colline level that would continue to mobilize communities post￾project. The MTE team used FGDs to interview 14 randomly selected CoProNuts and also included the topic of CoProNuts in almost all of the PI KIIs and in the FGDs with women participants and husbands of participants. According to these interviews: - The CoProNuts are widely respected and appreciated for their SBCC activities (health, nutrition, gender and agriculture) and for their role in organizing the monthly fairs. [14/14 FGDs with P1 women participants and 12/13 FGDs with P1 male participants] - CoProNut members are all volunteers and represent a broad spectrum of community interests. They are headed by the CHW, providing a direct link to the health center, and members include at least one administrative authority, usually the sous￾colline chief, which provides additional legitimacy and credibility in the community’s eyes. [14/14 FGDs with CoProNuts; corroborated by 26/27 FGDs with P1 participants and project reports] - CoProNut members mentioned that they devote considerable time and effort to Amashiga activities, yet they receive no compensation or concrete manifestations of appreciation; they also stated that they have not benefitted from any formal training events, just informal “orientations” on various Amashiga themes such as nutrition, kitchen gardens and other messages promoted through the project. [14/14 FGDs with CoProNuts] Men and women interviewed in approximately two-thirds of the 27 P1 participant FGDs also requested some form of compensation for the CoProNuts. - Although most CoProNut members stated that they are committed to fulfilling their mandates, approximately one-third indicated that they may not continue since their own economic activities are neglected because of the time spent on Amashiga. [14 FGDs] - There is no written guidance for rewarding and encouraging the CoProNuts and no sustainability strategy for ensuring that CoProNuts continue to function post-project. Figure 4: Rucikiri CoProNut representatives, September 2017 (By K. Tilford) Amashiga MTE - FINAL 25 - One of the planned activities for the CoProNuts was to organize nutrition demonstrations at the community level. As of the writing of this report, those demonstrations had not yet started. The delay to nutrition demonstrations at the community level is attributed to the August 2015 IEE approval Condition Number Four. At the time of its first complete IEE submission in April 2015, CRS requested that the output37 be classified under “Negative Determination with conditions” in IEE. The USAID BEO Environmental Threshold Decision (August 2015) Approval Condition Number 4 stated that CRS would need to effectively monitor performance indicators for fuel-efficient cookstove activities. This was because their effectiveness had not yet been demonstrated and there were no clear associated monitoring and evaluation activities. In July 2016, the Amashiga COP instructed team members that the activity (among others) could not be implemented prior to receiving FFP approval. In an email to USAID/FFP in October 2016 on the various pending IEE approvals, CRS accepted FFP’s assertion that cookstove interventions were highly complex and required close oversight but clarified that the use of improved cook stoves was a marginal activity in Amashiga designed to enable participants to consider a better cooking method and with the potential for positive environmental impact. Finally, CRS requested that FFP consider this use of improved cook stoves to fall under the “categorical exclusion” classification. In a response later that month, USAID/FFP addressed a separate issue but did not acknowledge this request for approval. Finally, in the August 2017 resubmission, CRS considered still that this was an activity likely to have some environmental impact and again recommended a classification of “Negative Determination with Conditions”. Approval was received in November 2017. Conclusions and Lessons Learned for Sub-Purpose 1.3 As a result of community-wide meetings and home visits focusing on WASH interventions, there have been some improvements in the physical environment: construction of drying racks and compost pits is widespread and some households are improving their latrines. According to the P1 FGDs and KIIs with project staff and government partners, the CoProNuts have also created a more supportive social environment for promoting better nutrition for young children and PLW. The CoProNuts have succeeded in reaching large numbers through mass SBCC carried out with MobiComs, CHWs, Ab’IRS and community authority figures. Because the CoProNuts represent a broad spectrum of community interests and religious denominations, they have also been able to disseminate project messages through the individual groups and associations that elected them to the CoProNut. Starting in Year 4, project plans include increased emphasis on nutrition demonstrations and WASH improvements at the community level. Given Mobicoms’ involvement in the fairs and their current workload with a minimum of 10 CoProNuts (and in more densely-populated areas a MobiCom may have more than 40 CoProNuts), a team of only 23 P1 field agents is inadequate to organize 33 fairs a month, provide adequate support and ongoing training to 800+ CoProNuts, and initiate quality work in nutrition demonstrations and additional WASH activities. In addition, five of the eight MobiComs interviewed in KIIs stated that project staff across purposes need a place at the commune level to meet for planning purposes and a shared computer. The project’s plan to provide training in budgeting, water fee management, and maintenance and small repairs for 3,300+ Water Associations is unrealistic. These Water Associations need more 37 Previously worded as follows: Output 1.3.4: CoProNut have extended nutrition messages and services at nutrition fairs. Amashiga MTE - FINAL 26 than one or two days’ training if there is to be a long-term, sustainable Outcome – they need qualified staff to work with them on an ongoing basis. The CoProNuts are active, engaged and represent a wide variety of community interests. However, those interviewed for the MTE are dissatisfied with the lack of incentives they receive for the level of effort they provide and certain CoProNut members expressed some doubts about their ability and/or willingness to continue Amashiga activities. The project does not have any concrete plans to encourage them or to ensure that they are sustainable structures post-project. P1 Overall Conclusions Overall, many of the P1 activities are significantly behind schedule, especially for Sub-Purposes 1.1 and 1.3; this is due to delays within the project’s control (e.g., slow start for activities such as Talking Books and Community Scoreboards) and to circumstances that to some extent are beyond the project’s control: no field activities in Year 1 due to insecurity; delayed IEE approval, affecting especially the WASH component; and delays on the MoH side in validating modules for IYCF and FAM. As for Sub-Purpose 1.2, many of the training events for facility￾based providers and CHWs have taken place but not according to the original calendar. In spite of these delays, there have been a number of achievements, including the following: - The food distributions are going well and it is a major achievement to ensure that over 45,000 participants receive the correct rations on a monthly basis. - The P1 team has been careful to harmonize its health and nutrition approaches with the GoB’s priorities, protocols, and procedures. During the first year, IMC team members collected and studied all the key documents relating to health and nutrition (MoH, UNICEF, WHO, other organizations, etc.) and designed Amashiga’s interventions to align with the MoH and international standards. - Although there is room for improvement, the CoProNuts’ activities and other SBCC initiatives have raised knowledge levels on key Amashiga themes such as nutrition, hygiene and sanitation, optimal child health, and gender equity. - There is a heightened awareness of the importance of husbands and wives working together to make decisions about family resources and family health, especially the health and nutrition of young children. - Before initiating major interventions, the P1 team has been conscientious about conducting assessments and carrying out studies to identify prior experiences in Muyinga Province, best practices that have worked, and possible challenges that may need to be addressed. Examples noted in the findings include the study on gender and SBCC in health centers, the radio assessment, the PD/Hearth survey, and the FAM study. The results of these studies, which are shared with communities and government counterparts, facilitate the design of new activities that have a greater chance of success since they are based on the current context and on lessons learned from previous experiences. - As for integration with other purposes, there is a strong link between P1 and the XSP for Gender. There is also a tenuous link to P2 as the CoProNuts promote kitchen gardens and other agricultural themes such as the importance of composting. - Project staff have worked to develop synergy with the other major USAID-funded initiative in the province, the Integrated Health Project/Burundi (IHPB). Project reports and staff describe this synergy as sharing responsibility for training in Integrated Management of Child Illness (IMCI) and family planning. Amashiga MTE - FINAL 27 Challenges that need to be addressed to improve P1 performance include the following: 7. Status of P1 staff: First and foremost, as of October 13, 2017, all 23 P1 MobiComs, the primary field agents who work at the community level, have been let go; most had served two years and had benefitted from multiple training sessions in SBCC, nutrition, kitchen gardens, and other topics. Although CRS is now recruiting new agents, it will be several months before the hiring process is completed and new MobiComs are trained in the essential topics. This will result in a serious disruption of community-level activities just as the project plans to gear up nutrition and WASH in the 800+ communities. Second, the number of field staff overall is inadequate compared to the activities, number of health centers, and number of communities. This includes the MobiComs, their supervisors, and the IMC Commune Supervisors who work with health facility providers and with the community volunteers such as CHWs. 8. Forward planning and setting realistic targets: The emphasis is currently on quantity of activities and not on quality. This means carrying out activities on a somewhat superficial level, setting unrealistic targets, and not having adequate plans for follow-up (e.g., training hundreds of providers, distributing thousands of Talking Books, training 3,300 + Water Associations, working effectively with 800+ communities to improve the physical environment, and constructing 200+ Community Scoreboards). Little attention is paid to considering all the steps to ensure that these interventions are implemented with the necessary follow-up, community involvement, and attention to quality and sustainability. If the project continues to focus primarily on achieving quantitative Outputs, this will produce few Outcomes and will negatively affect results and sustainability. 9. CoProNuts: The CoProNuts interviewed during the MTE all expressed dissatisfaction with the lack of recognition and remuneration for their efforts. Since Amashiga was designed with the CoProNuts as a major factor for project success and sustainability, this has serious implications for implementation during the remaining two years and for sustainability. 10. SBCC: The SBCC approaches for P1 are limited and some of the key channels to reinforce behavior change are not yet available (Talking Books, Community Scoreboards and FAM counseling) or reach only a small audience (radio and nutrition fairs). The focus on sensitization in large groups such as community meetings and fairs and not on negotiations and support for behavior change is not conducive to lasting change. The lack of didactic materials, teaching aids, other visual media, and practical demonstrations also hampers SBCC efforts. 11. WASH: Since much of the WASH component is starting three years into the project, it is unrealistic to expect that the original action plan and targets are still valid and can be achieved in the time remaining. This is especially true for working with the Water Associations and introducing PHAST. 12. Training providers at health facilities and CHWs (Sub-Purpose 1.2): The main concern with Sub-Purpose 1.2 is that providers at the facility level are not systematically applying the training; this constraint is clearly beyond the project’s control. However, three other areas also raise concerns: efforts to create a more gender-responsive environment at health centers have not been effective; FAM training is starting late and there is a lack of appropriate didactic materials. 13. Relations with the MoH: KIIs with the three district medical directors showed that two of the three directors appreciate the training that Amashiga has sponsored for health care providers but all three directors stated that the project does not keep them informed about activities at the community level, including the nutrition fairs. They do not receive Amashiga MTE - FINAL 28 information on who benefits from the rations, have not visited a fair, and although they have heard of the CoProNuts, they do not know their responsibilities and activities. 14. Sustainability: At this point, sustainability may include knowledge of key messages, the adoption of some practices that require few resources such as accessing MCHN services and building drying racks, reinforced capacities for health care providers and CHWs, and perhaps the continuation of some SBCC activities by CHWs, community leaders and other CoProNut members. The widespread sustainability of the CoProNuts as permanent structures is unlikely. P1 Recommendations (in relative order of priority) 1. Status of P1 staff: - Increase the number of P1 field agents (MobiComs) and supervisors so that there is adequate coverage for current and planned activities. This could include increasing the overall number of MobiComs, adding agents specifically for WASH activities, increasing the Commune Supervisors from one per commune to two, and increasing the number of MobiCom supervisors from two to four. - Prepare a set of streamlined, practical training modules and didactic teaching aids for all new staff, especially the MobiComs currently being recruited. 2. Forward planning and setting realistic targets: - Review the IPTT and where not already included, establish desired Outcomes for sets of Outputs. - Prepare a detailed 18-month calendar with all planned P1 activities. Determine if it is really possible to accomplish all activities in all 800+ sous-collines in a quality fashion with the necessary steps to achieve sustainable Outcomes. Consider a realistic workload for field agents and whether the CoProNuts are willing to increase their level of involvement. - Negotiate with USAID/FFP to reduce activities and lower targets so that the P1 team can do less but do it well. Don’t focus only on numbers and Outputs but rather on Outcomes and on all the steps needed to ensure that an activity is well-executed, appropriated by the MoH and/or the community, and sustainable. - Drop the FARN part of the planned PD/Hearth activity (nutritional recuperation). 3. CoProNuts: - Continue to give priority to CoProNut members for P2 activities and inputs. - Organize formal appreciation ceremonies for CoProNuts, including individual certificates describing their contributions to Amashiga and their communities. 4. SBCC (includes nutrition demonstrations, Talking Books, Scoreboards and radio): - Add community-level nutrition demonstrations (FAN) using local products and focusing on balanced meals for pregnant women and complementary feeding for different age groups of children. Ensure that groups remain small, that participants bring most of the food products, and that husbands participate in at least one demonstration. - Instead of demonstrating how to prepare rations at the fairs, substitute nutrition demonstrations using locally-available products. - Using the suggestions in the SBCC strategy document as a point of departure, develop a detailed plan and timeline for distributing the first Talking Book. - Begin developing the content for at least one additional Talking Book title. - Since it will be difficult if not impossible to complete all the steps for establishing Community Scoreboards in more than 200 communities, establish criteria for selecting a realistic number of communities in which to install the Scoreboards. Complete all the Amashiga MTE - FINAL 29 necessary steps to ensure that the Scoreboards are maintained post-project. Determine why the Scoreboards (Community Mirrors) set up by Pathfinder are not being maintained so that Amashiga can anticipate potential problems and devise solutions. - Conduct a rapid quantitative survey in all seven communes to determine if Amashiga radio messages are reaching a substantial portion of the population. If not, invest the resources in other SBCC channels. (Note: If it is already known where people cannot capture RTNB, then these areas do not need to be surveyed.) 5. WASH: - Establish criteria to select a manageable number of Water Associations to work with during Years 4 and 5. Develop i) a realistic training module that focuses on key Water Association functions, ii) a plan for follow-up work with each association to ensure that they are able to practice what they have learned and iii) a plan for engaging the community in supporting the Water Associations in their work. - Decide whether it still makes sense to conduct PHAST training for project staff and counterparts. If MoH staff, CHWs and/or P1 staff will not be able to apply this training in a concerted fashion at the community level, find a more efficient way to work with communities to improve the physical environment. This may require reducing the number of communities that benefit from this activity. - For future projects: Projects to improve nutritional status should include plans for assisting communities to improve their water and sanitation infrastructure either by using project resources or by securing a firm commitment from other agencies and organizations. 6. Training providers (gender, FAM, supportive supervision, and nutrition screening): - Before training any additional providers in gender, revise the module to make it more practical, including changes providers can make to the physical setting to provide more gender-responsive services. - Before training providers and CHWs in FAM, have didactic teaching and counseling aids available for each trainee to practice with and to keep. - Since Amashiga is only promoting two natural family planning methods, streamline the current FAM module for health center providers to include only those methods. A day-long workshop will be sufficient; include reinforcing counseling capacities. - Ensure that FAM training for CHWs is practical, focuses only on the two methods, and includes reinforcing counseling skills via simulations and role plays. - Increase the number of supportive supervisions, including joint supervisions with the District Health teams; revise the checklists to more accurately cover the key points for each training. - Designate one CHW to conduct MUAC screening at fairs (and only this activity) for children 6-23 months. Ensure that a CHW or Amashiga agent follows up all children in the red and yellow zones with a home visit and referral to the health center if needed. Train two CoProNut members on the MUAC measurement to assist the CHW at fairs. 7. Relations with MoH: - Establish a monthly meeting with each District Health Office to i) share the data collected at fairs (number and category of participants; total amount of rations distributed; and results of MUAC screening for children 6-23 months) and ii) work collaboratively to improve the referral system for malnourished children. - Establish a program of quarterly visits to fairs for MoH representatives from the provincial, district, and health center levels. Include one or two visits a year by the Governor. Amashiga MTE - FINAL 30 - Prepare a quarterly report for each District Health Office on community activities in their district. Include plans for the next quarter. Share with the Provincial Medical Director and the Governor’s office. - Improve planning with the health centers and the District Health Offices to ensure that Amashiga activities are included in the Annual Action Plans. 8. Sustainability: - In collaboration with CoProNut representatives, develop and implement a sustainability plan for the CoProNuts. Consider what activities they could continue post-project and what they need in terms of motivation, support, resources and capacities to carry out these activities. 4.2 Purpose 2: HH have continuous access to adequate nutritious food in Muyinga Introduction to P2 Activities P2 activities were designed to be complementary to P1 activities by improving sustainable access to more varied food and by increasing home production and incomes. P2 participants (about 800 Lead Farmers, 250 Farmer Groups, 32 Agro-Forestry nurseries, 39 Seed Multipliers) were not selected on the basis of P1 participation but rather on the basis of real or potential agricultural performance. In the case of FFA and seed fairs, however, the targeting was based on vulnerability criteria. There was no specific targeting for savings and credit (SILC) groups other than to target areas not yet served by such associations. CRS has a P2 coordinator who supervises four sector managers (agriculture, seed, NRM, SILC). The main partner is RBU2000+ who employs two project managers, 14 agronomists (2/commune) and seven SILC supervisors who supervise 42 SILC field agents (not staff). Bioversity International has one field staff member to supervise the banana research activities. ODEDIM has responsibility for the implementation of the FFA/Food for Work (FFW) activities through one FFA/FFW coordinator. The core activities have revolved around the CRS tried and tested method of introducing and multiplicating improved seed (bean, corn) and new plant varieties (cassava, OFSP) supported by improved in-field agro-ecological practices and the creation of hundreds of SILC. By the time of the evaluation there were three levels of progress: i) a few activities had proceeded nearly unimpeded and were on track to finish (SILC, FFA for Contour Bunds), ii) some had experienced delays (seed multiplication, increased food production) and, iii) some had not started (improved market access, goat solidarity chain, youth employment). Findings Field activities started in late 2015 due to the insecurity. Other delays in 2016 and 2017 are attributed to the issue of the IEE and PERSUAP approval. There was an uncommon drought between November 2016 and April 2017 which disrupted agricultural activities. Table 7: P2 Summary of Key Indicators Actual Target Comment Sub-Purpose 2.1 LF >1,000 810 PO 300 Not many operational POs in province Amashiga MTE - FINAL 31 Actual Target Comment Agro-forestry nurseries 28 850 4/commune SMG/IMS 39 ¼ not certified as non-compliant Seed fairs for vulnerable HH 12 21 8,100 participants, 65t distributed VitA banana testing 2 2 SDSR/BXW banana LF 62 62 Research, not dissemination Km of Contour Bunds 255 NA AEAs (1/colline) 135 135 Though working in 210 collines Goat solidarity chain 0 18,550 1,485 initial goats and HH in 2018 Sub-purpose 2.2 PSPs 35* 74 * from 42 Field Agents SILC groups 629 729 Now increased to 1,800 FFA participants 3,129 3,728 4 sessions so far Km of road rehabilitated 0 400 # of grain stores rehabilitated 0 12 Post Harvest infrastructure 0 670 m3 Special Food fortification (t/yr) 0 7,500 7,500t/yr provided by Amashiga Delays attributed by Amashiga to Environmental Compliance Considerations At the time of its IEE submission in April 2015, CRS requested that a number of outputs under P2 be classified under “Negative Determination with conditions” considering that they could have negative environmental impacts. CRS committed to commissioning a PERSUAP and using the results to ensure the appropriate use and handling of pesticides in project implementation areas. The USAID BEO Environmental Threshold Decision (August 2015) Approval Condition Number Three directed that the project comply with USAID’s Pesticide Procedures, Regulation 216.3 (b) before using or promoting pesticides. It specified that any use or promotion of pesticides, including biological pesticides, could not commence without a BEO approved PERSUAP. It further stated that “building linkages is the same as promotion.” In October 2016, in response to the August 2015 approval and condition from USAID BEO, CRS requested and obtained a waiver to continue selected P2 activities in Q1 of FY17 in parallel to the preparation of the Agriculture PERSUAP. CRS noted in its waiver request that USAID BEO’s statement that “building linkages is the same as promotion” led them to consider that five project outputs, (four of which fell under P2 and related to the increasing access to inputs for crops; the adoption of improved farming techniques, practices to reduce pre and postharvest losses; and sustainable forage/fodder and feeding practices) could all be seen to “build linkages” with pesticide use even though almost no farmers use pesticidesin Burundi because most could not afford them. CRS argued, therefore, that the proposed Amashiga activities would not be creating any new linkages with pesticide use but rather involved “working with people who are already doing the activities that could be linked to pesticide use.” The USAID BEO gave CRS permission for limited use during the 2016 planting season of 12 pesticides, as specified in the preliminary pesticide assessment report submitted by CRS in October 2016, pending the submission and approval of a complete Agriculture PERSUAP. As noted in Section 2.3, the Amashiga MTE - FINAL 32 Agriculture PERSUAP was submitted in February 2017 and approved in June 2017.38 The amended IEE was approved by USAID/FFP/BEO during the MTE in November 2017. Sub-Purpose 2.1: HHs have increased self-supply of diverse food Findings for Sub-Purpose 2.1 Findings for IR2.1.1 and corresponding output(s): IR2.1.1: Farmers and POs adopt improved agricultural and natural resource management techniques and choices Output 2.1.1.1: Farmers and POs have more knowledge of improved agricultural and NRM practices Output 2.1.1.2: AEAs and LFs provide gender-responsive agricultural advisory services and information to farmers - In October 2016, CRS included the activity of training in improved cultivation techniques and sustainable forage/fodder and feeding practices for livestock in its waiver request to USAID/FFP pending submission and approval of an Agriculture PERSUAP. This was on the stated understanding that these outputs39 (among others) fell under a “Negative Determination with conditions” classification in the IEE and that implementation was dependent on an approved Agriculture PERSUAP. - A few hundred farmers have adopted improved techniques, mostly those who received improved corn or bean seed and cassava or banana cuttings. [16 FGD] Most farmers, DPAE agronomists and AEAs interviewed negatively commented on the project's plant material distribution as either coming late by a few weeks (when others have already planted) or not coming at all. [14 KII, eight FGD] - The majority of Farmers and POs have knowledge of improved agricultural and NRM practices. [22 FGD] Most farmers cite line planting as an improved practice though few can describe the appropriate spacing and consider it applicable only to improved, bought or donated seed [20 FGD]. Most cite the value of fertilizing with manure but few have livestock or the means to buy it. [18 FGD] Most also cite the importance of contour bunds both for soil conservation and forage grass production. [25 FGD] - The project promotes the adoption of the climbing bean which has a 50-100% higher yield than the traditional dwarf bean. It was found that many farmers still preferred to grow dwarf beans as they have difficulty finding poles to support the climbing bean. The project has introduced the pole and string method of support but there is limited adoption of this practice. [Two KII, five FGD] - Quality vegetable seed for the kitchen gardens is available in Muyinga town and in some commune markets [5 FGD, one Observation]. For the first time since the beginning of the project, in October 2017, the project was planning to give: 42kg of vegetable seed to 4,200 participants (600/colline: 10 LF, 40PLW and 10 CoProNut members). They would each receive 10g from the following (no choice proposed): amaranth, cabbage, pepper, onion, carrot or eggplant. [Two KII] 38 Email from CRS on Oct 5, 2016 to USAID; Subject: Amashiga Responses to IEE Issues and Request for Waiver to Continue with Selected P2 Activities; Response by email granting waiver on Oct 19, 2016. 39 Previously worded as follows: O 2.1.4: Farming HH and communities have adopted improved approaches for gender￾responsive and climate-resilient land and water management systems and O 2.2.3: Farming HH have adopted sustainable forage/fodder and feeding practices for livestock. Amashiga MTE - FINAL 33 - Most of the POs are small with limited capacities; there was an active PO network for accessing microfinance institutions (MFIs) and markets in one instance only [Nine FGD]. - Most HH already have kitchen gardens and interviewees preferred the Amashiga keyhole gardens, which are also copied by neighbors. Water and access to vegetable seed are cited as the main constraints to their use. The main vegetable grown is amaranth with occasionally a few other vegetables. [Nine FGD/KII and seven observations] - Amashiga introduced the addition of a cover to the existing practice of compost holes to reduce evaporation. This is implemented about half the time due to the difficulty in obtaining poles. [Four FGD, three observations] - Poorer farmers are aware of the improved techniques exhibited by the Lead Farmers but have not put them into practice as they understand them to be contingent on having a large landholding and livestock. [11 FGD, five KII] - There are two trial banana nurseries for seven normal varieties and one field demonstration site for three vitamin A rich varieties. There are also four sites in two communes with 62 Lead Farmers for testing SDSR as an effective method for the reduction of BXW, with the incidence reduced from an average of 28% to below 0.7% in this group. The current contract with Bioversity International ends in October 2017 but a no-cost extension for 2018 has been agreed to start dissemination of the results. [Two KII, one observation, one FGD, document review] - Banana farming was largely the domain of men (for sale and beer making) but now women are participating both in the cropping and decision￾making.[Three FGD, two KII] - Distribution of free hybrid corn seed for the 2017 C season for 42 ha in the lowlands was a project adaption, as Amashiga initially had planned to work only during seasons A and B. This coincided with the arrival of the army worm infestation (with a very limited response capacity with pesticides) and in parallel to the government's effort to sell the seed at a very subsidized cost. [Three FGD, two KII, two observations] - The AEAs and LFs have not received training or coaching on gender-responsive service provision. “Gender responsive” is only understood to mean that women and men share in the Figure 5: Keyhole garden in Muyinga Province, October 2017 (by B. Crenn) Figure 6: Composting hole with a roof in Muyinga Province, October 2017 (by B. Crenn) Figure 7: Hybrid maize grown in the lowlands. Muyinga Province, October 2017 (by B. Crenn) Amashiga MTE - FINAL 34 decision making as promoted in XSP1. [Three KII] When possible AEAs are involved in Amashiga activities, although this causes problems with the DPAE as there is no formal cooperation agreement. The most frequently cited example is that the AEAs only work two days/week for the DPAE and the commune agronomists often find their AEAs already busy with Amashiga activities, without being informed. [Five KII] - There will soon be a distribution of improved varieties of sweet potato (2.5 million pieces) and cassava (1.3 million cuttings); the mode of dissemination to onward participants will be through a solidarity chain system and POs but details on this approach were not well understood/articulated by project staff. [Two KII] Thus far the project has been informally acting as an intermediary i) between the SMGs and the DPSP/ISABU by assembling and presenting the necessary documentation; ii) between ISABU and the SMGs by transporting the seed, and iii) between the ONCCS and the SMGs by paying all the certification costs. There are no formal long-term agreements between the project and any of these national institutions. [Five KII, document review] - Training and dissemination of knowledge to LF and PO consists mostly of theoretical training from project staff using only flipcharts and copybooks without actually showing the techniques or having participants practice in front of the trainers as might happen in a demonstration plot. At the time of the MTE field work (September/October 2017), demonstrations plots were planned in some areas. The fact that they have not yet been set up and that there was no apparent plan for how they could be used to change agriculture practices (e.g., by whom and with what frequency) means that they will be of limited value. There is no complete plan for cascade training to other LF and PO members. [16 FGD, five observations, document review] - Bioversity International's partner training in Integrated Pest Management, participatory learning approach and improved banana seed system used the only project teaching support material for the Banana Xanthomonas Wilt (BXW). - Although SBCC should be an integral part of P2 as the project is asking participants (and others) to change their attitudes and behavior, only one training was delivered to 9/23 of the RBU staff in 2016 and there has not been further SBCC activity in P2 since. [Three KII]The nurseries visited had 5-6 agro-forestry tree varieties but only one type of fruit tree, avocado and those were aimed for sale. [Three KII, observations] Findings for IR2.1.2 and corresponding output(s): IR2.1.2: Improved natural environment for agricultural production Output 2.1.2.1: Natural resources developed or rehabilitated - 255km of new contour bunds were constructed to offer protection for the lowlands (marais) but they were not constructed on sloping fields under cultivation. [Seven FGD, five observations]. The project tracked the quantity of contour bunds created but not the quality such as siting, vegetable cover and maintenance. [Two KII, document review] Figure 8: A contour bund with dried up stalks in Muyinga Province, October 2017 (by B. Baharanyi) Amashiga MTE - FINAL 35 Figure 9: A well-managed community grain store. Muyinga Province, October 2017 (by B. Crenn) - About 50% of the newly constructed contour bunds were planted with cover grasses, many of these dried out, so that currently it is estimated that only about 15% of the new contour bunds are protected. [Seven FGD, three KII, five observations] - There is no evidence of other natural resources developed or rehabilitated beyond the kilometers of contour bunds (2 KII and document review). - Three out of the four agro-forestry nurseries visited did not know how to sell their production (they assumed the project would buy it) and none had a business plan. [Four KII, observations] The project provides them with seed and materials and allocates 25% of their production to be used for project purposes (contour bunds, roadside trees, etc.). Findings for IR2.1.3 and corresponding output: IR2.1.3: HH goat ownership increased Output 2.1.3.1: Community Solidarity Chains (CSC) are established - This activity has not started; the delay is attributed to the August 2015 IEE approval condition Number 2. At the time of its first complete IEE submission in April 2015, CRS requested that the output40 be classified under “Negative Determination with conditions” in IEE. The USAID BEO Environmental Threshold Decision (August 2015) Approval Condition Number 2 directed that CRS take caution when importing non-native animal breeds into Burundi. In July 2016, the Amashiga COP instructed team members that the activity (among others) could not be implemented prior to receiving FFP approval. In an email to USAID/FFP in October 2016 on the various pending IEE approvals, CRS recalled the rationale for importing goats, described CRS experience in this domain and outlined measures in place to manage & monitor the distribution of goats, etc. Finally, CRS requested approval of FFP to proceed with purchase, importation, and distribution of goats from Uganda. In a response, USAID/FFP addressed a separate issue but did not acknowledge the request for approval. Finally, in the August 2017 resubmission, CRS considered still that this was an activity likely to have an environmental impact and again recommended a classification of “Negative Determination with Conditions” for the activity. Approval was obtained in November 2017. Findings for IR2.1.4 and corresponding output(s): IR2.1.4: Pre- and post-harvest losses reduced Output 2.1.4.1: Farmers trained in techniques to improve pre- and post-harvest management practices - As for IR2.1.1 above, CRS included a request to proceed with activities under this output41 so as to train farmers in integrated, natural pest management in its waiver request to USAID/FFP 40Previously worded as follows: Output 2.2.1 Farming HH have increased animal ownership. Activity: Distribution of Goats 41 Previously worded as follows: O 2.1.5: Farming HH and communities have adopted technologies and practices to reduce pre and postharvest losses Amashiga MTE - FINAL 36 pending submission and approval of an Agriculture PERSUAP on the understanding that this output42 (among others) fell under a “Negative Determination with conditions” classification in the IEE and implementation was dependent on an approved Agriculture PERSUAP. - Four out of five community grain stores visited were well managed (as illustrated in Figure 6; well built, well organized, clean, registry card for all members; good scales; reasonably rodent- and bird-proof, rain-proof; charging a 4% stocking fee; locked and with a security guard; drying apron out front; the stores are also monitored by DPAE agents. [Three FGD, five observations] - Little evidence of reduction of pre- and post-harvest losses at the HH level. A few theoretical trainings have taken place with little on-site demonstration and monitoring. [15 FGD, five KII, document review] - Project staff understand that the post-harvest loss reduction component applies primarily to institutional activities e.g., community stores and local food fortification with the WFP. [Two KII] Findings for IR2.1.5 and corresponding output(s): IR2.1.5: SMGs produce & market more certified seed locally Output 2.1.5.1: SMGs established and trained in improved seed production techniques Output 2.1.5.2: Seed fairs organized - As for IR2.1.1 and 2.1.4 above, CRS included a request to proceed with the purchase and supply of improved seed to farmers43 in its waiver request to USAID/FFP pending submission and approval of an Agriculture PERSUAP. This was on the stated understanding that this output (among others) fell under a “Negative Determination with conditions” classification in the IEE and implementation was dependent on an approved Agriculture PERSUAP. - The project has managed improved seed distribution twice since beginning implementation. In March 2016, the project worked in two communes and provided a total of 2.5t of breeder, foundation and certified bio-fortified bean seed to SMGs and ISMs for a total harvest of 2t of foundation, 1.2t of certified and 10.5t of commercial seed. At the same time it distributed about 75,000 cassava cuttings to 51 farmers (enough for 7.5ha); only about 10% survived. [Three FGD] In mid-October 2017, the project distributed16t of certified bean seed to OPAs and LFs, 1.7t of foundation bean seed to SMGs, 0.5t of certified soybean seed to OPAs and LFs and 14kg of foundation soybean seed to SMGs. [Four KII, document review] - There is no evidence of a long-term or strategic plan with all the seed multiplication actors (professional seed multiplication groups, DPAE, ISABU, ONCCS and DPSP). [Three KII staff, document review] - There are three types of seed multiplication actors: i) independent, mature and experienced individuals, with large landholdings and capable of breeder seed multiplication, who also work in groups 42Previously worded as follows: O 2.1.4: Farming HH and communities have adopted improved approaches for gender￾responsive and climate-resilient land and water management systems and O 2.2.3: Farming HH have adopted sustainable forage/fodder and feeding practices for livestock 43 Previously worded as follows: O 2.1.3: Farming Households (HH) have increased access to inputs for crops, especially for women. Figure 10: A handful of poor quality local bean seed in Muramba, Buhinyusa., October 2017 (by B. Crenn) Amashiga MTE - FINAL 37 [Two FGD, two KII] ii) POs and iii) individuals with small land areas. The latter two generally have little experience, are very dependent on the project and only able to produce certified seed from foundation seed. Four of the five Amashiga-created SMGs visited did not know how to sell their production and none had a business plan. [Five FGD, two KII, observations] Those with ONCCs support did not know the cost of seed certification, as it was funded by the project. - There have been seed fairs in February/March and October 2017. The two October seed fairs visited did not occur on time, the quality of the seed was sometimes very poor (mixed varieties and stones) and was rejected by the DPAE. During the fairs, there was no information dissemination on improved cultural techniques to participants. [Two KII, two observations] - The criteria for vulnerability of the seed fairs participants (widow or female- headed HH, orphan/youth-headed HH, HH with elderly or handicapped, landless HH) could be cited by all but, during four KII and two observations, the MTE team noted that some participants did not meet the criteria (strong men and Lead Farmers) and the staff acknowledged not carrying out any verification. [Two KII] Conclusions and Lessons Learned for Sub-Purpose 2.1 While only quantitative data can yield a credible and complete answer, there is little evidence that the project has directly contributed to an increase or diversification of HH food self-supply due to the low quantity and quality of project activities. There has been little attention paid to the quality of implementation and of the results/impacts and there have even been two counter-productive activities and results. First, the project gives rations for contour bund-making when established government practice is for this to be a regular and non-compensated community activity. This demotivated non-participants’ contribution, even for maintenance purposes and created the impression that those contour bunds “belong to the project.” Second, late provision of improved seed and cassava cuttings to LF and POs led to late planting, resulting in poor harvests due to reduced rainfall, shortened growing season and increased pest attacks, scorn from fellow farmers and distrust in the project. Lack of clarity leading to distrust of the project was exemplified where farmers did not always know which individuals would receive seeds or only learned a few days (or one-two weeks) before the distribution/fair). Furthermore, misinformation may have caused several farmers to wait longer in hopes of procuring seeds, leading them to plant seeds with inputs they had to procure themselves later on. Considering this, farmers who received seeds from the project may have had significantly lower net production if they were delayed three to five weeks as was reported. As the agricultural calendar does not wait for projects, it is essential to create and act on alternate plans to reduce negative impacts on participants, such as planning for delays in seed distribution While first trimester planning was completed at the beginning of FY4, no annual or to the end￾of-project plan exists. There are systemic weaknesses in the process of knowledge dissemination and change of attitude and practices. IR2.1.1: Farmers and POs adopt improved agricultural and natural resource management techniques and choices: Most Amashiga activities and concepts follow government and provincial priorities and have been promoted for many years by other projects and the DPAE such that it is difficult to attribute results to the project. Nevertheless, when not in conflict with participants' interest and government practice, Amashiga has given a boost to these improved Amashiga MTE - FINAL 38 practices. For example, there is a general level of knowledge of agricultural improved practices but much less detailed knowledge and only a few specific ones observed in practice. The only clear Amashiga contribution to this sub-objective are the Keyhole Gardens. Quality vegetable seed for kitchen gardens are available in Muyinga town and in some commune markets but the project has not made any effort to make those more widely available at lower costs and has concentrated its efforts on the free distribution of project-obtained vegetable seed to project participants. The lack of fruit tree production by the agro-forestry nurseries does not reinforce HH access to more nutritious food. Although the banana research is progressing very well, the widespread dissemination of the results and the new varieties will occur after the end of Amashiga as banana field trials take time. Thanks to a no-cost extension granted to Bioversity International [December 2017 communication], the first stages of the dissemination will occur during the life of the project and Bioversity International is also in the process of negotiating with donors and the DPAE for province-wide dissemination. IR2.1.2: Improved natural environment for agricultural production Although 255km of contour bunds were created, the quality of implementation (siting and cover) reduces the impact of this activity. By the end of the project, all the 12 selected grain storehouses and 33-50% of the selected roads should be able to be rehabilitated. IR2.1.3: HH goat ownership increased The goat distribution is two years behind schedule and the project will not be able to achieve the targets within the life of the project due to fewer birthing cycles. Based on similar activities in Burundi, with the support of the DPAE, the project can, nonetheless, achieve a threshold of sustainability by project's end if properly planned. IR2.1.4: Pre- and post-harvest losses reduced As currently implemented, the project will have very little impact on this IR by the end. IR2.1.5: SMGs produce & market more certified seed locally There is potential to achieve significant results by project's end but the project staff is not very clear on the process of certified seed multiplication from technical, management and economic standpoints. This is a demanding activity with high costs and needs careful and long-term planning. For practical and sustainable impacts, it is essential to work formally with the local authorities and other key actors from the start, especially if they have field activities and networks of their own, such as the DPAE, the ONCCS and the professional seed multipliers. The timing, roles and responsibilities of all actors need to be clearly defined. Sub-Purpose 2.2 HH have increased income Findings for Sub-Purpose 2.2 Findings for IR2.2.1 and corresponding output(s): IR2.2.1: Farmers’, especially women’s, engagement with markets improved Output 2.2.1.1: Market feeder roads improved or constructed - No market access infrastructure has yet been rehabilitated through FFW/CFW (roads and warehouses) and this delayed is attributed to IEE approval process. CRS considered at the Amashiga MTE - FINAL 39 time of its first complete IEE submission in April 2015 that the output that included the feeder road improvement activity44 had not been sufficiently defined to enable a thorough analysis of potential environmental impact. CRS therefore asked for a classification of “deferral” until such time as the details could be confirmed. The organization committed to submitting a request for modification to the USAID/BEO by 31 March 2016 at the latest. The MTE team did not identify any further correspondence or action on this issue prior to the CRS’s revised IEE submission in August 2017 submission at which time its revised IEE included a detailed Environmental Risk Analysis and Mitigation Measures for the feeder road rehabilitation. The Amashiga team expects the infrastructure rehabilitation activity to start in FY4. [document review] - There is no systematic approach or defined strategy to increase market access other than infrastructure rehabilitation and a few theoretical trainings on marketing. [Two KII, document review] - There is no evidence of specifically enabling women to better access markets, other than increasing the shared decision making. [Five FGD, two KII] Findings for IR2.2.2 and corresponding output(s): IR2.2.2: Farmers have increased, equitable access to financial services Output 2.2.2.1: PSPs trained in establishment, capacity building and functioning of SILCs Output 2.2.2.2: SILC membership expanded - More than 600 SILC groups have been formed; no free kits were provided to enhance the self￾replication process as new groups do not expect the project to provide them with free kits. [Six FGD, two KII, document review] - SILC and Village Savings and Loans Association groups are common throughout the province due to many other projects; Amashiga creates them where there are gaps in geographical coverage and there is still an unmet demand. [Six FGD, three KII staff] - This activity started in late 2016 and less than a dozen groups have had a share-out as a cycle takes nine to twelve months to complete. [Four FGD, two KII, document review] - Members who have had a share-out report being able to invest in goats, cows, roof sheeting and buying/renting farm land. [Two FGD, two KII] - 42 SILC Field Agents (FA) have been trained and, at the time of the MTE, 35 FA were in the process of becoming PSPs, to receive further training, continue forming new groups and support old groups. The project is planning to recruit another 30 FA and RBU2000+'s seven SILC supervisors' contracts, previously due to end in October 2017, have been extended by a year, until the PSPs can take over their responsibilities. [Three KII] - Amashiga is planning to link SILC groups to MFIs once the groups enter their second cycle. The two MFIs visited in Muyinga are aware of SILC groups and are keen to engage with them [Six FGD, four KII]. - The project tracks the number of total members only; it is a common practice for several members of the same HH to belong to the same or different SILC groups or for a HH member to belong to 2-4 groups simultaneously. [Nine FGD] Thus, HH represent approximately two thirds of the total SILC membership, though there is much variability due to economic and 44 The output (2.3.2) was previously worded as follows: “Youth and landless, especially females, have diversified off-farm livelihoods”. Amashiga MTE - FINAL 40 geographical contexts. This issue of HH versus individual membership is similar to that of other SILC groups in the region. - SILC groups have not received other systematic project inputs as described in the project proposal. For example, only members who are also CoProNut or Ab'IR have shared nutrition or gender information. [Six FGD, three KII staff] - Although this is not quantified by the project, various P1, P2 and XP1 volunteers (CoProNut members, Lead Farmers and Ab'IRs) are also members of SILC groups. [Nine FGD] - A few SILC groups also formed POs, so as to manage joint income generation activities. [Six FGD] Findings for IR2.2.3 and corresponding output(s): IR2.2.3: Youth and landless, especially women, have diversified off-farm livelihoods Output 2.2.3.1: Poorest Farm HH, Youth, and landless are involved in FFW Output 2.2.3.2: Youth and landless especially women complete vocational training - See also findings relative to IR2.2.1 regarding the rehabilitation of feeder roads (not yet started), which is planned to be FFW and not FFA. - 3,200 people have been involved in FFA activities adding 255km of contour bunds to protect watersheds. [Two KII] - The timing of the FFA activities sometimes coincided with a period when participants wanted to work in their own fields. [Seven FGD, three KII] - While the criteria for FFA participant selection were well known by all the FFA participants interviewed (see findings for IR2.1.5), participant substitutions have been made [Two FGD] and project staff did not verify that the new FFA participants met the selection criteria.[Two KII staff] The reasons often cited for the substitutions were political manipulation and the fact that many project volunteers were not compensated for their project related activities. For example, one group shared the requirement for FFA participation was to have previous experience with making contour bunds and having goats, neither of which are vulnerability factors. - The youth employment activities have not yet started. [KII staff, document review] CRS considered at the time of its first complete IEE submission in April 2015 that the output45 had not been sufficiently defined (more than six months after the start of the project) to enable a thorough analysis of potential environmental impact. CRS therefore asked for a classification of “deferral” until such time as the details could be confirmed. The organization committed to submitting a request for modification to the USAID/BEO by 31 March 2016 at the latest. In July 2016, the Amashiga COP instructed team members that the activity (among others) could not be implemented prior to receiving FFP approval. In October 2016, CRS provided clarification by email to USAID/FFP that this output consisted mainly of training and would be designed to include content on possible negative impacts to the environment (e.g. related to the use of improved cookstoves). CRS did not, however, formally request an IEE amendment and reclassification of the output, before its August 2017 submission at which time its revised IEE included a detailed explanation of the activities and proposed mitigation actions for the output. 45 Previously worded as follows: “2.3.2: Youth and landless, especially females, have diversified off-farm livelihoods”. Activities: Training of the Youth for brickworks and Training of the Youth for improved cook stoves Amashiga MTE - FINAL 41 Conclusions and Lessons Learned for Sub-Purpose 2.2 Although a change in HH income needs to be quantified through a HH survey, the project has likely not contributed significantly due to a delay in the start of SILC activities and a lack of linkages between SILC groups and other income-generating activities (IGAs). The project has likely had no impact on the diversification of off-farm income and improved market access due to a lack of activities. The overall impact of FFA activities other than giving food to participants is doubtful on several counts: i) participant selection, ii) against a sustainable government policy (although ultimately the government did concede to permit the project to continue with FFA) and iii) quality of implementation and maintenance issues. Though the SILC group activities started late, they are progressing well and are appreciated; however, they could use some improvements to be of maximum utility to their members. In particular, FFA/FFW activities should not interfere with participants' agricultural activities. The hunger season can also coincide with a time of farm work and poorer HH are often also labor poor. P2 Overall Conclusions and Lessons Learned Outside of the SILC component, overall progress on P2 is significantly behind schedule, cannot meet its original targets and will have difficulty making a sustainable impact. There are two external and four internal reasons for this: 1. The 2015 Burundi political crisis which severely limited field access for several months at the beginning of the project; 2. The early 2017 drought which hampered cropping and harvests (Season B 2017). This wiped out project-based agricultural advances and required them to be re-done in Season A 2018, while making it much more difficult to ensure continuous access to adequate nutritious food; 3. The PERSUAP was submitted only in February 2017, resulting in many delayed activities. See Section 5.5 for MTE conclusions on the management of environmental compliance requirements; 4. The lack of integrated partnership with the DPAE, despite reasonable cooperation in the field. Negotiations started in late 2016 and a cooperation agreement was signed in October 2017. and the two most important of all: 5. A lack of leadership and vision to adapt the project concept to the context. For example, while waiting for improved seed, the project could have strongly emphasized demonstration plots on how to produce more with current resources. The project could also better offer farmers a range of options to suit their circumstances, ranging from high yield/high cost and low risk/low cost options. 6. A lack of attention to the quality of implementation and results/impacts, such as failing to verify the eligibility of seed fair and FFA participants, not providing teaching aids for cascade training, providing plant material late or at the wrong time, etc. The project proposal states that P2 activities will reach about 138,000 people (about one quarter of the whole Muyinga Province population). SILC activities alone will reach 30,000 SILC members, about10,000 key agricultural producers (LF, POs, nurseries, SMGs), 6,000 FFA/FFW participants and 8,000 vulnerable people. In accordance with the layered project design, many participants belong to several categories; however, the MEAL system permits a significant amount of double counting. For example, most LF are also in POs, SMG/IMG and SILC and many PO members are in SILC. This counting also does not factor in the level of HH Amashiga MTE - FINAL 42 participation against that of individuals (i.e. mother and father are both members of two different SILC groups, mother is a CoProNut member and father is a LF and also in an PO - thus this one HH can be counted in five different ways and this is not an atypical scenario). Very strong participation from the DPAE AEAs, CoProNut members (P1) and Ab’IRs (P3) in most P2 activities provides the greater part of inter-component integration. There has also been good inclusion of women in P2 activities (a bit weak among Lead Farmers but very strong in SILC groups). The repeated delays in the provision of improved seed has reduced project credibility, and has damaged the reputation of Lead Farmers and the production capacity of POs who plant late with resulting reduced harvests. The project is very similar to several other projects and DPAE activities and offers few innovations or significant benefits other than paying for a few products and services. Given these considerations and broader findings, key concluding points include: 1. Forward planning & sustainability: There are no clear and detailed plans of how to achieve significant and sustainable impact/results by the end of the project. The project does not sufficiently consider the issues of sustainability (economic and organizational) and the potential for self-replication (cascade training and plant material propagation). The project makes many decisions on behalf of participants (agro-forestry nurseries), and gives too much away (vegetable seed). Planning could also be better integrated vertically across the project: while quarterly planning is theoretically done at the Amashiga level, with the full participation of partner staff, in practice, partner staff report a lack of engagement within the process and participation in only monthly and weekly planning demonstrates the lack of vertical integration. Field workers tend to do what their supervisors tell them, without broader understanding of the overall project. Finally, given that the project knew when writing the proposal that it would have to develop an IEE and a pesticide-compliance plan, these should have been organized and completed within three months of the official project start so as not to incur delays and create false expectations. 2. Credibility & “Do No Harm”: Due to the many delays and un-kept promises, the project has lost credibility among participants and stakeholders. The project does not sufficiently take into account the negative impacts and consequences on communities and individuals (“Do No Harm” 46) and accountability toward participants and stakeholders. These issues go beyond P2 and are explored further in Section 5.1 of the General Conclusions. Furthermore, in failing to make good use of local resources (SMGs, DPAE, etc.), the project sometimes competes with them, sometimes goes against local government practices (FFA, free seed), or even harms participants (encouraging farmers to delay planting) in a manner that exacerbates (rather than reduces) existing food security challenges.47 3. Quantity versus quality: There is too much emphasis on the quantity of activities and volume of participants and not enough on the quality of training, implementation and results. SILC and PSP activities have started well but can bring even more benefits to their members by timing the share-outs to coincide with the period for peak cash demand and linking the best performing SILC groups or some of their members to MFIs. With fixed resources and time, 46 Mitoma, Glenn and Kerry Bystrom, “Humanitarianism and Responsibility,” in Journal of Human Rights, 12:1-20, (Taylor & Francis Group, 2013). 47 Anderson, Mary, Do no harm: how aid can support peace--or war, (Lynne Rienner Publishers: Boulder, Colo., 1999). Amashiga MTE - FINAL 43 there is a tradeoff between the quality and quantity of activities. There should be a definition of project quality and an annual review along with the MEAL team to achieve a reasonable balance. The IPTT focuses mainly on quantities. Project activities should be considered from the participants' perspective with a view to sustainability, accountability and avoiding conflicts of interest. 4. KIIs and document reviews indicate the main issues for P2 are the lack of a useful budget/expense monitoring tool and a nearly complete lack of adequate teaching aids. 5. Staff: Staff numbers and capacity seem adequate for the work planned and implemented, though all could benefit from additional training on gender, SBCC, sustainability, business management and specific technical training such as seed multiplication for the agronomists. The gender balance among P2 staff is at most 70:30 male/female. 6. Inter-component integration: There is incomplete inter-component integration to ensure that P2 activities add to P1 impact. For example, agro-forestry nurseries do not produce fruit trees to improve nutrition at HH level, the SILC groups are not mobilized to reinforce P1 messages and P1 volunteers (CoProNut members) are rarely targeted for P2 activities. 7. Adaptation to changing environment: When the external and internal environments change significantly during implementation from the project proposal, it is essential to revise both strategy and logistics, annually and participatorily, and to work backwards from the desired results and outcomes. 8. Indicators: The project aims to have province level-impact but this cannot be measured if the project does not track both the absolute and relative numbers and sizes of project activities and what proportion of the whole population they reach. P2 Recommendations 1. Forward planning and sustainability: - Hold planning meetings with the DPAE, partners and professional seed multipliers as soon as possible to decide on concrete but realistic actions and methods to have a minimum of sustainable and self-propagating results by October 2019; concentrate on what is working well or can make a significant difference (vegetable seed, keyhole gardens, demonstration plots with intercropping, SILC, etc.) and avoid new or marginal activities; - Completely abandon the un-started youth employment activity (2.2.3); - Do not recruit staff or consultants needing a visa as they are getting more difficult to obtain and will add yet more delays; - Employ a small-business development expert to assist all groups with business planning, marketing, networking, financing and economic sustainability; - Reduce giving things away and use smart subsidies and other sustainable methods; - Ensure that all costings are known to all interested parties; - Ensure proper documentation as evidence for all the decisions made; - The banana research should continue and be assisted as much as possible as it is of essential importance to local farmers. No agency is able to do it independently as the DPAE has a moratorium on other food security projects in Muyinga due to the size and scale of the Amashiga project. 2. Credibility & “Do No Harm”: - Analyze and then develop the capacity of local actors to reach project objectives during project life so that they can continue afterwards on their own (SMGs, DPAE, etc.); Amashiga MTE - FINAL 44 - Restore trust by regularly informing participants and stakeholders of the activity timetable with deadlines by which to implement contingency plans so as to avoid any negative effects. 3. Improve MEAL processes so that: - Data collected are more independent from the implementers; - Analyses are interactively shared with the implementers; - There is greater emphasis on quality monitoring; - There is greater emphasis on project added-value; - Data provide both relative and absolute scales. 4. Staff: Add an SBCC officer in P2 so as to improve the quality, sustainability and self￾propagating aspects of changes in knowledge and attitudes. 5. Inter-component integration: - Include the CoProNuts and Ab’IR members in the distribution of plant matter and goats and consider making some of them Model Farmers, if this is practical. - Use the SILC groups to include more Health/Nutrition and Agriculture/NRM activities (as written in the proposal); change group share-out time to September if they prefer; link groups and better prepared individuals to MFIs. - Improve PSP usefulness to SILC groups by tailoring each group's mode of operation to be most useful to its members (share-out time, number of monthly group meetings, access to MFI, PO activities, links to other project activities, give advice by phone, etc.) and increase PSPs' IGA opportunities (vegetable seed selling, marketing and networking support, etc.) 4.3 Purpose 3: Decentralized government structures, civil society and private sector strengthen and implement effective and equitable nutrition strategies Introduction to P3 Activities Purpose 3 focuses on building the capacity of local government, civil society and private sector entities to mitigate risks to food security and nutrition, improving commune level services that contribute to food security and nutrition, and ensuring a local supply of fortified flour to replace USG goods. The three sub-purposes focus on: (i) improving community Disaster Risk Reduction (DRR) capacity related to food security and nutrition; (ii) improving the involvement of representative committees in the quality of public services related to food security and nutrition; and (iii) informing national policy related to food security and nutrition through lessons from Amashiga. The activities that have been planned fall into the following key areas: - Supporting the development of provincial and communal DRR plans; - Supporting the development of Early Warning System (EWS) Committees; - Strengthening the system of commune (and colline) planning and reporting, including public feedback meetings; - Strengthening quality of communal services through social audits using a scorecard methodology; - Facilitating the availability of locally available fortified maize flour. Table 8 below summarizes the various committees and how the project works with them. Amashiga MTE - FINAL 45 Table 8: Committees operating in Amashiga’s area of intervention Name Composition Origin/Mandate Link with Amashiga Commune Community Development Committee – CCDC 16 members: - Seven members elected from the CDC members grouped by Zone (a politico￾administrative layer between colline and commune which has limited functions) - Two members from civil society - One representative from private sector - One commune council member - Two representatives of decentralized services (usually health, education, or agriculture) - One commune Technical Advisor for Development Decentralization law from 2015; five-year term - Amashiga has supported them to develop Annual Investment Plans (PAI) linked to their five-year development plans developed before the project. - Members from the CCDC serve as mentors to one to three CDCs in their commune to ensure a means of support and advice to CDCs after the project exits. Colline Development Committee – CDC Seven members: elected from the different sous-collines in the colline Decentralization law from 2015; five-year term - Amashiga provided training regarding their role, basic planning and reporting functions. They provide monthly reports on colline development activities to the CCDC and the commune office. Commune Early Warning System (EWS) Committee 8-11 members: - Red Cross - Commune TA for Development - Two Data Collectors (paid by Red Cross/WFP) - Decentralized Services at the commune (Agronomist, Social Welfare Officer, Health Promotion Technician) WFP had introduced this system in two communes of the province prior to Amashiga; no specific mandate and not a government￾mandated structure. - Amashiga re-activated two pilot commune EWS committees that had ceased activities for seven months when WFP stopped support in late 2016. - WFP supports Red Cross to hire two data collectors per commune (who sit on the EWS Committee; whether this is official was unclear). - Four monthly analysis meetings have been held, producing and submitting reports to the Red Cross (who submit to the Provincial EWS Committee). Amashiga MTE - FINAL 46 Name Composition Origin/Mandate Link with Amashiga Provincial DRR48 Platform Approximately 16 members: - Representative from each ministry - One security/police - Governor or his/her advisor - Red Cross - Other NGOs Inter-ministerial Decree on DRR which then created an Inter-Ministerial Committee and a National Strategy (2011-2015) - The project has supported the DRR Platform to develop the 2016 and 2018 Contingency Plans. Commune DRR Platform Not yet in place. National Strategy does not specify the structure at the sub￾provincial level but speaks of the function/role at the commune and the colline levels. The project intends to put these in place replicating the structure of the provincial committees. According to the 2012-2015 strategy, there are five pilot committees in place. The project has not undertaken assessments to understand how many exist currently or if there are any lessons learned from those pilot experiences. 48 Project staff requested the use of “DRR Platform” as the acronym for Platforme de la Prevention des Risques et de Gestion des Catastrophes and this would be the standard English-language expression. It is notable that the original French describes prevention of risk and management of disasters. Although the subsector is relatively new in Burundi and still quite limited in its capacity and its interventions (by admission of the 2012-2015 strategy document of the GoB), the inclusion of “management” (implying response) is notable. Amashiga MTE - FINAL 47 Sub-Purpose 3.1: All communal community development councils (CCDC) and Platforms are prepared to mitigate risks to food security and nutrition Findings for Sub-Purpose 3.1 Findings for IR3.1.1 and corresponding output: IR3.1.1: Increased capacity of CCDCs and/or Platforms to develop & execute gender-responsive, conflict-sensitive action plans to mitigate natural and manmade shocks Output 3.1.1.1: CCDC/Platform members build their capacity to act to mitigate shocks with gender-responsiveness and conflict sensitivity - Amashiga supported the Provincial DRR Platform to develop a 2016 Contingency Plan and held a workshop in late September 2017 to develop the 2018 Contingency Plan (expected to be validated/finalized in October 2017 after the MTE field work). They did not prepare a Plan for 2017 due to CRS understanding of IEE requirements in the summer of 2016 when the Contingency Plan for 2017 would have been developed. - At the time of its IEE submission in April 2015, CRS considered that this output49 had not been sufficiently defined to enable a thorough analysis of potential environmental impact. CRS therefore asked for a classification of “deferral” until such time as the details could be confirmed. In July 2016, the Amashiga COP instructed team members that it (among others) could not be implemented prior to receiving FFP approval. In its October 2016 email, CRS reminded USAID/FFP that environmental issues have already been taken into consideration in Government of Burundi laws and implementation guides for DRR and requested approval from FFP to proceed with DRR activities as per the project proposal. The MTE team did not identify any further correspondence or action on this issue prior to the CRS’s revised IEE submission in August 2017 submission at which time its revised IEE included a proposed reclassification of the output under “Negative Determination with Conditions” and included a detailed Environmental Risk Analysis and Mitigation Measures for the feeder road rehabilitation. This IEE amendment was approved in November 2017. - The Provincial EWS Committee meets monthly and, with WFP support, analyzes the reports and data submitted to the province from the commune-level EWS committees. - The project supported seven communal Early Warning System committees put in place in April 2017; they have produced three-four monthly reports. - EWS Committees were unaware of the eventual plans to introduce Communal DRR Platforms. [Three of four EWS Committees; one participant of the one Committee had some awareness when prompted] - None of the four EWS Committees cited examples of interactions with the commune (other than the presence of the Commune Technician for Development as a Committee Member). - The representatives on EWS Committees from the decentralized services (e.g. agriculture, health, water, etc.) could not give examples of information gathered for the EWS that was not part of the data or information that their ministry already collected or that they included in their own reports to their provincial ministry. [Three of four FGDs with EWS committees where representatives of decentralized services were present] 49 The output (3.1.1) was previously worded as follows: “CCDCs have increased capacity to design and execute gender￾responsive, conflict-sensitive action plans to mitigate natural and manmade shocks”. Amashiga MTE - FINAL 48 - EWS Committees who were asked the question as to whether or not they had occasions to discuss their data or problems in the commune with other services confirmed that they did not. [Two of two FGDs with EWS committees who were asked the question] - At least three KIIs spoke of plans to introduce DRR Committees and EWS Committees at the colline level but were unable to confirm specifics of the structure (e.g., number of persons, which positions/roles, etc.). Findings for IR3.1.2: IR3.1.2: CCDC and/or Platforms implements risk reduction measures (no outputs) - Two accounts of an organized response to the poor bean harvest in early 2017 (weak production in Season A 2017 planted in October/November 2016) were shared with the MTE team [Four KIIs with project staff and Provincial stakeholders]:  It was reported that the Provincial DRR Platform took action; [One KII]  The governor called a meeting to discuss the poor harvest with two to three departments (rather than the full Platform) and they mobilized food distribution and bean seeds distribution to ensure a better harvest for Season B, with a major contribution of bean seeds coming from Amashiga (see Purpose 2 findings Section 4.2 for more detail on this distribution). Aside from the preparation of the contingency plan which was supported by Amashiga, Platform meetings rarely, if ever, meet formally or include even a simple majority of members. [Three KIIs; one KII implied that the full meetings happen but did not provide convincing levels of detail on timing and topics] - The commune-level EWS Committees are noted in the indicator for implementing this IR. The commune-level EWS Committees have reported findings and data to the Provincial EWS Committee (via the Burundi Red Cross, which chairs the Provincial and Communal EWS Committees and the WFP). - No other concrete measures were implemented. Conclusions and Lessons Learned for Sub-Purpose 3.1 This Sub-Purpose is still at an early stage of implementation and does not have a documented strategy through the end of the project (or it was not shared with the MTE team). CCDCs do not yet demonstrate knowledge of DRR and platforms do not exist at the commune level; the Provincial Platform does not meet regularly or as a full platform at the provincial level. This is particularly problematic given the delays, attributed to environmental compliance requirements, which led to the formation and support of Communal EWS Committees (part of a disaster risk mitigation approach) prior to the formation of their parent committees, the DRR Platform. Moreover, there were some, albeit not documented, plans to roll these structures out at the colline level without any discussion as to the required capacities and what functions they would ultimately fill. Considering only 18-20 months of implementation time remain for the project, the MTE team considers that the replication of the Provincial DRR Committees (16-members) at the commune level alongside a Communal EWS Committee will lead to confusion of roles and could run counter to the original intent of this sub-purpose, i.e., commune-level capacity to identify, monitor, and mitigate risks to food security and nutrition. There are also several questions about the sustainability and appropriateness of the approach. The current capacity of the CCDC and other commune structures is still relatively basic and there is no clear linkage (documented or verbal) between these new committees and the resource-allocation Amashiga MTE - FINAL 49 entities of the commune (i.e. the CCDC and the Communal Council). Particularly problematic is the fact that the project pays two data collectors to collect data that the decentralized services already have and should be analyzing. This approach offers no path towards sustainability; indeed, it potentially reduces the sense of responsibility for the existing ministries to share this data with the communes. Moreover, the Provincial Platform’s lack of activities (outside of the contingency plan activity supported by Amashiga in 2017 and in 2015), combined with the lack of research or assessments into the experience of the pilot Commune Platforms, and the lack of an overarching strategy towards Commune Platform’s membership, suggests there is no clear vision as to how these platforms could interact with one another and pre-existing commune structures to implement reduction or mitigation measures in a concerted, sustainable manner. In project reporting and discussions with stakeholders, there is confusion and a lack of clarity around which structures will exist at each level, with some stakeholders noting that that DRR Platforms and EWS Committees would be introduced at the colline level, as well. More importantly, there is no articulated vision (documented or verbal) as to how the Communal Platform would interact with the existing structures to implement mitigation or reduction measures. Disaster responses would require intervention/support from the Provincial Platform, the Governor, and/or international organizations, and how the EWS Committee would relate to these structures has yet not been documented, much less articulated, and there is only about 18 months of implementation to achieve such capacity developments. Moreover, in feedback to this report, project staff attributed the lack of knowledge of CCDCs on DRR to the fact that this sits within the mandate of the DRR Platform, which do not yet exist. Unfortunately, it is unclear how commune DRR Platforms will take such future actions when they have no clear vision, resources, nor access to a budget. Ultimately, it seems likely that they will have to go back to the CCDC and CC to integrate reduction measures into PAIs or to go to the commune administrator or commune staff, who have very limited financial resources not already allocated in the PAI (according to interviews with commune administrators and advisers). Overall, this sub-purpose lacks a clear strategy document for various structures across multi-year planning, weakening the ability to adapt to changes and clearly articulate to project stakeholders (and even to project staff) the evolution of activities. The lack of outputs contributing to intermediate results exacerbates this problem. The inclusion of jargon in proposals (e.g., disaster risk reduction, contingency plans, early warning systems) may appeal to reviewers but is often not widely understood by project staff members let alone participants and can lead to a focus on less important aspects like forming committees with these names to reflect this jargon rather than ensuring the function is understood by local actors and is fulfilled. This distracts from the inclusion of concrete activities required to strengthen the capacity of existing structures (e.g. supporting commune staff, CCDCs, and Commune Councils to carry out focused data collection, put in place area-specific monitoring systems, etc.). Amashiga MTE - FINAL 50 Sub-Purpose 3.2: CCDCs ensure gender-responsive, equitable delivery of services to support HH food security and optimal nutrition practices Findings for Sub-Purpose 3.2 Findings for IR3.2.1 and corresponding output: IR3.2.1: CCDCs and civil society actors have a shared understanding of prioritized list of key determinants of chronic malnutrition including gender barriers Output 3.2.1.2: Best practices and knowledge gained through Amashiga identified and shared - In April-May 2017, Amashiga conducted seven communal workshops (one for each commune of Muyinga Province) on the local determinants of malnutrition and produced workshop reports. [Document review and five of five KIIs with staff members] - The seven reports show little variation in the determinants included. They do not discuss specific gender barriers other than references to joint decision-making at the household level, which is part of the initial Amashiga project design and was part of programming activities and messaging prior to the analysis workshops. [Document review of completed reports] - Participants at the local workshops were primarily commune staff members and did not contain any significant numbers (e.g. more than two) of elected CCDC members or civil society members [based on three of four staff interviews and workshop reports containing participant lists; no CCDC FGDs revealed specifics of the workshops] - CCDCs did not cite the workshop on key determinants when asked about their understanding of the causes of malnutrition and examples given of key determinants were general examples at the household level (similar to project sensitizations messages). [Four FGDs with CCDCs] Findings for IR3.2.2: IR3.2.2: CCDC regularly reports publicly on the performance of programs and the current state of key determinants (no outputs) - Accountability meetings were held for all communes in June 2017 and were planned to happen every six months. These were run by the commune administrator and other Commune Council members and not the CCDC as stated in the IR. [six of eight KIIs and FGDs (the question was not directly asked in two of the eight meetings) with CCDCs and commune personnel in four visited communes] - Public accountability meetings were held at the colline level as well. At least four CDC FGD members mentioned the attendance of CCDC members. [All seven FGDs with CDCs where the question was asked; the question was not asked in two CDC FGDs] - The project provides “transport fees” for some participants for the commune meetings but not for colline meetings. - The accountability meetings do not cover directly the state of key determinants. Instead they discuss the commune PAI or commune-level achievements more generally (or the previous six months of colline monthly plans in the case of the colline meetings). [12 KIIs with CDCs, colline chiefs, and CCDCs] Amashiga MTE - FINAL 51 Findings for IR3.2.3: IR3.2.3: Progress on communal government action plans to address key determinants is monitored by civil society organizations (social audits) (no outputs) - Scorecards were done for the commune Ministry of Agriculture services in all four communes visited (and reported for all seven communes). [KIIs with project staff and project records] - The project trained local facilitators who are commune staff members who then guided the local civil society representatives to gather the information in the collines. [KIIs with project staff] - The scorecard workshops were done as “training workshops” whereby the training and the activity were done in the same four-day period. [KIIs with project staff] Conclusions and Lessons Learned for Sub-Purpose 3.2 IR3.2.1: Amashiga seems to feel that the one-off activity is adequate and yet, the MTE team does not believe that there is a shared understanding of truly local determinants or even an understanding of the specific and unique aspects of their community that contribute to food insecurity. The reports do contain determinants of malnutrition in a general sense, which largely correspond to the elements from the project proposal (e.g. low agriculture production, low water availability, etc.), as opposed to very specific analysis based on the area and the people and the structures (e.g. these collines in this commune have less water and higher diarrhea incidence than these collines in the Commune, or compared to neighboring communes). IR3.2.2: Amashiga has supported the communes initiating commune accountability meetings relative to the PAI; however, this is not directly related to the local determinants but rather the PAI. The extent to which PAIs include local determinants varies but is greater for the 2018 PAIs developed in 2017, after the project’s production of reports on local determinants. There appears to be an appetite to continue this activity after just one iteration in the summer of 2017, although the project should monitor these activities, as it may be used for political ends rather than true accountability (depending upon how much discussion and debate or questions is genuinely permitted during these sessions; the activity is notably led but the commune administrator and the Communal Council, which is a slight difference from the wording of the IR). Nonetheless, it is a good step towards an accountability mechanism, provided that it is monitored so that it is not coopted for political ends. This I.R. is poorly worded relative to what the project is currently working toward (or else the project activities are not aligned correctly). The PAI is not a plan whose purpose is specifically to address key determinants of malnutrition but rather to achieve development objectives of the commune, which should include addressing local determinants of malnutrition. IR3.2.3: Amashiga started to strengthen the review of services supporting food security with the Social Audits activity using a scorecard methodology in August-September 2017. The first scorecard for agriculture services is the sole tool the project has selected towards completing the social audit. Strong understanding of roles and responsibilities of elected, appointed, and civil service actors, along with strong development planning and reporting processes, are important pre-cursors to strengthening decentralized services. Much of this sub-purpose (FY17)’s first year of implementation was devoted to strengthening the understanding of roles and responsibilities and the planning processes at the commune and colline levels, an important contribution to the sub- Amashiga MTE - FINAL 52 purpose given that the communal PAIs are the mechanism by which resources are allocated from the commune budget to development activities in the commune. However, the activity does not appear in any of the IRs or Outputs, although the PAIs are referred to in Indicator 3.2.1.1. There have been some strong initial activities during the last four months (June-September 2017) of Year 3 although there is no clear vision for how they will come together during the remaining 18-20 months of implementation. Moreover, the links between annual plans, service quality and civil society monitoring appear disjointed, lacking clear linkages to the work done on the PAI planning process and between service quality and the new Commune Community Developments which expire in 2017. In addition, the process is delayed while the Province seeks funding to conduct the activity. Only one scorecard (for agriculture services) has been implemented in each of the seven communes. To ensure meaningful service accountability, at least two-three scorecards per service should be implemented to enable follow-up, learning, and adaption for the subsequent period., However, there is simply not enough time in the calendar to implement first and second scorecards for all the remaining services (or even those for health and water services, the most closely related to food security and nutrition after agriculture) in the remaining project timeline. The scorecard develops actions to be taken but there is no clear plan for how its use will evolve. It is feasible for the project to complete one more iteration during the summer of 2018, which would reflect the first progress report against an action plan This suggests that there will be only a partial achievement of this sub-purpose. Many activities with the communes are supported with “transport fees” which raise significant questions regarding the sustainability of these activities after the end of the project, particularly with regard to the social audits and the commune-level accountability meetings (e.g., “reunions de rendre-compte”). The project efforts to strengthen colline-level planning processes and public accountability meetings, as well as the mentoring system from CCDC members show stronger promise of continuing after the project’s end. Nonetheless, to achieve the sub-purpose sustainably by project end implies a need for local facilitators to conduct the scorecards with all communal services (at least water, agriculture, and health) at least two to three times, and at least one of those times without project resources. The sub-purpose is a worthwhile aspiration for the context and the overall project goal but the wording does not reflect adequately the logic of what the project has been seeking to achieve. It does not capture the fact that the activities are seeking to strengthen the capacity to analyze local determinants of malnutrition and food insecurity, to strengthen commune and colline planning and reporting processes, and to increase accountability of commune services to communities. As written, several key achievements to date are not clearly attributed as contributing to the project design, and the current sub-purpose is unlikely to be fully achieved, and certainly not in a sustainable manner, by the project end. In particular, the reference to “local determinants of malnutrition” is used liberally and loosely, and this activity does appear to have been more of a one-off formality, rather than a key building block for sustainability of local attention and action to reduce chronic malnutrition and food security, rather than just agriculture production or acute malnutrition which is often the focus of communities which are chronically poor with developing services. Amashiga MTE - FINAL 53 Sub-Purpose 3.3: Practices generated by Amashiga contribute to effective national policy implementation and increased gender-responsiveness of policies (related to reducing chronic malnutrition in under 5 children) Findings for Sub-Purpose 3.3 Findings for IR3.3.1 and corresponding output: IR3.3.1: Policy Agenda approved by FFP Output 3.3.1.1: Policy situation assessment carried out, validated, and disseminated - At the time of its IEE submission in April 2015, CRS also considered that this output50 had not been sufficiently defined to enable a thorough analysis of potential environmental impact. CRS therefore asked for a classification of “deferral” until such time as the details could be confirmed. In July 2016, the Amashiga COP instructed team members that it (among others) could not be implemented prior to receiving FFP approval. In its October 2016 email to USAID/FFP, CRS stated that this output would consist in the collection of best practices and lessons learned during the life of the project and so would have no unique environmental implications. It requested a “categorical exclusion” for this activity. The MTE team did not identify any further correspondence or action on this issue prior to the CRS’s revised IEE submission in August 2017 submission at which time its revised IEE included a proposed reclassification of the output under “Negative Determination without Conditions”. 51 This IEE amendment was approved in November 2017. - The MTE team never received nor saw a Policy Agenda prepared or submitted by the project. Activities at the sub-national level completed prior to the end of Year 3 included some public advertisements promoting the new law on fortified flour and the identification of 15 Farmer Producer Groups (FPG) from among those identified under purpose in August/September 2017. These FPG are expected to produce quality maize that could then be milled and ultimately fortified. - The project-provided list of 15 selected FPGs lists a production capacity per season of 550 MT of maize yet participants themselves in an FGD reported a production capacity of 100- 200 MT of maize. This was verified also by the MTE team against their land size during the FGD. This suggests that they only produce in the valley areas (only Season C, or the off￾season). They also had no knowledge of what Amashiga planned to do with them or what the project would commit or what would be required of them. [One FGD] - A one-day workshop with approximately 40 participants from government, Amashiga staff members, and representatives from the 15 FPGs, gave attendees a broad orientation to the law on fortified flour and the requirements of the staple food inputs (i.e. unprocessed maize) to meet regulations to be fortified. [One FGD with FPG participants at the workshop] - No millers are producing fortified flour in Muyinga Province. [KIIs with project staff] - Sale of commercially produced (unfortified) flour is limited to markets in Muyinga city and is not consumed often in the communes due to cost and/or availability. [At least three KIIs with 50 The output (3.3.3) was previously worded as follows: “Implementation of food policies focused on food access, prices, and fortification is strengthened by lessons learned from Amashiga”. 51 Page iii, CRS Burundi, Amashiga Title II Development Aid Program Amended IEE, Submitted August 29, 2017; the Threshold Determination on P22 of the same document recommends the classification of “categorical exclusion”. Amashiga MTE - FINAL 54 project staff or commune staff where this topic was discussed, confirmed with additional informal interviews, although not based on any market study] - There appear to be 0-4 small mill-owners in the three of the four visited communes. [Four of five KIIs or FGDs at the commune level asked about the presence of associations or individuals who can mill flour] Conclusions and Lessons Learned for Sub-Purpose 3.3 The only actions completed at national level to date have been the support for the development of a national strategy for the promotion of fortified flour. Activities at the commune level only began in October 2017 at the beginning of the Year 4. The remaining activities related to the sub-purpose as shown in the DIP and ARRs support the implementation of the national law on the fortification of flour. Otherwise, there is no project documentation or workplan that traces the activities or expected results of this sub-purpose until the project end. The MTE Team is skeptical that any activities of this sub-purpose will contribute in a sustainable way to the other project components. Finally, while the MTE team agrees that a supply of fortified flour in the province was a logical inclusion in the project’s theory of change in terms of increasing locally available alternatives to CSB, its impact would have been minimal given the local production issues identified in the findings. Given the nascent stage of project activities and design weaknesses, and the limited implementation time period remaining,, the MTE team does not believe that this sub-purpose is worth pursuing further, unless there can be guarantees of tangible results in terms of flour availability before the end of project activities. P3 Overall Conclusions and Lessons Learned Purpose 3, as with all other purposes, has experienced significant delays; commune-level staff were finally hired in October 2016. With the exception of a few activities at the provincial level (e.g. the Provincial DRR Platform) in Year 1 and national-level activities to support the development of a national strategy for the promulgation of the law on fortified flour in Year 2, the majority of activities under the purpose only began in Year 3. The delayed start is connected to insecurity in FY15 with subsequent delays attributed by Amashiga to the IEE approvals. Nevertheless, the MTE team fails to understand the logic of the timing of the suspension of activities and why the project would start with putting in place EWS committees prior to establishing and supporting the parent committees (the Commune DRR Platforms). There are zero outputs or only output for each of the IRs making it difficult to clearly know the originally intended logic of the project design in a concrete way although the indicators and ToC narrative give some clues. There are also differences in the wording of sub-purposes and what is being achieved as discussed above and in Table 9 below. Table 9: P3 Summary Conclusions by Sub-Purpose Sub-Purpose Summary Conclusion SP3.1: All communal community development councils (CCDC) and Platforms are prepared to mitigate risks to food These activities have only recently started and only with EWS committees so it is not entirely surprising to see that the CCDC has not really changed in its capacity to mitigate risks (other than it has stronger planning and reporting processes as part of SP3.2) and that the EWS Committees have some commune staff who are able to Amashiga MTE - FINAL 55 Sub-Purpose Summary Conclusion security and nutrition analyze some data (available to those services prior to the committees starting). SP3.2: CCDCs ensure gender-responsive, equitable delivery of services to support HH food security and optimal nutrition practices The use of the scorecard (the intervention and IR most closely related to this sub-purpose) is a good idea for introducing accountability measures into service provision. However, this is only beginning near the end of the project and they are known to require several iterations to be sustainable. Moreover, the project targeted only one service provider (agriculture) of the three services that are relevant to the project outcomes (health, agriculture, water). The work on supporting the communal planning processes (the PAI) and introduction of reporting and feedback mechanisms between the collines and the commune was one of the strongest aspects of this component but there appear to be few plans to consolidate this foundation to ensure that it pays particular attention to food security among the many competing issues that these committees must handle. There is limited capacity still among commune actors to analyze actively the current state of food security and malnutrition and to take actions accordingly. SP3.3: Practices generated by Amashiga contribute to effective national policy implementation and increased gender￾responsiveness of policies (related to reducing chronic malnutrition in under 5 children) The formulation of the sub-purpose is removed from actual implementation and is significantly behind schedule. If Amashiga continues with this purpose, it is unlikely to produce more than trace amounts of fortified flour, which will have no significant impact during the project and unlikely during the three to five years thereafter without significant additional support. The project is just starting efforts with large suppliers who take time to change and with small associations who are too small in number to have any impact; moreover, few of the latter, if any, have ever engaged in transformation. Amashiga supported the roll-out of a national strategy on food fortification but there is no connection to project interventions in Muyinga province. Overall, most of the elements of P3 are founded on sound conceptual logic. Further, the project undertook some important steps – not reflected in the higher levels of the results framework – that were timely, appropriate and an important foundation to the ultimate logic of the component. These include, most notably, the support to the CCDC (and indirectly to the Communal Councils) to strengthen their planning and accountability processes (how commune funds are allocated and budgeted). Nevertheless, there are a number of modifications that should have been considered in light of the lengthy delays to implementation. The lack of modifications means that various aspects of the current design are unlikely to be achieved within the remaining project period. The project, therefore, seems to be far from achieving the initial logic of the Theory of Change Purpose i.e., that commune government and non-government actors are capable (and engage in) actively analyzing food security and nutrition on an ongoing basis so as to take actions to improve the Amashiga MTE - FINAL 56 situation and to mitigate risks, so as to continue the role that Amashiga was expected to fill in the short term. P3 Recommendations 1. Document the governance strategy. Create at least a brief (four-six pages) strategy document which explains the evolution of the purpose and its sub-purposes and IRs through the end of the project with clear expectations for the expected results for each quarter as well as the budget. It would be linked to the initial theory of change but would be tied directly to planning documents and updated regularly to adjust for changes in context or planning. 2. DRR/EWS and sustainability: Focus on strengthening existing structures; reduce targets to create new ones accordingly - Concentrate efforts on introducing basic understanding of the value of identifying and mitigating food security and nutrition risks of disasters at the commune and colline levels to existing structures i.e., the CCDC, the Commune Council and other key commune staff. Formal members of the EWS Committee could be integrated as appropriate but not the collectors. This would be particularly timely if done prior to the development of the five-year Commune Community Development that are currently behind schedule (pending the Ministry of Local Government’s ability to raise the necessary funds to fund these planning processes). - In parallel, given the lack of national experience and project assessment of learning from the ten pilots, Amashiga should drop the activity to support the formation of Communal DRR Platforms as formal separate entities. The MTE team is conscious that the project team does not agree with this recommendation but reiterates that it is an unrealistic ambition to create and train new committees at this late stage in implementation particularly given the extent of other challenges faced by the project. Finally, this work would inevitably detract from efforts and potential impact to be made from working with the CCDC and the commune to ensure greater capacity to analyze and understand local determinants of malnutrition on an ongoing basis, to follow-up on social audits, and ensure the structures who allocate and manage resources understand the importance of disaster risk reduction as part of development planning. - The project should discontinue the use of data collectors who are receiving part-time monthly stipends to collect data that the decentralized services already have and should be analyzing themselves, instead focusing on bringing together difference decentralized services together with commune staff (and possibly some elected representatives from the CCDC or the CDC) to analyze their own data. - Activities between now and the end of the project, particularly those which the project hopes will continue, should phase out quickly the use of “transport fees” that are not already covered under commune policy (nearly all the current transport fees provided by the project for meetings would not be covered by the commune budget). 3. Social Audits and sustainability: Build local capacity to carry out social audits Focus on completing at least one more social audit of the agriculture services in each commune as the follow-on to the action plan developed during the initial social audit completed in August/September 2017. The project staff should hold refresher training and a planning session with the facilitators at last one month prior to the audit, act only as observers on site, and perhaps hold a one-day lessons-learned workshop for the facilitators and commune staff (and possibly also the CCDC members) to identify next steps and an action plan for the subsequent year. If it is possible to support the scorecard for one other service, it Amashiga MTE - FINAL 57 should be planned carefully not to conflict with project activities of other purposes, particularly as heath and water would be the next likely services, and there are already major activities planned under P1. Moreover, the subsequent recommendation suggests collaborating with P3 to do an abridged version of the scorecard for health services, linked to the indicators chosen for the Giant Community Scoreboards rather than a separate social audit for health. 4. Integrate activities across purposes via the Social Audit approach - Purpose 1 has already planned to introduce colline-level Giant Community Scoreboards which would have a public health indicator to be tracked to encourage collaboration between the community and health facilities on health-seeking behaviors. P1 and P3 teams could collaborate on this undertaking by combining this activity with a partial scorecard of services52 related to the chosen Giant Community Scoreboard indicators with mini-action plans for the service providers at the health facility, grouping a few collines together. At the end of each month/quarter, when the collines update their scoreboard, they could bring together representatives of the collines at the health center to review the indicators and also review the progress of the service providers on their action plan for that aspect of services. - An alternative, or additional, opportunity would be to introduce a social audit/scorecard for commune water services, to complement the intended P1 activity to reactivate the water committees. There are several of these per sous-colline and colline and they should have a link to the colline authorities, i.e. the CDC and Colline Council/Chief. As part of re-training activities in the roles and responsibilities of these water communities, Amashiga could introduce the performance indicator, scorecard, and follow-up plan for the commune level services to these water points (and the committees). There would conceivably be time to pilot some of the scorecards after one year of work. 5. Remove or revise Sub-Purpose 3.3. This Sub-Purpose should be removed, or it should be revised significantly to clearly demonstrate what concrete changes could be realized before the project end and how it can contribute to the project goal. The MTE Team recommends removing this sub-purpose to concentrate management and financial resources on the other components, unless the project partners can propose a strong alternative which focuses on supporting commune-level associations to increase milling capacity so that flour can be produced in greater quantities (even if it is not fortified). 4.4 Crosscutting Sub-Purpose (XSP) HHs and communities adopt gender-equitable decision making practices Introduction to XSP Activities This sub-purpose consists primarily of the identification and training of 15 aberemesha kiyago ibeterembere lidakumira (Ab’IR) couples per colline. They are trained in the Faithful House (FH) model developed and implemented by CRS in other countries. The couples then conduct house visits or provide advice couple-to-couple as well as carrying out awareness-raising on household decision-making at community meetings or events. During the second half of 2017, 52 It would be complicated to conduct a standalone social audit for health services as the health services do not align neatly with the commune. It would therefore take many years to strengthen lines of accountability between the service provider and the decentralized government entities. Amashiga MTE - FINAL 58 they also conducted some skits or discussion groups. The planned expansion to train Muslim couples to be Ab’IR was delayed for nearly a year pending the approval of the Islamic Community; this was finally obtained during the MTE field work in October 2017. XSP Sub-Purpose: HHs and communities adopt gender-equitable decision-making practices Findings for XSP Findings for IR X.1 and corresponding output: IRX.1 HHs with behaviors that threaten food security and nutrition are identified and supported to change Output X.1.1: Ab'IRs recruited, trained and supported - 15 Ab’IR couples per commune have been identified and trained. While it does not appear that all collines did the retraining in exactly the same way, it appears that the Ab’IR couples completed the training and the FGDs revealed no obvious differences in knowledge levels or engagement. [13 FGDs with Ab’IR couples and positive deviant couples and four KIIs with staff]  The primary training method implemented in the communes visited (and project-wide according to staff) was a cascade training whereby the 15 selected Ab’IR couples per colline selected five couples from among them to attend the initial FH training (three days); these five then trained the other ten couples in weekly sessions over the next month.  A second method of the cascade training was conducted whereby five Ab’IR couples were selected and trained and then a further ten couples were selected and trained by the initial five Ab’IR couples.[confirmed during the FGDs/KIIs] - There was greater appreciation of the work of Ab’IR couples’ work in the three of four communes visited where the first training method was employed. Fewer Ab’IR could cite details of the training approach in the commune where Ab’IR couples followed the second training model and local leaders and staff showed less enthusiasm and/or provided less detailed responses when explaining the second method. [Four FGDs/KIIs with Ab’IRs, confirmed by staff KIIs] - The second training (five days) was organized to cover peaceful resolution of conflict and GBV and involved sending three individuals from among the 15 Ab’IR couples (i.e. three of 30 individuals). Few participants in FGDs could give details on the re-training when asked when it happened or how it happened. In three of the four FGDs where the specific question was asked, couples noted that the FH training was much more valuable and could not cite “new things” that they had learned. The MTE team did not initially understand the difference between the training approaches (e.g. the difference in the way of selecting the five couples to follow the direct training) as it was not specified in documentation or interviews priorto the field work. - Ab’IR couples conduct house visits and sensitization sessions (e.g., communicating messages at community meetings called for other purposes. [14 FGDs with Ab’IR couples and positive deviant couples and four KIIs with staff] - Ab’IR couples report approaching couples “in difficulty” or couples “with disputes” but do not generally identify households based on behaviors directly related to food security and nutrition. Several respondents described, however, how the household dynamics of couples with disputes had negative consequences on household food security and nutrition (e.g., poor use of household resources, lack of discussion and communication or agreement about child care priorities, sharing of household responsibilities, etc.). [13 FGDs with Ab’IR couples and positive deviant couples] Amashiga MTE - FINAL 59 - Some Ab’IR in some collines developed and presented skits at nutrition fairs to communicate messages on household decision-making and appropriate care for children. [Four FGDs and KIIs with Ab’IRs and project staff, confirmed also by KIIs/FGDs for P1 field work.] - Discussion groups among Ab’IRs have started in some communes. [Three KIIs with project staff] - Some Ab’IR have begun forming associations to encourage solidarity among Ab’IR and encouraging them to continue. [Three FGDs/KIIs with Ab’IRs and/or project staff] - Nearly all Ab’IR couples are Catholic, although they provide home visits/advice to non-Catholics as well. [> 17 FGDs/KIIs with Ab’IR, positive deviants, local leaders, and project staff] - The Islamic House guide is ready for use to train Muslim Ab’IR couples (who have not yet been identified) and the approval letter from the national Islamic community in Bujumbura was received in October 2017 at the time of the MTE. [staff KIIs] Findings for IRX.2 and corresponding output: IRX.2 Couples receive training and support to make HH decisions together Output X.2.1: Couple groups receive FH trainings from Ab'IRs - It is difficult to distinguish IRX.2 from IRX.1: both focus on behavior change at the HH level. Couples receive house visits and advice based on FH themes rather than formal FH training. See IRX.1 for findings related to FH training and couple groups. - There are also overlaps between the outputs and output indicators for IRX.1 and IRX.2 as can be seen below in Table 10. Table 10: Comparison of X.1 and X.2 IR and Output Language X.1 X.2 IR HHs with behaviors that threaten food security and nutrition are identified and supported to change Couples receive training and support to make HH decisions together IR Indicator Average score of the perception of target population on Gender-Based Violence Number of households visited by Ab'IR couple per month. Output Ab'IR couples recruited, trained, and supported Couple groups receive FH training from Ab'IRs Output Indicator # Ab'IR couple trained and supported by Amashiga # couples who completed Faithful House training Findings for IRX.3 and corresponding output: IRX.3 Community members' standards for gender equity in decision making in homes and communities increased Output X.3.1: Leaders and community members trained or sensitized on the importance and benefits of gender-equity in making decisions for the community - Leaders at the colline level confirm that there have been reductions in the numbers of household disputes (“litiges”) that are brought to them for mediation. [Five of eight FGDs/KIIs with colline chiefs] Amashiga MTE - FINAL 60 - They spontaneously cited the advantages of improved decision-making and improved household dynamics more generally at the household level. [Ten of 15 FGDs/KIIs with colline chiefs or CDCs] - Some colline chiefs, possibly where land conflicts are infrequent, charge couples for mediation or levy fines to resolve domestic disputes, and therefore, saw the Ab’IR as a potential threat to their income. The relatively narrow focus of the project on husband-wife disputes and the overall benefits of fewer disputes between couples taking place appear to have overcome this barrier. - Some colline chiefs refer couples with disputes to the Ab’IR directly. Conclusions and Lessons Learned for XSP This sub-purpose was one of the highlights of the MTE: the project has made meaningful and significant progress in less than two years of implementation. Households have already begun to employ gender-equitable decision-making practices. This can likely be attributed to having a simple and well-executed model. Given the appreciation for the activity, as demonstrated for example by the spontaneous referrals by the colline chiefs of couples with disputes to the Ab’IR, it appears that the approach is well suited to Burundi, with potential for other contexts as well. The model integrates reference to elements of Christianity or Islam and then, using a couple-to￾couple peer approach, the Ab’IR give tools to improve communication, negotiation, and planning within the household. In rural communities where religion is a big part of life, it seems to have been well-received. The fact that the majority of the population in the province and all of its communes is Catholic is fortunate for the project given the gap in identifying and supporting Muslim and non-Catholic Christian couples. Reaching non-Catholic Ab’IR couples must be prioritized for the duration of the project. Implementation must be closely monitored and refined as necessary, particularly given that the two organizations (ODEDIM as lead implementer under CRS oversight) are associated with the Catholic faith. In the future, couples of various faiths should be identified from the beginning of such activities to ensure cohesion among couples of different faiths working in the same communities and to ensure that all couples of all faiths feel comfortable seeking and receiving advice and for the approach to be consistent. The narrow focus on decision-making at the household-level, combined with an approach that seems well-suited to addressing intimate household issues, seems to have had positive benefits for behaviors on nutrition and is promising should agriculture activities intensify in the remaining two years of the project. The project should, nevertheless, be careful to acknowledge the limits of the Ab’IR approach with regard to SGBV. Greater equality in joint household decision-making, logically and anecdotally, may contribute to prevention of some types of intimate partner violence (but likely limited to only this sub-category of SGBV). The project must not, therefore, overstate its impact in this regard especially given the fact that the project and field work SOW were not designed to measure or assess this in detail. There are eight indicators at the sub-purpose level for XSP.1 which seems excessive and unreasonably ambitious. Moreover, at the output and IR level, IRX.2.1 seems more appropriate as a second output for IRX.1. It is also worth noting that the IRX.1. indicator (“Average score of the perception of target population on Gender-Based Violence”) seems unreasonable in its coverage of “GBV” in general rather than focusing on household GBV which is more reasonably what might be impacted by the activities of this component. The wording of the sub-purpose and indeed IRX.3 Amashiga MTE - FINAL 61 suggest changes at the community level in terms of decision-making whereas this is an indirect process and there are few activities focusing on decision-making at the community level. This sub-purpose in general seems to be the furthest along in achieving its intended outcomes and therefore – although it was necessary to note these weaknesses – the MTE team does not encourage Amashiga and FFP to devote a lot of time to modifying it. XSP Recommendations 1. Develop a brief (four-six page) document that summarizes the logic of the component between now and the end of the project, including the identification and training of the Muslim Ab’IR couples. This document should specifically address integration with other Ab’IR couples and the long-term plan for how future Ab’IR couples of all religious confessions could be identified and trained. Given that the training in conflict resolution and GBV is less valued by current Ab’IRs and the time constraints before the project end, the project should consider adding the themes of conflict resolution (which are already themes in the Faithful House and Islamic House guides) and SGBV (especially the identification of complex SGBV cases and referral pathways) to a full week training on the Islamic Household rather than as a separate training. 2. Ensure that training materials and didactic materials are provided to the future trainers of Ab’IR couples (presumably Ministry of Gender and Social Affairs and/or representatives of different faiths). Ensure that the commune Social Welfare agents are involved in the training of the Muslim Ab’IR couples so that they might expand the identification and training of Ab’IR couples to non-Catholics. 3. Strengthen messaging to SILC groups around household management of financial resources particularly by focusing on PSP capacity and understanding of key themes regarding equitable household decision-making and appropriate nutrition for children (to link to household budgeting). The link between the SILC groups and household decision-making about resources seems to be a very natural one with potential to strengthen joint decision-making and reinforcing P1 and P2 outcomes. 4. Modify the results framework to consolidate (or clearly distinguish) IRX.1 and IRX.2. We recognize there may be other ways to simplify the design but a very basic simplification follows: Logical Framework Element Indicator IRX.1 HHs with behaviors that threaten food security and nutrition are identified and supported to change Average score of the perception of target population on Gender-Based Violence (8- 40)53 Output X.1.1 Ab'IR couples recruited, trained, and supported # Ab'IR couple trained and supported by Amashiga Output X.1.2 Couples receive training and support to make HH decisions together Number of households visited by Ab'IR couple per month. 53 As noted under Conclusions above, the IRX.1 indicator should certainly not focus on all types of SGBV but should focus on domestic violence or intimate partner violence. Having not included questions in the baseline, however, the project now has limited options without adding a new measurement tool (e.g. questionnaires to the couples that Ab’IR have counselled), something that at this stage in implementation the MTE team cannot recommend. Amashiga MTE - FINAL 62 5. Modify the results framework to focus IRX.3 on household decision-making by removing community decision-making from the formulation. Changes in community decision-making are likely to be an indirect effect and are unlikely to be feasible within the project timeframe. If Amashiga wishes to keep the current formulation (against the MTE team recommendation), it should introduce a series of activities to contribute to this aspect of the result. This would require significant time and human resources already strained to complete the expansion of the model to Muslim communities, the capacity development of commune-based trainers in addition to completing the previously planned activities of discussion groups, and the other recommendations noted above. 6. Track impact on broader, more equal decision-making by counting the number of male and female representatives of key committees, particularly the elected posts, such as the CDC, the Colline Councils, the CCDC, and the Commune Councils. This would help the project to learn if there is evidence of changing attitudes to women’s involvement in decision-making in the public sphere (although election to a committee is merely a first step and does not imply full integration in decision-making). It may also help to identify key informants (e.g., women who are newly elected) to better understand whether and how the project’s support to Ab’IR couples may or may not be contributing to more equal decision-making at the community level. 63 Amashiga MTE - FINAL 5. Overall Findings, Conclusions and Recommendations This section firstly presents some cross-cutting findings and conclusions. Secondly the MTE team provides a summary review against the evaluation’s objectives. These are organized according to an assessment of: i) the MTE’s five key questions and ii) the three main objectives of the MTE. Finally, the MTE team proposes a number of over-arching recommendations to optimize impact for the duration of the project and to support Amashiga preparations for the final project evaluation. 5.1 General Findings and Conclusions This section describes findings and conclusions related to cross-cutting project elements. Section 5.1 is organized as follows: - Social & Behavior Change Communication - Commodity Management - Financial Management - MEAL - Project Design & ToC - Environmental Compliance Social & Behavior Change Communication (SBCC) Introduction to SBCC activities54 In April 2016 Amashiga completed the SBCC strategy document (Amashiga Social & Behavior Change Communication Strategy 2015-2020), a detailed, evidence-based road map for a comprehensive multi￾sectoral SBCC component, complete with situation analyses, theoretical frameworks, suggestions for communication activities, and an implementation plan. It provides an excellent framework for effecting sustainable behavior change across all purposes. Illustrative examples of other documents that were developed to provide guidance for SBCC include: - A workshop report on developing key messages (Résultats d’un Atelier de l’Elaboration des Messages Clés d’Amashiga). - An analysis of gender barriers for each purpose (Analysis of Gender-related Barriers to the Reduction of Chronic Malnutrition and Stunting in Children <2 Years of Age). - A barrier analysis report on four key behaviors (Rapport de Recherche Formative Sur L’Analyse des Barrières au Changement de Comportement). - A barrier analysis on constraints related to adoption of BXW disease control techniques; and - An analysis of gender and SBCC at health centers (Analyse CCC et Genre dans les Centres de Santé à Muyinga). One of the first SBCC activities was to develop four messages to be disseminated across all purposes and groups: 1. Mothers of children 6-23 months feed their children three times a day with food from two or three food groups and provide two snacks between meals. 2. Farmers growing climbing beans use improved seeds from research institutions or other qualified structures such as the DPAE, ISABU, or NGOs. 3. CDCs and CCDC provide reports [on development plans] to the community twice a year during colline-level meetings. 54 This section summarizes SBCC for all Purposes. See sections on individual Purposes for additional information. 64 Amashiga MTE - FINAL 4. Husbands and wives with children <2 discuss their children’s health and nutrition every evening. Principal SBCC approaches are sensitization or awareness-raising (sensibilisation in French), which usually consists of giving information to groups of varying sizes, including large community meetings; radio messages with themes planned for all three purposes and the cross-cutting gender Sub-Purpose; distribution of some printed materials such as posters for health centers, nutrition training modules, and counseling cards for CHWs (P1) and visual aids on P2 interventions; and home visits by Ab’IRs to provide counseling to couples having marital problems and by CoProNut members and P1 MobiComs to reinforce messages to P1 participants and their neighbors. Skits, theater, music, dance, and contests have also been used on several occasions to disseminate messages, especially at the monthly fairs. Another important SBCC activity was to train certain project staff in SBCC: the P1 staff, including the MobiComs and their supervisors, participated in a week-long training and the XSP team also received training in behavior change principles. SBCC Findings Data collection for SBCC included: two KIIs with the SBCC Technical Advisor (SBCC TA); reviews of the SBCC strategy document and related materials such as formative research reports, studies, and training modules; consultations with other MTE team members; and observations of the monthly fairs, health centers, and P2 activities. Additional information on SBCC for P1 was obtained through KIIs with CHWs and project staff and SBCC questions were included in the 41 FGDs with P1 women participants, husbands of women participants, and CoProNuts. Principal findings are summarized here: - Of the four transversal messages selected for project-wide dissemination (see above), one message seems to have been heard by participants across all purposes: Husbands and wives with children <2 discuss their children’s health and nutrition every evening. While spouses may not actually discuss their children’s health every evening, FGDs and KIIs showed that there is a widespread understanding of the link between harmony at the household level and the positive impact on children’s health. - Although the SBCC strategy document covers all purposes, it has not been fully implemented due to resource constraints (financial and human resources) and to a lack of initiative and support from some senior program managers. [KIIs with program managers and the SBCC TA]. - There is a lack of variety and innovation in the SBCC activities. Planned interventions that would reinforce SBCC, especially for P1, have encountered delays: The Talking Books have not yet been distributed due to a procurement issue and no Community Scoreboards have been installed. Other activities that have not been tried include practical demonstrations, especially demonstration plots for P2, and using role models such as positive deviant farmers. [Document review, observations of P2 activities, KIIs with project staff, including the SBCC TA] - Amashiga messages cited by respondents in all the P1 FGDs with CoProNuts and women participants (28 FGDs) and almost half the FGDs with husbands (6/13) include:  The project’s basic theory of change – the need to increase agricultural production and diversification in order to make the transition from rations to using local products for a nutritious diet [also in P2 FGDs with participants];  The importance of a balanced diet (respondents may not always know which foods belong to which of the three food groups but the principle is understood) [also in P2 FGDs with participants];  Three key P2 messages (value of kitchen gardens, planting in a straight line, and using manure and compost to improve soil fertility) [also in P2 FGDs with participants];  General determinants of malnutrition (but not local, context-specific determinants); and 65 Amashiga MTE - FINAL  Basic tenets of IYCF and optimal child care (e.g., immediate and exclusive breastfeeding until six months; start complementary feeding at six months; importance of seeking qualified medical care for a sick child; hand washing; etc.). - The project developed a Vision Statement that is disseminated at the monthly fairs; P1 women participants and CoProNut members were able to summarize the statement. [asked in half the FGDs for these two groups – 14/28 FGDs] - SBCC is carried out at the monthly fairs but there are too many themes in the session and the methodology most often consists only of large group sensitization. The project agents conducting the sessions do not always demonstrate participatory animation and group facilitation skills. [Observations at 10 fairs] - Radio messages on themes promoted by Amashiga are broadcast on a regular schedule but as the P1 data collection showed, few people in the four communes visited had heard the messages. However, when people are able to hear the messages, their recall, especially of skits, is quite detailed. [2/14 FGDs with P1 women participants] - There are insufficient SBCC supports for staff and participants (e.g., didactic materials, teaching aids, posters, visual reminders, etc.). [Observations of P1 and P2 activities] - There is only one person for SBCC (the SBCC TA) and no dedicated counterparts for her to work with on any of the purpose teams. The two KIIs with her demonstrated that she knows the strategy document quite well, is continually researching SBCC options, and developing activities and budgets to implement them. - Some of the project staff and counterparts most directly involved in behavior change have received no training in conducting effective SBCC activities or in the basic principles of behavior change. This includes CHWs, CoProNuts, and Agriculture Monitors. [Document review; KIIs with staff, counterparts, and community volunteers] SBCC Conclusions 1. Inadequate resources and staff commitment: Amashiga lacks adequate resources (human and financial) to successfully implement a comprehensive, multi-sectoral SBCC strategy. Also missing is a strong understanding of and commitment to behavior change activities from a number of project staff, especially senior managers and those responsible for behavior change at the participant level. 2. Lack of variety in methods and activities and inadequate support for sustaining behavior change: The focus on providing information to large groups such as community meetings, group gatherings, and fairs and not on negotiations and support for behavior change is not conducive to lasting behavior change. Participants are hearing many of the key messages but lack the support, resources, and message reinforcement through multiple pathways to make the move from knowledge to change in attitude to sustained adoption of new practices. 3. Need for supporting materials and activities: The lack of didactic materials, teaching aids, other visual supports, and practical demonstrations hampers SBCC efforts. 4. Need for trained staff and other implementers: Many of the project staff, counterparts, and community volunteers have not been trained in the basic principles of behavior and/or in participatory group facilitation and effective animation techniques. SBCC Recommendations 1. Provide adequate human and financial resources to the SBCC component and ensure that senior program managers demonstrate a strong level of commitment for SBCC initiatives. Hire a second person for the SBCC team with a strong agriculture background to assist the SBCC TA in improving 66 Amashiga MTE - FINAL SBCC activities for P2, including the development of teaching aids and other supports. If other staff and partners are trained in basic SBCC (see #3 below), this will help ensure that SBCC becomes a priority across the project. 2. Have the SBCC TA review key sections of the SBCC strategy document with senior managers for each purpose. Jointly determine SBCC priorities for each purpose and develop an action plan including resources needed, a detailed timeline with outputs and ways to measure effectiveness and impact. In the action plan include a list of materials to be developed; this may include additional materials for promoting agriculture techniques, basic nutrition/hygiene and sanitation information for groups such as SILC and visual aids on good governance, including CDC and CCDC responsibilities toward the communities they represent. 3. Train CHWs, CoProNuts, Agriculture Monitors, new P1 MobiComs (once recruited) and other field level implementers in the basics of effective communication and facilitation and the principles of behavior change. Develop a streamlined version of a module such as Designing for Behavior Change and ensure that the training is practical, hands on and fun. 4. Conduct a rapid quantitative survey in all seven communes to determine if Amashiga radio messages transmitted by RTNB are reaching a substantial portion of the population. If not, explore using other stations or invest the resources in other SBCC activities. Commodity Management Introduction to Commodity Management Title II food commodities represent the main non-monetary resources used in Amashiga’s implementation. The total tonnage approved is 14,830MT of Corn Soy Blend Plus (CSB+) and Vegetable Oil (VO) for P1 activities, as well as Soy Fortified Bulgur Wheat (SFBW) and Yellow Split Peas (YSP) for FFA/FFW P2 activities. Table 1 in Annex J shows Amashiga has placed its calls forward (CF) against the total level approved. The last CF was on August 17, 2017 for an estimated arrival of January/February 2018. CRS/Burundi currently runs two warehouses for the storage of its Title II food commodities: one in Muyinga (1,400 MT capacity) and the other in N’Gozi (2,500 MT capacity), located 47 miles away. The commodities come through the port of Dar-es-Salaam where they are trucked either to Muyinga or N’Gozi based on space availability. Commodity Management Findings A summary of key findings follows. Additional findings related to commodities, including a summary table of the Calls Forward Schedule can be found in Annex J. Quality and cleanliness of storage facilities, nutrition fairs, and FFA/FFW activities - The warehouses in Muyinga and N’Gozi are not in an isolated location and are all fenced. [Observation] - The rooves are high enough to allow high storage of bags. The corrugated iron roof of the N’Gozi warehouse, however, has many small holes. - The inside and surroundings of the warehouses are clean. No trace of rodents. - The Commodity & Logistics Manager used the services of a certified firm to fumigate the warehouse in Isaka. The warehouses in Muyinga and N’Gozi have yet to be fumigated because, according to the Manager, food commodities are moving out quickly due to the steady implementation of the fair activities. 67 Amashiga MTE - FINAL - All the food commodities are stacked on pallets; however, current stacking of bags at the two warehouses is too wide with no alleys for air circulation around the stacks. - A substantial portion of the warehouse in N’Gozi is not well lit rendering data entry challenging. - In August 2017, three bags of CSB+ sent out for distribution (in Kiiryama) were returned to the warehouse due to weevils’ infestation and foul odor identified by participants. They were in the load of foods kept at the Tanzania border in trucks for about seven weeks due to tax exoneration problems. - On September 29, 2017, 4.47MTof unfit foods for human consumption were destroyed in an environmentally sound way with FFP’s and MOH’s approval. The foods were transformed into charcoal briquettes by Bioenergie Burundi. - On nutrition fair sites, the shelters are just big enough to accommodate participating women and children at the reception and registration stage of the fair. Others waiting for their turn have to stay under the sun along with their children. - Discussions with participants and observations showed the average time spent on site is three to five hours. Some women walk two hours one way to come to the site: five hours spent on site plus the four hour walk make a total of nine hours to collect the food rations. - The surroundings of the nutrition fair sites are littered with discarded sheets of paper and other debris. None of the latrines had a Tippy-Tap for handwashing. - CoProNut members use their bare hands to help participating women adjust their food rations in their buckets. - There are no signboards (Branding & Marking) to show FFA achievements. [all visited FFA sites] - FFA participants in Gakoni said they initially had a problem with Bulgur as it was not in their local diet and some people experienced minor gastric problems. [FGDs] Further complaints were not noted after a cooking demonstration by Amashiga staff. - The current location of FFA tools, such as spades, hoes and pickaxes, provided by the project to carry out activities is unclear. Adequacy of mechanisms to safeguard against loss or abuse - Both warehouses are monitored 24/7 by a security company. - The N’Gozi warehouseman and his assistant were fired in April 2017 for foods diversion (84 bags of CSB and 11 cartons of oil). - Monthly physical inventories were completed by different staff members each time. Their names and signatures are written down in the ledgers and on the stock cards. Common causes of delivery delays and commodity losses - Due to Burundi’s civic unrest in 2015, the tonnage of the first call forward was stored in Isaka (Tanzania) for almost seven months. This unusual situation required additional resources and a change in plans. A new storage location was rented and staffed with a warehouseman, a cleaner and security guards at a cost of $16,500 per month. A specific team of three to four persons traveled every month to Tanzania for the Figure 11: Too Wide Stacking-Muyinga Warehouse, October 2017 (by A. Tingbo) Figure 12: Women & children waiting for food rations under the sun in Nyamirambo. October 2017 (by A. Tingbo) Figure 11: Too Wide Stacking-Muyinga Warehouse, October 2017 (by A. Tingbo) 68 Amashiga MTE - FINAL physical inventories. This situation was temporary and, with the restoration of security in the country, the Isaka warehouse was no longer needed. - The clearing of the second shipment faced tax exoneration problems. Prior to receiving a tax exemption letter from the Minister of Finance in September 2016, trucks loaded with foods spent six weeks at the Kobero border before the clearance of the commodities. - 676 cartons of Vegetable Oil ($23,509) were declared short-landed and treated as maritime losses. The 676 cartons of oil were actually offloaded from the vessel and mistakenly stored at WFP’s warehouse. The WFP warehouseman returned them to the ship as an excess quantity according to their Bill of Lading. Division of roles and cooperation among project staff, external partners and community members and clarity of commodity records and reports - The last tonnage of the award is expected to arrive in February 2018 with the 5th call forward placed in August 2017. As per the Commodity & Logistics Manager, the new foods pipeline will have food rations available till September 2018. As a result, the current plan is to start the Voucher program in October 2018. The Commodity & Logistics Manager reported that CRS/Bujumbura was working with CRS/Baltimore to send a consultant by the 2nd week of December 201755 to assess the possibility of the Voucher program either with local or regional foods purchase. - There is a system in place with internal control mechanisms for the delivery of the food commodities from the warehouses involving P1 and P2 key actors. Their requests are sent to the Foods Accountant and Commodities & Logistics Manager who review them for accuracy of sites and number of beneficiaries against their existing files. - Commodity Status Reports, Recipient Status Reports and Loss Summary reports are submitted on a quarterly basis to FFP. There have not been any claims or corrections request from FFP. - Problem solving mechanisms: i) suggestion boxes were introduced in September 2017 and ii) two women in Gasenyi reported to the MTE team that they had been denied food rations for three and four months respectively due to their cards malfunction. Commodity Management Conclusions 1. The planned incorporation of vouchers by October 2018 does not appear to be feasible as local foods availability is not foreseen for the near future. See Section 4.2 for further findings and analysis of P2. The results of the flour fortification by private entities are not visible either. See Section 4.3 for relevant findings and analysis of P3. 2. The internal control mechanisms for warehousing and food deliveries meet FFP's requirements. 3. The fairs are well organized e.g., good site space, involvement of CoProNuts, group distributions, etc. but overly focused on food rations with little attention to the quality of the sensitization and message retention. See Section 4.1 for related analysis and conclusions. 4. Quality and cleanliness of storage facilities: - Small rooftop holes at the N’Gozi warehouse may allow leaks to damage the food commodities. 55 Subsequent to the period covered by this MTE. 69 Amashiga MTE - FINAL - Hygiene/Sanitation is a concern at fair sites especially as Amashiga is promoting hygiene. The MTE team observed paper debris (from empty bags) all over the ground, and utensils for cooking demonstrations in Gasenyi and Tura were kept on a bag on the ground instead of on the wooden racks being promoted by the project. Furthermore, CoProNut members helping participating women could use a plastic plate, rather than their bare hands, to assist in adjusting food rations. - Distances travelled and time spent on fair sites are relatively long for some women: eight to nine hours away from home or their petty trade/farm could be harmful to other family and existing livelihood. 5. Adequacy of mechanisms to safeguard against loss or abuse: - The 676 cartons of vegetable oil declared short-landed is an indication of a weakness of CRS’s maritime surveyor (Polucon) and a lack of coordination among key players receiving the same commodities at the same time. - The weevils-infested bags identified in August 2017 should have triggered a broader quality check. In foods management, the presence of one pest-infested bag might be an indication of a broader problem that needs quick actions. 6. Common causes of delivery delays and commodity losses: - The long storage of foods in Isaka combined with the situation of the trucks stranded at the border while waiting for the tax exoneration are good conditions for the proliferation of weevils and other pests. Even though the Best Use By Date was not affected, some participants’ reaction to infested foods to be given to them can take another interpretation or turn. People who dislike food programs might say CRS or the US is sending foods they cannot eat in their own country. 7. Division of roles and cooperation among project staff, external partners and community members: - Tax Exemption: CRS was very active when the tax exemption problem occurred. The COP, the Deputy Chief of Party (DCOP), the Commodities Manager and Progam Managers, etc. met at different levels with the government authorities and the International NGOs network in Bujumbura, as well as with the Governor in Muyinga. The coordinated effort was instrumental in finding a solution with the Minister of Finance who wrote a letter to reinstate the eligibility of CRS for tax exemption. However, in spite of the letter of the Minister of Finance that helps clear the foods since the 2nd call forward, the political situation in the country might trigger another reshuffling of the Ministries. This might lead to another interpretation of the tax exemption clause applicable to CRS’s Title II foods. - The lack of FFP claims or request for verification of Commodity Status Reports (CSR), Recipient Status Reports (RSR) and Loss Summary Reports (LSR) testifies to the reports’ accuracy. - There is good collaboration between P1 and food commodities staff as demonstrated by the management of the food requests and the timely dispatching of the foods to the sites. - Issues with non-functioning cards and linked communication to participants should be resolved within a month and not allowed to continue month-to-month. Food Distributor, Community Mobilizer, or End-use Checker should escalate such problems and treated them with urgency. Figure 12: Women & children waiting for food rations under the sun in Nyamirambo. October 2017 (by A. Tingbo) 70 Amashiga MTE - FINAL Commodity Management Recommendations 1. Strengthen relationship and contractual arrangement with State authorities to protect tax emption arrangements. Meet from time to time with Office Burundais des Recettes (OBR) and the Direction de la Politique Fiscale to ensure mutual shared interpretation of the clauses in the Minister of Finance’s letter about tax exemption. When the current CRS-GoB agreement Amendment56 comes up for renewal, ensure inclusion of specific clauses on tax exemptions for Title II foods. 2. Refine warehouse management - Fumigate warehouses twice a year by certified firm to prevent foods infestation. Even though no pests were found in the warehouses, the August 2017 incident where participants on a distribution site identified three weevils-infested bags with foul smell means the Amashiga must remain vigilant; even one pest-infested bag can be an indication of a broader problem; - Review current stacking pattern in warehouses and make smaller stacks that leave alleys in between. This would help air circulate and reduce proliferation of pests. It is more convenient to have stacks with the basis layer of 20-25 bags (length position). 3. Make fair sites more accessible and agreeable for participants - Build secondary shelters for protection of women and children waiting for their turn for food rations against sun & rain (no extra cost). - Explore ways to reduce travel time and time spent on site by participating women i.e. a couple of additional sites, reorientation of some participants to alternative sites. - Ensure hygiene and sanitation principles on sites:  Remove trash from fair sites prior to the beginning of activities (under CoProNut’s guidance);  Put a Tippy-Tap next to each latrine;  Provide CoProNut members with a plastic plate to help adjust the weight of the CSB+ when it is being shared among two participants instead of their bare hands. - Revisit FFA approach and establish how infrastructures should be maintained with community members and/or local authorities; make sure FFA activities do not impede farming. - Put Branding & Marking signs at FFA achievements or obtain a waiver. 4. Maintain open and consistent communication - Make sure to consistently inform participants and authorities alike on any unforeseen delays in activities implementation e.g., unavailability of food rations due to administrative problems. Financial Management Financial data and procedures are already available to CRS, partners and USAID/FFP through standard reporting mechanisms and the country program is expecting a standard visit from the CRS/Baltimore audit team. The MTE team believes that the challenges faced in the financial management of the project are a result of the implementation and general project management weaknesses outlined throughout this report. The MTE team has therefore chosen to present its high-level financial management findings and conclusions together in this same section. Recommendations related to financial management have been incorporated into the overall MTE recommendations (Section 5.3as they are tied to overall project management considerations. The financial management-related field work included eight KIIs with finance, administrative and program staff in Muyinga as well as email exchanges with the CRS Finance Manager in Bujumbura. 56 Food For Peace Host Country Agreement 71 Amashiga MTE - FINAL Additional data was collected through a literature review of budget documents and reports shared with the MTE team during the field work. Financial accountability Donor accountability is coordinated by CRS as the AMAHSIGA consortium lead. The responsibility for managing the four partners receiving funds (WFP, ODEDIM, RBU 2000+, Bioversity International and IMC) is held by the CRS Grant Manager who conducts frequent field visits with partners. Partners send their forecasts to CRS/Bujumbura (the coordinating finance department) and justify expenditure on a monthly basis before any new fund transfer. Aside from some late report and cash forecast submissions by WFP and Bioversity International, project staff did not report any particular challenges. Sufficiency of finances to assure good project implementation Given the substantial delays experienced by the project to date, it is unsurprising that the project is underspending compared to proposal projections – see Table 16 below for a summary of Amashiga Inception-To-Date Project Analysis including Variance column relative to the obligated amounts as well as relative to the agreement totals. During KIIs, finance staff reported that they did not face any particular challenges with regard to the budget and that it was sufficient through the end of the project. It is unclear to the MTE team, however, how Amashiga expects to allocate remaining funds for the full duration of the project; projections (as per donor requirement) shared by CRS with the MTE team extended to February 2018 only. It should be noted that the project has only spent 45% of the total budgeted amount (43% of 202(e); 67% of ITSH and 43% of Commodity In-Kind/Freight). This suggests that the project will have to spend more in each of the remaining two project years in the Direct Costs and ITSH categories than they have spent in any year to date, which raises some questions about the ability to use fully the budgeted amounts by the project end, although this could be due to the constraints of a number of activities being delayed due to IEE approvals during FY2017 which slowed spending relative to the true potential of the project. 72 Amashiga MTE - FINAL Table 11: Amashiga Inception-To-Date Project Analysis including Variance57 Category Agreement Value Obligated Amount (through FY17) Total Inception to Date Expenses (through FY17) Variance Relative to Obligated Amount Agreement Balance through end of project (FY19) 202('e) Direct Cost $ 23,865,888 $ 13,836,818 $ 10,172,091 $ 3,664,727 $ 13,693,797 26% 57% NICRA $ 3,525,652 $ 2,027,869 $ 1,473,969 $ 553,900 $ 2,051,683 27% 58% TOTAL $ 27,391,500 $ 15,864,687 $ 11,646,060 $ 4,218,627 $ 15,745,440 27% 57% ITSH Direct Cost/TOTAL $ 5,132,400 $ 4,064,253 $ 3,464,251 $ 600,002 $ 1,668,149 15% 33% Commodity In-Kind and Freight Direct Cost/TOTAL $ 17,026,500 $ 9,045,400 $ 7,318,213 $ 1,727,187 $ 9,708,287 19% 57% Grand Total Direct Cost/TOTAL $ 49,550,400 $ 28,974,340 $ 22,428,524 $ 6,545,816 $ 27,121,876 23% 55% Project Period: 09/26/2014-09/25/2019. Flexibility of the budget to respond to changing conditions/Overall expenditures made as allocated in the three funding sources Interviewees reported having sufficient flexibility to adjust spending either by using the standard available budget flexibility at line level or by requesting specific approval from USAID/FFP to respond to real project needs not originally anticipated e.g., i) the 57 Sources: i) Amashiga Proposal (for Agreement Value) and ii) Inception-to-date Finance Report at 30 September 2017 as shared with the MTE Team by Finance Manager 73 Amashiga MTE - FINAL unexpected storage of food commodities in Isaka (Tanzania) led to unexpected extra costs for the ITSH budget (rental cost, hiring of a warehouseman, a cleaner and security guards, monthly visits to Tanzania for physical inventories, purchase of a generator, etc.) and ii) the AmashigaCOP requested and obtained approval for the purchase of a central phone system, the construction of a shed for motorcycles and the construction of a fuel tank. Staff did not indicate any constraints. It must be noted that there have been few project decisions or strategy changes implying any particular need to modify the budget i.e., the budget has not needed to respond to changing conditions thus far because – as outlined elsewhere in this report – the project design has not been adapted to these changing conditions. In these circumstances, it is possible that staff have never had to fully assess financial constraints or limits to the project’s financial flexibility. MEAL MEAL Findings - The project proposal outlines a learning, knowledge and capacity building approach whereby process and outcomes of learning through Amashiga implementation will be systematically documented, communicated with stakeholders and applied to program implementation and capacity building via formal learning agendas and knowledge sharing processes. The proposal further references Technical Resource Partners to support the project’s research and facilitate the scaling up of findings. - Staff reported that work on the M&E Plan engaged significant amounts of senior staff time, in-country and in headquarters. The Plan was submitted initially in December 2014, and revisions were requested by USAID/FFP three times each in FY15 and FY16 before the plan was finally approved during Q2 of FY17, over two years later. The Cooperative Agreement stipulated that an M&E workshop be organized by FFP for the awardee to learn about FFP’s M&E requirements and to refine the project’s ToC, LogFrame and IPTT. It lays out a timetable for completion of these documents: the revised project Theory of Change, LogFrame, and IPTT would be due to FFP within 20 days after the completion of the workshop and awardees were to submit their comprehensive M&E Plan for the project within 60 days after the M&E workshop. After the FFP￾organized workshop in November 2014, the project consortium submitted “major revision[s]” in October 2015, April 2016, end of FY16 and early FY17. The FY16 ARR (two years into implementation) stated that “The process of developing and revising the Amashiga M&E Plan is not entirely clear to CRS”. - The majority of indicators in the LogFrame show values of zero and there are few indicators that serve to benchmark progress or show how outputs will lead to achievement of intermediate results. - The MTE team’s analysis of project data based on the initial desk review and on program data shared with the team during field workis presented in Annex I. 74 Amashiga MTE - FINAL MEAL Conclusions Amashiga has spent enormous amount of time and staff energy developing and revising its M&E Plan. While the lengthy development and ongoing revisions to the Plan cannot absolve the consortium of the responsibility of quality data collection for ongoing activities, it is clear to the MTE team that this whole processrepresented a significant challenge for the MEAL team as well as for Amashiga senior management and impeded them, and program staff, from focusing on application of systems and procedures for data gathering, management and quality control. Further, MEAL and program staff do not appear to have a shared understanding of the data quality issues. This is evident in the lack of clear criteria for data quality for activities, and limited indications of monitoring of quality of results or activities by program staff (or by MEAL or management staff). Finally, reporting to FFP58 concentrates on providing process updates on the development of and revisions to the M&E Plan, to the detriment of using data to inform implementation or sharing learning from Amashiga implementation in line with the holistic intentions described in the project proposal. MEAL Recommendations 1. Hold a meeting with the COP, DCOP, the MEAL and sector coordinators to: - Inventory the problems of (i) data availability; (ii) data quality; and (iii) quality of outputs/activities, particularly training events; - Agree on the problems or clarify where there is disagreement on the problems; - Conduct a longer follow-up meeting one to two weeks later where participants present proposals to address the problems (and further evidence of the problem if needed); and - Develop an action plan with deadlines and responsibilities. 2. Strengthen the collection of data in the existing M&E Plan rather than undertaking a process of further revisions. If the project is to be extended or if there is a modification to the LogFrame, there should be a very short deadline for both USAID/FFP and Amashiga to agree on the changes. It is essential to avoid another lengthy period of uncertainty and inaction. Project Design & ToC Design & ToC Findings - In accordance with FFP requirements, at the beginning of FY15 Amashiga started work on a revised Theory of Change (ToC). The FY15 and FY16 ARRs note that USAID/FFP requested extensive revisions to the ToC and associated revisions to the M&E Plan, IPTT and LogFrame three times in FY15 and three times further during FY16. - Annex I contains a description of the changes or refinements made through the process. - A ToC workshop was planned for November 2017 after the MTE to revise the ToC but was apparently postponed, perhaps to await the recommendations of the final MTE report. - Of the 26 IRs included in the Amashiga LogFrame, seven have no associated outputs, and an additional 14 have only one output. 58FY15 and FY16 ARR; FY17 ARR was not yet available or shared with the MTE team. 75 Amashiga MTE - FINAL Design & ToC Conclusions The discussion of the conclusions regarding the design and ToC is more extensive than the findings as such discussion necessarily involves the judgement of the MTE team rather than pure statements of fact. The team’s conclusions cover: - the minimal progress to date towards the project purposes and goal - the impact of particularly weak achievements under P2 on the overall design - the undermining of the design by revisions that are not grounded in learning from the specific area and challenges of implementation Following our review of the existing ToC and LogFrame, it is the MTE team’s opinion that the project will not be able to achieve its goal. The current design, while potentially logical prior to the extensive political violence in Burundi in FY15, was written for a significantly different implementing context (external and internal). Based on the actual and reasonably projected results (barring dramatic changes in project implementation), the existing ToC simply cannot hold and malnutrition rates are likely to increase as women and children graduate from the PM2A distributions. As is consistent with FFP programming, the entry point for activities is the distribution of commodities to pregnant women and mothers of children under two years old while, in parallel, the project works to i) build capacity of local populations to produce diverse nutritious food, ii) increase household access to diverse nutritious foods, and iii) ensure local ownership and appropriation of key project activities. The distribution component has a target to reach approximately 50,000-60,000 PLW with free distributions of 7,500 T/year of CSB. To replace these commodities, local annual production of at least 1,900 MT of soy (or equivalent protein) and 5,000 MT of maize (or equivalent calorie￾rich, and ideally vitamin-rich, carbohydrate) would be needed. Based on the findings of the MTE team, if the project support for the current planting season goes well (maize seed had not yet been acquired at the time of the MTE), Amashiga can project increased production of about 800 MT of maize, 100 MT of bio-fortified beans and 0 MT of soybean by the end of Season A 2018, thus leaving a significant shortfall between needs and production capacity. In addition, given that the project has not yet done demonstrations on how to prepare nutritious meals using locally available ingredients, it is difficult to be assured that participants would be capable of preparing nutritious diets even if they were to have access to the appropriate ingredients (which is doubtful given the lack of progress noted above). Given the limited P2 achievements at the time of the MTE, we can assume that only a small proportion of households will have increased their access to the key elements to replace the commodities currently being distributed. Moreover, without assurances of the sustainability of the CoProNuts, or some alternative or transition plan, it is unclear how women who become pregnant or become mothers each month will access the knowledge that women now benefitting from distributions have gained through project activities linked to distributions. Finally, many of the governance activities, including the linkages with the DPAE, have not yet started or have had limited reach so impact will remain theoretical or will only begin to be measurable at project end. The understanding of what constitutes truly local determinants of malnutrition seems quite limited and they do not have an apparent capacity to analyze local 76 Amashiga MTE - FINAL determinant on an ongoing basis. The scorecard of agricultural services, which does not appear to have been linked to P2 activities or follow-up, is a promising start but it takes significant time to build local ownership and capacity and will have very few follow-on scorecards (to follow up on actions taken and identify new actions) prior to the project end. It is a source of concern for the MTE team that the project’s conceptual design is still under regular time-consuming review without these modifications leading to adaptations linked to the context, to helpful milestones such as the MTE or to project planning and monitoring tools in a consistent way. While the MTE team commends investment in an intellectually sound Theory of Change, the process to develop it has been lengthy, open-ended and disconnected from the potential implications for indicators, data collection and evaluation. Further, the fact that the individuals involved in such an activity are the same as those responsible for detailed planning and targets achievement is an additional burden that can only have detrimental effects on implementation. Finally, the lack of outputs for a number of IRs in the IPTT and LogFrame is problematic, as it is inconsistent with basic principles of project design and may have contributed to the lack of recognition that the project design would not be fully achieved as written. While USAID FFP guidance stipulates that the ToC forms the foundation for the LogFrame and the IPTT, FFP’s guidelines dictate that “every application and M&E plan must include a LogFrame that is consistent with the TOC”59; that the LogFrame – with its specified performance indicators and data sources60 – forms the basis of ongoing monitoring efforts; and the IPTT is useful in “following activity performance and comparing it against planned progress.”61 The MTE team acknowledges that the Amashiga team factored such IRs without direct associated outputs into their ToC but the lack of linkages to the LogFrame and IPTT led to an unbalanced presentation of the LogFrame. This undermines both the capacity of Amashiga to measure the impact of its achievements and that of USAID/FFP to hold the project to account. Design & ToC Recommendations 1. Hold a two-day workshop for USAID/FFP and Amashiga to finalize together the ToC and any relevant changes to the LogFrame so that these documents can be finalized and linked to a clear plan of activities through the project end. With 18 months left in the project,62 the project cannot afford additional lengthy back-and-forth communications between USAID/FFP and the project staff. 2. If the Amashiga team and USAID/FFP agree that certain outputs contribute to multiple IRs, then these “linked” outputs should be included explicitly as such in the revised LogFrame. 3. For the remainder of the project, prioritize i) sustainable production of food (through livestock and crop-based agriculture, including seed systems) and ii) helping local communities learn appropriate feeding practices using locally-available ingredients. There are other valuable 59 USAID, “Office of Food for Peace Policy & Guidance for Monitoring, Evaluation, and Reporting for Development Food Security Activities,” December 2016, p. 21. Emphasis by Oxu 60 USAID, “Logical Framework,” http://usaidprojectstarter.org/content/logical-framework-lf 61 USAID, “Office of Food for Peace Policy & Guidance for Monitoring, Evaluation, and Reporting for Development Food Security Activities,” December 2016, p. 29. 62 At submission of this final MTE report in December 2017 77 Amashiga MTE - FINAL program elements but these two are the foundation of the design and are not yet visible in implementation. Environmental Compliance Environmental Compliance Findings - Table 2 in Section 2 summarizes the different submissions by CRS to USAID/FFP to meet environmental compliance requirements. - The LogFrame organization and wording of project outputs were modified after the project commencement but these modifications are not reflected in the IEE submissions. The MTE team understands that this may have been to meet USAID/FFP/BEO requirements for consistent presentation of outputs across submissions. - Project staff and senior CRS management cited lack of IEE/PERSUAP approvals, including implications for compliance with (non-specified) provisions of the Cooperative Agreement (CA), as a major justification for delays in implementation. This justification was also included in reports to USAID/FFP. - Similarly, in July 2016, the then Amashiga COP instructed team members by email that a number of activities could not be implemented until “FFP approval” was received. These activities included nutrition messages, WASH, kitchen gardens, the access to and use of agriculture inputs, improved agricultural techniques, goat distribution, pre- and post-harvest loss reduction, training of youth, support to CCDCs for the design and implementation of DRR and contingency plans, and the sharing of lessons learned and policy work around fortification, etc. It is unclear to the MTE team what new information precipitated this instruction at this particular time as no specific rationale or references were shared with the team, and the timing does not correspond with the dates of responses from USAID/FFP in the documents that were shared with the MTE team. This former COP left in Q1 FY17 and was not available for interview by the MTE team. - Considerable resources (including senior management, HQ and external consultant time) were expended to prepare a substantial volume (>300 pages) of environmental documentation (i.e. multiple IEE submissions, the Agriculture PERSUAP, the Fumigation PERSUAP and the Climate Risk Screening and Adaptation documentation). CRS management acknowledged, however, that the preparation and tracking of the ToC and the M&E Plan took priority over management of the IEE requirements and documentation. - In the documents shared with the MTE team, there is little formal, detailed communication describing the consequences of these delays for the project beyond occasional references in quarterly reports and a few follow-up emails by CRS to USAID/FFP. Only one example (dated October 2016) shared with the MTE team demonstrated a CRS challenge to stringent USAID/FFP/BEO conditions attached to the August 2015 approval. Project staff confirmed that meetings and conference calls were organized at times to discuss outstanding approvals or to confirm expectations for the environmental documentation, particularly in 2016 (almost two years into implementation) but the MTE team did not see notes or actions resulting from these meetings and calls. 78 Amashiga MTE - FINAL Environmental Compliance Conclusions Given that project staff frequently cited the delays in approvals of the IEE and Agriculture PERSUAP as reasonsfor delays in implementation, the MTE team chose to go beyond the elements outlined in the evaluation SOW63 and conducted a review of the management of these submissions and approvals. 64The MTE team found that the primary drivers of the cumulative delays were: - Excessive time to prepare documentation by CRS; - Vague and disconnected approvals containing disproportionate conditions by USAID/FFP/BEO; and - CRS’s conservative interpretation of the conditions associated with approvals. As can be seen in Table 2, the delays were mainly due to lengthy intervals between submissions of documentation by CRS (for a variety of reasons) rather than attributable to an exceptionally lengthy approval process by USAID/FFP/BEO. For example, there was a gap of 28 months65 between submissions by CRS of the second and third revisions to the IEE whereas approvals by USAID/FFP/BEO generally took three to four months. As explored below, however, the imprecise nature of the USAID BEO approvals did not facilitate decisive implementation action by CRS either. In the particular case of Amashiga, CRS erred on the side of caution when interpreting requirements and conditions and exhibited a tendency to suspend activities when faced with uncertainty. This is partially understandable given that no irreversible commitment of resources can take place until the environmental documentation is submitted by CRS and then approved by USAID/FFP/BEO. The conservative approach to avoid disallowable costs does not, however, seem to have been balanced by any sense that the delayed activities would have severe consequences for the fulfilment of the theory of change and were a disservice to project participants. Curiously, USAID/FFP does not appear to have raised any concerns about the delays in implementation as a result of CRS’s overly-cautious management of environmental compliance issues.66 The lack of direction and planning regarding the management of environmental compliance under Amashiga is quite surprising and relatively inexplicable to the MTE team, particularly given CRS’s international experience and HQ resources. In parallel, and to a lesser degree, the MTE found that the environmental conditions issued by USAID/FFP/BEO were disproportionate to the scale of the activities and the guidance provided to CRS was lacking in precision and therefore not conducive to the overall benefit of project participants. In Sections 4.1, 4.2 and 4.3 the MTE team has outlined steps taken related to the outputs most directly impacted by environmental compliance requirements and the impact of delays is also 63 Indeed, given the incomplete status of the Environmental Mitigation and Management Plan (EMMP) and IEE at the time the SOW was developed, the focus of the SOW (Adherence to and adequacy of the Environmental Mitigation and Management Plan (EMMP) and the Initial Environmental Examination (IEE), including how well planned and actual actions do/don’t comply; recognition/ avoidance of unforeseen environmental damage) is somewhat surprising. 64 The MTE team does not claim to have conducted an exhaustive review of the process or of the full rationale of the interpretations of requirements by CRS as these were simply not documented or consistently applied. 65 2nd submission: April 2015; 3rd submission August 2017. 66 If such concerns were shared, supporting documentation was not provided to the MTE team. 79 Amashiga MTE - FINAL discussed in those sections. To complement that analysis, the MTE team presents below i) two types of issues where CRS’s management of these compliance issues was inadequate and where lessons should be learned and ii) some weaknesses in the usefulness of the guidance and conditions issued by USAID/FFP/BEO. The MTE team understands that it has not accessed all exchanges between CRS and FFP/USAID on this matter nor did the team focus interviews with management only on this issue. The intent of the subsequent analysis is not, therefore, to interpret the environmental compliance requirements or to point out flaws in every decision but rather to take a step back and comment on the overall effectiveness of the process and the documented rationale (or lack thereof) for decisions affecting implementation: - Management of “deferred” outputs:  As outlined in Section 2 and referenced throughout Section 4, five outputs/activities were classified as “deferred” in the April 2015 IEE submission by CRS.67 This means that CRS determined that these activities had not been sufficiently defined (more than six months after the start of the project) to enable a thorough analysis of potential environmental impact. CRS therefore recommended a classification of “deferral” until such time as the details could be confirmed.  Condition Five of the August 2015 Environment Threshold Decision confirmed that CRS must submit and receive USAID/BEO approval for an IEE Amendment before starting implementation of currently-deferred activities.  In October 2016, CRS requested approval by email to proceed with these activities but did not submit a revised IEE.  The MTE team did not identify any further correspondence or action on this issue prior to the CRS’s revised IEE submission in August 2017 at which time its revised IEE included a detailed Environmental Risk Analysis and Mitigation Measures for these deferred outputs. As no consolidated analysis or explanation of the interpretation of the IEE process or conditions appears to exist as yet and the Amashiga team did not provide specific justifications, the MTE team can only deduce that CRS did not understand the IEE process and/or internally began to conflate the deferred outputs with the outputs on hold pending the preparation and approval of an Agriculture PERSUAP.68 - Management of USAID BEO Environmental Threshold Decision Conditions August 2015: The above interpretation (that CRS did not understand the IEE process and/or conflated the deferred outputs with the outputs on hold pending approval of an Agriculture PERSUAP) is further supported by how CRS managed two conditions: i) Condition Two: CRS must take caution when importing nonnative animal breeds into Burundi and ii) Condition Four: CRS will need to effectively monitor performance indicators for fuel-efficient cookstove activities.  As described previously in Section 4.1 and 4.2 above, in an email to USAID/FFP in October 2016 on the various pending IEE approvals, CRS recalled the marginal nature of 67 1.2.3: HH have increased use of hygiene behaviors, potable water and sanitation infrastructure; 2.3.2: Youth and landless, especially females, have diversified off-farm livelihoods; 3.1.1: CCDCs have increased capacity to design and execute gender￾responsive, conflict-sensitive action plans to mitigate natural and manmade shocks; 3.3.3: Implementation of food policies focused on food access, prices, and fortification is strengthened by lessons learned from Amashiga: Irrigation of Kitchen Gardens [2.1.3: Farming HH have increased access to inputs for crops, especially for women] 68 This was delayed for a variety of reasons including following difficulties obtaining a visa for an international consultant. 80 Amashiga MTE - FINAL the cookstove activity and the rationale for importing animals along with measures in place to manage and monitor the distribution of goats, etc. At the same time, CRS requested FFP approval to proceed with these activities. The basis for requesting approval is unclear to the MTE team but Amashiga management indicated it was a CA provision.  In a response to CRS’s email, the USAID BEO addressed one other issue from the email (waiver for the limited use of the specified pesticide, for the 2016 planting season) but did not acknowledge the request for approval for the goat and cookstove activities.  The MTE team did not see evidence of any further follow-up or requests for clarification from CRS before the August 2017 IEE resubmission (later approved in November 2017). In both of these instances, it is unclear to the MTE team why CRS suspended these activities for such extensive periods and why such a gap was allowed to persist between requests for an IEE amendment. Ultimately, whether it is through lack of understanding of requirements or lack of tracking of approvals and their implications or a combination of both, it is an unacceptable failure of management given the importance of these activities and approvals and the negative implications for participants. This failure risks being compounded by a lack of explicit learning from the process by CRS/Amashiga. Finally, the MTE team has concluded that the replies by USAID/FFP/BEO to CRS submissions and requests have, at times, exacerbated the lack of understanding and contributed to the delays in implementation. This is demonstrated by three main examples: - In October 2016, CRS wrote to USAID/FFP by email to respond to USAID’s August 2015 Environmental Threshold Decision. The email outlined (at least in part) CRS’s understanding of the USAID/BEO treatment of deferred outputs, IEE approval conditions and five outputs considered affected by the Agriculture PERSUAP under preparation. It also contained a request for a waiver to start implementation in Q1/FY17 of the five outputs pending submission and approval of the Agriculture PERSUAP. The USAID BEO response, however, only acknowledged and granted a limited, temporary waiver covering the use of 12 pesticides. From an outside perspective, it is difficult to deduce if USAID/BEO is ignoring superfluous approval requests from CRS (i.e. CRS did not need approval to proceed with all the outputs) or not considering the approval requests because they are not being submitted in the appropriate format (i.e., in the shape of a revised IEE with request for amendments) or for some other unexplained reason. In any event, the response does not adequately address issues that CRS is clearly finding challenging to manage and where support is being requested. At any rate, the provision of non-specific responses that do not acknowledge requests from CRS or correct assumptions of expectations are not conducive to resolving outstanding approval requests or meeting of requirements. - The non-specific and wide-ranging Condition Number 1 attached to the USAID BEO Environmental Threshold Decision (August 2015) as follows: CRS must sufficiently integrate climate screening and sensitivity into all relevant activities, in accordance with the US Presidential Executive Order 13677 [….] greater climate sensitivity must be mainstreamed and integrated into all relevant activities. All activities must reflect a sensitivity and awareness of the importance of climate change issues….” The use of vague terms in Conditions such as “sufficiently” and “all relevant activities” without providing any framework for assessment, next steps or potential consequences for the 81 Amashiga MTE - FINAL awardee is unhelpful. It implies a need by USAID/FFP/BEO to ensure administrative application of an Order without any genuine concern for implementation or implications. - Unlike for project reporting and M&E requirements outlined in the CA and general regulations, there does not appear to be a timeframe within which environmental compliance documentation must be finalized. This means that it is more challenging to hold awardees to account for submissions and – although implementation and meeting participant needs should be sufficient incentive for CRS and USAID/FFP – there appears to be no sense of urgency to the process. As a final general point, the MTE team has concluded that if additional approvals from USAID/FFP emanating from CA provisions were indeed necessary to proceed with activities (e.g., the distribution of goats) subsequent to the issuing of approvals with conditions for these activities by USAID/FFP/BEO, this step appears to be an arcane and unnecessarily complex one. If the requirement exists, 69 then it should at least be integrated into the USAID/FFP IEE approvals process. Environmental Compliance Recommendations 1. For CRS/Amashiga: Document lessons learned on the IEE and PERSUAP preparation and submission processes for management of future projects. These lessons should include: - Estimated resource needs for document preparation, including in-country assessments; - Library of relevant documents/references/explanations compiled by consultants when preparing environmental compliance documentation; - Workplan and SOW templates for potential consultants; and - Emphasis on the need for a designated CRS environmental compliance lead/coordinator reporting to the COP to develop required documentation and ensure pro-active follow-up for timely support and approvals. These elements would be of use to Amashiga partners, to CRS internationally and also for the final project evaluation. 2. For USAID/FFP/BEO: Issue internal guidance for specific deadlines/timeframes for completion of environmental documentation to all new awardees, as well as guidelines for the specificity of decisions and associated conditions. - If there are conditions attached to approvals, these must be specific to the project and describe comprehensively requirements and expectations for next steps by the awardee i.e., the implications of the condition must be easily inferable. - Modifications to output or activity wording in the project LogFrame should be carried through to IEE documentation to facilitate tracking and project management. - USAID/FFP should ensure that there is a process between the BEO and the FFP in-country focal points to ensure that decisions are adequately specific and responsive to the approval request and integrate local implementation or other cooperative agreement planning considerations. To that end, the MTE team suggests that USAID/FFP/BEO undertake an internal review of these processes with the objective of streamlining the process and adopting a holistic approach to environmental and agreement compliance requirements. 69 Despite requests, the Amashiga team did not provide to the MTE team specific references to the CA provisions understood to oblige them to obtain additional approvals before proceeding with implementation and so we were unable to establish from where this understanding emanates. In addition, it is unclear how activities impacted by conditions issued in August 2016 could continue to be on hold into FY18. 82 Amashiga MTE - FINAL 5.2 Summary Review of Evaluation Main Objectives and Key Questions This section presents an overall review of the evaluation’s key elements, and these are organized according to assessment of i) the evaluation’s five key questions to address; and ii) the three objectives of the evaluation. Review of Evaluation Questions The table below shows the evaluation’s five key questions and a summary of conclusions for each. Table 12: Conclusions Across MTE’s Five Main Questions Objective 1 1. How well have the project’s interventions met planned schedules, beneficiary numbers, and outputs? What factors promoted or inhibited adherence to schedules? How were problems and challenges managed? i) Schedules Amashiga has been dominated by numerous delays in project implementation, as discussed in Section 2, within the findings for each sector section in Section 4; and under Environmental Compliance in Section 5.1 above. Three principal factors inhibited adherence to schedules: - Due to the volatile socio-political situation in Burundi, few field activities were carried out in Year 1. In fact, the Amashiga team was not fully present in Muyinga until Year 2. - New (to CRS Burundi) USAID environmental compliance requirements to identify and classify risks by output and to develop appropriate mitigation measures coupled with a tentative and conservative management approach by CRS in interpreting USAID BEO conditions. - Complying with the MoH’s requirements for using approved training models and in some cases MoH trainers delayed two activities: FAM and IYCF. This includes reinforcing capacities and implementing community-level activities for both activities. Finally, some activities started late into implementation with little real justification offered by the project team for the delays, e.g., Talking Books, Community Scoreboards. ii) Participant numbers and outputs Some of the targets for participant numbers may be reached; however, even if they are reached, many of the activities are being implemented with an eye to meeting quantitative targets rather than on producing high-quality, sustainable outcomes. Detailed findings for outputs are described in Sections 4 and Section 5.1. Seven IRs from the LogFrame have no associated outputs; five outputs were classified as “deferred” by CRS in IEE submission and so could not proceed until an IEE amendment was submitted and approved by the USAID BEO. CRS did not submit this amendment until August 2017 (almost three full years after project commencement); this was approved in November 2017 (just prior to submission of this report). It is the MTE’s opinion that the response by the project to challenges including implementation delays due to the context and environment compliance conditions has been slow, passive and inadequate. Objective 1 continued 2. What are the strengths of and challenges to the overall project design, implementation, management, communication and collaboration so far? What factors appear to promote or challenge the project operations or effective collaboration and cooperation among the various stakeholders? i) Project design In general, the original project design is strong (aside from some elements of Purpose 3), evidence-based and focused on the interconnectedness of the different purposes and interventions. The strong emphasis 83 Amashiga MTE - FINAL on gender, using a proven approach (Ab’IR couples implementing The Faithful House module), is non￾threatening to men and appropriate to the context. The inconsistent inclusion of outputs in the LogFrame, however, hampers the analysis of change (see Section 5.1 above). Specific comments and suggestions to strengthen the design and the ToC are included in the Sector Conclusions and Recommendations sections 4.1-4.4 and in Section 5.1 Design & ToC. The main weakness, however, is that the design has not been adapted in response to the challenges faced during implementation (contextual challenges and donor-related delays). Rather than seeking to improve the design to meet the contextual challenges, the response of some project staff was to tell the MTE team: “That’s the way the project was designed”. This passivity and lack of critical thinking is an impediment to making the adaptations necessary to the project’s evolution. ii) Project implementation Challenges in the design and implementation include: i) if P2 does not succeed, P1 will not achieve its intended results; ii) the lack of resources for improving water and sanitation infrastructure means a felt need is not met and a critical component for nutrition programs is missing; and iii) not building in motivation and rewards for the CoProNuts from the beginning could negatively impact implementation and sustainability since they are seen as fundamental to long-term impact. Another implementation challenge is the sheer scale of the project: with 809 communities, many activities, especially for P2, cannot possibly reach even half the households. And not targeting the P1 households for P2 activities dilutes the overall potential for impact. There are some limited examples of responsiveness or adaptations by the project to changing contexts or the requests of stakeholders. These include i) the distribution of free hybrid corn seed for the 2017 C season for 42 ha in the lowlands; ii) the procurement and application of Orthene to effectively combat the new armyworm infestation; and iii) the heightened coordination and lobbying efforts deployed by CRS to secure a tax exemption for the project’s Title II foods. Overall, the MTE team found that the project’s focus (formal and in practice) is on achieving highly ambitious quantitative targets rather than on implementing quality activities. iii) Project management Further, despite very substantial delays in start dates, with some activities still not underway, there is no plan in place to review and modify plans/targets or to revise the project staffing or budget. Management’s approach to resolving outstanding questions regarding USAID/FFP requirements, especially with regard to environmental compliance, has been marked by extreme caution. When accompanied by the lack of resolve, creativity, and energy on the parts of the Amashiga consortium and USAID to find solution, this overly cautious approach (illustrated by management’s narrow interpretation of documentation and limited allocation of resources to the problem) has directly impeded or prevented altogether the timely delivery of services to participants and the monitoring and evaluation of project activities. Project delays seem generally to be accepted by all parties and there is little apparent consideration of the consequences for participants. It is surprising to the MTE team that this inactivity on the part of the consortium has not been challenged to a greater extent by USAID/FFP focal points. The MTE team refers in particular to three issues: i) the management of the environmental compliance requirements; ii) the development of the MEAL plan; and iii) the finalization of the project ToC. In summary, management’s concern with compliance to USAID/FFP requirements and apparent unwillingness to drive things more seem more important than accountability to project participants for actual service delivery. iv) Communication and collaboration Not only is there no long-term detailed updated project plan or strategy for project management 84 Amashiga MTE - FINAL purposes but the MTE team also found that local authorities and communities have little idea of what the project expects to do during the coming quarter, much less the coming year or during the remainder of the project. Partners and communities know the project is ending in 2019 but they do not know what activities are starting or stopping when. Objective 1 continued 3. In each technical sector, what are the strengths of and challenges to the efficiency of interventions’ implementation and their acceptance in the target communities? How well do implementation processes adhere to underlying principles and project protocols? What factors in the implementation and context are associated with greater/lesser efficiency in producing outputs of higher/lower quality? Which interventions and implementation processes are more or less understood by and acceptable to members of the target communities and why? P1: Implementation is particularly efficient around efforts to increase women’s visits to health centers, attributable to the strong encouragement women receive from multiple sources to attend four antenatal consultations and at least one postnatal consultation and to ensure that their children adhere to the immunization calendar. Additionally, the P1 team’s numerous assessments and studies to gather information before implementing major interventions have helped identify best practices that have worked and possible challenges that may need to be addressed. These efforts, which in a number of cases have helped to ensure a smooth, efficient rollout of new activities, include: a FAM study; a media assessment, especially for radio; a Positive Deviance Study; an assessment of gender and SBCC in the health centers; and a number of barrier analyses. The most widely-understood and appreciated intervention is the distribution of MCHN rations, intended primarily for PLW and children 6-23 months. Providing nutritious food supplements to the most vulnerable family members meets a strongly-felt need. Also appreciated is the focus on i) affordable ways to safeguard the family’s health (e.g., using free services at the health facility and taking precautions such as hand washing) and ii) how to improve the physical environment at the household and community level (affordable hygiene and sanitation improvements such as compost pits and drying racks). Finally, the credibility of the CoProNuts (headed by the CHW and a local administrator) and their engagement helps to ensure that participants understand the value of and are receptive to project interventions. Interventions are suitable to the needs of the participants and the local context; harmonized with current GoB priorities, policies and protocols; and in line with the project’s stated emphasis on i) improving gender relations; ii) involving men more fully in household decision-making around health: and iii) responding to the MoH’s requests for assistance with capacity-building of health care providers and health promotion through intensified SBCC activities. Challenges to effective implementation and ensuring sustainability include the lack of resources for participants to apply what they have learned (e.g., access to more nutritious foods and the means to improve water points and latrines) and the ambitious scale of the project (covering a whole province, working with 809 communities, supporting more than 13,000 CoProNut members and strengthening 3,300+ Water Associations.) This emphasis on quantity, rather than quality, prevents adequate attention to the steps needed to ensure interventions are sustainably implemented in a high-quality fashion. Another challenge to effective implementation is the fact that while many MCHN providers at health facilities have been trained on the MoH protocols supporting the growth of children, efficient (and consistent) integration of these protocols is limited by the understaffed nature of health centers, wherein overburdened nurses often have insufficient time to follow full protocols with patients. 85 Amashiga MTE - FINAL P2: There is very good participation of the DPAE AEAs, CoProNut members (P1) and Ab’IRs (P3) in most P2 activities and they provide the main level of inter-component integration. By the beginning of FY4, there was a recently signed cooperation agreement between the project and the DPAE, the key government partner. While SILC activities are widely appreciated and progressing well, the repeated delays in agricultural activities have reduced both the credibility of the project and its impact on improving food production, with late seed distribution leading to lower harvest production. The goat solidarity chain will start two years late but is eagerly awaited and well-known process; and it is now too late to start the livelihood enhancement activity. As the agricultural calendar imposes its own limits on project outcomes, the project should create and act on alternate plans for better outcomes for project participants. Additionally, while some progress has been made towards P2 activities – including increasing production and access to markets for nurseries and SMGs – there is a lack of understanding among participants and project staff alike regarding the process of certified seed multiplication and broadening market access from technical, management and economic standpoints. Increasing knowledge around these processes – and the steps requisite to accomplish project benchmarks – would ensure a more sustainable process for high-quality results. Furthermore, participant selection procedures – particularly for IR2.2.3. – have shown a lack of adherence to project protocols. While criteria of FFA participant selection were well understood by interviewees, new participant substitutions often did not adhere to these criteria, which has diminished the project’s credibility. P3: The strengthening of the colline- and commune-level development planning and reporting processes was widely understood and appreciated by local committees and elected officials. Similarly, a firm understanding of the roles and responsibilities of various government and civil society actors demonstrates a strong foundation for the purpose. However, there is a concerning possibility that project staff, let alone project participants, may not have a thorough grasp of key concepts and how local actors will fulfill the related functions (e.g., disaster risk reduction, contingency plans, early warning systems) to be able to move beyond relabeling existing committee structures to enact meaningful change. The other activities (e.g. local determinants analysis workshop, the EWS committees, the social audits using the scorecard methodology) have only started in the last 4-6 months or have not started at all (e.g. reinforcing capacity among commune entities on disaster risk reduction) so it is difficult to speak to effectiveness. There are significant concerns about the sustainability of some activities at the commune level, given the fact that the commune is unlikely to be able to support any transport fees for participants to facilitate activities which the project is currently supporting. XSP: The work of Ab’IR couples is well understood by the couples themselves, by those who have benefited from their home visits, by local leaders in the collines (and sous-collines), and by participants of other project components. The Ab’IR model is simple, well-executed, and appears well-suited to the Burundi context. While the project has successfully identified Catholic Ab’IR to date, it worked with Catholic structures to identify them, resulting in a few couples from other faiths. The effects are likely to be mitigated, particularly in the final survey, given that the majority of the province is Catholic. After the distribution of MCHN rations, it is probably the most appreciated intervention. Commodities: Overall, the management of the food commodities meets FFP/USAID’s requirements. The foods staff is knowledgeable about the regulations as illustrated by the lack of pending issues from FFP related to the quarterly Commodity Status Reports (CSR), the Recipients Status Reports (RSR) and the Loss Status Report (LSR). Some aspects of project implementation could better align with 86 Amashiga MTE - FINAL underlying project goals; for example, hygiene/sanitation at nutrition fair sites could be improved by not discarding empty bags and debris around the site and encouraging the washing of hands after use of existing latrines at each site. In terms of collaboration and synergy, the MTE team identified one example of synergy between Amashiga and another USAID-funded nutrition and health initiative in Muyinga Province (see Section 4.1) as well as some cross-purpose internal collaboration, notably between P1 and P2 and P2 and P3 (described in detail in section 4). The MTE team has also provided a number of recommendations to further strengthen integration across purposes. Objective 2 4. What changes do community members and other stakeholders associate with the project’s interventions? What factors appear to promote and deter the changes? How do the changes correspond to those hypothesized by the project’s ToC? Participants, community leaders and MoH stakeholders associate a number of positive improvements with the project and gave the following examples: - Improved maternal and child health: children are healthier; birth weights are higher; very few children are in the red zone during nutrition screening; women have easier pregnancies with fewer complications. - More women are going to ANC and delivering at the health center; parents take children to the health center at the first sign of illness rather than waiting; children are up to date on their vaccinations (corroborated by the MTE team in several communities). - Participants are more aware of the importance of hygiene and sanitation to good health and are making simple improvements at the household level: evacuating used water, digging compost pits, constructing drying racks and, to some extent, improving latrines. - There is a good understanding at the community level that production must be increased so that households can transition to having a balanced diet after commodity distribution is phased out. - Commodity distributions are generally well-managed and appreciated by participants and local authorities; the American origin of the food and USG support to the overall project is also widely known. Across all sectors, participants stated that they are more aware of the importance of harmony at the household level, mutual support between husband and wife and joint decision-making. They also described how improved gender relations can have a positive impact on family well-being, especially children’s health and increasing household production and revenue, and gave personal testimonials and other examples of how this knowledge has transformed families. On the other hand, practical application of key messages is less feasible, e.g. having a balanced diet without access to local foods or frequent hand washing when water availability is an issue. Further, there is little understanding among participants, local stakeholders or the Amashiga team as to what the project expects to happen to the project-supported structures and interventions after the current award ends. Overall, levels of activity have been too low and have started too recently to clearly attribute concrete change for or by participants to Amashiga. 87 Amashiga MTE - FINAL Objective 3 5. Based on the findings from the previous four questions, how could the project be modified to improve its acceptability to targeted communities or the efficiency and effectiveness of its implementation? How should the project’s ToC or results framework be refined or modified? The MTE team has provided detailed recommendations by sector and a number of overall recommendations in Section 5.2 below too. In summary, the MTE team recommends that the Amashiga consortium reviews its targets and workplans with a view to cutting targets and certain activities that have either not yet started or cannot have the desired impact in the remaining time period. This requires active involvement from and liaison with USAID-FFP to ensure timely discussions and approval of revisions. To improve effectiveness and efficiency of implementation Amashiga must: - Scale back targets and increase the number of field agents. - Improve the effectiveness, reach and scope of the SBCC activities. - Move from dissemination of information to providing the means for people to adopt new practices and change behaviors. - Communicate (in advance and on an ongoing basis) the project’s plans to local stakeholders and participants. - Integrate capacity building of local service providers into project design (in particular for agriculture and governance). Assessment of Amashiga Midterm Evaluation Objectives The MTE team has drawn conclusions related to the three MTE objectives: Objective 1: To evaluate the strengths and weaknesses of the implementation of project implementation and the quality of outputs, as regards the adherence to terms agreed by FFP and of their acceptability and perceived value to target communities, identifying factors that appear to enhance or detract from the quality, acceptability, and usefulness of implementation and outputs. The Amashiga Project started with an ambitious but very strong initial Theory of Change and design. Just as the project was starting, Burundi suffered major and widespread violence, affecting access to Muyinga Province, which understandably delayed the project start. On top of this, there seems to have been almost a perfect storm (with good intentions) of new USAID/FFP policies and guidelines that the project was expected to use, particularly with regard to the M&E Plan and environmental compliance. The high level of activity around these requirements masked the consequences of the delays; all of this may have inadvertently reinforced a passive atmosphere among the project team, providing a ready excuse for the delays and inaction. In addition, the turnover in CoPs and in staff at USAID/FFP has further masked the consequences, or has slowed any reaction to, the cumulated delays. The primary job of the MTE is to assess project progress relative to the proposed design and the intended impacts. The MTE team cannot soften the disappointing conclusions presented in this report as this would risk diminishing the urgency for USAID/FFP and CRS to work together quickly and with resolve (which has not happened to date) to agree on a revised implementation plan. The MTE team found very few examples where the project adapted to delays or sought to find alternate ways to implement. Given the relatively short period of time remaining, it is vital that manageme nt seize this opportunity to urgently review and plan the remaining quarters in detail. Ultimately, it is the project participants who suffer the consequences of the delays to implementation. The hope of the MTE team is that the Amashiga partners (including USAID/FFP) will work with new resolve to prioritize above all else the delivery of project results to communities in the time remaining. 88 Amashiga MTE - FINAL Objective 2: To present evidence of changes (intended and unintended) associated with project interventions and outputs, assess how well the observed changes reflect the Theory of Change (ToC), and identify factors in the implementation or context that impede or promote the observed and intended changes. Overall, levels of activity have been too low and have started too recently to clearly attribute concrete change for or by participants to Amashiga. See MTE team response to Question 4 in previous section for conclusions on some challenges to effecting change and a brief description of some promising starts. Objective 3: To recommend adjustments to the ToC, project design, resource allocation, project management, monitoring and evaluation (M&E) plan, or implementation that could improve the likelihood of achieving desired results by the project’s end – based on the evidence collected and conclusions drawn for the evaluation objectives above. Detailed recommendations pertaining to the ToC, project design and activities are provided in Section 4 and under Question 5 in the previous section. The overarching recommendation by the MTE team is that the Amashiga consortium reviews its targets and workplans, using three guiding principles: i) Emphasis on quality over quantity of activities; ii) Each activity has a plan by project quarter to the end of the project; each component has an exit- and long-term strategy; and iii) Communications to local stakeholders, including participants, are a priority. Timely and regular communications should clearly describe the project’s plans and the specific support to be provided. 5.3 Overall Recommendations This final section presents the MTE team’s overall recommendations for the remainder of the project, its final evaluation and for future programming. 1. Revise activity planning to determine realistic targets, focusing on quality of outputs and outcomes rather than on quantity of activities. It is important for the consortium and USAID/FFP to acknowledge that the substantial delays in implementation mean that it is not feasible to achieve the outstanding targets by the end of the project. It is vital to determine now what is realistic while working to improve the quality of interventions to maximize impact and enhance the potential for sustainability. i) Develop or update strategies for each purpose and key activity area with quarterly milestones until the end of the project. - Finalize revised purpose strategies and activity milestones/quarter by end of Q2 FY2018 - Anticipate potential delays; set absolute deadlines and establish back-up plans. - Communicate revised activity start and end dates, expectations for involvement and anticipated support to communities and partners; provide regular updates until project ends. ii) Reduce volume and increase quality of activities. - Nominate specific purpose and MEAL staff as quality focal points. - Document quality policies with detailed definitions. - Ensure management devotes time to redirecting focus of implementation from doing a lot of things to doing fewer things better. - Organize a workshop to include management, implementation and MEAL staff to assess and take decisions for ensuring quality quantitative data, formal learning agendas and knowledge sharing processes in line with CRS best practice. 89 Amashiga MTE - FINAL 2. Develop a project-wide sustainability plan with specific sustainability measures for all activities. i) Reduce free inputs. - Phase out provision of free seed, transport fees and per diem for participants. ii) Support local actors to conduct activities rather than having them participate merely as attendees so that they can take on appropriate responsibilities. This includes: - Working directly with the DPAE and other local stakeholders involved in the certified seed system to plan remaining activities in the project to ensure ownership and increase likelihood for sustainability. See also Recommendations for P3. - Developing plans for the ongoing maintenance of the Food-for-Assets (FFA) outputs. 3. Strengthen the integration of the project purposes by identifying points of intersection between sectors. For example, assess the feasibility of joint activities: - P1 and P3 teams could work together on social audits of health services and on Community Scoreboards; - P1 and P3 teams could collaborate on reinforcing the Water Associations (establishing selection criteria for which associations to work with and jointly conducting training with government counterparts); - CoProNut members could be targeted for P2 activities; and - SILC groups could be targeted for messages from P1, P3 and XP1. 4. Strengthen the SBCC component to enhance the potential for sustained behavior change and lasting adoption of improved practices across all components. - Devote adequate human and financial resources to SBCC. - Review the current SBCC activities and available materials for each activity or group of activities and determine how to improve the effectiveness and reach of SBCC interventions to achieve the desired outcomes. - Train field agents and volunteers such as CoProNuts in effective communication and facilitation techniques. Include the basic principles for effecting behavior change and ensure that training includes demonstrations and practice. - Prepare additional didactic materials that can serve as reference guides, aides-memoire and references for people to use now and once Amashiga ends. These materials should be simple, visual and durable. 5. [for USAID/FFP and Amashiga] Develop a proactive approach to donor relations, clarifying compliance requirements and obtaining approvals. - Track outstanding approvals or questions between Amashiga and USAID/FFP and both parties should follow up in a proactive manner. Ensure that multiple requests are clearly separated in communications and that each request is tracked until resolved; - Maintain close communication and proactive collaboration with clear decision-makers for both Amashiga and USAID/FFP to ensure timely approval of modifications to targets and associated modification to the Theory of Change and the M&E Plan within a predetermined fixed timeframe; - When seeking clarification regarding issues such as environmental compliance or other regulations, send questions and refutations to the donor no later than the quarter following the date of receipt of approvals/conditions. Document delays and the implications of these delays for participants and activity targets in the quarterly narrative reports. This includes (as noted in Recommendation 1) communicating to the donor absolute project deadlines beyond which commencement of a given activity will not be feasible. 90 Amashiga MTE - FINAL 6. Prepare for the Final Evaluation of Amashiga The final quantitative evaluation results should be available to the final qualitative evaluation team at least two weeks before qualitative field work begins to better inform the qualitative evaluation. Provide the following documents to the evaluation team: - Personnel lists by sector including name, post, phone number and/or e-mail, date started with project - Job description, especially for field agents - Annual surveys and data quality surveys - A spreadsheet showing which activities are at each site, including the start day for the activity in that site P1: Provide a list of all training participants from the government including their position (and facility) and the specific trainings received to the evaluation team. P2: Include a senior member of the DPAE in the evaluation of P2 for accountability and for joint learning. P3: Provide copies of the plans and reports of EWS Committees (if they continue), commune DRR Platforms (if they are initiated), Commune PAIs, scorecards, etc. to the evaluation team to permit a review of a sample of those documents to inform questions during qualitative field work. XSP: - Ensure that the final survey is specific in questions about SGBV and distinguishes domestic (or intimate partner) violence from other types of SGBV. - The project keeps track of “positive deviants” not of all couples that receive services from Ab’IRs; the survey should therefore include a mechanism for gauging whether or not respondents received services and their appreciation of those services. SBCC: Provide a complete set of SBCC materials for each purpose to the evaluation team. Environmental compliance: share documentation with the evaluation team on lessons learned by Amashiga on the IEE and PERSUAP processes.