Empowering New Generations to Improve Nutrition and Economic Opportunities (ENGINE), Ethiopia EXTERNAL MID-TERM PERFORMANCE EVALUATION REPORT September 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Tufts University. The authors of the report are James Levinson, Berhanu Admassu, Tanya Trevors and Amdissa Teshome. Agriculture Knowledge, Learning, Documentation and Policy Project (AKLDP), Ethiopia ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report Empowering New Generations to Improve Nutrition and Economic Opportunities (ENGINE), Ethiopia EXTERNAL MID-TERM PERFORMANCE EVALUATION REPORT USAID Contract number: 663-13-000006 Agriculture Knowledge, Learning, Documentation and Policy (AKLDP) Project Implemented by: Feinstein International Center Friedman School of Nutrition Science and Policy Tufts University Africa Regional Office PO Box 1078 Addis Ababa Ethiopia Tel: +251 (0)11 618014 www.fic.tufts.edu Disclaimer The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report CONTENTS List of Acronyms............................................................................................................................................................ i Acknowledgements ..................................................................................................................................................... ii Executive Summary .......................................................................................................................................................i 1. Introduction...........................................................................................................................................................1 1.1 Background to ENGINE and the evaluation ....................................................................................... 1 1.2 Project objective and intermediate results ........................................................................................ 1 1.3 Theory of change ........................................................................................................................................... 2 1.4 Project management, coverage and target groups ......................................................................... 3 1.5 Evaluation questions ................................................................................................................................... 3 1.6 Evaluation design .......................................................................................................................................... 3 1.7 Report limitations ......................................................................................................................................... 4 2. Evaluation Findings.............................................................................................................................................4 2.1 Evaluation Question 1: Performance against objectives .............................................................. 4 2.2 Evaluation Question 2: Impact of actions with government and USAID‐assisted partners ............................................................................................................................................................................ 9 2.2.1 IR1: Capacity and institutionalization of nutrition programs and policy strengthened ............................................................................................................................................................. 9 2.2.2 IR2: Quality and delivery of nutrition and health services improved ............................ 11 2.2.3 IR3.1: Prevention of under‐nutrition through community‐based nutrition ............... 12 2.3.4 IR3.2: Access to food and economic strengthening opportunities .................................. 14 2.3 Evaluation question 3: Contributions of FTF partners in reducing under‐nutrition ... 16 2.4 Evaluation question 4: Contributions to gender equity and female empowerment .... 18 2.5 Evaluation question 5: Balance of resources against activity priorities ............................ 20 2.6 Evaluation question 6: Management effectiveness and efficiency ....................................... 21 2.7 A critical question: Is ENGINE likely to achieve its Stunting Reduction Target? ............ 22 3. Summary and Conclusions.............................................................................................................................23 3.1 Evaluation question 1: Performance against planned objectives ......................................... 23 3.2 Evaluation question 2: Impact of actions with government and USAID‐assisted partners ......................................................................................................................................................................... 24 3.3 Evaluation question 3: Contributions of FTF partners in reducing under‐nutrition ... 24 3.4 Evaluation question 4: Contributions to gender equity and female empowerment .... 24 3.5 Evaluation question 5: Balance of resources against activities and priorities ................ 24 3.6 Evaluation question 6: Management effectiveness and efficiency ....................................... 25 4. Recommendations and Next Steps..............................................................................................................25 4.1 Evaluation question 1: Performance against objectives ........................................................... 26 4.2 Evaluation question 2: Impact of actions with government and USAID‐assisted partners ......................................................................................................................................................................... 26 4.2.1 IR1 Capacity and institutionalization of nutrition programs and policy strengthened 26 4.2.2 IR2: Quality and delivery of nutrition and health services improved ............................ 26 4.2.3 IR3: Prevention of under‐nutrition through community‐based nutrition ................... 26 4.3 Evaluation question 3: Contributions of FTF partners in reducing under‐nutrition ... 27 4.4 Evaluation question 4: Contributions to gender equity and female empowerment .... 28 4.5 Evaluation question 5: Balance of resources against activities and priorities ................ 28 4.6 Evaluation question 6: Management effectiveness and efficiency ....................................... 28 4.7 Additional recommendations ............................................................................................................... 29 ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report List of Tables Table 1: Progress and predicted achievement under IR1 Capacity for and institutionalization of nutrition programs and policies strengthened .................................................................................................................... 5 Table 2: Progress and predicted achievement under IR 2 Quality and delivery of nutrition and health care services improved .................................................................................................................................................................. 6 Table 3: Progress and predicted achievement under IR3 Prevention of under‐nutrition through community‐based nutrition care practices improved ...................................................................................................... 8 Table 4: Comparison of ENGINE livestock assistance activities in SNNPR ........................................................... 16 Table 5: FTF project nutrition activities ............................................................................................................................. 16 Table 6: Effect of ENGINE on Women's Decision Making (Amhara Region, small sample) ........................... 19 Annexes Based on the reporting and submission requirements detailed in the evaluation scope of work, a set of annexes are available as separate documents. The list of annexes is as follows: Annex 1 Evaluation Statement of Work Annex 2 Places visited; list of organizations and people interviewed, including contact details. Annex 3 Evaluation design and methodology. Annex 4 Data collection checklists Annex 5 Bibliography of critical background documents Annex 6 Meeting notes of all key meetings with stakeholders Annex 7 Statement of Differences Annex 8 Evaluation Team CV’s Annex 9 Disclosure of Conflict of interest Annex 10 Observations in a primary school in eastern Oromia Annex 11 Program Constraints Assessment Annex 12 Food insecurity measurement Annex 13 Dietary diversity measurement ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report i | Page List of Acronyms ADD AGE AGP AKLDP AMDe ANC ATVET CAADP CC CCA CHD CIAFS CMAM DA DHS DRMFSS EBF ENGINE EOC FAO FGD FTC FTF GHI GMP GOE GRAD HCs HCWs HDA HEIs HEWs HH HIV HMIS HP IFAD IFHP IFPRI IR IYCF LMD LQAS MAIYCN MI MIYCN MNCH MNP MOA MOE MOH MUAC MVH Addressing Dietary Diversity – Assess, Grow and Eat Agricultural Growth Program Agriculture Knowledge, Learning, Documentation and Policy Project, USAID Agribusiness Market Development Project, USAID Antenatal Care Agriculture Technical and Vocational Education Training Comprehensive Africa Agriculture Development Program Community Conversation Community Conversation Agents Community Health Days Capacity to Improve Agriculture and Food Security Project, USAID Community Management of Acute Malnutrition Development Agent Demographic and Health Survey Disaster Risk Management and Food Security Sector Exclusive Breast Feeding Empowering New Generations to Improve Nutrition and Economic Opportunities Project, USAID Ethiopian Orthodox Church Food and Agriculture Organization Focus Group Discussion Farmer Training Center Feed the Future Program Global Health Initiative Growth Monitoring and Promotion Government of Ethiopia Graduation with Resilience to Achieve Sustainable Development Project, USAID Health Centers Health Center Workers Health Development Army Higher Education Institutions Health Extension Workers Household Human Immunodeficiency Virus Health Monitoring Information System Health Post International Fund for Agricultural Development Integrated Family Health Program, USAID International Food Policy Research Institute Intermediate Result Infant and Young Child Feeding Livestock Market Development Project, USAID Lot Quality Assurance Sampling Maternal, Adolescent, Infant and Young Child Nutrition Micronutrient Initiative Maternal Infant and Young Child Nutrition Maternal Newborn Child Health Multi‐micronutrient powders Ministry of Agriculture and Rural Development Ministry of Education Ministry of Health Mid Upper Arm Circumference Most vulnerable household ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report ii | Page NCB NGO NNP OFDA OPDO ORDA OR ORS PATH PCA PEPFAR PHCU PMP PMTCT PRIME QI RFA PSNP SAM SBCC SNNPR SOW TIPS U3 UN UNICEF USG USAID VAS WASH WFP WHO WRA National Coordination Body Non Governmental Organization National Nutrition Program Office of Foreign Disaster Assistance Oromo Peoples Democratic Organization Organization for Rehabilitation and Development in Amhara Region Operations Research Oral Rehydration Solution Partnership for Appropriate Technology in Health Program Constraints Assessment President's Emergency Plan for AIDS Relief Primary Health Care Unit Performance Management Plan Prevention of Mother to Child Transmission Pastoralist Areas Resilience Improvement and Market Expansion Quality Improvement Request for Application Productive Safety Net Program Severe Acute Malnutrition Social and Behavioral Change Communication Southern Nations Nationalities and Peoples Region Statement of Work Trials of Improved Practices Under 3 years of age United Nations United Nations Children’s Fund United States Government United States Agency for International Development Vitamin A Supplementation Water, Sanitation and Hygiene World Food Program World Health Organization Women of Reproductive Age Note In Ethiopia, a woreda is an administrative unit, roughly equivalent to a district in other countries. A kebele is equivalent to a sub‐district, being the next level of administrative unit within a woreda. Acknowledgements The evaluation team would like to express its gratitude and sincere appreciation to the following:  The AKLDP staff and particularly Adrian Cullis, Andrew Catley, Fasil Yemane and Yemiserach Woldearegay,  Mary Harvey and her USAID colleagues,  The Feed the Future and Global Health Initiative partners,  Dr. Fekadu Habtamu and his ENGINE team, both those at ENGINE headquarters and those who provided assistance to us all at the regional, zonal, and woreda levels,  Officials of the Government of Ethiopia at all levels and from all sectors, and  The ENGINE beneficiaries for so generously providing the team with invaluable information and insights. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report i | Page EXECUTIVE SUMMARY This external mid‐term performance evaluation of the USAID/Ethiopia‐assisted Empowering New Generations to Improve Nutrition and Economic Opportunities (ENGINE) project was carried out in June‐July 2014 following the Statement of Work specifications and six specific evaluation questions provided by USAID. ENGINE is a five‐year integrated nutrition project launched in September 2011 that aims to improve the nutritional status of women and young children through sustainable, comprehensive, coordinated and evidence‐based interventions. The project was designed to strengthen the Ethiopian Government’s nutrition policy and programmatic efforts to implement its new National Nutrition Program (NNP) and to deliver on the nutrition objectives of the U.S. Global Health (GHI) and Feed the Future (FTF) initiatives in four regions of the country. ENGINE’s Results Framework includes four intermediate results (IR) areas, each accompanied by a set of strategic approaches: 1. IR1: Capacity for and institutionalization of nutrition programs and policies strengthened; 2. IR2: Quality and delivery of nutrition and health care services improved; 3. IR3: Prevention of under‐nutrition through community based nutrition care practices improved; 4. IR4: Rigorous and innovative learning agenda adopted. The evaluation covers only IRs 1‐3, based on the evaluation questions set by USAID. The evaluation team visited three of the four regions of the country where ENGINE is operating and interviewed a large cross section of government officials, ENGINE staff, external partners and beneficiaries. The evaluation recommendations should be viewed in the above context, and most recommendations are well within ENGINE’s ability to implement. Key findings and conclusions The evaluation report presents 29 specific findings, structured around the six evaluation questions, and detailed in section 2 of the report. The general conclusions of the evaluation, across IRs 1 to 3, are as follows:  ENGINE is performing well and is on track to achieve most of its performance targets by the end of the project.  The project is having major impacts at the policy and institutional levels, and on strengthening national, zonal and woreda‐level systems to scale up nutrition services across sectors.  USAID Ethiopia FTF projects have taken important strides to incorporate nutrition education and training into their programs and activities. But FTF projects should strengthen and re‐ orient their monitoring and evaluation efforts to measure explicitly the impact of their efforts on nutritional status and on food security with particular attention to dietary diversity. Outstanding challenges are mostly well‐understood by ENGINE, with strategies and plans in place or being developed accordingly. Key issues affecting future project impacts are:  To continue to address aggressively the primary determinants of stunting in Ethiopia, prioritizing and reshaping activities around critical areas such as WASH, food security and dietary diversity, while also continuing to support multi‐sectoral nutrition convergence across programs and actors.  In key activity areas such as training, information dissemination, demonstration and livelihoods support, there is need for a clearer understanding of whether, and if so, how specific activities translate into nutrition and food security impacts among target groups, and how these activities can be scaled up. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report ii | Page  Long‐term, the current momentum around nutrition in Ethiopia now requires further intensified efforts by ENGINE – and by USAID and other donors – to normalize nutrition‐related activity within the government, with less reliance on external assistance. ENGINE is well‐placed to work with GoE partners to integrate nutrition activities into government plans and budgets. Overall, ENGINE is making important contributions to nutrition policy and programming in Ethiopia ‐ which also are likely to contribute to wider learning around nutrition in other countries and regions. However, as indicated above, there are key areas of ENGINE activity where more needs to be done to measure and document the nutritional and food security impacts of its activities, as well as those of other FTF projects. Key recommendations The evaluation produced 40 specific recommendations, as detailed in section 4 of the report. The central recommendations are presented below:  ENGINE is on‐track to achieve most of the targets in its PMP, but specific issues with indicators noted in Tables 1 to 3 of the main report should be addressed on a priority basis by ENGINE management.  The ENGINE child stunting reduction objective is unrealistic and should be changed ‐ a target of 38% stunting prevalence might be reasonable.  ENGINE should continue to concentrate on maintaining Ethiopia’s rate of stunting reduction – and seek to prevent plateauing by intensively addressing the 3 potential limiting factors (WASH, maternal malnutrition and poverty/food insecurity) while continuing to focus on the “first 1000 days, and by introducing and emphasizing genuine multi‐sectoral nutrition convergence (interventions from each sector in every village of coverage areas) with diligent M&E, and impact measurement.  Testing of multi‐sectoral nutrition convergence should be initiated, both in existing AGP kebeles where FTF projects are co‐located, and in ENGINE/GRAD efforts in several PSNP woredas, and with results presented for consideration in the next NNP and FTF II.  Using food security and dietary diversity data collected annually on all livelihood component beneficiary households, USAID should leverage the GoE and other donors to participate in this enormously valuable undertaking.  ENGINE should invest strongly from a technical perspective in getting the SBCC program “right” and then go for scale‐up via the government using woreda and kebele government staff and the HDA as well as local NGOs.  Iron folate stockouts in the country need to be addressed as a top priority.  Utilizing the convergence approach where possible, FTF projects should seek to better align their nutrition counseling and production/value chain support to beneficiary farmers so that (a) nutrition messages are reinforced and (b) households benefit simultaneously from poverty reduction and dietary diversity/nutrition interventions. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 1 | Page 1. INTRODUCTION 1.1 Background to ENGINE and the evaluation This external mid‐term performance evaluation of the USAID/Ethiopia‐assisted Empowering New Generations to Improve Nutrition and Economic Opportunities (ENGINE) project was carried out in June‐July 2014 following the Statement of Work (SOW) specifications provided by USAID. The purpose of the evaluation was to examine what the ENGINE Project has achieved at the mid‐way point in its implementation; how it is being implemented; how it is perceived and valued; whether expected results are occurring or are likely to occur before the end of the project; and to assess the management and operation of the project. The findings, analysis and recommendations are intended to inform and improve implementation of ENGINE in the remainder of the Project’s life. The ENGINE Project is a cooperative agreement (663‐A‐00‐11‐00017) that began on September 27, 2011 and will end on September 26, 2016. The Life of Project Budget is $50,891,422. The ENGINE project emerged from a growing recognition within USAID of the important contribution of improved nutrition to national development, and the critical importance of assuring adequate nutritional status during “the first 1000 days” of life to minimize malnutrition’s negative and irreversible effects on physical and cognitive development with major implications, in turn, for active learning capacity, productivity and economic growth. The ENGINE Project was designed to strengthen the Government of Ethiopia’s nutrition policy and programmatic efforts to more comprehensively address the multi‐sectoral, underlying causes of maternal and child under‐nutrition in four regions of the country. ENGINE was designed to support the National Nutrition Program (NNP), both in its content and in its implementation. ENGINE fits within the nutrition objectives of the U.S. Global Health (GHI) and Feed the Future (FTF) initiatives, and is guided by principles of multi‐sectoral engagement, ensuring country ownership, partnerships and innovation. The project is also well positioned to deliver on USAID’s new global Nutrition Strategy (2014‐2025) and, globally, it is one of USAID’s largest nutrition investments, alongside Tanzania, Uganda and Nepal for FY 2013.1 1.2 Project objective and intermediate results As required within its cooperative agreement, ENGINE has a comprehensive Performance Management Plan (PMP) and Results Framework. The ENGINE strategic objective is “Improved nutritional status of women and young children through sustainable, comprehensive, coordinated, and evidence‐ based interventions”. Under this strategic objectives are four IRs, with related strategies as follows: IR1: Capacity for and institutionalization of nutrition programs and policies strengthened IR 1.1 Nutrition policy environment strengthened ‐ Strengthening existing coordinating mechanisms ‐ Supporting development and revision of policies, guidelines, and standards IR 1.2 Pre‐service and in‐service nutrition training for health care agents strengthened ‐ Pre‐service education ‐ In‐service capacity building ‐ Strengthening learning networks IR2 Quality and delivery of nutrition and health care services improved IR 2.1 Quality of nutrition services strengthened ‐ Facilitating integration of quality improvement processes ‐ Building capacity of health facility staff and frontline workers to provide high‐quality services IR 2.2 Enhanced health and nutrition service seeking behaviors ‐ Developing SBCC strategy ‐ Mobilizing communities ‐ Educating on micronutrients 1 Email correspondence with Newal Sheriff, USAID Washington. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 2 | Page IR2.3 Access to health and nutrition services increased ‐ Linking with other partners ‐ Strengthening referral system ‐ Establishing sites to treat acute malnutrition ‐ Supporting improvements in supply and logistics IR3 Prevention of under‐nutrition through community‐based nutrition care practices improved IR3.1 Maternal, IYCF knowledge and practices improved ‐ Developing SBCC strategy ‐ Promoting optimal nutrition practices through different communication channels IR 3.2 Access to food and economic strengthening opportunities through programming and cross‐sectoral linkages increased ‐ Applying economic strengthening interventions to meet households’ needs ‐ Facilitating community‐based learning on agriculture techniques for increased production of diverse foods IR 4 Rigorous and innovative learning agenda adopted Activity 4.1: Design and Delivery of Research Strategy: Research relevant to IR1: ‐ Strengthening the nutrition and HIV nutrition policy environment ‐ Research relevant to IR2 ‐ Improving the quality of nutrition services in Ethiopia ‐ Improving the effectiveness (coverage and efficacy) of programs to address acute malnutrition ‐ Improving coverage and quality of preventative nutrition and related services for mothers and young children Research relevant to IR3: ‐ Improving the effectiveness (coverage and efficacy) of interventions that aim to prevent under‐nutrition and micronutrient deficiencies ‐ Improving and strengthening explicit nutrition outcomes in multi‐sectoral programs Activity 4.2: Develop and Manage an Innovative Documentation and Dissemination Strategy This performance evaluation focuses uniquely on ENGINE’s IRs 1‐3 as IR4 was evaluated separately. 1.3 Theory of change The theories of change embedded in the project are multiple. Building on improved government policies, enhanced capacity building, and coordination (both within FTF and with external partners), ENGINE seeks to:  Increase agriculture production diversity in ENGINE’s target regions, improve the incomes of food insecure household plus those who are particularly vulnerable, help to translate that income, through counseling and demonstrations into improved food consumption with a particular focus on improved household food security and improved dietary diversity (taking advantage of improved production diversity), and, in turn, address a primary determinant of nutritional status;  Incorporate nutrition into all facets of health care delivery, improving caring and self‐care practices, hygiene and sanitation and reducing the prevalence of infectious disease, another primary determinant of nutritional status;  Facilitate women’s empowerment and household decision making, resulting in improved household food consumption and health care, and, in turn, nutritional status. IR1 focuses on the capacity development supporting each of the above‐mentioned change mechanisms. IR2 focuses on the health sector‐based interventions – bullet 2 above. IR 3 focuses both on behavioral change (SBCC) and on the agricultural sector – bullet 1 above. IR 4, ENGINE’s research agenda, supports each of the other IRs, but, as indicated, is not included in this evaluation. Bullet 3 ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 3 | Page above, women’s empowerment, is cross cutting, and was expected to be incorporated into each of the other IRs. The project has assumed that policy strengthening and capacity building activities would lead to improved field performance in the agricultural sector and in the health sector (at least down to the health center level.) The project also assumed that SBCC efforts, by improving knowledge, would regularly lead to behavioral change. Finally, the project assumed that pursuit of each of its IRs would result in sustainable stunting reduction. 1.4 Project management, coverage and target groups In terms of project management, Save the Children International (SCI) is the prime award recipient for ENGINE, and Land O’ Lakes, Jhiego and Tufts University are ENGINE’s current sub‐recipients. Valid International and the Manoff Group, together with three local NGOs, are contracted by SCI to manage the monitoring and evaluation, and SBCC components of ENGINE respectively. Other strategic partners for ENGINE include the Federal Ministries of Health (MoH) and Agriculture (MoA), USAID FTF partners, UNICEF, the World Bank, the World Food Programme (WFP), the USAID‐funded Integrated Family Health Program (IFHP), among others. ENGINE currently operates in 83 woredas included in Ethiopia’s Agricultural Growth Program (AGP) in four regions of the country, and will soon initiate activity in an additional 17 non‐AGP woredas – collaborating with the US Office of Foreign Disaster Assistance (OFDA)‐funded GOAL and the FTF GRAD projects, where the government’s Productive Safety Net Program (PSNP) is presently operating. The ultimate target groups of ENGINE’s programmatic efforts are 3.1 million children under the age of five, an equal number of reproductive‐age women, 500,000 pregnant and lactating women, and 2.7 million households. 1.5 Evaluation questions The evaluation addressed all six questions posed in the Statement of Work (Annex 1) as follows: 1. To what extent is the ENGINE Project progressing against planned objectives as embedded in its Performance Monitoring and Work Plan? Are the ENGINE project’s four intermediate results the most effective means to achieve the Project’s goal given recent evidence and changes in the Ethiopian context (i.e. the revised National Nutrition Program, burgeoning support for multi‐ sectoral collaboration at National and Regional level, and changes of emphasis within USAID/ Ethiopia)? 2. What has been the impact of the actions and activities of ENGINE’s efforts in fostering partnerships amongst the USAID Projects (FTF and IFHP) and in its multi‐sectoral engagement/actions with host country entities at the national, regional, district, and kebele levels? 3. What have been the contributions of the FTF Projects to IR 5 of DO 1 (under‐nutrition reduction) of the USAID/Ethiopia CDCS and its development hypothesis? 4. To what extent has this project contributed to gender equity and female empowerment and specifically addressed the role of gender in decision making on use of resources for maternal and infant feeding and increased women’s access to resources and services that will improve their nutrition and that of their children? 5. Given the findings in questions 1‐3, does the Project have the right balance of staff and funding given activity priorities? 6. How effectively and efficiently has the consortium of ENGINE sub‐partners performed as well as the leadership of the Prime. 1.6 Evaluation design The evaluation was based on a qualitative design, involving three main activities: 1. A review of literature 2. Key informants interviewed centrally in Addis Ababa 3. Field visits to observe project activities, and conduct interviews with local key informants and community members The validity of the evaluation findings was based on triangulation of information gathered from each of these activities, combined with an assessment of the technical plausibility of project activities ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 4 | Page leading to the expected results. Further details on the evaluation design are provided in Annex 3 and the CVs of the evaluation team members are provided in Annex 8. 1.7 Report limitations There are important socio‐economic and agro‐ecological differences between the four regions included in USAID’s “zone of influence.” However, given the relatively smaller number of woredas covered in Tigray region, that region was not visited. In addition, given the time limitations and SOW for the evaluation it was not feasible to use a statistically representative sample of areas and informants to examine ENGINE’s work. However, some quantitative analysis was in a Lot Quality Assurance Sample (LQAS) survey during the internal mid‐term evaluation of ENGINE conducted by SCI2, and these results have been utilized. USAID’s June 2014 “Qualitative program review of three FTF Activities using agriculture extension agents to promote nutrition: documenting the roll out of a new approach” report produced by Strengthening Partnerships, Results, and Innovations in Nutrition Globally (SPRING), and the ENGINE‐commissioned “Technical Report: Assessment of Quality of Nutrition Services” (April 2014) were also used as references. 2. EVALUATION FINDINGS The findings presented below are structured according to the six evaluation questions (section 1.5 above) posed to the team in the SOW (see Annex 1). ENGINE has four primary IRs, each accompanied by a set of strategic approaches (section 1.2). ENGINE’s performance under IRs 1‐3 is examined below under evaluation question 2, and progress against specific targets is discussed throughout this report, and detailed in Tables 1 to 3; each Table delineates progress toward each target, the likelihood of achieving these targets by 2016, and where off‐target, the steps underway or recommended to achieve them. 2.1 Evaluation Question 1: Performance against objectives  To what extent is ENGINE progressing against planned objectives as embedded in its PMP and work plan?  Are the ENGINE project’s four IRs the most effective means to achieve the project’s goal given recent evidence and changes in the Ethiopian context? Finding #1: Out of 33 still‐relevant indicators in the ENGINE PMP, the team has concluded that: – 21 targets are very likely to be achieved – 8 targets are likely to be achieved – 4 targets are uncertain In addition to this assessment of the relevance of the indicators, the evaluation carefully considered the target set at project inception for stunting reduction and found it unrealistic (see more detailed discussion in section 2.7). Through the interviews conducted and a review of data from ENGINE’s internal mid‐term evaluation report, the evaluators did not identify a significant difference in performance among the three regions for most of the 33 indicators reviewed. The only indicator where some regional differences were noted at the health center and health post level concerned iron folate supplementation. In the woredas of SNNPR and Oromia visited, existing stocks of iron folate supplements have been either unavailable or expired since December 2013. As discussed later in this report, if this issue is not urgently addressed, then ENGINE is unlikely to achieve its target for this indicator in these regions. 2 Save the Children. Internal Mid‐Term Evaluation of ENGINE. April 2014. The report’s “Dashboard Assessment”, its participatory impact assessment and its WASH commentary were particularly useful in this external evaluation. That report, like this one, lauded ENGINE for its assistance to the government in ways, which were not design‐directive. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 5 | Page Table 1: Progress and predicted achievement under IR1 Capacity for and institutionalization of nutrition programs and policies strengthened Indicator Base￾line Latest data Endline target Predicted achievement (1-5 scale) Actions being taken or recommended # multi-sectoral nutrition meetings with MoH using coordination framework 0 4 10 5 Multisectoral Nutrition Coordination Board now meets bi-annually. Next meeting will be August. Target likely to be overachieved. # regions with multi-sectoral nutrition coordinating body 0 4 4 5 NNP launched now in all 4 ENGINE regions. Target achieved. # ENGINE initiatives contributed to NNP 0 1 1 5 NNP completed. Target achieved. % of students at ENGINE supported HEIs who demonstrate sufficient knowledge in nutrition 0 n/a >80% 3 Jhpiego responsible for conducting surveys starting Y3. DA graduates now. Health graduates 2015. Viewed as high target. # of new graduates from ENGINE supported HEIs by cadre 0 4361 12,300 4 Burie TVET discontinuing DA training. May affect achievement of target # of HIE instructors who successfully completed training on nutrition QI methods at ENGINE supported HEIs 0 762 578 5 QI methods training completed. Target overachieved % of ENGINE supported HEIs that integrated the revised nutrition, HIV and food security syllabi into their curriculum 0 33% 85% 3 Achievement of targets may be affected if DA training through TVETs is discontinued. Need to engage Higher Education Bureau within MoE. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 6 | Page Table 2: Progress and predicted achievement under IR 2 Quality and delivery of nutrition and health care services improved Indicator Baseline Latest data Endline target Predicted achievement (1-5 scale) Actions being taken or recommended # HC’s with QI services integrating nutrition 0 0 50% of HCs 3 Had planned for QI to start in Y2 but has only started in Y3. Given the large number of HCs to cover, this target may not be achieved. # of HFs with SAM management capacity 334 46 359 * This indicator is no longer relevant to the ENGINE project. Not a program area of focus. Recommend cutting. 3 *# of children 6-59 mo who rec’d VAS from ENGINE supported programs 915,839 2,731,610 4,205,663 3.5 Target achievement may be adversely affected in some woredas by discontinuation of Child Health Days. *% of children 0-59 months with diarrhea who received zinc and ORS 0 ? 60% 4 Target should be achieved but logistics, delays with stock replenishments, could affect performance. Recommend push with MOH. *# of children 24-59 months de￾wormed in the previous 6 months n/a 1,704,962 29% 3,172,512 5 Target likely to be achieved but may be affected in some woredas by discontinuation of child health days. # of people trained in child health and nutrition through ENGINE programs 0 15,067 28,000 5 Achievable. # of HCWs who successfully completed in-service training (PMTCT) 0 718 2206 4 ENGINE is working to meet target by September with the PEPFAR funds ending. # of eligible clients who received food and/or other nutrition services 0 4864 15584 5 Achieved. 3 ENGINE initially included this indicator because the project was receiving PEPFAR funding. However, given the low SAM rates and the emergence of significant issues related to WASH, a management decision was taken to shift budget funding from SAM over to WASH. As a result this indicator is not longer directly relevant to the project. 4 Clearly a serious underestimate. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 7 | Page % of women15-49 years with children 6 -18 months who took Fe/Fol supplement during last pregnancy 21% 75% (LQAS) 60% 4 ENGINE has raised issue with Pharmacy Supply Agency of the MOH. In exceptional circumstances recommend ENGINE temporarily fill supply gap. Supplies will affect whether target achieved. % of pregnant women who attended 4 or more ANC visits during last pregnancy 18% n/a 40% * ENGINE is not directly providing TA relating to encouraging mothers to come for ANC. Recommend removing indicator. Prevalence of EBF of children < 6 months 52% n/a5 +8% 5 Achieved according to IFPRI baseline data % of children <1yr fully immunized 49% n/a >85% (maintain) * ENGINE is not directly providing TA related to immunizations. Recommend removing indicator. % of mothers of children 6-36 months who know 2 signs of childhood illness n/a n/a Increase to 50% from baseline * ENGINE is not directly providing TA related to this indicator. Recommend removing indicator. # of U5 reached by ENGINE￾supported nutrition program 0 3,094,029 5,753,229 5 Indicator subject to over-counting. To control ENGINE is counting VAS reach plus children U6 mos. Target likely to be achieved. 5 A figure of 67.6%, however, was found by an IFPRI survey. It appears that the target was inadequate. A revised target of 75% appears reasonable. These figures, however, should be interpreted with caution. International evidence indicates that exclusive breastfeeding (by definition absolutely nothing else given to the child at any time) estimates are frequently overestimates. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 8 | Page Table 3: Progress and predicted achievement under IR3 Prevention of under‐nutrition through community‐based nutrition care practices improved Indicator Baseline Latest data Endline target Predicted achievement (scale 1-5) Actions being taken or recommended % of children 6-24 months with breastfeeding initiated in the first hour 40% n/a 60% 5 Core program area for ENGINE. Target likely to be achieved. % of children 6-23mos who received solid, semi-solid or soft foods and frequency in the last 24 hours 69.3% n/a 80% 4 Part of ENGINE’s core work. Target likely to be achieved. Prevalence of children 6-23 months receiving MAD 13% n/a 40% 4 Part of ENGINE’s core work. Target likely to be achieved. Mean number of food groups consumed by WRA 2.8 2.86 4 3 Will be achievable for livelihood families but more difficult for population as a whole. Target may not be fully achieved. % of HH using iodized salt 67.4% n/a 80% 4 Target likely to be achieved. # of community conversation agents (CCAs) trained in CCs 0 240 3000 4 ENGINE working through local NGOs – target likely to be achieved. 6 A reflection of inadequate attention given to this indicator, until recently. The low figure may also relate to the fasting practices of women (see “Additional Recommendations.”). Among livelihood beneficiaries, evaluation interviews suggest significant improvements. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 9 | Page Finding #2: The evaluation found that 9 of ENGINE’s PMP indicators are not being directly addressed through ENGINE’s main intervention areas or do not have appropriate targets. For example, under IR2 ENGINE is no longer supporting interventions related to the management of Severe Acute Malnutrition (SAM) and thus the indicator “number of health facilities with SAM management capacity” is no longer relevant (see Appendix 1, column six for details on each indicator). Appropriate targets have also not been set for 4 of ENGINE’s 6 WASH indicators. However the evaluation team was informed by ENGINE management that new WASH targets would be established in August 2014 for the PMP. The FTF indicator for Vitamin A Supplementation, “Number of children under five who received Vitamin A from USG‐supported programs” was assigned a score of 3.5/5 by the evaluators given (a) the significant coverage gap remaining between the year three estimates7 and the Year 5 target and (b) the shift away from using Community Health Days to routine systems. The evaluation found that ENGINE has already achieved its target of 60% for exclusive breastfeeding (EBF) in light of the figure of 67.7% found by the recent IFPRI survey.8 These figures however need to be interpreted with caution recognizing that international evidence indicates that EBF rates are often overestimates. The other indicator noted with caution by the evaluation was the mean number of food groups consumed by Women of Reproductive Age (WRA). Based on data from the LQAS conducted for the internal ENGINE Mid‐Term Evaluation, little to no change has been observed on this indicator. The low figure may relate to the fasting practices of women (see section 2.4), which is an area being addressed through ENGINE’s partnership efforts at the community level with the Ethiopian Orthodox Church. Finding #3: Regarding the evaluation question of whether ENGINE’s four IRs are the most effective means to achieve the project’s goal, the evaluators are able to answer in the affirmative. With regards to gender, however, the evaluation team found that the project PMP does not contain indicators to accurately assess the impact of the project on gender mainstreaming and women’s empowerment as they relate to nutrition – this despite ENGINE’s baseline survey and operational research being strong on gender. Specific weakness in ENGINE’s PMP and recommendations were also noted in the project’s Gender Mainstreaming Strategy (October 2013). With regards to SBCC, the best indicator in the PMP is “number of community conversation agents trained in community conversations (CCs)”. However, while this indicator captures the number of CCs held, it does not indicate how many beneficiaries (RPA women and caretakers of young children) were reached or anything about the information provided. 2.2 Evaluation Question 2: Impact of actions with government and USAID‐assisted partners What has been the impact of the actions and activities of ENGINE’s efforts in fostering partnerships amongst the USAID Projects (FTF and IFHP) and in its multi‐sectoral engagement/actions with host country entities at the national, regional, district, and kebele levels? 2.2.1 IR1: Capacity and institutionalization of nutrition programs and policy strengthened Finding #4: ENGINE played a major role in the redesign of the NNP and is now actively involved in NPP rollout in ENGINE woredas of four regions and 12 zones of the country. The newly designed NNP, launched in 2013 and scheduled to operate through 2015, introduces a multi‐sectoral approach to nutrition across Ethiopia and lays out in detail the responsibilities of the nine national level ministries in this broad‐ based effort to reduce under‐nutrition in the country. 7 References: ENGINE Baseline Survey Report, 2014; ENGINE Mid‐Term Evaluation, 2014. 8 IFPRI. Ethiopia Strategy Support Program II. Agricultural Growth Program of Ethiopia. Baseline Report. 2011. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 10 | Page The evaluation team’s interviews with national level staff working in the concerned ministries and with sub‐national government officials indicate that ENGINE’s performance in this undertaking has been exemplary. ENGINE’s efforts in the NNP redesign work reflected both diligence and creativity, while, appropriate for such an undertaking, ENGINE functioned consistently as a team player with no attempt to undermine the government’s centrality in the work. At the regional level, ENGINE has facilitated rollout of the new NNP through patient explanations of the approach, the organizing of launch workshops and the development of inter‐ sectoral coordination mechanisms with zonal and woreda‐level staff. To date, ENGINE has supported the official NNP launch in all four of its operating regions and plans to continue its roll out into its 12 zones. Finding #5 ENGINE has established effective working relationships and partnerships at the national and regional levels with federal and regional government officials, FTF partners, UNICEF, the World Bank, USAID health partners and many other external stakeholders interviewed. Senior officials in the MoA at federal regional levels remain difficult to engage on the nutrition agenda. At the zonal and woreda levels however, ENGINE staff have well developed working relationships with MoA staff as they implement ENGINE’s agronomic and livelihood support programs together. Important inroads have also been made at the national level with the recent inclusion by the MoA of recommended nutrition interventions into the design of the new PSNP 2 and AGP 2 plans. Another positive move is the MoA willingness to consider the creation of a “nutrition team” under the Advocacy and Training Directorate under the State Minister for Extension. ENGINE’s diligent efforts to engage the MoA from the bottom up are continuing with a view to influencing officials at the regional and national levels, who hear and learn how well the nutrition interventions are working. Further advocacy efforts such as the ENGINE‐supported gathering of policy makers and parliamentarians in early 2014 at the national level will be key to increasing the nutrition commitment and involvement of other sectors, including education. One senior government official interviewed by the evaluation team at the regional level stated, “As long as they have bread to eat, we don’t care. Our focus is production.”9 Finding #6 Evaluating the sustainability of an effort like ENGINE’s – as with so many programs in Ethiopia – requires, at the outset, an examination of the planning and budgeting mechanisms at the national level and below. In nutrition, as in other sectors, sustainability, inherently, requires strategic means of assuring that national and sub‐national budgets exist and are not endlessly dependent on donor assistance. The issue, of course, goes well beyond an evaluation of ENGINE, but it is emphasized given the need for budgets and nutrition activity goals for each ministry to be established. In all three regions visited, ENGINE’s staff highlighted that efforts were underway at the zonal and woreda levels to bring ENGINE‐funded nutrition activities on‐line with annual plans and budgets. Many donor‐assisted development programs begin by providing models and then assist governments – and particularly sub‐national governments ‐ to extend these programs with their own budgets. The ideal ENGINE programmatic thrust for the final year and a half of its operation would be to facilitate such a handover – at least in a portion of its woredas, encouraging and facilitating planning, budgeting, implementation and systematic monitoring. While other determinants of sustainability, namely capacity development and systems strengthening, are being addressed by ENGINE and, to some extent, by the government itself, the absence of ENGINE‐supported activities in national and sub‐national planning and budgeting constitutes, at present, the primary constraint. 9 The same official went on to say, “If they’re not eating enough, it will show up in information on severe (acute) malnutrition and in death rates.” When pressed, he offered to delegate the nutrition issue to “my home economics and gender people.” ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 11 | Page 2.2.2 IR2: Quality and delivery of nutrition and health services improved Finding #7 ENGINE’s involvement in the design and strengthening of pre‐service and in‐service training of GoE health care and agriculture staff, and its implementation of such training in its coverage areas has often been impressive. Trained individuals spoke frequently of their “discovery” of nutrition and of the ways in which ENGINE has been “the engine” driving their nutrition‐related undertakings. Working with Jhpiego, ENGINE developed and delivered a Quality Improvement Assessment (QIA) for nutrition at community, health post and health centers and has recently started implementing a QI model as part of a systems building approach to improve the quality of nutrition services. ENGINE’s training “model” is addressing both short (in‐service, mentorship, supervision), medium (pre‐service) and long‐term (MSc/PhDs) HR capacity for nutrition. Its on‐the‐job Quality Improvement (QI) training, support and mentorship for HCWs/Health Extension Workers (HEWs) and Development Agents (DAs) working through zonal/woreda/kebele health and agriculture bureaus have been excellent. Finding #8 ENGINE has achieved some important successes with its pre‐service training component implemented by Jhpiego ‐ supporting the first ever competency‐based curricula for GoE agriculture and health staff cadres on nutrition in Ethiopia. Unfortunately due to unforeseen delays encountered with the start of the pre‐service training program, the number of Higher Education Institutions (HEIs) supported as well as the number of graduates by year three so far is lower than planned.10 That said, the number of instructors trained on nutrition is already higher than ENGINE’s projected target for Year 5, indicating that the student graduation targets may still be achievable. The target of achieving 80% of students at ENGINE‐supported HEIs who demonstrate sufficient knowledge in nutrition is viewed as a challenging one by ENGINE management – particularly for the agriculture technical and vocational schools – given the relatively new incorporation of the nutrition modules. The evaluation team visited the ENGINE/Jhpiego‐supported Burie Agriculture Technical and Vocational Education Training (ATVET) College in Amhara Region. One significant issue noted was the slow start‐up and initiation of curriculum development and training activities for DAs. While the project successfully provided in‐service training for two rounds of previously graduated DAs on nutrition, only 5 to 6 hours of nutrition content was provided to the recent class of 1300 DAs between November 2013 and their graduation in May 2014 following ENGINE support.11 Jhpiego has indicated that dialogue with national level TVET officials is ongoing to address the above situation with Burie College. However this is another policy area where ENGINE can engage the MoAFC and the Ministry of Education/TVET Agency to discuss ways to sustain the integration of nutrition into agriculture curricula and to strengthen nutrition sensitive agriculture pre‐service and in‐ service training over the long‐term. Finding #9 Attrition of health workers at the health center and health post level, a consequence of low salaries and inadequate job satisfaction, emphasizes the need for continued on‐the‐job training and supportive supervision (as ENGINE is doing) to ensure that new and existing staff keep up their training and that high quality nutrition services can be routinely delivered and monitored. ENGINE’s QI model and in‐ service training approach, described above, states that front‐line workers (HEWs) receive at least monthly supportive supervision visits from a Health Post Nurse, and quarterly supervision by the Zonal ENGINE Health Coordinator. From the 5 health centers and 8 health posts visited, the evaluation team noted that the nutrition counseling provided by health center staff was more likely to be tailored to the specific situation of the child, than that provided by HEWs. 10 Reference: ENGINE Internal Mid‐Term Evaluation Report (June 2014). 11 The vice‐dean of Burie also highlighted that the DA training program from Burie has now been shifted to Debre Markos University and, as a result, he did not expect that the nutrition curriculum developed at Burie for DAs will continue to be used in future unless some of the instructors decided individually to continue to use the content. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 12 | Page Finding #10 ENGINE has fostered effective partnerships with FTF implementing partners at the national level (see section 2.3). One of ENGINE’s most promising partnerships is its new strategic relationship with the FTF GRAD program. Over the past year, ENGINE staff have assisted the GRAD team to incorporate nutrition indicators into their PMP and, starting in November 2014, ENGINE, GRAD (and the OFDA funded GOAL Project) will start implementing together in 17 PSNP woredas where GRAD and GOAL are already operational. ENGINE has also formed a strategic partnership with the USAID‐funded Integrated Family Health Project (IFHP) in woredas where both projects have been operating and where QI training on nutrition is an overlapping issue. IFHP staff interviewed by the evaluation team provided examples of how IFHP and ENGINE share transportation to health centers and posts, and provide supportive supervision follow‐up visits to assist one another. Finding #11 While materials development and training relating to WASH were not part of the original ENGINE mandate, ENGINE‐supported research efforts in Ethiopia and new international findings resulted in the subsequent incorporation of a WASH‐orientation into ENGINE’s plan of work in Year 3. This new orientation, still in its early stages, shows considerable promise, but additional clarity regarding the means by which ENGINE will support households to access needed hardware inputs – perhaps through partnerships with relevant social‐marketing efforts ‐ is needed. The project, to its credit, has been actively seeking to coordinate its WASH efforts with national and sub‐national partners. Finding #12 While the evaluation team found consistently high quality nutrition‐related counseling at all levels of health services, other problems of concern were identified at health posts and below. Since the majority of households – including nutritionally at‐risk mothers and young children ‐ are reached at these sub‐ health center levels, and since the achievement of ENGINE’s major targets requires efficiency and consistency at these levels, the evaluation team has sought to better understand the problems faced and to develop recommendations designed to help address them.12 The first major problem identified at this level is what appears to be a serious under‐estimate of target beneficiaries (particularly pregnant women) at the health post level and, in turn, inadequate coverage of these target beneficiaries by HEWs at the community level.13 The second is a supplement supply problem. Iron folate and zinc supplements have been out of stock (expired) in SNNPR and parts of Oromia since December 2013. ENGINE is working with the Ministry of Health and partners at the national level to unblock this serious problem. As mentioned earlier, there are serious consequences for nutrition and for the achievement of ENGINE’s target. 2.2.3 IR3.1: Prevention of under‐nutrition through community‐based nutrition Finding #13: The third health sector problem at the health post and below relates to social and behavioral change communications (SBCC). Despite the Year 1 and 2 delays faced in its SBCC efforts (material design, training and roll‐out) and the ensuing management challenge to find an SBCC technical partner to replace Johns Hopkins University (JHU), the evaluation team was impressed both by ENGINE’s resiliency – utilizing new as well as existing materials (some designed by the GoE together with Alive and Thrive) in its training activities, and by the expeditious efforts by the Manoff team to complete the analysis and proceed with the development of particularly creative but practical and accessible materials . 12 As indicated below, the recommendations in this report may have implications not only for ENGINE’s undertakings in its final two years of operation, but also in the further redesign of the NNP (scheduled to function from 2016 to 2020) and in the design of AGP 2. 13 Health posts are provided with estimates from the woreda and the zone of the number of pregnant women who should be present at the kebele level. HEWs then often utilize these projections as targets, rather than seeking to cover all of the pregnant women in the kebele. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 13 | Page Finding #14 To date, ENGINE has distributed nearly 11,000 SBCC materials at health centers and health posts, to HEWs, DAs and other staff working at sub‐national levels. ENGINE is also assisting the Federal MoH to harmonize existing nutrition training materials (currently there are six sets, funded by different donors). ENGINE (through JHU) also supported local NGOs to conduct one round of CC groups, and the evaluation team was able to visit one group in Amhara region. A radio magazine on nutrition during “the first 1000 days” was also broadcast in three languages for four months before the JHU contract was terminated. ENGINE’s strategy is for the new ENGINE‐supported SBCC materials to become the core nutrition SBCC materials for use at community level by HEWs, for government training of the Health Development Army14 and for their subsequent use, and by local NGOs. The evaluation team, however, is concerned, about a fundamental shortcoming in SBCC message dissemination to its most important recipients – the mothers/caretakers of young children. The evaluation team observed repeatedly that health staff at all levels knew the messages well, and that many mothers were able to repeat them back nearly verbatim. But the problems are twofold:  There is a substantial gap between knowledge and practice. (Mothers can often repeat, nearly verbatim, BCC counseling messages on, e.g. exclusive breastfeeding, timely introduction of nutrient dense complementary food, continued feeding during infection, without putting this knowledge into practice.)  Messages provided to mothers are generic and not geared to the specific needs of their children. (This relates in part to the virtual absence of accurate GMP – discussed below – and, in turn an inability to identify growth faltering in children and determine its causes in specific children.) Bridging these gaps is likely to represent the key to ENGINE translating SBCC into actual reductions in stunting. Finding #15 In addition to the SBCC knowledge‐to‐practice gap noted above, the evaluators repeatedly observed, at all 13 health centers and health posts visited, the proforma and inaccurate weighing of children solely for the purpose of recording their weights for the HMIS, followed by the provision of generic SBCC messages to the mothers/caretakers. Although the evaluators recognize that growth monitoring and promotion (GMP) has not technically been part of ENGINE’s capacity building and QI with health facility staff, the weighing of children U3 is now a standard Government HMIS indicator, which is being poorly done and is generating inaccurate data on nutrition within HMIS. When done correctly, proper GMP has been demonstrated to address the knowledge practice gap because caregivers appreciate and see the relevance of the information they are being given for their specific child.15 ENGINE senior staff fully understands this situation and its consequences. Although the latest national survey indicates an underweight prevalence of 29%16, an informal collection of data from the log books reviewed by the evaluation team at 3 health posts visited in Amhara region found that only between 3‐4% of children U3 were underweight. 14 Health Development Army (HDA) members are community women who work with five community households each, focusing particularly on caring practices and encouraging the use of health facilities. The HAD members train ‘model families’ to implement health initiatives and to serve as role models for other households. 15 The issue of GMP has been hotly contested within the international nutrition community. While the 2008 Lancet “Nutrition Series” deprecated its value, a subsequent major survey of professionals involved in international nutrition strongly favored its use. Proponents of GMP indicate that negativity toward GMP is the negativity universally shared when GMP is done poorly and when it is done solely for reporting purposes. Opponents argue that GMP is not cost‐effective in areas of limited infrastructure and limited local resource capacity. Proponents compare the debate with that which took place over field‐based immunization in earlier years – when serious problems existed with cold chains, sterile needles, recording systems and information campaigns. Instead of discarding field‐based immunization, the problems were solved and, in the process, countless lives were saved. The issue surfaces here (a) because GMP had been an integral part of the NNP, (b) because it is an HMIS indicator, and (c) because it is being carried out poorly. 16 Ethiopia Demographic and Health Survey, 2011. Central Statistical Agency, Government of Ethiopia, March 2012 ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 14 | Page These observations reflect serious flaws in the reporting of U317 child weights in the HMIS at the health post and community levels and are further addressed in the Recommendations (section 4). Finding #16 ENGINE is working strategically with the Federal MoH to harmonize existing nutrition training and SBCC materials developed by government, UNICEF and NGO partners over the years. 2.3.4 IR3.2: Access to food and economic strengthening opportunities Finding #17 ENGINE employs approximately 5 to 6 zonal livelihood coordinators per region who work closely with DAs in their respective zones and woredas to implement their agronomic activities, and to provide support to households through the Most Vulnerable Households (MVH) Program. As highlighted in the SPRING “Qualitative Program Review of Three FTF Activities in Ethiopia Using Agriculture Extension Agents to Promote Nutrition” and also observed by the evaluation team, DAs are mainly male and when it comes to nutrition, their work focuses on the production of fruits, vegetables and small livestock, cooking demonstrations and promoting dietary diversity. Of all the woredas and FTC’s visited, the evaluation team met only one Home Economics Agent in one woreda, in SNNPR. The evaluation team observed that ENGINE has achieved some promising results through its agronomic and vulnerable household livelihood support activities and is interested to expand these efforts (see discussion below under Finding #18.) The participatory methodology used to identify and select beneficiary households for the MVH program, supervised by ENGINE’s Zonal Livelihood Coordinators, was found to be rigorous and transparent with multiple selection stages included to exclude households that did not meet the MVH criteria. Finding #18 ENGINE has been supporting (a) agronomic and cooking demonstrations carried out at Farmer Training Centers (FTCs) and schools, and (b) livelihood support for targeted low income households through garden and small livestock support and related counseling. Through its agronomic support, a subset of better‐off farmers (with larger holdings and adequate water) have been receiving training geared, in part to assist these farmers to diversify their production (from an earlier concentration almost solely on staple crops and beans to increased vegetable production.) Many of these farm families also are receiving some useful information on food preparation and utilizing these vegetables through ENGINE‐supported cooking demonstrations. The evaluation team’s concern with these demonstrations is adoption rates.While ENGINE has commissioned small‐ scale studies to assess the benefits to those attending demonstrations (with mostly positive findings),18 findings on adoption were not quantitative, and few demonstration sites are collecting such information systematically. The absence of such information has numerous consequences, among them the inability of trainers to re‐orient future training based on information from previous non‐adopting participants on constraints faced. 19 20 17 The team observed the collection of weight for age data in some areas for children U3, in other areas for children under age 2, and in still other areas for children under age 5. The team recommends data collection for children under age 2. 18 “Participatory Impact Assessment of ENGINE Livelihood Interventions in Oromia – Preliminary Findings.” May 2014. 19 The evaluation team’s own estimates of households and farming households in selected kebeles of Amhara region indicate a relatively small percentage attending agronomic demonstrations. This is explained by the fact that only farm households with adequate land and water were selected, with the idea that that they would serve, in turn, as models for other households. 20 In one woreda, when pressed, the agriculture staff indicated that, in the case of perma‐garden demonstrations, 96 out of a planned 240 households were represented, and, of these, six implemented the practice. In the case of composting demonstrations, only 10 of a planned 240 household representatives attended, but all 10 implemented the demonstrated practices at home. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 15 | Page All of the 10 DA’s interviewed by the evaluation team during visits to FTCs were asked to share: a) the number of farmers who receive ENGINE‐funded agronomic training; b) how many of them were women; c) how many households adopted the technologies they were taught; and d) which households in the kebele did not attend the training and why? While all of the DAs interviewed as part of this evaluation could provide the number of farmers who attended the agronomic training (male and female) in their catchment areas, none could provide the percentage of area farmers this number represented, or provide information on adoption rates. Finding #19 The team’s questions about adoption led to particular concerns about school gardens and school‐based demonstrations where, it appears, only a small portion of those students participating in the school gardening receive seeds and seedlings to take home (e.g. see Annex 10). On the positive‐side, the team did learn that in some woredas (particularly in SNNPR and Oromia) the school garden concept is expanding to other non‐ENGINE targeted schools using locally‐available resources.21 Finding #20 With respect to livelihood assistance through ENGINE’s MVH program, particularly poor households with young children or reproductive age mothers are receiving garden and livestock assistance, improving their incomes, and learning about the importance of assuring adequate consumption (particularly by vulnerable family members) of the food produced. This is an impressive activity. Some concerns were raised to the evaluators by DAs, ENGINE Zonal Coordinators and beneficiaries, however, about the sustainability of ENGINE’s quality seed provision efforts. The high quality seeds provided by ENGINE at these demonstrations – some of them imported – are not currently readily available in local markets. This problem has been identified by ENGINE management and a strategy to ensure sustainability is underway. Interviews with members of low income households being assisted by the MVH component indicated that they believed the garden and small livestock assistance provided to them have been helpful in improving their lives (and similar positive feedback was documented in the May 2014 PIA of ENGINE Livelihood Interventions in Oromia). While the project has collected baseline data on a sample of project MVH participants as part of the ENGINE baseline survey, and is expected to collect endline data on a comparable sample, there is no explicit measurement of the progress being made by individual households in their food security status or their dietary diversity. Such monitoring, in addition to guiding project implementation, would have considerable value both in USAID’s efforts to leverage additional donor support for this important and valuable undertaking, and in providing encouragement to individual households. The issue is further addressed in the recommendations. The team carried out some first year analysis of the three livestock strategies being pursued by ENGINE with these households: the provision of pregnant heifers, of sheep and goats, and of poultry. These findings are reported in Table 4. As indicated, the poultry option seemed to provide the best source of continuous food for these vulnerable households, and the best economic returns in the first year. The team however, is seriously concerned with the livestock disease issue. A 2013 ENGINE livestock assessment found mortality in sheep and goats (breeding stock and newborns) to be 7%, and the LQAS findings from ENGINE’s internal mid‐term evaluation found 11% mortality among sheep (due largely to sheep pox.) However, local livestock officials, DAs and beneficiaries interviewed for this evaluation across the three regions reported higher mortality figures, and a perception that roughly one third of all small livestock provided by ENGINE died within the first year. 21 The principal of Ejersa Kao Primary School in Girar Jarso woreda, Oromia indicated that ten other schools had come to observe their school gardens, and subsequently obtained DA support and seeds purchased from their own school budgets to initiate gardens at their respective schools. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 16 | Page Table 4: Comparison of ENGINE livestock assistance activities in SNNPR Livestock type Initial ENGINE investment (birr) Average annual household income (birr) Direct nutritional benefits to the households Risks 1 pregnant heifer 5000 4320 Daily milk, Cheese, butter Disease, delayed reproductive age Goats or sheep (3 female, 1 male) 3000 2700 Meat Occasional milk Food purchased with funds from livestock sales Disease, shortage of quality feed, absence of adequate treatment Poultry (10 hens, 2 cocks) 2640 5040 Daily eggs, occasional meat Food purchased with funds from livestock sales Disease, absence of effective medication, direct competition with human food 2.3 Evaluation question 3: Contributions of FTF partners in reducing under‐nutrition What have been the contributions of the FTF Projects to IR 5 of DO 1 of the USAID/Ethiopia CDCS and its development hypothesis? Finding #21 The evaluation team examined specifically the nutrition contributions of USAID FTF partners, the extent to which they are coordinated with ENGINE’s work, and the real and potential synergies of such cooperation. Table 5: FTF project nutrition activities Activity AMDe LMD GRAD PRIME Baseline survey Cost of Diet Survey IYCF baseline (selected woredas) Yes On dietary diversity Nutrition included in PMP Yes Yes Yes Yes MOA training 5 trainings (2014) 3 trainings (2014); 126 DAs Yes Yes HEW training No Planned Planned Yes Cooking demonstrations Yes 40 at health posts22 Yes # Yes # School education No School milk day (18,000 students) No Planned SBCC at community level Planned Planned Planned Yes New SBCC materials Brochures, sticks, manuals 3 planned Planned ENGINE has been providing technical leadership for USAID’s FTF partners on nutrition through its coordination of the national‐level FTF Partner Working Group, that meets quarterly, and its role in leading the March 2013 FTF Partners Meeting with a special focus on nutrition. As a result of USAID and ENGINE leadership in this area, all FTF projects now have national‐level nutritionist positions, and some also have regional level nutrition staff in place. GRAD has incorporated nutrition and appropriate indicators into its workplan and PMP, with a budget. Nutrition baseline information has also been collected by all FTF partners in their target woredas, and several joint field 22 The evaluation team was curious to see LMD, essentially a livestock project, carrying out cooking demonstrations (already being done by ENGINE) at health centers rather than, for example, promoting dairy product consumption through its existing value chain, and raised the issue with LMD. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 17 | Page visits among FTF partners have also been organized and conducted. Table 5 below highlights the primary nutrition activities of USAID Ethiopia’s FTF partners. Finding #22 By design, ENGINE was mandated to operate in FTF AGP woredas within Amhara, Oromia, SNNPR and Tigray regions to enable the overlap of USAID’s agricultural growth, livestock and economic strengthening programs with those promoting nutrition. As indicated, ENGINE provides technical assistance for all health centers and health posts to deliver improved nutrition services through the health sector within these AGP woredas. It also provides agronomic and livelihood support to selected farmer households in 2 to 3 kebeles per AGP woreda, and provides technical assistance for agriculture through FTCs and DAs . Over the first year and a half of implementation, FTF projects were co‐located in AGP woredas but there was little coordination among them. More recently however, coordination efforts at the national level have improved and efforts are being made to reach AGP farmers with activities designed to both reduce poverty and reduce under‐nutrition. As an example, AMDe, LMD, GRAD, PRIME and ENGINE have each mapped out their respective intervention woredas and are working to align and coordinate their nutrition activities (see Table 5 above). At the local level, however, the evaluation team observed that coordination of nutrition‐related services among the FTF partners requires additional attention. Finding #23 As part of a strategic effort to move ENGINE’s influence beyond its priority AGP woredas, USAID requested and obtained approval from the GoE within the past year for ENGINE to work with two USAID funded programs, GRAD and GOAL, in an additional 17 PSNP woredas, beginning in Sept 2014. This expansion presents an opportunity for USAID to provide a “package of interventions approach,” to improve nutrition, improve livelihoods and enable economic strengthening opportunities for some of Ethiopia’s most vulnerable households. As noted above, at present ENGINE is not providing agronomic and livelihood activities across all kebeles of its existing target woredas, so there is opportunity for greater collaboration between ENGINE and other FTF partners to expand coverage. The larger question is whether FTF partners are best positioned to carry out nutrition technical assistance, training and SBCC (focused on dietary diversity, not health sector‐related messages) or whether this is best done by ENGINE itself, with FTF partners complementing ENGINE efforts with nutrition sensitive activities that go along with their value chains. LMD’s nutrition advisor indicated that they are working to test the FTF “convergence theory”23 in some woredas by looking into downward value chain opportunities for the milk value chain, recognizing that many dairy farmers produce significant quantities of milk that are rarely consumed at the local level. Globally there is an increased understanding that the co‐location or “convergence” of nutrition‐ specific and nutrition‐sensitive interventions in a country’s vulnerable areas can produce synergies capable of dramatic decreases in stunting prevalence.24 The FTF program has tried to test this convergence theory in Ethiopia by overlaying USG‐funded economic growth and agriculture projects in AGP woredas, with those also addressing under‐nutrition. However, in Ethiopia (as well as in many other FTF countries) it is important to note that most FTF projects were not initially designed to include nutrition‐sensitive activities. Although ENGINE was purposefully co‐located in four regions with other FTF projects, the evaluation team found that coordinated programming overlap at the woreda and kebele levels, in line with the convergence theory, has not yet been achieved within the FTF program.25 23 But using other language. 24 See http://www.mdgfund.org/sites/default/files/ATddressing%20malnutrition%20multisectorally‐FINAL‐ submitted.pdf 25 True multi‐sectoral nutrition convergence, as indicated below, however, requires a yet broader array of interventions (including those from the health and WASH sectors) to be available in the same villages – not simply the same general geographic areas ‐ to permit synergistic effects. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 18 | Page More broadly, however, it became clear that efforts to implement nutrition‐sensitive agriculture have been hampered by:  limited understanding, despite ENGINE’s best efforts, of the concept of nutrition sensitivity, and means of designing agriculture programs in ways which actually improve food consumption and reduce food insecurity  some resistance at the national level to the idea that agriculture also has to “improve nutrition”  the absence of a common set of primary indicators in nutrition‐sensitive agriculture to monitor and to evaluate effectiveness.26 A recent AGP‐AMDe Cost of Diet Analysis report 27found:  daily cost of a minimally nutritious household diet in USAID “zone of influence” = 41.83 birr ($2.10)  daily cost of diet that is also “locally acceptable” (i.e. that minimizes the inclusion of less familiar foods or those considered by some culturally inappropriate) = 69.37 birr ($3.50.). Note that the latter figure is substantially higher than the daily income of an average rural Ethiopian household – indicating clearly the critical importance of the HH food insecurity constraints. At present FTF projects, including ENGINE, do address dietary diversity only indirectly. For example, agronomic demonstrations promote production diversity, clearly necessary for dietary diversity. Cooking demonstrations promote the utilization of a broader array of ingredients. Interviews with livelihood assistance beneficiary families suggest that this assistance has increased the dietary diversity of many participating households. Yet interviews with service providers indicate that the concept of dietary diversity is not yet promoted by these projects. There is little awareness, at present, that increased dietary diversity is necessary for normal child growth, healthy pregnancies and general good health. The provision of these messages explicitly, and the regular assessment of diversity28 can increase active consciousness and facilitate behavioral change. These issues are further addressed in the Recommendations section of the report. 2.4 Evaluation question 4: Contributions to gender equity and female empowerment To what extent has this project contributed to gender equity and female empowerment and specifically addressed the role of gender in decision making on use of resources for maternal and infant feeding and increased women’s access to resources and services that will improve their nutrition and that of their children? Finding #25 ENGINE’s overall mandate aims to improve gender equity and female empowerment through the specific means by which its nutrition activities are carried out. ENGINE completed a gender audit and analysis as well as a gender mainstreaming strategy in 2013. ENGINE’s PMP includes some gender‐ specific indicators, but fewer than would be desirable. Limited sample interviewing carried out by ENGINE in Amhara region suggests the possibility that local level ENGINE involvement in health and agriculture activities may be having some beneficial 26 The GoE uses income as a proxy for household food security. This is problematic because self‐reported income is more amenable to under‐ or over‐reporting depending on the assumed benefits which will accrue. The Government has no standardized dietary diversity indicator. FTF does not have a food security indicator per se, but does measure “prevalence of households with moderate or severe hunger” based on responses to three food security‐related questions. 27 USAID/Ethiopia, AGP‐AMDe. “Cost of Diet Analysis Results: Nutrition Sensitive Agriculture Tool (Nutri‐SAT) Ethiopia in Pilot Studies (undated.) 28 e.g. “Tell me the foods your child has eaten today. OK, your child has consumed grains and legumes, but has not consumed any vegetables – including green leafy ones – no fruit, and no animal food. Now tell me what you ate today.” International experience indicates that adults (men included) can easily memorize 7 food groups and then regularly check the consumption of other household members (see Appendix 13.) ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 19 | Page effect on women’s decision making (see Table 6). However, further monitoring and evaluation of ENGINE’s direct impact on gender and female empowerment is recommended. Table 6: Effect of ENGINE on Women's Decision Making (Amhara Region, small sample) Decision Type Proportion (%) of estimated household decision‐making by women Before project Mid‐2014 Crop production 46 73 Livestock production 62 73 Major HH expenditures 69 88 Borrowing money 62 73 Food purchases/meals 100 100 Children’s education 69 92 Family planning 73 85 ENGINE gender‐related creativity Among ENGINE’s understanding of gender issues is women’s workload – and particularly arduous labor, affecting not only the caloric expenditure of reproductive age women (sometimes exceeding caloric intake), but also their time available for childcare. In Dembeli Keta kebele in Wollisa woreda of Western Oromia, the ENGINE team was particularly creative. Sitting with a large group of women who are ENGINE livelihood beneficiaries and participants in ENGINE’s agronomic and cooking demonstrations, and in the context of discussing program‐related benefits and constraints, one of the ENGINE staff members asked the women, "What are the biggest problems you face overall in your lives?" The women responded immediately ‐ all basically saying the same thing ‐ that their workload is impossible ‐ and that the worst of the workload is having to carry water for more than an hour every day. On impulse, the ENGINE staff then called into the circle all the men who were standing on the periphery and told them what the women had said. And someone added, "Here's your chance to become the most famous village in Ethiopia, the most famous village in Africa. Get your men to start helping with the water carrying ‐ and people will be coming from everywhere to watch, to take pictures, and to tell your story." One older man stood up and said, "Yes, we understand, but we men also have lots of work to do, and men carrying water is against our cultural tradition." But then another man stood up and said, "OK, we'll do it " That man, it turned out, was Zerihun Neguse, the Village Chairman. Thirty minutes later, Neguse had gathered 200 men in the center of the village, telling them that the new gender‐sharing water carrying program would begin the next day! ENGINE staff are following up, and photos are available. This is virtually unprecedented in Ethiopia, and extremely rare in sub‐Saharan Africa as a whole. The evaluation team observed that ENGINE has been encouraging gender balance in local government administration, in DAs, and in other cadres of health and agriculture staff, and by assuring that many of its services are provided directly to women. ENGINE has also been creative on the ground in addressing nutrition‐related constraints facing rural women – see box above. Finding #26 ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 20 | Page Another important nutrition and gender issue noted by the evaluation team concerns the significant number of “fasting days” (220 per year for the Orthodox Church) – days during which no animal products may be eaten. These fasting days are likely responsible in large measure for the absence of larger increases in overweight and obesity in the population, and for the high rates of maternal under‐ nutrition and low number of food groups consumed reported in ENGINE’s baseline survey and mid‐ term evaluation LQAS.29 The religious fasting regulations impose significant problems for vulnerable groups. While pregnant women and children under the age of seven are excused from fasting, many fast anyway not wishing to be exceptions and desiring to be in solidarity with the rest of the family. ENGINE is already working with the Orthodox Church at the local level to identify and address the fasting issue as it relates to nutrition and SBCC. 2.5 Evaluation question 5: Balance of resources against activity priorities Given the findings in questions 1‐3, does the project have the right balance of staff and funding given activity priorities?30 Finding #27 ENGINE has a well‐developed management structure and organogram (see overleaf) and the evaluation team had few concerns. ENGINE’s move to adapt and allocate funding for WASH in Year 3 ‐ in place of further attention to severe acute malnutrition ‐ was particularly timely in light of the evidence. Finding #28 The primary staffing issue concerns the workload of zonal health and livelihoods coordinators who are typically seconded to zonal health and agriculture bureaus. Despite well‐crafted job descriptions, interviews with ENGINE’s zonal coordinators across the three regions made clear that these staff members are, in fact, heavily burdened with time consuming government meetings, required ENGINE monthly report writing, logistical responsibilities (e.g. livestock and seed procurement from local markets) and large catchment areas that often limit their ability to provide the critical supportive supervision, mentorship and monitoring at the health facility, post and village levels that is vitally needed. The ENGINE budget for Years 1 to 3 was allocated to IRs as follows: IR1 13%; IR2 23%; IR3 37%; IR4 26%. ENGINE’s administrative costs, 29% in Year 3, are within the GoE limit of 30%. 29 ENGINE Baseline Survey Report, 2014 30 See also discussions of budget and staffing in the Recommendations section. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 21 | Page 2.6 Evaluation question 6: Management effectiveness and efficiency How effectively and efficiently has the consortium of ENGINE sub‐partners performed, as well as the leadership of the Prime? Finding #29 The evaluation team was impressed both with ENGINE’s management efficiency and its resilience in responding to contracting problems and delays. The project has five field offices at the regional level ‐ two in Oromia, and works with five partners: Land O’Lakes, Valid International, JHU (whose activity was later given to three local NGOs) and now Manoff , Tufts University, and JHPIEGO. Of its total budget, 42% goes to these partners and 58% goes to SCI as the prime. In examining actual budgetary expenditure compared with original budgets, performance in IRs 3 and 4 were somewhat better than in IRs 1 and 2. The SCI office in Addis Ababa presently hosts 3 of the 5 partners, thereby facilitating communications among them. ENGINE holds bi‐monthly meetings, and Skype calls with those outside of Addis every quarter. ENGINE has faced and satisfactorily resolved the following management challenges:  Termination of the JHU communication contract has led to delays in launching the project’s SBCC activities. The Manoff Group, as indicated, has taken over the analysis and materials development energetically and creatively.  Valid International’s internal issues with the GoE were resolved when Valid’s status was changed from sub‐prime to technical assistant.  Land O’Lakes was able to overcome similar challenges. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 22 | Page  There were delays in initiation of the Tufts research, but these research projects are now underway or completed (see separate evaluation of IR4).  JHPIEGO’s procurement and budget utilization have been behind schedule because so many of its decisions are made at its HQ. Four local NGOs have been contracted with responsibility for implementation of the Community Conversation component: EOC/DICAC for Amhara and Tigray; FIDO and EMERDA for Oromia. It is too early to assess their performance. 2.7 A critical question: Is ENGINE likely to achieve its Stunting Reduction Target? The evaluation team believes that the 20% stunting reduction target in ENGINE was unrealistic in light of the primary limiting factors discussed below, and particularly the sanitation constraint – this is in light of recent international evidence, some of it released after the target was set.31 32 Contrary to expectations of continued reductions in stunting prevalence, the evaluation team is concerned that reductions in stunting may soon plateau – as has been the case in other countries – unless each of the potential limiting factors is adequately addressed. 33 Although the evaluators recognize that poverty reduction and other important interventions are in place in AGP woredas of Ethiopia where ENGINE is operating, the low uptake of some interventions – such as improved latrines ‐ is viewed as a rate limiting factor for ENGINE to achieve its overall stunting reduction target. An ENGINE IR4 study of stunting determinants in Ethiopia found that fully 29% of the difference between high and low stunting prevalence areas in the country was attributable to open defecation, and that this sanitation problem plus low maternal height (an indicator of maternal nutritional status) and poverty/food insecurity together explain 79% of the difference in stunting between areas.34 So where are ENGINE woredas on these three primary determinants – and potential limiting factors? Sanitation The sanitation issue, although now being addressed seriously by the government, continues to be a paramount problem with the improved sanitation facilities in ENGINE woredas not significantly better than the national average of 8.3% of households having improved sanitation facilities.35 Maternal Malnutrition In seven woredas of Amhara region covered in ENGINE’s baseline survey, maternal malnutrition, measured by mid‐upper arm circumference (MUAC), ranged from 45% to 59%. Rates of chronic under‐ nutrition among women using BMI ranged from 23% to 40%. These levels may have been reduced somewhat by ENGINE efforts encouraging maternal nutrition and day‐time rest. 31 WHO estimates that 50% of malnutrition is associated with repeated diarrhea or intestinal worm infections as a result of unsafe water, inadequate sanitation and/or insufficient hygiene. 32 While the 2014 “Mini DHS” found that stunting has been further reduced in the country as a whole and in most regions since the last survey, stunting prevalence in each of the regions where ENGINE operates is still reported at over 40%. (Government of Ethiopia. Central Statistical Agency. “Ethiopia Mini Demographic and Health Survey 2014. July 2014.) Additionally, even in the AGP woredas of these regions, the potential limiting factors remain serious issues. 33 Examples of countries where stunting rates plateaued after initial reductions include Congo (38% in 2001, 40% in 2007); Egypt (24% in 1992, 25% in 2008); Kenya (33% in 1993; 35% in 2005‐06); Madagascar (51% in 1992, 48% in 2003‐04); Mozambique (40% in 1997; 44% in 2003‐04); Niger (38% in 1985; 39% in 2008); Philippines (33% in 1993; 30% in 2003); and Vietnam (34% in 1998; 36% in 2006.) In most of these countries, sanitation has remained seriously problematic, as have other stunting determinants 34 Ghosh S, Suri D, Hiko D, Fentahun N, Griffiths J. “Factors associated with stunting in Ethiopian children under five.” ENGINE, Tufts U., Jimma U. (undated.) 35 The evaluation team recognizes the differences between Ethiopian and WHO standards in this regard, but also the dangers to children (falling into the larger latrine holes) of some facilities labeled as “improved,” ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 23 | Page Poverty and Food Insecurity There are numerous efforts in the country focused on poverty reduction, including the Government of Ethiopia’s AGP. ENGINE contributes through its livelihood activities, although on a small scale. Dietary diversity, an important component of food security, and an issue intensively addressed in this evaluation, is a particular problem in the country and in ENGINE woredas with most mothers consuming foods from fewer than 3 of the 9 food groups defined in the FTF Indicator Handbook. The latest international evidence on multisectoral nutrition indicates that the most significant reductions in stunting (as high as 4.5 percentage points per year) occur with multisectoral convergence, i.e.  the identification of highly vulnerable areas; and  the co‐location of interventions from each relevant sector (health, education, agriculture, WASH and social protection) in all villages of selected areas (not simply co‐locating at the regional or woreda levels as is being done through FTF). 3. SUMMARY AND CONCLUSIONS The general conclusions of the evaluation, across IRs 1 to 3, are as follows:  ENGINE is performing well and is on track to achieve most of its performance targets by the end of the project.  The project is having major impacts at the policy and institutional levels, and on strengthening national, zonal and woreda‐level systems to scale up nutrition services across sectors.  The GoE and USAID Ethiopia FTF projects have taken important strides to incorporate nutrition education and training into their programs and activities. FTF projects, however, need to improve monitoring and evaluation efforts to measure explicitly the effect of these projects on dietary diversity and household food security. Outstanding challenges are mostly well understood by ENGINE, with strategies and plans in place or being developed accordingly. Key issues affecting future project impacts are:  To continue to recognize the main determinants of stunting in Ethiopia, and to prioritize and reshape activities around critical areas such as WASH, food security and dietary diversity, while also continuing to support multi‐sectoral nutrition convergence across programs and actors.  To continue emphasizing the need for greater multisectoral nutrition convergence across all of USAID/Ethiopia’s FTF and GHI projects and measure the impact of this convergence.  In key activity areas such as training, information dissemination, demonstration and livelihoods support, there is a constant need for ENGINE to better document if and how activity translates into nutrition and food security impacts among target groups, and, where positive, to take action for scaling up.  Long‐term, the current momentum around nutrition in Ethiopia will need a range of ENGINE approaches and activities to become normalized within government, with less reliance on external assistance. ENGINE is well‐placed to work with GoE partners to integrate nutrition activities into government plans and budgets, and this effort should be intensified in years 4 and 5. Overall, ENGINE is making important contributions to nutrition policy and programming in Ethiopia. Through IR4, ENGINE also is contributing to broader understanding on key issues. However, as indicated above there are key areas of ENGINE activity where increased efforts are necessary to measure and document the nutritional and food security impacts of its activities, as well as those of other USAID‐Ethiopia FTF projects. 3.1 Evaluation question 1: Performance against planned objectives At the overall project level, there has been good progress against planned targets, with 29/33 targets very likely or likely to be achieved by the end of the project. However, a number of indicators/targets refer to measures of project implementation rather than project impact. In addition, there is a particular concern that the stunting reduction target of 20% is unrealistic (section 2.7). The project has started to ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 24 | Page address some of the stunting determinants more aggressively, drawing rapidly on research produced under IR4. Specifically, new WASH activities could contribute to stunting reduction but these activities need to be linked with hardware access mechanisms, and explicit targets are necessary. Similarly, potentially important work on fasting practices will require measurement, especially in terms of impact on maternal nutritional status. 3.2 Evaluation question 2: Impact of actions with government and USAID‐assisted partners ENGINE is achieving important impacts at the policy and institutional level such as contributing to the design of the NPP, and substantial capacity building of local government partners. Important challenges remain, such as raising awareness and commitment to nutrition across ministries at federal‐level, and normalizing nutrition activities in government plans and budgets as a means to support sustained nutrition programming and related benefits. ENGINE has established multiple and strategically relevant partnerships with key FTF implementers, and other projects and programs, providing responsive and practical technical assistance, and supporting coordination. The ENGINE support to nutrition and health services, implemented closely with government partners, has been extensive and of high quality. Further work is needed to better understand and address the gap between knowledge and practice, and the extent to which people receiving new information are able to act on it to improve nutrition e.g. local government workers with no access to nutritional supplements cannot recommend or deliver these supplements; economic or other constraints may limit how women respond to nutrition‐related counseling. ENGINE has implemented agronomic and cooking demonstrations, and livelihoods support. So far, the rate of adoption is unclear. Agronomic support has included support to farmers with adequate land and water, in the hope that demonstrations effects would increase adoption. ENGINE livelihoods assistance through the MVH program was impressive in terms of delivery, but limited data is available on nutrition or food security impacts. The provision of livestock – a relatively expensive activity – requires attention in terms of preventable livestock mortality, 3.3 Evaluation question 3: Contributions of FTF partners in reducing under‐nutrition There has been notable progress in integrating nutrition‐sensitive thinking and activities into FTF projects, despite the limited attention to nutrition during the design of these projects. Further work is needed to better coordinate these efforts, and to measure the actual rather theoretical impact of nutrition‐related programming (and also multisectoral nutritional convergence) on the nutritional status and food security of target groups, giving special attention to dietary diversity. 3.4 Evaluation question 4: Contributions to gender equity and female empowerment ENGINE has a clear commitment to assuring that many of the services and support under the project are provided directly to women at community level. Given the nature of these services and support, an indirect impact on gender equity and female empowerment can be expected, but with wide variations according to local contexts. In relation to the stunting reduction target of ENGINE and the importance of maternal under‐ nutrition as a primary determinant of stunting, the evaluation team concluded that fasting practices warrant greater attention. ENGINE’s existing links with the Orthodox Church provide a basis for further dialogue and action in this culturally sensitive area. 3.5 Evaluation question 5: Balance of resources against activities and priorities The evaluation team concluded that there was an appropriate balance of resources against prioritized activities in ENGINE. In light of the performance concerns noted in point 3.2 above, ENGINE management should give attention to the workload, catchment areas and supervision priorities of its health and livelihood coordinators in years 4 and 5 to ensure that QI and SBCC activities at the health post level and below are given sufficient time and attention. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 25 | Page 3.6 Evaluation question 6: Management effectiveness and efficiency ENGINE is a well‐managed project, especially considering the large budget it has to manage, and the geographical coverage, partnerships and breadth of activities involved from policy level to community level. The project has responded well at a management level to the partnership and contracting problems which in a project of this size and complexity, can be expected. 4. RECOMMENDATIONS AND NEXT STEPS The evaluation recommendations are drawn from the evaluation findings in section 2, and conclusions in section 3. Most of these recommendations fall well within ENGINE’s capacity. The general, over‐arching recommendations are as follows:  ENGINE is on‐track to achieve most of the targets in its PMP, but specific issues with indicators noted in Tables 1 to 3 should be addressed on a priority basis by ENGINE management.  The ENGINE child stunting reduction objective is overly ambitious and should be changed ‐ a target of 38% stunting prevalence might be reasonable.  Testing multi‐sectoral nutrition convergence (including in existing AGP kebeles where FTF projects are collocated, but also in ENGINE/GRAD efforts in several PSNP woredas) ‐ for consideration in the next NNP.  ENGINE should continue to concentrate on maintaining Ethiopia’s rate of stunting reduction – and seek to prevent plateauing by intensively addressing the 3 potential limiting factors (WASH, maternal malnutrition and poverty/food insecurity) in conjunction with other USAID partners while continuing to focus on the “first 1000 days) and by introducing multi‐sectoral nutrition convergence (interventions from each sector in every village of coverage areas) with diligent M&E, and impact measurement.  Deliberate testing and quantitative assessment of multi‐sectoral nutrition convergence (at kebele level where AGP, FTF and GHI projects are co‐located, and also in ENGINE/GRAD kebeles of PSNP woredas) is recommended to inform the direction of the next NNP and FFT II.  Using food security and dietary diversity data collected annually on all livelihood component beneficiary households, USAID should leverage other donors to participate in this enormously valuable undertaking.  ENGINE should invest strongly from a technical perspective in getting the SBCC program “right” and then go for scale‐up via the government using HDAs and local NGOs.  Iron folate stockouts in the country need to be addressed as a top priority.  Utilizing the convergence approach where possible, FTF projects should seek to better align their nutrition counseling and production/value chain support to beneficiary farmers so that (a) nutrition messages are reinforced and (b) households benefit both from poverty reduction and dietary diversity/nutrition interventions. Future FTF or GHI projects (and USAID Requests for Applications) should incorporate nutrition‐ sensitive interventions with appropriate, international standard indicators from the design stage to avoid current challenges faced by projects in having to “retro‐fit” nutrition into already established food security/agriculture projects. Priority should be given to development of food security and dietary diversity strategies for future FTF projects. As indicated, these recommendations pertain not only to ENGINE’s own current activity portfolio, but also to the design of any follow‐up to the ENGINE project should this materialize, and to ENGINE inputs into larger government systems: the design of AGP II and PSNP II currently underway, and the design of NNP 2016‐20, likely to begin soon. While the specific focus of recommendations is sometimes specified, ENGINE itself and the USAID mission are better positioned than the evaluation team to assess which of the recommendations might best fit into each of these envelopes. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 26 | Page 4.1 Evaluation question 1: Performance against objectives  ENGINE management should pay close attention to the eight targets that are “likely to be achieved” and the four targets which are “uncertain” – noting where mid‐course corrections need to be made in years four and five to ensure achievement. In particular, the indicators “mean number of food groups consumed by WRA” and “VAS coverage” should both be followed closely and specific barriers to achieving improvements addressed in years 4 and 5.  Nine of the indicators highlighted in Tables 1 to 3 are no longer relevant to ENGINE and should be dropped from the PMP. WASH indicators must be updated with appropriate baseline and target information, and gender mainstreaming indicators should be integrated into ENGINE’s PMP as a priority.  ENGINE’s EBF target should be revised to 75%. 4.2 Evaluation question 2: Impact of actions with government and USAID‐assisted partners 4.2.1 IR1 Capacity and institutionalization of nutrition programs and policy strengthened  The sustainability of ENGINE’s initiatives requires explicit inclusion of nutrition into national and sub‐national annual plans and budgets, guided by the specific sectoral nutrition responsibilities delineated in the NNP. As an advocacy priority, ENGINE should continue to advocate for nutrition‐ specific budgeting during the final year and a half of the project – particularly as annual plans are developed at regional, zonal and woreda levels.  ENGINE should continue advocating with the MOA for a designated unit or directorate for nutrition within the Ministry able to work with all three sectors (extension, livestock and DRMFSS/food security.) Recent developments with the Agricultural Extension Directorate are particularly promising. 4.2.2 IR2: Quality and delivery of nutrition and health services improved  There is a need for structural change in the health sector’s system of beneficiary projections and targeting, requiring assistance from USAID and other development partners engaged with this sector.  ENGINE should, to the extent possible given government regulations, increase its attention to health post and community‐based services, utilizing local NGOs as necessary  ENGINE should continue working intensively at the national level with all concerned parties to address the serious problem of inadequate iron folate supply, as well as zinc for treatment of diarrhea.  If stockouts persist, ENGINE should explore with USAID’s implementing partners involved in maternal and child health drug procurement (e.g. JSI Deliver) possible means of addressing bottlenecks, and even consider purchasing iron folate for neediest woredas on a stop gap basis. The iron folate stock out seriously impedes ENGINE’s efforts to achieve its primary objectives.  ENGINE should continue to move forward with its WASH activities – focusing on those that are easily implementable within the scope of ENGINE’s other activities and partnering with other projects that can help address broader WASH challenges requiring hardware inputs that are not affordable for ENGINE. ENGINE’s efforts to become part of the national WASH Coordinating Committee are commendable.  ENGINE field staff should reinforce the work that HEW/HDA’s are doing to inspect latrines and counsel households on necessary latrine improvements and on WASH practices, and assure follow up on compliance.  ENGINE should continue its efforts to work with social marketing partners able to assist HHs with the hardware needed to conduct repairs/upgrades – and to initiate the social marketing of soap. 4.2.3 IR3: Prevention of under‐nutrition through community‐based nutrition  ENGINE and its NGO partners working at the community level should work to strengthen HEWs/HDA capacity to: - Properly weigh and assess young children ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 27 | Page - Use weights as a tool to help them identify growth falterers quickly, and - Provide messages tailored specifically to these children.  If woreda health bureaus do not support ENGINE’s efforts to strengthen growth monitoring and promotion, the evaluation team recommends that child weighing – despite its inclusion in NNP and as an HMIS indicator – be discontinued in the country. The present process of faulty weighing and the recording of flawed data ‐ with no benefit to mother or child ‐ is a waste of time for all concerned, including the overworked HEWs.  Re SBCC materials, the evaluation team is pleased to see that ENGINE and the Manoff Group are committed to avoiding expensive “Cadillac” materials, recognizing also that these materials should be sufficiently practical for use by the Health Development Army  Strategic community‐based SBCC should be a top priority for ENGINE’s workplans in years 4 and 5 in light of the major knowledge‐practice gaps observed in this evaluation, and which constitute a major constraint on malnutrition reduction. The highest priority within the SBCC component, in turn, should be the utilization of a Trials of Improved Practices (TIPS) approach to counseling and group sessions where families and communities are encouraged to try new practices and tailor them to their local context. These approaches have produced highly positive results in Indonesia, Honduras, El Salvador, Nicaragua and Uganda. Strengthening the work that has begun through local NGOs to encourage follow‐up to and more sustained programming in support of the CCs would enhance peer learning and community engagement and, in turn, generate actual behavior change. Building the confidence of HEWs to deliver non‐generic messages designed to address the needs of individual children, and to support the core practices articulated in the CCs would be a first step in an integrated SBCC program. Carrying out these essential steps designed to address knowledge‐practice gaps is far more important than racing to distribute materials.  ENGINE should continue its committed efforts to generate nutrition‐sensitive interventions (livelihoods, agriculture, education and WASH) at local levels. 4.3 Evaluation question 3: Contributions of FTF partners in reducing under‐nutrition  Utilizing the convergence approach where possible, FTF projects should seek to better align their nutrition counseling and production/value chain support to beneficiary farmers so that (a) nutrition messages are reinforced and (b) households benefit both from poverty reduction and dietary diversity/nutrition interventions.  Where possible, FTF projects should use ENGINE’s nutrition‐ agriculture training and SBCC materials rather than developing their own.  Monthly coordination meetings of all FTF project nutrition focal persons at regional levels should be held until coordination issues are adequately addressed.  Future FTF or GHI projects (and USAID RFAs) should incorporate nutrition‐sensitive interventions with appropriate, international standard indicators from the design stage to avoid current challenges faced by projects in having to “retro‐fit” nutrition into already established food security/agriculture projects.  FTC and school agronomic and cooking demonstrations should be coupled with appropriate nutrition training guidelines and explicit attention to adoption rates and to constraints impeding adoption.  ENGINE should continue to facilitate the private sector production of seeds equal in quality to those distributed at ENGINE‐assisted FTC and school agronomic demonstrations.  Future livestock transfers should be preceded by health inspection. Efforts also should be made to provide recipients with access to affordable and reliable basic veterinary care, plus regular deworming and vaccinations after distribution.  USAID & ENGINE can together advocate for a redefinition of agriculture’s role in improving nutrition – both with FTF partners and with the government (next iterations of NNP and AGP) around two primary indicators (a) household food security scores, and (b) dietary diversity scores. ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 28 | Page  ENGINE and its FTF partners should therefore have explicit strategies to reduce household food security and to improve dietary diversity.  These two indicators also should be utilized at entry and in annual monitoring of households being assisted in ENGINE’s livelihoods initiative. Note: This evaluation report contains specific instruments (Annexes 12 and 13) which can be used by ENGINE and all FTF partners to monitor and to evaluate household food security and dietary diversity.  To systematically improve production and dietary diversity in AGP woredas, ENGINE and its FTF partners need to carry out a mapping in each geographic area ‐ and in each of the four major seasons of: - major food group commodities not regularly available in the area; - major food group commodities available but too expensive for purchase by most rural households, and - major food group commodities available but not regularly consumed in the area. A dietary diversity strategy then requires systematic agricultural production efforts and dietary diversity counseling to address the seasonal and geographic area shortfalls identified in order to increase both availability and consumption of a broader diversity of food in the country. Should ENGINE, USAID and/or FTF desire an acronym for this undertaking, the team’s suggestion is ADD AGE (suggesting the actions will increase longevity). The acronym stands for Addressing Dietary Diversity – Assess, Grow and Eat. 4.4 Evaluation question 4: Contributions to gender equity and female empowerment  ENGINE’s Gender Strategy is strong. The project now needs to establish a timeframe for implementing its recommendations, and also report on performance / challenges faced.  During review of the ENGINE PMP, it will be useful to include appropriate gender‐specific indicators to measure, e.g., how women supported by ENGINE have been able to gain greater decision making power within their households.  ENGINE should continue with the MVH program, and if possible in years 4 and 5, document the specific intervention programs (e.g. goats versus a heifers versus chickens) likely to have the greatest impact on female empowerment.  Through the First Lady, ENGINE, which already works closely with the Orthodox Church on this and other issues, may be in a position to influence fasting practices for women of reproductive age, and, at the same time, elicit increased Church support for gender equity. 4.5 Evaluation question 5: Balance of resources against activities and priorities In order to maximize convergence across FTF projects, the evaluation team recommends for both current and future USAID FTF project design:  USAID partners and the government identify a small set (e.g. 3) of woredas where all relevant nutrition‐specific and nutrition‐sensitive projects can provide inputs in all kebeles.  Follow progress carefully with M&E data including frequent direct observations.  ENGINE’s Zonal Coordinators should understand that their primary responsibility is for first hand observation of local level health and agriculture service delivery to determine the real effect of ENGINE training on ultimate service delivery. Other zonal coordinator responsibilities should be reduced (or the number of zonal coordinators increased) to assure that adequate attention is given to this primary responsibility.  For the design of future projects – the lion’s share of operational research funds should be utilized during the project to address, rapidly, problems and questions which arise, and, in turn, to facilitate necessary implementation changes – rather than specify all research activities at project inception. 4.6 Evaluation question 6: Management effectiveness and efficiency No recommendations ENGINE, Ethiopia - External Mid-Term Performance Evaluation Report 29 | Page 4.7 Additional recommendations  The Ethiopian Government, at the urging of ENGINE, now uses the term Maternal, Adolescent, Infant and Young Child Nutrition (MAIYCN). This suggests that ENGINE continue to encourage its active promotion in the next NNP iteration in addition to the present focus on the first 1000 days.  To further enhance the involvement of fathers in the young child’s psycho‐social development (closely associated with nutrition), ENGINE should consider incorporating into its SBCC strategy a highly successful Jamaican model, with ways for a father to mark on paper each time he plays with, tells stories to, or sings to the young child (6 months to 2 years.)  Regarding the ENGINE endline evaluation presently scheduled for 2015/2016, the evaluation team questions whether it will be useful to repeat at that time the expensive and time‐consuming survey protocol used in the baseline survey. With baseline data collection completed only in 2013, and with the full rollout of the new SBCC activity and WASH not likely until late 2014,36 the question becomes more pertinent. In order to permit observation of the impact of the entire project on stunting, full implementation of all project components at scale will be necessary for at least three years.37  The First Lady is a genuine champion of nutrition in Ethiopia, but has only one assistant. It is suggested that ENGINE consider seconding a nutrition assistant to her office. 36 Note additionally that adoption of even the best SBCC messages rarely takes place quickly. 37 A decision on evaluative data collection may be influenced by decision making on extending the ENGINE contract. An affirmative decision would permit extensive data collection in, e.g. 2016‐17 with minimal data collection in 2015‐16 (e.g. limited perhaps to anthropometric measurements and food security/dietary diversity assessments.)