EXTERNAL MID-TERM EVALUATION of USAID Ethiopia’s Empowering New Generations to Improve Nutrition and Economic Opportunities (ENGINE) Project Intermediate Result 4: Rigorous and Innovative Learning Agenda Adopted August 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Tanya M. Trevors. 2 3 External Mid-Term Evaluation of USAID Ethiopia’s Empowering New Generations to Improve Nutrition and Economic Opportunities (ENGINE) Project Intermediate Result 4: “Rigorous and Innovative Learning Agenda Adopted” FINAL REPORT Prepared under Purchase Order AID-663-O-14-00035 Submitted by: Tanya M. Trevors, Independent Consultant Front Cover Photos: ENGINE staff conducting household level survey interviews using tablets for data collection as part of the Birth Cohort Study in Amhara Region. Photo credit: Tanya Trevors DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 4 ACKNOWLEDGEMENTS The evaluator deeply appreciated the support that she received from Mary Harvey at USAID/Ethiopia as well as from the ENGINE headquarter and regional staff teams. In particular, the time and assistance of Dr. Habtamu Fekadu, ENGINE Chief of Party; Niguss Berihan, M&E Coordinator, Amhara Region; Ketema Kebebew, M&E Coordinator, Oromia Region; and Solomon Sermisa, ENGINE Livelihoods Coordinator, SNNP Region, and Dr. Beyene Wondafrash, Principal Investigator, Birth Cohort Study, Jimma University. She would also like to thank Mr. Basia Benda from Valid International and Peter Walker and Jennifer Coates from Tufts University for their time and sharing of documents. 5 CONTENTS Executive Summary………………………………………………………………………………………………….…..…. 5 Introduction………………………………….………………………………..………………………………………….…… 7 Background…………………………………………………………………………………………………….………. 7 Methodology………………………………………………………………………………………………............. 7 Scope of Work…………………………………………………………………………………………………........ 9 Limitations of the Study………………………………………………………………………………….………. 9 Evaluation Findings………………..……………………………………………………………………………………….. 9 ENGINE’s progress towards its IR4 targets………….………………………………………………….. 9 Management of ENGINE’s baseline survey and operational research components. 12 Balance of Resources: Operational Research and Knowledge Sharing ………………….. 14 ENGINE’s engagement of local research institutions and academic institutions …… 15 Effectiveness of ENGINE’s PhD scholarship program under IR4…………………………….. 15 Balance between ENGINE’s health-nutrition and agriculture-nutrition research agendas ……………………………………………………………………………………………………………….. 16 How well is gender being addressed by IR4 …………………………………………………………. 17 Evaluation Recommendations…………………….…………………………………………….…………………… 17 Annex A: Evaluation SOW……………………………………………………………..……………………………… 20 Annex B: Evaluation Interview Questions………………..……………………………………………………. 35 Annex C: List of documents reviewed for the mid-term evaluation………………………………… 36 Annex D: Agriculture-Nutrition Panel Study Priority Research Questions ……………………… 37 6 EXECUTIVE SUMMARY The overarching goal of the USAID funded ENGINE project is to contribute to the reduction of maternal and infant mortality by improving the nutritional status of women and children under-5 through sustainable, comprehensive and coordinated evidence-based interventions. ENGINE’s Performance Management Plan (PMP) has four Intermediate Results (IRs) areas within which activities have been designed and are being implemented to achieve targets on agreed indicators between July 2011 and September 2016. ENGINE’s IR4, “Rigorous and Innovative Learning Agenda Adopted”, is the focus of this external mid-term evaluation report. It encompasses ENGINE’s monitoring and evaluation (M&E), operational research (OR) and knowledge management components, which collectively aim to foster evidence-based learning and practice both within and beyond the project, and to track whether ENGINE is on course to achieve its overall goals and objectives. Evaluation findings ENGINE is on track to achieve its performance targets under IR4. Tufts University and Valid International have provided high caliber technical leadership to develop and guide ENGINE’s M&E and OR agendas, engaging global nutrition experts as principal investigators for the research design and partnering effectively with local Ethiopian research institutions. 26% of ENGINE’s budget for years 1-3 was used for activities within IR4, a sizeable investment within the overall budget. The design of ENGINE’s primary research agenda, while technically very strong, was found to be weak in two minor areas: a) it lacks a knowledge management and dissemination plan linked directly to the OR and M&E agendas to summarize and package information generated by the project for both internal and external stakeholders; and b) the design, financing and timing of the operational research program does not lend itself to easily address small project-related questions that arise during the course of implementation to enable mid-course corrections to be made. The evaluator found ENGINE to have a well-designed, statistically robust baseline survey. Preliminary and final results have been and continue to be shared internally and externally to inform the development and roll-out of the project. Extensive dissemination of appropriately summarized and translated baseline survey findings to relevant stakeholders at the sub-national level however has not yet been carried out in a comprehensive way. The ENGINE funded PhD capacity building program managed by Tufts University is underway for seven Ethiopian students to pursue studies related to ENGINE’s mandate. Tufts University significantly under-budgeted for the PhD program initially and Jimma University faced difficulties in finding placements for students with international university research advisors due to budget and planning constraints. It was also noted that despite specific guidance from ENGINE to select female PhD candidates, none were found. 7 Recommendations Within IR4, the ENGINE project has made important investments in high quality OR and rigorous M&E. In order to maximize ENGINE’s return on these investments, this mid-term external evaluation recommends that the following actions be taken by ENGINE and USAID during the remaining two years of the project, and beyond: 1. Within ENGINE’s year 4 and 5 workplans, allocate a specific budget for the development and implementation of a very strategic and targeted knowledge management and dissemination plan for the project (see page 18 for detailed recommendations). 2. The timing of ENGINE’s endline survey should be carefully considered to align precisely with the months when the baseline survey was conducted to ensure comparability of the two surveys and avoid bias related to seasonality. 3. For the design of future USAID-funded nutrition projects, this evaluation recommends a modified OR design which would allow for ad hoc research questions to be studied throughout the course of project implementation and used to make mid-course corrections. 4. Specific lessons learned from the budgeting, planning and implementation of ENGINE’s capacity building and PhD components of IR4 should be documented and shared by Tufts University with ENGINE management and USAID to inform the OR aspects of future USAID-funded nutrition projects involving partnerships between US-based and local research institutions. 5. As pointed out in ENGINE’s Gender Mainstreaming Strategy (March 2014) the project should move swiftly to integrate gender sensitive indicators into the project PMP in advance of year four. Concrete actions to address specific barriers related to women’s poor nutritional status in woredas (as documented in the ENGINE baseline survey) should also be prioritized within the upcoming year four workplan. 6. For future USAID-funded nutrition projects, this evaluation recommends that the approach of the PhD program be modified to encourage more female participation and require the selection of thesis topics more directly related to practical implementation aspects of the nutrition project funding the candidates. 8 INTRODUCTION Background The external mid-term performance evaluation of the USAID/Ethiopia-assisted ENGINE program took place from June 3rd to July 2nd 2014. The evaluation was guided by a set of six questions provided by USAID which had the overall purpose of assessing the direction of the program, its management, the progress being made toward its 2016 targets, and its integration, harmonization and coordination with other nutrition￾related programs in Ethiopia, including USAID-assisted Feed the Future (FTF) and health projects, and the Government of Ethiopia’s National Nutrition Program (NNP). The USAID Agriculture Knowledge Learning and Documentation Project (AKLDP), implemented by Tufts University, was contracted by USAID Ethiopia to lead the external evaluation looking specifically at performance under intermediate results (IRs) 1-3 of ENGINE’s Performance Measurement Plan (PMP). ENGINE’s Results Framework is included on page 8. Another consultant (not affiliated with Tufts University), was hired by USAID Ethiopia to work with the team on IR’s 1-3 and to also independently evaluate ENGINE’s performance under IR 4, “Rigorous and Innovative Learning Agenda Adopted”. This was done to overcome the potential for conflict of interest given Tufts University’s role as an implementing partner under IR4. Methodology The consultant conducted interviews for the IR4 evaluation at the national, regional and sub-regional levels with ENGINE project staff, government officials, local university faculty and students, IR4 implementing partners (e.g., Tufts University and Valid International) and collaborating partners. Questions used were determined in advance by the consultant and applied systematically throughout all interviews conducted. See Annex A for the list of interview questions. The IR4 interviews were carried out separately from other interviews, without other Tufts University evaluation team members being present. Stakeholders not in Ethiopia during the evaluation period were sent a list of interview questions by email and where possible were also interviewed using Skype. The consultant also carried out a secondary data review of relevant ENGINE project documents related to IR4. The list of documents reviewed is included in Annex C. 9 ENGINE Results Framework: Scope of Work This report IR1: Capacity for and institutionalization of nutrition programs and policies strengthened 1R2: Quality and delivery of nutrition and health care services improved IR3: Prevention of undernutrition through community-based nutrition care practices improved IR 1.1 Nutrition policy environment strengthened  Strengthening existing coordinating mechanisms  Supporting development and revision of policies, guidelines, and standards IR 1.2 Pre-service and in￾service nutrition training for health care agents strengthened  Pre-service education  In-service capacity building  Strengthening learning networks IR 2.1: Quality of nutrition services strengthened  Facilitating integration of quality improvement processes  Building capacity of health facility staff and frontline workers to provide high￾quality services IR 2.2 Enhanced health and nutrition service seeking behaviors  Developing SBCC strategy  Mobilizing communities  Educating on micronutrients IR2.3: Access to health and nutrition services increased  Linking with other partners  Strengthening referral system  Establishing sites to treat acute malnutrition  Supporting improvements in supply and logistics IR3.1 Maternal, IYCF knowledge and practices improved  Developing SBCC strategy  Promoting optimal nutrition practices through different communication channels IR 3.2 Access to food and economic strengthening opportunities through programming and cross sectoral linkages increased  Applying economic strengthening interventions to meet households’ needs  Facilitating community-based learning on agriculture techniques for increased production of diverse foods Strategic Objective: Improved nutritional status of women and young children through sustainable, comprehensive, coordinated, and evidence-based interventions IR4: Rigorous and innovative learning agenda adopted Activity 4.1: Design and Delivery of Research Strategy: Research goal: To generate rigorous learning outcomes on key questions and innovative policy and operational approaches that will support the successful scale up of programs to improve nutrition in Ethiopia and a subsequent reduction in maternal and child undernutrition. Themes to be considered: Research relevant to IR1:  Strengthening the nutrition and HIV nutrition policy environment Research relevant to IR2  Improving the quality of nutrition services in Ethiopia  Improving the effectiveness (coverage and efficacy) of programs to address acute malnutrition  Improving coverage and quality of preventative nutrition and related services for mothers and young children Research relevant to IR3  Improving the effectiveness (coverage and efficacy) of interventions that aim to prevent under-nutrition and micronutrient deficiencies  Improving and strengthening explicit nutrition outcomes in multi-sectoral programs Activity 4.2: Develop and Manage an Innovative Documentation and Dissemination Strategy 10 focuses exclusively on ENGINE’s mid-term progress and performance under IR4. It compares results achieved over the first three years of implementation (September 2011 to June 2014) against the five-year targets detailed in ENGINE’s Results Framework (above). The report also points out key management and performance factors to date that have either contributed to or inhibited the achievement of agreed objectives in a timely fashion under IR4. Finally, the report makes recommendations for how the Project and its implementing partners may be able to adjust and improve performance over the remaining period of the agreement or for future USAID-funded nutrition projects, looking also at the balance between the resources (staff and budget), their management and the activities the project has accomplished so far under IR4. Limitations of the Evaluation Due to the timing of the evaluation it was not possible to interview all stakeholders in person in Ethiopia. Where this was not possible, stakeholders were contacted and interviewed either by phone or an exchange of emails with the standard questions. EVALUATION FINDINGS ENGINE’s progress towards its IR4 targets Overall ENGINE is on track to achieve its performance targets under IR4. Table 1 below provides a summary of the progress made to date against each indicator. Table 1: ENGINE indicators for IR4 Indicator Baseline Latest Data Target Likelihood of achievement1 (0-5 scale) # of operations research studies conducted 0 3 10 5 # of new policy/strategies/interventions adopted as a result of learning generated 0 1 1 5 # national review meetings conducted 0 1 5 4 # of regional review meetings conducted 0 0 8 4 # of program managers trained in data utilization 0 431 500 5 As shown above, the weakest performance areas under IR4 pertain to the number of national and regional review meetings conducted. According to ENGINE management only one national review meeting has 1 A score of 0 means the target is likely to not be achieved whereas a score of 5 means that there is a very high likelihood that the target will be achieved. 11 been conducted so far and regional review meetings have not yet started. The evaluator was informed that the late initiation of the regional review meetings was due to a misunderstanding by regional management about the objective of these meetings in the workplans and has since been corrected. ENGINE management expressed confidence that both of these indicator targets could still be achieved before the end of year five by accelerating and increasing the number of review meetings within the remaining two year workplans. Given this information, these indicators were assigned a scoring of 4 out of 5. In terms of ENGINE’s OR agenda, three secondary data analysis research studies (see Table 2 on page 11) have been completed so far and five additional primary research studies are ongoing, each of which encompasses a number of different research sub-questions being studied. For example, the Agriculture￾Nutrition Panel Survey led by Tufts and Jimma Universities includes ten studies(see Annex D for the complete list). According to ENGINE, Tufts University and Valid International staff interviewed, data collection for all of the primary research studies is due to be completed before the end of 2015, and formal reports are to be submitted in advance of August 2016. Given this information, the OR study indicator was assigned a score of 5 out of 5. ENGINE’s other significant PMP target under IR4 was to develop at least one new policy or strategy. This target has been achieved under ENGINE’s support in years one and two to help the Government of Ethiopia draft and finalize its NNP. Through a review of ENGINE’s annual reports and interviews with ENGINE staff, the evaluator noted two main areas where ENGINE staff worked successfully to incorporate their learning into the NNP: 1. Evidence sharing: ENGINE presented its baseline and secondary research analysis findings to NNP planning meetings and different working groups2 to emphasize: a) the magnitude of stunting in both food secure and insecure woredas; b) the importance of maternal nutrition and the 1000 day window; and c) the key determinants of stunting in Ethiopia requiring multi-sectoral interventions to be undertaken. 2. Experience sharing: Lessons learned from ENGINE’s first two years of implementation experience were shared via NNP technical working groups as they developed the nutrition sensitive component of the NNP. ENGINE’s experience appeared to be most influential with regards to informing the key roles of the agriculture and education. ENGINE is also on its way to achieve its 5th PMP target, “ Number of program managers trained in data utilization”, ahead of schedule as this is a routine activity that has been well implemented thus far at the sub-national level. 2 ENGINE staff participated in 3 out of the 4 NNP Working Groups. 12 Table 2: ENGINE funded research Operational Research Purpose Principal Investigator Institution(s) Completed? Secondary Data Analysis Research Factors associated with stunting in Ethiopian children under-five To examine risk factors for stunting in Ethiopian children using DHS data from 2011, and to examine the trends, variability and changes in these factors over time usingi the 2000 and 2005 DHS. Dr. Shibani Ghosh Tufts Yes. Findings presented to USAID (March 2014) and 3 abstracts submitted to the journal “Experimental Biology”. Agriculture commercialization, production and consumption diversity To examine whether agriculture commercialization, production and consumption diversity are each associated with greater household dietary diversity. Dr. Jennifer Coates Tufts Yes. Publications forthcoming. Nutrition policy study To provide evidence to explain why large-scale policies and programs targeting improved nutrition achieve their goals or not in various settings. Dr. Elileen Kennedy Masersha and Tesfays Tufts, EPHI Yes. Abstract presented at June 2014 Micronutrient Forum Conference. Primary Research Agriculture-Nutrition Panel Survey To examine the role of ENGINE in affecting nutrition, food security and livelihood outcomes through its integrated programming. Tufts: Dr. Jennifer Coates Jimma: Dr. Beyene Hawassa: Alemzewed Tufts, Jimma To be completed by August 2016. Birth Cohort Study To assess the effectiveness of direct and indirect interventions targeting maternal and child nutrition and their health outcomes Tufts: Dr. Shibani Ghosh Jimma: Prof Tefera Hawassa: Debebe Tufts, Jimma, Hawassa To be completed by August 2016. SAM Cohort Study To determine the long-term health outcomes of children aged 6-59 months who are successfully treated for SAM in a community-based management program. Valid: Plauke and Kate Sadler Jimma: Dr. Tsinuel Valid, Jimma To be completed by August 2016. MAM Cohort Study To provide evidence of the need for a Targeted Supplementary Feeding Program in food-secure settings of rural Ethiopia. Valid: Kate Sadler Jimma: Professor Tefera Valid, Jimma To be completed by August 2016. 13 Management of ENGINE’s baseline survey and operational research components Baseline survey The ENGINE baseline survey was well designed, statistically robust, and all required FTF indicators were incorporated into the design alongside more specific ENGINE project indicators. Of note was that the survey used a stepped-wedge approach, rolling out as the ENGINE project scaled up, and allowing for early survey results to feed into program management decisions during years one and two. Valid International and Tufts University staff interviewed all emphasized that the endline survey must follow the same monthly data-collection timeline as the baseline survey, allowing seasonality to be properly accounted for. Data collection for the ENGINE baseline survey started in June 2012 and was completed in September 2013. The interim year one report on the baseline survey was provided to ENGINE management on time, while the final baseline survey report was completed in February 2014, approximately five months behind schedule. According to the information shared with the evaluator, delays encountered with initiating the baseline survey were mainly related to: a) additional time required to obtain consensus between USAID and ENGINE partners on which USAID/FTF indicators should be included in the survey, and sampling methodology3 ; and b) time required to finalize the sub-agreement between SCI and Valid International because Valid’s registration as an international NGO was not accepted. Some expected delays were also incurred during the survey roll-out due to difficult access encountered in some communities caused by rains during the data collection period; and additional time was required by Valid International during the final report writing period to address a request for geographical representation of all of the baseline indicators to be included in the final report. The delays incurred were mainly technical in nature and, from the perspective of this evaluation, did not affect the quality of the data collected or the robustness of the overall survey. See figure 1 for a visual timeline of the survey roll out. Operational Research (OR) This evaluation found that Tufts University and Valid International did a formidable job in managing ENGINE’s research agenda development through a consultative process in country with government stakeholders, the Ethiopia Public Health Institute (EPHI) and local universities (Jimma and Hawassa Universities). As summarized in Table 2, the consultation process resulted in a set of comprehensive research studies that are being explored through three secondary data analysis research studies (already completed) and four primary research cohort studies and a panel survey (ongoing). As mentioned in the above section, the project is on track to meet its target of producing ten studies. Not captured by this indicator however is the late timing of when these primary OR studies will be finalized and available for knowledge sharing and dissemination to key stakeholders both internal and external to the 3 FtF M&E guidance on nutrition indicators and population based surveys came out after ENGINE had finalized its baseline survey indicators and design in early 2012. 14 project. As faced with the baseline survey, ENGINE’s OR agenda also faced delays in getting started. Two main reasons noted by ENGINE stakeholders for the OR start-up delays included: 1. Institutional (local and international partner) delays involved in: a) establishing partnership agreements between universities; b) obtaining internal process and ethical review board approvals; and c) reaching consensus among partners regarding research design and methodology. 2. For all primary research studies, the recruitment of survey staff and households took longer than anticipated. One minor downside noted by the evaluator about the design of ENGINE’s OR component was that there is little room (e.g., budget or time) for other research questions that come up during the course of project implementation to be explored or studied in detail. All three of the regional ENGINE M&E Advisors interviewed expressed that they were not conversant with ENGINE’s OR agenda and/or the findings of the studies which had recently been completed by Tufts University. They were familiar however, with the baseline survey findings and expressed interest in doing more OR that could help them to improve the effectiveness, reach and sustainability of, for example, ENGINE’s livelihood-support programs. Figure 1: ENGINE timeline for baseline survey and primary research: Other management issues In the areas of staffing and budget management, Tufts University faced budget pipeline challenges in years one and two tied to the slower than anticipated start up of ENGINE’s research activities (see above). Given the situation, Tufts University increased its spending in year two which had consequences for outward year budgeting and planning for the primary research, the PhD program as well as for Tufts staff salaries and international travel. Sept 2011 Sept 2012 Sept 2013 Sept 2014 Baseline survey conducted June 2012 – Sept 2013 Planned start  of baseline survey Feb 2012 Project  start Date Project  end Date Sept 2016 Endline survey to be conducted Initiation of SAM/MAM studies (Aug/Sept 2013) Baseline Survey Delay Start of data collection: Birth Cohort Study and Ag-Nut Panel Survey (Mar/Apr 2014) Cohort & Panel Study Delays Final Baseline Survey Report Feb 2014 15 As a cost saving measure for years 4 and 5, Tufts University terminated its Senior OR Advisor position, Tarik Kassaye, who has been based in the ENGINE project since year one. As of September 2014, the ENGINE research advisory role will thus be covered by a US-based Tufts University assistant researcher, Meghan Loraditch, who will oversee the primary research studies and travel on a part time basis to Addis Ababa. Dr. Cherinet, ENGINE’s Senior Research Advisor, will also continue to serve as a liaison between Tufts University and ENGINE management at the office level. Based on interviews with ENGINE and Tufts University Staff, the evaluator concluded that cutting the OR Advisor Position will not likely have negative implications on the quality of the research but the additional workload falling on Ms. Loraditch and Dr. Cherinet will need to be supervised appropriately by Tufts University and ENGINE management. The other significant management issue faced by the ENGINE project concerned its partnership agreement with Valid International which learned in 2013 that it would not be able to continue working as an international non-governmental organization in Ethiopia. Given the situation, SCI was forced to alter its working agreements with Valid International and Valid International’s partnership sub-agreements signed with local universities to implement of the SAM and MAM cohort studies were terminated. This situation resulted in a delay in the SAM/MAM research studies and an additional time burden for SCI in assuming their management responsibility. Balance of resources allocated: Operational research and knowledge sharing ENGINE’s IR4 budget in years 1-3 represents 26% of its total budget for that timeframe. IR4 covers activities related to: a) the monitoring of all project activities (e.g., national, regional and woreda level review meetings, supportive supervision at woreda level, and special assessments such as the Participatory Impact Assessment); b) project impact evaluation (e.g., the baseline and endline surveys), c) support to the PhD program and iv) the OR. The original technical proposal for the ENGINE project stated: “The work under IR4 will include design and delivery of an OR strategy and of an innovative and effective documentation and dissemination strategy for learning generated under ENGINE”. It also mentions that the project would “develop an innovative range of verbal and non-verbal materials and approaches designed to reach local stakeholder groups”. This evaluation of IR4 however found that the knowledge management and dissemination components – planned to meaningfully ensure that the research and information generated by ENGINE would be disseminated to appropriate stakeholders both internal and external to the project at country level - were not actively followed up on by Tufts University or ENGINE management within annual workplans and budgets. Communication with Tufts University revealed that annual stakeholder workshops to expand dissemination of ENGINE’s OR were discussed internally, but never formalized in their annual budgets and/or workplans. Related to knowledge management, Save the Children International did contract a consultant in 2013 to develop a “Recommendations Paper for Developing ENGINE’s Overall Communication Strategy” in 2013 and subsequently also hired a Communications Officer in early 2014 to: a) oversee the overall project communications strategy; b) enhance internal project communications; and c) strengthen external project 16 communications. The evaluator noted however that neither the terms of reference for the Communications Officer nor the communications recommendations paper contain wording related to the knowledge management or dissemination needs of the OR component of ENGINE. Further review of ENGINE’s annual workplans and semi-annual reports for the project’s first three years revealed some knowledge management related activities, including joint biannual review meetings between ENGINE and government (the next one planned for September 2014) and periodic regional review meetings conducted to discuss lessons learned with government and other partners at the sub-national level. What appears to be missing from the outset however was a specific activity in the workplan for ENGINE to develop a guiding plan for knowledge management and dissemination – particularly given its sizeable M&E and OR investments. Despite the absence of a formal knowledge sharing strategy within ENGINE, interviews with ENGINE zonal coordinators by the evaluator in all three regions visited revealed an informal culture of “learning by doing” and sharing best practices with government health and agriculture staff during quarterly zonal and woreda level review meetings. The interest and eagerness of ENGINE’s zonal coordinators to share and apply what is working within the project, reinforces the need for at least a basic knowledge sharing and dissemination plan to their level within the project. ENGINE’s engagement of local research institutions and academic institutions IR4 was appropriately designed and implemented from the outset to ensure that Tufts University and Valid International partnered meaningfully with local universities to foster capacity building and ensure that local context was considered. Partnership sub-agreements were signed between Valid International and Tufts University with Jimma and Hawassa Universities and EPHI, and overall positive working relationships appear to have been developed throughout the research program implementation. Principal Investigators from Tufts University and Valid International provided technical support and capacity building to faculty and students primarily in the areas of research design and methodology. ENGINE partners interviewed all noted that the capacity building component has been a time consuming effort but one that has been very appreciated by all. Capacity building through the ENGINE funded PhD program is described below. Effectiveness of ENGINE’s PhD scholarship program under IR4 Under IR4 Tufts University managed a PhD scholarship program for seven male Ethiopian students from Jimma and Hawassa Universities, EPHI and Tufts University to pursue studies related to ENGINE’s mandate. Despite specific guidance from ENGINE management to identify female PhD candidates for the scholarships, none reportedly were found. Some not insignificant delays were faced by Tufts University in matching ENGINE funded PhD students with advisors from international universities but as of July 1, 2014 all seven students were enrolled and had advisors. An important hurdle noted by the evaluator was that Tufts University initially under-budgeted for 17 the PhD program and requested a budget increase4 . Even with the increase, efforts to partner some Ethiopian students with US universities (e.g., the Oklahoma University) were not possible due to their relatively high tuition costs. In the end, Jimma University successfully matched all of the PhD students with suitable advisors from (more affordable) non-US universities, such as Ghent University in Belgium. In terms of the research topics studied, five of the seven PhD students chose topics related to traditional health-nutrition areas while the remaining three are studying nutrition questions related to intra￾household dietary diversity and allocation of household financial resources between men and women. Across the PhD candidates funded by ENGINE there appears to be a bias towards nutrition-specific study topics over nutrition-sensitive ones. This bias should be addressed within the PhD program of future projects. Balance between ENGINE’s health-nutrition and agriculture-nutrition research agendas The evaluator reviewed and found there to be an adequate balance of funding for ENGINE’s health￾nutrition and agriculture-nutrition research agendas. The agriculture panel survey, the birth cohort study and the SAM/MAM cohort study have each been designed to assess ENGINE’s performance under both agendas as summarized by some of the questions below. The agriculture panel survey, for example, is assessing ENGINE’s role in affecting nutrition, food security, and livelihoods outcomes through its integrated programming, by answering the following questions: o To what extent, and through which pathways, do ENGINE’s nutrition-sensitive agricultural interventions improve food security, dietary intake, and nutrition outcomes, and for whom? o What factors predict the adoption and sustainable application of nutrition-sensitive agricultural practices? Part of this analysis will explore household decision-making related to market engagement and the use of income from agricultural sales for improved dietary quality. o To what extent have agriculture extension workers integrated nutrition-sensitive approaches and messages into their work? What do they perceive as key barriers or facilitators to doing so? and o Why does stunting persist in areas (and in households) of relative food surplus? ENGINE’s birth cohort study is assessing how and why specific strategies and approaches work to address nutrition and health concerns of vulnerable groups, namely, pregnant women and infants. o What is the added effect of (improved) livelihoods on maternal and infant nutrition and health outcomes? (e.g. ENGINE/AGP interventions). o What is the effect of direct nutrition interventions on maternal and infant nutrition and health outcomes? o What is the added effect of WASH on maternal and infant nutrition and health outcomes? 4 According to ENGINE staff, initially USD 100,000 was budgeted by Tufts University but by the end of the program approximately USD 398,000 was needed for the 7 PhD students to complete their studies by 2016. As a result the project was forced to adjust other budget line items so that the additional balance of USD 298,000 would be covered. 18 o What are the factors associated with adherence/participation in integrated programs such as ENGINE? o What are the barriers and facilitators to effective uptake of services delivered by integrated programs such as ENGINE? How well is gender being addressed under IR4? Gender and women’s empowerment have been strongly rooted in the design of the secondary data analysis and primary research studies funded by ENGINE. The PhD (and MSc) programs funded by ENGINE, although designed to encourage women applicants, however did not succeed in having an appropriate balance of male and female candidates. As mentioned above, there are no female students in the ENGINE funded PhD program. ENGINE finalized its Gender Mainstreaming Strategy (March 2014), which identified the need for additional gender sensitive indicators to be included in the PMP. Another important gender-related effort within ENGINE noted by the evaluator was that the baseline survey was well designed to highlight woredas where specific gender-related program considerations are needed (for example, woredas with exceptionally high maternal undernutrition and low dietary diversity). Information on women’s nutritional status, infant and young child feeding, sanitation practices, dietary diversity and healthcare coverage have important potential to inform ENGINE’s implementation and workplannng over the remaining two years. RECOMMENDATIONS Within IR4, the ENGINE project has made significant investments in high quality operational research and rigorous monitoring and evaluation. In order to maximize ENGINE’s return on these investments, the evaluator recommends that the following actions be taken by ENGINE and USAID during the remaining two years of the project, and beyond: 1. Under IR4, the ENGINE project is generating a vast amount of cutting edge, high quality research on nutrition and related best practices in Ethiopia. Having a simple tool in hand to manage and share this information with relevant stakeholders both internal and external in a meaningful way is both desirable and beneficial to ENGINE’s overall mandate. That said, developing yet another strategy or plan in year four has the potential to be costly, time consuming and yield risky returns. Bearing these factors in mind, the evaluation recommends that ENGINE management start with a simple team discussion or brainstorming session to: a) identify who are the most important two internal and two external target audience groups for the OR and M&E information being generated by ENGINE? b) what tools does ENGINE already have at its disposal to reach these 4 audience groups in an easy, appropriate and sustainable way?; c) what additional financial resources are available for this effort?; d) how will the knowledge management and dissemination link with and reinforce ENGINE’s broader communication efforts and new Social and Behavior Change 19 Communication (SBCC) Strategy; and d) who is best positioned within the team to spearhead the work?5 . In terms of resources, the knowledge management plan generated could utilize existing relevant on-line platforms and tools (e.g, webinars, knowledge databases, social media etc) and engage other appropriate USAID and/or Tufts University partners also involved in knowledge management and dissemination (e.g., USAID Learning Lab, SPRING). 2. The timing of ENGINE’s endline survey should be carefully considered to line up exactly with the months when the baseline survey to ensure comparability of the two surveys and avoid bias related to seasonality. 3. For the design of future nutrition projects, allow space within a smaller overall research agenda for ad-hoc research questions to be studied throughout the course of project implementation to inform project operations and allow for mid-course corrections to be made. 4. Lessons learned from the budgeting, planning and implementation of ENGINE’s capacity building and PhD components of IR4 should be documented and shared by Tufts University to inform the OR design within future USAID-funded nutrition projects involving partnerships between US-based and local research institutions. 5. As pointed out in ENGINE’s Gender Mainstreaming Strategy (March 2014) the project should give priority to identifying and integrating gender sensitive indicators into the project PMP. Concrete actions to address specific barriers to women’s nutritional status at the woreda-level generated by the ENGINE baseline survey should also be addressed as a priority within the upcoming year 4 workplan. 6. For future projects, in place of the current PhD program model, the evaluator recommends a nation-wide PhD and MSc Scholarship competition model whereby eligible students would be invited to put forward thesis ideas that link directly with the goals and objectives of the project. To address the gender imbalance faced by the current ENGINE funded PhD program, the new model should require that at least 50% of successful students be women and could also allow for successful graduates to intern with the project either during or immediately following completion of their studies. 5 USAID and on-line resources to assist with developing a knowledge management and dissemination plan: http://www.usaid.gov/results-and-data/information-resources/knowledge-management-supportl http://www.nccmt.ca/registry/view/eng/111.html 20 ANNEX A STATEMENT OF WORK PERFORMANCE EVALUATION SUPPORT TO EMPOWERING NEW GENERATIONS TO IMPROVE NUTRITION AND ECONOMIC OPPORTUNITIES (ENGINE) PROJECT I. INTRODUCTION The purpose of this performance evaluation is to examine what the ENGINE Project has achieved at the mid-way point in implementation; how well it is being implemented; how it is perceived and valued; whether expected results are occurring or are likely to occur before the end of the project; and to assess the management and operation of the project. The findings, analysis and recommendations are intended to inform and improve implementation of ENGINE in the remainder of the Project’s life. Cooperative Agreement: 663-A￾00-11-00017-00 Start Date September 27, 2011 End Date September 26, 2016 LOP Budget $ 50,891,422 USAID’s evaluation policy encourages independent external evaluations to increase accountability to inform those who develop programs and strategies, and to refine designs and introduce improvements into future efforts. In keeping with that aim, this evaluation will be conducted to review and evaluate the performance of the USAID-funded Empowering New Generations to Improve Nutrition and Economic Opportunities (ENGINE) Project implemented by Save the Children International. The evaluation will focus mainly on assessing, performance in achieving the Project’s goal, objectives, and results. In addition, the evaluators will assess the progress made with the integration of nutrition into the Feed the Future Projects that are contributing to IR 5, Nutritional Status of Women and Young Children Improved, of Development Objective One, Increased Growth and Resiliency in Rural Ethiopia, of the Mission’s Country Development Cooperation Strategy (CDCS). Theory of Change The determinants of malnutrition are multifaceted stemming from individual health status to household food access, to social, economic, political, and environmental factors at national and global levels. The most immediate causes of under-nutrition in children are insufficient energy and other nutrient intake combined with infectious diseases, especially diarrhea (Black et al., 2008; Black et al., 2013). A household’s access to safe, healthy, and diverse foods, health services, and safe water and sanitation underlie adequate nutrition (UNICEF, 1998). Fundamental to these factors is a complex array of determinants including women’s empowerment, care giving practices, education, the economy, political situation, socio cultural factors, and the environment. 21 CONCEPTUAL FRAMEWORK Adapted from UNICEF and Lancet 2013 Optimal Nutrition Adequate dietary intake Low disease burden Sustainable Household food security Appropriate maternal and infant and young child feeding and care Effective health services Socio-cultural, economic, and political context Healthy environment Access to water& sanitation Stability and Resilience Country commitment and capacity, leadership, financial resources for nutrition, knowledge and evidence, health and food systems, trade/private sector development Availability of Nutritious Foods Delivery of essential health & nutrition services, including family planning and reproductive health Adequate time, space, & support for care. Appropriate education & knowledge Appropriate Hygiene Practices Access to Nutritious Foods Gender equality& women’s empowerment Nutrition Enabling Environment Morbidity & Mortality Cognitive, Motor, and Socio-Emotional Development School Performance and Learning Capacity Adult Stature Obesity Work Capacity/Productivity Illustrative examples of interventions/services • Agriculture Production/Income Generation for Dietary Diversity • Food Processing • Postharvest Storage • Food Fortification • Targeted Livelihood Activities • Risk Mitigation Interventions • Social Protection and Safety Nets • Bio fortification • Early, Exclusive, &continued Breastfeeding • Appropriate Complementary Feeding • Feeding During Illness • Dietary Diversity for Pregnant and Lactating Women • Maternal supplementation • Caregiver support and protection • Early childhood care &development • Treatment of Moderate and Severe Malnutrition • Micronutrient Supplementation or Fortification • Nutrition Management of Infectious Diseases (e.g. HIV/AIDS, TB, Diarrhea) • Prevention and Treatment of Infectious Diseases (e.g. ITNs, antimalarials in pregnancy) • Family Planning and Reproductive Health Services • Deworming in Children • Nutrition assessment and • Clean Water Sources • Sanitation Facilities • Hand Washing • Clean Family Living Environment • Nutrition Advocacy • Nutrition Resources Mobilization • Multi-sectoral coordination • Human resources for nutrition • Gender Analysis • Accountable policies that enable participation and transparency • Systems: QI/QA, management, financial, logistics, M&E, nutrition surveillance 22 II. BACKGROUND A. Nutrition in Ethiopia Malnutrition remains one of the main public health problems in Ethiopia contributing to 53% of infant and child mortality. Although significant improvements have been made over the past ten years, according to the 2010 Ethiopian Demographic Health Survey, 44% of children are stunted, 10% of children are wasted and 29% of children are underweight. The 2010 National Nutrition Survey (NNS) found that 29% of women of reproductive age were malnourished. The survey showed that overall maternal, infant and young child feeding practices, and maternal and child health remain weak. B. USAID/Ethiopia Investments in Nutrition The USAID/Ethiopia Country Development Cooperation Strategy (CDCS) 2011- 2015: Accelerating the Transformation toward Prosperity lays out a strategy to assist Ethiopia to increase economic growth with resiliency in rural Ethiopia, increase the utilization and quality of health services and improve learning outcomes. Increasing nutritional status among Ethiopians is an integral part of USAID’s development objectives and is expressed as IR 5: “Improved nutritional status of women and young children”. Ethiopia has struggled to integrate nutrition with agriculture and food security programs. While nutrition is the mandate of the Ministry of Health (MOH), programs such as PSNP are overseen by the Ministry of Agriculture. Although the NNP provides clear roles and responsibilities with respect to coordination, communication between the two Ministries has been rare until recently. GOE efforts are now underway to link NNP with PSNP, increasing focus on integration of food security and nutrition through an ongoing accelerated stunting reduction strategy. In addition, MOH is now attempting to integrate nutrition across all sectors through the establishment of the National Nutrition Technical working group and the National Nutrition Technical Committee. USAID/Ethiopia is taking steps to ensure that its resources from diverse sectors are used to maximize achievement of IR5 as articulated in the (CDCS), Feed the Future (FTF) and the Global Health Initiative (GHI.) Specific GHI and FTF goals to” improve nutritional status of Ethiopians and reduce child under nutrition by 20%” will contribute to accomplishment of the overall USAID goal to “Transform Ethiopia to a Prosperous and Resilient Country”. USAID/Ethiopia is clear that dealing with early childhood undernutrition is key to breaking the cycle of poverty in the country. Mission plans lay out specific steps toward prioritizing cost effective, proven interventions to prevent undernutrition in mothers and children under two years of age (the 1,000 day window of opportunity). Operational strategies are included to promote joint programming across offices and funding streams. The Health, AIDS, Population and Nutrition Office developed a Mission Nutrition Strategy in February 2011, which served as a planning document and justification for the Mission’s Flagship Nutrition Project: ENGINE. In addition, the four large FTF Projects, AMD e , LMD, PRIME and GRAD have integrated nutrition activities in their current agriculture development programs. 23 Through integration of nutrition resources, USAID/Ethiopia will leverage the skills of the large number of agriculture and health extension workers to deliver behavior change communication messages focused on proper food utilization, preparation, and storage to improve household nutrition. Within the context of the Three Ethiopia’s (productive, hungry and pastoral), the USG’s FTF Development Hypothesis for Ethiopia is as follows: To achieve Increased Growth with Resiliency in Rural Ethiopia, and attain the interconnected FTF objectives of accelerating inclusive agriculture sector growth and improving nutritional status, a sustained and focused effort – coordinated across multiple stakeholders and aligned with Ethiopia’s development plans and aspirations – must be undertaken. The USG’s role in delivering these results must leverage its experience and comparative advantage and comprise a comprehensive but focused set of linked and integrated activities. By strengthening nutrition programming into AMDE, LGP, PSNP GRAD and PRIME, it is assumed USAID/Ethiopia will leverage the skills of the large number of agriculture extension workers. Through value chain programs, funding will assist agriculture extension workers to deliver better dietary diversity through behavior change communication messages focused on proper utilization, preparation and storage of food to improve household nutrition to farmers, a segment of the population not traditionally reached by nutrition programming. 24 C. ENGINE Project ENGINE is a five year (September 27, 2011 to September 26, 2016) integrated nutrition project, which builds upon the GOE’s initiatives and renewed commitment to nutrition as well as the U.S. Government’s Global Health and Feed the Future Initiatives. ENGINE supports the implementation of the National Nutrition Program (NNP) and aims to strengthen multi-sectoral coordination; build capacity at the policy and implementation levels and for pre-service education and training; support large￾scale behavior change communication for nutrition; link nutrition, livelihoods and food security interventions; and integrate health and nutrition with private-public partnerships. ENGINE’s innovative interventions, including a robust learning agenda, will support and guide effective nutrition policy and practices to reduce undernutrition. Furthermore, the project is expected to contribute to the Agriculture Growth Program (AGP) as articulated in Ethiopia’s Comprehensive African Agriculture Development Plan (CAADP) by strengthening nutrition components. Total anticipated funding over the five years of the Project is $50.8 million USD. In addition to the Feed the Future Initiative, funding from the Global Health Initiative includes support from the President’s Emergency Program for HIV/AIDS Relief (PEPFAR) and in FY 13 WASH funds. . ENGINE is operating in 100 woredas (districts) and at the national level. Eighty three of the woredas are “productive” woredas also targeted by the Agricultural Growth Program (AGP) and supported by Feed the Future in the Amhara, Oromia, SNNP and Tigray regions. The remaining 17 woredas are non-AGP woredas and the implementation of activities with the USAID OFDA funded GOAL Project and with the FTF GRAD Project, began in February 2014 The Project targets 3.1 million children under five years of age; half a million pregnant and lactating women, 3.2 million women of reproductive age, and 2.7 million households. III. PROJECT GOALS AND OBJECTIVES ENGINE’s goal is to improve the nutritional status of Ethiopian women and children less than five years of age through sustainable, comprehensive, coordinated, and evidence-based interventions, enabling them to lead healthier and more productive lives. The expected results for the program include: a. Capacity for and institutionalization of nutrition programs and policies strengthened. b. Quality and delivery of nutrition and health care services improved. c. Prevention of under nutrition through community-based nutrition care practices improved. d. Rigorous and innovative learning agenda adopted. 25 Results Framework The results framework below illustrates the ENGINE project’s anticipated results. IR1: Capacity for and institutionalization of nutrition programs and policies strengthened 1R2: Quality and delivery of nutrition and health care services improved IR3: Prevention of undernutrition through community-based nutrition care practices improved IR 1.1 Nutrition policy environment strengthened  Strengthening existing coordinating mechanisms  Supporting development and revision of policies, guidelines, and standards IR 1.2 Pre-service and in-service nutrition training for health care agents strengthened  Pre-service education  In-service capacity building  Strengthening learning networks IR 2.1: Quality of nutrition services strengthened  Facilitating integration of quality improvement processes  Building capacity of health facility staff and frontline workers to provide high￾quality services IR 2.2 Enhanced health and nutrition service seeking behaviors  Developing SBCC strategy  Mobilizing communities  Educating on micronutrients IR2.3: Access to health and nutrition services increased  Linking with other partners  Strengthening referral system  Establishing sites to treat acute malnutrition  Supporting improvements in supply and logistics IR3.1 Maternal, IYCF knowledge and practices improved  Developing SBCC strategy  Promoting optimal nutrition practices through different communication channels IR 3.2 Access to food and economic strengthening opportunities through programming and cross sectoral linkages increased  Applying economic strengthening interventions to meet households’ needs  Facilitating community-based learning on agriculture techniques for increased production of diverse foods Strategic Objective: Improved nutritional status of women and young children through sustainable, comprehensive, coordinated, and evidence-based interventions IR4: Rigorous and innovative learning agenda adopted Activity 4.1: Design and Delivery of Research Strategy: Research goal: To generate rigorous learning outcomes on key questions and innovative policy and operational approaches that will support the successful scale up of programs to improve nutrition in Ethiopia and a subsequent reduction in maternal and child undernutrition. Themes to be considered: Research relevant to IR1:  Strengthening the nutrition and HIV nutrition policy environment Research relevant to IR2  Improving the quality of nutrition services in Ethiopia  Improving the effectiveness (coverage and efficacy) of programs to address acute malnutrition  Improving coverage and quality of preventative nutrition and related services for mothers and young children Research relevant to IR3  Improving the effectiveness (coverage and efficacy) of interventions that aim to prevent under-nutrition and micronutrient deficiencies 26 IV. PURPOSE AND USE OF THE EVALUATION The purpose of this performance evaluation is to examine what the ENGINE Project has achieved at the mid-way point in implementation; how it is being implemented; how it is perceived and valued; whether expected results are occurring or are likely to occur before the end of the project; and to assess the management and operation of the project. The findings, analysis and recommendations are intended to inform and improve implementation of ENGINE in the remainder of the Project’s life. In addition, the evaluation will assess the contribution of ENGINE and the four major FTF Projects in terms of their contribution to IR 5, as previously stated above. V. EVALUATION QUESTIONS Key evaluation questions, listed in order of priority, are the following: • To what extent is the ENGINE Project progressing against planned objectives as embedded in its Performance Monitoring and Work Plan? Are the ENGINE project’s four intermediate results (see Section III) the most effective means to achieve the Project’s goal given recent evidence and changes in the Ethiopian context (i.e. the revised National Nutrition Program, burgeoning support for multi￾sectoral collaboration at National and Regional level, and changes of emphasis within USAID/Ethiopia)? The evaluation should map out the Project’s Performance Monitoring Plan based on five year targets, including identifying the major factors leading to the achievement of its objectives and the recommended adjustments in the Project that would improve its performance in the remaining period of the agreement. 2. What has been the impact of the actions and activities of ENGINE’s efforts in fostering partnerships amongst the USAID Projects (FTF and IFHP) and in its multi-sectoral engagement/actions with host country entities at the national, regional, district, and kebele levels? The extent of synchronization among this Project, other USAID Projects and host country entities is of great importance to the Mission and the evaluation should document the approach in a way that adds to the evidence base and offer recommendations for strengthening this effort. 1 What have been the contributions of the FTF Projects to IR 5 of DO 1 of the USAID/Ethiopia CDCS and its development hypothesis? In other words, recognizing the links between economic growth and nutritional status, how have the FTF and Global Health Initiative (GHI) programs supported geographically-aligned efforts to achieve IR5, “Increase nutritional status among Ethiopians, with focus on women and young children and in particular dietary diversity”? 2 To what extent has this project contributed to gender equity and female empowerment and specifically addressed the role of gender in decision making on use of resources for maternal and 27 infant feeding and increased women’s access to resources and services that will improve their nutrition and that of their children? Reaching both men and women remains critical to the success of this Project given that the balance in decision making on the use of resources is important in achieving improved maternal and infant feeding at a household level. As women are uniquely marginalized in many aspects and they and their children are also facing malnutrition, this evaluation is expected to assess the role of gender in maternal and infant feeding and provide recommendations based on existing situations where ENGINE is operating. - Given the findings in questions 1-3, does the Project have the right balance of staff and funding given activity priorities? Given that this is a formative evaluation, the evaluation should include a review of the appropriateness of the balance between the resources (staff and budget), their management and the activities the project intends to accomplish. - How effectively and efficiently has the consortium of ENGINE sub-partners performed as well as the leadership of the Prime. VI. EVALUATION DESIGN AND METHODOLOGY The evaluation team will be responsible for developing an evaluation strategy and methodologies that include mixed approaches. The team should present an evaluation questions matrix showing the source of data, method of data collection and also the tool to be used to answer each of the evaluation questions. The methodology will be presented as part of the draft work plan as outlined in the deliverables below, approved by USAID/Ethiopia and included in the final report. The evaluation team will have available for their analysis a variety of program implementation documents, baseline surveys and reports. Methodology strengths and weaknesses should be identified as well as measures taken to address those weaknesses. All data collected and presented in the evaluation report must be disaggregated, as appropriate, by gender and geography. (a) EVALUATION DESIGN This is a non-experimental design that will focus on measuring project results before and after project implementation using project monitoring and survey data. The before project data should be drawn from the baseline survey report produced by the implementing partners. (b) METHODOLOGY As stated above, the evaluation team will be responsible for defining an appropriate evaluation methodology. The team should consider mapping the research questions against the quantitative and qualitative data in a matrix/table to show how each research question will be answered. However, the suggested methodology should include the following: Use of quantitative data, which includes, but is not limited to:  Comparison of current indicator values to baseline data for select output and outcome indicators. ENGINE will be conducting its internal assessment in 28 May and will conduct a mini-LQAS using selected indicators/questions from the baseline impact survey in the projects fiver supervision areas (Amhara East Oromia, West Oromia, SNNPR and Tigray). THE LQAS will assess coverage of iron/folate supplements for pregnant women, de-worming and vitamin A supplementation for children 6-36 months, maternal and child dietary diversity, and use of iodized salt. The ENGINE baseline team leader ( now Tufts/ENGINE M&E data quality manager) will oversee the planning and implementation of the LQAS survey, including recruitment of data collectors from ENGINE’s baseline survey that have the practical, hands-on experience required to rapidly complete the survey in May. The External evaluation team will be provided with the results of the survey and other findings from this internal review for their use.  Map out the Project Results against performance measure indicators to show the total number of indicators under each result and whether performance is met/on target (90-110%), exceeded (>110%), or not achieved (<=89%) Use of qualitative data, which includes, but is not limited to:  Document Reviews  Key Informant Interviews  Focus Group Discussions (FGDs) VII. EXISTING DATA SOURCES The consultants will review the following documents: a) Program Descriptions and Modifications b) Work Plan c) Quarterly Reports d) Annual Reports e) Budget and financial reports f) PMP and other M&E documents g) Baseline surveys and formative research h) Project performance data i) Project-generated assessments j) GOE National Nutrition Program and performance data (if available) VIII. TEAM COMPOSITION The evaluation team shall consist of three independent international experts (with one serving as the team lead and primary coordinator with USAID) as well as three high level Ethiopian experts, at least one of whom can also serve as an interpreter. All team members must have professional-level English speaking and writing skills. The technical team members must all have significant experience in international nutrition and agriculture related programs. They should have Ethiopian country or East Africa regional experience, along with comparative experience in nutrition, MCH service delivery and/or food and nutrition programming in other countries or regions of the world. Sound experience in conducting evaluations or research is expected of all members, and experience in developing strategies would be useful. Ability to conduct interviews 29 and discussions in Amharic and provide accurate translations into English for at least one team member is essential. A statement of potential bias or conflict of interest (or lack thereof) is required of each team member. USAID may propose internal staff members from USAID/Ethiopia as well as from Washington to accompany the team in this evaluation as observers. As observers their role will be to provide, when asked, background information and to reply to the external evaluators questions, when asked. They will review and comment on the report for its accuracy, but the evaluators will be free to accept or decline their comments as this is an external review. Team Leader (one): The team leader, with at least 10 years work experience, will be responsible for overall management of the evaluation, including coordinating and packaging the deliverables in consultation with the other members of the external team. The team leader must have strong team management skills, and sufficient experience with evaluation standards and practices to ensure a credible product. The team leader will develop tools for the evaluation and a design plan and share it with USAID/Ethiopia for their approval. The team leader will develop the outline for the draft report, present the report and after incorporating USAID/Ethiopia staff comments, submit the final report to USAID/Ethiopia within the prescribed time line. The team leader must be fluent in English and have strong writing skills. International Experts (two): Should be senior-level technical analysts (with at least eight (8) years work experience in evaluation and performance monitoring), specialized in the areas of nutrition, agriculture/food security and monitoring and evaluation. These experts must be fluent in English and have strong writing skills. Local Ethiopian Experts (three): The Ethiopian experts should have experience with nutrition and agriculture in Ethiopia, with at least five (5) years work experience in monitoring and evaluation. The Ethiopian experts should also be proficient in English and Amharic. IX. EVALUATION SCHEDULE The estimated time period for undertaking this evaluation is 50 work days for all the team with the exception of the Team leader who will have 60 working days, of which at least 20 work days for all team members should be spent in Ethiopia. The ideal arrival time is June 8, 2014; however, the arrival date will be finalized between USAID and AKLDP, the organization conducting the evaluation. The evaluation team is required to work six days a week. The team is required to travel to selected provinces in each region where project activities are being implemented. While the teams are in Ethiopia, at least 50% of their time will be spent outside Addis Ababa to conduct interviews with project staff, government partners, and project beneficiaries. The evaluation team will prepare an exit briefing and presentation of the findings, which it will deliver to USAID staff before the consultants depart Ethiopia. Also, the evaluation team will submit a draft report 48 30 hours in advance of the exit briefing for review and comments by USAID. Comments from USAID will be incorporated before the submission of the final draft. The final report with USAID and consultants revisions should be submitted by August 1, 2014, assuming the work starts as planned on June 8, 2014. The findings from this report will be used in the development of the ENGINE 2015 work plan which will take place in mid-August. An illustrative table for Level of Effort (LOE)—Dates may be modified based on availability of consultants, key stakeholders and time for fieldwork. The Contractor is expected to submit a detailed LOE. Activity Expat Team Leader Ethiopian Evaluation Specialist (2) Document review, work plan, draft questions, data analysis plan, suggested list of interviewees, finalized data collection tools/instruments for the interview protocols. 5 10 Travel to/from Ethiopia 4 In-briefing with USAID in DC and Addis 2 2 Interviews in Addis Ababa 4 8 Interviews or survey work in project/activity areas 18 36 Mid-term briefing and interim meetings with USAID 2 4 Data analysis, preliminary report and presentation preparation 7 14 Draft evaluation report 10 8 Final exit presentation to USAID (with PowerPoint presentation and draft evaluation report) 2 2 Final exit presentation to relevant partners (with PowerPoint presentation) 1 2 USAID comments on the initial final report (10 days) - - Final evaluation report 4 4 One-page briefer preparation and translation 1 0 Totals 60 90 Travel over weekends may be required during site visits. Note that May 01, 05, 26 and 28; July 04 and 28; and September 01, 2014 are national holidays and thus the US Embassy and USAID are closed on these days. X. DETERMINATION ON THE TECHNICAL PROPOSAL USAID/Ethiopia will determine the soundness of proposal based on the contractor’s overall technical understanding and approach, proposed team members, and cost realism, reasonableness, completeness, and allowability. 31 XI. AKLDP CONFLICT OF INTEREST Tufts University is one of the sub-primes for the ENGINE and also manages AKLDP, which will conduct this evaluation. In order to avoid a conflict of interest, AKLDP will only be evaluating the components of ENGINE that are not managed by Tufts University. The activities under IR4 of the ENGINE project’s results framework, “Rigorous and innovative learning agenda adopted,” for which Tufts University is responsible, will be evaluated by one of the International Experts, to be contracted under a separate mechanism, and insofar as these activities relate to the evaluation questions. This International expert will be responsible for ensuring the integrity of the external evaluation as it related to IR4. XII. USAID MANAGEMENT The evaluation team will officially report to AKLDP managed by TUFTS University. From a technical management perspective, the evaluation contractor is responsible for all direct coordination with the USAID/Ethiopia AOR Mary Harvey and alternate AOR Fisseha Merwai and Program Office. In order to maintain objectivity, all final decisions about the evaluation will be made by the Program Office. XIII. LOGISTICS The USAID funded AKLDP Project will provide the administrative and logistics support. XIV. REPORTING REQUIREMENTS AND DELIVERABLES A. DESCRIPTION AND TIMELINE OF DELIVERABLES 1. In-briefing: Within 48 hours of arrival in Addis Ababa, the evaluation team, will have an in-brief meeting with USAID/Ethiopia’s Program Office and Mary Harvey and Fisseha Merwai of the EG&T Office for introductions; presentation of the team’s understanding of the assignment; initial assumptions; review of the evaluation questions, survey instruments, and initial work plan; and adjustment of the SOW, if necessary. 2. Evaluation Work Plan: Prior to their arrival in-country, the evaluation team shall provide a detailed initial work plan to the Program Office and EG&T Office and a revised work plan three days after the in-briefing. USAID will share the revised work plan with GOE for comment, as needed, and will revise accordingly. The initial work plan will include (a) the overall evaluation design, including the proposed methodology, data collection and analysis plan, and data collection instruments; (b) a list of the team members indicating their primary contact details while in-country, including the e-mail address and mobile phone number for the team leader; and (c) the team’s proposed schedule for the evaluation. The revised work plan shall include the list of potential interviewees, sites to be visited, and evaluation tools. 3. Mid-term Briefing and Interim Meetings: Schedule a mid-term briefing with USAID to review the status of the evaluation’s progress, with a particular emphasis on addressing the evaluation’s questions and a brief update on potential challenges 32 and emerging opportunities. The team will also provide the Acquisition Officer’s Representatives for Save the Children and the ENGINE Project with periodic written briefings and feedback on the team’s findings. Additionally, a weekly 30 minute phone call with the Program Office and the EG&T Office and Team Leader will provide updates on field progress. If there are any problems these should be immediately addressed and not to wait for the phone calls. . 4. PowerPoint and Final Exit Presentation to USAID and relevant partners that will include a summary of key findings and key conclusions. To be scheduled as agreed upon during the in-briefing, and five days prior to the evaluation team’s departure from Addis Ababa. A copy of the PowerPoint file will be provided to the Program Office prior to the final exit presentation. 5. Draft Evaluation Report: The content of the draft evaluation report is outlined in Section XIV.B, below, and all formatting shall be consistent with the USAID branding guidelines and 508 compliance. The focus of the report should be answering the evaluation questions and may include factors the team considers to have a bearing on the objectives of the evaluation. Any such factors can be included in the report only after consultation with USAID. The draft evaluation report will be submitted by the evaluation team leader to the Program Office 24 hours in advance of the exit briefing for review and comments by USAID. USAID’s Program Office and EG&T will have ten business days in which to review and comment and the Program Office shall submit all comments to the evaluation team leader. 6. Final Evaluation Report will incorporate final comments provided by the Program Office. The length of the final evaluation report should not be more than 30 pages, not including Annexes and the Executive Summary. USAID comments are due within ten days after the receipt of the initial final draft. The final report should be submitted to the Program Office within three days of receipt of comments by the evaluation team leader. All project data and records will be submitted in full and shall be in electronic form in easily readable format; organized and fully document for use by those not fully familiar with the project or evaluation; and owned by USAID and made available to the public, barring rare exceptions, on the USAID Development Experience Clearinghouse (http://dec.usaid.gov). 7. One-page briefer on key qualitative and quantitative findings and conclusions relative to the evaluation questions for each municipality is included in the evaluation’s scope—to be given to the appropriate government counterpart(s) so that they have the opportunity to review evaluation findings and share them with the larger community. Each briefer will be reviewed by the Program Office and EG&T prior to distribution and will be translated into Amharic. 33 B. FINAL REPORT CONTENT The evaluation report shall include the following: 1. Title Page 2. Table of Contents (including Table of Figures and Table of Charts, if needed) 3. List of Acronyms 4. Acknowledgements or Preface (optional) 5. Executive Summary (3-5 pages) The executive summary should succinctly capture the evaluation purpose and evaluation questions; project background; evaluation design, methods; and limitations; and the findings, conclusions, and recommendations. 6. Introductory Chapter a. Brief description of FTF, ZOI, and ENGINE intervention areas b. A description of the project evaluated, including goals and objectives. c. Brief statement on purpose of the evaluation, including a list of the main evaluation questions. d. Brief statement on the methods used in the evaluation such as desk/document review, interviews, site visits, surveys, etc. e. Explanation of any limitations of the evaluation—especially with respect to the methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.)—and how these limitations affect the findings. 7. Findings: This section should include findings relative to the evaluation questions. The information shall be organized so that each evaluation question is a sub-heading. Any findings examining group differences ( i.e. sex, region, etc.) should indicate instances in which differences are statistically significant. 8. Summary and Conclusions: This section must answer the evaluation questions based upon the evidence provided through the Findings section. The information shall be organized so that each evaluation question is a sub-heading. 9. Recommendations and Next Steps: Based on the conclusions, this section must include actionable statements that can be implemented into the existing program or included into future program design. Recommendations are only valid when they specify who does what, and relate to activities over which the USAID program has control. For example, recommendations describing government action is not valid, as USAID has no direct control over government actions. Alternatively, the recommendation may state how USAID resources may be leveraged to initiate change in government behavior and activities. It should also include recommended future objectives and types of specific activities based on lessons learned. The information shall be organized so that each evaluation question is a sub-heading. 34 10. Annex: The annexes to the final evaluation report should be submitted as separate documents—with appropriate labels in the document file name (e.g., Annex 1 – Evaluation SOW), and headers within the document itself—and may be aggregated in a single zipped folder. a. Evaluation Statement of Work b. Places visited; list of organizations and people interviewed, including contact details. c. Evaluation design and methodology. d. Copies of all tools such as survey instruments, questionnaires, discussions guides, checklists. e. Bibliography of critical background documents. f. Meeting notes of all key meetings with stakeholders. g. “Statement of Differences” h. Evaluation Team CV’s i. Disclosure of Conflict of interest (signed by each member.) C. REPORTING GUIDELINES  The format of the report shall be consistent with the USAID branding guidelines and 508 compliance.  The evaluation report should represent a thoughtful, well-researched and well￾organized effort to objectively evaluate what worked in the project over the given time period, what did not, and why.  Evaluation reports shall address all evaluation questions included in the statement of work.  The evaluation report should include the statement of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the Program Office.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an annex in the final report.  Evaluation findings will assess outcomes and impact on males and females, and data will be disaggregated by gender, age group, and geographic area wherever feasible.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative and/or qualitative evidence.  Sources of information, including any peer-reviewed or grey literature, will be properly identified and listed in an annex.  Recommendations will be supported by a specific set of findings. They will also be action-oriented, practical, and specific, with defined responsible parties for each action. 35 Annex B: Evaluation interview questions 1. Has the management of the research component of ENGINE been appropriate and adequate? 2. Has the balance of resources allocated for operational research and knowledge sharing been adequate and appropriate to date? 3. How have local institutions and academic institutions been engaged in the research conducted thus far? 4. Is there an appropriate balance between the health-nutrition and the agriculture￾nutrition research agenda? 5. How well has the PhD scholarship program worked? Has there been a balance of study topics across the IRs? 6. How well is gender being addressed under IR4? 7. In what ways has the research strategy contributed so far towards the scaling up of broader nutrition programs and activities? 36 Annex C: List of documents reviewed for the mid-term evaluation of IR4 of the ENGINE Project Project Documents 1. ENGINE Technical Proposal Submitted to USAID (August 2011) 2. ENGINE Baseline Survey – Proposal Submitted to Oromia Ethics Board 3. ENGINE Internal Mid-Term Evaluation Report (June 2014) 4. ENGINE Communication Strategy (August 2013) – Recommendations for Developing and Implementing ENGINE’s Overall Communication Strategy 5. Valid International Final Performance Report (January 2013) 6. Jimma University – Year 3 Semi Annual Report (March 2014) 7. ENGINE – Tufts University Annual Meeting Report (March 2013) 8. Tufts University – Progress Report (March 2014) 9. ENGINE budget by IR; Years 1-3 (June 2014) 10. Final ENGINE Revised PMP (August 2013) 11. Summary of ENGINE Operational Research (May 2013) 12. Website communication piece: Being the Voice to Improve Maternal and Child Nutrition: ENGINE Supports National Nutrition Advocacy Workshop for Parliamentarians and Policy Makers (March 14-15, 2014) 13. ENGINE Gender Mainstreaming Strategy (March 2014) Research Studies 1. One pager: Prospective Operational Birth Cohort Study 2. One pager: Nutrition policy research 3. One pager: Moderate and Severe Acute Malnutrition Studies 4. Prospective Operational Birth Cohort Study – Research questions included in Cohort 1 and Cohort 2 5. Agriculture-Nutrition Panel Study – Evaluating Multisectoral Strategies for Improving Nutrition and Food Security in Ethiopia 6. Agricultural Commercialization, Production Diversity and Consumption Diversity among Small-Holders in Ethiopia. Results from the National Ethiopia Integrated Survey on Agriculture, Rural Socioeconomic Survey 2012 (2014) 7. Factors associated with stunting in Ethiopian children under five. Tufts University and Jimma University (2014) 8. MAM and SAM Operational Research Updates from Valid International (2014) 9. Evaluation of dynamics of the National Nutrition Program Implementation in Ethiopia – presentation at the Micronutrient Forum (June 2014) 10. List of ENGINE funded M.Sc. and PhD thesis topics Meeting reports, presentations and notes 1. Meeting notes: Tufts University – ENGINE Annual Meeting (March 19-23, 2013; Executive Hotel, Adama) 2. Evaluation of the dynamics of national nutrition program implementation in Ethiopia – Presentation by Masresha Tessema, EHRI, at the 2014 Micronutrient Forum in Addis Ababa (June 2014). 3. ENGINE-Tufts University Year 3 Semi-annual Report (April 2014) 4. ENGINE-Tufts University Progress Report (October 2013) 5. ENGINE - Tufts University Year 2 Semi-annual Report (April 2013) 6. ENGINE-Valid Budget (June 2014) 7. ENGINE - Valid Final Performance Report (January 2013) 8. ENGINE - Valid Mid Term Evaluation Report (June 2014) 37 Annex D: Agriculture-Nutrition Panel Priority Research Questions Question Data Source # Manuscripts as Deliverables Implementation Research Questions 1. a. How do ENGINE-trained development agents’ knowledge, attitudes, and practices regarding nutrition-sensitive agriculture extension services differ from those of non-ENGINE-trained development agents? b. What do they perceive as barriers/facilitators of incorporating ENGINE training into their work and applying it in practice? DA Survey+ Qualitative inquiry 1 2. a. What are the factors associated with program exposure to, participation in, and uptake of ENGINE’s nutrition sensitive activities? b. What are barriers and facilitators of this process, and how do these change inter and intra-annually? c. What are women’s perceptions of the ENGINE nutrition sensitive messaging and how are these messages translated/absorbed into the decision making process of smallholder farming households? What barriers to do women face in translating these messages into action? (Marion Min￾Barron PhD question) Ag-Nut Panel+ Qualitative inquiry 1 (a+b) Outcome/Impact Research Questions 3. a. Are those households that adopt nutrition-friendly agricultural practices more likely to achieve positive production, consumption, and nutrition outcomes? b. Which nutrition-friendly practices appear to yield the greatest benefits and why? Ag-Nut panel+ Qualitative inquiry 1 4. a. What is the added effect of nutrition-sensitive programming on a package of nutrition-specific interventions? b. Through which pathways (eg. nutrition sensitive v. nutrition specific/income or direct consumption) did ENGINE’s interventions achieve its impacts? Ag-Nut panel+ Qualitative inquiry 1 Other Research Questions (these research questions are not project deliverables, but we will aim to address them if time permits) 5. Under what circumstances does smallholder commercialization lead to improved household dietary diversity? LSMS-ISA secondary data 1 6. What factors predict the share of own production that is sold versus directly consumed from homestead gardens and smallholder farms? Ag-Nut panel 1 7. What is the reliability and internal validity of the Women’s Empowerment in Agriculture Index (WEAI) within the context of Ag-Nut panel 1 38 Question Data Source # Manuscripts as Deliverables Ethiopian smallholder farming households and different female subgroups? (Marion Min-Barron PhD Question) survey Qualitative inquiry 8. What are the female nutritional benefits associated with different patterns of crop production, crop diversity and other decisions regarding smallholder commercialization? (Marion Min-Barron PhD Question) Ag-Nut panel survey 1 9. a. To what extent do nutrition-sensitive agricultural behaviors, once adopted, persist inter-seasonally and inter-annually? b. What factors explain their persistence, or lack thereof? Ag-Nut panel+ Qualitative inquiry 10. a. Are food security levels among households located in FTF￾designated “productive areas” higher and less variable than those between households situated in “productive” and “hungry” areas? b. What are the targeting implications, if not? LSMS-ISA Qualitative inquiry