BUREAU FOR FOOD SECURITY RESULTS CHAIN ANALYSIS PROJECT FEED THE FUTURE PROGRAM IN ETHIOPIA September 24, 2018 BUREAU FOR FOOD SECURITY RESULTS CHAIN ANALYSIS PROJECT FEED THE FUTURE PROGRAM IN ETHIOPIA This publication was produced at the request of the United States Agency for International Development Developed under Contract/Task Order Number: AID-OAA-1-15-00024/AID-OAA-T0-16-00008. Prepared by: Luis Guillermo Placci – Team Leader Armando Valdes-Velasquez – Senior Expert Contractor: Program Evaluation for Effectiveness and Learning (PEEL) ME&A 1020 19th Street NW, Suite 875 Washington, DC 20036 Tel: 240-762-6296 www.MEandAHQ.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS INTRODUCTION 1 Background 1 Rationale For A Results Chain-Based Study 1 STUDY APPROACH 1 Using Results Chains 1 Study Design 3 Results Chain-Based Analysis: An Example From the Feed the Future Ethiopia Program 4 MAIN FINDINGS AND RECOMMENDATIONS 7 Main Findings 7 Recommendations 9 ANNEXES 11 Annex A: List of Consulted Documents 12 Annex B: Feed the Future Program High-Level Results Chain, Key Hypotheses, and Questions 16 Annex C: Indicators and Measurements for the Feed the Future Program High-Level Analysis 19 1 INTRODUCTION BACKGROUND Feed the Future is a whole-of-government presidential initiative led by the United States Agency for International Development (USAID) Bureau for Food Security (BFS). The original goals of Feed the Future were to sustainably reduce global poverty and hunger by promoting inclusive agriculture sector growth and improved nutritional status—especially for women and children. Eight specific intermediate results supported these two goals. Results included 1) Improved agricultural productivity; 2) Expanded markets and trade; 3) Increased investment in agriculture and nutrition-related activities; 4) Increased employment opportunities in targeted value chains; 5) Increased resilience of vulnerable communities and households; 6) Improved access to diverse and quality foods; 7) Improved nutrition-related behaviors; and 8) Improved use of maternal and child health and nutrition. The Feed the Future initiative is based on the proposition that supporting smallholder farmers is key to unlocking the transformative potential of agricultural production and expanding the markets, which builds economic growth and paves the way out of poverty and hunger. Feed the Future has implemented programs in 19 focus countries1 and many “aligned” countries across Africa, Asia, and Latin America and the Caribbean. The focus countries were selected based on their level of need, their commitment to increasing food security and improving nutrition, and their potential for agricultural growth.2 RATIONALE FOR A RESULTS CHAIN-BASED STUDY In their efforts to learn and improve the program, USAID/BFS was interested in exploring how useful results chain could be in supporting Feed the Future plausibility analysis. Results chains are used to illustrate underlying assumptions in the program’s theory of change (ToC). The ToC is an assumed causal linkage between interventions and desired impacts that is achieved through a series of expected intermediate results (Margoluis et al. 2013). Making the ToC explicit and aligning it with existing information helps clarify whether the activities were implemented as planned and if the logic supporting the TOC was upheld. An explicit TOC also illustrates the degree to which interventions contributed to measurable changes in the desired outcomes and impact. Results chains will help support mapping of interventions in Feed the Future programs, make ToCs explicit, and help identify useful indicators to track results and progress. STUDY APPROACH USING RESULTS CHAINS Results chain is one of the tools used under the Open Standards for the Practice of Conservation3 and one that explicitly shows underlying assumptions in a project’s strategy. A results chain diagram depicts activities, interventions, and intended effects, and illustrates how interventions may lead to expected impacts. Figure 1, below, is a simplified, illustrative example of a results chain that maps out how improving the value chain of key commodities may reduce poverty and stunting. 1 Bangladesh, Cambodia, Ethiopia, Ghana, Guatemala, Haiti, Honduras, Kenya, Liberia, Malawi, Mali, Mozambique, Nepal, Rwanda, Senegal, Tajikistan, Tanzania, Uganda, Zambia. 2 Feed the Future (2016): Progress Report: Growing Prosperity for a Food-Secure Future. (http://feedthefuture.gov/progress2016) 3 The Open Standards for the Practice of Conservation. The Conservation Measures Partnership.V.3.0 April 2013 (www.conservationmeasures.org) 2 Figure 1: Illustrative Example of a Results Chain The yellow hexagon represents the intervention or strategic approach. The blue boxes represent the expected results. The purple boxes represent the expected impact, and the green oval represents the final goals. Each arrow represents a causal link and can be read as an “if… then” relationship. For example (counting from the hexagon) boxes 3 and 4 say: if more economic opportunities and jobs are created - through an improved value chain - then vulnerable households will engage in those economic opportunities. Each arrow also represents a hypothesis that can be tested and a question that can be posed to clarify and assess a project’s effectiveness. For example, to what extent (or under which conditions) do vulnerable households access new jobs or engage in new economic opportunities created through the project? How Can Results Chains Support Retrospective Analyses? A results chain diagram clarifies the key hypotheses that underlie a program or an activity and identifies the most appropriate indicators to help test hypotheses and answer key questions about the program or activity. Every relationship between two results (indicated by the arrows in figure 1) has four possibilities: (A) Both results are achieved; (B) The first result is achieved but the second is not; (C) The first result is not achieved but the second is achieved; and (D) Neither result is achieved. Case A represents the “success” scenario, which happens when the intervention is advancing as predicted and results are being achieved. In these successful cases, there is either evidence that the intervention is working well or, at least, no evidence to the contrary. This being the case, and in the event that there are plans for the intervention to continue, there is no apparent reason to adjust the intervention. Note that this does not mean that the second result can be directly attributed to the first. To determine causality, the project team would need to do an impact evaluation with an appropriate design, set of indicators, collection methods, and data disaggregation. Case B is somewhat more complicated. It could indicate that achieving the second result will require more time. However, it could also indicate that the logic is flawed, and the second result will never be achieved through this intervention. In the event of program continuation (or the intervention being replicated by another program), the project team should re-evaluate the timing, and consider the risk of faulty logic. The team may also have to consider that the project could come to an end without (or before) knowing if the second result could ever be achieved, increasing the uncertainty of the final impact of the project. Case C illustrates a situation where the second result is achieved, but the evidence indicates that project interventions may not have contributed to the result. The logic did not hold, and the positive results are unexpected consequences of unplanned factors. The project team should re-examine the logic behind the project and the context within which the logic operates before considering its continuation and/or replication by another program. Case D demonstrates an essential program failure where no results are achieved. This failure could be due to a faulty logic or poor program execution. Ideally, results chains should be developed during the program design phase. Developing the results chains 3 early allows for a set of indicators (and measurement methods), tailored to testing the key hypotheses, to be developed early as well. When results chains are used retrospectively, they help reconstruct the ToC and identify key hypotheses; however, testing the hypotheses depends on available information and the potential to collect new data. Collecting information during a retrospective analysis is challenging and not always feasible. In these cases, the analysis should rely on proximate information (i.e., proxy indicators like expert opinions among other possibilities). In a results chain-based retrospective study, associating success with program interventions (as in case A, above) is challenging. The retrospective analysis can, however, identify several A, B, C, and D cases, and the project team can use these findings to learn, adjust interventions, or reassess and redesign all or part of the program's ToC. STUDY DESIGN This results chain-based analysis of the Feed the Future program in Ethiopia used a retrospective study design. The study team used three primary data sources to assess progress toward achieving the expected results: (a) Feed the Future program monitoring data, (b) relevant reports and published materials, and (c) key informant interviews with stakeholders. The study team reviewed all of the key planning and implementation documents (a complete list of reviewed documents is in Annex A) as a means of reconstructing the ToC—the foundation of the Feed the Future program in Ethiopia. Once complete, the team developed the high-level results chains depicted in the ToC. Based on the results chains generated, the study team identified a set of key hypotheses and potential study questions. During a country visit to Ethiopia, the study team held a consultative meeting with the Feed the Future team to discuss, validate, and prioritize the draft results chains, hypotheses, and questions (Annex B includes a high-level program results chain, hypotheses, and questions). Feed the Future Program Monitoring Data: The team gathered this information from the Feed the Future Monitoring System (FTFMS) database, and progress and evaluation reports for different activities within Feed the Future program. Review of Relevant Reports and Published Materials: The team obtained this information by reviewing third-party reports4. The reports were not necessarily specific to the Feed the Future program, but related indicators were extracted and used to measure progress. Key Informant Interviews: The study team interviewed key informants in high-level positions within implementing partners, the Government of Ethiopia, USAID, and research institutions. Informants were selected based on the depth of their understanding of Feed the Future program design and its evolution. The study team acknowledges that more information exists about the program and activities than this study considers5 and that the limited scope taken by this report may lead to erroneous interpretations of results. The team also recognizes that Feed the Future partners have evolved in their understanding of the ToC and as such the program and activities have changed over time. Also, because this is a retrospective analysis, the results chains that were developed for this study attempted to reflect the original thinking of the Feed the Future team and their partners. Hence, this results chain analysis of Feed the Future program in Ethiopia retrospectively assesses the progress based on the original ToC, not necessarily the current thinking. Using the MIRADI software6, the study team developed the results chains and matched the indicators and existing data with specific results in each chain. The data source used to assess a specific results chain is 4 For example: Ethiopia Strategy Support Program (ESSP) Working papers or the impact evaluation report for the Agricultural Growth Program (AGP) of Ethiopia (2018) by AB. Weldesilassie, and colleagues from the Ethiopian Development Research Institute. While these reports and papers do not have the same zone of influence as Feed the Future Ethiopia, they include relevant information that can be used as proxy data (See annex A for a complete list of consulted documents). 5 The study team did review the key information that the Feed the Future program collects in a structured way through annual reports and the indicator database. The study team did not analyze the methodologies behind data collection or how data were synthesized and summarized. The study team assumed that the data provided in annual reports and found in the FTFMS database are accurate. 6 MIRADI is a program allowing nature conservation practitioners to design, manage, monitor and learn from their projects to more effectively meet their conservation goals. It closely follows the Open Standards for the Practice of Conservation (http://www.miradi.org). 4 identified with an icon that includes a three-letter code. Icons also show the level of achievement of the specific result (see the legend in Figure 3). RESULTS CHAIN-BASED ANALYSIS: AN EXAMPLE FROM THE FEED THE FUTURE ETHIOPIA PROGRAM The study team developed 15 results chains at two different levels. The first level was at a programmatic, or high-level that included four results chains depicting the whole Feed the Future program ToC for Ethiopia. The second was at the activity level and included 11 results chains depicting two flagship projects—the Livestock Market Development (LMD) and the Empowering New Generations with Improved Nutrition and Economic Opportunities (ENGINE). The study team used these result chains to assess progress on the interventions, note what results were achieved and determine whether the logic supporting the ToC was upheld. An example of a results chain analysis at the programmatic high-level is presented in steps 1-4. Step 1: Reconstructing the Theory of Change (Programmatic High-Level) The two highest-level indicators of the Feed the Future goals are the prevalence of poverty and the prevalence of stunting in children under five-years-old. The Feed the Future program in Ethiopia defined targets for these two high-level indicators: 1) Within five years, reduce household poverty by 30 percent in the zone of influence (ZOI); and 2) Within five years, reduce stunting by 20 percent in the ZOI. The Feed the Future strategy in Ethiopia applied a market-based agricultural development approach to reduce poverty and promote sustainable livelihoods for chronically food insecure households. To link agricultural economic growth with vulnerable households, Feed the Future designed a push-pull model (represented in a simplified results chain in Figure 2). This approach sought to strengthen the capacity of vulnerable and chronically food-insecure populations to the point where they could participate in economic activities. This was the “push.” Mobilizing market-led agricultural growth in high-potential areas and generating economic opportunities through increasing demand for smallholder production, labor, and services was the “pull.” The overall strategy assumed that the smallholders—especially those in the most vulnerable households—would increase their incomes and access to more food and better quality food (i.e., move towards food security) by improving agricultural productivity or engaging in the new economic opportunities the pull created. Figure 2: Simplified Results Chain of High-Level Overall Feed the Future Strategic Approach in Ethiopia 5 Every activity was designed to enable either the push or the pull. A fundamental underlying assumption was that all the efforts had to be integrated to capitalize on synergies and maximize impact (see Annex C for a synthesis of the overall program ToC with data on specific results from this study). Step 2: Identifying Key Hypotheses and Questions from the Theory of Change Each arrow of the results chain represents a hypothesis, and each hypothesis can encompass several questions. From a program management point of view, not all questions are equally important. For the results chain in Figure 2, staff from Feed the Future Ethiopia selected three hypotheses for the high-level program design. Each hypothesis was considered critical when following the logic of the ToC. That is, they selected hypotheses that complied with two criteria: (i) there was little knowledge and high uncertainty about the underlying assumption, therefore a higher likelihood of the hypothesis being invalid; and (ii) if the hypothesis was invalid, there was a high risk of it limiting the program’s or activity’s success. We present one of the three hypotheses along with the results chain constructed to depict that hypothesis (Figure 3). Hypothesis 1: If vulnerable households are either more productive or more engaged in new economic opportunities (improved livelihoods), they will see a significant increase in income or assets (other benefits) sufficient to take them out of poverty, hunger, and chronic malnutrition. Figure 3: Simplified Results Chain Highlighting Hypothesis 1 Some key questions related to this hypothesis are: • • How much increase in income or assets is sufficient to reduce poverty, hunger, and chronic malnutrition? What conditions are necessary for a household to spend the increased income (or assets) to reduce their poverty, hunger, and chronic malnutrition? Step 3: Identifying Information Needs The program’s ToC noted three paths to improved livelihood and income (see Figure 3). These are: 1. Improvement of households’ agricultural-based livelihood (agricultural production); 6 2. Engagement of households and businesses in new or improved agricultural-based economic opportunities derived from improved key commodities value chains; and 3. Engagement of households and businesses in new, non-agricultural, economic opportunities (mainly deriving from the country’s economic growth). To discuss this hypothesis, the study team needed to assess progress on four expected results: • • • • Improved livelihoods: Was the program successful in improving livelihoods? Improved households’ income: Did households’ income or assets increase significantly? Reduced poverty: Did beneficiary households get out of poverty? Reduced hunger and chronic malnutrition: Did beneficiary households get out of hunger and chronic malnutrition? Step 4: Answering Questions with Available Information The available Feed the Future documents and database provided limited information about the expected program results (blue boxes in the results chains). To test the identified key hypotheses, the study team used information from non-Feed the Future reports and information from key informants from the Government of Ethiopia and organizations linked to the implementation (see the legend in Figure 3). However, the study team found good information to support the job creation and economic opportunities (i.e., improved quality and number of business to business relations, market expansion, innovation grants, access to finance, etc.) generated by different Feed the Future program activities. Within some activities, jobs and opportunities were explicitly targeted to benefit poor and vulnerable people. Other activities did not disaggregate information between vulnerable and non-vulnerable people, smallholders, and businesses and so lacked precise information about who benefited from new jobs and opportunities. The study team was not able to identify and access any income or assets data for households specifically participating in the Feed the Future activities (beneficiary households). However, data from the Agriculture Growth Program (AGP) of Ethiopia’s7 2018 Impact Evaluation Report show higher income increases for AGP households than for non-AGP households. Several key informants (KIs) mentioned that the observed income rise was barely enough to make up for inflation. The Feed the Future Ethiopia ZOI Interim Assessment Report (2016) notes a slight increase in the average daily expenditures of three percent in two years. Some KIs stated that traders saw increased profitability but that the farmers benefited less than traders and consumers suffered the impact of higher prices. The study team could not find data to determine what percentage of the households that had been below the poverty line or faced malnutrition conditions before interventions have seen improved income after interventions. Therefore, it was impossible to know how many of them got out of poverty or malnutrition because of program interventions. There is information—quantitative data from FTFMS database and M&E reports and qualitative data from key informant interviews—suggesting that Feed the Future contributed to improved households’ agricultural-based livelihoods by creating new economic opportunities and jobs. There is data supporting the claim that farm-level sales increased and that this increased smallholders’ and enterprises’ income and contributed to economic growth. However, there are reasonable doubts about the contribution these activities made to increasing vulnerable households’ income and pulling them out of poverty, hunger, and chronic malnutrition. As of yet, there is no information to support the assumption that these households will not relapse after jobs and economic opportunities pull them out of poverty or malnutrition. 7 The Agricultural Growth Program (AGP) is a Government of Ethiopia program that seeks to increase agricultural productivity and market access for key crop and livestock products. AGP is implemented in 25% of the productive highlands of Ethiopia. The impact evaluation compares the effects of the program on the households in this 25% (AGP households) with the rest (non AGP households). Feed the Future is one of the partners that supports AGP through its program and specific activities. 7 MAIN FINDINGS AND RECOMMENDATIONS MAIN FINDINGS The study team developed and analyzed 15 results chains for the Feed the Future program in Ethiopia. The analyses included reconstruction of the ToC at the program-level and activity level (for two activities), generating visual depictions of the ToC with results chains, identifying hypotheses and key questions, and then reviewing available information to test or answer them. Based on the construction of the results chains and analysis of available quantitative and qualitative data, the study team identified patterns in the Ethiopia program and documented examples from the retrospective analysis to explain the main findings. 1. Program Design. The Feed the Future Ethiopia program was designed to address the main challenges in reducing poverty and malnutrition through an innovative approach that linked the generation of job opportunities with capacity strengthening in the population applying for those jobs. The facilitation of, and active participation in, multi-stakeholder platforms allowed continuous reinforcement of this program design and precise alignment with country context and priorities. 2. Program Impact. There is evidence that the Feed the Future program contributed to increasing smallholder income and businesses, which then added to overall agricultural growth in Ethiopia. While there is some good information (quantitative and qualitative data) that supports the Feed the Future's contribution to agricultural economic growth in Ethiopia, there is less evidence on the contribution this overall agricultural growth made to increasing vulnerable households' incomes and pulling these households out of poverty and chronic malnutrition8. The nature of the Feed the Future interventions are such that—even if data were disaggregated between vulnerable and non-vulnerable households— recognizing and determining Feed the Future’s impacts in Ethiopia will take time. 3. Link Between Activity Goals and Program Goal. In many cases, the link between the Feed the Future Ethiopia program activity’s end goal and the overall program end goal is missing or not explicit. For example, some activities focus on economic growth and lack an explicit, logical link to reducing hunger and chronic malnutrition. Lack of these links limits precise measurements for how each activity contributes to the overall program’s end goal. 4. Theories of Change. Even when the ToC for the activities is explicit and clear, the way different interventions within each activity are expected to reach the end goals is not always clear. This lack of clarity can result in at least three consequences: • • • The way each intervention contributes to a common result is unclear or cannot be assessed. For example, there is evidence that the Feed the Future value chain approach in the productive regions of Ethiopia improved agricultural sector performance. However, it is not clear which interventions contributed to this performance. Critical intermediate results are not clearly defined and measured. Without clarifying who is accessing new jobs and opportunities created by the program (i.e., vulnerable households, smallholders, business people), it is not possible to know whether or not they are getting out of poverty because of program intervention. Some interventions respond to external demands and do not grow from logical, strategic prioritization of potential program interventions. For example, interventions aimed at addressing the situation of people living with HIV within the Livestock Market Development (LMD) Activity 8 The study team could not access any specific income or assets data for households that participated in Feed the Future activities. However, data from Ethiopia’s AGP I Impact Evaluation Report (2018) show higher income increases for AGP households than for non-AGP households. Still, information was not disaggregated between vulnerable and non-vulnerable people, smallholders and businesses. The study team could not find data to determine what percent of households that saw improved income had been below the poverty line or faced malnutrition conditions before interventions (Annex C for complete list of indicators with data). 8 seem to be independent of other interventions. The inclusion of these interventions without explicit explanation and link to the other interventions reduces the strength of the programmed actions and the activity’s ToC. 5. Synergies Among Activities and Interventions. Interactions among program activities were not always made explicit. For example, by separating “push” and “pull” activities, there was a tendency toward programmatic isolation and expertise silos. When activities were implemented within different geographic areas, isolation increased. This created gaps in linking vulnerable households from deficit areas to markets opportunities in more productive regions. This weak interaction also occurred among interventions within a given activity. 6. Justifications for Actions. Some interventions included unrelated actions not explicitly aligned with the expected outcomes, and some appeared to be a “fit-it-all” bucket of actions not logically prioritized and justified. For example, the LMD project had an intervention to build the capacity of public and private stakeholders to create an enabling environment. This is a broad take on capacity building with a wide range of actions not explicitly and systematically prioritized. 7. Availability of Information. Despite the considerable effort the Feed the Future program in Ethiopia put into monitoring and reporting data, the available FTFMS datasets at the time of the study were insufficient in assessing program effectiveness, testing the key hypotheses underlying the program design, or answering the essential questions that were identified and prioritized by the Feed the Future Ethiopia team during the study to guide program management. Furthermore, the monitoring system was not designed to address these data needs. Some indicators such as the total value of incremental sales and the total value of exports are useful in aggregating results. However, many indicators focus on implementation or short-term products such as the number of people trained (outputs). Few indicators measure intermediate expected results or outcomes (blue boxes in results chains) like the number of people using the knowledge they gained, or the number of people who improved their income, assets, or quality of life due to training they received. Indicators tend to be isolated and, therefore, not useful for analyzing cause-and-effect relationships where bivariate analysis is needed. There are gaps in the indicator coverage along the ToC, which make it difficult to: a) assess the interventions and the combination of results that contribute to the goals; b) identify the interventions that do not help; and c) note the interventions that could be generating adverse effects and obscuring the positive impact of the "good" interventions. The level of disaggregation and the detail of the indicators limited how helpful the information was in answering key questions. Data are not disaggregated in a way that allows for an accurate assessment of how different strategies contribute to the same result. This is exemplified by the lack of detail in the number of jobs created through specific interventions or strategies. Another example was the lack of disaggregation among vulnerable and non-vulnerable people, which limited the power of data to answer some key questions. 8. Complexity of Expected Outcomes. Some essential intermediate results or outcomes require elaborately designed multiple indicators. For example, the expected outcome “improved key commodities' value chains” (i.e., crop-related and livestock/dairy value chains) involves numerous layers and variables. The key variables differ from one value chain to another, and there is no simple way of assessing whether a particular value chain was improved. 9. Consistent Use of Terminology. Some terms do not have a standardized definition across all documents. For example, the words “goals” and “objectives” have different meanings in different documents—sometimes even within the same document. This adds unnecessary complexity when developing program design or assessing the impact (see first and second main findings) and makes it harder to identify components included in ToCs that are means to an end or an end in and of themselves. Other terms, like “activity,” “action,” “intervention,” and “strategy,” are also used in different ways in different documents. 9 RECOMMENDATIONS Based on this study’s main findings, the team has recommendations for improving the Feed the Future Ethiopia program, enhancing the possibility of answering key questions and allowing the program’s key hypotheses to be tested. Feed the Future Ethiopia should invest more efforts to identify and track indicator measures along the pathways that households take to move out of poverty and chronic malnutrition. It would be important for the specific activities to make explicit in their planning documents what the goals of the Feed the Future Ethiopia program are, and present how their interventions (with aligned indicators) may contribute to these goals. Feed the Future should also consider further standardizing terminology and definitions used across all projects and documents. Some key expected results are more complex than others. Developing a dashboard with simple measures that allow a quick view of the state of key variables—the ones most critical to achieving competitiveness—within each value chain would benefit Feed the Future Ethiopia program. Changes in the state of these key variables on the dashboard, would help guide, prioritize, and adjust interventions where needed. The Feed the Future Ethiopia program could use results chains to address some key issues noted in this report’s findings, define goals and objectives clearly and consistently across activities, make interventions and assumptions explicit, and identify indicators that can help answer key evaluation questions and test hypotheses. The strengths and limitations of results chains are discussed below and include some conditions that might improve the approach’s uptake. 1. Strengths of Using Results Chains The following benefits result from using Results Chains approach as a tool during program planning and implementation and for evaluation purposes: a. It helps make the end goals (impact, purposes, and sub-purposes) explicit. b. It makes clear the relationships linking the intervention to the end goal (purpose and sub-purpose), which allows critical assumptions, bottlenecks, challenges, and opportunities to be identified. c. It highlights the need for coordination among implementation partners and activities, and between interventions within the same activity. Identifying results throughout the ToC depends on this coordination. d. It helps identify, organize, and prioritize activities that lead to each outcome in the results chain. e. It defines the selected and prioritized indicators so that teams can measure effectiveness in the progress towards identified goals, test the assumptions, and answer critical management questions. f. It matches indicator reporting to specific actions, strategies, and activities, and identifies who needs to report which indicator. More importantly, it illuminates the reason a specific indicator needs to be reported and shows the required level of disaggregation, which allows partners to see each indicator’s utility as reported by their team and requested by the donor. g. It uses standardized terminology in a simple, graphic way, which helps address the issue of the Feed the Future program and its activities using terms in multiple ways—even within the same document. 2. Limitations that Might Affect the Use of Results Chains Results chains are powerful tools that support the design, implementation, monitoring, and evaluation of programs and projects. However, when results chains are only used in one step of the program cycle, their usefulness is limited. In the case of the Feed the Future Ethiopia program, the study team used the approach solely for a retrospective assessment of the program and two of its activities. For reconstructing the ToC and identifying key hypotheses and questions, results chains were valuable tools. However, the lack of a specifically designed monitoring system based on those results chains, and study team’s limitations to collect new data, 10 limited their power for testing and answering those questions and hypotheses. One drawback of using results chains is that they can quickly identify too many indicators. Using results chains helps identify the most important expected results in a ToC. However, assessing each result’s level of achievement requires one or more indicators. The relationships between results lead to identifying hypotheses, and each hypothesis can lead to several questions. Testing those hypotheses and answering those questions demands more data than assessing the status or level of achievement that the two related results can provide. The study team is not suggesting that Feed the Future Ethiopia, or any other program, follow a path that could lead to an unlimited number of indicators. Instead, results chains should be used as a way to identify the most important results, the most important hypotheses, and the most relevant questions. When those are determined, the program should strategically prioritize the key hypotheses and questions that, once tested and answered, will guide future project management decisions. The program should also identify the minimum number of indicators that would allow for hypotheses and questions to be addressed. 3. Conditions That Improve the Use of the Results Chains To improve results chains use, the program team must have sufficient capacity to use the approach efficiently throughout all project phases—design, implementation, monitoring, and evaluation. In the case of the Feed the Future Ethiopia program, sufficient capacity would mean that both the Feed the Future staff and partner institution staff have trained personnel in developing and using results chains. Institutional commitment also improves the use of results chains. The strength of the approach is amplified when different institutional layers (partners, activities, program, Missions, and Bureaus) adopt a similar approach. When this commitment occurs, it allows for standardizing, aggregating, reporting, comparing and, most importantly, learning. The approach should be incorporated into different business processes to ensure coordination among institutional tools and practices, such as work plans, monitoring systems, procurements, and reporting instruments. It is critical that teams use the approach as early as the design phase. By implementing the approach at the inception, they can overcome most of the limitations described above. Results chains are being used successfully by several USAID Offices and Missions around the world. Using the Measuring Impact mechanism, the USAID Forest and Biodiversity Office (FAB) is developing best practices for implementing the Program Cycle. Their best practices are specific to building biodiversity programs that enhance the evidence base and help USAID understand and comply with the Collaboration, Learning and Adapting policy. However, Feed the Future could build on FAB’s experience and adapt the use of results chains to its own needs. 11 ANNEXES 12 ANNEX A: LIST OF CONSULTED DOCUMENTS AGP-Agribusiness and Market Development- Life of Project Report: 2011-2016. Agricultural Growth Program – Agribusiness and Market Development (Agp-Amde) Project – Performance Management Plan (PMP) – November 2012. Agricultural Growth Program – Agribusiness and Market Development (AMDe) Project, Ethiopia. External Mid-Term Performance Evaluation Report. Feinstein International Center. Friedman School of Nutrition Science and Policy. Tufts University Africa Regional Office. Agricultural Growth Program (AGP) of Ethiopia: AGP I Impact Evaluation Report (Final). 2018. Alebel B. Weldesilassie, Tigabu Getahun, Jemal Ahmed, Hailemariam Teklewold, and Gebrehiwot Ageba. EDRI Research Report. Addis Ababa: Ethiopian Development Research Institute. Agricultural Growth Program – Livestock Market Development – Expanding Livestock Markets for the Small￾holder Producers – Work Plan (April 1, 2013-June 30, 2014). Agricultural Growth Program – Livestock Market Development – Expanding Livestock Markets for the Small￾holder Producers – Work Plan (July 1, 2014-June 30, 2015). Agricultural Growth Program – Livestock Market Development – Expanding Livestock Markets for the Small￾holder Producers – Year 4 Work Plan – (July 1, 2015-June 30, 2016). Agricultural Growth Program – Livestock Market Development – Year 5 Work Plan – July 1, 2016-July 31, 2017. Agricultural Growth Program – Livestock Market Development – Expanding Livestock Markets for Small￾holder Producers – Annual Report – (September 17, 2012-June 30, 2013). Agricultural Growth Program-Livestock Market Development – Expanding Livestock Markets for Small-holder Producers – Annual Report – (July 1, 2013-June 30, 2014). AGP-Livestock Market Development Project – Annual Report July 2015-June 2016. Agricultural Growth Program – Livestock Market Development – End Market Analysis for Meat/Live Animals, Leather and Leather Products, Dairy Products Value Chains – Expanding Livestock Markets for the Small￾Holder Producers. April 5, 2013. Agricultural Growth Program – Livestock Market Development. (AGP-LMD) Project. External Mid-Term Performance Evaluation Report. Feinstein International Center. Friedman School of Nutrition Science and Policy. Tufts University Africa Regional Office. May 2015. Agricultural Growth Program – Agribusiness and Market Development (AGP-AMDE) Project. Value Chain Strategy and First Annual Work Plan. November 2011–September 2012. Agriculture Growth Program – Agribusiness and Market Development (AGP-AMDe) – Nine Months Report – October 2012-June 2013. Agriculture Growth Program – Agribusiness and Market Development (AGP-AMDe) – USAID FY2014 Annual Report – July 1, 2013-June 30, 2014. Agriculture Growth Program – Agribusiness and Market Development (AGP-AMDe) – USAID FY2015 Year Four Annual Report – July 1, 2014-June 30, 2015. Bart Minten, Ermias Engida, and Seneshaw Tamru. 2016. How big are post-harvest losses in Ethiopia? Evidence from teff. ESSP Working Paper 93. 18 pp. Causal Mapping of the Gender Integration Framework. July 15, 2015. 13 Climate Change and Development: Experiences from Ethiopia and the Mercy Corps’ PRIME program. MercyCorps. Empowering New Generations to Improve Nutrition and Economic Opportunities (ENGINE): Impact Assessment. VALID Internacional. December 2016. Empowering New Generations to Improve Nutrition and Economic Opportunities (ENGINE). Year two work plan narrative (2012-2013). Save the Children Federation, Inc. September 2012. ENGINE: Empowering New Generations to Improve Nutrition and Economic Opportunities. Year III Work Plan Narrative. October 1, 2013-September 30, 2014. Save the Children. November 2013. ENGINE: Empowering New Generations to Improve Nutrition and Economic Opportunities. Year V Work Plan Narrative. October 1, 2014-September 30, 2015. Save the Children. September 2014. ENGINE: Empowering New Generations to Improve Nutrition and Economic Opportunities. Year IV Work Plan Narrative. October 1, 2015-September 30, 2016. Save the Children. December 2015. Ethiopia Development Trends Assessment. January 20, 2017. Ethiopia Performance Monitoring and Evaluation Services (EPMES). Fanaye Tadesse, Alemayehu Seyoum Taffesse, Tassew Woldehanna, Thomas Woldu, Ibrahim Worku, and Yisehac Yohannes/International Food Policy Research Institute. 2016. Feed the Future Ethiopia [2015] ZOI Interim Assessment Report. Ethiopia. Fantu Bachewe, Guush Berhane, Bart Minten, and Alemayehu Seyoum Taffesse. 2016. Non-farm income and labor markets in rural Ethiopia. ESSP Working Paper 90. 31 pp. Fantu Nisrane Bachewe, Guush Berhane, Bart Minten, and Alemayehu Seyoum Taffesse. 2015. Agricultural Growth in Ethiopia (2004-2014): Evidence and Drivers. ESSP Working Paper 81. 44 pp. Feed the Future Ethiopia 2011. Ethiopia FY 2011-2015 Multi-Year Strategy. Feed the Future Ethiopia External Mid-Term Performance Evaluation Report. Feinstein International Center. Friedman School of Nutrition Science and Policy. Tufts University Africa Regional Office. May 2015. Performance Evaluation: Graduation with Resilience to Achieve Sustainable Development (GRAD). Ethiopia Performance Monitoring and Evaluation Service (EPMES) for USAID/Ethiopia Project. March 2017. FY16 Feed the Future Ethiopia Key Issue Performance Narrative. March 1, 2017. Graduation with Resilience to Achieve Sustainable Development (GRAD). Draft Baseline Report. Burns & Bogale. February 2013. Graduation with Resilience to Achieve Sustainable Development (GRAD). Mid-Term Evaluation. Hirvonen, Kalle, Feiruz Yimer, Kibrewossen Abay, Fantu Bachewe, Guush Berhane, John Hoddinott, Guush Berhane, Mekdim Dereje, Bart Minten, and Seneshaw Tamru. 2017. The rapid – but from a low base – uptake of agricultural mechanization in Ethiopia: Patterns, implications and challenges. ESSP Working Paper 105. 32 pp. Joachim Vandercasteelen, Mekdim Dereje, Bart Minten, and Alemayehu Seyoum Taffesse. 2016. Row planting teff in Ethiopia: Impact on farm-level profitability and labor allocation. ESSP Working Paper 92. 25 pp. John W. Mellor and Paul Dorosh. 2012. Agriculture and Ethiopia’s Economic Transformation. February 2012. Kibrom A. Abay, Guush Berhane, Alemayehu Seyoum Taffesse, Bethelhem Koru, and Kibrewossen Abay. 2016. Understanding farmers’ technology adoption decisions: Input complementarity and heterogeneity. ESSP Working Paper 82. 21 pp. Levinson, J., Admassu, B., Trevors, T., and Teshome, A. (2014) Empowering New Generations to Improve Nutrition & Economic Opportunities. External Mid-Term Evaluation. 14 Margoluis, R., C. Stem, V. Swaminathan, M. Brown, A. Johnson, G. Placci, N. Salafsky, and I. Tilders. 2013. Results chains: a tool for conservation action design, management, and evaluation. Ecology and Society 18(3):22. Pastoralist Areas Resilience Improvement through Market Expansion Project. Year 2 Report FY 2014. Reporting Period: October 2013-September 2014. Mercy Corps. October 2014. Pastoralist Areas Resilience Improvement through Market Expansion Project. FY 2015 Report. Reporting Period: October 2014-September 2015. Mercy Corps. October 2015. Pastoralist Areas Resilience Improvement through Market Expansion Project. FY 2016 Annual Report. Reporting Period: October 2015-September 2016. Mercy Corps. October 2016. Pastoralist Areas Resilience Improvement through Market Expansion Project. Year 2 Work Plan Narrative. October 1, 2013 –September 31, 2014. USAID. Pastoralist Areas Resilience Improvement through Market Expansion Project. Year 5 Work Plan Narrative. October 1, 2016 –September 31, 2017. USAID. PRIME: Annual Household Survey. 2016. Report. Kimetrica. January 2017. Rui Benfica and James Thurlow. 2017. Identifying Priority Value-chains in Ethiopia. ESSP Working Paper 110. 15 pp. Sagara, B. and Hudner, D. 2017. Enhancing Resilience to Severe Drought: What Works? Evidence from Mercy Corps’ PRIME Program in the Somali region of Ethiopia. Seneshaw Tamru, Bart Minten, Dawit Alemu, and Fantu Bachewe. 2016. The rapid expansion of herbicide use in smallholder agriculture in Ethiopia: Patterns, drivers, and implications. ESSP Working Paper 94. 18 pp. USAID Ethiopia Country Development Cooperation Strategy 2011-2016. Accelerating the Transformation Toward Prosperity. March 2012. USAID Ethiopia Agriculture Value Chain and Private Sector Project (AVC&PS) [DO-1, IR-1 and IR-3]. Project Appraisal Document. July 6, 2015. White, S., Gebru, G., and Akilu, Y. (2015) Pastoralist Areas Resilience Improvement through Market Expansion (PRIME), Ethiopia. Mid-Term Evaluation Report. 15 ANNEX B: FEED THE FUTURE PROGRAM HIGH-LEVEL RESULTS CHAIN, KEY HYPOTHESES AND QUESTIONS High-Level Program’s Results Chain High-level program key hypotheses and questions Feed the Future key staff selected the following hypothesis as the most critical ones for High-Level program design. Hypothesis 1: If vulnerable households are either more productive or engaged in new economic opportunities (improved livelihoods), they will have a significant increase in income or assets (other benefits) sufficient to take them out of poverty, hunger and chronic malnutrition. Figure 5: Key hypothesis 1 in the High-Level Program’s Results Chain 16 Some key questions related to this hypothesis are: • • What is the level of increase income (or assets) sufficient to contribute to reducing poverty, hunger, and chronic malnutrition? Under which conditions does a household spend the increased income (or assets) to reduce their poverty, hunger and chronic malnutrition? Hypothesis 2: If the push-pull model generates new economic opportunities by increasing supply and demand for agriculture produce, then vulnerable households will engage in these opportunities. Figure 6: Key Hypothesis 2 in the High-Level Program’s Results Chain ● ● ● ● For this hypothesis to hold true, there needs to be: Enough new opportunities effectively available for vulnerable households; and, Vulnerable households need to: 1) have the capacity (tools, resources, knowledge and time) to engage in those new opportunities; 2) have a competitive advantage over other households to access those opportunities; and 3) be effectively linked to the new opportunities (geographical distribution, access to information). Therefore, some key questions related to this hypothesis are: Do vulnerable households have access to the new economic opportunities? Under which conditions (capacity, competitive advantage, functionally linked) do vulnerable households access these opportunities? 17 ANNEX C: INDICATORS AND MEASUREMENTS FOR THE FEED THE FUTURE PROGRAM HIGH-LEVEL ANALYSIS Compiled from multiple sources with MIRADI software: - - - - EG/HL: Feed the Future Monitoring System database IAR: Feed the Future ETHIOPIA – Zone of Influence – Interim Assessment Report (Hirvonen, et al., 2016). VAL: Empowering New Generations to Improve Nutrition and Economic Opportunities (ENGINE): Impact Assessment (Valid International, 2016). AGP: Agricultural Growth Program I – Impact Evaluation Report (Alebel B. Weldesilassie, et al, 2018) Expected result Indicator Measurement (Date Value) Trend Item Trend Details Reduced hunger and chronic malnutrition IAR. Prevalence of stunted children < 5 yrs (%) 2015-12-31: 47.1 Mild Decrease 2013-12-31: 49.2 Not Specified IAR. Prevalence of underweight children < 5 yrs (%) 2015-12-31: 26.4 Mild Decrease 2013-12-31: 31.9 Not Specified VAL. % of Woredas with sig. decrease in stunting . 2016-12-31: 35 Not Specified The stunting prevalence was reduced from 3.0 ppt to 24.3 ppt Reduced poverty IAR. Percent of people living on less than $1.25 (%) 2015-12-31: 35.0 Mild Decrease 2013-12-31: 39.9 Not Specified Improve nutritional status of women and young children VAL. % of Woredas with positive change in prevalence of mothers undernutrition 2016-12-31: 19 Not Specified Change in prevalence (% mothers) of maternal MUAC <230 mm Increased agriculture sector profitability EG.3.2-21. Number of firms (excluding farms) or civil society organizations (CSOs) engaged in agricultural and food security-related manufacturing and services that have increased profits or become financially self-sufficient with USG assistance 38 out of 40 FCUs are profitable. 2016-12-21: 1 Mild Increase 2016 Target: 63 – LOP Target: 244 2013/2016: 74 – Target: 225 18 2015-12-31: 68 Strong Increase Target: 60 2014-12-31: 5 Mild Increase Target: 53 2013-12-31: 0 Flat Target: 49 Increased vulnerable households income AGP. % increase in household revenue 2016-12-31: 37 Strong Increase 41% due to increased price and 59% due to increased quantity AGP. Mean HHs revenue (Birr) from the sum of crop, livestock and dairy sales 2016-12-31: 7118 Strong Increase AGP HHS Baseline: 4951 – Midline: 5967 - Endline: 7118 Non AGP HHs Baseline: 3725 – Midline: 5056 – Endline: 5046 – This increase was higher for female 2013-12-31: 5967 Strong Increase AGP HHs Baseline: 4951 – Midline: 5967 Non AGP HHs Baseline: 3725 – Midline: 5056 2011-12-31: 4951 Not Specified AGP HHs Baseline: 4951 Non AGP HHs Baseline: 3725 IAR. Average daily per capita expenditure (USD) Daily per capita expenditure as a proxy for income in ZOI 2015-12-31: 2.04 Mild Increase 2013-12-31: 1.98 Not Specified More agriculture related economic opportunities and jobs created EG.3-9. Number of full-time equivalent (FTE) jobs created with USG assistance 2016-12-31: 32.349 Strong Increase 2016 Target: 22.977 – LOP Target: 145.200 2013/2016: 90.886 – 2013/2016 Target: 137.763 2015-12-31: 20.639 Strong Increase Target: 7.905 2014-12-21: 2.975 Flat Target: 67.878 2013-12-31: 34.923 Strong Increase Target: 39.003 Increased value and volume of sales in domestic and export markets EG.3.2-19. Value of small-holder incremental sales generated (USD) 2016-12-31: 34.783.774,88 Strong Increase 2016 Target:16.311.529,12 – LOP Target: 47.910.408 2013 / 2016: 242.979.005 – Target: 32.969.456 2015-12-31: 387.447,26 Flat 2015 Target:13.902.878,94 2014-12-31: 45.606.016,81 Strong Increase 2014 Target: 2.755.047,57 2013-12-31: 162.201.765,64 Not Specified