Feed the Future Ethiopia EXTERNAL MID-TERM PERFORMANCE EVALUATION REPORT May 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Tufts University. The authors of the report are George Gray, Laura Kuhl and Demese Chanyalew. Agriculture Knowledge, Learning, Documentation and Policy Project (AKLDP), Ethiopia Feed the Future Ethiopia - External Mid-Term Performance Evaluation Report Feed the Future Ethiopia EXTERNAL MID-TERM PERFORMANCE EVALUATION REPORT USAID Contract number: AID-663-C-14-00002 Agriculture Knowledge, Learning, Documentation and Policy (AKLDP) Project Implemented by: Feinstein International Center Friedman School of Nutrition Science and Policy Tufts University Africa Regional Office PO Box 1078 Addis Ababa Ethiopia Tel: +251 (0)11 618014 www.fic.tufts.edu Disclaimer The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Feed the Future Ethiopia - External Mid-Term Performance Evaluation Report CONTENTS List of Acronyms and Abbreviations .................................................................................................................... i Acknowledgements ..................................................................................................................................................... ii Executive Summary ......................................................................................................................................................1 1. Introduction................................................................................................................................................................6 1.1 Purpose ..................................................................................................................................................................... 6 1.2 Methodology ........................................................................................................................................................... 6 1.3 Background to FTF projects in Ethiopia ..................................................................................................... 7 2. Results Framework ..................................................................................................................................................8 2.1 Outline of Framework and Indicators ......................................................................................................... 8 2.1.1 First Level Objective 1: Poverty Reduction ................................................................................... 10 2.1.2 First Level Objective 2: Improved Nutritional Status ............................................................... 11 2.2 Project Components ......................................................................................................................................... 11 2.2.1 FTF in relation to the Agricultural Growth Program ............................................................. 11 2.2.2 Geographical categories and reaching the poor ...................................................................... 12 2.2.3 Role of policy and enabling environment ................................................................................... 13 2.3 Assumptions and Constraints ...................................................................................................................... 13 2.3.1 Traction of production–focused interventions amongst vulnerable households......... 13 2.3.2 Potential for the generation of agricultural employment ....................................................... 14 2.3.3 Potential for off‐farm income generating activities ................................................................... 14 2.3.4 Timeframe of interventions and impacts ....................................................................................... 15 2.3.5 The importance of counterpart performance ............................................................................... 15 2.3.6 Potential for reduction of transaction costs .................................................................................. 16 2.3.7 Other constraints ...................................................................................................................................... 16 2.3.8 Summary ....................................................................................................................................................... 17 3. Progress against Indicators................................................................................................................................17 3.1 Overview of Progress To‐Date ..................................................................................................................... 17 3.2 Progress against Indicators .......................................................................................................................... 18 Goal: To sustainably reduce poverty and hunger in USG‐assisted areas ..................................... 18 First‐level Objective 1: Inclusive agricultural sector growth ............................................................ 19 IR 1: Improved agricultural productivity .................................................................................................. 20 IR 2: Expanded markets and trade ............................................................................................................... 22 IR 3: Increased investment in agriculture and nutrition‐related activities ................................ 24 IR 4: Increased employment opportunities in targeted value chains ........................................... 25 IR 5: Increased resilience of vulnerable communities and households ....................................... 25 First‐level FTF Objective 2: Improved nutritional status (women and children) ................... 26 IR 6: Improved access to diverse and quality foods .............................................................................. 27 IR 7: Improved nutrition related behaviours .......................................................................................... 28 IR 8: Improved use of maternal and child nutrition services ........................................................... 28 Summary ...................................................................................................................................................................... 29 4. Cross‐Cutting Issues ..............................................................................................................................................30 4.1 Gender .................................................................................................................................................................... 30 4.2 Climate Change ................................................................................................................................................... 31 4.3 Knowledge Management ................................................................................................................................ 33 5. Investment Quality ................................................................................................................................................35 5.1 Successful Investments ................................................................................................................................... 36 5.1.1 Resource Constraints ................................................................................................................................... 38 5.1.2 Relative Cost‐Effectiveness of Projects ................................................................................................ 39 5.1.3 Summary ........................................................................................................................................................... 41 Feed the Future Ethiopia - External Mid-Term Performance Evaluation Report 5.2 Opportunities for Reprogramming ............................................................................................................ 41 5.2.1 Immediate Reprogramming ................................................................................................................. 41 5.2.2 Redesign ........................................................................................................................................................ 42 6. Management Opportunities ...............................................................................................................................45 6.1 Portfolio Management ..................................................................................................................................... 45 6.2 Project Management ........................................................................................................................................ 47 7. Summary of Conclusions and Recommendations ......................................................................................47 Annexes A set of annexes were submitted as a separate document to the main report as follows: Annex A: Scope of Work Annex B: Bibliography Annex C: Interviews and field visits Annex D: Results frameworks and indicators Annex E: Statement of differences Annex F: Curriculum vitae for team members Annex G: Conflict of interest forms Feed the Future Ethiopia - External Mid-Term Performance Evaluation Report i List of Acronyms and Abbreviations AGP AKLDP ALT AMDe (or AGP‐AMDe) ATA BCC (or SBCC) CBO CDCS CIAFS COP CSO CVCA DAP DO DQA EGT EGTE ENGINE ETB FTF FTFMS GDP GEM GoE GRAD HABP ha IFPRI IR LAND LUI LMD (or AGP‐LMD) LOE LOP M&E MAD MFI MOTI MSME MTE PLI PMP PPP PRIME PSNP P4P qt RAIN REST SMFI SOW SPSNP TOPs Agricultural Growth Program Agriculture Knowledge, Learning and Documentation Project Asset Livelihood and Transition Office, USAID Agribusiness and Market Development (Agricultural Growth Program‐AMDe) Agricultural Transformation Agency Behavior Change Communication or Social Behavior Change Communication Community‐based Organization Country Development Cooperation Strategy Capacity to Improve Agricultural and Food Security Chief of Party Civil Society Organization Climate Vulnerability and Capacity Assessment Development Assistance Program Development Objective Data Quality Assurance Economic Growth and Transformation Office, USAID Ethiopian Grain Trade Enterprise Empowering New Generations to Improve Nutrition & Economic Opportunities Ethiopian Birr Feed the Future Feed the Future Monitoring System Gross Domestic Product Global Entrepreneurial Management Government of Ethiopia Graduation with Resilience and Development Household Asset Building Program Hectare International Food Policy Research Institute Intermediate Result Land Administration to Nurture Development Livelihoods Integration Unit Livestock Market Development (Agricultural Growth Program‐LMD) Level of Effort Life of Project Monitoring and evaluation Minimally Acceptable Diet Microfinance Institute Ministry of Trade and Industry Micro, Small and Medium Enterprises Mid‐term Evaluation Pastoral Livelihoods Initiative Project Monitoring Plan Public Private Partnership or Purchasing Power Parity Pastoral Resilience and Improved Market Expansion Productive Safety Net Programme Purchase for Progress Quintile Revitalizing Agricultural/Pastoral Incomes and New Markets Relief Society of Tigray Somali Microfinance Institution Scope of Work Special Productive Safety New Programme Transitioning out of pastoralism Feed the Future Ethiopia - External Mid-Term Performance Evaluation Report ii UNICEF USAID USG USGS VESA VSLA WATER WEAI ZOI United Nations Children’s Fund United States Agency for International Development United States Government United States Geological Survey Village Economic and Social Association Village Saving and Loan Association Water, Sanitation, and Hygiene Transformation for Enhanced Resilience Women’s Empowerment in Agriculture Index Zone of Influence Note In Ethiopia, a woreda is an administrative unit, roughly equivalent to a district in other countries. A kebele is equivalent to a sub‐district, being the next level of administrative unit within a woreda. Acknowledgements The evaluation team would like to express its gratitude and sincere appreciation to the following:  Mission staff for providing access to reports and data and describing their experience of working with FTF.  FTF project COPs and deputies for responding to our requests for interviews and providing access to their projects.  FTF project staff for organizing field trips and interviews and for providing first‐hand experience of the impacts of interventions both within and often way beyond working hours.  The MTE team was impressed by the consistent dedication of all stakeholders to the achievement of the overall FTF goals and is grateful for the time and assistance that was so readily made available. 1 EXECUTIVE SUMMARY This evaluation of Ethiopia’s Feed the Future (FTF) Program consists of an overview and assessment of program activities and achievements to date. The evaluation was tasked to: 1. Assess progress against objectives and goals specifically the extent to which planned results (both the quantitative and qualitative) have been achieved; 2. Assess the appropriateness (or effectiveness) of USAID investments in different program components and integrated programs areas at the activity level; 3. Identify actionable recommendations for reprogramming of funds to achieve program goals; 4. Identify actionable management recommendations to maximize the impact of the existing portfolio of investments, including the interactions among activities across components and integrated programs. The methodology applied to this evaluation has been based upon the detailed review of all documents directly associated with the program, combined with a qualitative assessment of program and project interventions through interviews and field visits. Neither a survey nor any other form of quantitative data collection was undertaken. Ethiopia’s FTF program has as its goal “To sustainably reduce poverty and hunger in USG‐ assisted areas”. This is to be attained through achievement of two first‐level objectives namely: 1) “Inclusive agricultural sector growth” and 2) “Improved nutritional status (women and children)”. The program builds upon investments under the Assets and Livelihoods in Transition office, which parallel the Government of Ethiopia’s PSNP initiative, and which are also funded from FTF. Beyond these investments, the FTF program1 has been based upon 16 core projects and a large number of smaller projects2. Nevertheless, achievement of the FTF results framework is largely dependent upon interventions under five main projects that utilize $214 million (78%) of the $273 million committed to the program viz. AGP‐AMDe, AGP‐LMD, GRAD, PRIME and ENGINE. It is these projects that are the main focus of this evaluation. An area of 149 woredas covered by these five projects is considered to be the FTF zone of influence, and it is amongst the population of this zone of influence that program impacts and outcomes are to be assessed. In undertaking this evaluation it became clear that the FTF results framework and that of the Country Development Strategy (Development Objective 1) under which projects had been originally designed, were not the same and that an assessment of performance using the FTF indicators would differ from a performance assessment based upon the framework and indicators of Development Objective 1. Accordingly the second part of this assessment analyses the two results frameworks, considering differences between the two frameworks, the value of indicators collated by the FTF management system and the relevance of the causal logic to the immediate FTF goals. Differences between the results frameworks suggest consequent disparities between program design and FTF priorities that reduce the likelihood of achieving first level FTF objective 1. While some FTF indicators are considered to be appropriate to the projects, others are not, and some key elements of project performance are omitted altogether if FTF indicators alone are used to assess progress. As its contribution to the Government of Ethiopia’s Agricultural Growth program (AGP), USAID has taken on the specific sub‐component of agribusiness and market development (and in some areas, support to production). Nevertheless, for a program such as FTF that is focused upon vulnerable households, the direct impact of activities designed to increase productivity, and enhance marketing and trade is limited. The degree of impoverishment of Ethiopia’s most vulnerable households, and in particular their limited access to land, prevents them from participating in many of the agricultural interventions proposed by AGP which the marketing component of Ethiopia’s FTF program is designed to support. A typical AGP smallholder cultivates 0.8 ha of land and can access ETB 5,824 Birr for inputs. By contrast, very few vulnerable households own more than 0.5 ha of land and are able to access more 1 Throughout this report, FTF is referred to as a “program”, while AGP‐AMDe, GRAD and other activities that contribute to FTF have been described as “projects”. 2 In addition to the above, the program includes four activities with FTF funding managed in other offices, six field support activities, six DCA projects, four grants to local organisations, and 21 different projects (mainly innovation laboratories) implemented through Bureau for Food Security central mechanisms. 2 than ETB 4,500 in loans. The successful performance of the AGP‐AMDe and LMD projects will certainly contribute to the AGP goal of agricultural sector growth, but it is not as clear that it will contribute to the FTF poverty reduction goal in the short‐term. While some vulnerable smallholders may become successful entrepreneurs, job creation in the value chains targeted by AGP‐AMDe and LMD ‐ may be the only sustainable poverty reduction strategy for many of the country’s poor. It is unlikely however that significant and visible progress can be made in this area within the five‐year time period of FTF. The above notwithstanding, the two results frameworks include very similar components to achieve the first level FTF objective 2 and there are no comparable challenges with translating the FTF indicators to DO1 of the CDCS. The evaluation considers progress against indicators. From an individual project perspective, AMDe and possibly LMD are on track to achieve the majority of their objectives. Similarly, GRAD and to a lesser extent PRIME might be expected to be effective in supporting the limited number of beneficiaries in the 46 woredas in which they operate. The extent to which ENGINE may be able to influence the first level FTF objective 2 is currently a matter of expert debate3 beyond the expertise of the MTE. It is difficult to draw rigorous conclusions from what is essentially a qualitative review, albeit based upon both individual project mid‐term evaluations, the results of the FTFMS, and the FTF MTE’s own observations. It is therefore helpful that parallel assessments have recently been made that can provide some indication as to the expected impacts of FTF interventions. In particular, the World Bank survey of poverty4 undertaken in 2014 showed that reductions in poverty that have occurred between 2005 and 2011 as a result of agricultural growth were exclusive of the most vulnerable households (those in the lower 15 percentile). The same report indicated that the positive impact of agricultural growth on poverty was only experienced by households living close to urban centers of more than 50,000 people, and that there was on average no impact on poverty from the use of improved inputs.5 A separate report has been produced by WFP on their Purchase for Progress program (P4P) in Ethiopia6, which has provided direct support to cooperative unions in the form of marketing contracts and facilitation of credit (working in concert in a number of cases with AGP AMDe). This showed no discernible impact of P4P interventions on four different indicators of household welfare (income, household assets, food consumption score and livestock ownership) over the period 2009‐2013. In the light of these observations, from the perspective of the goal and objectives of FTF as a whole, the MTE considers that when the mid‐line and end‐line survey results are compared on a rigorous basis with those of the baseline, the intermediate results, objectives, and program goals of the FTF program are unlikely to be met. This is primarily due to the dilution of individual project impacts across the entire FTF ZOI. The process of assessing progress against indicators raised a number of concerns regarding the nature and selection of indicators themselves. These included the FTF management system’s approach of combining indicator values from different projects that describe different variables, the absence of targets in some cases altogether and the observation that while business development and employment are critical to the push/pull model of development out of poverty, they are measured by only one FTF indicator in each case. Such observations suggest that the current framework of indicators does not provide a particularly strong basis for the assessment of progress towards the program IRs, Objectives and overall Goal. Program management is not facilitated by such a framework. 3 The MTE was presented with two contrasting perspectives on this issue, both of which were voiced by experts in their field whose experience exceeded that of any MTE team member. Accordingly no assessment could be made on this issue by this MTE other than to note the differences in expert opinion. 4 World Bank (2015).Ethiopia Poverty Assessment. Report No. AUS6744. World Bank Poverty Global Practice, Africa Region. 5 Positive impacts of improved input usage on poverty reduction were evident when the weather was good, but over the long term, on average there was no significant impact. 6 Kieger, D. (2014). The Impact of P4P on FOs and Smallholder Farmers in Ethiopia. World Food Programme, P4P Global Learning Series. 3 Nevertheless, overall, the MTE found no strong evidence to suggest that levels of effort and resources are not appropriately matched to the achievement of the output targets specified in the FTF management system and output indicator targets are generally on target to be met. Regarding impact and outcome targets, the picture is more variable. Targets in key areas of finance (especially for GRAD), employment (all projects) and off‐farm business development (GRAD and PRIME) are less likely to be achieved. The achievement of high‐level impact targets (where such exist) is more problematic still. It is unlikely that any of these targets will be met as a result of program interventions alone. The evaluation observed that in most cases, the resources constraining project achievements have been beyond project’s manageable interests. These have included inadequate production (AMDe and LMD), weak or shifting institutional counterparts (AMDe, LMD and ENGINE) as well as the limited availability of finance (all projects). Such a situation may in part reflect USAID’s recognition of L’ Aquila obligations to align with national programming, but it is evident counterpart capacity to perform must be realistically assessed if a program’s results are dependent upon it. Experience would suggest that the contrasting approach of funding parallel interventions, as applied to both PSNP and HABP (where USAID funds independent interventions that parallel government programs) is more robust than funding complementary interventions, as within AGP. Cross cutting issues of gender, climate change and knowledge management are assessed. Gender is being integrated into FTF through explicit strategies to promote women’s empowerment as well as efforts to mainstream gender into all FTF activities. Explicit strategies include GRAD’s work with VESA groups, the Women in Agribusiness Leadership Network established by AMDe and supported by LMD, and the activities of PRIME in the formation of the Somali Microfinance Institution (SMFI) (whose clients are 90% women), and support to the Women Traders Association in Jigjiga. Nevertheless, in terms of gender mainstreaming, no consistent or integrated FTF strategy could be discerned and the results appear to be more individual project outcomes, than a program level impact. Although there are some initiatives specifically targeting youth in GRAD and PRIME, a stronger emphasis on the needs of this vulnerable group is required. Particular attention should be paid to lessons learned from the pilot projects with TOPs in PRIME, which could help inform future programming. All project activities are designed to ensure minimal negative impacts on the environment and to address climate change. Nevertheless, only two of the five major FTF projects have a specific climate change component: GRAD and PRIME. In both cases, the primary achievement to‐date has been the development of climate vulnerability assessments. Each of the five main FTF projects has a learning and knowledge management component, but AKLDP is tasked to use experiences culled from FTF as a whole (as well as experiences from projects outside of FTF) to undertake knowledge management as a means to support improved agriculture and resilience programming, So far AKLDP knowledge management activities within FTF have focused primarily on evaluation as well as internal learning and coordination. It will be critical to ensure that a process for capturing the FTF experience and lessons learned is developed in the remaining stages of projects. The evaluation proposes a number of key areas for investigation and a possible model for knowledge management in the future. One challenge to this model is the role of AKLDP as the external evaluator for FTF projects. The evaluation noted a number of achievements worthy of remark, as summarized below. a) The success of GRAD in promoting the replication of its comprehensive support package across the future PSNP4 zone of influence.7 b) ENGINE’s substantial formative research that has led to the revision of behavioral change communication modules. c) The coordination between GRAD and LMD in the production and marketing of sheep and goats in Oromiya and Tigray. d) The work undertaken by PRIME in developing the innovative Sharia‐compliant Somali MFI. e) The Women in Agribusiness Leadership Network supported by AMDe and LMD. f) The support by AMDe for the construction of Ethiopia’s first fertilizer blending plant for Bicho 7 The next iteration of the PSNP is expected to expand into productive as well as food deficit woredas to meet individual household needs. 4 Woliso Cooperative Union and others yet to be completed. g) The promotion by ENGINE of water‐carrying by men in the village of Dembeli Keta is also a notable success that deserves further support. The quality of FTF investments is assessed in terms of the direct impact of investments, there is little consistency amongst FTF projects in terms of financial investment procedures or the principles upon which such investments are based, leading in some cases to large investments that are expected to result in “trickle down” benefits to households, although the extent and nature of such is not always clear. This is not unexpected when project management is required to achieve a targeted rate of disbursement and it is recommended that project output targets for grant disbursement should be avoided when the desired outcome (of business development) can be measured in other more direct ways. Evaluation of the relative cost‐effectiveness of projects is confounded by the fact that different projects within the FTF have very different objectives. Nevertheless, from the narrow perspective of contributing towards the achievement of FTF objectives, the successful implementation of GRAD and its consequent replication in the new PSNP4 has enabled it to leverage relatively modest USAID funding into national level finance. From this perspective, the project can be assessed as highly cost effective. PRIME has the potential to achieve the same results as GRAD, but from the narrow and short‐term FTF perspective, LMD and AMDe are unlikely to achieve the same level of cost effectiveness.8 ENGINE is similarly compromised through its obligation to work through the Ministry of Agriculture as a development partner, which may limit its impact. This does not mean that LMD, ENGINE and AMDe are not cost effective, but within the limited scope of the FTF objectives and the lifetime of the current FTF program, the immediate contributions that these programs have made when set against the budgets that have been expended are relatively lower than those of GRAD. It was noted however that investments made through the marketing components of FTF have been designed within a multi‐year framework that exceeds the FTF five‐year time frame. Both financial investments and the introduction of improved technologies will require ongoing support beyond the current program if they are to be effective in supporting the FTF objectives and goal. In evaluating program management, the evaluation found there to be little evidence of the specific and focused management required by a program of the size and complexity of FTF. The considerable number of demands upon the time of portfolio management at the Mission level has meant that the necessary capacity for oversight and response in the event of unforeseen developments is limited. Issues associated with indicators, targets, data reporting and data management all contribute to reduce the effectiveness of monitoring and management. Recommendations to improve the flow of management information are made in each case including the provision of additional resources to portfolio M&E. It is also suggested that the greater involvement of ALT in the oversight of the FTF program would allow that office’s experience and knowledge regarding poverty and food security to provide useful direction to the FTF program, especially insofar as it relates to the immediate needs of beneficiaries in the FTF ZOI. At the project level the performance of project management was generally professional and competent. While some concerns have been raised by individual project evaluations regarding the centralization of some projects, COPs and their managers were generally well informed of their project interventions and results and understood the causal pathways that contributed to their project goals. An understanding of the extent to which individual project goals might contribute towards the FTF objectives and goals was also both evident and realistic. The evaluation assessed potential areas for short and long‐term reprogramming. Potential areas for the immediate reallocation of funds with minimal disruption to the interventions that are already in place are suggested. These include the wheat value chain, chickpea, honey and meat export markets as well as further large scale grants. Funds freed up by curtailments in these areas could be used to 8 While both AMDe and LMD have been instrumental in the redesign of AGPII, this has not resulted in the leveraging of funds. AGPII design documents specify that USAID is expected to continue in its role as the major donor in the area of market development. 5 support domestic market strengthening as well as the training and mentoring required to strengthen the long‐term sustainability of interventions that are already in place. In the longer term, a future FTF program can take advantage of FTF project participation in the ongoing design processes for key government programs, most notably the PSNP4 and AGPII. These offer new opportunities for a program based upon the following principles: 1. A primary focus upon vulnerable households. The scaling up of PSNP4 to become a national rural program covering not only less productive woredas, but high potential areas as well, will allow a future program to integrate different interventions that impact both productive and less productive households within a single geographic area, with the ultimate goal of enhancing the incomes and food security of the most vulnerable. This change would help reduce the geographic constraints that have limited the effectiveness of FTF Ethiopia’s “push/pull” mechanism in the past. 2. The use of parallel as opposed to complementary programming to minimize those aspects of development beyond the manageable interest of the FTF program. 3. A stronger emphasis upon vertical integration to achieve the same end (see 5 below). Including the layering of development activities, including a) PSNP‐type food or cash transfers, b) GRAD‐ type aspirational development, fundamental financial literacy, small business development and social transformation (VESA‐type activities) and asset transfers, c) job‐creation and employment facilitation, as promoted by PRIME, d) ENGINE‐sponsored and directed nutritional activities covering all aspects of the stunting syndrome to achieve demonstrate the benefits of convergence, and e) AMDe/LMD market and business development to support the effective marketing of produce on the one hand and to facilitate business development and job creation on the other. 4. The restriction of all but marketing activities to a limited zone of influence commensurate with the resources available for investment, on the basis that demonstrable success can lead to wider scale replication and leverage of other donor finance. Such a restriction cannot apply to the systemic marketing components, which will require a broader ZOI if their interventions are to be effective. Nevertheless, the primary focus of such projects should remain impact the vulnerable household level 5. A strong emphasis upon training and capacity development within all projects, but recognizing that investment as asset transfer is an essential first step in the development process at the GRAD/PRIME level. The immediate objective of such a program would be to achieve the FTF Objectives within a limited ZOI. The broader goal would be to catalyse national‐level investment by government and donors so as to achieve FTF Objectives at a national level. Overall the MTE found that while individual projects are operating effectively, the FTF program is flawed by the assumption that support to agricultural production, marketing and trade through the AGP could reduce poverty amongst vulnerable households in Ethiopia within the context of a five‐year FTF program. That this does not appear to be the case does not imply that such activities are not essential to economic growth and consequent poverty reduction, or that they will not bear fruit over the long term, but they will have little impact upon the immediate Goal and Objectives of FTF. To be most cost effective, the investments made under the systemic marketing projects will require continued support beyond the initial five‐year time frame. A future FTF program to provide such support would be best provided within a limited area and would integrate the activities of current components of the FTF program in a layered approach, focused more directly upon vulnerable households and implemented in parallel with government interventions. 6 1. INTRODUCTION This evaluation of Ethiopia’s Feed the Future (FTF) Program9 has been undertaken at the midpoint of the program and consists of an overview and assessment of activities and achievements to date. It is not a review or compilation of the individual FTF project mid‐term evaluations that have been or are in the process of being undertaken at the moment. While this study has drawn upon the results and conclusions of those individual project evaluations where possible, the primary focus of this work has been the assessment of progress against the overarching FTF goal and objectives. The individual and coordinated contributions of projects toward FTF objectives has been the primary metric of this Mid‐ Term Evaluation (MTE). Its results and conclusions may therefore differ somewhat from those of individual project evaluations, reflecting the different contexts within which the different exercises have been undertaken. 1.1 Purpose The primary objective of the evaluation is to provide a perspective on the level of progress regarding the program’s planned results against stated output targets and strategic goals, the appropriateness of the Economic Growth and Transformation (EG&T) department’s investments in different project components and activities, and the linkages among components and integrated programs. The evaluation is also designed to provide specific information that will feed into EG&T’s Bureau of Food Security portfolio review in March and produce actionable management recommendations to aid future implementation. In line with these objectives, the evaluation is intended to achieve the following:  An assessment of progress against objectives and goals, specifically the extent to which planned results (both the quantitative and qualitative) have been achieved.  An assessment of the appropriateness (or effectiveness) of USAID investments in different program components and integrated programs areas at the activity level;  The identification of actionable recommendations for reprogramming of funds to achieve program goals;  The identification of actionable management recommendations to maximize the impact of the existing portfolio of investments, including the interactions among activities across components and integrated programs. The scope of work for this MTE is provided in Annex A. 1.2 Methodology The methodology applied to this evaluation has been based upon the qualitative assessment of program and project interventions through a detailed review of all documents directly associated with the program as well as to those peripheral to it (including Government of Ethiopia (GoE) policy and program documentation), in conjunction with interviews of key stakeholders in GoE counterpart institutions, program and project management and implementation as well as program beneficiaries, together with field visits to witness specific project interventions. Neither a survey nor any other form of quantitative data collection was undertaken, although extensive use was made of baseline and other annual survey data. A list of documents consulted is provided in Annex B, and a list of interviews and field visits in Annex C. The work was undertaken in four parts:  An initial visit to all the main project offices to discuss the key areas of interest for further investigation with Chief of Parties (COPs) and their deputies, and to request all possible program documentation. Interviews with some key stakeholders were also undertaken at this time.  A period of literature review to assess the documents provided including project proposals, program budgets, program monitoring plans (PMPs) quarterly and annual reports and spreadsheets downloaded from the web‐based Feed The Future Monitoring System (FTFMS). 9 Throughout this report, FTF is referred to as a “program” comprising a number of “projects” including AMDe, LMD, PRIME, GRAD and ENGINE. 7  A second visit incorporating a field visit to PRIME, GRAD, AMDe and LMD interventions and further interviews with COPs and key stakeholders.  A further period of results assessment and report writing. The evaluation has been a relatively complex process reflecting on the one hand the broad nature of the FTF program within which many different projects have been implemented, and on the other, the extensive volumes of data collected through the FTFMS that has been applied across all of those projects. The results obtained from the FTFMS were the most up‐to‐date that were available to the MTE team, but may not always reflect the latest data available at the individual project level. Nevertheless, it is not expected that such differences will substantially alter the conclusions in the narrative. 1.3 Background to FTF projects in Ethiopia Ethiopia’s FTF program is based upon the Mission’s FTF strategy generated in 2011. That document guided the incorporation of a broad suite of projects (some entirely new in concept, others already at a planning stage) into an overall program to achieve the original Development Objective (DO): “Increased growth and resiliency in Rural Ethiopia”.10 Most FTF projects are managed under the EG&T office, but some (PSNP DFAPs and GRAD) are managed by the ALT office, reflecting both the growth and resilience aspects of FTF. The objective would be accomplished by promoting a market‐based value‐chain approach to increasing rural on‐and‐off farm productivity, expanding domestic and international market access, increasing the capacity of businesses, strengthening financial markets, improving the regulatory environment for trade, and increasing that ability of safety‐net beneficiaries to participate in the market by graduating them off assistance and into productive agriculture or employment opportunities. All of the above build upon the assistance provided to vulnerable households through the Productive Safety Net Program (PSNP). The PSNP is critical to the achievement of FTF goals. USAID does not fund the PSNP directly but using FTF funds, provides parallel support to three Cooperating Sponsors mainly in the form of food, which is then transferred to targeted beneficiaries in specific woredas through interventions that mirror the GoE PSNP activities. The importance of the PSNP is two‐fold, on the one hand it substantially improves the food security of targeted beneficiaries, and on the other by removing the need to make “fire‐sales” to raise cash for food, it protects household assets. In this way, the PSNP and parallel USAID interventions create the environment in which households can positively respond to GRAD, PRIME, ENGINE and the more systemic marketing initiatives, AMDe and LMD. Ethiopia’s FTF projects have been implemented within the context of the PMP results framework for the DO outlined above. Section 2 highlights the differences between the USAID Ethiopia PMP results framework and that of the global FTF, the latter being more production orientated. The foundation underlying the USAID Ethiopia PMP results framework is the PSNP, which utilizes approximately US$100 million annually. The USAID Ethiopia PMP framework builds upon the impacts of the PSNP through the implementation of 16 different projects, of which nine might be considered to be the main FTF components namely Agricultural Growth Program‐Agribusiness and Market Development (AGP‐AMDe), Agricultural Growth Program‐Livestock Market Development (AGP‐LMD), Graduation with Resilience and Development (GRAD), Pastoral Resilience and Improved Market Expansion (PRIME), Empowering New Generations to Improve Nutrition & Economic opportunities (ENGINE), Capacity to Improve Agriculture and Food Security (CIAFS), Land Administration to Nurture Development (LAND), Water, Sanitation, and Hygiene Transformation for Enhanced Resilience (WATER), and Agriculture Knowledge, Learning and Documentation Project (AKLDP). An additional seven projects have undertaken various interventions that have supported the achievement of the results framework, but the majority (92%) of the FTF budget and most of the interventions have fallen under these nine projects. Both CIAFS and LAND have made important contributions towards the strengthened agricultural policy and land management, and the WATER program which ended in 2014 has improved pastoralists’ access to clean and sustainable water sources, hygiene awareness, and access to 10 This development objective differs somewhat from that of FTF, which has two objectives i.e. “Inclusive agricultural sector growth and “Improved nutritional status (women and children)”. 8 sanitation.11 Nevertheless, achievement of the FTF results framework is largely dependent upon interventions under the five projects listed below:  AGP‐AMDe (implemented from May 2011, budget of $49.8 million) is designed to complement the production‐focused interventions of the GoE Agricultural Growth Program (AGP) through the strengthening of markets and market linkages within six specific value chains (wheat, maize, chickpea, coffee, sesame, and honey). It operates throughout 96 of the woredas in which the AGP is also functional.  AGP‐LMD (implemented from September 2012, budget of $38.0 million) is intended to achieve similar objectives to AGP‐AMDe but with a focus on meat and dairy production and is similarly operational throughout 46 AGP woredas.  GRAD (implemented from December 2011, budget of $23.4 million) works in 16 woredas of which nine are not included under the AGP, but are located close to AGP woredas, to assist households within the Productive Safety Net Programme (PSNP) to achieve sustainable food security.  PRIME (implemented from October 2012, budget of $52 million) works in 30 pastoral woredas (one only within the AGP) to facilitate both livestock production and marketing on the one hand and to help those transitioning out of pastoralism (TOPS) to achieve food security.  ENGINE (implemented from September 2011, budget of $55.7 million) is mandated to work with the Ministries of Health and Agriculture to strengthen health delivery services with a specific focus upon improved nutrition and to promote nutrition sensitive agriculture through DAs, FTCs and woreda. Its Zone of Influence (ZOI) covers all AGP woredas as well as 6 GRAD and 16 PRIME woredas. In addition to the above, AKLDP (budget of $8.5 million) is intended to support evidence‐based learning as a means to improve policy and programming, drawing on both USAID‐funded projects and the projects of other donors and implementers. Due to both time constraints and the economic importance of the five largest projects, this evaluation is limited to an assessment of these five projects. The overall area of project implementation includes the 111 AGP woredas that fall under the aegis of AGP‐AMDe, AGP‐LMD and ENGINE and the additional 38 woredas beyond the AGP woredas where GRAD and PRIME are active. This somewhat diverse group of 149 woredas, drawn from both the most productive and some of the least productive areas altogether constitute the Feed The Future Zone of Influence (FTF ZOI) and it is the population of vulnerable households within the FTF ZOI who are the ultimate targeted beneficiaries of the FTF program. 2. RESULTS FRAMEWORK 2.1 Outline of Framework and Indicators There are two results frameworks of relevance to the Ethiopia Feed the Future initiative: the global FTF results framework and the results framework used in the PMP for Development Objective 1 of the Country Development Cooperation Strategy (referred to as the USAID Ethiopia PMP). These are detailed together with relevant indicators in Annex D and shown diagrammatically overleaf. Each individual project also has its own project‐level results framework. This section addresses the logic of these results frameworks in terms of the likelihood that activities designed within these frameworks will achieve the two FTF goals of poverty and stunting reduction. It also looks at the compatibility of the two frameworks and the appropriateness of the FTF indicators for measuring the progress within the Ethiopian portfolio. The USAID Ethiopia PMP has many areas of complementarity with the global FTF results framework, but there are several areas where the USAID Ethiopia PMP departs from the global FTF strategy, with important implications for design, monitoring and reporting. 11 See: USAID: Final performance evaluation of water sanitation and hygiene transformation for enhanced resiliency (WaTER) project. IMAWESA: The voice of an Agent of Change http://imawesa.info/2013/04/02/the‐ voice‐of‐an‐agent‐of‐change/ accessed May 2015. 9 Development Objective 1 Feed The Future 10 2.1.1 First Level Objective 1: Poverty Reduction The FTF results framework posits that within the project timeframe of five years, increasing agricultural productivity, marketing and trade will lead to poverty reduction through two main causal pathways: a) increasing productivity and resulting incomes for smallholder producers, and b) job creation in the targeted value chains. IR5 of the FTF results framework acknowledges the role of social safety nets and resilience measures for vulnerable households as necessary components of a poverty reduction strategy, but this component addresses the need for measures to maintain assets and ensure that vulnerable households don’t fall into greater poverty, and are not intended to increase incomes directly. The USAID Ethiopia PMP presents a similar logic, although it is important to note that the key development objective for the USAID Ethiopia PMP is increased growth and resiliency, which is broader than the specific poverty reduction goal of FTF. In the Ethiopian context, the validity of the first causal pathway that activities focused on increasing productivity, marketing and trade can directly lift smallholder households out of poverty (defined as below the global $1.25 per day poverty line) has to be questioned. Even in the comparably better‐off AGP regions, which comprise the majority of the FTF zone of influence, the average landholding size is less than half a hectare. Analysis by AMDe of the potential for growth in Ethiopian agriculture suggests that most growth has and will continue to come from farms of at least 0.75 hectares, suggesting that efforts focusing on marketing of agricultural surpluses will primarily engage producers that are better‐off than the average household, and are unlikely to impact smaller households directly. While there are opportunities for productivity initiatives to reach smallholders below the poverty line, and evidence elsewhere12 has suggested that agricultural sector growth can lead eventually to poverty reduction13,14, marketing initiatives such as AMDe or LMD are less likely to do so directly within the timeframe of the FTF program. The second causal pathway to poverty reduction posited by the FTF strategy is job creation in the targeted value chains. Because of Ethiopia’s high rural population density and the lack of productive potential in some parts of the country, job creation is the only sustainable poverty reduction strategy for many of the country’s poor, a point emphasized in a recent report by Ethiopia’s New Climate Economy Partnership (EDRI and GGGI), which stated that “urbanization will increasingly play an important role in realizing our ambition to achieve lower middle income status by 2025”.15 However, it is questionable whether efforts focused on the FTF value chains will produce enough jobs in the short‐term for this to be a substantial pathway out of poverty in the 5‐year time period of FTF. Empirical evidence suggests that significant increases in productivity are required before producers hire outside labor, relying first on their own household labor.16 Indeed, with increases in productivity, there is a tendency for increased investment in labor‐saving technologies, (some of which are promoted by FTF), so that the elasticity of employment of agricultural growth is often low.17 In terms of value addition, the promotion of increased investment in processing and marketing will undoubtedly lead to increased employment opportunities, but will not occur unless the production of a reliable commercial surplus can be achieved (a key area in which the AGP has yet to fulfill its mandate) and will in any case result primarily in urban and peri‐urban rather than rural employment. In the long term, new opportunities, outside of agriculture but based upon the increased wealth of profitable agricultural producers are recognized to be the most common pathway out of poverty18, but such developments are not immediate. They require both agricultural growth and subsequent 12 Dorosh, P. and Mellor, J. (2013). Why agriculture remains a viable means of poverty reduction in Sub‐Saharan Africa: The case of Ethiopia. Development Policy Review, 31(4): 419‐441. 13 Irz, X., Lin, L., Thirtle, C. and Wiggins, S. (2001). Agricultural productivity growth and poverty alleviation. Development Policy Review, 19(4): 449‐466. 14 Thomas, G. and Slater, R. (2006). Innovation, agricultural growth and poverty reduction. International Journal of Technology and Globalisation 2(3/4): 279‐288. 15 Neway Gebreab (2015). “Unlocking the Power of Cities in Ethiopia”. Forward in: Ethiopia’s New Climate Economy Partnership (EDRI and GGGI), Addis Ababa. 16 USAID (2013). “Evaluation of the Push/Pull Hypothesis”, USAID Addis Ababa. 17 Dorosh and Mellor (2013) suggest that this elasticity is approximately 0.3. 18 Ibid. 11 investment in the provision of goods and services. Such investment is very important but commercial logic dictates that it should be concentrated in peri‐urban rather than rural areas.19 Regrettably, there is no obvious short‐term solution that could be expected to significantly increase the incomes of the most vulnerable smallholder households in rural Ethiopia. While small reductions in poverty can be achieved through technical, financial and social interventions, significant and sustainable reduction is fundamentally dependent upon the overall growth rate of the national economy. This is largely beyond the manageable interest of FTF and for this reason, the broad‐based poverty reduction objective is unlikely to be achieved within the five‐year timeframe stipulated in the program design. This is not to suggest that interventions to promote growth are not essential to addressing poverty in the long‐term; indeed this analysis suggests that it likely to be the only approach to sustainably improve the incomes of the poor, but such structural changes will not be fully measurable in the timeframe of the FTF initiative. 2.1.2 First Level Objective 2: Improved Nutritional Status FTF and DO1 of the Ethiopian Country Strategy include very similar components to achieve the nutrition goal of stunting reduction. Access to diverse foods, behavior change and utilization of maternal and child health services represent the key components of both strategies. As a result of this alignment, there are no comparable challenges with translating the FTF indicators to the USAID Ethiopia PMP. At the same time, there are several other issues with the nutrition strategy in both frameworks. The contrast between the level of complexity in the poverty reduction and the nutrition components is noteworthy. In the USAID Ethiopia PMP, four IRs contribute to the poverty reduction target, while only one IR addresses nutrition. In the global FTF framework, there are three IRs for nutrition, but none of them have any sub‐IRs. Both approaches suggest that either a) the stunting target is less complex and requires fewer types of interventions to achieve, or b) the causal pathways to achieve stunting reduction are less well‐understood. Evidence suggests that stunting is a multi‐faceted issue.20 The implementers of ENGINE have identified at least eight actors that may contribute to stunting including: 1. Exclusive breastfeeding 2. Complementary nutrition 3. Animal protein intake 4. Maternal age and body weight 5. Potability of water 6. Sanitation 7. Vitamin A, iron and zinc intake levels 8. Aflatoxin contamination. The development of outcome sub‐IRs which reflect progress in any or all of the above would be helpful in managing for results under objective 2. 2.2 Project Components 2.2.1 FTF in relation to the Agricultural Growth Program The Ethiopian FTF strategy can only be understood in relation to the Agricultural Growth Program (AGP) of the GoE. AGP is a government‐led, multi‐donor supported initiative designed to increase the productivity of “high potential” designated regions of the country. Increased productivity in these regions is supposed to drive Ethiopia’s economic growth, and is thus a central pillar of the national Growth and Transformation Plan. As its contribution to the AGP, USAID has taken on the specific sub‐ component of agribusiness and market development. As its name suggests, the primary goal of the AGP is agricultural growth, with a strong emphasis on exports, which is central to the country’s development 19 A view strongly endorsed by the New Climate Economy Partnership between the Ethiopian Development Research Institute and the Global Green Growth Institute in their report “Unlocking the Power of Ethiopia’s Cities”. 20 Stewart, C.P., Iannotti, L., Dewey, K.G., Michaelsen, K.F. and Onyango, A.W. (2013). Contextualising complementary feeding in a broader framework for stunting prevention. Maternal and Child Nutrition 9 (Suppl. 2):27‐45. 12 priorities and need for foreign exchange. While the AGP strategy seeks to ensure that such growth will be inclusive, it is important to recognize that the ultimate metric of success is agricultural growth, and thus, while complementary, the goal of AGP and of Feed the Future are not the same. The design of the AGP‐AMDe and LMD projects will certainly contribute to the AGP goal of agricultural sector growth, but it is not as clear that they will contribute to the FTF poverty reduction goal in the short‐term. Due to the role of FTF within the larger AGP, a discrepancy arises between the USAID Ethiopia PMP and the global FTF results framework regarding the role of increasing productivity of the agricultural sector. The first IR of the CDCS is “Performance of the agriculture sector improved.” At a superficial level, this appears very similar to the FTF IRs 1 and 2 “Improved agricultural productivity” and “Expanded markets and trade,” but a key difference is that the USAID Ethiopia PMP purposefully does not address increasing productivity directly. The first Sub‐IR of the USAID Ethiopia PMP: “Increased productivity and income through expanded market opportunities” explicitly identifies that the increased productivity and incomes will come from expanded market opportunities, not from efforts or activities on production. The other Sub‐IRs that contribute to FTF IR 1 include: “Increased commercial viability of small and medium agribusinesses,” “Increased technology transfer” and “Improved agricultural enabling environment.” The only one of these that leaves an opportunity to address productivity for smallholders directly is “increased technology transfer.” While the decision to focus on these aspects of the agricultural sector is reasonable in light of the role that the government and other donor‐led components of AGP are designed to take in addressing the agricultural productivity components of the agricultural sector and is in line with a division of responsibilities among the Ethiopian government and its development partners, it presents a challenge for mapping the FTF indicators to the country strategy, since a significant number of the FTF indicators fall under the FTF IR of increasing productivity. Since the Mission, in collaboration with the Ethiopian government and other donors, is not responsible for the productivity component of the AGP, it is challenging to understand how the Ethiopian FTF activities will achieve ambitious targets in those FTF indicators designed to address productivity. The approach taken by the Mission has been to place a number of these indicators under the “increased technology transfer” component of the USAID Ethiopia PMP, but it is an imprecise translation and it is not clear exactly how technology transfer for productivity fits into the agribusiness and marketing projects in the AGP geographic areas. Neither AMDe nor LMD are well‐designed to deliver on the agricultural productivity indicators of FTF. Their “middle of the value chain” interventions are not particularly well‐suited to address the household‐level productivity indicators included in the global FTF results framework. Such indicators fail to capture the types of “system‐level” changes marketing projects like AMDe and LMD are designed to address, and instead projects are reporting on indicators that are only tangentially related to their core activities. In addition to unrealistically expecting productivity results from marketing and agribusiness projects, judging AMDe and LMD using these indicators does not do justice to the core activities of these projects, which are intended to address barriers at the “system‐level” and not address productivity‐focused indicators. 2.2.2 Geographical categories and reaching the poor Another challenge is the strong and explicit geographic division between the productive areas and vulnerable areas in the USAID Ethiopia PMP results framework. The global FTF strategy mirrors this distinction between productivity and vulnerability, but does not make any reference to geographic divisions. The geographic division between productivity and vulnerability present in the USAID Ethiopia PMP is driven by the political categorization of the country into PSNP and AGP woredas, but doesn’t necessarily speak to the realities of the target beneficiaries of FTF. The poverty reduction targets are based on the poverty rate within the zone of influence (the majority of which is within AGP woredas, although GRAD woredas, which are PSNP, and PRIME woredas, which include both PSNP and non‐PSNP woredas are also included). High levels of poverty also exist in the AGP woredas, and the achievement of the FTF poverty‐reduction target is dependent upon interventions reaching the poor within the AGP woredas. The current FTF strategy however, is aligned with the CDCS, which distinguishes between productive activities (i.e. marketing and agribusiness activities) in the AGP woredas and resilience activities (i.e. livelihood activities designed to target the poor) in the PSNP woredas. This distinction essentially means that the poor (or those who are less productive) within the 13 AGP woredas (who comprise the majority of the poor that need to be reached to achieve the FTF goals) are not explicitly addressed by the current portfolio of FTF programs. It also means that the full range of activities undertaken by GRAD and PRIME are not captured by metrics under the resilience component of the global FTF results framework, as neither project is purely about protecting against shocks (this is the role of PSNP). PRIME, and to a lesser extent, GRAD, have important marketing and business development components. 2.2.3 Role of policy and enabling environment Apart from these important differences in the two results framework, and corresponding development strategies, there are a number of less significant but still incongruous issues with the results frameworks and corresponding indicators. One is that under the “agricultural enabling environment,” the FTF indicator on agricultural policies (4.5.1(24)) is not included, but rather placed outside the USAID Ethiopia PMP results framework, which suggests a lack of importance given to policy within the concept of the enabling environment. This limited clarity on the role of policy can also be seen in a number of projects, where policy processes are not well‐integrated, nor is a sufficient evidence‐base being generated in a systematic way to inform policy processes. Although a number of projects are reporting on their impact on agricultural policies others, particularly GRAD, are not, although GRAD has probably had the most impressive impact on government and donor policy through the PSNP4 redesign. In addition, the FTF indicator, which is specific to agricultural policy, provides no space for projects (particularly ENGINE) to report on nutritional policy impacts, even though this is just as central to the overall FTF goals. The policy indicator is also problematic because it focuses on the “number of policies that might be improved” irrespective of the nature of such policy improvements. These might range from the trivial to the substantial or from being controversial to being straightforward to implement, (e.g. being already agreed, requiring only technical drafting). Moreover, in some cases, targets were specified prior to the completion of the identification of policies to be addressed, counterparts to be involved, or the extent of cooperation required in each case. This is scarcely a realistic approach to either program design or monitoring and it is recommended that the empirical identification of actual policy weaknesses and counterpart needs be completed prior to the setting of future policy targets. This critique is not intended to suggest that projects are not contributing to policy processes, but rather to note that the results frameworks and reporting mechanisms do not provide a clear logic for the role of policy reform in the achievement of the goals of FTF nor are they being reported consistently. 2.3 Assumptions and Constraints A number of assumptions are implicit in the design of the FTF program, some of which have proved to be correct, although a substantial number have not been borne out. Many have been effectively encapsulated in the “push/pull” hypothesis, which suggests that support to agricultural development and marketing would create opportunities for production, labor and the provision of good and services that could be exploited by vulnerable households to increase their resilience. While this hypothesis may be correct in general terms, specific aspects of its implementation have proved problematic. 2.3.1 Traction of production–focused interventions amongst vulnerable households A primary constraint which was well understood from the program outset, and which has been subsequently verified by baseline surveys, is that the most vulnerable households lack adequate productive resources (especially land) to allow them to take advantage of agriculturally focused interventions. Thus, the AGPII program design targets a smalholder of 0.8 ha and requires the investment of ETB 5,824 in crop inputs. Even in AGP woredas, a significant proportion of households have less than 0.5 ha of land21 and MFI’s will provide access to credit of no more than ETB 4,500. In order for a household producing cereal crops, such as maize or wheat, on a landholding of 0.5 ha, to rise 21 The CSA Agricultural Survey for 2010/11 reports that 34% of smallholdings are less than 0.5 ha in size and 61% are 1.0 ha or less. A further 25.5% are between 1.0 and 2.0 ha and only 13.5% of all holdings are more than 2.0 ha in size. 14 above the FTF poverty line of $1.25 per person per day, yields would either need to increase by more than 100 or 200% of current yields, which would be a remarkable, if not impossible, achievement. See discussion in section 3 on IR5 for more details. This constraint together with the inaccuracy of the first assumption has meant that initiatives such as those of AMDe and LMD which rely upon the promotion of yield and strengthened markets to increase the agricultural gross margins of producers are unlikely to have any direct benefit to the most vulnerable households in the FTF ZOI. Indeed, the immediate effects of such interventions upon market dependent households may actually be to reduce food security by increasing the farm‐gate prices that the poorest are obliged to pay22, which are already among the highest in Africa. Moreover, the assumption that a “pull” from strengthened markets for production from vulnerable households who would also be “pushed” by facilitating their capacity to produce, did not fully accommodated the costs of aggregation and of the management and coordination required to develop sustainable linkages. Neither did it allow for the geographical separation of the different projects generating the “push” and “pull”. This implies that while projects such as AMDe and LMD are essential to the overall development of the agricultural sector, their beneficial impact upon vulnerable households in the FTF ZOI will be primarily through the creation of agricultural labor and other economic opportunities to provide goods and services to an increasingly affluent section of the rural economy, i.e. smallholders with larger landholdings23, who will be better able to take advantage of the enhanced production and marketing opportunities that AGP, AMDe and LMD can provide. The same logic underpins the emphasis on support to those transitioning out of pastoralism in PRIME. This indirect “wealth” effect is not immediate and those investing in response prefer peri‐urban as opposed to rural locations for their businesses. 2.3.2 Potential for the generation of agricultural employment General experience elsewhere in the world suggests that the employment elasticity of agricultural income is less than unity, implying that as the agricultural sector develops, the proportion of the rural population employed within it will ultimately decrease. Thus in the long term, the primary benefit of enhanced agricultural production and marketing to vulnerable households will be the indirect business and employment opportunities that the demands of an affluent agricultural sector will create. While some opportunities may develop immediately as a result of investment in adding value to agricultural products, such direct benefits should not be confused with the more general and widespread economic opportunities that will arise through the increased flow of finance to efficient producers. Indeed, such direct employment opportunities are as much drivers of rural economic development as the growth of the agricultural sector itself. Hence, while projects such as AMDe and LMD may report the number of jobs increased as a result of specific investments, in the long term these represent a relatively small proportion of the overall economic development that will be the ultimate benefit of these programs, being not so much impacts as drivers of further change. 2.3.3 Potential for off‐farm income generating activities One key assumption underpinning both GRAD and PRIME has been that off‐farm income generating opportunities would be available to rural smallholders as alternatives to traditional agricultural livelihoods. This would allow income to be generated both from new small businesses and from the employment opportunities that such new businesses might generate. In practice it has been observed that such opportunities are more restricted than anticipated for four reasons:  First, the proportion of the population willing to risk their savings in small businesses is low (the 22 Jayne, T.S., Yamano, T., Nyoro, J. and Awuor, T. (2001). Do Farmers Benefit From High Food Prices? Balancing Rural Interests in Kenya’s Maize Pricing and Marketing Policy. Tegemeo Working Paper 2B. Tegemeo Institute of Agricultural Policy and Development, Nairobi. 23 The threshold landholding size is a matter of some debate. It has been suggested (Dorosh and Mellor, 2013) that it could be as low as 0.75 ha. While this might be possible in conjunction with other sources of off‐farm income, a minimal size of 2 ha is considered necessary to sustain an exclusively agricultural livelihood. 15 Global Entrepreneurial Management (GEM) survey undertaken for Ethiopia in 201324 estimated that only 7.4% of the population were successful entrepreneurs) and may be even lower amongst poorer households, so that employment opportunities associated with such small businesses is also low.  Second, the economic issue of services, utilities and other supports, as well as the market for new goods and services is not yet adequately developed to support much new business development.  Third, the availability of finance necessary to develop such businesses is substantially less than current demand.  Fourth, a significant proportion of those entrepreneurs who do obtain the necessary finance to engage in off‐farm income generation immediately relocate to a peri‐urban environment where the availability of both utilities and markets is far greater than in small rural communities. For all of these reasons, the development of rural off‐farm income generating activities has not proceeded at the rate anticipated at the program design stage when it was envisaged that such off‐farm IGAs would contribute significantly to rural economic development in the short term. This has not occurred and the development of off‐farm IGAs (other than sheep and goat rearing and/or petty trade) continues to be a challenge facing both GRAD and PRIME. 2.3.4 Timeframe of interventions and impacts Most importantly however, the push/pull hypothesis postulates an increased demand for goods and services generated by the increased incomes of more productive farmers, which is widely recognized to be the most substantial impact of agricultural development upon vulnerable households, but this development is by no means immediate. Although various stages of development may be concurrent, there is clearly some element of sequencing. The process of increasing agricultural production will in itself take some years. The secondary process of linking agricultural growth to the development of a goods and services sector will take longer, and the tertiary process of expanding that goods and services sector to become a major source of employment may take longer still. This raises the issue of time as a key constraint to the effectiveness of the systemic projects (especially AMDe and LMD) whose pro‐poor impacts are predicated mainly on the impacts of secondary and tertiary development outlined above. Throughout this report, emphasis has been placed upon the extent to which various interventions are likely to meet FTF targets within the time frame of the program time. From this perspective, many of the systemic project interventions will yield few benefits in five years alone, but over the longer term of 10 to 15 years, the systemic changes introduced by these projects will be essential to pro‐poor development. Targets set at the inception of FTF expecting change within five years were not realistic and that short time frame is of itself a constraint to the success of these projects as it might be perceived in the immediate term. 2.3.5 The importance of counterpart performance Both AMDe and LMD were designed on the premise that smallholders would increase their yields as a result of AGP interventions and that the two marketing projects would work with increasing commercial surpluses that could be readily marketed. In practice that has not been the case. Despite two favourable seasons over much of the country25 when yields have increased, anticipated commercial surpluses have not materialized. This suggests that the observed increases in yields may have had an impact on food security, but have not increased sales or incomes significantly. Despite the fact that agricultural production has increased by over 4.5% per annum, prices have rarely fallen below export parity so that anticipated export markets for maize, chickpeas, meat and honey have been weak, (although domestic markets as well as the market for live animal exports have been strong and farm gate prices are above those in most countries). This has hindered export marketing initiatives which have consequently been constrained in their impacts. 24 Amha,W., Woldehanna, T.,Tamrat, E. and Gebremedhin, A. (2014) “Characteristics and Determinants of Entrepreneurship in Ethiopia”. Ethiopia Inclusive Finance Training and Research Institute (EIFTRI). 25 With the exception of north western Tigray, which experienced a poor sesame harvest in 2014. 16 As a consequence, possibly of the lack of production of a nature and volume suitable for export, both AMDe and LMD have extended the reach of their projects to include production‐focused interventions, especially training in enhanced technologies for both crop and livestock production. While such activities may exceed the mandate of either project as they were originally conceived, they are nevertheless logical developments under the circumstances and highlight the inadequacy of the assumption that liaison with AGP production initiatives would be sufficient to develop effective and sustainable value chains. Instead, the MTE observed that there was a strong tendency for both projects to become involved throughout the full length of each value chain, to ensure both the quality and volume of produce entering the marketing chain. Observation would suggest that such vertical integration might play an important role in reducing the risk of non‐performance by counterparts and enhancing the overall effectiveness of the assistance provided. With regard to counterparts, a second assumption underlying both of the marketing projects was that there would be effective and stable counterpart institutions in place, to ensure the effective coordination of project interventions with GoE activities. In practice this has not been the case. In 2013, the GoE moved responsibility for the marketing of agricultural products from the Ministry of Agriculture to the Ministry of Trade and Industry (MOTI). This obliged AMDe to establish new linkages within MOTI. At the same time, the new Livestock State Ministry established in 2013 has been in the process of developing its own institutional infrastructure so that dialogue with LMD has been less directed than one would hope. This has reduced the extent to which either AMDe or LMD have been able to work effectively with the GoE, although the situation is now improving considerably. 2.3.6 Potential for reduction of transaction costs The extent of market imperfections was originally judged to be sufficient that improvements in market efficiency could, by reducing transaction costs, simultaneously reduce prices to the consumer while maintaining (or even increasing) prices to the producer. In practice this has not been the case. In the case of teff26, the disparity between producer and consumer prices has been found to amount to only 20% of the retail price allowing limited room for improved prices to either party through increased market efficiency. This is a significant error since a reduction in prices to the consumer was considered to be a potentially major aspect of improved food security amongst the most vulnerable households who are predominantly market‐dependent to complete their annual food needs. If as recent studies suggest27 a sustainable reduction in food prices is unlikely to occur, then increased food security must be achieved through increased agricultural production or other sources of household income. 2.3.7 Other constraints The two projects focusing on more direct support to households have also faced constraints. As noted in section 3.2, some of GRAD’s target households have been unable to access finance to the extent anticipated and this has limited the speed with which graduation might be achieved, but two other constraints have been equally important. The first has been the vast predominance of shoat fattening as an off‐farm income generating activity. This may reflect the desire of households to own livestock either as a stepping stone towards the purchase of an ox for draft power, as well as for the nutritional and/or economic benefits. As a result, many households who have received shoats have tended to deviate from the fattening program upon which their business plans were based towards less profitable, but more favoured shoat rearing (under which a higher proportion of the animals would be retained – thus reducing profitability). This tendency was not unexpected, but has highlighted the need for enhanced formative research to understand people’s aspirations (in this case to sell fewer and retain more 26 Although teff is not an FTF value chain, it has been extensively studied and robust data on the various margins are available. Data for other cereal crops is less robust but analyses of farm gate and retail prices provide similar results. 27 Minten, B., Tamru, S., Engida, E. and Kumar, T. (2013). Ethiopia’s Value Chains on the Move: The Case of Teff. ESSP Working Paper 52, IFPRI. http://www.ifpri.org/sites/default/files/publications/esspwp52.pdf accessed May 2015. 17 shoats) if project interventions are to achieve their targets. The trend has also highlighted the second substantial constraint of livestock feed as a fundamental input to shoat rearing as an off‐farm IGA. The widespread perception that shoats and other livestock can be find adequate feed from communal grazing lands, field and roadside margins is no longer valid. Only those households with clear rights are able to access such grazing and for the most part, shoats are either tethered or housed to be fattened and fed using the cut and carry system. Feed is a substantial cost of sheep and goat fattening28 which is increasing as the practice becomes more prevalent and as grazing land is increasingly brought under the plough. The profitability of one of the main elements of the GRAD methodology is thus coming under increasing pressure and it will be necessary to address the issue of feed production and marketing more comprehensively if the impacts of GRAD are to be sustained. 2.3.8 Summary Overall, the MTE found that key assumptions regarding the potential for market strengthening to enhance poverty reduction in the short term, the impact of agricultural improvement upon the most vulnerable households and the capacity of counterpart institutions have not yet been validated. While definitive results will be provided by the mid‐line survey to be undertaken later this year (2015), the qualitative assessment of the market developments, impacts on vulnerable households and counterpart capacities suggest that these factors all constrain the impact of market development projects such as AMDe and LMD as far as the immediate FTF objectives are concerned. At the same time it was observed that assumptions regarding access to finance, opportunities for off‐farm income generation and availability of livestock feed have also been inaccurate, limiting the effectiveness of interventions of GRAD and PRIME. Most importantly however, the push/pull hypothesis, while potentially valid in the long‐term has but few examples of its successful application to date, suggesting it to be of limited relevance to short‐term development objectives. 3. PROGRESS AGAINST INDICATORS This section first considers the general progress of the FTF program before assessing progress against individual FTF indicators. 3.1 Overview of Progress To‐Date At the time of the MTE, all of the five main FTF projects had been operational for at least 28 months so that it is feasible to comment on the general progress made under the program. It is evident that from an operational perspective, AMDe, GRAD and ENGINE have been able to deploy their available human and financial resources efficiently and in line with project expectations, while LMD and PRIME have been engaged in less definitive activities as part of their learning curve. Nevertheless in all cases, the baseline survey across the FTF ZOI has highlighted the extent and depth of poverty and the substantial levels of stunting that the FTF program seeks to address. The inclusion of households outside the FTF ZOI will allow for a rigorous assessment of impact, but already shows the endemic nature of many aspects of vulnerability, which are prevalent within the productive AGP woredas as well as outside them. The baseline survey clearly indicates the considerable impacts required if project interventions are to meet the FTF program goals. Thus it would appear that while AMDe and possibly LMD are on track to achieve the majority of their individual project objectives, they are unlikely to be as successful in achieving FTF impacts at a scale necessary to achieve program objectives. This is due in part to constraints arising from invalid assumptions made during FTF program design, (as outlined in section 2.3) and in part to the length of time required for agricultural market‐focused impacts to influence the income and food security of the most vulnerable households. Conversely, while GRAD and to a lesser extent PRIME might be expected to have a direct impact in supporting the limited number of beneficiaries in the 46 woredas in which they operate, their effects will nevertheless be diluted within the overall FTF ZOI, which is three times 28 According to data provided by GRAD, feed costs vary substantially according to circumstances from 16% to 44% of input costs. 18 greater in size. The extent to which ENGINE may be able to influence the second FTF objective is currently a matter of debate beyond the expertise of the FTF MTE.29 It must be emphasized that this evaluation is not a quantitative assessment based upon primary data, but a qualitative exercise using whatever data has been collected to date. A rigorous quantitative assessment can be made once the mid‐line data collection exercise (planned to begin in June 2015) has been completed. Nevertheless, progress achieved to date, provides only limited evidence that the capacities of component projects are adequate to achieve impacts across that same area within the program timeframe. The projects that focus directly upon vulnerable households (PRIME and GRAD) are limited in their geographic scope, while the more systemic impacts of the market‐focused projects (LMD and AMDe) have yet to reach the vulnerable households that are the primary targets of FTF. As a result, when the mid‐line and end‐line survey results are compared on a statistically rigorous basis with those of the baseline, the intermediate results, objectives and program goal of the FTF program are unlikely to be met within the limited five‐year time frame specified by the program design. In addition to providing a qualitative assessment of the likelihood that project activities will lead to the high‐level FTF targets, the MTE can assess the progress toward output indicators based on project reporting. As discussed in section 2, there are some logical challenges linking progress on the outcome indicators to the likelihood of achievement of the higher‐level targets, and thus the significance of success or failure for the outcome targets is dependent on the logic by which they contribute to the higher‐level objectives. 3.2 Progress against Indicators Goal: To sustainably reduce poverty and hunger in USG‐assisted areas Progress towards this goal is measured using two high‐level population impact indicators:30 1. Prevalence of Poverty: Percent of people living on less than $1.25. 2. Prevalence of underweight children under five years of age. Such population indicators are assessed only by the baseline, mid‐line and end‐line surveys. A statistically valid survey protocol, adequate to identify significant differences of 10% or more, has been developed and will be used to assess progress. The survey samples both treatment and control groups to determine the impact of interventions through a matching pair “difference in difference” methodology that allows for the impact of changes to both groups caused by factors other than the FTF interventions (provided such factors act equally upon both groups). The methodology substantially enhances the extent to which observed changes might then be attributed to the FTF interventions. This means that changes in indicators due to a nationwide increase in per capita GDP, or to a change in cereal prices (which might impact the poverty line) will be ignored. Since the midline survey has not yet been undertaken, it is impossible to provide objective estimates of progress against the two goal indicators. Nevertheless, the baseline survey results together with some of the other lower level impact and outcome indicators allow some preliminary conclusions to be drawn, as below: 1. The target for the first indicator is to reduce the prevalence of poverty by 30% from the baseline level of 34.5% to a LOP target of 24.2%. Such a reduction could be achieved through increased agricultural incomes (due to enhanced production or farm gate prices), increased off‐farm income 29 The MTE of ENGINE concluded that the objective of reducing stunting by 20% was unlikely to be achieved. Project management disagrees, noting that convergence of many factors is reducing stunting rapidly. Both opinions are those of experts in nutrition. The FTF MTE team lacks the expertise to assess which expert opinion is more valid. The mid‐line survey to be undertaken this year will provide an objective assessment. 30 Throughout this analysis, the term impact indicator is used to denote impacts beyond the direct influence of an intervention. Outcome indicators measure the direct results of project interventions and output indicators measure the performance of the intervention itself. Thus, “number of farmers trained in new technologies” is an output indicator, “number of farmers applying new technologies as a result of training “is an outcome indicator and “increased yield due to the application of new technologies” is an impact indicator. 19 generating activities, or increased availability of employment. Assessment of the relevant indicators below suggests that productivity amongst the poor households will not be sufficiently improved to meet the target, that labor opportunities are limited, and that off‐farm income generating activities, while moderately successful in increasing household income, are currently limited in their scope and may be constrained by external factors, so that this target is unlikely to be met. Ethiopia’s target of a 30% reduction in poverty is very ambitious, and exceeds the targets set by other FTF focus countries, most of which have set a 20% poverty reduction target. 2. The target for the second indicator of the FTF Goal is not specified in the PMP. The baseline survey of the FTF ZOI indicated that in mid‐2013 this prevalence was 32.1%, which is high, but not significantly different from the prevalence in non‐FTF woredas. Given the observed improvement in food security that has occurred through 2013 and 2014 as a result of two consecutive good Meher harvests, it is quite probable that the prevalence of underweight children will have fallen, but it is impossible to say if the impact of the FTF program will have been sufficient to result in a significant difference between the levels within and beyond the FTF ZOI. First‐level Objective 1: Inclusive agricultural sector growth This first‐level objective is assessed by three indicators: 1. Percent change in agricultural GDP. 2. Daily per capita expenditures (as a proxy for income) in USG‐assisted areas. 3. Women’s Empowerment in Agriculture Index (WEAI) Score 1. Agricultural GDP has grown substantially and continuously over the last nine years at rates of between 4% and 11%. The extent of reported agricultural growth exceeds that of all other developing countries with the exception of Egypt (which benefitted from extensive installation of irrigation systems) and the accuracy of national statistics has been questioned on more than one occasion. This impact indicator is measured at a national level and as such will be similar both within and outside the FTF ZOI so that attribution to FTF interventions will be impossible. 2. The second indicator measures the purchasing capacity of the household. This impact indicator was directly measured at the baseline survey and will be reassessed in subsequent mid‐line and end‐line surveys. The FTF PMP does not specify a target for this indicator. The baseline figure for mean daily expenditure in the FTF ZOI is ETB 17.49 per capita. On an adult equivalent Purchasing Power Parity (PPP) basis, this amounts to $2.19 per day. This mean value is 75% higher than the $1.25 specified as the poverty threshold. Indeed when disaggregated by poverty status, the average daily expenditure of the 34.5% of people below the poverty line was found to be ETB 8.1, while the average expenditure of the 65.5 % above was ETB 28.1 (i.e. 246% greater than that for the poor population). These disparities highlight the inadequacy of the indicator in measuring the impact of FTF interventions upon poverty. It may be possible to affect the incomes of the poor without raising them above the FTF poverty line, given the large gap between the current incomes of the poorest households. Given fundamentally commercial interventions, it is quite possible for mean incomes to increase substantially in the short term without measurable impact on the incomes of the poor if those interventions act primarily upon the wealthier households who have the assets and capacity to respond. This is not a concern across all of the FTF ZOI, within which both GRAD and PRIME focus specifically upon poor households. Nevertheless, both AMDe and LMD are more income‐neutral in their immediate impacts and pro‐poor activities are less evident in these projects. 3. The third indicator uses a specific index (Women’s Empowerment in Agriculture Index ‐WEAI) that is estimated using 6 different parameters collected by the baseline, mid‐line and end‐line surveys. The baseline score estimated for the FTF ZOI was 0.698. There is no PMP target. This is not unreasonable given that the WEAI has not yet been widely used and there is no body of empirical knowledge to link WEAI scores with different degrees of development of food security. As a result, it is impossible to determine what WEAI might reasonably be expected to result from the five year FTF program. Moreover, the WEAI combines scores from five different domains so that the root cause of disempowerment remains unknown unless the index is disaggregated. As a result, this 20 impact indicator is of limited value to current program management, although the indices used to compile it may provide useful data for future programming. The First Level Objective 1 is supported by five intermediate Results (IRs). Progress towards each IR is assessed as follows: IR 1: Improved agricultural productivity There is only one indicator for this IR, along with indicators for a number of Sub‐IRs, discussed below. The indicator for IR 1 is: Gross margin per hectare, animal or cage of selected product. This is measured by both AMDe and GRAD, but not by LMD, PRIME or ENGINE (although all three of the latter projects contain elements of agricultural/livestock production that can be subjected to gross margin analysis). Since gross margins can be subject to so many factors related to both productivity and price, the usefulness of this impact indicator either as a reflection of productivity, or as an indicator of project management performance is questionable. The FTF PMP LOP targets for this indicator are: wheat $1028; maize $675; coffee $1972; sesame $1302; chickpea $770; cow $100; sheep $30; and goat $30. The baseline figures recorded by AMDe were: wheat $551; maize $387; coffee $1807; sesame $555; and chickpea $715. Those recorded by GRAD were: cow $40; sheep $10; and goat $10. Given that gross margins are subject to so many influences, it is not surprising that progress to date has been variable. AMDe results extrapolated from a recent (2014) results survey are not comparable with the baseline survey31 and some gross margins that had exceeded targets in 2013/14 (e.g. sesame) have declined substantially in 2014/15. GRAD results show moderate progress to date. Overall, it is not possible to assess progress on the basis of available data. Nevertheless, given the limited progress achieved towards sub IRs, as measured by outcome and output indicators and described below, it is unlikely that the few targets that have been set within the PMP for this indicator will be met, but the significance of this indicator is weak. Sub‐IR 1.1 “Enhanced human and institutional capacity development for increased sustainable agriculture sector productivity” is assessed on the basis of five indicators. 1. “Number of farmers and others who have applied improved technologies or management practices as a result of USG assistance” ‐ the LOP target for this outcome indicator is 273,022 while the actual figure to date is assessed at 341,038 i.e. substantially above target. Nevertheless, these figures should be compared with a baseline estimate of rural households within the FTF ZOI of 3,577,837, i.e. at the mid‐point of the program, less than 10% of households have adopted improved technologies or management practices that might lead to an increase in gross margins. Future interventions and indeed program design would be enhanced if this indicator could be disaggregated by wealth group to determine if improved technologies being used mainly by wealthier farmers. 2. “Number of individuals who have received USG supported short‐term agricultural sector productivity or food security training” ‐ the LOP target for this output indicator is 387,954. To date, 335,883 have received training. While this number is already 86% of LOP target, it represents both agricultural (80%) and food security (20%) training, some of which is of extremely short duration (field days and cooking demonstrations). Moreover, the number of people trained is slightly less than the number that have applied new technologies, implying a remarkably high adoption rate of over 100%. Again however, the data should be compared with the total number of households within the FTF ZOI, on such a basis it is hard to see how training at a frequency of less than 10% could yet have significantly affected gross margins across the ZOI. Literature on technology adoption suggests that repeated engagement in training is necessary for adoption of new technologies or practices. Thus, one‐time training is unlikely to lead to sustained changes in agricultural practices.32,33 31 The annual survey sampled a population of relatively well‐educated high‐performing farmers, who would be expected to have higher‐than‐average yields, which makes it impossible to extrapolate from these results to the general population that was sampled in the baseline survey. 32 Foster A. and Rosenzweig, M. (2010). Microeconomics of technology adoption. Annual Review of Economics 2, 395‐424. 21 3. “The number of food security private enterprises (for profit), producers’ organizations, water users’ associations, womens’ groups, trade and business associations, and community‐based organizations (CBOs) receiving USG assistance” ‐ the LOP target for this output indicator is 7,579, while the number to date is 11,542. GRAD and AMDe have substantially over performed, while LMD has achieved 71% success and PRIME has yet to make significant progress against this indicator. Unfortunately it is not possible to assess the extent to which USG assistance will impact either the sub IR or the IR itself. In some cases, a relatively low LOP has resulted in major impacts (such as the achievement of certification through LMD assistance, to permit exports from abattoirs), while in others, substantial effort has yet to provide significant benefit (such as the coordination of co‐operatives for the sale of wheat to EGTE in an undersupplied market). The significance of this output is weak, especially when compared with the following indicators. 4. “Number of private enterprises, producers organizations, water users associations, women’s groups, trade and business associations and community‐based organizations (CBOs) that applied improved technologies or management practices as a result of USG assistance” ‐ this outcome indicator is formally reported on by AMDe and GRAD only. In practice, LMD and PRIME are also engaged in business development and have made some progress in this area, but their individual project results are not incorporated within the FTFMS. The LOP target for the indicator is 2,825. Progress to date exceeds this target by 12%. According to the FTFMS, the result is strongly biased by the success of GRAD, although in practice, the other projects may also have had more of an impact than has been quantified. 5. “Number of people implementing risk‐reducing practices/actions to improve resilience to climate change as a result of USG assistance” ‐ the LOP target for this outcome indicator is 71,827. Only GRAD and PRIME report on this indicator and the two projects have already exceeded this target by 19%. Neither AMDe nor LMD report on this indicator. AMDe in particular might consider the impact of some of its interventions on resilience to climate change. The introduction and widespread uptake of single varieties of maize, wheat or chickpea can have potentially negative impacts upon resilience. Sub‐IR 1.2 “Enhanced technology development, dissemination, management and diffusion” is assessed on the basis of four indicators: 1. “Number of hectares under improved technologies or management practices as a result of USG assistance” ‐ the LOP target for this outcome indicator is 2,939,600 ha. This has already been surpassed due to the reporting by PRIME of 4,862,773 ha under improved rangeland management.34 The other project reporting (AMDe) has met 86.7% of its individual LOP target of 55,658ha. The inclusion of large pastoral areas that are extensively managed together with small areas belonging to highland smallholders generates a statistic that has little meaning. In particular, the significance of the contribution of improved land management to the Sub‐IR is impossible to determine. 2. “Number of vulnerable households benefiting directly from USG interventions” ‐ the overall FTF LOP target for this output indicator is 1,257,868, and the actual level of achievement is 1,254,787. Theoretically, the FTF LOP target has almost been met. In practice however, the bulk of these results are made up of PSNP, Pastoral Livelihood Initiative (PLI), and Peace Corps beneficiaries. GRAD and ENGINE are the only two of the main five FTF projects reporting on this indicator. The LOP target for these is 81,170. Both projects have exceeded their targets already with a combined level of achievement to date that is 73% above target. It is unclear why LMD and PRIME do not report on this indicator since they both provide some assistance to vulnerable households, but their project data is not included in the FTFMS. AMDe also impacts households through the provision of seed and training programs, but it is debatable whether or not such households are actually vulnerable. 3. “Number of technologies or management practices in various phases of development as a result of USG assistance” ‐ AMDe is the only project of the main five FTF projects reporting on this 33 Feder. G., Just, R.E. and Zilberman, D. (1985). Adoption of agricultural innovations in developing countries: A Survey. Economic Development and Cultural Change 33(2): 255‐298. 34 In this case, the inclusion refers only to the process of mapping units of rangeland. 22 output indicator with an LOP target of 25 different technologies and practices and a current level of achievement of 10. It would appear that LMD, GRAD or PRIME should also be reporting on the new technologies that they have developed (e.g. PRIME’s work with Sharia‐compliant MFIs, or LMD’s work in conjunction with GRAD on livestock feed utilization), but these are not included in this indicator. 4. “Number of water resources sustainability assessments undertaken” ‐ is reported only by the USGS Groundwater project. This project was not evaluated in detail, but is reportedly successful. In addition to the above, it was noted that a fifth FTF indicator: “Number of climate vulnerability assessments conducted as a result of USG assistance”, might also be considered. Such assessments have been conducted by both GRAD and PRIME, but no targets for this output indicator have been set and neither project is reporting upon it. Overall, it is evident that the indicators used to assess progress towards Sub IR 1.2 do not provide a sound basis for such a determination. Reporting on some indicators is incomplete, while other data is potentially misleading. In terms of the Sub‐IR itself, the MTE team assessed that there had been some diffusion of enhanced technologies, especially of improved seeds, and improved rangeland management, but neither the real extent of such diffusion nor its impact could neither be assessed from the indicators available. Sub‐IR 1.3 “Improved agricultural policy environment” is assessed on the basis of a single outcome indicator namely, “Number of agricultural and nutritional enabling environment policies completing various processes/steps of development as a result of USG assistance”. This indicator is reported by AMDe, LMD and PRIME but not by GRAD. The LOP target for the three projects is 31. The actual achievement to date is substantially higher at 83, based upon the inclusion of work by CIAFS, PLI and ATA, which have together contributed an additional 46 policies, but even without this supplementation, the three main FTF projects have already exceeded the LOP target by 20%. While this result appears impressive, counterpart stakeholders did question the process by which relevant policies were selected for “improvement” and the ultimate meaning of these statistics in the absence of final adoption is unclear. Conversely, GRAD has clearly had a substantial impact upon GoE policy as evidenced by the incorporation of many of its key principles into the redesign of the PSNP, yet this impact is not at all evident from this indicator which GRAD does not report upon. Indeed, even if it were to do so, it would be inadequate to describe the potential extent of the project’s influence, which can be expected to be much greater than the addition of a single policy to this indicator could possibly convey. It is perhaps inevitable that the simplistic numerical indicator used to describe progress towards this Sub‐IR should be inadequate to capture the real extent of policy improvement, which is essentially qualitative in nature and whose progress is by not always immediately evident. This is especially the case in a country such as Ethiopia, where policies that are forward looking in nature, may be implemented from an unexpected perspective that can lead to very different results from those anticipated originally. IR 2: Expanded markets and trade Progress towards this IR is measured by two impact indicators, as well as indicators for various sub‐ IRs: 1. “Value of incremental sales (collected at farm‐level) attributed to Feed the Future implementation” ‐ this indicator could be reported upon by AMDe, LMD, PRIME and GRAD, but in practice GRAD is not reporting this data. Results for LMD are based upon detailed assessment of beneficiary sales, but for AMDe and PRIME, results are extrapolated from samples. Although current reported incremental sales are more than 97% of the LOP target, the MTE team was concerned that figures extrapolated from sampling that is not representative of the total FTF ZOI population may tend to overestimate the result reported for this indicator. 2. “Value of exports of targeted agricultural commodities as a result of USG assistance” ‐ this indicator is reported upon by both AMDe and LMD. Currently reported incremental exports stand at 50% of the LOP target, but results to date are subject to the same concerns as the value of incremental sales, i.e. both DQA assessments and inspection of data suggest that total rather 23 than incremental sales are being counted in some cases at present. Moreover, given that the main export crops are sesame and chickpeas and that the production and price of sesame have both declined dramatically over the last twelve months, while exports of Kabuli chickpeas have been weak, it is quite possible that this indicator may actually decline over the course of the next year. For these reasons, while LMD estimates may be realistic, those for AMDe may not reflect the actual extent of market expansion well so that the accuracy and predictive value of the combined indicator is open to question. It is again recommended that this indicator should be reassessed more rigorously before it can be used to assess actual progress. Sub‐IR 2.3 “Improved market efficiency” is assessed by a single indicator, the “Total increase in installed storage capacity”. This output is reported mainly by AMDe with an LOP target of 155,520 M3 and a 36% achievement to date of 55,424 M3, of which 2,504 M3 has been generated by PLI. Given that the budget through which this increase in storage capacity has been achieved is almost exhausted, it appears unlikely that the LOP target will be met. Nevertheless, the recommendation made in the MTE for AMDe, that greater emphasis should now be placed upon the consolidation of investments made to date appears to be sound and would probably result in as great an enhancement of markets and trade as would further physical expansion of storage. Sub‐IR 2.4 “Improved access to business development and sound and affordable financial and risk management services” is assessed by one impact and two output indicators: 1. “Value of agricultural and rural loans” ‐ is reported as an impact indicator by AMDe, GRAD and LMD. The LOP target is set at $76,838,238 and the value achieved to date is $66,109,903, i.e. 86% of target. Most of this is due to the activities of AMDe. Significantly PRIME does not report on this indicator, despite its activities in support of the Somali Micro Finance Institute, suggesting that this indicator is underreported. Overall, however, it would appear that notwithstanding difficulties experienced by GRAD in obtaining loans from MFIs, and the generally expressed reluctance on the part of MFIs to lend to PSNP beneficiaries, progress towards the achievement of the FTF LOP target has been good. 2. “Number of MSMEs, including farmers, receiving USG assistance to access loans” ‐ is reported upon by AMDe, LMD and GRAD, but again, not by PRIME. The LOP target for this output indicator is 59,634 MSME’s of which 40,499 (68%) have been assisted to date. Examination of the data shows that more than 99% of the MSMEs assisted have been GRAD beneficiaries. While progress towards the target appears good, the activities and impact of AMDe and LMD working with processors, cooperatives and small businesses are almost completely obscured in the FTF headline reporting. The impact of PRIME’s work on micro finance and business development is ignored. 3. “Number of MSMEs, including farmers, receiving business development services from USG assisted sources” ‐ is reported by AMDe and GRAD, but not LMD or PRIME. The LOP target is 64,952 and the current level of achievement is 57,740 (87.7%). This indicator is once again dominated by GRAD, which has an individual LOP target of 59,000 MSMEs and has so far assisted 51,819. AMDe has a target of only 254 MSMEs and has already assisted 223. The balance is due to the activities of PLI and other small programs. Overall, it is evident that all four of the main agriculturally focused FTF projects have improved access to business development services. Progress towards the achievement of targets for each indicator appears good. Nevertheless, the numerical data hides three key weaknesses of the program:  First, as remarked above, the data refers primarily to the activities of GRAD, which has assisted individual households to access loans to develop MSMEs. In doing so, the project (and indeed most of the AGP) encountered the obstacle of MFIs’ refusal to make finance available until previous loans (made through the earlier Other Food Security Programme, under uncertain conditions and subject to some controversy) had been repaid. This significantly reduced the amount of finance that could be made available to loanees. Until GRAD can negotiate either the write‐off or at least a moratorium on these debts, this credit constraint will remain a significant stumbling block to the access of finance.  Secondly, the sustainability of access to loans is not yet secure. While more than 80% of all GRAD 24 applicants received loans in the first round of borrowing, the group collateral system35 has meant that less than 30% have been able to borrow a second time and less than 10% a third.  Finally, the development of business plans has been a key aspect of assistance, but there has not always been a good understanding of the process by beneficiaries. Given the high rate of illiteracy (slightly above 70%) within the FTF ZOI, this is not unexpected, but it does mean that the value of business plan preparation is limited and that smallholders may not benefit from the assistance provided to the extent expected. IR 3: Increased investment in agriculture and nutrition‐related activities Progress towards this IR is measured by two FTF indicators: 1. “Value of new private sector investment in the agriculture sector or food chain leveraged by Feed the Future implementation”‐ this outcome indicator is reported by AMDe, LMD, and PRIME. The FTF LOP target is $85,577,424 and the current value of leveraged investment is reported to be $29,861,065 (34.9% of LOP target). The results suggest that the program is on‐track to meet its target, although mid‐term evaluations raised concerns regarding PRIME’s accounting of assets provided by the project as part of the leveraged investment. 2. “Number of public‐private partnerships36 formed as a result of Feed the Future assistance”‐ this indicator is reported upon only by AMDe, although it might be expected that both LMD and PRIME might also be active in this field.37 Nevertheless, for AMDe the LOP target is 90 PPPs and achievement to date is 171 (including 14 formed through the ATA). The target has therefore been comfortably exceeded, largely by the development of partnerships with USAID through the signing of innovation fund agreements. The sustainability of such partnerships beyond the life of the FTF program is as yet uncertain. As well as the two indicators listed above, IR 3 is also monitored through one of two Sub‐IRs which contribute towards it, namely Sub‐IR 3.2 “Increased private sector investment”. The indicator for this Sub‐IR is outcome impact indicator, “Number of firms (excluding farms) or Civil Society Organizations (CSOs) engaged in agricultural and food security‐related manufacturing and services now operating more profitably (at or above cost) because of USG assistance”, which is reported upon by AMDe alone, although both LMD and PRIME might also be expected to be able to report progress against this indicator. The LOP target is 5, but no achievements are yet recorded in the FTFMS. Such progress is surprising given the reportedly expansionary nature of the Ethiopian agricultural sector. It is remarkable that none of the AMDe interventions, including the provision of training and investment to a considerable number of cooperative unions, cooperatives and other beneficiaries should not have resulted in the more profitable operation of even a small number of businesses. It is also remarkable that none of the activities of PRIME or LMD would be considered as leading to the increased profitability of firms that could be reported under this indicator. It is possible that it is too early in the program to witness the development of profitable agribusinesses. Nevertheless, the limited response against this indicator raises two concerns. The first is that only one of the FTF programs is reporting against what might be construed to be a central element of the “push/pull” mechanism upon which Ethiopia’s FTF strategy relies. The development of profitable agriculture‐based industries is also central to the GoE rural development strategy. It is therefore disconcerting to see this indicator considered relevant by only one project. If rural poverty is to be alleviated through more than agricultural production alone, the development of ancillary businesses will be essential for wealth creation and as a source of employment. The monitoring of that development should be a key aspect of the FTF strategy and should be undertaken by all projects that 35 MFIs disburse credit to groups of GRAD borrowers, all of whom must repay before any one member of the group can access new finance. 36 The FTF indicator definition of a PPP is much broader than the more commonly understood definition of an arrangement under which goods or services are provided by the private sector to meet the accepted responsibilities of the public sector. 37 LMD is working with AKLDP on a PPP for a quarantine facility for live animal exports. 25 support business development. The second concern is that the very limited development of profitable agribusinesses that has occurred must reflect to some extent upon the other interventions that have been made both by FTF projects. If, despite all of the investment and training that has been brought to bear, it is impossible to record a single business operating more profitably than it was prior to the FTF program, then the nature and targeting of FTF interventions must be called into question. IR 4: Increased employment opportunities in targeted value chains Progress towards this IR is assessed through a single, critical, impact indicator “Number of jobs attributed to Feed the Future implementation”, which is monitored by AMDe, LMD, PRIME and GRAD. The LOP target for these four projects is 101,481 jobs, and to date 34,671 (34%) have been created. While the basic data suggests reasonable progress towards the LOP target, there are three areas of concern over the reporting of this indicator:  GRAD has interpreted the definition of a job in such a way as to include almost all of its beneficiaries and is consequently responsible for 80% of the LOP target. By contrast, AMDe, LMD and PRIME have adopted a more stringent definition requiring conversion of all jobs to “full time employment” equivalency, but on this basis contribute only 20% of the target. If the indicator is restricted to these three projects, the current level of achievement is 19%. In practice, a further 8,966 jobs have been created under FTF by PLI and other projects. If these are included, performance increases to 44%.  The number of jobs reported by PLI is identical over two years.  In a meeting of M&E managers of FTF projects, some confusion was expressed over whether or not self‐employment counted towards the number of jobs and whether the indicator should be based upon jobs created or jobs filled. Until these issues are resolved, it is hard to assess progress against this indicator. This is of concern, since the creation of employment through the development of the agricultural sector under IR4 is as critical an aspect of the push/pull strategy as the creation of profitable agribusinesses under IR3. If the FTFMS is unable to monitor this key aspect of the program effectively, it will be difficult for those responsible for the co‐ordination and management of the program to operate effectively. This indicator should be revisited to ensure consistent reporting across all projects and revised historical data should be posted on the FTFMS. IR 5: Increased resilience of vulnerable communities and households This IR also contributes towards First Level Objective 2. Progress towards the result is assessed on the basis of four indicators, the first two of which are clearly impact indicators: 1. “Prevalence of households with moderate or severe hunger” was recorded by the program baseline survey at 4.9% for households in the FTF ZOI. It will be reassessed in 2015 and 2017. The FTFMS does not show any targets for this indicator. Given the recent good Meher harvests experienced by most households, it is quite probable that the prevalence of hunger will have declined, but whether it will be possible to detect a difference attributable to the FTF program is less certain. Given the low levels estimated both within (4.9%) and outside (4.5%) the FTF ZOI, it is unlikely that a significant difference will be observed in either the mid‐line, or the end‐line survey. 2. “Depth of poverty: mean percent shortfall relative to the $1.25 poverty line” was also measured by the baseline survey at 11.3%. Again, no target is shown in the FTFMS for this indicator. Since the poverty headcount ratio was 34.5%, a depth of poverty score of 11.3% implies that the actual difference between the mean income of households below the poverty line and the line itself was 33% or ETB 4.5 per capita per day. This is equivalent to ETB 6,178 per household per year, or at current prices, 14 qt of maize or 6.7 qt of wheat. In 2013, the mean daily income per adult equivalent in the FTF ZOI was ETB 21.59 and the mean area cultivated to cereals was assessed at 0.67 ha. Households at or below the poverty line can be expected to cultivate correspondingly less. On the simplistic but not unreasonable premise that income is proportional to area cultivated, the average cultivated area of a household at the poverty line would be no more than 0.39 ha. For such a household to increase its income by ETB 6,178 would require it to increase its productivity by 36 qt/ha of maize or 17 qt/ha of wheat. These 26 figures represent yield increases of approximately 200% and 100% over baseline yields for maize and wheat respectively. The probability of such increases being achieved within a five‐ year timeframe is remote. There is not only the physical improbability of yields being increased by the required amounts, but also and perhaps more importantly, the limited propensity of smallholders to adopt techniques requiring increased investment since this is invariably associated with increased risk. Thus for smallholders on 0.5 ha, costings reported by farmers in the key maize growing area of Wellega would result in seed and fertilizer at ETB 2800, and at ETB 600, i.e. a cash investment of ETB 3400. A basic analysis indicates that if they consumed only maize, they would need about 900 kg38, and to cover their costs they would need another 8.5 qt at ETB 400/qt39 or 10 qt at ETB 340/qt.40 This implies total per ha yields of 35qt/ha to 38 qt/ha. As average yields are well below these figures, and the poorer households can be expected to be in the lowest tercile of productivity, the chances of obtaining such yields are slim. By contrast, the household could go the low technology route of home‐saved seed and animal manure and get a yield of 17 qt per ha without the risks associated with increased investment. Such analysis illustrates the extreme difficulty of reducing the depth of poverty through increased agricultural production alone. Instead, additional off‐farm income generation will need to be facilitated if this indicator is to be significantly reduced. 3. “Number of USG social assistance beneficiaries participating in productive safety nets” is a function of USAID’s participation in the PSNP. Targets for this indicator are predetermined and readily met. USAID’s consistent parallel support for the PSNP over the last ten years has resulted in a degree of proficiency amongst the cooperating sponsors that allows assistance to be delivered in a timely and effective manner. The actual level of assistance might be expected to increase from 2015 onwards as the new PSNP program, designed to support a larger number of beneficiaries over a wider area than before is progressively implemented. 4. Within the narrow scope of five main FTF programs, the fourth indicator, “Number of vulnerable households benefiting directly from USG assistance” is measured only by GRAD, and ENGINE. Although PRIME works with TOPS, it has not reported on this indicator. The LOP target is 81,170 households. Actual performance to date is 75,920 (93.5%), largely due to the contribution of GRAD, which has already exceeded its target. It is quite probable that this target will be met. At a broader level, this indicator also includes all PSNP beneficiaries, as well as PLI, Peace Corps and WATER project beneficiaries so that the actual numbers recorded in the FTFMS is about six times higher than the figures for GRAD and ENGINE alone. As such, the value of this indicator as a management tool to measure the impact of the core FTF program is limited. The higher numbers obscure the level of progress that has been made by the core FTF projects. Fluctuations in the coverage of the PSNP or WATER can significantly alter the indicator result without affecting the fundamental implementation of FTF within the ZOI. First‐level FTF Objective 2: Improved nutritional status (women and children) Progress towards this objective is measured using three impact indicators namely: 1. Prevalence of stunted children under five years of age (Baseline 50.6%) 2. Prevalence of wasted children under five years of age (Baseline 12.1%) 3. Prevalence of underweight women (Baseline 26.8%) These indicators were assessed by the baseline survey of June/July 2013 and will be reassessed when the survey is repeated in 2015 and 2017. The baseline results for the FTF ZOI are somewhat higher than the estimate made for AGP woredas in 2011 (IFPRI 2011) of 46.2%, although similar to estimates of wasting (12.0%) from the same survey. While ENGINE targets a 20% reduction in stunting, a 15% reduction in wasting, and a 10% reduction in underweight women, FTF targets for these indicators across the ZOI are not specified. 38 15 kg per person per month, with five people in the household. 39 EGTE maize price in Nekempt, March 2015. 40 Farm‐gate maize price outside Nekempt, March 2015. 27 It is hard to estimate progress against these indicators. First, there is no current data available. Secondly, expert opinions are divided. The target for ENGINE (the main project responding to this indicator), of a 20% reduction in stunting (to 40%) is considered feasible by project management on the basis of the converging influences of improved national agricultural production, health, education and WASH service provision, which appear to be reducing national stunting levels by 1% per year, so that in the ZOI, stunting would fall to 45% over the life of the program irrespective of FTF program interventions. Consequently a further reduction to 40% as a result of ENGINE interventions is considered plausible by project management. This view is countered by the mid‐term evaluation of ENGINE, which considers that such a reduction would be unusual. Similar observations might apply to the second and third indicators. From the perspective of FTF, if the anticipated “difference in difference” methodology is used to assess program impacts, external influences can be expected to be discounted, thereby reducing the observed impacts of ENGINE in the ZOI and limiting the possibility of reaching the targets. ENGINE project management considered that while it had developed an effective modality to change nutritional household level nutritional practices, it was still necessary for the other projects (AMDe, LMD, GRAD and PRIME) to develop complementary SBCC techniques to ensure that the extra income generated from production and sales would be used to purchase diverse and quality to improve nutrition further. This observation, together with the as yet unresolved issues associated with fasting and the limited incorporation of WASH messaging and infrastructure development within the FTF program, would suggest that the impact of ENGINE (together with the other projects undertaking BCC and SBCC activities in the ZOI) will be insufficient to achieve the targeted differences in these indicators between woredas within and beyond the FTFZOI. The first IR contributing to First Level Objective 2 is IR 5 “Increased resilience of vulnerable communities and households”, which is considered in more detail under Objective 1 above. IR 6: Improved access to diverse and quality foods This is the second IR contributing to Objective 2 and is measured by the following two impact indicators (with baseline results for the FTF ZOI in brackets): 1. Prevalence of children 6‐23 months receiving a minimum acceptable diet (MAD): (Breast fed 1.56%, non‐breast fed 0.0%) 2. Women’s Dietary Diversity: Mean number of food groups consumed by women of reproductive age: (1.57) FTF ZOI baseline results for both of these indicators are low and do not correspond with individual project data. ENGINE reported a baseline of 13% for children receiving a MAD, and a women’s dietary diversity score of 2.8, while GRAD reported a prevalence of 6.6% for children receiving MAD and a women’s dietary diversity score of 2.9. The disparities between these results suggest that different definitions or methodologies may have been used to assess these indicators. The FTF target for women’s dietary diversity is an average score throughout the ZOI of 4.0. This represents a 30% increase over the ENGINE and GRAD baseline results, and a 150% increase over the FTF ZOI baseline estimate that will ultimately be used on a “difference in difference, matching pair” basis to assess progress. Similarly, the LOP target of 30% for children receiving a MAD throughout the FTF ZOI represents a massive increase of 1,900% over the initial baseline to be achieved within the five‐ year LOP. In both cases, it appears unlikely that the targeted results will be achieved. The interventions to effect both increases comprise mainly widespread SBCC activities championed by ENGINE, together with more direct but limited nutrition interventions on the part of the other FTF projects. Given the limited geographical scope of both GRAD and PRIME, and the focus of LMD upon only 9 woredas for “deep” nutrition interventions, their contribution to these indicators will inevitably be constrained. AMDe may have a broader impact but its limited nutrition interventions are unlikely to achieve changes on the scale envisaged. It is therefore primarily ENGINE that is expected to achieve impact operating indirectly through health workers. While SBCC is recognized to be a potentially highly effective methodology for nutritional development, its level of achievement is generally more modest than that 28 expected of the FTF Program, particularly as such interventions typically take a long time to reach scale.41,42 IR 7: Improved nutrition related behaviours This IR is measured by a single outcome indicator: 1. Prevalence of exclusive breastfeeding of children under six months of age. This indicator has been measured for the FTF ZOI at a baseline level of 67.6% and an LOP increase of 8% was originally targeted in the ENGINE PMP. While the FTF PMP reports no specific target for the overall program, the 2014 Portfolio Review targets an increase in the number of exclusively breastfed children of 20%, so that the targeted LOP prevalence across the FTF ZOI would be 81%. UNICEF records only six countries in the world achieving over 70% exclusive breastfeeding up to 6 months43, of which one has achieved a level above 80% (Rwanda at 85%). The target is therefore possible, but it is unlikely that the current FTF interventions which rely primarily upon BCC, indirectly mediated through health workers, will be adequate to allow this indicator to be met within the next three years across the ZOI. Indeed, given the current high levels already prevailing, the original ENGINE target of an 8% increase to 75% seems more realistic and in line with experience elsewhere, and even that might be hard to effect in the limited time available. IR 8: Improved use of maternal and child nutrition services This IR is measured by four indicators: 1. Number of people trained in child health and nutrition through USG‐supported programs. 2. Number of health facilities with established capacity to manage acute under‐nutrition. 3. Number of children under five who received Vitamin A from USG‐supported programs. 4. Number of children under five reached by USG‐supported nutrition programs. FTF targets for these indicators are compiled from individual project targets. For the first indicator, the five main projects are targeted to train 80,389 people in child health and nutrition. The contributions to the total expected of different projects are somewhat variable. GRAD is targeted to contribute 2%, while ENGINE (which might be expected to contribute the most) stands at 35%, PRIME at 19% and LMD at 44%. This last figure is remarkable given LMD’s focus on only 9 woredas. AMDe has no reported target. Targets for this indicator do not appear to reflect the capacity of individual projects to undertake the required training. AMDe has no target for this indicator, the target for GRAD appears to be relatively low, while that for LMD appears relatively high. As of November 2014, the number of people trained had reached 35% of the FTF target. GRAD has exceeded its target by 5.5%, ENGINE has achieved 40%, PRIME 24% and LMD has achieved 11% of LOP target. While overall progress has been sound, the LOP target for LMD is very high as compared with its coverage of woredas and the likelihood of this indicator being met is questionable. The second and third indicators are entirely the preserve of ENGINE, which has already reported a total of 46 health facilities as against a target of 25. The LOP target has thus been already met. The target for the number of children receiving vitamin A is 4,205,663, as against a reported figure of 2,883,673, or 69% of target. If current rates of progress can be maintained, this target should also be met. The final indicator in this series is also very largely driven by the activities of ENGINE, whose project target contributes 96.5% of the overall FTF program target. Currently the LOP FTF target of 41 Duflo, E. (2012). Women empowerment and economic development. Journal of Economic Literature 50(4): 1051–1079. 42 Lamstein, S.T., Stillman, P., Koniz‐Booher, A., Aakesson, B., Collaiezzi,T., Williams, K. and Anson, M. (2014). Evidence of Effective Approaches to Social and Behavior Change Communication for Preventing and Reducing Stunting and Anemia: Report from a Systematic Literature Review. Arlington,VA: USAID/ Strengthening Partnerships, Results, and Innovations in Nutrition Globally (SPRING) Project. 43 Data available at: http://data.unicef.org/nutrition/iycf 29 reaching 5.96 million children under five has been 57% achieved, and it is possible that the final figure may be reached. Overall, the MTE team found there to be a substantial gap within the FTF results framework as regards Objective 2, in that while the indicators that are most frequently reported are output indicators under IRs 7 and 8, the most critical indicators are the impact indicators for the objective itself and for IR6, which have not yet been assessed since baseline data was first recorded. The causal pathway between the output and impact indicators may be logical, but it is by no means unequivocal. For example, trained people may still be unable to access health or nutrition services, while the physical presence of an improved‐capacity health facility doesn’t necessarily indicate improved use. Ideally an indicator might describe accessibility to improved health facilities and the percentage of households at risk of acute under‐nutrition who can access a facility in the ZOI. Without appropriate outcome indicators that are more directly relevant to the objective, it is possible that the program might not reach its LOP targets for Objective 2 without any foreknowledge of the impending shortfall. The FTF PMP does not provide the data necessary to assist program or project management to adjust activities to achieve FTF Program Objective 2. Differences between the assessments of management and the ENGINE MTE highlight this uncertainty, and while the mid‐line evaluation might be expected to shed some light upon the rate of progress towards this objective, more frequent reporting using smaller samples would support more effective management for results. It is broadly accepted that stunting is a multifaceted problem, being impacted by at least eight different factors including: maternal nutrition and weight at birth, prevalence of exclusive breast feeding, adequate complementary nutrition, animal protein intake, vitamin A, Zinc and iron intake levels, sanitary practices, availability of potable water and aflatoxin intake. It might be helpful to program and project management if IRs, sub IRs and indicators could be developed within the FTFMS which allowed progress with regard to these different factors to be tracked and their impact on the program objective to be assessed. Summary Assessment of progress against indicators was confounded by the following factors:  The absence of targets for some indicators, especially population‐based indicators determined by baseline, mid‐line and end‐line surveys, prevents any meaningful assessment of progress, even when it might be possible to estimate the probability of different rates of growth, since there is no framework within which to determine the significance of any observed change.  The fact that some projects have not reported on indicators that were clearly within their manageable interest has meant that the extent of progress may have been underestimated in some cases.  The different interpretations of indicators by different projects has rendered some statistics meaningless at the program level (e.g. numbers of jobs)  The compilation of different sources and types of data to provide an indicator of a single number has also obscured the meaning of the result (e.g. areas under improved management)  The extrapolation of survey data based upon samples that are not representative of the baseline has led to unrealistic assessments of progress (e.g. yield and gross margin data).  Confusion of incremental and actual sales and exports within the FTFMS  Eleven out of 39 indicators were of limited value in that their results were based upon data of different sources/types could not be readily interpreted, or that the attribution of impact to FTF interventions was impossible  Eight out of 39 indicators had no targets. Under such conditions it is difficult, to determine the level of effort or resources that should be allocated to the relevant interventions, and the objective evaluation of progress is limited to positive/negative/no change. It is also surprising that relatively little attention has been paid to the two key outcomes of business development and employment which, although critical to the push/pull model of development out of poverty, are measured by only one indicator in each case. 30 In the light of these concerns, it is evident that the current framework of indicators provides a weak basis for the assessment of progress towards the program IRs, Objectives and overall Goal. Program management is not facilitated by such a framework. Nevertheless, an overall assessment of progress against indicators would suggest that output indicator targets are generally on target to be met. In some cases, initial under or over estimation of targets has meant that they have already been met or will not be met at all. In such cases, the FTFMS has been appropriately adjusted. There is no strong evidence to suggest that levels of effort and resources are not appropriately matched to the achievement of output targets. For outcome targets, the picture is more variable. Targets in key areas of finance, employment and business development are less likely to be achieved (when such targets have been specified). This may be due to factors beyond the manageable interest of the FTF program, but it may also reflect the diffuse nature of market‐based interventions and the limited coordination between such projects and more geographically focused interventions such as GRAD or PRIME. The achievement of high‐level impact targets is more problematic again, especially when considered from the perspective of a statistical analysis that uses a difference in difference methodology to ensure proper attribution of the results. Under such rigorous analysis, while it is possible that there may be significant improvement in the high level (objective, population‐based) indicators to the targeted levels and beyond, it is unlikely that any of these targets will be met as a result of program interventions alone. 4. CROSS‐CUTTING ISSUES 4.1 Gender Gender is being integrated into FTF in two ways: explicit strategies to promote women’s empowerment and efforts to mainstream gender into all FTF activities and track this progress by collecting gender‐ disaggregated data. It is well‐known that women play a key role in agricultural production and nutrition, but frequently lack access to productive resources and decision‐making authority.44 In addition to addressing equity, empowering women is viewed as central to achieving the FTF goals of agriculture‐ led poverty reduction and improved nutrition. By collecting gender‐disaggregated data (which is being done by multiple projects), projects should be able to identify the gendered impacts of their programming and adjust their strategies to ensure that women’s needs are addressed. The key indicator for women’s empowerment, the WEAI, is only measured using the population‐ based surveys, so it is difficult to track the contribution of individual projects to this goal directly. A report by IFPRI examined the baseline results for the WEAI, but unfortunately, Ethiopia’s baseline data was not available in time for it to be included in the global analysis. Globally, however, the largest constraint to women’s empowerment was found to be access to credit, particularly for East African FTF countries.45 Secondary education and exclusive breastfeeding were also found to be strongly correlated with women’s empowerment. Women’s role in household decision‐making, control over productive assets, and leadership in the community and comfort speaking in public are also important components of the index, and project components that support each of these aspects is considered in our analysis. GRAD’s work with VESA groups has probably made one of the strongest contributions to women’s empowerment. The VESA group model is designed to help poor households save money, gain access to loans provided by the group, and access loans from microfinance institutions. Because of the high female membership in VESA groups, this model is an ideal mechanism for improving women’s access to credit and decision‐making regarding spending. It also offers a forum for women to take on leadership positions and speak in the group meetings. GRAD has facilitated group discussions on women’s empowerment and the role of women in household decision‐making. Both men and women are present in these conversations, making them an ideal format to discuss the issues of inter‐household decision‐ making and resource allocation that can be difficult to address simply through interventions targeting women. The VESA model, which focuses on both economic and social capital, represents an important 44 Malapit, H.J. et al. (2014). Measuring Progress toward Empowerment: Women's Empowerment in Agriculture Index: Baseline Report. IFPRI. http://www.ifpri.org/publication/measuring-progress-toward-empowerment accessed May 2015. 45 Ibid. 31 departure from previous VSLA models in that it explicitly addresses intra‐household dynamics and decision‐making by including both husbands and wives, as opposed to the VSLA model which was more focused exclusively on women. The Women in Agribusiness Leadership Network established by AMDe and supported by LMD is another interesting initiative to address women’s empowerment. The network supports female entrepreneurs through training and mentoring, recognizing the unique hurtles women face establishing, operating, and expanding their businesses. By matching female entrepreneurs with successful, senior women in agribusinesses, the network seeks to provide women with the confidence, skills and resources they need to be entrepreneurs. Although it is too early to see the impact of the network, it is an innovative approach that should be monitored closely. Particularly in light of the importance of access to credit, PRIME is potentially contributing significantly to women’s empowerment, although some initiatives have not been undertaken as part of an explicit gender strategy. PRIME (and its predecessor RAIN) have been instrumental in the formation of the Somali Microfinance Institution (SMFI) and the newly established Afar Microfinance Institution, which are helping fill a large unmet need for Islamic finance in the Somali and Afar regions. Approximately 90% of SMFI’s clients are women (compared with a national average of 41%), and SMFI has expanded rapidly since it opened in 2011, now with over 19 million birr in outstanding loans, 41 million birr in savings and almost 5,000 clients (Association of Ethiopian Microfinance Institutions (AEMFI)). These statistics suggest that there is a large demand for access to credit in the region, particularly among women. Building on the historical role of Somali women as entrepreneurs and traders could further strengthen PRIME’s gender work. PRIME has begun to work with a Women Traders Association in Jijiga, connecting them to the National Women Traders Association and helping them host a trade fair. There are clear opportunities to build on this work to further strengthen this, and potentially other, groups, including through the small grants component of the project, but these opportunities have not been fully exploited to‐date. In terms of gender mainstreaming and the collection of gender‐disaggregated data, it is clear that projects are taking pains to collect this data, and were aware of the targets that had been set for female participation in various project activities (for some projects it was unclear how much of this was driven by FTF priorities, and how much was driven by the GoE’s mandate to have at least 30% participation of women in trainings and cooperatives). There were some concerning examples, however, that measures are being taken to ensure that targets are met that do not necessarily contribute to project objectives. For example, in order to encourage women to become members of cooperatives, AMDe gave them an umbrella or scarf for signing up, and the cooperative that signed up the most new members won an international exchange visit. While such incentives can be justified as providing visibility to women that join, one has to question whether women joined because of their inherent interest in becoming members, and if such superficial mechanisms for promoting membership will lead to lasting changes in women’s empowerment. Such strategies are highly effective in helping reach project targets, but may potentially have very little actual impact. The FTF gender component is primarily focused on women, but another vulnerable or disempowered group of particular relevance for FTF is youth. In terms of job creation, this is one of the key target groups. Due to the large number of landless youth, addressing the needs of this group will likely require initiatives outside of the agricultural sector, an area that has proven particularly challenging across FTF. Although there are some initiatives specifically targeting youth in GRAD and PRIME, overall, a stronger emphasis on the needs of this vulnerable group is needed. REST, through the GRAD project, has organized landless youth to form “cut and carry” forage businesses and beeswax businesses, but it is not clear if this is a widespread component of the program. Particular attention should be paid to lessons learned from the pilot projects with TOPS in PRIME, which could help inform future programming. 4.2 Climate Change Two of the five major FTF projects have a climate change component: GRAD and PRIME. In both cases, the primary achievement to‐date has been the development of climate vulnerability assessments. For GRAD, these assessments have helped identify risks in the value chains they were promoting, and led to a greater emphasis in the project on ways to reduce or minimize these risks. For PRIME, the 32 assessments were undertaken as part of the work with the rangeland management councils to help identify climate vulnerability and adaptation priorities for each rangeland unit using CARE’s climate vulnerability and capacity assessment tool (CVCA). In GRAD, climate change is one of the four issue areas addressed in the VESA groups: gender, nutrition, climate change and aspiration to graduation. According to REST, climate change adaptation is the area where they have been least successful, perhaps because many of the climate adaptation options that have been identified are in value chains that they have struggled to promote (over 85% of beneficiaries prefer to invest in shoat rearing or fattening, and it has been challenging to get clients to invest in other income‐generating activities, particularly in Tigray). For PRIME, climate adaptation is the second IR of the project. According to the draft PRIME MTE report46, the quality of the CVCA assessment was high, but now the priority should be on implementation of the recommendations that emerged from the assessments. Although there are few explicit climate adaptation activities to‐date, resilience is a key component of both projects, featuring in the project titles. Many of the activities undertaken in relation to climate change are perhaps better understood as addressing resilience rather than climate change per se. Viewed from this perspective, both projects are contributing significantly to resilience through a wide range of activities, including income diversification, savings, informal insurance (focus group participants identified informal insurance as one of the primary benefits of the VESA groups), commercial destocking systems, and natural resource management activities. In spite of progress addressing some aspects of resilience, one key vulnerability issue stands out for PRIME and GRAD, as well as LMD and ENGINE: the issue of feed and fodder. The livelihood, marketing and nutrition activities in all four projects depend on the productivity of livestock and livestock products, which is not feasible without sufficient feed and fodder. Project staff as well as government officials have all identified the lack of feed and fodder as a barrier to the development and sustainability of the livestock sector. Each project has been marginally engaged in the feed and fodder issue, but none have addressed it as the central issue it is becoming for the sector. In light of the new USG Executive Order on resilience47, all future projects will need to think more critically about the role of resilience and climate change in their activities. An examination of AMDe and LMD suggests that there are opportunities for integrating a climate or resilience perspective into such projects, and opportunities to improve programming that have been missed. The geographic division of the country into productive and vulnerable areas masks the reality that all agricultural activities, particularly rain‐fed agriculture, are vulnerable to climate‐related shocks, including both droughts and floods, and consideration of risk‐reducing measures is an appropriate component of any agricultural project. In addition to considering climate‐related risk, a resilience perspective would encourage projects to think more critically about the many risks smallholder producers face. As the COP of AMDe expressed, a project focusing on marketing of commodities, needs to have enough diversity in its portfolio, because inevitably some value chains will do better than expected and others will not. While the project has covered its own risk through diversification, it is not clear that it is encouraging the same risk‐ spreading among smallholder producers. In fact, in some cases it is potentially encouraging farmers to take on higher risk (albeit higher return) strategies. Hybrid maize is one such example. Although the yields associated with hybrid maize exceed local varieties, it also requires investment in more fertilizer and other inputs to be successful. In a good year, the increased cost of seed and other inputs is a good investment, but in a drier year, low‐yielding local varieties may do better, or at a minimum, producers may lose less. Commercial farmers have sufficient access to finance to manage these risks, but projects need to think critically before promoting such risky strategies for smallholder producers without sufficient access to credit, savings or safety nets in the case of crop failures. While hybrid maize may potentially still benefit smallholder producers, to ensure that its adoption does not make vulnerable 46 USAID. Mid Term Performance Evaluation of the PRIME Project 47 In September 2014, the Obama Administration issued an Executive Order on climate‐resilient international development that, “… requires the integration of climate‐resilience considerations into all United States international development work.” https://www.whitehouse.gov/the‐press‐office/2014/09/23/executive‐order‐ climate‐resilient‐international‐development/ accessed May 2015. 33 households more vulnerable, complementary investments in increased education about the risks, risk‐ mitigating measures such as crop insurance, and encouragement of reinvestment of profits into livelihoods with no covariant risk are required. Each project does not need to develop its own climate change strategy independently. A wide range of resources within the FTF global portfolio exist that can be drawn upon to help projects. For example, the FTF Innovation Lab for Collaborative Research on Adapting Livestock Systems to Climate Change48 could provide support and technical expertise to PRIME and LMD (and potentially GRAD) in their work in the livestock sector. Another example is the Climate‐Resilient Chickpea Innovation Lab, which seeks to harness traits from native chickpea strains to improve the resilience of domesticated chickpea. The Ethiopian Institute for Agricultural Research is one of the main partners in the Innovation Lab, and thus, it should be easy to establish collaboration. AMDe is working to promote Kabuli varieties of chickpea for export, but chickpea has traditionally been grown as an opportunistic crop, deriving its moisture from stored soil moisture rather than rainfall.49 Considering the role that chickpea has historically played in the resilience of smallholder farmers, one would hope that this would factor into the decision‐making to promote a non‐native variety for export in AMDe, but this does not appear to have been the case. Aside from resilience concerns, the Chickpea Innovation Lab could be a helpful partner to address what AMDe has identified as a key barrier for the marketing and export of chickpeas: the insufficient size of the Kabuli varieties. 4.3 Knowledge Management While each of the five major projects has a learning and knowledge management component, AKLDP also plays a major role in the provision of collaborative learning across the whole Feed the Future program. So far AKLDP knowledge management activities have focused primarily on monitoring and evaluation as well as internal learning. This may have been appropriate for the first half of projects, as lessons learned may just now be emerging, but it will be critical to ensure that a process for capturing the FTF experience and lessons learned is developed in the remaining stages of projects. A wide range of activities are included in the knowledge management component of projects, including monitoring and evaluation for reporting and project management, dissemination of experiences for internal and external use, research, and policy work.  Monitoring and evaluation: In response to the requirement to report on FTF indicators, and a perception on the part of projects that these are the indicators of greatest interest to the Mission, the M&E efforts of projects are strongly focused on the FTF indicators. Traditional activity, output and outcome indicators specific to projects have been given less priority, and in some cases, abandoned entirely. One of the key goals of M&E is to inform programmatic activities and make adjustments to project direction, focus or strategy. While in some cases, M&E efforts are contributing to this goal, the emphasis on FTF indicators has affected the ability of projects to focus on their own activities. As discussed previously, FTF indicators are not always well‐aligned with direct project activities and are challenging for projects to collect, requiring a large effort on the part of project M&E teams. Some projects felt that their ability to engage in “knowledge management” was hindered by their focus on the FTF indicators.  Dissemination of learning: Projects are beginning to document project learning, but across the board, this is not sufficiently systematic. There is a lack of commonality in terms and procedures of knowledge gathering and reporting within projects that hinders the dissemination of experience amongst them. In some cases, there appears to be a lack of distinction between “success stories,” reporting for annual reports, and documentation of lessons learned. While including lessons learned and challenges in annual reports are important components of reporting, documentation of lessons learned should be a separate and distinct process, focused more on the fundamental 48 http://lcccrsp.org/ accessed May 2015. 49 Shiferaw, B. et al. (2007). Analysis of production costs, market opportunities and competitiveness of Desi and Kabuli chickpeas in Ethiopia. ICRISAT/EIAR. http://oar.icrisat.org/5273/1/BBBS‐WorkingPaper3f.pdf accessed May 2015. 34 knowledge gained and experience of relevance to other projects or organizations, including the government, with a broader audience than program/project management alone.  Research: A distinction needs to be made between formative research, designed to inform project design and activities, and fundamental research, designed to inform the field more broadly, and potentially future projects. Failure to distinguish between these types of research may lead to unrealistic expectations for research activities, as was seen in some projects. Some projects, most notably ENGINE, have systematically incorporated research into the project, designing research to inform policy work and project design. Other projects appear to have undertaken research in an ad‐ hoc fashion, as the need arose, with less strategic focus, and could benefit from clearer direction.  Policy: FTF projects have an important role to play in the policy process, particularly by advocating for policy reform and providing examples of successful approaches to address key policy issues. Both roles require a strong evidence base upon which to build policy, but as yet the policy components of most projects are not drawing significantly upon the evidence bases that they themselves are developing (with the exception of ENGINE which has drawn extensively on its own formative research to develop recommendations for implementation by the NNC and others). This may be a reflection of the stage of implementation, i.e. that evidence bases are still under development. In which case, more effective policy reform might be expected in the latter half of the program. As one of the largest FTF portfolios globally, FTF Ethiopia is in a unique position to provide lessons learned not only for future USAID Ethiopia projects, but to inform FTF more broadly. Currently neither the knowledge management components of projects, nor efforts at the Mission, are sufficient to capture these experiences and share them. AKLDP has a key role to play in capturing the knowledge and lessons learned by projects and translating them to a broader audience. AKLDP can potentially play several roles in the remainder of the project period: 1. Compiling useful project experiences, particularly on issues of relevance to more than one project. One example would be to conduct an assessment of different saving group modalities, as this has emerged as a key approach for addressing not only savings and credit interventions but also the BCC messaging at the center of the nutrition components of FTF. 2. Conducting research on issues or challenges identified by projects. For example, characterizing the feed and fodder chain would provide valuable information to multiple projects. Understanding the credit requirements and constrains for GRAD beneficiaries would inform the potential of such approaches to be scaled‐up, and assessing the pro‐poor impact of the various value chains would help inform both current and future FTF design. Characterising the assemblers, traders and brokers involved in all aspects of inter‐ and intra‐Regional trade (within Ethiopia) would help to understand the needs of, and most effective project‐based supports for the trading sector. Recent initiatives to establish a Livestock Working Group as well as Nutrition Working Group were highlighted by projects as a useful first step in identifying such areas of mutual interest. 3. Documenting and understanding those initiatives that have not resulted in the anticipated benefits. Recognising that failures can provide as much useful experience as successes, the documentation and analysis of such initiatives can provide useful information for future program design. Establishing a model similar to faculty‐student research mentorship could be one productive way to encourage more learning from the FTF projects. In such a model, staff in the AKLDP project could work with projects to identify research questions of broader interest to FTF, as well as project experiences that should be explored in further detail. Based on the types of questions and the capacity of the individual projects, AKLDP could then either take the lead on researching or documenting these topics, or they could play an advisory role for the projects to conduct their own research or document their own findings. Such a role would not duplicate the knowledge management components of individual projects, as AKDLP would help identify topics that are of broader interest to FTF, rather than those specifically within the scope of individual projects, and would work to bring together multiple partners if relevant. To support this work, AKLDP could potentially leverage more resources of relevance to FTF 35 at Tufts University than it has to‐date. For example, Tufts University also hosts the Feed the Future Innovation Lab on Nutrition.50 One challenge to this model of knowledge management and learning is the role of AKLDP as the external evaluator for FTF projects. It is potentially detrimental to the close collaborative relationship that needs to be established if projects are going to be open about the challenges they are facing and the lessons they have learned (including failures) to have AKLDP serve as an external evaluator. Although this has not been overly problematic for the mid‐term evaluations, perhaps because the learning agenda is only just getting underway, we recommend that a different arrangement be established for final evaluations. 5. INVESTMENT QUALITY The quality of FTF investments can be considered in terms of both direct impacts and contribution toward FTF goals. The latter consideration is strongly dependent upon the validity of the causal pathway, which is considered in section 2. In this section, the MTE considers the direct impact of investments, particularly the financial investments made through the various grant processes. It was observed that there is little consistency amongst FTF projects in terms of financial investment procedures. GRAD and ENGINE beneficiaries, in keeping with principles established under HABP do not receive any financial support beyond assistance to access micro finance. AMDe, LMD and PRIME provide no grants to individuals, but community organizations and businesses may receive finance on a matching grant basis. That basis may vary according to circumstance (as for LMD or PRIME), or it may be fixed (AMDe). From a financial perspective therefore, considerations of investment quality are limited to AMDE, LMD and PRIME. Within these three projects the MTE team found that some investments that were not fully aligned with the development principles underlying FTF. This is not unexpected when project management is required to achieve a targeted rate of disbursement. Under such circumstances it is easier for a project to invest larger sums in a small number of businesses that are better able to account for and utilize finance, rather than to seek out and train potential beneficiaries from the ranks of emergent MSMEs. The latter group can be expected to have a higher failure rate and to require a higher level of technical effort to support, but failure to encourage such emergent businesses can lead to the development of oligopolies or conversely of a restricted number of buyers, a result that directly contradicts the principles of market development for the benefit of either consumers or smallholder producers. As a consequence, while the MTE team observed that most investments were well targeted, some investments were noted that were: 1. Reportedly made for political expediency, and when the beneficiary stated to the MTE team that the finance could have been obtained from other sources (PRIME). 2. Supporting the expansion of infrastructure in the reported absence of adequate working capital to utilize the new equipment (LMD, PRIME).51 3. Made on the basis that “without the provision of these investments, our training would not have been effective” (AMDe). 4. Undertaken on an individual basis to promote cooperative membership (AMDe). The above investments are somewhat akin to the use of aid to “buy economic growth”, a concept that has been criticized on both theoretical and empirical grounds.52 There are also proponents of such a policy ‐ indeed, AMDe management asserted that investment in warehouses would be the most outstanding aspect of the project for which it would be remembered in years to come. It is certainly true that training alone in the absence of resources that trainees might utilize leads nowhere, and that with the possible exception of training in financial literacy and business plan development, training of itself does little to enhance access to finance, so that an argument for some financial support can be well made. 50 http://nutritioncrsp.org/ accessed May 2015. 51 It was noted however, that capital assets provided through the FTF program (especially by AMDe) might then be used as collateral to obtain finance that would be otherwise impossible to access. 52 Easterly, W. (2003). Can foreign aid buy growth? The Journal of Economic Perspectives 17(3): 23‐48. 36 AMDe reported that the call for applications for grant finance received over 400 applicants with a total value of over $40 million for an available fund of $14 million. These statistics may demonstrate a high demand for finance, but the situation is complex. A preceding USAID program (Agribusiness and Trade Expansion Program) implementing a similar but smaller investment facility was similarly inundated proposals for funding, but many were assessed as non‐viable. At the same time, it was noted by that where financial assistance or other support had enabled beneficiaries to obtain loans53, the disbursement rates were significantly lower than rates of loans approved. This was reportedly due to the inherent risk management of SMEs and cooperatives and bank loan officers who only disbursed loans as they were needed and could be absorbed. Such a pragmatic approach reflects the limited real capacity of the agricultural sector to absorb finance even though it is severely underfinanced. The capacity of institutions to absorb, utilize and account for funds effectively is still relatively underdeveloped. This results in those businesses or institutions that can respond appropriately being often targeted by a number of different projects in their search for effective and potentially successful beneficiaries. Under such circumstances where, real absorptive capacity is limited, it would appear that pressure to achieve disbursement targets has resulted in a tendency towards the “purchasing of growth”, and it is recommended that project output targets for grant disbursement should be avoided when the desired outcome (of business development) can be measured in other more direct ways. It is also quite evident that the need still remains for the development of private sector business capacity to the point where the absorption of funds is not focused upon a limited number of beneficiary institutions or businesses. 5.1 Successful Investments Notwithstanding the above criticisms, the MTE also noted a number of substantial achievements that demonstrated the quality of specific investments. These were generally associated with investments in training and mentoring rather than financial support. In particular the following stood out: 1. The success of GRAD in demonstrating the effectiveness of the comprehensive support package, including especially savings and loans groups (aka: VSLAs/VESAs/Development Groups), but also the concept of training in business development together with ongoing support and mentoring: The methodology adopted by GRAD has been taken up to a considerable extent by the GoE in the redesign of the PSNP/HABP program that will run from 2015 onwards indicating that GRAD has been effective in influencing policy, This has not been well captured by the FTF indicators, and yet it is exactly the result that a focused program such as GRAD is designed to achieve. Being limited to only 65,000 households, the GRAD project could never significantly affect rural poverty on a large scale directly, but it could achieve a much wider impact through its demonstration effect. This is indeed what has occurred, and as a result the limited investment in the GRAD project (the smallest of the main five FTF programs) can be considered to have been well justified. Budget restrictions may prevent the expansion of GRAD to cover a wider area, but there is considerable scope to build upon key principles of GRAD, namely incorporating PSNP support and development assistance, working closely with GoE staff and MFIs, and focusing on development in a limited area that is largely within manageable interest. Such developments are considered further in section 5.2.2. 2. In a similar manner, ENGINE has instrumental in incorporating nutrition into both PSNP 4 and AGP 2. It has also worked successfully with UNICEF and others to improve the policy environment for multi‐sectoral nutrition, while both AMDe and LMD have influenced the design of the upcoming second AGP program. 3. The ENGINE project has undertaken substantial formative research that has led to the revision of behavioural change communication modules. This again is not well reflected in FTF indicators, but has led to a significantly improved understanding of the social and religious constraints to an adequate diet that is not only being incorporated into BCC programs but has also led to positive discussions with religious leaders. Such formative research could play a similar role in the development of appropriate BCC to address such issues as low rates of 53 AMDe for example reported that it had facilitated access to over $90 million in loans, although a significant proportion of this was not disbursed. 37 livestock off‐take and low adoption of improved agricultural technologies. 4. The coordination between GRAD and LMD in the production and marketing of sheep and goats in Oromiya and Tigray is an example of a successful push/pull dynamic that can inform other situations. Although the end‐buyers of the animals were in both cases outside the GRAD woredas, geographical disparities were overcome through the development of a close working relationship between GRAD and LMD project staff who were together able to support the entire length of the value chain (from feed production through livestock fattening to end marketing). Situations where a similar dynamic might prevail could be limited in number, but the success of this interaction between the two projects has provided a demonstration of what can be achieved and some lessons as to the extent of the cooperation needed to achieve it. Other efforts at coordination and joint implementation are in the works, and project teams appeared to have a genuine interest and commitment to such efforts, but geographical and institutional challenges have made it more complicated than originally envisioned in the FTF design. 5. The work undertaken by PRIME in developing the Somali MFI has been a groundbreaking success that was initially viewed with a degree of skepticism by MFI experts in Ethiopia, but has now won their admiration. The speed with which this MFI has developed over less than two years to the point where it is already the 19th largest MFI in Ethiopia reflects the effectiveness of the project’s support. Perhaps the strongest indication of the value of this intervention is the fact that the initiative is already being independently replicated. This MFI, whose clients are predominantly women, will substantially improve access to finance for MSMEs in Somali Region. It will be interesting to see if this success will also be replicated in PRIME’s recent work with the Afar Microfinance Institute. 6. One aspect of PRIME that is poorly reflected in the FTF PMP, but is critical to success in the long‐ term is the learning nature of many of its interventions. While support to pastoral producers has been undertaken by previous projects (e.g. PLI I and II and RAIN), the provision of support to facilitate the transitioning out of pastoralism into new commercial activities or employment is a new development that has to date received little attention either in Ethiopia or elsewhere in the Horn of Africa. The different project interventions designed to promote employment creation, to enhance skills development and to bring together employers and potential employees, as well as those designed to promote and support small business development, form a range of activities designed to facilitate transition that will ultimately include both successes and failures. Experience would suggest that the failures will initially be greater in number than the successes, but it is important to recognize that PRIME represents the first in a series of projects designed to assist TOPS. That series will increasingly refine the suite of appropriate interventions necessary to achieve success, in a manner similar to that, which has been achieved by the project series: SPSNP ‐ PSNP Plus ‐ and now GRAD. As such, PRIME is working with innovation and already with some successes to lay the foundation for future projects that will be critical for the support of TOPS. 7. The Women in Agribusiness Leadership Network supported by AMDe and LMD), while unlikely to be of direct impact to the most vulnerable households, has nevertheless created an impetus for gender equity that can spread from the top down. The network has proven extremely successful in mobilizing businesswomen throughout the FTF ZOI. This would appear to be a sustainable initiative that can be used in the future for gender‐based advocacy, for training and perhaps more importantly for the development of aspiration amongst women. If in the remaining period of the AMDe project, the Women in Agribusiness Leadership Network can be provided with the necessary support to ensure its long term stability, it is possible that this more than any other development will be the most remarkable achievement of the AMDe program. 8. The support by AMDe for the construction of Ethiopia’s first fertilizer blending plant for Bicho Woliso Cooperative Union is an exciting development, and the availability of improved fertilizers along with complementary activities of soil testing, research, field demonstrations and stakeholder meetings on the topic of fertilizer distribution will undoubtedly benefit 38 commercial producers. Past experience54 suggests that campaigns to increase and improve the use of fertilizer are prone to unsustainability and that it is the complementary activities that AMDe has initiated that will in the long term be more critical to the success of this initiative. It will be necessary for AMDe to ensure the sustainability of its investment through both training and if possible, the institutionalization of its soil testing, demonstrations and stakeholder meetings to ensure that these complementary activities can be continued and adoption is sustained after the project has ended. 9. One last notable achievement which, although small in its immediate impact is potentially of far‐ reaching significance, has been the promotion by ENGINE of water carrying by the men of the village of Dembeli Keta. This development allows the women of the village significantly more time both to relax and to look after the rest of their families and so impacts both gender equity and nutrition. If only 1% of the FTF budget were to be focused on ongoing support to this single activity and the promotion of its replication elsewhere, it might have no impact at all, but could equally have an impact out of all proportion to the costs involved that would be the defining characteristic of the first Ethiopian FTF program. In addition to the above, it must be noted that FTF through AMDe in particular, has supported a range of technical innovations including new seeds, blended fertilizer and diverse support to the activities of the ATA. All of the above represent progress towards an enhanced agricultural sector. Nevertheless, the potential of such investments can only be realized if ongoing support is provided for their effective management over at least the next five years and possibly beyond. Past experience in southern Africa has shown that technical innovations even when supported by extensive trials and demonstrations are rarely enough to promote sustainable development. Continued training and reiteration of the promotion of improved technologies will be required if these extensive efforts are to have the impacts upon agricultural production required to eventually generate opportunities for economic improvement amongst the most vulnerable households. 5.1.1 Resource Constraints In almost all cases, the resources constraining project achievements have been beyond project’s manageable interests, varying according to methodology. While this is an inevitable reality of working in collaboration with government‐led initiatives and in a country with a strong development partner landscape, it presents challenges for the implementation of FTF. Both AMDe and LMD rely upon increasing production as well as sustainable markets for their focus value chains. These aspects have not always been present. Sesame yields and prices have fallen substantially in the last few months. Wheat markets have been distorted by GoE intervention and chickpea production has failed to achieve export quality. Similarly, abattoirs working with LMD have reported that their targeted export markets are unable to match domestic market prices. For these and a number of other reasons, both AMDe and LMD have broadened their scope to include not only marketing, but also various aspects of production in response to the fundamental constraint of inadequate or unprofitable production volumes, but it remains evident that a lack of progress in the production component of AGP has constrained the achievements of both AMDe and LMD. In the case of the LMD, progress has been limited by the lack of a clear counterpart within the GoE. Staff within the newly created State Ministry for Livestock were confident that as the new Directorates become increasingly operational, this constraint might be expected to diminish. GRAD has faced two resource constraints. The intensity of the interventions has required a substantial number of trained facilitators in order to apply the GRAD methodology effectively so that it has not been possible to extend the project beyond16 woredas. The same concern may become evident for PRIME in its support to TOPS, but it is too early to identify such a trend. This is primarily a budgetary issue and does not affect the replicability/efficacy of overall project designs. More fundamentally however, the availability of MFI credit for small business development has been 54 Not only in Ethiopia where the Sasakawa Global 2000 initiative was initially highly successful, but subsequently experienced high dis‐adoption rates. Similar experiences have been recorded over the last 50 years in Zimbabwe, Zambia and Malawi all of which have seen fertilizer usage rise and fall in line with the introduction and subsequent cessation of various promotional interventions. 39 significantly less than anticipated. This has been partly due to previous non‐performing loans, but it was also reported that MFIs remain reluctant to avail credit to PSNP participants, preferring instead to finance larger MSMEs and/or sectors outside of agriculture. This issue will continue to constrain the graduation of PSNP beneficiaries unless alternative sources of finance can be found. While ENGINE has a clear institutional framework to work with in MOH at all levels, the project has faced the constraint of an unclear counterpart institutional framework within the Ministry of Agriculture at the Federal level that has limited the effectiveness of its outputs at the household level. The project has been proactive in seeking to resolve such confusion through its activities as the secretariat of the National Nutrition Planning Committee, but the institutional challenges of multi‐ sectoral programming will take a long time to resolve. 5.1.2 Relative Cost‐Effectiveness of Projects The MTE is required to comment on the relative cost‐effectiveness of projects. Such an assessment is confounded by the fact that different projects within the FTF have very different objectives. Moreover, some projects are more self‐contained than others, although all depend to some extent upon developments that are beyond their manageable interest. Nevertheless, from the narrow perspective of contributing towards the achievement of FTF objectives, some assessment of cost‐effectiveness can be made. The support by FTF of interventions that parallel the PSNP has been evaluated on a number of occasions with a general consensus that the interventions contribute substantially to the enhanced food security of targeted households. On average PSNP‐type interventions have reduced the number of months of food insecurity for targeted households by 1.5, which has made a substantial difference to the protection of household assets. The cost per household exceeds US$350 per household, an amount that must be continually reinvested each year, and it is evident that investment in complementary programs such as GRAD that would allow households to graduate out of the PSNP by becoming sustainably food secure is a much needed development. Nevertheless, FTF support for PSNP‐type interventions has not only had a clearly beneficial direct impact in the past, but would also be an essential component of projects such as GRAD in the future, since the provision of a “consumptive stipend” remains critical to the early development of the beneficiaries of such projects. GRAD has been successful in its area of operation. It is meeting its targets and has a clear impact upon vulnerable households, even though its influence over the entire FTF ZOI can be expected to be small. Given the number of households (65,000) that have been targeted for change, and at a cost of approximately US$23.4 million, the level of investment per household appears large at US$ 360 per household, especially when compared with more systemic projects such as AMDe or LMD. Nevertheless, from a purely financial perspective, such a level of investment is entirely realistic, if not optimistic. Few businesses achieve greater than a 25% annual return on their investment, yet the expectation of GRAD, which aims to increase household income by US$365 per year, is that annual returns on GRAD’s investment per household should slightly exceed 100% ‐ this is potentially a very impressive result, but requires further monitoring. There remain a number of concerns over GRAD which may temper this assessment in the future. The fact that so many beneficiaries have opted for a production‐based approach to increased income (focusing so heavily on sheep and goat fattening) is a concern given the limited availability of feed, and the restricted availability of finance from a system that is strongly risk averse also raises doubts about the scaleability and sustainability of the GRAD approach. The limited extent to which GRAD has been able to develop other off‐farm income generating activities, also suggests that the broad‐based rural economic development that it was intended to foster amongst vulnerable households in its target woredas is not so easily achieved. From these perspectives, GRAD appears to be more of a step toward economic empowerment than the actual achievement of that goal, and there is clearly a need for further interventions to build upon the foundation that a program such as GRAD has created. Nevertheless GRAD has achieved a key objective of providing a successful model that is now in process of replication through PSNP4, leveraging funds from other donors. By doing so, it has justified the limited and intensive approach to development. PRIME has the potential to achieve the same results as GRAD; its overall budget of US$52 million results in 25% lower level of investment when applied to the 194,000 households targeted to receive USG assistance (i.e. US$268 per household). Many of the 40 PRIME interventions are also more systemic in nature and it inevitable that some of these will be more cost effective than others. Given that PRIME is breaking new ground in its TOPS‐focused interventions, (especially those addressing employment creation) it is too early to judge what may be most cost effective at this stage. LMD has worked effectively in a limited number of areas, but the fact that the project is entirely new (as opposed to GRAD or PRIME, which were able to build upon PSNP Plus and PLI/RAIN respectively) and an observed high level of rigor in its reporting appears to have contributed to lower than average achievement of targets. The differences in reporting across projects means that while LMD has not contributed as much numerically to targets as some other projects, this is not necessarily indicative of its actual on‐the‐ground impact. At the same time, staff noted that it had not been easy to identify beneficiaries and interventions that might contribute directly towards the immediate FTF objective of reaching those below the poverty line other than through partnership with GRAD. ENGINE has similarly worked well to strengthen national nutrition policy but has itself recognized that in terms of developing the successful BCC programs to promote the second FTF objective, there are still obstacles that remain to be overcome to influence income utilization in support of improved behaviors. Finally, AMDe has achieved much in terms of economic development in selected areas and value chains. The project has reached 650,000 SHF households using strict definitions, and by a somewhat broader definition over 1.7 million SHF households have received significant support through capacity building, training, investments, loan and market facilitation to over 50 FCUs, but its primary emphasis upon those households that can participate in agricultural marketing and value chain development ignores the majority of vulnerable households within the FTF ZOI. This does not mean that LMD, ENGINE and AMDe are not cost effective, but within the limited scope of the FTF objectives and the lifetime of the current FTF program, the immediate contributions that these programs have made when set against the budgets that have been expended are relatively lower than those of GRAD. For AMDe and LMD, this situation may be quite different when viewed from a long‐ term perspective. It is widely accepted that supporting equitable economic development will ultimately reduce rural poverty and that in the absence of significant growth in services, industry or manufacturing, the agricultural sector is the only candidate for such support. Systemic marketing initiatives that are begun now may well bear fruit in 5 to 10 years, but cannot be expected to impact poverty in a shorter timescale. For ENGINE, the limiting factor is that of manageable interest. It is widely recognized that the use of BCC can be an extremely cost‐effective tool55, but for as long as the project is prevented from working directly with health workers or households, it will be difficult for it to develop effective BCC models that could be replicated elsewhere. All of the above imply that from an immediate FTF perspective GRAD has been the most cost‐ effective program to date. This finding may appear counter‐intuitive given the high cost of household level interventions as compared with a more systemic marketing approach. Basic economic modeling suggests that the level of investment required to achieve the targeted increment in income is inevitably high, whether this be provided through commercial investment or (as a stop gap, pending the availability of such investment) through programs such as GRAD. BRAC’s experience in Bangladesh shows that when combined with a market development approach, household level interventions can be perfectly sustainable. Under the current circumstances both are required, especially when targeting the most vulnerable households and looking for results within five years. Nevertheless, the comparison is of relatively limited significance. What is significant however is the recognition of the limitations faced by each project that have reduced their immediate cost effectiveness so that future program designs can allow for these constraints. 55 Bhutta, A.Z., Das, J. K., Rizvi, A., Gaffey, M. F., Walker, N., Horton, S., Webb, P., Lartey, A., and Black, R.E. (2012). Evidence‐based interventions for improvement of maternal and child nutrition: what can be done and at what cost? The Lancet 382: 452. 41 5.1.3 Summary The analysis above would suggest that the most effective investments have been those that have focused upon human development, within a limited area and where factors beyond the project’s manageable interest have been minimal. Under such circumstances, project activities have achieved their goals. Infrastructural investments, while potentially positive in their long‐term impact have yet to demonstrate immediate benefits, while evidence to date shows that a strong reliance upon counterpart performance can lead to a reduction in investment impact. As emphasized throughout the report, an analysis of cost effectiveness is highly dependent on the goals considered. For this MTE, achievement of the high‐level FTF goals by the end of the 5‐year program is the ultimate measure of success, but this perspective is obviously limited and an analysis of a broader set of development objectives might come to slightly different conclusions. 5.2 Opportunities for Reprogramming 5.2.1 Immediate Reprogramming Some areas exist where immediate reprogramming and reallocation of resources within the FTF portfolio could enhance the probability of reaching the FTF goals and the sustainability of progress made to date. The following recommendations are designed to minimize disruption to initiatives already in place and are feasible in the remaining timeframe of existing projects without additional allocation of budgeting. Based on the concerns raised earlier in this section on the investment grant component of several projects, but particularly AMDe, we recommend ending the allocation of new large grants. Small grants targeted in specific projects to address key barriers are still appropriate (such as the PRIME grants to private veterinary pharmacies). Sufficient investments have now been made, as noted in the mid‐term evaluation, and future efforts should focus on strengthening the capacity of grantee institutions to maximize the effectiveness of the grants and ensure their sustainability and profitability. As identified in the AMDe mid‐term evaluation, the balance between financial investments in grants and capacity‐building has been heavily skewed towards grants, and significant increases in capacity‐ building is required for those grantees in the remaining lifetime of the project. A number of grants have already been awarded, and are currently in the process of being dispersed, which likely are politically infeasible to halt, but very clear processes (potentially drawing on the experiences of PRIME and LMD) should be employed to ensure grantees meet certain milestones and targets before receiving more money. PRIME’s experience suggests that grants can be allocated based on performance, and additional components of the grant can even be provided to other members of the supply chain, further distributing the benefits of the investment grants and ensuring a more competitive and sustainable supply chain. Efforts in the wheat value chain are also unlikely to contribute significantly to the FTF goals, and we recommend suspending activities in this value chain. AMDe has recognized that there are fewer opportunities in the wheat value chain, and has allocated fewer resources to it, which is appropriate, but remaining resources would be more effectively allocated elsewhere. Wheat production remains below national targets, but national prices are approximately double the international price of wheat, suggesting that marketing initiatives will not lead to increased production, and indeed, if the market allowed, imported wheat would be sourced by many processors at a lower price. The price of bread is also controlled by government, leaving little room for agribusinesses to operate profitably in the wheat value chain. In the chickpea value chain, although the Kabuli variety has been strongly promoted as an export crop by AMDe, problems appear to remain regarding the quality of the Kabuli varieties currently being produced. Although larger than the Desi varieties, the product is still too small for the international market. AMDe staff expressed the opinion that more research is needed to improve the seed quality before Ethiopian Kabuli chickpea is ready for export markets. At the same time, the project has provided a number of cooperatives with the quality cleaning and grading equipment required for export, which appears inappropriate considering the production challenges. We recommend that AMDe focus on production for the domestic market rather than promoting export of chickpea, at least until the quality issues have been resolved, which may take a number of years, and stop distributing equipment 42 that is only required for export production. The domestic market for Kabuli varieties of chickpea appears strong, but if marketing and agribusiness opportunities for domestic consumption are limited, chickpea activities should be scaled back in light of the export quality challenges. Concerns about quality should be communicated to the Feed the Future Innovation Lab for Climate‐Resilient Chickpea and potential for collaboration should be explored. For a number of export‐focused value chains, most notably honey and meat, particularly in the highlands, a key challenge is that the domestic price far exceeds the international price (This is especially the case when the domestic market contains a large element of informality, as in the case of meat, for which formal processing adds substantially to the final price of the product, reducing its competiveness). Both AMDe and LMD designs are focused on export marketing strategies for these value chains, but have struggled to find sufficient supply for their export processors. Rather than trying to force the market towards export production, efforts would be better directed towards ensuring that smallholder producers and those below the poverty line are able to participate in the market, locally or nationally. Both honey and small ruminants are among the strongest pro‐poor value chains in the FTF portfolio, but the poor are not currently prepared to produce in the quality and consistency required by the export market, nor does the market pay a sufficient premium for such producers to be attracted to these markets. Based on local market conditions there are important marketing opportunities and constraints in these value chains that could have immediate pro‐poor benefits, and resources would be better spent working on these aspects of the value chain rather than on export promotion. Budget resources freed up from the investment grants and wheat value chain should be reallocated to strengthen the cooperatives and other businesses that have already received grants, and on linking smaller producers to markets. As grantee businesses expand, their training needs will shift, and continued training, capacity‐building and mentoring are essential to ensure sustainability. As the AMDe mid‐term evaluation highlights, efforts for the remainder of the project should focus on deepening and strengthening progress, rather than seeking to expand the projects reach more broadly. This is particularly true for AMDe, but is an important lesson for other projects that are not yet as far along. In addition to reallocating budgets to strengthen existing investments and focus on more pro‐poor aspects of value chains, it may be appropriate to reconsider the division of effort between ENGINE and other projects. While in AGP woredas it may have been appropriate for ENGINE to engage directly in livelihood activities to promote nutrition‐sensitive agriculture, (since LMD and AMDe are not designed to work directly at the household level), the project is now poised to expand into woredas where GRAD and PRIME both have direct contact with vulnerable households, allowing new and potentially more synergistic approaches to collaboration to be developed. The GRAD/ENGINE collaboration appears to have a clear division of labor that would allow each project to focus primarily upon its core competencies, but to nevertheless work together to ensure both effective SBC and resilience. The collaboration between ENGINE and PRIME is still in the early stages, but could be developed on the same basis. In particular it is important that lesson’s such as those derived from “Milk Matters”56 should be successfully incorporated into PRIME’s market‐oriented activities, in a way that can ensure the commercial sustainability of the desired nutritional outcomes, especially in relation to the development of the feed and fodder value chain. 5.2.2 Redesign In addition to opportunities for immediate reprogramming, the MTE team offers several recommendations for the redesign of future FTF programs. These recommendations are based on our understanding of the goals and objectives of FTF as well as the unique opportunities and constraints in the Ethiopian context. The recommendations build on both the successes and lessons learned to‐date 56 Sadler, K., Mitchard, E., Abdullahi Abdi, Yoseph Shiferaw and Catley, A. (2012). Milk Matters: The Impact of Dry Season Livestock Support on Milk Supply and Child Nutrition in Somali Region, Ethiopia. Feinstein International Center, Tufts University, Addis Ababa. http://fic.tufts.edu/assets/Milk‐Matters‐2.pdf accessed May 2015. 43 from FTF. When designing future FTF programming, several principles should inform the design:  Re‐orientate to a genuine pro‐poor focus FTF programming should explicitly target poor households and projects should be designed to recognize the unique risks these households face, which can be very different from the challenges of better‐off households. The introduction of improved technologies to increase yields is of little benefit to the most vulnerable households if they are unwilling to adopt those technologies. Experience has shown that this can often be the case and the importance of formative research to understand barriers to adoption of improved technologies amongst the most vulnerable cannot be underestimated. Such an approach likely requires interventions targeted at the household level, as well as interventions designed to address systemic barriers to economic development. A strong pro‐poor focus would recognize that the underlying causes of poverty and under‐nutrition are deep‐seated. There are no “silver bullets” or easy wins, and targets should reflect this reality. While programming will necessarily engage households beyond the most vulnerable, progress against indicators should measure results for this population, and targets should be set accordingly.  Quality is preferable to quantity One of USAID’s greatest strengths is the ability to demonstrate strategies that can be scaled‐up by government and other donors. Particularly in the Ethiopian context where there are many development partners and government programs, as well as a very large population of relevance to FTF, tradeoffs have to be made between program depth versus breadth. It is not possible for FTF to engage directly with all vulnerable households, a reality acknowledged in the current strategy. Instead of attempting broad coverage, or high numerical targets, FTF investments will be more sustainable if efforts are more concentrated to ensure high quality. USAID Ethiopia experience has shown that when USAID introduces new innovations, government and other development partners are receptive to these new approaches. Through its role in government program and development partner platforms, USAID is well‐positioned to transfer best practices and experiences to government for scale‐up, but first their effectiveness must be demonstrated through FTF.  Build on complementary programs To maximize the overall impact of investments, FTF should build on strong, existing programs with synergistic objectives. The newly designed PSNP4 program provides a clear opportunity for this approach. In addition to supporting the same target population, FTF could benefit from the logistical and institutional experience of implementing partners locally if these programs were to be co‐located. The current FTF Ethiopia portfolio contains a range of projects designed to address key constraints in the agricultural system and contribute to government programs, particularly the AGP, and to some extent, the PSNP. While the current design includes important enabling environment initiatives, market development, and livelihoods support for the poor and vulnerable, the portfolio is widely dispersed, both geographically and in terms of objectives, diluting its impact, and the pro‐poor targeting of the program is not as direct as it could be. Feed the Future II should seek to deliver a more integrated, targeted pro‐poor program. Ongoing design processes for key government programs, most notably the PSNP and AGP, but also SLMP and the new Livestock program, offer new opportunities for the next phase of Feed the Future. Key developments include:  Expansion of PSNP from selected woredas to become a national rural safety net program  Integration of the PSNP and HABP program, expanding the scope of PSNP from a asset‐maintenance to an asset‐building program  Expansion of AGP woredas, including more overlap with the new PSNP coverage area  Inclusion of new value chains in the AGPII, such as pulses, horticulture, and poultry, that have potential for a pro‐poor focus. Each of these developments represents an opportunity for USAID to support government priorities in new ways that are closely aligned with FTF priorities, and will alleviate some of the constraints 44 currently faced by FTF. The geographic expansion of PSNP4 will allow for overlap of PSNP and AGP interventions, and creates the opportunity to layer Food for Peace and FTF funded interventions beyond those of GRAD alone in the same geography. Similarly, with the merging of PSNP and HABP, the previous focus of PSNP on asset‐maintenance will expand to asset‐building, which aligns much more closely with FTF objectives, and offers even greater synergies between FTF and Food for Peace activities than have existed before. While the geographic expansion not only of PSNP but also AGP will stretch resources and may potentially dilute government’s ability to deliver the full programs, it provides an opportunity for USAID to engage in both market‐focused and livelihood‐focused interventions in the same areas. Opportunities also arise from the expanded scope in AGPII to cover additional value chains that may have greater potential to raise the incomes of poor households with limited land holdings57. This expanded scope will allow USAID to focus on those interventions that are strongly pro‐poor while still supporting AGP priorities. A potential FTF redesign could build on the strengths and successes of USAID with the PSNP, using a model similar to PSNP+ and GRAD, in which USAID projects influence national policy through example, while simultaneously contributing to government program objectives in specific geographic areas.58 Such a program would include both highland and pastoral coverage, with tailored programming to the socio‐cultural, economic and environmental realities of different regions. Title II Food For Peace funding would provide the foundation to ensure that assets are maintained and the poor can begin investing in increased productivity and income‐generating activities. Layered on top of this would be a series of programs funded by FTF including:  Household interventions focused on production (including technical assistance, land management and inputs), access to finance, market linkages and pro‐poor employment, Mechanisms such as the VESA group model have been proven to be effective for promoting a wide range of agriculture and non‐agriculture‐based interventions in a scalable fashion, and can be designed to link closely with the asset protection offered through Food for Peace support and the market‐facilitation interventions. Household interventions should focus on a diverse range of income‐generating activities, and not be limited to specific value chains (although certain value chains will necessarily feature prominently in household strategies based on their resources and market demand). Particularly for the target population of FTF, most of whom own less than 0.5 ha of land, it is probable that multiple income streams will be required to meet income targets.  The programming of household‐focused interventions would also allow the implementation of activities designed to influence stunting directly (as opposed to indirectly through the Health Army). The experience of ENGINE in BCC could be used to combine proven BCC/SBCC with the critical elements of WASH, vitamin fortification and all of the eight different factors affecting stunting could be combined within a single project and in a limited area to demonstrate the benefit of convergence for subsequent replication by other parties.  Demand‐driven market facilitation interventions focusing on pro‐poor agricultural, livestock and non‐agricultural opportunities, as well as the supply and marketing of key inputs needed to ensure livelihood sustainability, particularly feed and fodder can be layered on top of household‐level interventions. These marketing initiatives would target the poor as their beneficiaries but would engage a broader range of actors in order to strengthen markets and ensure that all linkages in the value chain are supported. Systemic support for nutrition outcomes, including better sanitation and healthcare facilities is also needed for a holistic approach. Such interventions would build on the experience with AMDe and LMD, but would be tailored to cover the geographic areas of overlap between AGP and PSNP, and the agricultural, livestock, and non‐farm value chains which poor households are engaged in. 57 While the examples of pulses, horticulture, and poultry look promising from a pro‐poor perspective, a comprehensive analysis of the pro‐poor potential of each opportunity is needed. 58 Reference to a “PSNP+ or GRAD model” is not intended to suggest that all activities would resemble a GRAD project. Rather, the example is given to highlight the model of working intensively in a limited geographic area to achieve results that are then replicated by government. FTF can effectively serve as a pilot for holistic demand‐ driven agricultural and nutrition focused development programming. 45  Policy, coordination, and enabling environment support, as well as technical assistance for key government institutions, including the Ministries of Agriculture, Trade and Health would constitute the final “layer” of FTF programming. While specific interventions targeting the poor are necessary to lift households out of poverty in the short‐term, long‐term development requires economic growth and stronger agricultural and non‐agricultural sectors. These interventions, while not large in number or financial resources, would strategically address capacity and policy priorities that need to be addressed to ensure the sustainability of FTF investments. The conceptual framework for such a redesign is shown below: 6. MANAGEMENT OPPORTUNITIES The Ethiopian FTF program is conceptually simple, but complex in its implementation, requiring a substantial LOE dedicated to its management if it is to be effective. The size of the program is sufficient to justify its own management structure and tools both within the Mission and amongst the implementing partners. Nevertheless, evidence of such specific and focused management is limited. Instead, it would appear that FTF management is one of the many activities that occupy different mission staff, while coordination amongst implementing partners, although occurring regularly amongst COPs, does not always occur at the level of implementation and is in fact only specified in the SOW of one of the five main projects. Under such circumstances, the capacity of management to achieve the necessary oversight and to respond in the event of unforeseen developments is limited. The following sections highlight issues of management that were observed by the MTE team and the opportunities for improvement that exist. 6.1 Portfolio Management Causal pathways: The value chain approach to development features strongly as an element of the causal pathways of DO1 and much of FTF. This is appropriate when the target group has the capacity to participate effectively in those value chains, but many of the most vulnerable households are unable to do this. Nevertheless, while this suggests a need to modify the causal pathways to account for the specific needs and capacities of the target group, there is little evidence that management has 46 attempted such modification. The MTE suggest that a greater emphasis upon formative research to validate and amend the causal pathway and the flexibility to do so would assist in the management of what is after all an innovative program. Indicators and targets: While some FTF indicates had well‐specified targets, the MTE team was unable to determine the targets set for some potentially critical indicators (see section 3.2). The absence of such targets makes it impossible to determine the adequacy of resources to meet the overall FTF objectives and goal. Even though some targets might have been impossible to set until the baseline data had been collected, the opportunity now exists to specify targets based upon both baseline data and experience in implementation. Data reporting and management: The quality of the reported data assessed by the MTE team varied considerably from rigorous counts to broad extrapolations from samples. The definitions applied by different projects were not always consistent and the aggregation of different types of data has tended to obscure the real nature of achievements. Portfolio management could strengthen the quality of data reported by coordinating with individual project M&E managers to ensure consistency in data collection procedures and in definitions. The MTE team was provided with an FTF PMP, which although described as a “living document” was apparently last updated in 2010 and has been effectively superseded by the FTFMS. This is a web‐ based spreadsheet record of targets and actual performance for each indicator disaggregated as appropriate. Its structure (running to over 5,000 lines of data on a single worksheet) does not facilitate either the accessibility or the interpretation of data, and it is not surprising that the MTE encountered numerous contradictions between FTFMS data and that provided by individual projects in their own PMP matrices. The FTFMS as it was supplied to the MTE team does not facilitate an assessment of progress or any other management requirement. It would be more useful if it could be presented as a structured workbook that would allow both headline reporting and drilling down to individual IRs so that information could be shared in an accessible form. The size and complexity of the quarterly compiled FTF data set does not lend itself to immediate comprehension by Mission staff. As a result, although data is compiled and dispatched to FTF management in Washington, its accuracy and/or significance is not always immediately understood at Mission level. This can lead to overstating expectations59 or simple inaccuracies60 in reports and presentations. It can also result in under reporting if indicators are not used to capture relevant developments on the ground.61 In all cases, the effectiveness of management is compromised. Use of M&E resources: The MTE team found that the M&E resource available for portfolio management is under‐utilized so that management’s appreciation of developments on the ground is limited as is capacity to react to such developments or to external changes. The role of the M&E manager does not extend much beyond the collection and compilation of data. There is little evidence that M&E management is able to feed its analysis of data into ongoing portfolio management in a manner that might facilitate proactive program management. Coordination with ALT: Given that one of the five main FTF programs is managed by ALT (GRAD), it would be helpful to overall portfolio management if there were to be greater involvement of ALT in the oversight of the FTF program. Much of the strategy that originally guided Ethiopia’s FTF was based upon the observations made through ALT projects, especially RAIN, SPSNP and the LIU, which led to the recognition that many vulnerable households had a) limited capacity to engage in agriculture and were in fact net buyers of food, and b) become destitute not as a result of a specific shock, but due to the pressure of inexorable population growth acting within a limited resource (land). As such, the perspective on poverty and food security that has been developed within ALT can provide useful direction to the FTF program, especially insofar as it relates to the immediate needs of beneficiaries in the FTF ZOI. 59E.g. the anticipated 300% increase in yield of chickpea based upon a variety that is recognized to be generally unsuited for the export market. 60 The distribution of ”frost” resistant wheat seed that was expected to increase wheat production was in fact “rust” resistant wheat, and was susceptible to the entirely new variety of rust that emerged in Ethiopia in 2014. 61 Comments from the Mission highlighted the fact that LMD in particular was undertaking a considerable number of activities that were not captured by relevant FTFMS indicators. 47 Portfolio management would be further strengthened if adequate resources could be provided to M&E to ensure the collection of accurate data, to allow the analysis of compiled results, and to generate succinct reports on ongoing developments that can allow portfolio management to make informed decisions. Portfolio management itself appears to be under‐resourced and requires the dedicated capacity to manage the overall FTF program if it is to be effective. 6.2 Project Management Coordination between individual FTF projects at the COP level is considered to be good, but the MTE team found there is nevertheless a focus on individual project targets that overrides or at least ignores the overarching FTF objectives and goal. Given that the individual SOW’s of each project do not focus on these objectives or goal and (with the exception of LMD) do not require co‐ordination between projects, this is to be expected. To enhance program coordination, future project SOW’s might not only specify such cooperation, but might also stipulate the LOE required to achieve it. Coordination between projects at the level of implementation is not always as effective and the MTE found occasional examples where individual project staff were not implementing coordinated activities that had been agreed at COP level. Nevertheless, overall, the MTE concurred with individual project mid‐term evaluations who found only minor issues of management and noted that overall that the performance of project management was professional and competent. COPs and their managers were generally well informed of their project interventions and results and understood the causal pathways that contributed to their project goals. An understanding of the extent to which individual project goals might contribute towards the FTF objectives and goals was also both evident and realistic. 7. SUMMARY OF CONCLUSIONS AND RECOMMENDATIONS The following section summarizes the main conclusions of the MTE and presents recommendations (in italics) where appropriate. Progress to date An overall assessment of progress against indicators suggests that output indicator targets are generally likely to be met. There is no strong evidence to suggest that levels of effort and resources are not appropriately matched to the achievement of output targets. Impact and outcome targets in key areas of finance, employment and business development are less likely to be achieved (when such targets have been specified). This may be due to factors beyond the manageable interest of the FTF program, but it may also reflect the diffuse nature of market‐based interventions and the limited coordination between such projects. The achievement of objective‐level impact targets is more problematic still. While it is possible that there may be significant improvement in indicators to the targeted levels and beyond, it is unlikely that any of these targets will be met as a result of program interventions alone. Although AMDe and possibly LMD are on track to achieve the majority of their individual project objectives, they are unlikely to be as successful in achieving FTF impacts at a scale necessary to achieve the program objectives. Similarly GRAD and to a lesser extent PRIME might be expected to be effective in supporting the limited number of beneficiaries in the 46 woredas in which they operate, but their effects will be diluted within the overall FTF ZOI. The extent to which ENGINE may be able to influence the second FTF objective is currently a matter of debate beyond the expertise of the MTE. Because the application of the goal and objectives of the FTF program across the entire FTF ZOI is not aligned with the capacities of the component projects to achieve impacts across that same area, the intermediate results, objectives and goal of the FTF program are unlikely to be met. Evaluation of the FTF contribution of each project should be restricted if possible to those woredas in which each project is operational. Evaluation against FTF targets across the entire ZOI will have little meaning. The success of GRAD within its operational area has not been well captured by performance indicators. Nevertheless, the project has made a significant impact upon GoE policy, sufficient to achieve replication via PSNP4 over a substantially greater area than the project itself. The example of GRAD is a strong argument for the development of future programs over a limited area with the necessary resources, 48 focus and scope to achieve measurable success within the timeframe of programs such as FTF. This does not suggest that programs such as AMDe or LMD are unnecessary. They serve an essential purpose, but will achieve results over a longer time frame and through a target group that is not the direct focus of FTF. While there is little point in simply scaling up GRAD, there is considerable scope for developing new programs around similar project design, but extending the scope to focus not only on vulnerable households but on the development of the markets with which those households might interact, in a manner similar to the coordination between GRAD and LMD. Program design The USAID Ethiopia DO1 and FTF development strategies have not been completely aligned and as a result, the causal pathway linking project interventions to the FTF objectives and goal is not always robust. Because a significant proportion of households in the ZOI have less than 0.5 ha of land, initiatives such as those of AMDe and LMD to increase the agricultural gross margins of producers are unlikely to have any direct benefit to the most vulnerable households in the FTF ZOI and may actually reduce their food security in the short term. Program design should take the economic characteristics of households within the ZOI into account prior to designing interventions. A baseline survey undertaken across the ZOI after program design has been completed and the work begun has little formative value and is useful mainly for monitoring. Ongoing formative research into vulnerable household needs and preferences regarding pathways for development is required to inform program design. In particular, barriers to the adoption of new technologies need to be understood and addressed with appropriate measures (e.g. crop insurance). Reliance upon AGP counterpart programs has constrained the impacts of AMDe and LMD interventions which have been limited by factors beyond their manageable interest, especially factors associated with the increased production of commercial surpluses that have not generally materialised. Similarly, institutional changes in counterpart organizations have restricted the ability of both AMDe and LMD to coordinate effectively with GoE. FTF projects should align with GoE guidelines, but should endeavour to ensure that all aspects essential to successful development are incorporated within the manageable interest of each project, or ‐ given effective coordination between projects ‐ of the program overall. Observation would suggest that vertical integration might play an important role in reducing the risk of non‐performance by counterparts and enhancing the overall effectiveness of the assistance provided. GRAD has demonstrated how a project may align with the principles of the GoE HABP program, and yet be independent of it. PRIME has the opportunity to follow the same path. If this is not the case, (i.e. key factors affecting the success of a project lie outside the control of the implementing agency, then chances of success are compromised, and USAID must make a determination as to whether or not the investment is worthwhile. Overall, the MTE found that key assumptions regarding the potential for market strengthening to immediately enhance food security, the impact of agricultural improvement upon the most vulnerable households and the capacity of counterpart institutions to achieve expected levels of production have not been validated, so that, although systemic market development projects might be successful in their own right, their impact upon FTF objectives was weakened. At the same time it was observed that assumptions regarding access to finance, opportunities for off‐farm income generation and availability of livestock feed have also been inaccurate, limiting the effectiveness of interventions of GRAD and PRIME. Most importantly however, the push/pull hypothesis, while potentially valid in the long‐term has but few examples of its successful application in the short term, suggesting that while push/pull may be a key driver of long‐term poverty alleviation, it is unlikely that many benefits of push/pull will be realized within the short time frame of a five year program. Program management The introduction of a coordinated interdepartmental FTF program should have been undertaken in parallel with the introduction of a dedicated portfolio management system. This did not occur and as a result, the capacity of management to achieve the necessary oversight and to respond in the event of unforeseen developments has been compromised. Portfolio management needs to be strengthened through support to dedicated M&E to ensure the collection of accurate data, the analysis of compiled 49 results, and the generation of succinct reports on ongoing developments that can allow portfolio management to make informed decisions. Portfolio management itself appears to be under‐resourced and requires dedicated capacity to manage the overall FTF program if it is to be effective. Program monitoring The value of reported data for FTF program management is compromised by a number of factors. There are no PMP targets for a number of FTF indicators, which limits the usefulness of such indicators and prevents direct assessment of the extent to which available resources are adequate to meet program objectives. Some indicators aggregate data from sources that do not bear strict comparison, while others are subject to varying interpretations by different projects. Still others may be incorrectly entered into the FTFMS. Indicators should be revisited by Mission program monitors in conjunction with project M&E staff to ensure consistency and accuracy of data entry. Those indicators that inappropriately aggregate data from different sources should be reported on a disaggregated basis. Where targets are missing, they should be agreed upon and set as a matter of priority. The majority of indicators for nutrition are only collected based on the population‐based surveys. Thus it is difficult to monitor progress towards the ultimate goal of stunting reduction or change the course of activities if progress is not being made. A clear plan for the types of outputs and outcomes needed to achieve the high‐level indicators was not well‐defined at the time the strategies were developed. The lack of targets for some key indicators makes it impossible to monitor performance or to allocate an appropriate level of resources to achieve the desired result. It may even suggest that management does not know what exactly it is trying to achieve. Using experience to date, targets should be set for all indicators. Project coordination Much of the strategy that originally guided Ethiopia’s FTF was based upon the observations made through ALT projects, especially RAIN, SPSNP and the LIU, which led to the recognition that many vulnerable households had a) limited capacity to engage in agriculture and were in fact net buyers of food, and b) become destitute not as a result of a specific shock, but due to the pressure of inexorable population growth acting within a limited resource (land). Given that one of the five main FTF programs is managed by ALT (GRAD), it would be helpful to overall portfolio management if there were to be greater involvement of ALT in the oversight of the FTF program. The perspective on poverty and food security that has been developed within ALT can provide useful direction to the FTF program, especially insofar as it relates to the immediate needs of beneficiaries in the FTF ZOI. Specific interventions A number of other developments have occurred within FTF, which although not well captured by indicators may nevertheless result in significant positive change. These include the development of the Somali MFI, substantial formative research, the development of the Women in Agribusiness leadership Network and the promotion of water carrying by men in Dembeli Keta. Ongoing support should be provided to all of the above initiatives for the remaining duration of the program, and in particular that the successes achieved should be replicated in other projects where possible. ENGINE has undertaken a considerable body of formative research which has illuminated the issues surrounding stunting. The skills and approach that ENGINE has developed in this area should be used in other aspects of FTF projects, including the understanding of low rates of livestock off take and barriers to the adoption of new technologies. The predominant off‐farm IGA (sheep and goat fattening) is highly susceptible to feed constraints. A detailed assessment of the feed and forage supply sub sector should be undertaken and additional program interventions to strengthen this aspect of the value chain should be undertaken if necessary. Investment Some investments are potentially vulnerable to the criticism that aid has been used “buy economic growth”, a concept that has been criticized on both theoretical and empirical grounds.62 While such a 62 Easterly (2003), Ibid. 50 policy can sometimes be justified, nevertheless, it would appear that pressure to achieve disbursement targets has resulted in a tendency towards the “purchasing of growth” reducing the long term impact of the investments made. Project output targets for grant disbursement should be avoided when the desired outcome (of business development) can be measured in other more direct ways. The allocation of new large grants should be discontinued. Small grants targeted in specific projects to address key barriers are still appropriate (such as the PRIME grants to private veterinary pharmacies). Sufficient investments have now been made, and future efforts should focus on strengthening the capacity of grantee institutions to maximize the effectiveness of the grants and ensure their sustainability. Improved technologies have been introduced by a number of projects, and it is tempting to consider these as ends in themselves. Nevertheless, the application of such technologies elsewhere in the past has shown that their full impact is only realized if ongoing support to their effective utilization and management can be sustained over a period of at least five years. Resources should be programmed for the ongoing support over at least another five years, of those initiatives to introduce improved technologies that have been undertaken to date. Cross‐cutting issues Although there is a consistent gender component throughout FTF and despite the fact that all data was gender disaggregated, it was not always evident that the gender concerns were fully addressed. In some cases, when indicators showed female participation in activities to be lower than planned, project teams did not appear to have reflected on why this had occurred, and whether the activities they were promoting met female needs. GRAD’s work with VESA groups has probably made one of the strongest contributions to women’s empowerment. The VESA group model is designed to help poor households build social and human capital and therefore save money, gain access to loans provided by the group, and access loans from micro‐finance institutions (build financial capital). Because of the high female membership in VESA groups, this model is an ideal mechanism for improving women’s access to credit and decision‐making regarding spending. As one of the largest FTF portfolios globally, FTF Ethiopia is in a unique position to provide lessons learned not only for future USAID Ethiopia projects, but to inform FTF more broadly. Currently neither the knowledge management components of projects, nor efforts at the Mission, are sufficient to capture these experiences and share them. AKLDP can play a key role to play in capturing the knowledge and lessons learned by projects and translating them to a broader audience. AKLDP can potentially play several roles in the remainder of the project period including the compilation of useful project experiences, particularly on issues of relevance to more than one project and the undertaking research on issues or challenges identified by projects. Opportunities for immediate reprogramming In the short term, opportunities exist to curtail work on the wheat value chain, reduce the emphasis on meat and chickpea export market development, and concentrate instead upon meeting domestic market needs. There is also the opportunity to cease any further large scale grant disbursements and concentrate resources on smaller grants and increased training and mentoring to ensure the sustainability of progress achieved to date. Long‐term reprogramming The fact that systemic projects such as AMDe and LMD are unlikely to have greatest impact within a five‐year timeframe suggests the need for a longer‐term programmed approach to the development of food security. Evidence in other countries over the last 40 years suggests that the impact of a five‐year systemic program will be lost unless it can be reinforced by continued intervention (although this might be on a reduced basis) to ensure that policy and institutional changes are maintained. There is little evidence to suggest that programs such as AMDe or LMD can be expected to generate significant and sustainable change unless they are able to exert an influence over a period of at least 10 years. Similarly, it is evident that programs such as GRAD and PRIME are only able to impact a small number of beneficiaries during the course of a five‐year program and in the longer term, such programs may need to be developed further to enhance outreach and improve targeting. 51 This perspective strongly suggests that a long‐term layered approach to the challenges of FTF, in which direct interventions with beneficiaries are undertaken in conjunction with more systemic market development initiatives might provide a more robust solution than has been seen to date. Such an approach would also include a direct nutritional component that addressed each of the eight different factors affecting child nutrition (and especially stunting). An opportunity now exists through the introduction of PSNP4 to build upon both the successes and lessons learned to date, through the implementation of a model similar to GRAD that could influence national policy through example. Such a program would include both highland and pastoral coverage, with tailored programming to the socio‐ cultural, economic and environmental realities of different regions. Title II FFP funding would provide the foundation to ensure asset maintenance and food security, while layered on top of this would be a series of projects funded by FTF including:  Household‐level interventions to strengthen production and income  An integrated approach to nutrition, incorporating response to all the various facets of the stunting syndrome  Demand‐driven market facilitation interventions targeting both the poor as well as others necessary to ensure the viability of value chains  Policy and enabling environment support, as well as technical assistance for key government institutions. Such an approach, by minimizing factors beyond the manageable interest of the program and focusing resources upon a limited number of beneficiaries would be most likely to achieve success. Such success can then be used to leverage other donor and GoE resources necessary to achieve program replication at a scale that would otherwise be impossible through the use of FTF resources alone. FEED THE FUTURE ETHIOPIA EXTERNAL MID‐TERM PERFORMANCE EVALUATION REPORT May 2015 Annexes Annex A. Scope of Work   Annex B: Bibliography   Annex C: Interviews and Field Visits   Annex D: Results Frameworks and Indicators    Annex E: Statement of Differences   Annex F: CVs Annex G: Disclosure of Conflict of Interest STATEMENT OF WORK:  ANNUAL FEED THE FUTURE PERFORMANCE EVALUATION IN ETHIOPIA I. INTRODUCTION In 2009, President Obama pledged at least $3.5 billion dollars over a period of three years as part of                                       Feed the Future (FTF), a global initiative to improve agricultural development and food security in 19                                 countries in need. Through Feed the Future, the U.S. supports the Government of Ethiopia’s                             commitment to country-led development programs that facilitate economic growth and development.                     These initiatives also provide USAID and collaborating U.S. Government agencies with a unique and                             promising opportunity to assist in implementing a transformative food security strategy aligned with                           Ethiopian objectives.  In line with the overall objectives of FTF globally and with a total investment portfolio of about $370  million, USAID/Ethiopia’s Economic Growth and Transformation (EG&T) and the Alternative  Livelihoods and Transition Office (ALT) works to reduce poverty and childhood stunting measurably  with its geographic zone of influence. To do this, EG&T works in three project components: (1)  agriculture growth enabled food security; (2) linking the vulnerable to markets; and (3) fostering policy  and capacity improvements. EG&T links these three components with the “push­pull” framework,  which seeks to catalyze market growth in productive agricultural areas while building viable economic  links to vulnerable populations in less productive areas. Four complementary integrated programs also  support the larger initiative in Ethiopia, which are Nutrition, Climate Change and Adaption, Private  Sector Development, and Humanitarian Assistance.   EG&T, on behalf of USAID/Ethiopia, expects the annual FTF performance evaluation to provide  feedback on the level of progress regarding the initiative’s planned results (both the quantitative and  qualitative) against stated objectives and goals, the appropriateness of EG&T’s investments in different  project components and activities, the linkages among components and integrated programs.  The  evaluation will provide specific information that will feed into EG&T’s Bureau of Food Security portfolio  review in March and produce actionable management recommendations to aid future implementation.  The analysis will cover area 4.6 Agriculture activities funded with 150 account funds, as defined in the  USAID Standardized Program Structure, but not activities that Title II majority funds.   USAID’s evaluation policy encourages independent external evaluation to increase accountability to  inform those who develop programs and strategies, and to refine designs and introduce improvements  into future efforts.  In keeping with that aim, this evaluation will be conducted to review and evaluate the  performance of the entire FTF portfolio in Ethiopia.  The evaluation will focus on assessing the  program’s mid­term performance (May 2011 through November 2014) in achieving its goals,  objectives, and results.  II. BACKGROUND USAID/Ethiopia’s Country Development Cooperation Strategy (CDCS) goal—“           Ethiopia’s Transformation to a Prosperous and Resilient Country Accelerated”—               is aligned with the         1 Annex 1 Government of Ethiopia’s (GOE’s) five year Growth and Transformation Plan (GTP), which builds on                             the significant expansion of the economy and basic services over the past five years. The                               USAID/Ethiopia CDCS comprises three Development Objectives (DOs) and one Support Objective.                       The Economic Growth and Transformation (EG&T) office, in coordination with the Alternative                         Livelihoods and Transition (ALT) office, implements DO1 (Increase economic growth with resiliency in                           rural Ethiopia). As shown in the DO Results Framework (see annex I), the achievement of this DO                                   depends upon the combined success of five interdependent intermediate results (IRs). The strategy will                             demonstrate the potential of market-based agricultural development to reduce poverty and promote                         sustainable livelihoods for chronically food insecure households.   The Feed the Future (FTF) Strategy, which is the key component of DO1, aims to sustainably reduce                                   poverty and hunger through investments in the performance of the agricultural sector, in improved                             nutrition and the improved capacity of vulnerable households to meet their food needs. FTF                             investments will lead to improvements in food availability, access and utilization. A better performing                             agriculture sector should improve both food availability and access, while improved food access and                             utilization will improve nutrition. The strategy demonstrates the potential of market-based agricultural                         development to reduce poverty and promote sustainable livelihoods for chronically food insecure                         households. The strategy utilizes a                                 Push-Pull Model that seeks to build the capacity of vulnerable and chronically food insecure households to participate in economic activity (the                           “push”), while mobilizing market-led agricultural growth to generate relevant economic opportunity and demand for smallholder                         production, labor, and services (the “pull”). In Ethiopia, over twenty-six direct implementing partners conduct activities in support of the initiative’s                             goal to reduce extreme poverty by 30 percent and childhood stunting by 20 percent in the geographic                                   zone of influence. Progress against these ambitious goals is tracked at multiple levels. At the highest                                 level, EG&T conducts a population-based survey (PBS), which is a cross-sectional survey of over                             7200 households in the zone of influence and statistically matched control groups that lie outside of the                                   zone of influence. The first wave of the survey was in 2013. The subsequent waves will occur in 2015                                       and 2017. Congress, USAID, and other stakeholders will ultimately judge FTF in Ethiopia on the                               results of the 2017 survey wave compared to the relevant survey control groups.    In addition to the PBS, all of the largest FTF activities in Ethiopia collect rigorous (albeit sometimes                                   inconsistent) baselines, including, in some cases, baseline data on counterfactuals. ENGINE and                         PRIME are prime examples. Finally, EG&T collects 86 annual output indicators. EG&T collects these                             indicators every November. In practice, EG&T typically combines activity expenditure data, output                         data, performance evaluations, and technical judgment to inform resource allocation decision in the                         absence of rigorous impact evidence.    Since its inception, FTF has produced notable achievements against output targets. In 2013, for                             instance, FTF achieved the following: ● Reached 1.675 million children with nutrition interventions ● Leveraged $70 million USD in new investments into the agricultural sector ● Stimulated over $20 million in new lending ● Created more than 30,000 new jobs among vulnerable people 2 Results Framework The results framework below illustrates FTF’s anticipated results in Ethiopia. III. PROJECT GOALS AND OBJECTIVES In line with the overall objectives of the global FTF strategy, USAID/Ethiopia’s FTF objective is to                                 reduce extreme poverty by 20 percent ($1.25 USD 2005 at PPP) and to reduce childhood stunting by                                   20 percent with the zone of influence, which is an area that includes about 19 million people. The                                     population-based survey will measure the size of these effects against statistical controls in the survey.                               To accomplish this, the initiative consists of three components and four integrated programs.                           Component and integrated program has associated activities and output metrics. Each out-put metric                           has baseline data and targets. All told, the assumption is that the output level targets will be sufficient to                                       achieve DO1 and the poverty and stunting reduction targets.   For the analysis, EG&T will make available a full list of all activities and their associated output metrics.    3 IV. PURPOSE AND USE OF THE EVALUATION FTF in Ethiopia is entering a critical time. The initiative has less than three years to show dramatic results in the last wave of the PBS. To do this, EG&T will need to make intelligent new investments, get the maximum impact out of its existing investments, and make some very difficult management decisions. This annual performance assessment will be a key input into all these difficult tasks.   EG&T expects the annual FTF performance evaluation to provide a new perspective on the level of output targets and strategic goals, the appropriateness of EG&T’s progress regarding the initiative’s planned results (both the quantitative and qualitative) against stated investments in different project components and activities, and the linkages among components and integrated programs.  The evaluation will provide specific information that will feed into EG&T’s Bureau of Food Security portfolio review in March and produce actionable management recommendations to aid future implementation. Specifically, the evaluation will: 1. Assess progress against objectives and goals specifically the extent to which planned results (both the quantitative and qualitative) have been achieved;  2. Assess the appropriateness (or effectiveness) of USAID investments in different  program components and integrated programs areas at the activity level;  3. Identify actionable recommendations for reprogramming of funds to achieve program goals;  4. Identify actionable management recommendations to maximize the impact of the existing portfolio of investments, including the interactions among activities across components and integrated programs.    V. EVALUATION QUESTIONS A. Performance Results 1. To what extent is the FTF Initiative progressing against its outcome and portfolio review targets?  The evaluation should analyze actual results versus the targets outlined in the FTFMS and the intermediary targets established in the 2013 BFS portfolio review, including identifying the activities and approaches that are succeeding or failing relative to their budget expenditures.   2. Are the Initiative’s outcome targets and intermediary BFS targets sufficient to achieve the higher level poverty and stunting reduction targets? The evaluation should provide some perspective on the ability of USAID to affect its higher level targets given its approach, targets, and funding levels.      4 B. Investment Quality 3. Has EG&T made cost effective investments with its budget?   The evaluation should analyze the relative cost effectiveness vis-a-vis output targets and higher level goals of different activities across program components.       4. Are there clear opportunities (perhaps identified in question 3) to improve the chances of accomplishing output targets and higher level goals by reprogramming funding among activities? The assessment should make clear recommendations about the reallocation of scarce resources, especially budgets, moving forward. The assessment’s recommendation should take into consideration constraints, such as directed fund categories and the time associated with reprogramming. C. EG&T Portfolio Management 5. Are there clear management changes at the portfolio or activity level that EG&T can execute to improve its chances of success? The assessment should make clear recommendations about portfolio, component, and activities management changes that would increase FTF chances of accomplishing high-level objectives and the BFS intermediary goals. EG&T is especially interested in management changes that would better help us better coordinate with activities that fall outside of this evaluations’ scope, such as USAID Global Health Programs and the Productive Safety Net Program, and changes that would advance gender-intelligent programming approaches. 6. How should USAID refine the push-pull hypothesis? The assessment should revisit the push-pull model and make recommendations about improving the link between the “push” and the “pull.” As part of that analysis, the assessment should consider interventions not traditionally associated with agriculture, such as interventions in the financial or macroeconomic sectors.   VI. EVALUATION METHODS A recommended evaluation design and data collection methods are presented below. However, the                         evaluation team will be responsible for refining the design as recommended below or proposing an                               alternative design for consideration and approval by USAID. The evaluation strategy and data                           collection methodologies should include mixed approaches for better triangulation and validation of                       findings. The team should present an evaluation questions matrix showing the source of data, method                               of data collection and also the tool to be used to answer each of the evaluation questions. The                                     methodology will be presented as part of the draft work plan as outlined in the deliverables below,                                   approved by USAID/Ethiopia and included in the final report. The evaluation team will have available                               for their analysis a variety of program implementation documents, baseline surveys and reports. With the                               exception of classified materials, the team will have access to internal USAID reports. Methodology                           strengths and weaknesses should be identified as well as measures taken to address those weaknesses.                               5 All data collected and presented in the evaluation report must be disaggregated, as appropriate, by                               gender, geography and value chain. (a) EVALUATION DESIGN It is recommended that a non-experimental design be used that will focus on measuring project results                                 before and after project implementation using project monitoring and survey data. The before project                             data should be drawn from the baseline survey report produced by the implementing partners. The                               evaluation team is expected to strengthen this design to make it as rigorous as possible or propose                                   alternative evaluation designs for consideration.   (b) DATA COLLECTION METHODS As stated above, the evaluation team will be responsible for proposing an appropriate evaluation design                               and data collection methods. The team should consider mapping the research questions against the                             quantitative and qualitative data in a matrix/table to show how each research question will be answered.                                 However, it is also recommended that the data collection methods should include the following: Use of  quantitative data should include, but are not limited to:   ● Comparison of current indicator values to baseline data for select output and outcome indicators.   ● Map out the project results against performance measure indicators to show the total number of indicators under each result and whether performance is met/on target  (90-100%), exceeded (>100%), or not achieved (