Agricultural Growth Programme – Agribusiness Marketing and Development Project (AMDe), Ethiopia MID-TERM EVALUATION REPORT February 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Tufts University. The authors of the report are John Fox, Amare Ghizaw, Amdissa Teshome and Nigussie Alemayehu. Agriculture Knowledge, Learning, Documentation and Policy Project (AKLDP), Ethiopia Agricultural Growth Programme ‐ Agribusiness and Market Development (AMDe) Project, Ethiopia EXTERNAL MID‐TERM PERFORMANCE EVALUATION REPORT USAID Contract Number: AID-663-C-14-00002 Agriculture Knowledge, Learning, Documentation and Policy (AKLDP) Project Implemented by: Feinstein International Center Friedman School of Nutrition Science and Policy Tufts University Africa Regional Office PO Box 1078 Addis Ababa Ethiopia Tel: +251 (0)11 618014 www.agri‐learning‐ethiopia.org Disclaimer The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development (USAID) or the United States Government. Contents Executive Summary .........................................................................................................................................i 1. Introduction............................................................................................................................................. 1 1.1 Ethiopia’s agriculture sector .................................................................................................................... 1 1.2 Feed the Future and AMDe in Context .................................................................................................. 1 1.3 The Mid‐Term Evaluation .......................................................................................................................... 2 1.4 Methodology .................................................................................................................................................... 3 2. Evaluation Findings .............................................................................................................................. 3 2.1 Progress made under each of the four major results .......................................................................... 4 Result 1: Improved Competitiveness of the Value Chains ..................................................................... 4 Result 2: Improved Access to Finance and Investment ....................................................................... 10 Result 3: Improved Enabling Environment .............................................................................................. 11 Result 4: Stimulate Innovation and Investment ..................................................................................... 13 2.2 Improvements in gender equity ........................................................................................................... 14 2.3 Improvements in nutritional outcomes ............................................................................................ 15 2.4 Effective resource use .............................................................................................................................. 16 2.5 Project design and management .......................................................................................................... 18 2.6 Sustainability ............................................................................................................................................... 19 2.7 Management ................................................................................................................................................. 21 3. Recommendations...............................................................................................................................21 Annexes A set of Annexes were submitted as a separate document to the main report as follows: Annex 1 Statement of Work Annex 2 Evaluation Team Members – Curriculum vitae Annex 3 Conflict of Interest forms Annex 4 Work plan Annex 5 Bibliography Annex 6 List of places visited and people interviewed Annex 7 Tools used List of Tables Table 1: Average yield increases to December 2014 Table 2: Value of incremental sales attributed to FTF interventions Table 3: Progress against indicators and LOP targets for Result 1 Table 4: Policies, regulations, administrative procedures by their stages of development Table 5: Innovation grants awarded by region Table 6: Innovation grants awarded by value chain Table 7: Incentive‐based women membership drive Table 8: Number of jobs created List of Figures Figure 1: Number of clients benefiting from financial services Figure 2: Budget use by value chain Figure 3: Illustration of the collaborative mechanisms between FTF partners Acknowledgements The Evaluation Team wishes to thank the staff of the Agribusiness and Market Development (AMDe) Project for providing monitoring reports and related documentation, answering questions, arranging meetings with implementing partners and beneficiaries in the regions and Addis Ababa, and commenting on the draft Mid‐Term Evaluation report. The Evaluation Team also wishes to thank the staff of the Agriculture Knowledge, Learning, Documentation and Policy (AKLDP) Project for their insights into Ethiopia’s agriculture sector and their support in finalizing the Mid‐Term Evaluation report. Thanks also to Tufts University Africa Regional Office staff in the Addis Ababa office for their logistical and administrative support. In particular, the Evaluation Team would like to express their thanks to Fasil Yemane, Tsion Fisseha, Yemiserach Woldearegay, and Tewaba Kehinet. The authors are grateful to Adrian Cullis, Chief of Party, AKLDP, for technical editing of the draft report. Finally, the Evaluation Team wishes to thank the many representatives of federal and local government, the Agriculture Transformation Agency, industry associations, Federal Cooperative Agency, Farmers Cooperative Unions, Primary Cooperatives, and smallholder farmers—both men and women—who gave of their time so freely to attend focus group discussions or be interviewed individually. Their perspectives on Ethiopia’s changing agricultural sector and on the contribution made by the AMDe project have been of particular importance in the writing of this report. This cooperation notwithstanding, the views expressed in this Mid‐Term Evaluation are those of the Evaluation Team and any mistakes or misrepresentations are their responsibility alone. Note on Ethiopia’s administrative structure Kebele—the smallest administrative unit in Ethiopia. Kilil or Region—consists of a number of zones; Ethiopia comprises eight main regions. Woreda—equivalent to a district in other countries and the second‐lowest tier of local government. Zone—consists of a number of woredas. Acronyms ACDI/ VOCA Agricultural Cooperative Development International/Volunteer Overseas Cooperative Association AGP Agricultural Growth Programme of the Ministry of Agriculture AKLDP Agriculture Knowledge, Learning, Documentation and Policy Project AMDe Agribusiness and Market Development AMSAP Advanced Maize Seed Adoption Program ATA Agricultural Transformation Agency ATVETs Agricultural Training for Vocational Extension Trainers CAADP Comprehensive Africa Agriculture Development Programme CASCAPE Capacity Building for Scaling up of Evidence‐Based Best Practices in Agricultural Production in Ethiopia CIAFS Capacity to Improve Agriculture and Food Security EAB Ethiopian Apiculture Board ECEA Ethiopian Coffee Exporters Association ECX Ethiopian Commodity Exchange EGTE Ethiopian Grain Trade Enterprise EHBPEA Ethiopian Honey and Beeswax Producers and Exporters Association EIAR Ethiopian Institute of Agricultural Research ENGINE Empowering New Generations to Improve Nutrition and Economic Opportunities Ethiopia Project FAO Food and Agriculture Organization (UN) FCA Federal Cooperative Agency FCU Farmers Cooperative Union FTF Feed the Future Strategy of the United States Government GoE Government of Ethiopia GRAD Graduation with Resilience to Achieve Sustainable Development Ha Hectare (10,000 m2 ) LMD Livestock Marketing Development LOP Life of Project M&E Monitoring & Evaluation MoA Ministry of Agriculture MoI Ministry of Industry MoT Ministry of Trade MT Metric Tonnes MTE Mid‐Term Evaluation M4 Membership, Money, Management, and Marketing Assessment NGO Non‐governmental Organization PC Primary Cooperative PIF Policy Investment Framework PMP Performance Management Plan PSNP Productive Safety Net Programme PRIME Pastoralist Areas Resilience Improvement through Marketing Expansion P4P Purchase for Progress RuSACCO Rural Savings and Credit Cooperative SLMP Sustainable Land Management Project SM4M Sell More For More (a training module) SNNPR Southern Nations, Nationalities, and Peoples’ Region SoW Statement of Work USAID United States Agency for International Development USG US Government VC Value Chain WALN Woman in Agribusiness Leadership Network WASH Water, Sanitation, and Hygiene WFP World Food Programme (UN) ZoI Zones of Influence i Executive Summary This External Mid‐Term Evaluation (MTE) of the USAID Ethiopia Mission‐funded Agricultural Growth Program—Agribusiness and Market Development (AMDe) Project was carried out in two phases, November and December 2014 and January 2015. The Scope of Work was developed by USAID and addressed by a team of three consultants—an international team leader and three local consultants. During the review, the Evaluation Team visited 26 AGP woredas in Ethiopia’s four main regions, interviewing 215 individuals from 66 organizations that included the Ministry of Agriculture (MoA)—at federal and local level—local government, industry associations, private sector businesses, Farmers Cooperative Unions (FCUs), Primary Cooperatives (PCs), smallholder farmers—men and women—and AMDe staff. The AMDe is funded through USAID Ethiopia’s Feed the Future (FTF) program with a budget of USD 50 million.1 Launched in June 2011, the AMDe is funded for five years, with an end date of May 2016. The review findings cover the period from the launch to December 31, 2014, or three and a half years. It is planned that the AMDe will reach one million smallholder farmers. This MTE was delayed past the mid‐point, with the result that the AMDe has 15 months of implementation left following the submission of the first draft report in March 2015. This is unfortunate, as the period in which adjustments can be made is rather short. For this reason, the recommendations section includes a sub‐section of recommendations for a follow‐on project. The AMDe is aligned with the MoA’s, Agricultural Growth Program (AGP),2 funded by the Government of Ethiopia (GoE) together with the World Bank, Canada, Netherlands, Spain, and the Food and Agriculture Organization of the United Nations (FAO)3 and supported technically by the Agricultural Transformation Agency (ATA).4 The AMDe implements AGP’s Component 1, sub‐component 1.3: Market and Agribusiness Development. Key findings The MTE review findings are generic and specific. At the generic level, the Evaluation Team welcomes USAID’s engagement in Ethiopia’s high‐potential agriculture areas, including that this support is aligned with the MoA’s AGP. In this way, USAID is supporting government efforts to balance development investment across Ethiopia’s three main agro‐ecological areas: high rainfall areas, low rainfall areas, and the pastoral lowlands. The AGP is implemented in Ethiopia’s high rainfall areas, which are dependent on mono‐modal kiremt rains5 and a single meher growing season. Ethiopia has made substantial progress in recent years, with per capita income rising to USD 570,6 and 2.5 million people have been lifted out of poverty (using a poverty threshold of USD 1.25 per day). While these gains are impressive, Ethiopia remains one of the poorest countries in the world, and per capita incomes remain roughly half of the regional average USD 1,257. National per capita incomes are therefore in the ten lowest globally. Also, because of Ethiopia’s high population growth, the absolute number of poor remains unchanged over the past 15 years at around 25 million.7                                                              1 USAID Ethiopia’s FTF portfolio is valued at around USD 270 million over five years. It is implemented in 154 Zone of Influence woredas that are home to an estimated 17 million people. 2 The AGP is implemented in 96 woredas (AGP2 will be implemented in 157 woredas) and structured: Component 1—Agriculture production and commercialization, with sub‐components on institutional strengthening and development, scaling‐up good practice, and market and agribusiness development Component 2—Rural Infrastructure Development and Management, with sub‐components on small‐scale agriculture water management and market infrastructure development. The project is led by a Coordination Unit at federal and regional levels and is guided by Steering and Technical Committees. 3 The AGP is estimated to cost about USD 265 million, of which USAID provides 19 percent through AMDe and Livestock Marketing Development (LMD) projects. 4 The ATA—Agricultural Transformation Agency, www.ata.gov.et. 5 The single kiremt rains of June to September and the associated meher growing season. 6 International Monetary Fund 2014. 7 Poverty and Growth in Ethiopia (1995–2011), Ministry of Finance and Economic Development, 2013. ii The agriculture sector plays a central role in the Ethiopian economy—45 percent of total output and 80 percent of all exports—and the lives and livelihoods of an estimated 13 million smallholder farmer and pastoral households, or nearly 80 percent of the national workforce. Agriculture growth rates over a decade have been impressive,8 and the result is more land being farmed and increases in productivity per farmed area. Productivity increases are the result of public investment9—inputs, roads, agriculture extension—and public policy reform related to land tenure. Despite the progress made, more will need to be done to strengthen the capacity of Ethiopia’s extension services, build stronger links between research institutions and farmers, and improve access to inputs—seeds, fertilizers and agro‐chemicals for plant protection. There are also both infrastructure and policy‐related challenges in agriculture marketing. Finally, despite the increase in the amount of land under some form of irrigation,10 Ethiopian agriculture is primarily rain‐fed and therefore subject to the vagaries of the weather. The AMDe was launched in June 2011 and became fully operational in 2012. The AMDe has therefore been operational for the 2012, 2013, and 2014 meher seasons. Within these three seasons and therefore learning cycles, the Evaluation Team finds the AMDe has performed exceptionally well in some areas, less well in others, and has yet to address some issues that are of importance. Progress in each of these areas is presented in this Mid‐Term Evaluation. Overall, the Evaluation Team finds the AMDe has performed well and has the potential to achieve “high performance” over time. This is impressive, within the relatively short period of implementation and Ethiopia’s diverse and complex agro‐ecology, farming systems, and agriculture sector institutional and policy environment. The Evaluation Team finds it of strategic importance for USAID to fund a follow‐on phase. To accelerate learning, analysis, and documentation and identify evidence‐based good practice that can be taken to scale, the Evaluation Team finds that the FTF ZoI should be smaller and more focused and that AMDe interventions should be better supported by other USAID programs—social protection, WASH, nutrition, health, and education. Implemented as it is, the Evaluation Team finds it unlikely the AMDe will make a significant contribution to FTF’s poverty and stunting reduction objectives. The Evaluation Team also finds the AMDe could do much more to target smallholder farmers with holdings of one hectare or less and women in agriculture—both women‐headed households and women in male‐headed households—and in this way make an increased contribution to meeting FTF outcomes. Finally, in this overview section, the Evaluation Team finds that USAID could have exercised stronger leadership to contextualize FTF indicators to Ethiopia’s agriculture sector and agro‐ecology. This was not done, with the result that AMDe staff has wasted considerable time and resources monitoring and reporting on rather meaningless indicators, e.g., gross margins per hectare, value of incremental sales, and number of hectares under improved management. Importantly also, FTF does not require the monitoring of a control group, with the result that attribution is unclear.11 Moving to the specific, the Evaluation Team presents its findings under the questions in the SoW. Key Question 1: To what extent is the AMDe Project progressing against planned objectives as outlined in its performance management plan and work plan? As mentioned, the Evaluation Team finds some of the FTF indicators rather meaningless. For this reason, progress reported in this section should not be interpreted as a statement of AMDe's overall performance but rather progress against the current indicators.                                                              8 Official figures record agriculture growth rates of around 8 percent. 9 Ethiopia is investing an estimated 16 percent of its national budget in agriculture and has therefore exceeded the CAADP investment target of 10 percent. A considerable proportion of this investment is through the Productive Safety Net Programme. 10 Estimated by the MoA to be around two million hectares, or16 percent of total cultivated land although a considerable proportion is allocated to sugar. 11 USAID confirms that an end‐of‐project evaluation is planned. While this is welcomed, the Evaluation Team considers statistically significant impacts unlikely in the relatively modest period of AMDe implementation. iii Result 1: Improved Competitiveness of the Value Chains For Result 1, the AMDe tracks 13 indicators with progress recorded by value chain. The Evaluation Team finds that 21 percent of the value chain‐related targets have been fully achieved, 20 percent are very likely to be achieved, 14 percent are likely to be achieved with focused additional support, and 45 percent are thought unlikely to be achieved. For example:  The number of hectares under new technology (indicator #1) against LOP targets across all crop‐based value chains is low. This said, LOP targets are modest, e.g., the target for wheat is 12,000 ha, and these LOP targets may well be achieved through the cumulative effect of AMDe funding in seed, fertilizer, and equipment.  The numbers of farmers benefiting from access to new technology—seed, fertilizer, and training—(indicator #3) against LOP targets are variable. For example, the number of beneficiary farmers in the coffee and sesame value chains has been achieved; the number of farmers in the maize value chains is 57 percent of the LOP target and may be achieved; but it is unlikely the number of farmers for the wheat, chickpeas, and honey value chains will be met.  The increased value of improved productivity and production (indicator #4) appears to be progressing well, but contrasts with progress reported under indicator #1. As the two indicators are linked—the number of hectares under improved management contributes to increased productivity—the progress reported will need to be reviewed.  The value of incremental sales (indicator #6) is 96 percent of the LOP target and is likely to be exceeded.  The value of exports (indicator #10) is 62 percent of the LOP target and may be met. Result 2: Improved Access to Finance and Investment Specifically:  The value of agricultural and rural loans target (indicator #14) has been exceeded.  The number of clients benefitting from financial services (indicator #18) is 60 percent of the LOP target and may be met. Result 3: Improved Enabling Environment For example:  The seven policies/regulations/administrative procedures targets are progressing: Stage 1: Analyzed (seven targets met) Stage 2: Drafted and presented for public/stakeholder consultation (four targets met) Stage 3: Presented for legislation/decree (two targets met) Stage 4: Passed/approved (two targets met) Stage 5: Passed for which implementation has started (zero targets met) The Evaluation Team finds the process associated with the selection of the seven policy issues to be robust, including the engagement of multi‐sectoral platforms. However, the Evaluation Team is concerned that some of the selected policy targets appear to duplicate policy change to which the MoA is already committed through the Agriculture Sector Policy Investment Framework (2010–2020), while others appear overly optimistic. The Evaluation Team also finds that FTF policy work could have been better coordinated. Result 4: Stimulate Innovation and Investment For example:  The AMDe has disbursed USD 11.6 of the planned LOP target of USD 14.2 million (indicator #22) and is therefore well on track to meet the LOP target. Key Question 2: To what extent has this project contributed to gender equity in terms of access to credit, capacity‐building support, improved inputs and technologies resulting in an increase in sales of agricultural commodities? Are there evidences supporting positive changes in the aforementioned areas? iv The AMDe launched the Women in Agribusiness Leadership Network (WALN)12 in 2014 and in the same year launched a women membership drive with the FCA resulting in an additional 78,000 members (not all new). As recommended by the AMDe behavior change implementing partner, the membership drive was supported by small incentives. While respecting the value of this approach, the Evaluation Team finds the continued use of incentives to be unsustainable. The Evaluation Team however finds both of these interventions to be very positive. The Evaluation Team finds that USAID gave inadequate attention to gender in the AMDe design phase, as four of the original five value chains13 have a strong export focus. While the importance of increasing agriculture exports is recognized as a national priority, the Evaluation Team is reminded that FTF funding has clear poverty and under‐nutrition reduction development objectives and therefore that FTF implementing partners are required to address poverty reduction, including through gender equity and women empowerment approaches. The Evaluation Team finds that USAID might have given more consideration to the fact that the agriculture export sector is dominated by men and that increased emphasis could therefore have been given to value chain development for domestic markets and thus to women smallholders, traders, and agro‐processors. USAID also did not designate specific funds to support a strong gender equity component. Key Question 3: What has been AMDe’s contribution to the improvement of nutritional status of women and children? What is the evidence? Stunting levels in Ethiopia, including in AGP woredas, are among the highest in the world and, as mentioned, accelerated stunting reduction is one of two FTF development objectives. This said, nutrition was not part of the original design and nutrition targets were “bolted‐on” during AMDe’s first year of implementation. It is perhaps not surprising that progress on nutrition is mixed: on the one hand, AMDe has contributed to improved nutrition outcomes through the chickpea value chain development and launch of three chickpea shiro (a local sauce)products;14 on the other hand, the nutrition cascade training15 appears to have achieved little that will be sustained after the life of the project. Therefore, the Evaluation Team finds that agribusiness and market development projects offer few real opportunities to contribute to improved nutrition outcomes, as opportunities for nutrition‐sensitive agriculture are minimal unless they are an integral part of the design phase and linked to a well‐funded and supported agriculture sector production and productivity component. Key Question 4: What is the impact of the resources spent (financial and human) and performance per value chain? What is the relative value generated in terms of productivity and income increase, employment generation per value chain to resources spent? Which value chains have the highest returns per dollar spent? The AMDe monitoring appears weak and at times progress reports lack consistency and rigor. As some of the data sets lacked consistency and USAID had conducted its own “Cost Benefit Analysis” study across FTF value chains in 2014 that provided a performance ranking, the Evaluation Team did not address this question in detail.16 In order to address this question fully, the AMDe will need to complete a data cleaning and verification exercise.                                                              12 The network has received national recognition through the January 2015 National Conference that was attended by more than 100 women entrepreneurs. 13 USAID added the chickpea value chain in the first year of implementation. 14 Through GUTS Agro‐Industry. 15 The cascade training was supported with nutrition training posters, training manuals, a cookbook, and other materials, which have been used by other projects. 16 It may be that USAID’s Data Quality Assessment (DQA) team could help address the issue of data consistency. However, to address the issue of rigor and impact, mention has been made of the need to undertake more detailed studies that include control groups. v Key Question 5: Which among the following AMDe’s partnerships have been the most effective in terms of their collaboration and coordination to implement AGP‐AMDe? Which ones have been the least effective? In both cases, what is contributing to these partnerships’ success and challenges? The AMDe works with an impressive range of stakeholders: MoA/regional Bureaus of Agriculture, Ministry of Trade (MoT), ATA, Federal and Regional Cooperative Agencies (FCA/RCA), Ethiopian Institute of Agriculture Research (EIAR) and Industry Associations.17 The AMDe also works effectively with private sector organizations, international and local NGOs, and other FTF implementing partners. The Evaluation Team finds that the AMDe’s strongest partnerships are with the MoA/regional Bureau of Agriculutre BoA AGP, ATA, FCA/RCA, and MoT Export Promotion Directorate with which it shares a common vision for agribusiness development and the associated growth of cooperatives. At an operational level, the AMDe annual work plans are reviewed and endorsed by regional and federal AGP Technical Committees. In addition, USAID, ATA, and AMDe have forged a tripartite coordination plan with assigned activity focal points. Although AMDe is developing effective partnerships, the Evaluation Team finds that it will take time before USAID and AMDe play a full and central role in AGP decision‐making. In part, this can be attributed to organizational differences—donor‐government, culture, and salary and related incentive structures. The Evaluation Team also recognizes challenges associated with USAID’s parallel funding arrangement to the pooled AGP fund. While recognizing the challenges, the Evaluation Team also finds that parallel funding offers opportunities for innovation and the identification of emerging good practice that can inform AGP planning and operations. This has already happened within the context of AMDe and other FTF projects.18 Ethiopia’s cooperative movement aggregates and markets less than 10 percent of Ethiopia’s cereals,19 with small‐scale private sector traders playing the major role in cereal aggregation, transport, marketing, and processing. While the AMDe works well with industry associations and some 500 private sector actors, commercial farmers, service providers, traders, and processors, the Evaluation Team finds that, in order to address long‐term issues of sustainability, the AMDe will need to achieve a better balance of support between cooperatives and private sector actors, with a particular emphasis on small‐ and medium‐sized businesses.20 The Evaluation Team finds the AMDe works well with other FTF implementing partners at federal and regional levels, but as the AMDe deploys few field staff at woreda level, the quality of partnership inevitably tails off at woreda and community level. The Evaluation Team however finds that USAID could have done more to forge a coordinated and better‐integrated FTF portfolio, including that where possible USAID implementing partners support the same communities, and the opportunity for synergy is maximized. Key Question 6: Which of the components and/or project activities can easily be scaled up in the future based on measurable, practical, and sustainable results? The AMDe is well established in the four AGP regions and across the six value chains. In view of the AMDe’s strong marketing focus, the Evaluation Team also finds that the project is doing important and strategic work to address productivity issues through its improved seeds, blended fertilizers, and crop protection work that is supportive of other AGP sub‐components. The Evaluation Team finds AMDe's training and capacity‐building work on post‐harvest management (threshing, winnowing, cleaning, storage, and pest control), aggregation, and the marketing of agricultural surpluses to be of primary importance, in particular where the                                                              17 Ethiopia Apiculture Board, Ethiopia Honey and Beeswax Producers and Exporters Association, Ethiopia Pulses Oil Seeds and Spices Processers and Exporters Association, and the Ethiopia Coffee Exporters Association. 18 Engine—mainstreaming approaches in nutrition and GRAD—mainstreaming village savings and loans approaches. 19Cereals account for more than 60 percent of all arable cropping in Ethiopia. 20 The AMDe points out that cooperative membership can help smallholders with access to inputs, information, training, and finance and that the government is committed to increase the effectiveness and marketing and capacity of cooperatives. The AMDe also notes its M4 Capacity Assessment carried out in summer 2014 documented increased capacity in half the cooperative unions. vi beneficiaries are poorer smallholder farmers—with less than one ha—and therefore planned FTF beneficiaries. The Evaluation Team is also informed by lessons learned by the AGP that one‐ off, short‐term training has limited impact.21 Learning from this, the AMDe should review its training and capacity‐building work and channel resources to fewer PCs, FCUs,22 and private sector partners in order that all trainings can be supported by follow‐up visits, impact assessments, and, as appropriate, refresher training. The Evaluation Team learned that cereal, honey, and some pulse prices are typically higher in Ethiopia than on the international markets.23 While recognizing that increasing agriculture exports is a national priority, the Evaluation Team is concerned that AMDe’s focus on exports may well ultimately benefit larger commercial farmers more than the smaller and poorer farmers who are FTF primary planned beneficiaries. The Evaluation Team therefore finds that additional consideration could be given to developing value chains for domestic markets, which are more accessible to poorer smallholders and PCUs, as they obviate the need for achieving export standard production, aggregation, cleaning, and marketing. The Evaluation Team finds that AMDe business management training support for cooperatives to be effective, in particular where it is integrated into broader capacity‐building work. The Evaluation Team also finds this work could be mainstreamed through the Ardaita Cooperative ATVET,24 Oromia Region, which provides similar training and capacity‐building support to PCs and FCUs in that part of Oromia. By partnering with Ardaita—including as appropriate the AMDe investing further in capacity building of Ardaita itself—the AMDe can minimize duplication and address sustainability issues.25 The Evaluation Team finds that AMDe’s grant work is widely appreciated. Grants have has supported the construction of Ethiopia’s first blended fertilizer plant, warehouses, and improved input supply. The Evaluation Team finds that the grant work should be continued, with an increased focus on small‐ and medium‐sized businesses. The Evaluation Team has learned that the AMDe support to FCUs for warehouse construction has been replicated and that it is proposed that the AGP2 will construct 135 additional warehouses, storage, and grading facilities.26 The Evaluation Team finds that the AMDe should complete its existing commitments and thereafter disengage from funding warehouse construction. The Evaluation Team recognizes the AMDe’s support to cooperatives to improve credit ratings and secure loans from banks and Rural Savings and Credit Cooperatives (RuSACCOs). The AMDe is supporting World Food Programme’s (WFP) P4P initiative through the delivery of 32,898 mt of maize from 13 FCUs. The Evaluation Team finds this a positive intervention, as maize farm gate prices are volatile. To address the issues of volatility sustainably, however, the Evaluation Team finds that AMDe's maize marketing support for small‐ and medium‐sized private sector traders should be continued. Key Question 7: Given the findings above, does the Project have the right balance of staff and funding? Is there an appropriate balance between the resources (staff and budget), their management, and the activities the project intends to accomplish? The AMDe employs 81 full‐time staff and 19 part‐time or pooled staff (64 men and 36 women). The pooled staff includes the Deputy Chief of Party Operations and senior finance, HR, administration, and support staff who are shared with other ACDI/VOCA projects (AMDe funds 70 percent of these staff salaries). The Evaluation Team finds the relationship between AMDe and some of the pooled staff is unclear and would benefit from clarification.                                                              21 World Bank, 2015. 22 Perhaps as few as 50 FCUs. 23 The price of wheat, for example, in Ethiopia at the time of the MTE was nearly twice world prices. The domestic price of honey was also considerably higher than the price on the world market. 24 The Cooperative Sector Development Strategy (FCA, 2012), developed with the support of ATA, identifies Ardaita as a potential “center of excellence” for cooperative development. While not centrally located, it is planned that the center will establish branches in other regions and operate as a “college without walls.” 25 During the Evaluation Team’s visit to Ardaita, it was learned that the AMDe had visited once but without follow‐up. 26 World Bank, 2015. vii Almost half of the AMDe's staff is based in Addis Ababa.27 In order to consolidate progress and ensure sustainability, increased attention needs to be given to staff AMDe's work in the regions. Recommendations Recommendation for Result 1 #1: The AMDe continues to work on the six value chains in order not to disrupt project implementation during the remainder of the project, but gives increased focus to incomplete Results:  Wheat—productivity and value addition through processing  Maize—productivity, including open pollinated varieties, improved post‐harvest handling, cleaning, storage, and diversifying domestic market outlets  Coffee—post‐harvest loss reduction and quality standards  Sesame—post‐harvest loss reduction and processing  Chickpeas—domestic and niche markets (Sudan, for example) for the currently used varieties  Honey—production and productivity through the distribution of improved equipment, including modern beehives and training and equipping of beekeepers and new beekeepers Recommendation for Result 2 #2: Continue and strengthen AMDe's engagement in agriculture sector finance, in particular capacity building for cooperatives and small‐scale private businesses to develop business plans,28 negotiate with banks, and improve accounting and auditing systems, but with a focus on fewer partners. Recommendation for Result 3 #3: Recognize the need to work more closely with other FTF implementing partners including the AKLDP.29 With FTF partners, forge a common, integrated, and coordinated approach to agriculture sector policy work. Recommendation for Result 4 #4: Reduce the focus on grants for new partners and consolidate the capacity of existing partners to improve impact and sustainability. Recommendations for gender equity #5: Continue to support women empowerment, including women membership of cooperatives— though discontinue the use of incentives—and WALN, with increased emphasis on women‐led family agribusinesses in the regions. Set aside funding for gender equity work. #6: Document AMDe’s work with women in Tigray and SNNP Regions and share with FTF projects within and beyond Ethiopia. Recommendation for nutrition #7: Hand over the nutrition work to ENGINE. Recommendations for improved collaboration #8: Continue to strengthen partnerships with key stakeholders, in particular regional BoA AGP teams, ATA Agriculture Commercialization Clusters, and Regional Cooperative and Industry Association Offices to accelerate training and capacity building of regional, zonal, and woreda                                                              27 This figure includes AMDe’s Oromia Region staff who are based in Addis, as their regional counterparts in the Oromia Bureau of Agriculture are based in Addis. 28 The business plans prepared jointly by FCUs and AMDe are of a high quality. For example, the Sidama Elto FCU business plan secured loans of Eth Birr 7.6 million in 2013, Eth Birr 14.4 million in 2014, and Eth Birr 6.35 million in 2015. 29 The AKLDP project provides agriculture knowledge, learning, and policy support to USAID’s FTF portfolio. viii staff in agriculture marketing that includes components on the private sector and its potential role in transforming Ethiopian agriculture. Ensure that all training and capacity building is focused and an integral part of a holistic capacity‐building approach. #9: Collaborate with ATA and Federal Cooperative Agency (FCA) to develop a single PC/FCU Certification system that integrates and harmonizes accreditation criteria. Recommendations for scaling up #10: Support the MoA and ATA to improve productivity through the improved distribution of certified seeds (with a minimum 60 percent open pollinated varieties), blended fertilizer, credit, and extension support. #11: Continue to support the WFP‐P4P initiative but continue to strengthen and develop links between smallholder farmer groups and responsible private sector traders. Recommendations for management #12: Recruit an organizational change expert to reorientate AMDe staffing, specifically increased support for: farmer level production and productivity;30 domestic market development; and gender. Clarify the responsibilities of the AMDe and ACDI/VOCA pooled staff and as appropriate move staff to the regions. #13: Recruit an M&E specialist to improve AMDe M&E capacity. Ensure all project data are cleaned and that increased consideration is given to project outcomes and attribution, ahead of the final evaluation. #14: Strengthen AMDe’s capacity development work to 15 percent of the Year 5 project budget to build the capacity of priority PCs and FCUs and better equip them to continue work started with AMDe. To support this reorientation, complete all warehouse construction‐related commitments and subsequently disengage from warehouse construction. #15: Include representatives from the MoA and possibly the MoT and MoI in the final evaluation. General recommendations for a follow‐on project The Evaluation Team recommends that USAID provide follow‐on funding for a minimum of another five years and therefore welcomes USAID’s commitment to fund AGP Component 4: Agriculture Marketing and Value Chains, sub‐component d—strengthening of selected livestock and crop value chains. While welcoming this commitment, the Evaluation Team recommends that USAID support for AGP be more focused to achieve FTF development objectives; specifically that a follow‐on program operate in a smaller ZoI and with fewer partners. Within these ZoI, it is recommended that, in addition to continuing to support agriculture marketing, significant increased emphasis is given to: smallholders with holdings of less than one hectare— perhaps 70 percent of planned project beneficiaries; sustainable agriculture production and productivity, including soil health, land management, and land tenure; women in agriculture— women‐headed and women in male‐headed households; and the private sector, in particular small‐ and medium‐sized businesses. In addition, the Evaluation Team recommends that USAID provide better layered and sequenced support within its ZoI, including social protection, WASH, nutrition, health, and education, and that, through this integrated and coordinated approach, USAID play a lead role in poverty and stunting reduction that will be replicated by others. The Evaluation Team finds that USAID should exercise increased leadership to contextualize global FTF indicators and guide FTF implementing partners to key indicators that will track progress to address Ethiopia’s poverty and stunting challenges. Specific recommendations for a follow‐on project include: #16: Re‐orientate the value chains: hand off coffee and wheat to the industry associations and other development partners, including as appropriate the EU, a USAID‐funded innovation lab                                                              30 The Evaluation Team is keen to see that the USAID Ethiopia Mission strengthens its support of production and productivity and therefore achieves a more balanced agriculture sector portfolio. The Evaluation Team is of the view that the Mission is at present too market‐focused. ix project holder, and ATA; build a more integrated approach to cereal value chains that includes maize,31 malt barley, and sorghum—grown by large numbers of poorer smallholders; expand the honey value chain—assisting larger numbers of poor women with little or no land; and integrate chickpeas into a legumes/pulses value chain for domestic markets—legumes offer important soil health and nutrition outcomes. USAID could therefore support the following value chains:  Cereals— maize, malt barley, and sorghum  Honey—for domestic markets (with an emphasis on women)  Legumes—for domestic markets #17: Ensure that capacity building, institutional development, and gender are key components of a follow‐on project, are appropriately funded, and are delivered within a holistic capacity‐building approach. #18: Reduce managerial complexity—sub‐contract the productivity, value chain development, aggregation, and domestic market development components to specialist NGOs with a proven track record in Ethiopia. The project holder can then better focus on management, coordination, monitoring (data collection, collation, analysis, documentation), learning and championing evidence‐based good practice, including with AGP2. The follow‐on project would be expected to leave Ethiopian institutes, organizations, and businesses better equipped to lead the transformation process of Ethiopian agriculture.                                                              31 With a focus on open pollinated varieties. 1 1. Introduction 1.1 Ethiopia’s agriculture sector Ethiopia has made substantial progress in recent years, with per capita income rising to USD 570.1 Two and a half million people have been lifted out of poverty (using a poverty line of USD 1.25 per day). While this progress is impressive, Ethiopia remains one of the poorest countries in world; per capita incomes are roughly half the regional average of USD 1,257 and amongst the ten lowest internationally. As a result of Ethiopia’s high population growth, the absolute number of poor remains unchanged over the past 15 years at about 25 million2 despite concerted efforts to reduce this number. The agriculture sector continues to play a central role in the Ethiopian economy—45 percent of total output and 80 percent of all exports3—and the lives and livelihoods of an estimated 13 million smallholder farmer and pastoral households, which is nearly 80 percent of all employment.4 Ethiopian agriculture can be characterized under three broad rainfall regimes: high rainfall, moderate rainfall, and the pastoral lowlands. The AGP woredas are concentrated in the high rainfall belt of western Ethiopia and are primarily dependent on the kiremt rains of June to September and the associated single meher growing season. The principal crops include: cereals, pulses, oilseeds, coffee, roots, and tubers—potatoes, sweet potatoes, cassava, other vegetables, and sugarcane. Growth rates of all crops have been impressive over the last decade, due to a combination of more land being farmed5 and increases in productivity per farmed area. Productivity increases are primarily the result of public investment, which is 16 percent of the national budget and considerably more than the CAADP‐recommended 10 percent bench‐ mark.6 Investment has resulted in improved access to inputs, rural roads, expanded agriculture extension services, and policy reform, including certification of land holdings. Despite the impressive level of investment, more needs to be done to increase the capacity of the extension services, including to work across different agro‐ecological zones and to strengthen the link between research institutions and farmers.7 Progress also needs to be made towards improving smallholder farmer access to inputs—seeds,8 fertilizers, and agro‐ chemicals for plant protection—including opening up new distribution networks through small agribusiness dealers that will address the current over‐dependence on cooperatives. It is also widely recognized that there are challenges to marketing that relate to poor market infrastructure—aggregation and storage and market centers—and trade policy issues related to both domestic and export markets. Finally in this section, despite the increase in the amount of land under some form of irrigation,9 Ethiopian agriculture is primarily rain‐fed and therefore subject to the vagaries of the weather. 1.2 Feed the Future and AMDe in Context Informed by the Growth and Transformation Plan, 2010–2015 (GTP), 10 the MoA launched the Policy and Investment Framework (PIF) 2010–2020.11 The Agriculture Growth Program (AGP) was launched as a “flagship” project in 2010 to increase agricultural productivity and market performance of selected crop and livestock value chains in 96 high‐growth woredas in Amhara,                                                              1 International Monetary Fund 2014. 2 Poverty and Growth in Ethiopia (1995–2011), Ministry of Finance and Economic Development, 2013. 3 Agriculture exports include coffee, oilseeds, some pulses, and livestock. 4 AGP Project Appraisal Document, September 2010, World Bank. 5 Large areas of communal grazing have been put to the plough in recent years. 6 The PSNP is, however, the largest‐funded government program, and most of the investment is in the form of food and cash payments. 7 Some universities now require MSc students to undertake action research with smallholders in the vicinity. 8 In particular, certified seed of the crops and varieties that they prefer and are appropriate for their holdings and farming systems. 9 Estimated by the Ministry of Agriculture to be around two million hectares or 16 percent of total cultivated land (although a considerable proportion is allocated to sugar). 10 Growth and Transformation Plan, 2010/11–2014/15, Federal Democratic Republic of Ethiopia. 11 Agriculture Sector Policy Investment Framework, 2010–2020, Ministry of Agriculture. 2 Oromia, SNNP, and Tigray Regions. The AGP is funded by the GoE, World Bank, and bilateral development partners—Canada, Netherlands, Spain, and FAO. The AGP is also technically supported by the ATA. The US Government’s FTF program in Ethiopia is valued at USD 270 million.12 FTF’s development objective is to increase economic growth with resiliency in rural areas, specifically to reduce sustainably poverty13 and hunger14 through improvements in food availability, access, and utilization. A FTF implementing partner, the AMDe is aligned with the AGP and supports the implementation of AGP sub‐component 1.3: Market and Agribusiness Development. The AMDe was funded in June 2011 for a period of five years to July 2016, with a budget of USD 50 million. It was planned that the AMDe benefit a total of one million farmers. The AMDe development goal is: sustainably reduce poverty and hunger by improving the productivity and competiveness of agricultural value chains that offer jobs and income activities for rural households. 15 AMDe's work is structured around improving the competitiveness of six value chains: maize, wheat, sesame, coffee, honey, and chickpeas16 through technical and managerial assistance, increased access to finance, and private sector investment. AMDe partners with 2,554 PCs and 251 FCUs, with a total membership of 1.9 million. The project design included a 12‐month Inception Phase during which time ACDI/VOCA17 hired staff, established offices in each of the AGP regions, refined the implementation strategy, and developed detailed work plans. The AMDe submitted its first annual work plan in November 2011. The result of an iterative process, the first work plan was approved by USAID in May 2012. Follow‐on plans were submitted and accepted without revision in October 2012, July 2013, and July 2014. The AMDe has therefore been operational for three agriculture cycles and is about to enter the fourth. 1.3 The Mid‐Term Evaluation This Mid‐Term Evaluation (MTE) is delayed well past the mid‐term point. The draft report was submitted in March 2015 at a time when the AMDe had 15 months of implementation remaining.18 While therefore the primary purpose of this MTE is to assess project performance and recommend adjustments during the life of the project, the SoW (see Annex 1) also requires the Evaluation Team to offer recommendations to inform the design of a follow‐on project. The full list of questions includes the following: 1. To what extent is the AMDe project progressing against planned objectives as outlined in its performance management plan (PMP) and work plan? 2. To what extent has this project contributed to gender equity in terms of access to credit, capacity‐building support, improved inputs, and technologies, resulting in an increase in sales of agricultural commodities? Are there evidences supporting positive changes in these areas? 3. What has been AMDe’s contribution to the improvement of nutritional status of women and children? 4. What is the impact of the resources spent and performance per value chain? What is the relative value generated in terms of productivity and income increase, employment generation per value chain to resources spent? Which value chains have the highest returns per dollar spent? 5. Which among the AMDe’s partnerships have been the most effective in terms of their collaboration and coordination to implement AGP‐AMDe? Which have been the least                                                              12 Eighty‐five percent of the investment is, however, in five main projects: two implemented in AGP woredas, one in pastoral areas, one in PSNP woredas, and one a specialist nutrition project. 13 To reduce household poverty by 30 percent in the Zone of Influence within five years. 14 To reduce stunting by 20 percent in the Zone of Influence within five years. 15 Task Order AID 663 TO 11‐00003. 16 This was added in Year One of implementation by USAID as PepsiCo expressed interest in purchasing chickpeas from Ethiopia. This in fact did not happen. 17 The PRIME implementing agency. 18 The project will close out in May 2016. 3 effective? In both cases, what is contributing to these partnerships’ success and challenges? 6. Which of the components and/or project activities can easily be scaled up in the future based on measurable, practical, and sustainable results? 7. Given the findings to these questions, does the Project have the right balance of staff and funding? Is there an appropriate balance between staff and budget, their management, and the activities the project intends to accomplish? 1.4 Methodology The Evaluation Team comprised a Team Leader and three local consultants19 (see Annex 2— Team CVs) and carried out the MTE in two phases, mid‐November to mid‐December 2014 and January 2015. The team was briefed by the AKLDP team and USAID in the first week of the mission. The Evaluation Team’s work plan, also presented for discussion in these meetings, was approved by USAID (see Annex 4). The Evaluation Team reviewed key documents, including the project proposal and monitoring and other major reports (see Annex 5 – Bibliography). The Evaluation Team also visited 26 woredas in AGP regions, interviewed 215 informants from 66 organizations including farmer organizations, primary cooperatives, farmer cooperative unions, partner organizations, industry associations, and the private sector (see Annex 6). Interviews were structured using a guideline developed by the Evaluation Team (see Annex 7). Limitations Despite its efforts—the review of a large amount of project information and interviewing more than 200 informants—the Evaluation Team recognizes its limitations. For example, time did not allow for a detailed comparative study between beneficiary and non‐beneficiary households, and therefore issues of impact and attribution have not been adequately addressed. Time constraints also resulted in bias to stakeholders and activities that were accessible by all‐ weather roads. The shortcomings aside, the Evaluation Team is confident that the findings and recommendation are useful both in the short term and the design of a follow‐on project. 2. Evaluation Findings The MTE review findings are generic and specific. At the generic level, the Evaluation Team welcomes USAID’s engagement in Ethiopia’s high potential agriculture areas, including that this support is aligned with the MoA’s AGP. In this way, USAID is supporting government efforts to balance development investment across Ethiopia’s three main agro‐ecological areas: high rainfall areas, low rainfall areas, and the pastoral lowlands. The AGP is implemented in Ethiopia’s high rainfall areas, which are dependent on mono‐modal kiremt rains20 and a single meher growing season. The AMDe was launched in June 2011 and became fully operational in 2012. The AMDe has therefore been operational for the 2012, 2013, and 2014 meher seasons. Within these three seasons and therefore learning cycles, the Evaluation Team finds the AMDe has performed exceptionally well in some areas, less well in others, and has yet to address some issues that are of importance. Progress in each of these areas is presented in this Mid‐Term Evaluation. Overall, the Evaluation Team finds the AMDe has performed well and has the potential to achieve “high performance” over time. This is impressive, within the relatively short period of implementation and Ethiopia’s diverse and complex agro‐ecology, farming systems, and agriculture sector institutional and policy environment. The Evaluation Team finds it of strategic importance for USAID to fund a follow‐on phase. To accelerate learning, analysis, and documentation and identify evidence‐based good practice that can be taken to scale, the Evaluation Team finds that the FTF ZoI should be smaller and more focused and that AMDe interventions should be better supported by other USAID programs—social protection, WASH, nutrition, health, and education. Implemented as it is, the                                                              19 John Fox, together with Dr. Nigussie Alemayehu, Dr. Amare Ghizaw, and Dr. Amdissa Teshome. 20 The single kiremt rains of June to September and the associated meher growing season. 4 Evaluation Team finds it unlikely the AMDe will make a significant contribution to FTF’s poverty and stunting reduction objectives. The Evaluation Team also finds the AMDe could do much more to target smallholder farmers with holdings of one hectare or less and women in agriculture—both women‐headed households and women in male‐headed households—and in this way make an increased contribution to meeting FTF outcomes. Finally, in this overview section, the Evaluation Team finds the USAID could have exercised stronger leadership to contextualize FTF indicators to Ethiopia’s agriculture sector and agro‐ecology. This was not done, with the result that AMDe staff has wasted considerable time and resources monitoring and reporting on rather meaningless indicators, e.g., gross margins per hectare, value of incremental sales, and number of hectares under improved management. Importantly also, FTF does not require the monitoring of a control group, with the result that attribution is unclear.21 Moving to the specific, the Evaluation Team presents its findings under the questions in the SoW. 2.1 Progress made under each of the four major results Question 1: To what extent is the AMDe project progressing against planned objectives as outlined in its performance management plan (PMP) and work plan? At times, the Evaluation Team found it difficult to confirm progress as the quality of the project data appears to be inconsistent. The Evaluation Team therefore presents “best estimates.” Result 1: Improved Competitiveness of the Value Chains For Result 1, progress is summarized in Table 3: Progress against indicators and LOP targets for Result 1. Progress is color coded as follows:  Blue—fully achieved or 100 percent of the LOP target  Green—very likely to be achieved or 80–100% percent of the LOP target  Yellow—likely to be achieved but requiring additional focus or 60–79 percent of the LOP target  Red—unlikely to be achieved or <60 percent of the LOP target  Uncolored—not assessed Under Result 1, the AMDe tracks 13 indicators and associated value chain targets. The Evaluation Team finds that AMDe has fully achieved 21 percent, an additional 20 percent are very likely to be achieved, 14 percent are likely to be achieved with focused additional support, and that it is thought that 45 percent are unlikely to be achieved. As has been mentioned, however, the Evaluation Team finds some of the FTF indicators rather meaningless, and therefore this section of the report should not be interpreted as a statement of AMDe performance. Indicator #1: Number of hectares under improved technologies or management practices as a result of USG assistance At the end of the 2014 growing season, progress against LOP targets is low across all value chains: 7 percent of the 12,000 ha target for wheat, 1 percent of the 15,670 ha for maize, and 11, 20, and 36 percent for coffee, sesame, and chickpeas respectively. AMDe staff, however, suggests they are confident of meeting, and in some cases exceeding, the LOP targets, as the targets are modest and AMDe systems—access to seeds (maize—BH66122 and sesame—Humera‐1 and Setit‐1), fertilizer, and equipment—are in place as a result of its grants facility and AMDe’s                                                              21 USAID confirms that an end‐of‐project evaluation is planned. While this is welcomed, the Evaluation Team considers statistically significant impacts unlikely in the relatively modest period of AMDe implementation. 22The maize BH661 hybrid is being popularized through the Advance Maize Seed Adoption Program (AMSAP) partnership of which AMDe is a member. While recognizing the value of increasing yields, the Evaluation Team is concerned the AMDe is being unwittingly pulled into a narrow focus on yields that is potentially unsustainable, as open pollinated varieties are often better suited to the needs of poorer smallholder farmers in the more marginal areas of the AGP’s four main regions. 5 support to farmer training, field demonstrations, farmer‐to‐farmer experience sharing forums, and exchange visits. AMDe is confident that this will result in a rapid scaling‐up of progress in the current growing season. The Evaluation Team, however, wishes to draw attention to the character of Ethiopian agriculture: 13 million smallholder farming families, each with two, three, or more scattered plots that typically total less than a hectare. Smallholders also prize cropping diversity to meet household food security needs and (understandably perhaps) many are resistant to standardized extension packages that are invariably designed for model farmers with larger holdings and for higher rainfall areas. The Evaluation Team therefore finds that this indicator and associated LOP targets should be better contextualized. Indicator #3: Number of beneficiaries supported by AMDe‐assisted value chains The LOP targets for sesame and coffee have been surpassed: the target of 74,980 beneficiaries for sesame was exceeded by 47,752 or 164 percent and the target of 110,515 beneficiaries for coffee exceeded by 17,963 or 116 percent. Progress across other value chains has been slower, and it may be that the LOP targets will be achieved: 23,130 of the planned 286,951 beneficiaries for wheat or 8 percent; 167,290 of the planned 295,404 beneficiaries for maize or 57 percent; 30,704 of the planned 143,974 beneficiaries for chickpeas or 21 percent; and finally 19,755 of the planned 88,282 beneficiaries for honey or 22 percent.23 AMDe staff suggest the targets are achievable for reasons already outlined. Indicator #4: Average yield per hectare/hive (kg) The LOP targets for increased yields per hectare for wheat and coffee have been achieved at 110 and 104 percent respectively, and the LOP targets for maize, sesame, and honey are likely to be achieved with progress at 95, 84, and 80 percent of LOP targets respectively. The LOP target for chickpeas is unlikely to be achieved, as progress is 44 percent of the LOP target. Again, the AMDe staff is confident targets will be reached and surpassed for the reasons outlined. Table 1: Average yield increases to December 2014 Value chain LOP yield targets (kg/ha) Achieved increased yields (kg/ha) Achieved increased yields (%) Wheat 2,969 3,266 110 Maize 3,703 3,517 95 Coffee 976 1,015 104 Sesame 800 672 84 Chickpeas 3,305 1,454 44 Honey 15 12 80 Source: Abstracted from project PMP reports While recognizing the progress made, the Evaluation Team is concerned that average yield increases presented in Table 1 are at variance with the number of hectares under improved management (Indicator #1). For example, the reported progress for wheat under improved management is 7 percent of the 12,000 ha LOP target, yet despite this and the known impact of wheat rust in 2014, the AMDe reports a 110 percent increase in yield. In parenthesis, such anomalies are found across AMDe’s progress reports. Indicator #5: Gross margin per unit of land/hive (USD) The target LOP gross margins for wheat and sesame have been achieved at 101 and 186 percent respectively, and progress against LOP targets for maize and chickpeas are currently 86 and 85 percent respectively. Gross margin progress for coffee is more modest at 66 percent of the LOP target despite the progress made to address production constraints. The Evaluation Team,                                                              23 There are an estimated 1.5 million households keeping bees in Ethiopia. 6 however, finds the AMDe reporting on gross margins misleading, as the reports do not address attribution.24 It is therefore not clear to what extent AMDe played a central role in improving gross margins or to what extent the improvements were the result of other projects or simply the result of good rainfall or Ethiopia’s expanding road network or improvements in input supply. To illustrate the problem of measuring gross margins, the Evaluation Team learned the Gozamen FCU, Debre‐Markos is holding several thousands of quintals of maize that were purchased at prices calculated on the basis of production. Prices are now well below the cost of production, and the FCU is understandably reluctant to purchase more maize. As a result, cooperative members were forced to sell maize to traders at the costs of production or even at a loss. According to when the gross margin indicator is measured, specifically before or after price collapses, the FTF implementing partner can be seen to be either performing well or much less well. At a minimum, the Evaluation Team finds that gross margins performance data should be updated monthly in order to more accurately reflect the complexity of the commodity markets.25 Indicator #6: Value of incremental sales at farm‐level attributed to FTF implementation Coffee sales have reached USD 32 million or 127 percent of planned LOP target. The LOP targets for incremental sales of maize and sesame appear achievable, as progress is respectively 87 and 80 percent. In contrast, the targets for wheat and chickpeas are unlikely to be achieved, as progress against the LOP targets is respectively 20 and 19 percent. See Table 2 below. Here again, the Evaluation Team finds this indicator inadequate, as the value of incremental sales is primarily informed by fluctuating commodity prices. Accordingly, the timing of the data collection can play a significant role in showing progress against this indicator in a positive or negative light. The value of incremental sales therefore appears to be another relatively meaningless indicator. Table 2: Value of incremental sales attributed to FTF interventions Value chain LOP targets— increased sales (USD) Actual increased sales (USD) Actual increased sales (% of LOP target) Wheat 9,120,344 1,824,068 20 Maize 12,449,951 10,831,457 87 Coffee 25,274,365 32,098,443 127 Sesame 37,544,955 30,035,964 80 Chickpeas 8,403,233 1,596,614 19 Honey 1,904,833 1,276,238 67 Total 94,697,681 77,662,784 82 Indicator #10: Value of exports of targeted agricultural commodities as a result of USG assistance The LOP target for coffee has been surpassed at 120 percent of the target or value of exports worth USD 120 million. The LOP export target for sesame was USD 74 million, and exports to date are USD 46 million or 63 percent. The target might therefore be achieved. In contrast, it would seem the LOP targets for chickpeas26 and honey27 will not be met. Despite the limited progress made under this indicator in the honey value chain, the Ethiopia Beekeepers Association has, with the support of its donors, including AMDe, assisted Ethiopia in securing International Organization for Standardization accreditation to export honey to niche markets                                                              24 In general, the Evaluation Team finds that the AMDe fails to give adequate attention to attribution. 25 Coffee farm gate prices have, for example, fluctuated by as much as 50 percent in just two years. At the time of the MTE, domestic wheat prices were roughly twice international prices. 26Ethiopia does not yet produce export‐quality chickpeas for export to the Middle East. 27 The domestic price for honey is higher than export prices. 7 in Europe, including the Fair Trade outlets. For more information on AMDe support to Trade Associations, see Text Box 1. The LOP targets for this indicator are informed by an export market assessment carried out by USAID and the ATA. It would appear the assessment did not adequately address domestic and international quality and price issues that inevitably impact on performance. Bearing in mind the focus of FTF development objectives, the Evaluation Team finds that the AMDe might have performed better if it had been tasked to develop value chains for domestic markets that not only are more compatible with a pro‐poor focus, but are also less risky. Indicator #12: Volume of processed products (mt) Progress against planned LOP targets for the volume of processed products is as follows: honey—79 percent of the planned 1,916 mt; coffee—57 percent of the planned 29,627 mt; and maize—52 percent of the planned 19,265 mt. Progress for wheat and sesame are 20 percent of their respective targets and chickpeas only 3 percent. It is likely the LOP target for honey, coffee, and maize will be met, as good progress was made in Year 3 when annual targets were exceeded by 287, 224, and 121 percent respectively. The LOP targets for wheat, sesame, and chickpeas are unlikely to be met. Indicator #13: Percentage decrease of post‐harvest losses as a result of AMDe intervention The planned reductions in post‐harvest losses: 14 to 9 percent for wheat; 23 to 7 percent for maize; 25 to 10 percent for coffee; 10 to 5 percent for sesame; and finally 20 to 3 percent for chickpeas. To date, reported progress is impressive: wheat losses to 8 percent, maize losses to 11 percent and chickpeas losses to 6 percent. In contrast, progress in sesame and coffee are more modest, losses down by only 20 and 7 percent of planned targets. The Evaluation Team, however, learned from lead researchers that post‐harvest loss data in Ethiopia is probably overstated and that losses are more modest. If losses are in fact more modest, it will be necessary to revisit the AMDe baseline in order for the progress reported by AMDe to be verified. Text Box 1: Support to trade associations AMDe is providing support to a number of industry associations: Coffee Exporters Association; Coffee Producers and Exporters Association; Beekeepers Association; Honey and Beeswax Producers and Exporters Association; Ethiopia Apiculture Board; and Ethiopian Pulses, Oilseeds and Spices Processers and Exporters Association. AMDe's primary support is capacity building and technical support. AMDe also supports its partners to attend international trade fairs. AMDe, together with Oxfam, SNV, and SOS Sahel assisted the Ethiopia Apiculture Board (EAB) in hosting the Third Api‐Africa International Conference that was attended by international experts and beekeepers from all over Africa, including Ethiopia. AMDe also assisted the Ethiopia Beekeepers Association (EBA) to secure EU accreditation to export honey to Europe, as it appears that there is an expanding market for Ethiopian honey in Europe. 8 Table 3: Progress against indicators and LOP targets for Result 1 Indicator Value chain LOP target Cumulative (Years 1–3) Year 4 target % Target Result % Achievable Result %LOP IR 1: Improved competitiveness of selected VCs 1 Number of hectares under improved technologies or management practices as a result of USG assistance Wheat 12,000 6,017 846 14 7% 7,501 70% Maize 15,670 8,527 106 1 1% 9,984 64% Coffee 3,200 900 362 40 11% 1,380 54% Sesame 3,788 2,252 762.5 34 20% 2,075 75% Chickpeas 21,000 15,822 7,554 48 36% 15,600 110% 3 Number of beneficiaries supported by AGP‐AMDe‐assisted value chains Wheat 286,951 173,968 23,130 13 8% 72,995 33% Maize 295,440 176,948 167,290 95 57% 74,995 82% Coffee 110,515 70,529 128,478 182 116% 19,994 134% Sesame 74,980 57,984 122,732 212 164% 9,997 177% Chickpeas 143,974 53,997 30,704 57 21% 44,989 53% Honey 88,282 37,292 19,755 53 22% 29,994 56% 4 Average yield per hectare/hive (kg) Wheat 2,969 3,276 110% 2,801 94% Maize 3,703 3,517 95% 3,300 89% Coffee 976 1,012 104% 976 100% Sesame 800 674 84% 775 97% Chickpeas 3,305 1,470 44% 2,754 83% Honey 15 12 80% 15 100% 5 Gross margin per unit of land/hive of selected crops (USD) Wheat 822 833 101% 748 192% Maize 648 558 86% 589 177% Coffee 2,453 1,630 66% 2,336 162% Sesame 581 1,079 186% 571 284% Chickpeas 750 636 85% 736 183% 9 Table 3 (continued) Indicator Value chain LOP target Cumulative (Years 1‐3) Year 4 target % Target Result % Achievable Result %LOP 6 Value of incremental sales (collected at farm level) attributed to FTF implementation (USD) Wheat 9,120,344 17,127,039 1,830,662 11 20% 3,500,000 58% Maize 12,449,951 7,665,348 10,817,419 141 87% 2,046,875 103% Coffee 25,274,365 8,074,365 32,129,763 398 127% 5,000,000 147% Sesame 37,544,955 22,236,251 29,951,240 135 80% 9,221,550 104% Chickpeas 8,403,233 1,508,233 1,599,002 106 19% 2,016,000 43% Honey 1,940,883 1,750,079 1,292,527 74 67% 85,684 71% 10 Value of exports of targeted agricultural commodities as a result of USG assistance (USD) Coffee 63,998,000 37,837,000 76,732,508 203 120% 20,000,000 151% Sesame 74,644,250 38,398,000 46,859,399 122 63% 20,475,000 90% Chickpeas 6,073,807 4,390,337 ‐ ‐ Honey 3,520,000 1,373,600 907,211 66 26% 1,000,000 54% 12 Volume of processed products (mt) Wheat 132,582 79,095 25,867 33 20% 37,541 48% Maize 19,265 8,317 10,071 121 52% 3,918 73% Coffee 29,627 7,565 16,944 224 57% 15,617 110% Sesame 171,927 93,335 34,503 37 20% 46,850 47% Chickpeas 36,019 25,107 915 4 3% 8,999 28% Honey 1,916 530 1,519 287 79% 574 109% 13 Percentage decrease of post‐ harvest losses as a result of AMDe intervention Wheat 9 (14) 8 6 120% 1 200% Maize 7 (23) 4 12 75% 3 100% Coffee 10 (25) 7 1 7% 3 27% Sesame 5 (10) 4 1 20% 1 40% Chickpeas 3 (20) 12 14 82% 3 100% 10 Text Box 2: AMDe support to FCUs to build warehouses for the sesame value chain Based on a request from the Government, USAID requested AMDe to provide support to FCUs in northern Ethiopia to support the aggregation, cleaning, and export of sesame. The initial recommended standard AMDe 50 percent grant was rejected by the MoA in favor of a 70 percent grant. Each of the 4 warehouses costs around Eth Birr 9 million or USD 450,000. The balance of the construction cost was secured by loan from the Ethiopia Commercial Bank, using the land certificate as collateral. The loans included a one and a half year construction “grace period” and then repayment over between five and eight years. To date, repayments are on track and total 23 percent. The warehouses each have a capacity of 5,000 mt, and AMDe has also supplied sesame cleaning equipment under the same grant facility. Result 2: Improved Access to Finance and Investment Originally, this result was titled Improved Access to Finance. However, following the sub‐ contracting of Renew Strategies (a private equity firm), the result was renamed, as is indicated in the sub‐title. Indicator #14: Value of agricultural and rural loans The main activities carried out under this indicator are twofold: to build the capacity and understanding of banks and agriculture sector stakeholders, resulting in increased trust and investment; and to attract equity investors to invest in Ethiopia’s agriculture sector. To December 2014, this resulted in USD 63 million59 of new investment, or more than 150 percent the LOP target of USD 40 million. An analysis of the loans confirms that 69 percent were made by the Commercial Bank of Ethiopia, while 16, 7, and 5 percent were made by the Cooperative Bank of Oromia, Awash International Bank, and Wogagen Bank respectively. The loans were made to 76 major recipients that indirectly benefited an estimated 2,300 other firms or PCs. The Evaluation Team finds this work of strategic importance in a sector that has historically struggled to attract investment from banks60 and financial institutions in Ethiopia, with the result that smallholder farmers, PCs, and FCUs61 have been slow to take up new technologies. Some of the most successful investments have been the result of a combination of loans and AMDe grants (see Result 4). For example, the AMDe provided 70 percent grants to FCUs for the construction of warehouses and cleaning equipment for sesame that will increase exports. The AMDe staff also worked with FCUs to develop business plans and the Ethiopia Commercial Bank (ECB) that resulted in the balance of the 30 percent construction costs being covered by the ECB as a repayable loan. Based on the success of this model (see Text Box 2 and 3), the AMDe has received grant applications for an additional nine warehouses from FCUs for the aggregation, cleaning, and storage of maize. The quality of the new warehouses will enable the FCUs to compete for WFP P4P contracts. It is also reported that the World Bank is supporting the construction of 40 more warehouses across the four AGP regions. Under the second activity, Renew Strategies has secured USD 250,000 of private equity funding for METAD PLC (a local coffee value chain actor) that has also benefited from an AMDe grant of USD 172,000 for improved storage, a washing station, and cleaning equipment. The AMDe investment team also reported that Renew Strategies has an additional USD 2.5 million in the pipeline, with options of an additional USD 6 million. While very encouraging, it remains to be seen if this level of investment will be realized.62 Indicator #18: Number of clients (households and/or microenterprises) benefitting from financial services provided through USG‐assisted financial intermediaries, including non‐ financial institutions or actors                                                              59 At the time of the finalization of the review, the level of facilitated investment had increased to more than USD 80 million. 60The Evaluation Team learned that only 6 percent of Ethiopia Commercial Bank loans are made to the agriculture sector. 61 It should be noted that cooperatives in Ethiopia have historically struggled to secure bank loans as a result of poor credit ratings. 62 Investors continue to be discouraged by the cost of doing business in Ethiopia. 11 The planned number of LOP clients benefiting from financial services is 64,520, of which 29,080 or 45 percent has been achieved (see Figure 1). The number of planned beneficiaries for Year 5 is 17,720, and, provided this number is achieved, the end of the project total will be 46,800, or 72.54 percent of the LOP target, which is satisfactory. The Evaluation Team finds this indicator of central importance, as it reflects the number of people benefiting from USG assistance and is therefore a proxy for employment. In a country where unemployment is estimated to be over 17 percent63 (Central Statistics Agency, 2014)— and higher in the age group 18 to 25—increasing the number of clients and therefore supporting the creation of seasonal and full‐time jobs is of primary importance. To further increase the number of clients and support increased employment opportunities, support would need to be focused on PCs and small‐ and medium‐sized businesses. In order to ensure this reorientation, the AMDe would need to disengage from further support for large infrastructure projects such as warehouse construction after it has met its current obligations. As a result of increasing interest from other development partners, including as proposed through the AGP2, it would seem that FCUs are increasingly able to attract investment for warehouse construction from other sources. Figure 1: Number of clients benefiting from financial services Result 3: Improved Enabling Environment Indicator #21: Numbers of policies/regulations/administrative procedures improved as a result of USG assistance The AMDe contracted a team of international and local consultants to undertake a “Business Enabling Environment Assessment” that is informed by the World Bank’s “Doing Business” assessment tools. In addition, the seed and fertilizer policy targets were informed by the PIF. The Assessment was carried out in 2012 and the findings presented to key stakeholders in a validation workshop. As a result, the AMDe identified seven policy processes that it planned to inform, with a view to creating a more favorable environment for agribusiness. The Evaluation Team learned that progress across the seven policy processes is mixed, with good progress made in some areas while slower progress has been achieved in other areas. This is perhaps not surprising, as the agriculture policy environment in Ethiopia is amongst the most challenging in sub‐Sahara Africa. Detailed progress is presented in the Table 4 below. Using the                                                              63 Ethiopia measures unemployment as the number of people actively looking for a job as a percentage of the labor force http://ieconomics.com/ethiopia‐unemployment‐rate. 12 Text Box 3: Embaba Haya RuSACCO, Tigray The Embaba Haya RuSACCO was established in December 2002 with 51 members (43 men and 8 women), with savings of Eth Birr 1,020. The membership increased steadily to 1,109 (515 men and 594 women) in 2012. With the support of AMDe, membership has increased further to 1,332 (620 men and 712 women) in 2013, and 1,479 (690 men and 789 women) in 2014. Savings are now valued at Eth Birr 5,196,000. Support provided by AMDe since March 2013 includes training on:  savings methods and membership mobilization  financial management  risk analysis in saving mobilization  basic computing—Peachtree Accounting Software  leadership With this support and AMDe’s support for office equipment— desktop computers, laptop, photocopier, printer, LCD projector, Sony camera, chairs, tables, filing cabinet, and fax machine—the Embaba Haya has graduated from a RuSACCO to a rural bank. FTF enabling environment tracking system, four policy processes have been progressed to Stage 2, one to Stage 3, and two to Stage 4. The Evaluation Team recognizes the good work done to create a more favorable environment for agribusiness but notes that the MoA had already identified the need for seed and fertilizer policy reform and that these were progressing as planned. In contrast, AMDe engagement in the wheat subsidy and maize export bans policy processes appear overly optimistic in view of the complexity of the policy processes, the number of different government institutions involved, and the importance to national food security. Irrespective of the engagement of John Mellor Associates to lead the wheat subsidy policy processes, the Evaluation Team finds it unlikely that these policy processes will be easily informed by the AMDe, or indeed that it could play anything other than a minor role. Rather, the Evaluation Team finds that AMDe’s policy work could have been better informed by and coordinated with other FTF partners, with the support of USAID. Table 4: Policies, regulations, administrative procedures by their stages of development Specific policy targets Development stage* Remark/note Seed system that supports the development of a private seed industry Stage 3: Presented for legislation/decree National Seed Regulations have been submitted by MoA to the Prime Minister’s Office for passage by Council of Ministers. Lift restrictions for private sector engagement in fertilizer production and marketing Stage 2: Drafted and presented for public consultation The MoA presented a revised policy proposal for the re‐launch of the National Fertilizer Industry Agency—dissolved in 2006. Regulatory framework for third party warehouse and grading Stage 4: Passed/approved A Public Enterprise Warehouse regulation was passed by the Council of Ministers and implementation will begin once the Regulation is gazetted. The regulation will separate the current ECX trading and warehousing system. This will be an important first step to third party or private sector warehousing. Removal of wheat import subsidies and maize export bans Stage 4: Passed/approved The MoT issued a letter to MoA on November 22, 2014 to remove the export ban on maize for commercial farmers and the EGTE. Provision of a National Agricultural Finance system to meet the needs of small commercial farmers Stage 2: Drafted and presented for public consultation Develop an institutional structure for coffee marketing Stage 2: Drafted and presented for public consultation Develop a strategy for the improved transport and handling of commodities to improve competitiveness Stage 2 :Drafted and presented for public consultation A strategy for the improved transport and management of fertilizer imports has been presented to key stockholders. Domestic transport cost analysis is underway. 13 *The various FTF policy enabling stages are: Stage 1—Analyzed, Stage 2—Drafted and presented for public/stakeholder consultation, Stage 3—Presented for legislation/decree, Stage 4—Passed/approved, and Stage 5—Passed for which implementation has begun. Result 4: Stimulate Innovation and Investment This result speaks specifically to AMDe's grant support. The grant fund was established at USD 14.2 million and to December 31, 2014 had disbursed a total of USD 11.6 million to 240 grantees. Grants are typically accessed by either a competitive solicitation process or specific requests from multi‐stakeholder value chain platforms with which the AMDe is involved. As has however been noted, at times AMDe responds to requests from the MoA routed through USAID. AMDe grants are funded under a cost share arrangement, with AMDe contributing between 30 and 50 percent. Individual grants vary considerably from as little as USD 817 to the Wirtu Kachama Seed Multiplication Cooperative Society for a Sell More for More Training— Phase 1, to grants of USD 1.8 million to the Yidnekachew Dabessa Coffee Plantation and USD 1.3 million to the Ethiopia Commodity Exchange for a Digital Traceability System. Other large grants include USD 970,000 for the Becho Woliso FCU—for Ethiopia’s first blended fertilizer plant—and five grants of USD 198,000 for the five management contracts of the blended fertilizer plants.64 Of the 240 grants completed or under process of payment to December 31, only 22 are above USD 100,000, with the vast majority around USD 50,000. Grants have been awarded to industry associations, cooperatives, RuSACCOs, private sector, public research institutions, and the Ethiopia Commodity Exchange. The majority of grants were awarded in 2014 and by year end, 9 percent had been fully completed, 61 percent were partially completed, and 2 percent were at the signature stage. In‐ kind grants of equipment and machinery accounted for 28 percent of grants, and all were under procurement. Additional analysis by region and value chain is provided in Tables 5 and 6 below. As can be seen, Oromia benefited from 88 grants totaling USD 4.7 million or 40 percent of all grants, with roughly half this amount awarded to the other three regions. Grants to the different value chains varied from USD 783,837, or 6 percent, for honey to 21 percent for sesame to support almost anything from warehouse construction to aggregate, clean, and store sesame for export, through to irrigation equipment, tractors, and seed cleaners for EIAR centers to increase the supply of basic seed and washing stations for coffee exporters. Similarly, PCs and FCUs have benefited from machinery—seed and grain cleaners, tractors, planters, honey homogenizers— and washing stations and have as a result been able to expand the range of services they provide their members. Grants have also been used to support common interest groups— women’s cooperatives and women entrepreneurs—including the purchase and distribution of coffee seedlings, modern beehives, and processors for chickpea milling. Table 5: Innovation grants awarded by region Region # of grants awarded to‐ date Amount awarded, to‐ date (USD) # of grants awarded 2014 Amount awarded, this year (USD) Amhara 47 2,207,141 43 1,406,613 Tigray 42 2,044,182 40 1,313,117 SNNPR 52 2,196,241 49 2,137,844 Oromia 88 4,751,127 81 4,410,354 Addis 11 416,506 7 237,719 Total 240 11,615,197 220 $9,505,648                                                              64 The World Bank funded the other four blended fertilizer plants. 14 Table 6: Innovation grants awarded by value chain Value chain # of grants awarded to‐ date Amount awarded, to‐ date (USD) # of grants awarded 2014 Amount awarded, this year (USD) Sesame 36 2,536,228 31 1,030,329 Coffee 40 2,178,866 32 1,999,084 Chickpeas 32 1,328,039 26 1,079,539 Maize 40 1,047,778 40 1,047,778 Wheat 58 1,189,210 58 1,189,210 Honey 26 783,837 25 608,467 Finance 3 461,905 3 461,905 Inputs (Fertilizer Blending) 5 2,089,335 5 2,089,335 Total 240 11,615,197 220 $9,505,648 In the remaining life of the project, the Evaluation Team find that it would be beneficial for AMDe to continue to focus its majority grant support to PCs and FCUs on the one hand and family businesses on the other, with grants of below USD 50,000 and that reach down the value chain towards productivity and first‐level aggregation or first‐level processing in smaller market towns. 2.2 Improvements in gender equity Question 2: To what extent has this project contributed to gender equity in terms of access to credit, capacity‐building support, improved inputs and technologies resulting in an increase in sales of agricultural commodities? Are there evidences supporting positive changes in the aforementioned areas? In accordance with GoE policy targets for women’s participation in Ethiopia’s economic, political, and social development, the AMDe set 30 percent women’s participation as a quota for all AMDe‐supported training, exchange visits, and investment support.65 Although the funding was delayed, the Evaluation Team has identified AMDe’s support to the launching of the Women in Agribusiness Leadership Network (WALN) in the four AGP regions as one of the project’s main gender equity successes. The Network offers business development training—negotiation, marketing, networking, financial planning, and communication skills— leadership training, mentoring and coaching support, and networking opportunities for women leaders operating in Ethiopia’s agribusiness sub‐sector.66 The primary purpose of AMDe’s support is to increase the profitability of women‐led businesses and to promote women as sector leaders.67 While recognizing the good work done, the Evaluation Team is concerned that, within a year of the launch, the AMDe channeled grants of USD 1.5 million for only 450 women members. The AMDe has also supported a membership drive to increase the number of women members and the number of women in leadership positions in PCs and FCUs. With the support of a behavior change organization, the AMDe used an incentive‐based model that offered small                                                              65 Women‐headed households account for 28 percent of households in AGP woredas, and women in male‐headed households typically constitute 50 percent of family labor. The 30 percent figure therefore under‐represents women in agriculture. 66WALN currently supports 100 women in its training program. The AMDe has invested USD 1.5 million in support of the Network to‐date. 67 The network has received national recognition through the January 2015 National Conference that was attended by more than 100 women entrepreneurs. 15 gifts to individual women who joined cooperatives and travel grants to the cooperative leaders who attracted the most women. As a result of the initiative, an additional 78,000 women joined cooperatives in the nine‐month period from March to December 2014 (see Table 7 below). Table 7: Incentive‐based women membership drive Region Number of FCUs Number of PCs Women registering Number registering for the first time % registering for the first time Oromia 14 761 14,161 2,781 19.6 Amhara 10 548 30,573 10,009 32.7 SNNPRs 9 237 19,227 13,066 68.0 Tigray 9 129 14,408 10,744 74.6 Total 42 1,675 78,369 36,600 46.7 While recognizing the impressive gains made to address the gender imbalance in cooperatives, the Evaluation Team is concerned AMDe’s behavior change advisory organization recommended the use of incentives, as when these are discontinued future women membership drives may prove less successful. What is really required is that cooperatives are given support to take on new ways of working that are more supportive of women membership and participation in decision‐making. For this reason, the Evaluation Team finds that the use of incentives should be immediately discontinued so that future increases in women membership are attributable strictly to women’s perceptions of membership‐related benefits. The Evaluation Team finds that USAID gave inadequate attention to gender in the AMDe design phase, as four of the original five value chains68 have a strong export focus. While aligning with the AGP is important, USAID might have given increased consideration to the fact that with some notable exceptions, agriculture sector exports are generally dominated by men. The selection of other and complementary value chains focusing on domestic markets might therefore have afforded increased opportunities for women smallholders, traders, and agro‐ processors. USAID also did not designate specific funds to support AMDe’s gender equity work. 2.3 Improvements in nutritional outcomes Key question 3: To what extent has AMDe contributed to the improved nutritional status of women and children? The original AMDe design did not include improved nutrition outcomes for women and children, and nutrition targets were added later at the request of USAID. In the first year of implementation, the AMDe commissioned a Cost of Nutritious Diet Baseline Study against which it is planned that nutrition‐related progress can be measured at the end of the life of the project. The findings of the study also inform AMDe’s cascade nutrition training through cooperatives. While the Evaluation Team recognizes the importance of improving nutrition outcomes in Ethiopia69 and that nutrition is a development objective of FTF, the Evaluation Team is concerned that the AMDe’s agribusiness and market development focus does not naturally lend itself to improved nutrition outcomes. Specifically, there are relatively few opportunities to develop                                                              68 USAID added the chickpea value chain in the first year of implementation. 69 Stunting rates are above 40 percent in AGP woredas and amongst the highest in the world. Text Box 4: Processed chickpeas for improved nutrition outcomes Together with Guts Agro Industry (GUTS), the AMDe has supported the launch of three chickpea‐based shiro products and roasted chickpea snacks. These new and nutritious chickpea products are processed from chickpeas purchased through FCUs that have a Memorandum of Understanding (MoU) with GUTS for 4,000 metric tonnes of chickpeas. It is estimated that the MoU benefits 52,000 smallholder farmers. Source: AGP‐AMDe Annual Report (2014) 16 and promote nutrition‐sensitive agriculture within the current scope of the project. The Evaluation Team is therefore not surprised to learn that nutrition‐related progress is mixed. On the one hand, the AMDe has contributed to improved nutrition outcomes through the chickpea value chain development and the launch of three chickpea shiro products (see Text Box 4). On the other hand, the Evaluation Team can find little evidence to support the continuation of the cascade‐based nutrition training, as it appears in some cases to be little more than an AMDe requirement for cooperatives to secure grants. It is also not clear to the Evaluation Team how successful the training is in changing attitudes and behavior or how gains will be sustained beyond the life of the project. The Evaluation Team finds that the nutrition component might be better implemented through FTF’s ENGINE project.70 2.4 Effective resource use Question 4: What is the impact of the resources spent and performance per value chain? What is the relative value generated in terms of productivity and income increase, employment generation per value chain to resources spent? Which have the highest returns per dollar spent? In order to address this question, it would be necessary to have full and reliable data sets on the following: increases in land area under new technologies; what the technologies were and what each contributed to yield improvements; an analysis of any increased costs associated with these changes; the value of incremental sales, with prices reported regularly throughout the marketing season; and the number of jobs, both seasonal and year‐round, that have been generated. This was not possible, as the AMDe monitoring lacks consistency and rigor. For this reason, this question was not addressed. The decision not to address this question was also informed by USAID’s own “Cost Benefit Analysis” study of 2014 that has provided a value chain performance ranking.71 In order to address this question fully, the AMDe will need to complete a thorough data cleaning and verification exercise. The Evaluation Team has, however, learned that the AMDe has created an estimated 4,231 jobs (see Table 8). While it is clear from the field work that not all these jobs are full‐ time/permanent, in a country where unemployment is so high, the Evaluation Team finds that employment creation should be a priority for FTF and that implementing partners should track and report in detail on employment creation, including both seasonal and full‐time/permanent employment, and ideally offer USAID gender‐ and age‐disaggregated data that can be shared with regional and federal government. Table 8: Number of jobs created Indicator Value chain LOP target Total # of jobs created (Years 1‐3) % LOP Number of jobs attributed to FTF implementation Wheat 2,268 143 6 Maize 2,060 2,690 130 Coffee 2,533 575 23 Sesame 1,556 538 34.5 Chickpeas 1,238 3 ‐ Honey 1,100 282 25.6 Total 10,755 4,231 39.3 Source: Abstracted from AMDe PMP reports                                                              70 Empowering New Generations in Improved Nutrition and Economic Opportunities implemented by Save the Children International. 71 It may be that USAID’s Data Quality Assessment (DQA) team could help address the issue of data consistency. However, to address the issue of rigor and impact, mention has been made of the need to undertake more detailed studies that include control groups.   17 While the information in these tables offers useful performance data across the six value chains, the Evaluation Team finds there is more work to be done to re‐check and verify some of the data—and in particular to confirm attribution, i.e., that the progress made can be attributed to the support provided by the AMDe. Unable to confirm the necessary information during the period of the review, the Evaluation Team did not fully answer this question. In the remaining life of the project, therefore, it is important the AMDe monitoring staff plays an active role in verifying all progress data, including addressing potentially challenging issues such as the impact of disease—rust in wheat and maize lethal necrosis—fluctuating farm gate prices, and the seasonality of some jobs. Unless this is done in the next few months, the Evaluation Team finds the AMDe will not be appropriately prepared for a rigorous end‐of‐project evaluation, which would be unfortunate, as the AMDe has clearly achieved a great deal. While unable to address this question fully, the Evaluation Team wishes to highlight the impact of USAID’s decision to support a MoA request for warehousing, storage, and cleaning equipment for sesame to increase exports. As presented in Figure 2, investment across six value chains across four years varies between 1 and 7 percent. The exception is support to the sesame value chains in Year 2, when 35 percent of the AMDe annual budget was invested. While recognizing the strategic importance of this investment, the Evaluation Team finds this skewing of investment had a deleterious effect on the development of the other value chains. Figure 2: Budget use by value chain Further analysis of budget use also reveals that “pass‐through grants” to partner organizations and value chain actors account for 45 percent of the project budget and are managed by ACDI‐ VOCA headquarters in Washington. While this grant management mechanism may reduce demands on local staff, the Evaluation Team finds many advantages in local grant management, as local staff better understands the context in which the project is working, and are in a better position to provide technical and managerial mentoring support. Finally in this section, the Evaluation Team learned that the AMDe invests only 3 percent of project costs in training and capacity development. The Evaluation Team finds that this level of investment is inadequate to consolidate and mainstream gains made by AMDe. 18 2.5 Project design and management Question 5: Among AMDe's partnerships, which are being most effective in terms of their collaboration and coordination? As mentioned, the AMDe supports the MoA AGP to deliver its Component 1, sub‐component 1.3: Market and Agribusiness Development.72 In order to do this, the AMDe works most closely with the MoA, ATA,73 and the FCA at federal and regional levels, with which it shares a common vision of the commercialization of agriculture and growth of cooperatives. At an operational level, the AMDe annual work plans are endorsed by regional AGP Technical Committees before they are presented at the federal AGP Technical Committee. In addition, the AMDe has also forged a range of partnerships, a sample of which includes the following:  Capacity Building for Scaling‐Up of Evidence‐based Best Practice in Agricultural Production in Ethiopia (CASCAPE): The CASCAPE project is a joint initiative between Ethiopia and the Netherlands to improve agricultural productivity. The AMDe partners CASCAPE on soil testing and improved seeds.  Capacity to Improve Agriculture and Food Security (CIAFS): The AMDe is working with CIAFS on fertilizer policy, honey and coffee value chains, capacity building for cooperatives, and a coordinated initiative to promote private sector‐led seed multiplication and certification.  Ethiopia Commodity Exchange (ECX): The AMDe is working with the ECX to train ECX staff, warehouse operators, and management firms on grading and coffee certification. In addition, the AMDe is working with the ECX to revise coffee standards, including the planned launch of a traceability system. Finally, the AMDe is supporting cooperatives to become permanent members of the ECX.  Ethiopia Institute of Agriculture Research (EIAR): The AMDe carried out a seed production and supply survey74 with the EIAR that resulted in AMDe grants for irrigation equipment, tractors, and seed cleaning equipment being awarded to 11 research centers to increase the supply of certified seed of varieties that will help increase smallholder farmer productivity.  Industry associations: The AMDe is working closely with a number of industry associations, including the Ethiopia Apiculture Board (EPA), Ethiopia Honey and Beeswax Producers and Exporters Association (EHBPEA), the Ethiopia Pulses, Oilseeds and Spices Processers and Exporters Association (EPOSSPEA) and the Ethiopia Coffee Exporters Association (ECEA).  Ministry of Trade: The Agricultural Marketing Directorate of MoT is mandated to build capacity in the supply, distribution, and marketing of agricultural inputs. AMDe is collaborating with the MoT to assess regional and international marketing opportunities and warehouse regulations.  Private Sector: The AMDe is working with several hundred private sector organizations, including WALN. The Evaluation Team learned from other FTF implementing partners that at times the AMDe leadership plays a lead and catalytic role amongst FTF implementing partners in Ethiopia, in particular in developing value chain thinking and promoting market solutions. This said, in a meeting of FTF implementing partners that the Evaluation Team facilitated on December 16, 2014,75 it became clear that much more could be done to strengthen levels of coordination and integration, including:                                                              72Through the delivery of the MoA’s Agriculture Sector Policy Investment Framework, 2010–2020 (PIF), the MoA has sought to reorientate donor funding from funding emergency food security projects to social protection (PSNP) and longer‐ term development (AGP and SLMP). USAID’s funding for AMDe is welcomed by the MoA as part of this commitment. 73 USAID, ATA, and AMDe have forged a tripartite coordination plan with focal points assigned for each joint activity. 74 Strengthening Source Seed Production and Supply System of the National Agriculture Research System in Ethiopia.   75 The meeting was attended by three Chiefs of Party (COPs), four DCOP (Deputy COPs) and four other senior staff representing the AMDe, ENGINE, GRAD, LMD, and PRIME projects.     19  Coordinate activities in AGP woredas in order to reduce duplication where projects are working in the same value chain, e.g., LMD and GRAD—sheep and goat fattening and AMDe and GRAD—honey.  Share social behavior change communication materials on nutrition and coordinate interventions.  Share lessons in gender mainstreaming.  Collaborate in the organization/participation in local and international trade fairs. Ways in which the FTF might better collaborate are presented diagrammatically below in Figure 3. Figure 3: Illustration of the collaborative mechanisms between FTF partners Key: Collaboration based on geographic overlap Collaboration based value chain or shared objectives Although the AMDe has built effective partnerships, the Evaluation Team finds that USAID and AMDe play complementary as opposed to central roles in AGP decision‐making. In part, the Evaluation Team attributes this to partner differences—organizational culture, timing of planning cycles, incentive structures—but this is also the result of USAID’s support for AMDe and LMD as opposed to its support for AGP pooled funding. While there are inevitable constraints associated with USAID’s funding arrangements, so too the Evaluation Team finds strengths and opportunities, including increased opportunities for innovation, piloting, and demonstration that over time can be documented and inform evidence‐based good practice. The Evaluation Team learned that Ethiopia’s cooperative movement aggregates and markets less than 10 percent of Ethiopia’s cereals,76 and, in contrast, small‐scale private sector traders play the major role in cereal aggregation, transport, marketing, and processing. While recognizing the AMDe works well with industry associations and some 500 commercial farmers, service providers, traders, and processors, the Evaluation Team finds that issues of sustainability will be better addressed by achieving a better balance of support between cooperatives and private sector actors. The Evaluation Team finds the AMDe works well with other FTF implementing partners at federal and regional levels but that the quality of partnership appears to tail off at woreda level. The probable reasons are outlined above and do not need repeating. The Evaluation Team, however, finds that USAID could have done more to forge a coordinated, integrated, and harmonized FTF portfolio and that the current re‐design of the MoA’s PSNP4 and AGP2 potentially offers USAID and lead FTF implementing partners77 new and potentially exciting opportunities for improved alignment, integration, and coordination. 2.6 Sustainability                                                              76Cereals account for more than 60 percent of all arable cropping in Ethiopia. 77 The lead FTF implementing projects are AMDe, LMD, GRAD, and PRIME.   ENGINE AMDe GRAD LMD AGP woreda Non‐AGP woreda 20 Question 6: Which of the components and/or project activities can be scaled up in the future based on measurable, practical, and sustainable results? As expected at this stage of the project cycle, the AMDe is well established in each of the four AGP regions and across the six value chains. In order to address the issue of sustainability and specifically which project activities in which regions can be scaled up, the Evaluation Team has structured its findings around the following four themes as required by the SoW:  Enhancing agricultural productivity  Access to finance and  Access to markets  Cross‐cutting Enhancing agriculture productivity: The Evaluation Team recognizes the good work done by AMDe to increase smallholder farmer productivity, including: input supply—improved seeds, blended fertilizers, crop protection—and post‐harvest handling practices. The Evaluation Team finds that this work should be continued and expanded. Under the G8 New Alliance for Food Security and Nutrition,78 the AMDe is supporting the MoA, ATA, and DuPont Pioneer79 in a three‐year partnership under the Advanced Maize Seed Adoption Program (AMSAP), which seeks to transition smallholders from open pollinated to hybrid maize varieties to increase yields. Since the launch with 320 model farmers in 2013, the program has expanded to 3,200 smallholder farmers in 2014. The AMDe also supports the BH‐661 Program that was launched with 100 model farmers in 16 woredas in Amhara and Oromia and has been subsequently expanded into SNNP Region. While recognizing the importance of improving access to quality seeds, the Evaluation Team is concerned the AMDe is narrowing its focus on hybrid maize. Certainly, hybrid maize out‐yields open pollinated varieties, but the appropriateness, affordability, and sustainability of the hybrid maize can be questioned. The Evaluation Team finds the AMDe should provide more balanced support to maize growers, including open pollinated varieties and support for farmer‐ to‐farmer exchanges and community seed fairs. Access to finance: It is widely recognized that in order to increase and sustain Ethiopia’s impressive agriculture sector growth rates, additional investment is needed. The Evaluation Team finds that AMDe’s work in this area is of critical importance and should be a priority during the remaining life of project. Specifically, however, the Evaluation Team finds the AMDe should focus on support to business plan development, negotiations with banks, and improved financial management, accounting, and auditing. The Evaluation Team also finds that the AMDe should increase its level of support to small‐ and medium‐sized businesses. Access to markets: The Evaluation Team finds that, while continuing to complete existing commitments, the AMDe should focus its remaining work to strengthen value chains for domestic markets, as these have the potential to benefit poorer smallholder farmers, in particular women, who are FTF’s primary beneficiaries. As part of this reorientation, the Evaluation Team find an important role for AMDe to continue to strengthen the technical, managerial, and administrative skills and capacities of PCs and FCUs to secure finance, invest in infrastructure and equipment, and therefore more effectively aggregate, grade, clean produce, and supply traders.                                                              78Launched in Ethiopia in the autumn of 2012. 79 DuPont Pioneer is investing USD 2.3 million to provide hybrid maize. AMDe is supporting this investment with USD 1 million to support training, field demonstrations, direct retail distribution networks, commercial credit schemes, post‐harvest storage, soil sampling at demonstration sites, logistics, and market access and development.    Text Box 5: AMDe support to Sidama Elto FCU Sidama Elto FCU was established in 2004 with 1,000 members (only 22 women). It has grown to 15,000 members (1,400 women), while its capital has increased from Eth Birr 6,100 to 15 million. AMDe facilitated links to WFP P4P in 2010. Initially, it was not easy to meet quality standards. The FCU is, however, now able to supply maize to WFP, the result of AMDe support—improved storage (including moisture testing), shellers, cleaning, and fumigation sheets. 21 Specific to the maize value chain, the Evaluation Team finds that, while continuing to facilitate links between FCUs and World Food Programme’s (WFP) P4P initiative (see Text Box 5), the AMDe also identify alternative markets that link FCUs to private traders and in this way spread risk and build more sustainable market options. Cross‐cutting: The Evaluation Team finds the AMDe cascade‐style training modules—Sell More for More (SM4M)—that AMDe has pioneered can be scaled up and mainstreamed. One way to do this is to partner with the Ardaita Cooperative ATVET,80 Oromia Region, as it provides similar training and capacity‐building support to PCs and FCUs in that part of Oromia. Through such a partnership arrangement, the AMDe could not only build the capacity of Ardaita, but also address issues of sustainability.81 In addition, the Evaluation Team finds that all training should be part of an integrated and holistic capacity‐building initiative that provides ongoing support to a more focused number of organizations and individuals, as only in this way will real impact be ensured. 2.7 Management Question 7: Does the project have the right balance of staff and funding given activity priorities? The AMDe employs a team of 81 full‐time staff and 19 part‐time staff (64 men and 36 women) as follows:  Senior managers—10 percent  Administration, finance, and support—11, 10, and 27 percent respectively  Component 1: Access to markets and improved competitiveness—25 percent  Component 2: Access to loans—5 percent  Component 3: Enabling environment—a single staff member  Component 4: Innovation and capital grants—2 percent  Cross cutting—10 percent The Evaluation Team learned that almost half of AMDe staff is based in Addis Ababa.82 In order to consolidate the gains made during the life of the project, the Evaluation Team finds that AMDe will need to invest more resources in capacity building and institutional support in the regions. To support this reorientation, the AMDe might benefit from the support of an organizational change expert who can also address other tasks, including the review and harmonization of job titles and clarification of roles and responsibilities amongst pooled workers. The Evaluation Team also recommends the AMDe strengthen its M&E capacity, with a view to improving the quality of its reporting and ensuring all data are cleaned ahead of the final evaluation. 3. Recommendations Recommendation for Result 1 #1: The AMDe continue to work on the six value chains in order not to disrupt project implementation during the remainder of the project, but give increased focus to incomplete Results:  Wheat—productivity and value addition through processing  Maize—productivity, including open pollinated varieties, improved post‐harvest handling, cleaning, storage, and diversifying domestic market outlets                                                              80 The Cooperative Sector Development Strategy (FCA, 2012), developed with the support of ATA, identifies Ardaita as a potential “center of excellence” for cooperative development. While not centrally located, it is planned the center will establish branches in other regions and operate as a “college without walls.” 81 During the Evaluation Team’s visit to Ardaita, it was learned that the AMDe had visited once but without follow‐up. 82This figure includes the AMDe Oromia Region staff based with the national team, which is not unusual as the Bureau of Agriculture, Oromia Region is similarly based in Addis Ababa. 22  Coffee—post‐harvest loss reduction and quality standards  Sesame—post‐harvest loss reduction and processing  Chickpeas—domestic and niche markets (Sudan, for example) for the currently used varieties  Honey—production and productivity through the distribution of improved equipment, including modern beehives and training and equipping of beekeepers and new beekeepers Recommendation for Result 2 #2: Continue and strengthen AMDe's engagement in agriculture sector finance, in particular capacity building for cooperatives and small‐scale private businesses to develop business plans,83 negotiate with banks, and improve accounting and auditing systems. Recommendation for Result 3 #3: Recognize the need to work more closely with other FTF implementing partners, including the AKLDP.84 With FTF partners, forge a common, integrated, and coordinated approach to agriculture sector policy work. Recommendation for Result 4 #4: Reduce the focus on grants for new partners and consolidate the capacity of existing partners to improve impact and sustainability. Recommendations for gender equity #5: Continue to support women empowerment, including women membership of cooperatives— though discontinue the use of incentives—and WALN, with increased emphasis on women‐led small‐ and medium‐sized agribusinesses in the regions. Set aside funding for gender equity work. #6: Document AMDe’s work with women in Tigray and SNNP Regions and share with FTF projects within and beyond Ethiopia. Recommendation for nutrition #7: Hand over the nutrition work to ENGINE. Recommendations for improved collaboration #8: Continue to strengthen partnerships with key stakeholders, in particular regional BoA AGP teams, ATA Agriculture Commercialization Clusters, and Regional Cooperative and Industry Association Offices to accelerate training and capacity building of regional, zonal, and woreda staff in agriculture marketing that includes components on the private sector and its potential role in transforming Ethiopian agriculture. Ensure that all training and capacity building is focused and an integral part of a holistic capacity‐building approach. #9: Collaborate with ATA and Federal Cooperative Agency (FCA) to develop a single PC/FCU Certification system that integrates and harmonizes accreditation criteria. Recommendations for scaling‐up #10: Support the MoA and ATA to improve productivity through the improved distribution of certified seeds (with a minimum 60 percent open pollinated varieties), blended fertilizer, credit, and extension support. #11: Continue to support the WFP‐P4P initiative but continue to strengthen and develop links between smallholder farmer groups and responsible private sector traders. Recommendations for management                                                              83 The business plans prepared jointly by FCUs and AMDe are of a high quality. For example, the Sidama Elto FCU business plan secured loans of Eth Birr 7.6 million in 2013, Eth Birr 14.4 million in 2014, and Eth Birr 6.35 million in 2015. 84 The AKLDP project provides agriculture knowledge, learning, and policy support to USAID’s FTF portfolio. 23 #12: Recruit an organizational change expert to reorientate AMDe staffing, specifically increased support for: farmer level production and productivity;85domestic market development; and gender. Clarify the responsibilities of the AMDe and ACDI/VOCA pooled staff and as appropriate move staff to the regions. #13: Recruit an M&E specialist to improve AMDe M&E capacity. Ensure all project data are cleaned and that increased consideration is given to project outcomes and attribution, ahead of the final evaluation. #14: Strengthen AMDe’s capacity development work to 15 percent of the Year 5 project budget to build the capacity of priority PCs and FCUs and better equip them to continue work started with AMDe. To support this reorientation, complete all warehouse construction‐related commitments and subsequently disengage from warehouse construction. #15: Include representatives from the MoA, and possibly the MoT and MoI, in the final evaluation. General recommendations for a follow‐on project The Evaluation Team recommends that USAID provide follow‐on funding for a minimum of another five years and therefore welcomes USAID’s commitment to fund AGP Component 4: Agriculture Marketing and Value Chains, sub‐component d—strengthening of selected livestock and crop value chains. While welcoming this commitment, the Evaluation Team recommends that USAID support for AGP be more focused to achieve FTF development objectives, specifically that a follow‐on program operate in a smaller ZoI and with fewer partners. Within these ZoI, it is recommended that, in addition to continuing to support agriculture marketing, significant increased emphasis is given to: smallholders with holdings of less than one hectare—perhaps 70 percent of planned project beneficiaries; sustainable agriculture production and productivity, including soil health, land management, and land tenure; women in agriculture— women‐headed and women in male‐headed households; and the private sector, in particular small‐ and medium‐sized businesses. In addition, the Evaluation Team recommends that USAID provides better layered and sequenced support within its ZoI that includes social protection, WASH, nutrition, health, and education and that, through this integrated and coordinated approach, USAID play a lead role in poverty and stunting reduction that will be replicated by others. The Evaluation Team finds that USAID should exercise increased leadership to contextualize global FTF indicators and guide FTF implementing partners to key indicators that will track progress to address Ethiopia’s poverty and stunting challenges. Specific recommendations for a follow‐on project include: #16: Reorientate the value chains: hand off coffee and wheat to the industry associations and other development partners, including, as appropriate, the EU, a USAID‐funded innovation lab project holder, and ATA; build a more integrated approach to cereal value chains that includes maize,86 malt barley, and sorghum—grown by large numbers of poorer smallholders; expand the honey value chain—assisting larger numbers of poor women with little or no land; and integrate chickpeas into a legumes/pulses value chain for domestic markets—legumes offer important soil health and nutrition outcomes. USAID could therefore support the following value chains:  Cereals—maize, malt barley, and sorghum  Honey—for domestic markets (with an emphasis on women)  Legumes—for domestic markets #17: Ensure that capacity building, institutional development, and gender are key components of a follow‐on project, are appropriately funded and delivered within a holistic capacity‐building approach.                                                              85 The Evaluation Team is keen to see that the USAID Ethiopia Mission strengthens its support of production and productivity and therefore achieves a more balanced agriculture sector portfolio. The Evaluation Team is of the view that the Mission is at present too market‐focused. 86 With a focus on open pollinated varieties. 24 #18: Reduce managerial complexity—sub‐contract the productivity, value chain development, aggregation, and domestic market development components to specialist NGOs with a proven track record in Ethiopia. The project holder can then better focus on management, coordination, monitoring—data collection, collation, analysis, documentation—learning, and championing evidence‐based good practice, including with AGP2. The follow‐on project would be expected to leave Ethiopian institutes, organizations, and businesses better equipped to lead the transformation process of Ethiopian agriculture. Annexes Annex 1 Statement of Work Annex 2 Evaluation Team Members – Curriculum vitae Annex 3 Conflict of Interest forms Annex 4 Work plan Annex 5 Bibliography Annex 6 List of places visited and people interviewed Annex 7 Tools used 2 ANNEX 1: STATEMENT OF WORK MID-TERM PERFORMANCE EVALUATION USAID/ETHIOPIA’S AGRICULTURE GROWTH PROGRAM—AGRIBUSINESS AND MARKET DEVELOPMENT (AGP-AMDe) PROJECT Contract Start Date End Date LOP Budget AID-663-TO-11-00003 May 02, 2011 May 01, 2016 USD 49,885,436 I. INTRODUCTION Agricultural Growth Program—Agribusiness and Market Development (AGP-AMDe) is a flagship project under USAID’s Feed the Future (FTF) strategy for Ethiopia and is USAID’s largest contribution to the Government of Ethiopia’s (GoE) Agricultural Growth Program (AGP). It is a five￾year project aimed at stimulating agricultural growth to improve national food security, generate employment, and reduce poverty in a sustainable manner. AGP is a unique initiative designed to coordinate and integrate the efforts of all development partners to ensure a shared understanding of priorities and objectives, and to avoid the duplication of efforts. The project was initiated in 2011 and will close in 2016. In line with the overall objectives of AGP, AGP-AMDe works to sustainably reduce poverty and hunger by improving the productivity and competitiveness of value chains that offer jobs and income opportunities for rural households. AMDe uses a value chain approach to bring together relevant actors to create and take advantage of market opportunities and address shared constraints. The value chain approach, it is postulated, stimulates agricultural productivity and the growth of small-scale businesses, increasing incomes and reducing the poverty and vulnerability of smallholder farmers and rural inhabitants. This market-led approach builds cooperation among stakeholders with divergent interests, fast tracks the pace of development, and ensures the sustainability of results. Target value chains are maize, wheat, sesame, coffee, honey, and chickpeas. The project is implemented in the high rainfall regions of Amhara, Oromia, Southern Nations, Nationalities, and People's Region (SNNPR), and Tigray by strengthening agricultural productivity and markets. The AMDe performance evaluation is expected to provide important information on the level of progress regarding planned results (both quantitative and qualitative) of the project vis-à-vis the project objectives and goal, effectiveness of the functions of the established collaboration/coordination mechanisms, and the working relationship among the implementing partners and stakeholders. The evaluation will also identify key lessons learned (both positive and negative) to inform the project’s implementation in the remaining years ahead. USAID’s evaluation policy encourages independent external evaluation to increase accountability to inform those who develop programs and strategies, and to refine designs and introduce improvements into future efforts. In keeping with that aim, this evaluation will be conducted to review and evaluate the performance of the USAID-funded AMDe project activities implemented by ACDI/VOCA. The evaluation will focus on assessing the program’s mid-term performance (May 2011 through June 2014) in achieving its program goal, objectives, and results. II. BACKGROUND USAID/Ethiopia’s Country Development Cooperation Strategy (CDCS) goal—“Ethiopia’s Transformation to a Prosperous and Resilient Country Accelerated”—is aligned with the Government of Ethiopia’s (GOE’s) five-year Growth and Transformation Plan (GTP), which builds on the significant expansion of the economy and basic services over the past five years. The USAID/Ethiopia CDCS comprises three Development Objectives (DOs) and one Support Objective. The Economic Growth and Transformation (EG&T) office, in coordination with the Alternative Livelihoods and Transition (ALT) office, implements DO1 (Increase economic growth with resiliency in rural Ethiopia). As shown in the DO Results Framework (As shown in the DO Results Framework 3 in Annex I), the achievement of this DO depends upon the combined success of five interdependent intermediate results (IRs). The strategy will demonstrate the potential of market-based agricultural development to reduce poverty and promote sustainable livelihoods for chronically food-insecure households. The Feed the Future (FTF) Strategy, which is the key component of DO1, aims to sustainably reduce poverty and hunger through investments in the performance of the agricultural sector, in improved nutrition, and in the improved capacity of vulnerable households to meet their food needs. FTF investments will lead to improvements in food availability, access, and utilization. A better￾performing agriculture sector should improve both food availability and access, while improved food access and utilization will improve nutrition. The strategy demonstrates the potential of market-based agricultural development to reduce poverty and promote sustainable livelihoods for chronically food￾insecure households. The strategy utilizes a Push-Pull Model that seeks to build the capacity of vulnerable and chronically food-insecure households to participate in economic activity (the “push”), while mobilizing market-led agricultural growth to generate relevant economic opportunity and demand for smallholder production, labor, and services (the “pull”). AGP-AMDe project is designed to achieve USAID’s overall goal to sustainably reduce poverty and hunger by improving the productivity and competitiveness of value chains that offer jobs and incomes for rural households. The project has applied a comprehensive approach to strengthen the private sector and link farmers to sustainable livelihoods along key agricultural value chains. Activities have been driven by gender-sensitive value chain analyses that incorporate a dual focus on tangible aspects such as infrastructure, institutions, technologies, and prices, together with intangibles such as trust, industry leadership potential, and the value placed on learning. This analysis enabled AGP-AMDe to provide the right mix of targeted technical assistance, capacity building and institutional strengthening, and critical public and private partnership development. The project promotes innovation and incentives through smart subsidies to enhance knowledge, increase opportunities, and inform choices that catalyze and drive changes in behavior, investment, and production. ACDI/VOCA, the prime contractor to the AGP-AMDe project, has assembled a team of sub￾contractors with diverse skills and broad experience that offers the specific competencies to deliver results across AGP-AMDe. ACDI/VOCA is the lead organization of the team as well as technical lead on the value chain and financial services components. IFDC offers its extensive experience in the development of commercial agro-input systems. Booz Allen Hamilton leads the enabling environment component, supported by technical assistance from John Mellor & Associates on agriculture policy reform. Danya provides a strategic communications plan which will guide AGP￾AMDe’s use of communication tactics to increase behavior change, adopt technology, facilitate learning, and increase overall project outcomes. Kimetrica brings its ki-projectsTM platform to strengthen the monitoring and evaluation system. CQI provides targeted technical support to development of the coffee value chain. Crown Agents provides demand-driven technical assistance. Since its inception, the AGP-AMDe project produced notable achievements against targets set. Under the Task Order, ACDI/VOCA has had success addressing systemic constraints through the following activities:  Building the internal financial and risk management capacity of Farmer Cooperatives Unions (FCUs), commercial farmers, and agri-businesses  Facilitating debt financing through intensive technical support to FCUs and Primary Cooperatives (PCs) on business plan development and meeting loan application requirements, linking potential borrowers to financial institutions for loans to invest in agricultural technology and machinery upgrades  Promoting and facilitating uptake of new products and technologies, such as leasing, in collaboration with other programs and pilots for community warehouse receipt systems and mobile banking  Supporting marketing programs, such as the WFP P4P program in which we provide direct technical assistance to FCUs to meet sales contract agreements of quality and quantity, and supporting Agricultural Transformation Agency (ATA) to increase FCUs’ access to agricultural machinery to improve their productivity 4  Stimulating equity investment in agro-processing in partnership with RENEW Strategies, drawing increased international, private sector investors to Ethiopia Intended Project Results The following chart summarizes the various anticipated results for the AGP-AMDe project. Activities Outputs Outcomes Impacts  Conduct value chain analysis with financial, enabling environment, nutrition, and gender lenses; strategy and vision development  Initial analysis and vision/strategy document; plan for integration of cross-cutting themes; regular updates and ongoing documented learning  Project follows clear strategy for achieving vision; project is able to adjust interventions based on observations Value chain level:  Increased competitiveness of value chains  Increased responsiveness to end-market opportunities through new technologies and relationship structures  Supporting markets serving value chain actors and adapting to trends in demand for inputs, training, ICT, transport, and other services  Expanded reach and variety of financial services available to value chain actors  More supportive environment for financial sector and value chain growth Firm level:  Increase in number and activity of agricultural enterprises including input suppliers, aggregators, exporters, and service providers  Increased profits from agriculture, especially for smallholders  Improved cash flow and access to finance for investment Household level:  Increased income  Increased employment  Reduced poverty  Improved food security and nutrition  Build relationships that enable value chain actors to respond to end-market opportunities  Promote transparency and trust in vertical linkages  Activities to strengthen horizontal linkages  Activities to strengthen supporting markets, including input supply, storage, ICT, transport, and training/BDS  Develop TOT and materials on health and nutrition themes; Train GoE AGP implementers who will train DAs to deliver health/nutrition messages  Buyers identified and engaged with suppliers  Stakeholders’ workshops and smaller meetings held  Increased and repeated transactions between buyers, suppliers, and support service providers  Cooperatives and associations assessed and trained  New commercial input suppliers entering the market  Storage facilities rehabilitated or established; possibility of private sector warehouse operators explored  Information about ECX grading system disseminated  Trade fairs attended, trainings in export promotion held  Market information delivered through local entities  Service providers linked to clients, trainings co-financed  Health/nutrition materials developed; TOTs held; follow-up TA conducted to ensure messages reach smallholders through DAs  Increased sales and learning flows between buyers and suppliers, including smallholder farmers and vulnerable groups  Increased trust and transparency between value chain actors  Smallholder farmers consistently able to understand and achieve quality standards  Balanced distributions of benefits in the value chain, leading to incentives for ongoing upgrading at multiple levels  Increased use of support services by smallholders (including embedded services)  Private sector input suppliers selling improved inputs and providing embedded services  Households receive and apply new information about improved health/nutrition practices  Build capacity of financial institutions  Develop new financial products and services  Build capacity of financial sector associations  Stimulate lending through guarantee funds  Work with agricultural enterprises to establish alternative forms of collateral  Facilitate policy and regulatory improvements  Leverage the investments of RUFIP  Trainings conducted for commercial banks and MFIs  TOT held with EBA and AEMFI  TA and training held for MFIs to improve marketing to women  New products designed and piloted to meet the needs of smallholders and other value chain actors  Products rolled out; staff of financial institutions trained in new products  TA delivered to financial sector associations in agricultural lending, risk management, and product development  Increased creditworthiness of agricultural enterprises  Public-private dialogue stimulated; reforms achieved  Meetings and joint activities carried out with RUFIP  Increased ability and willingness of commercial banks and MFIs to lend to the agricultural sector  Financial institutions able to tailor products to the needs of target clientele including women and smallholder farmers  Financial institutions participating in the DCA program  Financial sector associations facilitate increased focus on agricultural lending  Policy and regulatory framework conducive to growth  RUFIP adequately reaching and serving value chain actors  Promote public-private dialogue  Build the capacity of the public sector  Design improvements in policies and regulations  Meetings and workshops held with trader and exporter associations, cooperative unions, and other civil society institutions  Stakeholder buy-in to reform process  Reduced enforcement costs 6  Facilitate access to international markets  TA and training provided to MOARD staff  Reforms to administrative procedures  Training and TA provided to Ministry of Trade and other relevant institutions  More efficient administrative processes  Policies that better support value chain growth  Increased government ability to analyze and plan policy change  Ministry of Trade better able to analyze and alleviate trade constraints  Develop, disburse, and monitor strategic grants through Innovation and Demonstration Funds  Large and small grants disbursed to address systemic constraints related to the policy environment, lack of access to finance, disincentives for investing, etc.  Partial subsidies introduced and phased out as value chain actors demonstrate willingness to invest  Perceived risk reduced through observed benefits of investment in upgrading  Innovation becomes the norm as more value chain actors invest in new technologies 7 III. PROJECT GOALS AND OBJECTIVES In line with the overall objectives of AGP and the Agricultural Transformation Agency (ATA) national strategies, AGP-AMDe is designed to sustainably reduce poverty and hunger by improving the productivity and competitiveness of value chains that offer jobs and income opportunities for rural households. AGP-AMDe project activities are intended to directly contribute to USAID’s development objective of Increased Economic Growth with Resiliency in Rural Ethiopia through targeted intermediate results in the areas of:  Improved competitiveness of selected value chains  Improved inputs—both seed and fertilizers  Improved access to finance and investment  Business enabling environment  Innovation fund (focus on implementation rather than disbursement) The program consists of four major components: 1. Improved competitiveness of selected value chains: Under Component 1, the AGP-AMDe is increasing market competitiveness and improving market access for smallholder farmers by facilitating improvements in capacity of the cooperative sector, private traders, processors, and service providers, as well as creating specific market linkages between various industry actors. The project is supporting an increased value and volume of sales in domestic and export markets. 2. Improved access to finance: Under Component 2, AGP-AMDe is building on Component 1 activities to improve access to finance for input and output purchases and investment for cooperatives and agro-dealers through the development of business plans, financial plans, and loan applications for input/output and medium-term capital. The project is also developing new or enhancing existing financial products for cooperatives and agribusinesses that commercial banks provide. Additionally, improving access to production and output financing for smallholder producers is also included here. 3. Improved enabling environment of selected value chains: Under Component 3, AGP-AMDe is largely focusing on fertilizer and seed policy changes to be officially adopted by the GoE, and subsequently rolled out throughout the country. Additionally, the project is addressing agricultural finance, streamlining regulations supporting trade of inputs and outputs, warehouse management regulations, and gender-enabling environment constraints. These constraints have been identified and prioritized with ATA and GoE, and joint initiatives are being planned and implemented to address them. 4. Stimulate increased innovation and investment: The Innovation Fund, a key resource available to the AGP-AMDe project, is facilitating investments in a broad range of competiveness￾enhancing activities, with specific focus placed on innovative approaches and technologies. It cuts across all areas and supports many of the activities under Components 1 and 2. 8 AGP-VC Results-Oriented Integration: IV. PURPOSE AND USE OF THE EVALUATION The main purpose of this midterm performance evaluation is to provide the AGP-AMDe team and USAID with an objective assessment of project progress towards achieving its objectives, the effectiveness of the AGP-AMDe partnerships, and any strengths and challenges. The findings and recommendations from this mid-term evaluation are therefore expected to inform management decision￾making processes and improve performance towards the achievement of AGP-AMDe’s key results and goal. The evaluation findings will also help inform the design of future similar projects. Specifically, the evaluation will: 1. Assess the extent to which planned results (both the quantitative and qualitative) of the project have been achieved vis-à-vis the project objectives and goal 2. Assess the programmatic and cost effectiveness of the project’s approaches during implementation towards achievement of its objectives and key results 3. Assess whether or not the project activities are replicable and sustainable with supporting evidence 9 4. Identify lessons learned and challenges and make clear, explicit, and actionable recommendations. Suggestions will be made for improving not only project performance but also the design and implementation of future similar projects. V. EVALUATION QUESTIONS A. Performance Results 1. To what extent is the AGP-AMDe Project progressing against planned objectives as outlined in its Performance Management Plan (PMP) and Work Plan? The evaluation should analyze actual results versus the targets outlined in the Project PMP within the activity’s different technical areas (value chain competitiveness, access to finance, influencing the enabling environment, innovation grant disbursements, and anticipated outcomes and cross cutting themes), including identifying successful and unsuccessful approaches leading to the achievement of its objectives. In addition, any evidence that suggests AMDe may be under- or over-reporting results against its indicators should be identified. 2. To what extent has this project contributed to gender equity in terms of access to credit, access to capacity building support, access to and adoption of improved inputs and technologies, and income increase as the result of incremental sales of agricultural commodities? Are there evidences supporting positive changes in the aforementioned areas? 3. What has been AMDe’s contribution to the improvement of Nutritional Status of Women and Children? What are the evidences of that? 4. What is the impact of the resources spent (financial + human) and performance per value chain, i.e., what is the relative value generated in terms of productivity and income increase, employment generation per value chain to resources spent? (i.e., Which value chains have the highest returns per dollar spent?) B. Project Design and Management 1. Which among the following AMDe’s partnerships have been the most effective in terms of their collaboration and coordination to implement AGP-AMDe? Which ones have been the least effective? In both cases, what is contributing to these partnerships’ success and challenges? a. Other AGP program components implemented by government agencies b. Other AGP program partners: both GoE ( this includes Ministry of Agriculture, Ministry of Trade, Ministry of Industry, Federal Cooperative Agency, etc.), ATA, and other donors c. Other Feed the Future projects, especially LMD, ENGINE, and GRAD C. Sustainability 1. Which of the components and/or project activities can easily be scaled up in the future based on measurable, practical, and sustainable results:  Enhancing agricultural productivity  Access to finance  Access to markets D. Management 1. Given the findings in the above questions, does the Project have the right balance of staff and funding given activity priorities? Is there an appropriate balance between the resources (staff and budget), their management, and the activities the project intends to accomplish? 10 VI. EVALUATION METHODS A recommended evaluation design and data collection methods are presented below. However, the evaluation team will be responsible for refining the design as recommended below or proposing an alternative design for consideration and approval by USAID. The evaluation strategy and data collection methodologies should include mixed methods for better triangulation and validation of findings. The team should present an evaluation questions matrix showing the source of data, method of data collection, and also the tool to be used to answer each of the evaluation questions. The methodology will be presented as part of the draft work plan as outlined in the deliverables below, approved by USAID/Ethiopia and included in the final report. The evaluation team will have available for their analysis a variety of program implementation documents, baseline surveys, and reports. Methodology strengths and weaknesses should be identified, as well as measures taken to address those weaknesses. All data collected and presented in the evaluation report must be disaggregated, as appropriate, by gender, geography, and value chain. (a) EVALUATION DESIGN It is recommended that a non-experimental design be used that will focus on measuring project results before and after project implementation using project monitoring and survey data (primary and secondary data). The before-project data should be drawn from the baseline survey report produced by the implementing partners. The after-project implementation data should consider both primary and secondary information and need to be both qualitative and quantitative in nature. The evaluation team is expected to strengthen this design to make it as rigorous as possible or propose alternative evaluation designs for consideration. (b) DATA COLLECTION METHODS As stated above, the evaluation team will be responsible for proposing an appropriate evaluation design and data collection methods.The team should consider mapping the research questions against the quantitative and qualitative data in a matrix/table to show how each research question will be answered. However, it is also recommended that the data collection methods should include the following: Use of quantitative data should include, but not be limited to:  Comparison of current indicator values to baseline data for select output and outcome indicators  Map out the project results against performance measure indicators to show the total number of indicators under each result and whether performance is met/on target (90–100%), exceeded (>100%), or not achieved (