Photo by Habibul Haque/USAID Feed the Future Global Performance Evaluation Report December 2016 Prepared for the United States Agency for International Development under USAID Contract Number AID-OAA-I-15-00019/ AID-OAA-TO-16-00003. This publication was prepared independently by: Lee Briggs, Evaluation Team Leader; Patricia Vondal, Senior Evaluation Specialist; Charu Vijayakumar, Evaluation Specialist; Michael Maxey, Senior Technical Advisor; Allyson Bear, Senior Technical Advisor; Derek Byerlee, Senior Research Advisor; Lauren Rosapep, Evaluation Specialist; Amun Nadeem, Evaluation Specialist; and William Fiebig, Senior Technical Advisor. USAID/Bureau for Food Security (BFS) Contact: ehogue@usaid.gov Dexis Consulting Group Contact: 1412 Eye St NW, Washington, DC, 20005 202-625-9444 evaluations@dexisonline.com Table of Contents Executive Summary ........................................................................................................................ i I. Evaluation Purpose and Evaluation Questions .................................................................... 1 II. Initiative Background ............................................................................................................. 4 III. Evaluation Methods and Limitations ..................................................................................12 IV. Findings, Conclusions and Recommendations ..................................................................19 Question 1: Adherence to and progress across the Feed the Future Results Framework across the initiative. How well have strategies and the implementation of programs followed the Results Framework and the causal pathways identified therein? ....................................................... 21 Question 2: How has the Feed the Future initiative performed against each of its eight intermediate results outlined in the Results Framework across focus countries? Does the progress made against the eight Intermediate Results to date, coupled with changes in impact-level indicators, provide evidence that Feed the Future is on track to achieve its five￾year goals of contributing to reductions of 20% in poverty and stunting? ................................... 31 Question 3a: How have value chain approaches been applied and what have been the successes and challenges to focusing resources on strategic and limited value chains. .......... 49 Question 3b: How well has the initiative focused implementation and concentrated resources in appropriate (in terms of size and agro-ecology) geographic zones? .......................................... 71 Question 3c: Are the proportions of households being reached in each ZOI sufficient to plausibly achieve the targeted impacts? ...................................................................................................... 75 Question 4: How and to what extent have Feed the Future interventions, both Mission- and centrally-managed, helped build human and institutional capacities for the agricultural and nutrition/health sectors? .................................................................................................................................... 82 Question 5: How and to what extent have Feed the Future interventions promoted gender￾inclusive agricultural sector growth and improved nutritional status of women through equitable and strategic integration of women and men in agriculture and nutrition programs? 105 Question 6: How have Feed the Future interventions integrated nutrition into value chain activities? Do results differ if nutrition objectives are an integral part of the value chain work? If so, how? .............................................................................................................................................................. 118 Question 7: What is the effectiveness of the initiative’s focus on country ownership? How well has Feed the Future fostered country ownership of the Feed the Future program in Focus Countries to support sustainability of outcomes. .................................................................................. 136 Question 8: What has been Feed the Future’s contribution to influencing and leveraging multi￾lateral institutions and initiatives, specifically: i) the G8 and the New Alliance; ii) the Comprehensive Africa Agriculture Development Program (CAADP); iii) the Global Agriculture and Food Security Program (GAFSP); and, iv) the Consultative Group on International Agricultural Research? ...................................................................................................................................... 148 Question 9: The effectiveness, relevance, and collaboration in Feed the Future’s implementation of its research strategy. How well are Feed the Future-supported research activities designed to address major global challenges and spur agricultural development across initiative Focus Countries? ................................................................................................................. 165 Question 10: How well has the initiative leveraged private sector participation to support agricultural and nutritional outcomes? ...................................................................................................... 176 Question 11a: How well is Feed the Future identifying and promoting policy reform at the national and regional level? ............................................................................................................................ 186 Question 11b: How well is Feed the Future implementing policy reform? .................................. 191 Question 11c: What kinds of regional policy interventions have been the most effective at contributing to bilateral impacts and why? .............................................................................................. 198 Question 12a: How well have Feed the Future MEL approaches achieved .................................. 201 accountability for commitments Feed the Future has made? ........................................................... 201 Question 12b: Has the Feed the Future MEL system supported improved programming and how? ......................................................................................................................................................................... 213 TABLE OF FIGURES FIGURE 1: FEED THE FUTURE RESULTS FRAMEWORK ......................................................................... 9 GRAPH 1: VALUE OF EXPORTS OF TARGETED AGRICULTURAL COMMODITIES AS A RESULT OF USG ASSISTANCE. REPORTED VALUES AND LINEAR DIRECTIONAL TREND 2011 -2015........39 GRAPH 2: NUMBER OF PUBLIC-PRIVATE PARTNERSHIPS FORMED AS A RESULT OF FTF ASSISTANCE -REPORTED VALUES AND LINEAR DIRECTIONAL TREND 2011-2015..................40 GRAPH 3: WOMEN’S DIETARY DIVERSITY: BASELINE AND INTERIM STATISTICALLY SIGNIFICANT RESULTS .............................................................................................................................................43 TABLE 1: PROGRESS ON PBS INDICATORS ...........................................................................................46 TABLE 2: DIVERSITY OF NUTRITIOUS CROPS PROMOTED AND NUMBER OF IPS PROMOTING EACH ............................................................................................................................................................130 FIGURE 2 TRENDS IN USAID/BFS FUNDING OF AGRICULTURAL RESEARCH ($ US) .....................168 TABLE 3: PRIVATE SECTOR PARTICIPATION FY 2012 – FY 2015 .......................................................180 TABLE 4: CONTRIBUTING FACTORS AND BARRIERS TO EFFECTIVE POLICY IMPLEMENTATION194 Acronyms A3PSP Agriculture Policy Support Project aBi Agricultural Business Initiative ACE Agriculture Commodity Exchange ADVANCE Agricultural Development and Value Chain Enhancement Project Africa RISING Research in Sustainable Intensification for the Next Generation AFSI L’Aquila Food Security Initiative AgN-GLEE Agriculture and Nutrition Global Learning and Evidence Exchange AGP Agriculture Growth Program AIP Agricultural Innovation Partnership AOR Agreement Officer’s Representative ARP Office of Agricultural Research and Policy at USAID/Bureau for Food Security ARS Agriculture Research Service ATT Agriculture Technology Transfer AUC AWARD BEO African Union Commission African Women in Agricultural Research and Development Bureau Environmental Officer BFS USAID/Bureau for Food Security BHEARD Borlaug Higher Education for Agricultural Research and Development BIFAD Board for International Food and Agricultural Development BMGF Bill and Melinda Gates Foundation CAADP Comprehensive Africa Agriculture Development Programme CAFTA Central America Free Trade Agreement CDCS Country Development Cooperation Strategies CGIAR Consultative Group on International Agricultural Research CI Confidence Intervals CIP Country Investment Plan CILSS (French: Comité permanent inter-état de lutte contre la sécheresse au Sahel) The Permanent Interstate Committee for Drought Control in the Sahel CIMMYT (Spanish: Centro Internacional de Mejoramiento de Maíz y Trigo) The International Maize and Wheat Improvement Center CLA Collaborating, Learning, and Adapting CMA/WCA Council of Ministers of West and Central African States COMESA Common Market for Eastern and Southern Africa COMIECO (Spanish: Consejo de Ministros de Economía de Centroamérica) Council of Ministers of Economy of Central America COOPPAV Cooperative for the Promotion of Fishing and Commerce Activities of Isambaza (Coopérative pour la Promotion de Pêche et des Activités de Vente d’Isambaza) COR CORAF Contracting Officer’s Representative (French: Conférence des responsables de recherche agronomique africains et français) Conference of African and French Leaders of Agricultural Research Institutes CRP CRSP CGIAR Research Program Collaborative Research Support Program CSISA CSO CTP CTF Cereal Systems Initiative for South Asia Civil Society Organization Center for Transformational Partnerships Child Trust Fund DA DAES DEC Development Assistance District Agricultural Extension System Development Experience Clearinghouse DFAP Development Food Aid Program DfID UK Department for International Development DHS DIME Demographic and Health Survey Development Impact Evaluation Group DQA Data Quality Assessment ECAM USAID/Central America and Mexico Regional Mission ECCAS Economic Community of Central African States ECDPM European Centre for Development Policy Management ECOWAS Economic Community of West African States EG E-HEP EPA ESF EU Economic Growth Enhanced Homestead Food Production for Improved Food Security and Nutrition Project Environmental Protection Agency Economic Support Funds European Union FANTA Food and Nutrition Technical Assistance Project FAO FED United Nations Food and Agriculture Organization Food and Enterprise Development Program (Liberia) FAS Foreign Agricultural Service FEWsNet Famine Early Warning System FFP Food for Peace FinGAP Financing Ghanaian Agriculture Project FPMU Food Planning Monitoring Unit FSIC Food Security Innovation Center FSMA Food Safety Modernization Act FSN Foreign Service National FTFMS Feed the Future Monitoring System FUNDESA (Spanish: Fundación para el Desarrollo de Guatemala) Development Foundation of Guatemala FY Fiscal Year G8 The Group of Eight Industrialized Nations, which includes the United States, the United Kingdom, Germany, Japan, Canada, France, Italy, and Russia G20 The Group of 20, a forum for international economic cooperation and decision-making. The members of the G20 are: Argentina, Australia, Brazil, Canada, China, France, Germany, India, Indonesia, Italy, Japan, Republic of Korea, Mexico, Russia, Saudi Arabia, South Africa, Turkey, United Kingdom, United States, and European Union. GAFSP Global Agriculture and Food Security Program GCC Global Climate Change GDA GDP GH Global Development Alliance Global Domestic Product Global Health GHG GHI GIF Greenhouse Gases Global Health Initiative Gender Integration Framework GIZ (German: Deutsche Geselleschaft für Internationale Zusammenarbeit GmbH) The German international development agency GLEE GNAFF GPRA Global Learning and Evidence Exchange Ghana National Association of Farmers and Fishermen Government Performance and Results Act HICD Human and Institutional Capacity Development HRSA Health Resources and Services Administration (Kenya) HUB RURAL IA (Francophone) The Rural Hub, a knowledge management resource for Western and Central Africa Institutional Architecture ICD Institutional Capacity Development ICT Information and Communication Technology IEA IEE Independent Evaluation Arrangement Initial Environmental Examination IFAD International Fund for Agricultural Development IFC International Finance Cooperation IFDC International Fertilizer Development Center IFPRI International Food Policy Research Institute IL Innovation Lab IM Implementing Mechanism InnovATE Innovation for Agricultural Training and Education IP Implementing Partner IR Intermediate Result IWYP International Wheat Yield Partnership KDAD KFIE Knowledge Driven Agricultural Development Kenya Feed the Future Innovation Engine KII KM Key Informant Interview Knowledge Management LEO Leading Economic Opportunities LSMS Living Standards Measurement Survey M4P MAD Making Markets Work for the Poor Minimum Acceptable Diet MCC Millennium Challenge Corporation MDGs MDTF M&E Millennium Development Goals Multi-Donor Trust Fund Monitoring and Evaluation MEAS Modernizing Extension and Advisory Services MEL Monitoring, Evaluation, and Learning METSS Monitoring and Evaluation Technical Support Services MIT Massachusetts Institute of Technology MOA/MOFA Ministry of Agriculture/Ministry of Foreign Affairs MPI MSME USAID/BFS Office of Market and Partnership Innovation Micro, Small, and Medium Enterprise MYS Multi-Year Strategy MYAP Multi-Year Assistance Program NAIP National Agricultural Investment Plan NARS National Agricultural Research Service NASS National Agriculture Statistics Service NBCRI Norman Borlaug Commemorative Research Initiative NEPAD New Partnership for Africa’s Development NIFA National Institute for Food and Agriculture NGO Non-Governmental Organization NORAD Norwegian Aid NPCA NEPAD Planning and Coordinating Agency NUCAFE National Union of Coffee Agribusiness and Farm Enterprises OCA Organizational capacity assessment ODA Official Development Assistance OCDI Overseas Development Institute OECD Organisation for Economic Cooperation and Development OFSP Orange flesh sweet potato OP Operational Plan OPI Organizational Performance Index OPIC OU P4I PACE Overseas Private Investment Corporation Operating Unit Partnership for Innovation Partnering to Accelerate Entrepreneurship PBS Population Based Survey PLW Pregnant and Lactating Women PMP Performance Monitoring Plan PPP PPR PROCOMIDA QDDR R Public-Private Partnership Program Performance Report Preventing Malnutrition in Children under 2 Approach Quadrennial Defense and Diplomacy Review Required R&D Research and Development REC Regional Economic Community RF RFP RiA RING Results Framework Request for Proposal Required if Applicable Resiliency in Northern Ghana ROPPA S (French: Réseau des organisations paysannes et des producteurs de l’Afrique de l’Ouest) Network of Farmers’ and Agricultural Producers’ Organisations of West Africa Standard SADC Southern African Development Community SARI The Savannah Agricultural Research Institute SBCC Social and Behavior Change Communication SIDA SME Swiss International Development Agency Small and Medium Enterprises SOW SPPM Scope of Work Bureau’s Office of Strategic Planning and Performance Management SPRING Strengthening Partnerships, Results and Innovations in Nutrition Globally Project SPS Sanitary and Phytosanitary Standards SRF Strategy and Results Framework Sub-IR Sub-Intermediate Result SWAPs TA UN USADF Sector Wide Approaches Technical Assistance United Nations U.S. African Development Foundation USAID U.S. Agency for International Development USDA USGS U.S. Department of Agriculture U.S. Geological Survey UEMOA (French: Union Economique et Monétaire Ouest-Africaine) West African Economic and Monetary Union USG U.S. Government USTR VC U.S. Trade Representative Value Chain VCN Value Chains of Nutrition VSLA WASH WB WEAI Village Savings and Loans Association Water, Sanitation, and Hygiene World Bank Women’s Empowerment in Agriculture Index WECARD West and Central Africa Council for Agricultural Research and Development WFP United Nations World Food Programme WOG Whole-of-Government ZOI Zone of Influence i Executive Summary Feed the Future Background Feed the Future is a U.S. Government (USG) presidential initiative launched in 2010 that is based on a common approach emerging from the Group of Eight Industrialized Nations (G8) Summit in L’Aquila, Italy in 2009 and guided by the Rome Principles for Sustainable Global Food Security. It is a whole-of-government (WOG) initiative that leverages the work of 11 USG departments and agencies and is coordinated by the U.S. Agency for International Development (USAID) to address food security globally. The initiative is designed to sustainably reduce the prevalence of poverty and hunger in 19 focus countries. Over its first five years of implementation, the initiative has focused on reducing the prevalence of poverty and of stunting of children under five years of age – a key indicator of hunger and undernutrition – by an average of 20% in targeted Zones of Influence (ZOIs). Feed the Future programming is implemented in 19 focus countries, five regionally-based programs, three strategic partners, and various aligned countries.1 Feed the Future pursues unified strategic planning across agencies through Multi-Year Strategies (MYS) in focus countries, facilitated by a Feed the Future Coordinator at country level. As the approach matures, greater cooperation among the agencies has been evolving over the last five years, through an alignment between food security and strategies implemented by other sectors supporting development. The need for “selectivity and focus” has also shaped the initiative’s strategic approach and coordinated agencies’ efforts. In July 2016, Congress passed the Global Food Security Act,2 which reaffirmed the USG’s commitment to ending global hunger, poverty, and child malnutrition. Evaluation Purpose The purpose of this global performance evaluation of Feed the Future is to provide an assessment of the initiative’s progress and achievements to date, for the purposes of both learning and accountability. This independent evaluation addresses questions on Feed the Future’s implementation and results, including the program’s contribution to the goals of reducing poverty 1 Aligned countries receive some assistance for agricultural development, but unlike focus countries, do not have Feed the Future country strategies. The aligned countries can change from year to year, and in FY 2014 Feed the Future aligned countries included: Burma, Democratic Republic of the Congo, Egypt, Georgia, Kyrgyz Republic, Lebanon, Nigeria, South Sudan, Timor-Leste, Yemen, and Zimbabwe. 2 “Global Food Security Act of 2016,” US Congress, July 16, 2016. ii and stunting by 20% as an average across focus countries. The evaluation’s findings and conclusions inform a set of recommendations for improvement in the design and implementation of the ongoing and future efforts for global food security and nutrition. Evaluation Methods and Limitations This evaluation is a mixed methods evaluation that uses a combination of quantitative and qualitative methods and data sources. Key informant interviews (KIIs) and group interviews were conducted in Washington, D.C. and five focus countries: Bangladesh, Ghana, Guatemala, Malawi, and Uganda. Interview participants included personnel from USG agencies, staff of implementing partners (IPs), host country governments, and other relevant stakeholders. A total of 327 persons were interviewed. Surveys were sent out to all 19 focus countries, five regionally-based programs, and the 10 various aligned countries. Each mission was asked to assign up to two staff members to respond to the surveys, totaling up to 68 potential responses. A total of 28 survey responses was received from the field. Evaluation questions were addressed through a combination of data from interviews and surveys, data from the Feed the Future Monitoring System (FTFMS) and Population Based Surveys (PBS), and extensive document review. To address various threats to validity, triangulation was used throughout the research process. This triangulation included the use of multiple research methods, multiple data sources, and regular and continuous peer review of findings, analysis, and conclusions. USAID was consulted on a regular basis, including in an initial review of emerging findings and conclusions, to aid in validation of the research process and support utilization-focused analysis. Additionally, emerging findings and conclusions were shared with an External Oversight Committee. Evaluation Findings Progress on expected results The Feed the Future Initiative is performing well against each of its eight Intermediate Results (IRs), providing evidence that the Feed the Future Initiative is contributing to reductions in global poverty and hunger. Overall, performance on Implementing Mechanism (IM) indicators is moving in a positive direction. However, performance across countries and indicators varies. Nepal, Honduras, and Rwanda have moved in a positive direction for more than 90% of the indicators they reported on. Cambodia, Ethiopia, Ghana, Guatemala, Kenya, Liberia, Malawi, Mali, Senegal, Tanzania, Uganda, and Zambia also moved in a positive direction for at least 70% of the indicators they reported on. iii While FTFMS data show that most value chains are on target to meet fiscal year (FY) 2016 and/or FY 2017 outcome-level targets and that Feed the Future assisted many smallholders to expand their production of staple and micronutrient-dense crops, the sustainability of those outcomes with respect to agricultural productivity in some countries has been negatively impacted due to the effects of the recent El Niño. Effects of severe drought and flooding in focus countries around the globe have resulted in reductions of agricultural productivity in certain crops in Cambodia, Malawi, Honduras, Tanzania, and Zambia. Focus country programs in Haiti, Bangladesh, Zambia, and Cambodia report that productivity for certain key value chain crops will see reductions in the coming harvest. All four indicators measuring IR 8 (improved use of maternal and child nutrition services) displayed a positive trend in aggregate performance and overall, the portfolio reviews show that almost all focus countries are on track to meet the FY 2016 portfolio review target on nutritional impact and activity-level results. In 2015, 113 out of 19 countries achieved 90% or more of their targets for the number of children under five reached by a USG-supported nutrition program. Major achievements The Feed the Future initiative is performing well against each of its eight IRs outlined in the Results Framework (RF) across focus countries (Evaluation Question #1). Focus country strategies and implementation of those strategies show close adherence to the Feed the Future Results Framework and its causal linkages. Results of IM-level indicators, coupled with changes in impact￾level (PBS) indicators, provide evidence that Feed the Future is contributing to reductions in global poverty and hunger. For 11 of the 17 countries, there were statistically significant decreases in the prevalence of poverty in the ZOI, and for eight of the countries, there were statistically significant decreases in the prevalence of stunting in the ZOI (Evaluation Question #2). The review of MYS documents shows that strategies were developed using an interagency process involving USAID and other participating USG agencies, leveraging the WOG approach (Evaluation Question #1). All 19 focus country strategies work within geographic areas and on specific value chains, and with identified target populations, that are guided by and consistent with local governments’ stated priorities. Feed the Future regional strategies align directly with priority issues and agendas established by regional bodies, and regional missions consult with country missions 3 Bangladesh, Ethiopia, Guatemala, Kenya, Liberia, Malawi, Nepal, Rwanda, Senegal, Tajikistan, and Zambia. iv to ensure these regional strategies align with locally defined government priorities (Evaluation Question #7). FY 2014 and FY 2015 portfolio review presentations show that important sources of field-level learning, generated by recent evaluations, special studies, assessments, and surveys conducted in individual countries, are used to strengthen the effectiveness of focus country strategies and activities and to design new activities. Feed the Future has developed a highly relevant research portfolio that strikes a good balance between research on global challenges and providing relevant research findings to focus country programs. Research programs address the major value chains targeted in focus countries (Evaluation Question #9). The applied policy research conducted under the Feed the Future initiative represents a very important contribution to efforts promoting the adoption and implementation of policy and regulatory reforms by deepening the evidence base (Evaluation Question 11b). There are benefits from the concentration of resources on a few value chains; this approach appears to have helped focus and reinforce efforts (Evaluation Question #3). Based on MYSs, portfolio reviews, and several evaluations, WOG resources were largely focused on supporting consistent and complementary interventions that addressed key actors and functions along the value chains, as well as within the larger enabling environment. In general, value chains successfully engaged a wide range of private sector, civil society, and institutional actors to address constraints and stimulate strategic linkages, improve coordination, and increase efficiency. Numerous sources suggest that synergies made possible through Feed the Future investments appear to have enhanced market efficiency in general, and facilitated local and regional food trade. Additionally, Feed the Future has implemented a variety of human and institutional capacity development (HICD) activities ranging across sectors and areas, including the public sector, the agricultural research and higher education sectors, agricultural extension systems, the private sector, civil society, and the broader Feed the Future community, including USAID and implementing partners (Evaluation Question # 4). The Feed the Future initiative has effectively leveraged private sector participation in support of agricultural and nutritional outcomes (Evaluation Question #10). Both quantitative and qualitative data indicate significant increases in private sector relationships (number of public-private partnerships or PPPs), commitment (private sector investment), market access (incremental sales), engagement (private enterprises assisted), and feasibility (agricultural loans). These combined outputs signal increasing commercial growth and value chain development, which was previously identified by USAID as a key indicator of the value of private sector engagement. v Feed the Future has identified a policy approach based on supporting specific policy agendas, mutual accountability systems, and institutional architecture, and in each focus and regional mission, has identified an initial set of crucial policies and regulatory changes needed to support Feed the Future investments and achieve the initiative’s goals while aligning to support policy agendas detailed in country-owned plans (Evaluation Question #11). Initiative-wide, there has been considerable success in the adoption of policy and regulatory reforms, major actions, and measures completed to support policy implementation, but implementation of policies has been slow and uneven. The coordinated approach among interagency policy specialists – drawing on ongoing policy work and updated analysis as the basis for extensive consultations with governments and a wide range of stakeholders to rank-order the most pressing policy change needs in each focus country and region – led to agreements with governments for the finalization of Feed the Future policy agendas. Progress has been made in the adoption of reforms in all seven areas of policy changes identified by the interagency sub-committee on policy. Feed the Future’s understanding of the complex underpinnings of malnutrition has evolved substantially from the beginning of the initiative, as is evident in the development of the first USAID Multi-Sectoral Nutrition Strategy and the first USG Global Nutrition Coordination Plan. These USG documents represent agricultural interventions under Feed the Future as a part of a larger and more complex USG response to malnutrition (Evaluation Question #6), which was groundbreaking and influential beyond the USG. The first five years of Feed the Future have been a significant learning period for the USG – and in particular, USAID – in how to design and implement multi￾sectoral nutrition programming. This learning curve is reflected in the quality of programming during this period. Feed the Future has modeled approaches to accountability focused on rigorously defined metrics and common reporting, and the joint reporting occurring under the FTFMS contributes to solidifying the WOG approach (Evaluation Question #12). Systems and processes that coordinate annual reviews of Feed the Future portfolio performance, and the reporting of performance data used for annual progress reports from USAID and other USG agencies, have been institutionalized across all operating units where the initiative is implemented. Additionally, the introduction of the innovative Women’s Empowerment in Agriculture Index (WEAI) has increased the scope of incorporating and tracking gender considerations in Feed the Future programming (Evaluation Question #5). The process for conducting portfolio reviews has developed over time, becoming more streamlined and highlighting key issues and learning. Global Learning and Evidence Exchanges (GLEEs) have been convened to disseminate learning and best practices. These diverse vi venues for regular engagement have increased awareness among WOG actors, facilitated their engagement, and increased opportunities for even greater collaboration. Obstacles 1. Operationalizing integration between agriculture and nutrition Although Feed the Future has dual goals of reducing poverty and undernutrition (measured through reduction of underweight and stunting), there is a lack of clarity on what constitutes desirable programmatic integration of agriculture and nutrition. The RF delineates discrete causal pathways that lead to each of the two goals, with only one IR overlapping between both the agriculture and nutrition sides. The agriculture side of the RF is much more developed, both in terms of sub-IRs as well as indicators, which is reflected by the fact that most IPs have a stronger focus/emphasis on agriculture. The funding for agriculture was much larger than for nutrition, thereby affecting Feed the Future’s ability to integrate programs. Since the development of the RF, Feed the Future has invested in the production of the “Understanding and Applying Primary Pathways and Principles”4 in 2014, and newer Feed the Future activities should reflect this latest thinking. Although IPs often incorporate a combination of agriculture and nutrition elements in their activities, the nature of the combination varies across countries and contexts. It is unclear whether and which of these variations, such as the layering of multiple focused activities in a ZOI or a single activity working across both sectors, achieves the synergistic “integration” envisioned by the initiative. A lack of facilitation for collaboration leads to missed opportunities to leverage greater impact. Additionally, there is a need to reconcile the widely held perception that the target populations for most agriculture growth and nutrition interventions are different segments of the population with the push for integration that is often interpreted to assume a homogeneous population resulting in artificially integrated activities. 2. Varied range of understanding of / approaches to value chain activities There is a lack of cohesiveness underpinning the wide and varied range of activities identifying with value chain approaches, across the initiative as well as within focus countries. Given the fluid bounds of the definition of value chains, the term is often used as more of a catch-all phrase than a programmatic guiding element. Although the selection of value chain crops provides parameters, there is a lack of clarity on what it means to take a value chain approach. That lack of clarity means 4 Herforth, Anna, and Jody Harris. 2014. Understanding and Applying Primary Pathways and Principles. Brief #1. Improving Nutrition through Agriculture Technical Brief Series. Arlington, VA: USAID/Strengthening Partnerships, Results, and Innovations in Nutrition Globally (SPRING) Project. vii the various interventions may not be working together cohesively and strategically to address the various links in the value chain, including affecting the tensions in targeting socio-economic segments of the population more suited to involvement along the different points in the value chain. There is a perceived lack of alignment between working within higher levels of the value chain, systems-level and/or policy work that cannot be effectively bounded to a limited set of value chain crops, and the geographic boundaries of ZOIs. Methods to operationalize the value chain approach can vary, from strategically positioning different activities along various points in the value chain to one activity working across multiple points on a value chain, or some interpretation in between. 3. Aligning inclusivity with activity objectives There is a lack of a clear framework or common understanding of how to effectively and systematically incorporate the range of vulnerable populations throughout the agriculture sector growth portfolio. While the Resilience Policy and Program Guidance provides direction, there are often competing incentives, driven by indicator results, to prioritize sales and production-focused objectives over inclusivity measures unless the activity is designed specifically to target vulnerable populations. Person-specific indicators are disaggregated by gender, but not necessarily with women, youth or other vulnerable group-based targets, which incentivize inclusion. Given that vulnerable populations are often not well-positioned to immediately link to higher levels of the value chain, their incorporation necessitates addressing constraints and longer timelines that are not particularly conducive to short-term results/outputs that align with activity objectives or periods of performance. 4. Disconnect between systems level work and reporting requirements While Feed the Future supports systems-level activities, indicators and reporting requirements are output-focused, presenting a disconnect between a push to affecting systemic change to facilitate agriculture sector growth and improvement of nutritional status and incentives to produce output heavy results. Detecting and assessing change at the systems level often requires longer timelines and non-traditional tools and approaches that do not necessarily align with the current programming and reporting structure. Recommendations 1. Provide clear guidance on integrating agriculture and nutrition Given the global scope of Feed the Future, the integration of agriculture and nutrition should inevitably manifest in a variety of ways to maintain applicability within the local context. However, it would be beneficial to develop a common understanding across the initiative of the most viii impactful ways to achieve meaningful integration of agriculture and nutrition. This could take the form of a framework that lays out the various models of integration, highlighting preferred models within different contexts and Feed the Future’s understanding of how integrating agriculture and nutrition affect the various target populations. It would be important for this work to be part of the learning and research agenda in the future. This guidance would provide missions and IPs with conceptual and programmatic parameters with which they can easily and clearly identify, while maintaining the flexibility to adapt to local contextual needs. Current funding levels of Feed the Future restrict its ability to implement some of this work. 2. Revise monitoring systems to capture and reward systems-level work In order to incentivize and track systems-level activities and interventions, the monitoring systems and associated elements such as reporting obligations and timelines must be revised to include non-traditional and appropriate measurement and learning tools. To capture outputs and outcomes defined by elements such as complexity, inter-connectedness, facilitation, and indirect beneficiaries, reporting processes should allow for the flexibility to communicate results through tools such as directional and perception-based indicators and social network analysis. 3. Reconcile systems-level work with selectivity and focus of value chains and ZOI Feed the Future should provide initiative-wide guidance to reconcile the tension between the selectivity and focus approach promoted by the selection of value chain crops and ZOIs and the promotion of the systems-level work that is by nature difficult to limit within geographic boundaries and single commodities. Official guidance should delineate the areas within both approaches where there can be more flexibility for programming and areas that need to be more rigid, to allow IPs to identify and align interventions to most effectively meet Feed the Future goals. 4. Encourage synergistic collaboration at the country level Feed the Future should foster synergistic collaboration among the IPs within a country through a USG/Mission-led process or dedicated mechanism that is mandated to identify and capitalize on areas of overlap to amplify program impacts. By design, Feed the Future activities include a wide variety of intervention types, target populations, approaches, and sectors with significant overlap to achieve activity specific and larger initiative objectives. While there are instances of effective collaboration occurring in various focus countries and contexts, due to time, budget, and reporting constraints, most IPs are not necessarily well-suited to effectively collaborate beyond sharing lessons learned and coordinating on a short-term basis. Such a facilitation mechanism should also leverage whole-of-government resources. 1 I. Evaluation Purpose and Evaluation Questions Evaluation Purpose The purpose of this global performance evaluation of Feed the Future is to provide an examination of the initiative’s progress and achievements to date, for the purposes of both learning and accountability. The evaluation addresses questions on Feed the Future’s achievements since 2009, how it is being implemented, and whether or not expected results are occurring. While this is not an impact evaluation, it also examines the extent to which programming and implementation have contributed to the initiative’s goals of reducing the prevalence of poverty and undernutrition by an average of 20% in the areas where it has worked. To ensure the quality and success of any future Feed the Future programming, it is necessary to understand how Feed the Future programs are functioning in country settings. Therefore, it is also necessary to understand if programs are achieving targeted results in their focus regions; how successful approaches are scaling up across broader communities and regions to achieve high level goals of sustainably reducing global poverty and hunger; and if there are any issues or challenges or common problems that should be understood and addressed to ensure success in future programming. However, it is important to note an inherent limitation that conditions the ability of Feed the Future to achieve results – and more importantly to demonstrate achievement – and this is the inescapable "lag" that is inherent in many elements of the approach. This includes not only time lags from design phase to the start of activities, but also lags in results, even when activities have been flawlessly implemented. These time intervals between project design and changes in the wider system often – probably usually – take more than five years. Given this, many of the results of Feed the Future are only partially evident, or the evidence available for this evaluation may in fact be out of date. The findings and conclusions from this performance evaluation, which examine Feed the Future performance across the years 2009 – 2016, will provide opportunities for learning which approaches and implementation actions work well – and which ones do not. These will inform a set of recommendations for improving the design and implementation of the next five-year iteration of the Feed the Future initiative. This evaluation will be used to support resource allocation decisions and to provide information to Feed the Future partners that they can adapt to modify projects and activities in ways that will improve their effectiveness in specific country settings. The key audiences for this report include the U.S. Agency for International Development (USAID) and other participating U.S. Government (USG) agencies; USAID missions, as well as their government counterparts and implementing partners (IPs) in Feed the Future countries; the U.S. Congress; the Office of Management and Budget; other bilateral donors; and multilateral institutions and other initiatives addressing food security, hunger, poverty, and malnutrition. 2 Evaluation Questions This global performance evaluation is intended to assess the Feed the Future initiative’s progress against its strategic goals and objectives. The Evaluation Questions covered by the evaluation team are as follows: 1. Adherence to and progress across the Feed the Future Results Framework (RF) across the initiative. How well have strategies and the implementation of programs followed the Results Framework and the causal pathways identified therein? 2. How has the Feed the Future initiative performed against each of its eight intermediate results (IRs) outlined in the Results Framework across focus countries? Does the progress made against the eight IRs to date, coupled with changes in impact-level indicators, provide evidence that Feed the Future is on track to achieve its five-year goals of contributing to reductions of 20% in poverty and stunting? 3. The implementation and effectiveness of the “selectivity and focus” approach within given country or regional portfolios. a. How have value chain approaches been applied and what have been the successes and challenges to focusing resources on strategic and limited value chains? b. How well has the initiative focused implementation and concentrated resources in appropriate (in terms of size and agro-ecology) geographic zones? c. Are the proportions of households being reached in each Zone of Influence (ZOI) sufficient to plausibly achieve the targeted impacts? 4. How and to what extent have Feed the Future interventions, both Mission- and centrally-managed, helped build human and institutional capacities for the agricultural and nutrition/health sectors? 5. How and to what extent have Feed the Future interventions promoted gender-inclusive agricultural sector growth and improved nutritional status of women through equitable and strategic integration of women and men in agriculture and nutrition programs? 6. How well have Feed the Future interventions integrated nutrition into value chain activities? Do results differ if nutrition objectives are an integral part of the value chain work? If so, how? Using the Feed the Future Guide as a reference point, assess the performance and management of the Feed the Future initiative as a whole, specifically, in terms of relevance, efficiency, effectiveness, and sustainability. Consider the following Evaluation Questions: 3 7. The effectiveness of the initiative’s focus on country ownership. How well has Feed the Future fostered country ownership of the Feed the Future program in focus countries to support sustainability of outcomes? 8. What has been Feed the Future’s contribution to influencing and leveraging multilateral institutions and initiatives, specifically i) the G8 and the New Alliance; ii) the Comprehensive Africa Agricultural Development Program (CAADP); iii) the Global Agriculture and Food Security Program (GAFSP); and iv) the Consultative Group on International Agricultural Research (CGIAR)? 9. The effectiveness, relevance, and collaboration in Feed the Future’s implementation of its research strategy. How well are Feed the Future-supported research activities designed to address major global challenges and spur agricultural development across initiative focus countries? 10. How well has the initiative leveraged private sector participation to support agricultural and nutritional outcomes? 11. How well is Feed the Future promoting policy reform at the national and regional level, including implementation of policy reform? 12. How well have Feed the Future monitoring, evaluation, and learning (MEL) approaches achieved accountability for commitments Feed the Future has made? Has the Feed the Future MEL system supported improved programming and how? 4 II. Initiative Background Feed the Future is a presidential initiative launched in 2010, and is the USG’s contribution to the intensified global effort focused on reducing high rates of hunger and poverty around the world. This effort is based on a common framework that was developed as an outcome of the 2009 Group of Eight Industrialized Nations (G8) and Group of 20 (G20) Summits. This framework serves as a blueprint for coordinated and comprehensive action among host governments, donors, civil society, private sector, and other stakeholders. The framework is comprised of five principles first articulated at the G8 Summit in L’Aquila, Italy in 2009 to embrace and support the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action. These principles were adopted unanimously by 193 countries at the 2009 World Summit on Food Security as the Rome Principles of Sustainable Food Security and were intended to accelerate progress toward the United Nations (UN) Millennium Development Goals (MDGs) of halving the proportions of people living in extreme poverty and hunger by 2015. The Rome Principles call on stakeholders to: 1) Invest in country-owned plans (also known as Country Investment Plans, or CIPs, and Implementation Roadmaps) that support results￾based programs and partnerships between host country governments and other key actors; 2) Strengthen strategic coordination; 3) Ensure a comprehensive approach; 4) Leverage the benefits of multilateral institutions; and 5) Deliver on sustained and accountable commitments. These key principles are still evident in the form and function of Feed the Future (originally called the Global Hunger and Food Security Initiative). These include a focus on Managing for Development Results, an emphasis on “education, research, science and technologies,” along with principles of country ownership, donor coordination, and working with multi-lateral institutions.5 There were a number of additional factors that contributed to the genesis of Feed the Future, conditioned the initial design, and influence operations to this day. To begin, the USG sent a multi￾agency delegation to L’Aquila – which was unusual in itself; this occurred in the very early days of the presidency of Barack Obama, in effect capturing the attention of the Office of the President. This multi-agency diplomatic engagement at L’Aquila was due to a recognition at that time that food security was a very complex challenge that had grave potential consequences (as evidenced by the world food price crisis), and as such required specialized expertise from across numerous government agencies. The L’Aquila Summit also produced similar whole-of-government (WOG) engagement with other G8 member countries, spawning the involvement of Ministers of Agriculture in international and diplomatic forums around the globe. Because of the high-level spotlight that L’Aquila shone on the issue, and due to multi-agency mobilization and engagement 5 “L’Aquila Food Security Initiative (AFSI). 2012 Report.” http://www.state.gov/documents/organization/202922.pdf. 5 in this diplomatic process, various senior leaders at cabinet level also saw roles for their agencies with greater clarity: these included Secretary of State Hillary Clinton, who appointed Cheryl Mills to lead the process at the Department of State; Tom Vilsack, the Secretary of Agriculture; leaders at USAID (initially acting administrator Alonzo Fulgham, but followed by Rajiv Shah, who had come from the U.S. Department of Agriculture (USDA) and had strong connections there); and Gayle Smith, then sitting on the National Security Council. There was a concerted push to elevate the initiative above the individual interests of the various agencies involved, and clear roles were established to ensure coordination at high levels: the Coordinator of Feed the Future, a position represented by the USAID Administrator; the Deputy Coordinator for Diplomacy, who sits at the Department of State and is charged with managing the diplomatic aspects of Feed the Future; and the Deputy Coordinator for Development, who sits in USAID/Bureau for Food Security (BFS), charged with managing delivery of foreign assistance funds. In late 2007 the World Bank (WB) issued a World Development Report focused on agriculture, that was produced through wide consultation with other donors, non-governmental organizations (NGOs), academia, etc., and thus had great authority. The central messages of the report were that renewed investment in agricultural growth is essential for poverty reduction, food security, and environmental sustainability, and that agricultural growth is much more effective in reducing poverty than non-agricultural growth. The Bill and Melinda Gates Foundation (BMGF) was also getting started on a large new initiative on agriculture, and were looking to justify their investments, and they relied heavily on the WB report. At about the time the report was published, world food prices started spiking, ultimately resulting in the world food crisis of 2008. The NGO community and other interest groups in the United States were also very aware of the scope of the challenge, and were putting domestic pressure on the USG to reform a disjointed and sprawling approach to funding countless programs across multiple agencies that all addressed some element of food security assistance, but lacked a coherent or unified policy strategy. Ultimately, in order to streamline food security funding and programs at USAID, diverse streams of funding and operational approaches were integrated under one bureau created for this purpose: the Bureau for Food Security. New resources were also devoted to food security, and these were primarily provided to USAID. The first Quadrennial Defense and Diplomacy Review (QDDR), conducted in 2010, highlighted three key initiatives for the USG: Feed the Future, Global Health (GH), and Global Climate Change (GCC). Furthermore, the QDDR responded to the Paris Declaration on aid effectiveness, and included key themes such as the use of science and technology, country ownership, a focus on results, and elevation of economic growth. Regular meetings among the US Agencies participating on Feed the Future in these early days further reinforced the idea of and supported recognition of the necessity for a shared effort. At a 6 fundamental level, according to numerous interviewees, the approach worked. Feed the Future has pursued unified strategic planning through interagency involvement in the 2010/2011 development of Multi-Year Strategies (MYSs) and 2012 country/region-specific policy agendas. This approach is driven by and facilitated by USAID at country level, but is not owned by USAID; USAID has a dominant role – as opposed to the normal jockeying for position and resources that typically occurs among near-peer agencies – but the role is primarily one of convening. It appears that five years on, for a number of reasons, there has been a certain loss of focus on the clear need for coordination, and perhaps a lack of enthusiasm has emerged. However, many components of Feed the Future clearly play to the strengths of the different agencies, therefore the commitment to coordination varies across the individual components. It appears that where shared interest is strong, coordination remains robust, but steady management attention and strong leadership is required to solidify joint action. Feed the Future is an initiative in which there has been a real framing of exactly where each agency truly has a comparative advantage, and thus can work together in a cohesive manner, and the scope of the initiative requires flexibility and recognition of how varied approaches other than providing development assistance are required. This recognition of how comparative advantage should be mobilized to address shared objectives under Feed the Future creates pressure for meaningful joint planning, and coordination that goes beyond simply sharing information, and this has resulted in a shifting of what agencies do to maximize their contribution according to their unique comparative advantage. As one example, USDA created annual strategies for the first three years of the initiative that fed into the Global Food Security Action plan. They contributed value by mobilizing expertise to work with national agricultural data, through engagement of the National Agriculture Statistics Service (NASS), and research through the Agricultural Research Service (ARS); they also provided support through the Foreign Agricultural Service (FAS) that aligned with trade￾related objectives. Because USDA is primarily appropriated funds for domestic agriculture, this required some budget shifts. In Bangladesh, Haiti, and Guatemala, numerous activities across agencies were occurring that aimed to address food insecurity to some degree. These included, in addition to agricultural development programming through the BFS, USAID’s Food for Peace (FFP), USDA Food for Progress, the McGovern-Dole Food for Education program, and other Title II programs. Joint assessments were conducted and an integrated approach was devised that involved layering of these programs in target areas, and an approach to joint-governance manifest in memoranda between the actors – which had never occurred before. A similar approach occurred within the Department of State as they worked with various United Nations (UN) actors and the G20, with other agencies feeding into the strategic approach the Department of State took at the diplomatic level. This integrated approach, pioneered under Feed the Future, has resulted in significant learning among key WOG actors. One example of this evolving recognition of the 7 importance of a joint approach, provided in an interview, is the way the USG has developed its approach under Power Africa. At country level, there are numerous examples of WOG leveraging and joint planning, and this is occurring very successfully where it is most needed, in the policy support and reform space. There are numerous other examples: for example, Millennium Challenge Corporation’s (MCC) work with infrastructure has been augmented by capacity development activities provided by USAID. Another example is the development of a Participating Agency Program Agreement between USAID and Peace Corps, with Peace Corps receiving funding for volunteers in focus countries, who then work at a grassroots level in support of USAID program objectives – e.g., by providing training to farmers and extension agents. The Aflatoxin partnership is another example, where objectives aligned well with agency priorities, including nutrition objectives at USAID and trade and food safety objectives at USDA. This was further driven by the fact that the BMGF had recently identified aflatoxin as a major issue in developing countries, and the World Food Programme (WFP) was unable to procure local food commodities because of high levels of the toxin. As a final example, USDA has asked USAID personnel to manage its wheat research activity, but using USDA money. Greater alignment is occurring through maturation of the approach, with evolutionary strategic linkages occurring through an intentional alignment between food security and development models. The need for “selectivity and focus” in limited countries, and in ZOIs within countries, on limited objectives, is also seen by interviewees to have contributed to success. Feed the Future modeled approaches to accountability focused on rigorously defined metrics and common reporting, evolving principles that were originally defined in the Rome principles and articulated at L’Aquila and underpinned the MCC’s model; the joint reporting occurring under the Feed the Future Monitoring System (FTFMS) is one way that the WOG approach is solidified. Additionally, the process for conducting portfolio reviews under Feed the Future has developed over time, becoming more streamlined and highlighting key issues and learning. Portfolio reviews, led yearly by the Feed the Future Coordinator in each focus country, are facilitated discussions and presentations about annual progress, results, opportunities and constraints. They are a common platform to share results and adapt programming to the changing landscape. Global Learning and Evidence Exchanges (GLEEs) have been convened to disseminate learning and best practices on Nutrition, Food Security, Gender, and Climate-Smart Agriculture, in a programmatic support mode aimed at improving knowledge and practice among WOG partners. Through these various venues for regular engagement, greater clarity around operational modalities, incentives, and operational priorities and comparative advantage have emerged. Over time, this increased awareness among WOG actors has further facilitated WOG engagement and increased opportunities for greater collaboration. 8 Feed the Future encompasses the work of 11 USG Agencies, led and coordinated by USAID’s Bureau for Food Security. Though it was conceived as a presidential initiative, Feed the Future will have institutional longevity beyond the end of the Obama Administration. In July 2016, Congress passed the Global Food Security Act,6 which reaffirmed the USG’s commitment to ending global hunger, poverty, and child malnutrition by permanently authorizing Feed the Future in statute. Feed the Future is designed to sustainably reduce the prevalence of poverty and undernutrition. Over its first five years, the initiative has aimed to reduce the prevalence of poverty and of stunting of children under five years of age – a key indicator of hunger and undernutrition – by an average of 20% in targeted Zones of Influence by tackling root causes, focusing resources, and employing proven strategies for achieving large scale and lasting impact. Feed the Future programming is implemented in 19 focus countries, five regionally-based programs, and various aligned countries.7 Feed the Future also funds cooperative agricultural development programs with three “strategic partner” countries: Brazil, India, and South Africa. Across all of these contexts, the overarching intervention approach, which is laid out in the Feed the Future Guide, is based on an assessment of the drivers of food insecurity, poverty, and malnourishment in developing countries around the world where poverty and malnutrition levels are the highest. An additional distinguishing feature of the initiative is its WOG approach, which is designed to leverage existing U.S. international development programs to create synergies and cross-agency coordination in pursuit of Feed the Future’s high-level goals. The Feed the Future Results Framework depicts a theory of change through a hierarchy of results that represent the causal pathways that initiative stakeholders undertake to achieve sustainable reductions in poverty and hunger (see Figure 1: Feed the Future Results Framework). 6 “Global Food Security Act of 2016.” PL 114-195, Jul. 20, 2016. https://www.congress.gov/bill/114th￾congress/senatebill/1252?q=%7B%22search%22%3A%5B%22global+food+security+act%22%5D%7D&res ultIndex=1 7 Aligned countries receive some assistance for agricultural development, but unlike focus countries, do not have Feed the Future country strategies. The aligned countries can change from year to year, and in FY 2014 Feed the Future aligned countries included: Burma, Democratic Republic of the Congo, Egypt, Georgia, Kyrgyz Republic, Lebanon, Nigeria, South Sudan, Timor-Leste, Yemen, and Zimbabwe. 9 Figure 1: Feed the Future Results Framework The 19 Feed the Future focus countries were selected based on five criteria: 1) Level of need, 2) Opportunity for partnership, 3) Potential for agricultural growth, 4) Opportunity for regional synergy, and 5) Resource availability. The 19 focus countries and five regional programs are identified in the text box at right. According to the Congressional Research Service, USAID invested approximately $4.7 billion in direct food security and agricultural development activities under Feed the Future from FY 2010 through FY 2014. During the same period, USAID invested an additional $2.3 billion for nutrition-specific programming through the Global Health Programming and Food for Peace Title II Development Aid. Other federal agencies participating in Feed the Future are estimated to have invested as much as $4.3 billion in development activities related to the initiative.8 19 focus countries: Bangladesh, Cambodia, Ethiopia, Guatemala, Ghana, Honduras, Haiti, Kenya, Liberia, Malawi, Mali, Mozambique, Nepal, Rwanda, Senegal, Tajikistan, Tanzania, Uganda, and Zambia. Five regional programs: Asia, Central America and Mexico, East Africa, Southern Africa, and West Africa. 8 Lawson, Marian L., Randy Schnepf, and Nicolas Cook. “The Obama Administration’s Feed the Future Initiative.” Congressional Research Service. January 29, 2016. p. 20. 10 Six key approaches emerge from a review of Feed the Future strategic material: ● Climate-smart development: Feed the Future is helping governments, researchers, and farmers assess the potential vulnerabilities and impacts of climate change, sustainably intensify their production,9 develop and deploy climate-smart technologies and management practices, and support policies that encourage the necessary investments and enabling environments to mitigate risk, improve resilience, and increase food security despite changing climate patterns. ● Gender integration: Women’s access to land and other key productive resources can be limited, and women rarely have legal control over the land they farm. There is consistent and compelling evidence that when the status of women is improved, agricultural productivity increases, poverty is reduced, and nutrition improves. Feed the Future integrates gender-based analysis into all of its investments and employs a new tool called the Women's Empowerment in Agriculture Index (WEAI),10 which measures the empowerment, agency, and inclusion of women in the agriculture sector in an effort to identify ways to overcome obstacles and constraints. ● Improved nutrition: Feed the Future works with a host of partners, including the Global Health Programming11 and USAID Office of Food for Peace, 12 to help families improve their nutrition by providing adequate nourishment for pregnant and lactating women and their children, especially in the critical period from pregnancy to a child’s second birthday, when poor nutrition can have lifelong adverse effects. ● Inclusive agriculture sector growth: Feed the Future strives to increase agricultural production and the incomes of both men and women in rural areas who rely on agriculture for their livelihoods. Investments in inclusive agriculture-led growth encompass improving agricultural productivity, expanding markets and trade, and increasing the economic resilience of vulnerable rural communities. ● Private sector engagement: Feed the Future engages the private sector to meet the global food security challenge. These partnerships aim to advance the impact of sustainable development and foster private sector-led growth in emerging markets. Feed the Future invests in creating enabling policy environments and physical infrastructures that facilitate private sector investment. 9 For more information, see: https://feedthefuture.gov/resource/feed-future-food-security-innovation￾center-sustainable-intensification. 10 For more information, see: https://feedthefuture.gov/lp/womens-empowerment-agriculture-index. 11 For more information, see: https://www.ghi.gov/results/focus_areas/nutrition/nutrition.html#.V9LoFigrK01. 12 For more information, see: https://www.usaid.gov/what-we-do/agriculture-and-food-security/food￾assistance. 11 Feed the Future serves as the principal vehicle through which the United States contributes to the New Alliance for Food Security and Nutrition, 13 a global partnership between African governments, donors, and the private sector to increase investment in agriculture and reduce poverty. ● Research and capacity building: Feed the Future supports research to develop solutions to enhance agricultural production, with an emphasis on improving nutrition and reducing adverse impacts on natural resources and the environment. The Feed the Future Research Strategy14 supports a global research agenda focused on creating more productive crops, sustainably intensifying agricultural production systems, ensuring food security, and enhancing access to nutritionally improved diets. Implementation of the research strategy is achieved through numerous funding streams that advance research in support of Feed the Future goals: including support for 24 Feed the Future Innovation Labs (ILs) led by U.S. universities, and an annual investment in the Consultative Group on International Agricultural Research. Across the Feed the Future programmatic areas, the U.S. Government is making core investments that focus on agricultural sector growth and improved nutritional status. Areas of investment include diet quality and diversification, post-harvest infrastructure, high-quality agricultural inputs (such as seed and fertilizer), and financial services. Regardless of the investment area, investments emphasize the provision of technical assistance and development or improvement of accountability systems. According to the Feed the Future Guide, investments are, overall, striving to: ● Support the food security priorities of partner countries and help build capacity for sustainable development ● Promote collaboration at the U.S. and international levels ● Empower women ● Embrace innovative partnerships with the private sector, civil society, and the research community ● Foster policy environments that enable private investment ● Advance climate-smart agriculture through research and innovation ● Integrate agriculture and nutrition, with a particular focus on mothers and children ● Maximize cost-effective results that create the conditions where assistance is no longer needed 13 For more information, see: https://feedthefuture.gov/lp/new-alliance-food-security-and-nutrition. 14 For more information, see: https://www.feedthefuture.gov/resource/feed-future-research-strategy. 12 III. Evaluation Methods and Limitations This USAID performance evaluation was implemented between February and December 2016. The evaluation is a mixed methods evaluation that uses a combination of quantitative and qualitative methods and data sources to address each question. A full description of the evaluation design and methodology is included in Annex II of this report. The evaluation began with a review of literature that relates to Feed the Future. USAID directly provided the evaluation team with many of the documents that were included in this review; additional documents were sourced from the Feed the Future Website, Agrilinks, and a wider Internet search. In parallel with this ongoing literature review, the evaluation team conducted key informant interviews (KII) and group interviews in Washington, D.C. and in five Feed the Future focus countries: Bangladesh, Ghana, Guatemala, Malawi, and Uganda. Interview participants included USAID personnel, other USG implementing agencies, representatives of various implementing partners, and other relevant stakeholders. Interviews conducted with Washington￾based participants were both face-to-face and via telephone. Evaluation team members also completed field visits to five purposively-selected focus countries to conduct in-person interviews with implementing partners, USAID mission staff, and other stakeholders. These other stakeholders included representatives from other participating USG agencies, bilateral and multilateral donors, and focus country governments. A total of 325 persons were interviewed: 101 in KIIs, and 224 in 64 separate group interviews. Additional field perspectives were obtained using three online surveys. Survey links were sent to participants in the 19 focus countries, five regional missions (or programs), and all 11 FY 2014 aligned countries via SurveyMonkey. Most of these participants were invited to complete the survey through an email sent out by the Assistant Administrator of the Bureau for Food Security, which contained a weblink to the survey. A total of 28 survey responses were received from the field. The evaluation team used mixed methods data analysis procedures to develop findings for each evaluation question. In most cases, the team addressed questions through a combination of FTFMS data analysis, review of documents germane to the specific research question, the analysis of quantitative and qualitative data from the online surveys, and the thematic analysis of key informant and group interview data. A matrix of the diverse data sources used in this evaluation is provided in Annex IV. a. FTFMS Data 13 The primary source of quantitative data used for this evaluation is performance monitoring data extracted from the Feed the Future Monitoring System database. The FTFMS is part of an interagency effort to consolidate USG reporting on Feed the Future activities. Reporting for Feed the Future indicators began in 2011. These indicators were designed to measure progress at each level of the Feed the Future Results Framework. The FTFMS database was used to assess performance to-date compared to targets, based on data reported for Feed the Future indicators, and to identify trends in each focus country and across focus countries. b. Population Based Survey (PBS) Data PBS data are collected in ZOIs selected within each focus country. The ZOIs represent specific geographic areas within a country where Feed the Future resources and interventions are being focused. Each focus country is responsible for impact-level results only within the ZOIs they have selected. Baseline data were collected after the ZOIs were established. Interim surveys were carried out approximately three years after baseline; the majority of interim surveys were conducted in FY 2015 to enable measurement of progress toward impact-level indicator targets, to measure achievements against goals and objectives related to poverty and nutrition, and to demonstrate progress toward some outcome-level indicator targets at the IR level. At the time that this evaluation was conducted, a subset of PBS data was available for analysis; each of the indicators available included data from four to 10 focus countries. c. Document Review The complete list of the documents reviewed is attached in Annex IV. The team used documents provided by USAID/BFS, Feed the Future representatives in participating USG agencies, and documents the evaluation team identified and downloaded from the Feed the Future website. Among the wide variety of documents used for this evaluation were the Feed the Future guidance documents, Multi-Year Strategy (MYS) documents, portfolio reviews, annual reports, the Feed the Future Policy Guide, policy matrices, and Country Investment Plans. Additional documents were collected for each of the five Feed the Future focus countries selected for field work, including Country Development Cooperation Strategies (CDCS), Performance Monitoring Plans (PMPs), activity assessments and evaluations, and project activity reports from USG agencies conducting programming in these countries. The team also searched the Agrilinks and Development Experience Clearinghouse (DEC) websites to identify impact and performance evaluation and assessment reports as well as other information products relevant to the BFS Research Agenda and the Feed the Future Learning Agenda. d. Online Surveys 14 The team designed three online surveys to collect additional initiative-wide data on topics of interest to this evaluation from the perspectives of focus, regional, and aligned country programs. The evaluation research questions were used to establish the frame of response categories for the survey instruments. Links to these surveys were sent directly by USAID/BFS in an email request to the designated Feed the Future coordinators or points of contact in all 19 focus countries, the five regional missions, and 11 aligned countries. In the event that Feed the Future coordinators felt that the survey could be better answered by other team members (such as a monitoring and evaluation (M&E) specialist or subject-specific point of contact), they had the option of designating up to two additional members of their team to respond to the survey. Among the 35 country and regional missions invited to participate in the survey, we ultimately received survey responses from 18 of the invited missions (13 focus countries, two aligned countries, and three regional missions). Six missions (four focus countries, one aligned and one regional mission) opted to have more than one staff member respond to the survey. In all, we received a total of 28 individual survey responses. In addition to the country and regional surveys, online elicitation techniques were used to obtain information from Feed the Future Innovation Lab directors who were unable to be interviewed in￾person due to scheduling and time constraints. A total of 18 IL directors were e-mailed the list of questions about research funded by USAID under the Feed the Future Research Strategy and were requested to respond in writing to these questions. A total of 10 responses were received from IL directors. e. Key Informant and Group Interviews Key informant and group interviews were the primary means by which the evaluation team obtained qualitative data to explore the perspectives of stakeholders involved in the design, management, or implementation of the Feed the Future initiative in Washington, D.C. and five focus countries visited by field teams.15 Broadly speaking, there were two general aims of the qualitative data collection phase of the evaluation: 1) to obtain a description of Feed the Future activities, and 2) to elicit perspectives on the effectiveness and accomplishments of these activities. Interview guides were developed to orient interview teams to the key information sought from each type of informant; the interviews themselves were relatively unstructured and relied on open￾ended questions. Interviewers aimed to facilitate a stimulating discussion of Feed the Future activities that would provide information useful for answering the specific evaluation questions. This allowed interviewees to steer the discussion to whatever seemed to be of greatest interest 15 A complete list of interviews conducted by country (including those conducted in the U.S.) is found in Annex IV. 15 from their perspective and allowed them to establish the course and content of the discussion based upon their own understanding and self-identified priorities. This design was selected to allow a broad range of perspectives and issues to emerge. In Washington, D.C., 56 key informant and group interviews with 84 individuals were conducted in￾person or by telephone with USAID, other participating USG agencies, and additional stakeholders. At USAID/Washington, group interviews were conducted with staff from each technical office in the Bureau for Food Security, the Center for Resilience, and the Global Engagement Strategy Team, and with key informants in the Office of Food for Peace and in the Global Health Bureau. Outside of USAID, key informant interviews were conducted with Feed the Future representatives from the African Development Foundation, the Department of the Treasury, the Department of Commerce, the Overseas Private Investment Corporation, the Peace Corps, and the MCC, State Department16 , and the USDA.17 Interviews were also conducted with representatives of the GAFSP, the CGIAR, the CAADP, the New Alliance for Food Security and Nutrition, Fintrac, the Strengthening Partnerships, Results and Innovations in Nutrition Globally (SPRING) project, and directors from seven Feed the Future Innovation Labs. At total of 112 key informant and group interviews with 243 individuals were also conducted across a purposively selected sample of five Feed the Future focus countries: Bangladesh, Ghana, Guatemala, Malawi, and Uganda.18 To obtain multiple points of view and facilitate triangulation, we interviewed a variety of stakeholders in each fieldwork country. Within USAID missions, we interviewed the Feed the Future Country Coordinator, relevant Agreement Officer’s Representatives (AORs) and/or Contracting Officer’s Representatives (CORs) and program managers, subject matter specialists (such as gender officers and health officers, and M&E officers), and Food for Peace officers. Outside of the missions we interviewed implementing partners, government partners, interagency staff from participating USG agencies, other donors, multilateral institutions, local research institutions and universities, and local private sector actors. The country visits provided an opportunity to collect rich data that could be used to contextualize quantitative results from the analysis of FTFMS and online survey data, and explore how the initiative is implemented across very diverse development contexts at the country level. 16 The State Department provided written answers to questions. 17 The evaluation team also attempted to schedule interviews with additional Feed the Future participating agencies, including the U.S. Geological Survey (USGS) and the U.S. Trade Representative (USTR), but were unsuccessful. 18 See Annex IV: Interviews Conducted for list of interviewees 16 A total of 327 persons were interviewed: 101 in KIIs, and 226 in 67 separate group interviews. Where informed consent was obtained, individual and group interviews were audio recorded; these audio files were professionally transcribed to facilitate analysis. During the fieldwork phase, evaluation team members wrote up and shared field notes to facilitate rolling analysis and identify emergent themes and trends for further analysis. f. Quantitative Data Analysis For analysis, the team used Microsoft Excel to store, organize, and analyze the FTFMS data. All relevant indicator data from the FTFMS was compiled into a master dataset aggregated at the country level for each indicator. Two primary types of analysis were conducted on this data: 1) Aggregated trend analysis was performed to understand the directional trajectory of performance on each indicator, and 2) Aggregated targets vs. actual performance were analyzed to understand to what extent focus countries did or did not achieve the targets set for each indicator. In order to analyze each country's performance against targets for any given indicator, the evaluation team aggregated matched pairs of targets and actuals from the implementing mechanism (IM) level to the country level. Matched pairs refer to only including values for IMs that reported both a target and an actual value for any given year per indicator. This allowed the team to analyze whether a country missed, met, or exceeded targets for indicators informing progress against the goals and objectives listed in the RF. The compiled dataset was also analyzed to understand the spread of data and identify patterns and inconsistencies in the data that could affect the overall analysis and/or interpretation of the analysis. Statistical analysis included examining correlations between various indicators. In addition, SurveyMonkey was used to produce summary statistics on the quantitative data collected from the online surveys of focus country missions, aligned country missions, and regional missions. g. Qualitative Data Analysis Team members reviewed documents for their responsiveness to specific evaluation questions and identified material within these documents to aid in the analysis. To analyze fieldwork data, interview transcripts were systematically coded with the aid of the qualitative analysis software Dedoose. Coders used an inductive and deductive approach based on a coding scheme informed by research questions, evaluation team members’ hypotheses based on preliminary document review, and extant subject expertise. The coding scheme was revised throughout the analysis process and additional themes and concepts emerged from the data; this was done to ensure that the coding phase was sensitive to and could account for expected and unexpected themes. Once coding was complete, a secondary thematic analysis of the coded excerpts was conducted to identify common conceptual categories as they relate to each evaluation question and themes within each category that emerged from each dataset. Thematic patterns were examined within 17 and across countries and informant types. Summary findings from this stage of the analysis were provided as an input be used alongside quantitative analysis and document review results to inform findings for each question, as applicable. h. Managing Bias and Data Limitations Self-reporting through survey questionnaires or qualitative interviews is a common and widely￾accepted approach to conducting performance evaluations at USAID. However, an overarching issue related to the methods used concerns the challenges associated with accurately self￾assessing and/or self-reporting. Various self-reporting biases – stemming from variations in respondents’ recollection, understanding of the questions posed, attitudes, and emotions – may have interfered with an accurate assessment of the true situation of Feed the Future programming. Additionally, in each field country, teams were only able to spend approximately 90 minutes on each interview. No independent verification was conducted of what was reported to the team by interviewees, although reports were triangulated across interviewees. Because of the dynamic and emergent nature of the interviews, each interview varied widely in terms of content and structure. Because of the need to identify patterns in the data, interviews were necessarily coded by the researchers in a way that required, to some degree, subjective interpretation, which can result in various forms of researcher bias. Coders received consistent training and were supervised throughout by the qualitative methods specialist in order to mitigate this potential for bias. Internet surveys can be unreliable for a number of well-documented reasons.19 Due to time, budget, and format limitations, the survey instruments did not undergo rigorous development and reliability testing, and they were self-administered online rather than being administered by a trained survey enumerator. As a result, there is a possibility that varying understanding and subjective interpretations of questions by respondents renders some responses invalid, and there is no way to estimate this error. However, when considered in combination with data emerging from interviews and reinforced by additional data that resides in the literature, the survey data becomes one piece of information that, in combination with all the others, allows the evaluation team to construct a reasonably reliable picture of the Feed the Future initiative. 19 See, for example, Karen J. Jansen, Kevin G. Corley, and Bernard J. Jansen. “E-Survey Methodology.” Chapter 1, in “Handbook of Research on Electronic Surveys and Measurements,” by Rodney A. Reynolds, Robert Woods and Jason D. Baker. IGI Global: 2007. 18 Furthermore, the timeframe, budget, and scope of work envisioned by USAID for this evaluation obviated the practicality of conducting fieldwork in all 19 focus countries. Given this context, a purposive sample of five focus countries was drawn, based on a range of specific criteria to examine how Feed the Future programs are being implemented in the field under varying country conditions, rather than through a random selection process. Accordingly, it is important to not view data collected from the field as representative of the total population of focus countries. The team has limited the use of findings from the field to providing examples, serving as an additional source of data for the triangulation of findings, and enriching the understanding of quantitative data. Additionally, it became quite evident through the course of field interviews that wide variation exists among Feed the Future countries. Factors that influence the success of any given Feed the Future country program include: operating culture within a USAID Mission and cultural parameters within the broader society; personality and capacity among key personnel; relative development, strength, and incentives within the commercial sector; relative capacity, engagement, and political will of the host country government; levels of agricultural or market infrastructure development; and natural phenomena like droughts or floods which have been exacerbated by El Niño or climate change. All of these factors vary widely from country to country. This complexity confounds the process of identifying initiative-wide patterns. In addition, as only five out of the 19 countries were visited, the issue is further compounded by data limitations. All researchers involved in the research process have extensive and complementary experience in the subject matter of this evaluation. The team includes researchers with extensive experience in the design and implementation of program evaluation, applied social science research, and qualitative and quantitative research methods; expertise in health and nutrition programming; and expertise in the design and implementation of agricultural development programs, with experience in several of the five countries selected for fieldwork. All members of the team regularly met to discuss and share their observations, review and check each other’s emerging analysis, and contribute to data collection, analysis, and preparation of the final report. To control for any methodological weaknesses, triangulation was used throughout the research process. Information obtained from multiple sources, using multiple methods, can be assumed to be more reliable, especially where the data converge. This triangulation included the use of multiple research methods, multiple data sources, and regular and continuous peer review of findings, analysis, and conclusions. In addition, USAID was consulted on a regular basis, including an initial review of emerging findings and conclusions, to aid in validation of the research process and support meaningful, utilization-focused analysis. Additionally, emerging findings and conclusions were shared with an External Oversight Committee to provide an additional layer of oversight to the analysis. Every attempt has been made to ensure that all interpretation is objectively based on 19 the data and to minimize the potential bias emerging from subjective interpretations of individual evaluators. i. Limitations to FTFMS Data Analysis of FTFMS and PBS data revealed some dataset limitations, which affected the team’s ability to use these sources to address some of the evaluation questions: Aggregation limitations for initiative-level analysis. To assess performance at the initiative level for the IM indicators, the evaluation team aggregated data upwards from the IM level. However, aggregating across IMs, and then across countries was complicated by the fact that each of the Feed the Future focus countries report on certain required as applicable indicators; each country’s performance is measured by a unique set of indicators, including custom indicators that best suit the Feed the Future programming in that country. As such, each country’s performance toward the overarching Feed the Future goals is defined by a different configuration of indicators, which is not particularly conducive to aggregation and analysis at the initiative level across all 19 countries. IV. Findings, Conclusions, and Recommendations The Feed the Future initiative has accomplished much and demonstrates many worthy achievements, and is overall viewed to have been successful up to this point, despite inevitable challenges and varied success across results areas. Overall, the investments are well-targeted at systems drivers and underlying conditions that determine poverty and under-nutrition, with a few areas where additional work is required. In spite of the enormity of the task, much progress has been made. In terms of tracking progress and ensuring accountability, the trends are generally positive although challenges remain in terms of measuring all results completely and fully capturing the successes that have been achieved. Feed the Future has generally aligned with its RF and remained faithful to its strategic approach, although there are weaknesses in the current framework that are evident in gaps in the RF, various difficulties with associated indicators, and processes within M&E systems that require additional work. It is worth noting here that many of these issues are currently being addressed. Nonetheless, Feed the Future has made significant contributions to developing the capacity of actors across the agricultural development space to improve their ability to monitor activities, identify needs, demonstrate results, and learn from experience. The multifaceted body of research that has been produced is generally of very high quality and cuts across multiple dimensions of relevance, providing strong evidence to support improved agricultural development and nutrition. 20 Furthermore, Feed the Future has been engaged in a systematic process of learning from experience and has itself evolved significantly over time, intentionally seeking to refocus activities on what works, address emerging needs, and continuously improve. Feed the Future has made significant contributions to improved agricultural productivity in focus countries through a variety of approaches, including much significant work with value chains, consultation and engagement with varied actors all across the agriculture space, and improvements in crop varieties and agricultural technologies. There is a need to re-assess how Feed the Future should best address resilience issues and the unique needs of people at the bottom of the socio-economic ladder – and this process is a work in progress. Feed the Future has developed capacity in multiple actors, including in the agricultural research sector, government institutions at multiple levels, and the private sector, as well as among numerous individuals. Much work remains to be done, however, and this requires development of additional guidance and strategic approaches that are cognizant of the critical importance of larger systems, rooted in optimal practices learned from the wider field of capacity development, and informed by robust mechanisms for feedback that inform learning and adaptation. Feed the Future has contributed significantly to improved systems and strengthened institutions that are responsible for developing and implementing policy at multiple levels, and has supported many noteworthy improvements in policies related to agricultural development – both regionally and at country level. While Feed the Future has made a coherent effort to address gender disparities and reduce inequities, and has developed tools to support this that are widely regarded as cutting-edge, additional focus on gender issues at an operational level is required. Feed the Future has developed an evidence base that supports meaningful integration between nutrition and agriculture, and has evolved the approach over time to improve the effectiveness of Feed the Future activities towards addressing nutrition. Tensions remain, however, between varying conceptualizations of nutrition and agricultural integration that have not been fully resolved, and additional work is required to complete this process. Feed the Future has worked hard to ensure alignment with country priorities and processes and these approaches not only ensure local ownership, participatory engagement, relevance, and most likely will contribute significantly to long term sustainability, but also ensure that Feed the Future conforms with development best practices as outlined in the Paris Declaration and the Accra Agenda, as well as general principles of aid effectiveness. Local political and other systems dynamics do not necessarily always align with or support Feed the Future goals, and there is a long-term time horizon required to fully realize required changes – but the strengths of the approach far outweigh any weaknesses. Feed the Future has engaged 21 with or supported a wide range of other donors and multilateral organizations, and is generally perceived to be an effective partner and a strong leader. Feed the Future has engaged the private sector vigorously, both from Washington via public-private partnerships, and in focus countries through a variety of approaches. Although current metrics do not adequately capture the scope and scale of this engagement, the initiative is clearly robust and is generally effective. The reader will note many areas of research interest that seem to recur, or perhaps that seem somewhat out of place, given the specific evaluation question being addressed. This is a result of the fact that many elements of the Feed the Future approach recur across multiple programmatic areas. For example, human and institutional capacity development (HICD) occurs through training provided to agricultural research institutions and researchers; it is also a primary intended result of many of the policy reforms being implemented at multiple levels, and it also reflects and supports the Feed the Future emphasis on local ownership and supporting local priorities. Similarly, private sector engagement is relevant to policy reform, local ownership, and HICD, and each focal area has elements focused on this. These are just a few examples. Rather than attempting to artificially segregate activities, the authors have attempted to address multiple areas of research concern, as appropriate and in adequate detail, in order to fully answer each specific evaluation question – while recognizing this results, to a certain degree, in duplicative content occurring under more than one question. Where noted, the evaluation team has alerted the reader to relevant and/or more detailed discussion occurring in other sections of the report. Question 1: Adherence to and progress across the Feed the Future Results Framework across the initiative. How well have strategies and the implementation of programs followed the Results Framework and the causal pathways identified therein? Focus country strategies and implementation of those strategies show close adherence to the Feed the Future Results Framework and its causal linkages during FY 2011 – FY 2013; adjustments and addition of new interventions were introduced in FY 2014 and FY 2015 to adapt strategies based on learning following ongoing implementation experience. The majority of these changes do not represent significant departure from the RF, but several include new areas of investment. Gaps in the RF and causal pathways at the sub-IR level are being filled by focus country programming as these countries continue to learn from implementation experience, evaluations, surveys, and discussions with peers. USAID’s Resilience Policy and approach is increasingly being adopted by missions to address IR 5, Increased resilience of vulnerable communities and households. Youth have emerged as a subpopulation that should be included and plans to develop 22 specific activities focused on youth represent new areas of investment. The incorporation of programming to address environmental degradation and climatic fluctuations, already occurring under Feed the Future and strengthened in response to an Executive Order on Climate Resilient Development, has been further heightened. BFS is engaged in developing a more comprehensive RF, which it incorporates the 2014 USAID multisectoral nutrition strategy and the new (2016) USG interagency nutrition coordination plan. The Feed the Future Guide (2010) was used to guide the analysis for this evaluation question. The Guide depicts the Results Framework, and the narrative description of strategies that comprise causal pathways to reach targeted objectives and goals (see Figure 1, pg. 9 for the Feed the Future Results Framework). The overarching goal of Feed the Future is to sustainably reduce global hunger and poverty by tackling their root causes and employing proven strategies for achieving large scale and lasting impact.20 Feed the Future identified two key objectives that have direct causal linkages to the goal – “Accelerated Inclusive Agriculture Sector Growth” and “Improved Nutritional Status.” The Guide discusses areas of investment to support causal pathways leading to key results at the IR level that are hypothesized to lead to results at the Objective level. The RF includes sub-IRs describing results that must be achieved to support three of the eight IRs: increasing agricultural productivity, improving markets and trade, and increasing investment. There are currently no specified sub-IRs in the RF that depict the causal pathways to achieve results for IR 4 (Increasing employment opportunities from selected value chains), IR 5 (Increasing the economic resilience of vulnerable communities and households), or the three Intermediate Results (6, 7, and 8) to support the objective of Improved Nutritional Status. The Guide does provide a narrative on the types of potential investments that should be employed to achieve these Intermediate Results. Directions in the Guide also require focus countries to incorporate three cross-cutting priority areas: gender equity, environmental stewardship and protection, and climate change. Detailed guidance is provided on strategies and investments to support these cross-cutting areas. USAID’s regional missions were guided to follow the Feed the Future RF as well. The Guide describes complementary investments that are regionally based to support programs in focus countries. Areas of investment for regionally-based programming are described for expanding access to regional markets; mitigating risks associated with drought, natural disasters, and disease; 20 USG Interagency Feed the Future Guide, 2010, p. 10. 23 and building long-term capacity of regional government and private sector organizations to increase the effectiveness and sustainability of regional investments. Importantly, the Guide indicates that some variation in the composition of strategic investments among focus countries is expected based on specific country contexts, evidence-based data from assessments, and through adherence to the Rome Principles – specifically, the requirement to support country-led and regional strategies. In Africa, African country strategies are linked to the CAADP; in Central America and the Caribbean, to the Latin America and Caribbean End Hunger 2025 initiative, and in Asia, to the ASEAN Integrated Framework for Food Security. Specifically, the Guide states: “Our investments will vary by country and will depend on the country’s highest priorities and the investments of the country itself, as well as the investments of other donors and key actors. They will build on existing successes, our expertise and strengths, and be concentrated in strategic areas based on our comparative advantages. They will be designed in close coordination with governments and other development partners in order to maximize our collective impact.”21 Building on this statement, the guidance for country strategies to promote the initiative’s objective of Inclusive Agriculture Sector Growth states: “The specific package and focus of investments will vary by country and will depend on the key priorities articulated in the country’s strategy.”22 The guidance for the Feed the Future objective of Improved Nutritional Status states that strategies will also be based on work with Global Health Programming to ensure that USG nutrition investments have maximum impact on target populations: “Through both initiatives, we will implement a nutrition strategy based on country-specific needs and opportunities.”23 1a. How well do strategies follow the Feed the Future Results Framework and causal pathways? Multi-year strategy documents24 and strategy change memos were used by the evaluation team to conduct a comparative assessment of how well focus country strategies followed the Feed the Future RF and causal pathways. QUESTION 1a FINDINGS 21 Feed the Future Guide, 2010, p. 10. 22 Ibid. 23 Ibid. 24 Each MYS was reviewed and approved in 2011 by an interagency committee in Washington, D.C. 24 The review of MYS documents shows that strategies were developed using an interagency process involving USAID and other participating USG agencies. The documents describe links to country￾owned plans and the specific conditions related to food security at the time MYSs were developed, building on prior USG investments. The articulation of selected investment areas in all 19 focus country MYS documents are drawn from the range of investment areas described by the Feed the Future Guide. Narrative descriptions and data from recent assessments, evaluations, and surveys are presented, and MYS documents clearly describe how findings from these assessments led to the selection of specific investment areas and the elements of their strategies to support results based on these investment areas. The assessment of strategies presented in MYS documents from the 19 focus countries shows adherence to the basic Feed the Future RF and areas of investments that support causal pathways to achieve reductions in the prevalence of poverty and stunting in the ZOI. There is variation in investment areas selected and the details of approaches across focus country strategies as expected, based upon the requirement to develop evidence-based plans that support country￾owned strategies and country context. There are a few notable areas of variation in the application of the Feed the Future RF and causal pathways, which are explained below. Regional Feed the Future strategies follow areas of investment prescribed by the Feed the Future Guide. A review of MYS documents for regional programs in Latin America and the Caribbean, East, South, and West Africa show that regional strategies are in alignment with the areas of investment described in the Guide to support achievement of results in focus countries. The greatest area of variation in the application of the Feed the Future Results Framework and its causal pathways was found in some country MYS documents related to IR 5, Increased resilience of vulnerable communities and households. This is not surprising given the Guide allows focus countries to identify which communities and households are vulnerable based on the country context. For example, Tanzania developed approaches to increase the resilience of vulnerable communities and households and strategies based on hunger and nutritional status. Rwanda focused on environmental conditions where the identified communities live. Nine of the focus countries integrated development and humanitarian resources from Feed the Future and Food for Peace in a strategy that is designed to increase the economic base and resilience of the rural poor, thereby enabling them to contribute to, and benefit from, the growth of the agriculture sector. Another variation to the application of the Feed the Future RF is illustrated by the incorporation of an additional intermediate result based on country-specific environment and global climate change issues instead of treating these topics as cross-cutting issues (Zambia, Senegal, Rwanda, and Haiti.) Explicit strategies are described based on interventions for improved natural resource 25 management of key resources required for the agriculture sector, and for increasing the ability to adapt to climate change to reduce vulnerability. The MYS documents from these focus countries provide evidence to support the centrality of addressing environmental and climatic conditions within their overall strategy to achieve the initiative’s objectives, and sustainable reductions in the prevalence of poverty and stunting. Some focus countries introduced additional cross-cutting areas based on local constraints and emerging issues. Citing very high youth unemployment rates, lack of opportunities in rural areas, and the growing youth bulge, Tanzania, Cambodia, Kenya, Liberia, and Malawi incorporated youth as an additional cross-cutting area and as a part of the strategy for achieving inclusive agriculture sector growth. Strengthening governance is included as a cross-cutting issue by Mozambique, Kenya, Ethiopia, and Cambodia. Additional cross-cutting areas include local capacity development (Bangladesh), land tenure (Liberia), business environment (Mali), and private sector engagement (Haiti and Kenya). 1b. How well has strategy implementation followed the Feed the Future Results Framework and Causal Pathways? To assess the extent to which the implementation of strategies follow the Feed the Future RF and its causal pathways, the analysis of strategies from MYS documents was extended to include a detailed review of annual portfolio review presentations spanning FY 2012 – FY 2015, Summaries of Issues and Action Items memos based on discussions of portfolio reviews (FY 2012 – FY 2014), and official Strategy Change Memos. QUESTION 1b FINDINGS Strategy implementation data closely links to the Feed the Future RF and its causal pathways during the initial years of implementation. Data presented in annual portfolio reviews from 19 focus countries in FY 2012 and FY 2013 demonstrate that implementation of Feed the Future portfolios is based on their original strategies designed and approved in 2011. Strategy Change Memos from the same time period indicate a need to adjust based on learning from data collected, especially in FY 2013, as well as key events that occurred. However, these adjustments still follow the main strategy. There are official Strategy Change Memos for 10 of the 19 focus countries. Six of the memos – from Tanzania, Tajikistan, Rwanda, Mali, Liberia, and Senegal – show approved changes for MYS documents, but do not entail fundamental changes in the country RF or causal pathways. Approved strategy changes are primarily based on: 1) changes to the existing ZOI (retraction or expansion), and 2) changes to the value chains (VCs) that were originally selected (either dropped or new value chains added). An example of changes to the existing ZOI is the temporary retraction of the zone in Mali following the coup in the north. 26 Approved changes to value chains were based on evidence showing low contribution to poverty goals and low market demand for the product. FY 2014 and FY 2015 portfolios show more frequent adjustments to the Feed the Future RF and causal pathways following four to five years of learning from implementation experience. Sources of change include the establishment of a second ZOI, the application of USAID’s Resilience Policy and Program Guidance, and the effects from the severity of the FY 2014 – FY 2015 El Niño. Four of the Strategy Change Memos describe approvals to establish a second ZOI in Kenya, Uganda, Nepal, and Malawi. The purpose of these additional zones was to address food security issues among extremely vulnerable populations in remote locations with little infrastructure connecting them to services and markets, and in locations with degraded environments. Kenya is an early example: creation of a second ZOI was approved in 2012, focusing on extremely vulnerable pastoral populations suffering from the effects of prolonged and recurrent drought in arid and semi-arid areas of the country. In 2015, Nepal, Uganda, and Malawi received approval for the creation of a second ZOI based on country-specific issues – specifically, extreme poverty and malnutrition in certain areas of Nepal that were not in the existing ZOI, which were exacerbated by a highly destructive earthquake, and conditions of extreme poverty in post-conflict areas in northern Uganda; in Malawi, the change was based on an area with populations experiencing very high levels of vulnerability and persistence of extreme poverty. FY 2015 portfolio reviews for Nepal and Uganda noted implementation of an integrated resilience strategy coordinated and facilitated by USAID’s Center for Resilience, which has just begun; in Malawi, the strategy was reportedly in the design stage.25 The application of USAID’s Resilience Policy and Program Guidance expands Feed the Future areas of investment to support IR 5, Increased resilience of vulnerable communities and households. The Center for Resilience coordinates and facilitates the application of USAID’s Resilience Strategy,26 which is designed to address communities and households in areas of chronic vulnerability such as the locations described above, with a long history of receiving humanitarian assistance. Populations are marked by conditions of instability, recurrent shocks, persistent poverty, high rates in the depth of poverty of among households, high rates of malnutrition among women of reproductive age and children under five years of age, and stunting in children at age five, and 25 Interview with Center for Resilience, May 2016. 26 USAID, Building Resilience to Recurrent Crisis, USAID Policy and Program Guidance, December 2012. Details of how this approach is developed, layered, and sequenced during phases of strategy implementation is described in this Guidance. 27 other measures of malnourishment and hunger.27 This approach expands the areas of investment to IR 5 outlined in the Feed the Future Guide to include, for example, conflict resolution and prevention interventions, short-term disaster assistance as needed, and diversified risk management practices to reduce risk from economic and climate shocks based on a collaborate interagency assessment of the drivers of chronic vulnerability of these populations. Moreover, in addition to Feed the Future and FFP programming, the strategy comprises coordinated intra￾agency and interagency interventions from, for example, the U.S. Geological Survey, and the U.S. African Development Foundation (USADF). USADF works directly with the most vulnerable and marginalized African farmers, often in remote locations, to unlock constraints for moving out of poverty; they have an established history of working with local populations in these newly established ZOIs in northern Uganda and Malawi.28 Strategies also incorporate investments from the World Food Programme. Changes based on agro-climatic events also were portrayed as a reason to expand areas of investment and causal linkages. To cope with and recover from the adverse effects of El Niño experienced globally, in countries ranging from Guatemala and Haiti to Zambia and Cambodia, focus countries presented plans in their annual portfolio reviews for increased investments in Global Climate Change programming and climate-smart agriculture that represents additions to GCC activities that were already incorporated in their strategies. Data are presented linking serious declines in agricultural productivity in value chain crops, damaged seed stocks, death of livestock, and concomitant drops in income to severe, prolonged drought and/or profound flooding related to El Niño. For example, Malawi’s recent drought and flooding requires the country strategy to readjust and integrate additional investments to withstand certain environmental and climatic conditions. Focus countries are also using Feed the Future and other USAID resources to address programming gaps and issues highlighted by new data. Portfolio review presentations in FY 2014 and FY 2015 provide information on new activity designs to fill in gaps, make activity adjustments, or to address specific issues based on learning from recent assessments, surveys, and performance evaluations 27 Data on the characteristics of these populations is collected and analyzed by the Center for Resilience using panel studies conducted by the World Bank, DHS data, and from other sources (Interview with Center for Resilience, May 2016). 28 USADF manages 19 country offices across the continent. USADF complements, collaborates and extends Feed the Future programming by USAID and other participating USG agencies in nine of the 12 African Feed the Future countries, (specifically, Liberia, Mali, Malawi, Senegal, Tanzania, Zambia, Kenya, Rwanda, and Uganda) and through satellite programming in Ethiopia related to food/energy. Interview with USADF, May 2016. 28 or impact evaluations. Actions taken are still closely related to their original strategies based on the Feed the Future RF, and the presentations did not include requests for a formal Strategy Change Memo. Presentations include brief descriptions of actions taken (or planned) to address those findings, such as the completion of designs for new activities to increase off-farm economic opportunities for youth related to value chains; more layering of nutrition and agriculture interventions in the same areas; and the design of new projects to include additional interventions to address specific problems experienced by beneficiary populations. An example of the latter is Guatemala’s plan to integrate water, sanitation, and hygiene (WASH) messaging into every project and to assist municipalities in increasing chlorination rates and to protect drinking water sources based upon preliminary survey results showing that most households in the ZOI are consuming biologically contaminated water. Changes have also been made, or are being planned, based on recent poverty and malnutrition data. FY 2015 portfolio reviews from several focus countries present adjustments to their strategies and new activity designs based on evidence from portfolio-wide or program-specific evaluations as well as preliminary, estimated results from recent interim PBS data showing that progress in reducing the prevalence of poverty and/or stunting within the ZOI was not being met to the degree expected.29 While in Kenya there were statistically significant reductions in depth of poverty in the northern part of the ZOI, the prevalence of poverty increased from baseline in the southern and central part of the ZOI, though those results were not statistically significant. Countries used findings from recent evaluations to understand why progress was not on track to meet goal-level targets. The most common conclusion reached by missions from these focus countries that was described in their portfolio review presentations can be summarized as the need to design more holistic, integrated, and focused approaches, including additional programmatic areas of investment. However, evaluation findings and assessments pointed out the complexity of situations external to Feed the Future confounding slow progress, including the effects of severe droughts, flooding, diseases affecting value chain crop production, and political and economic events at the national and international levels. For example, Cambodia’s Portfolio Review presented an example where there is not the same level of progress in reducing malnutrition among children under five and women at reproductive age among the poorest of the poor living in pockets of deep poverty within the ZOI, although estimated data from the interim PBS show reductions in the overall prevalence of poverty and stunting has already surpassed the targets set 29 These countries include Uganda, Ghana, Senegal, Kenya, Nepal, Honduras, Tajikistan, and Ethiopia. Note that several of these countries did have statistically significant reduction in poverty at interim, but average rate of reduction is not enough to meet 2017 target. Some of the interim PBS data presented for the annual portfolio review were estimated (findings were not yet verified.) Tanzania and Rwanda did not yet have data from their interim PBS. Mali presented plans to conduct its interim PBS in June 2016. 29 for FY 2017. Uneven progress within the ZOI was also noted in Senegal and Kenya. Countries that did not yet have the findings from the interim PBS drew on recent evaluations that provided some data on poverty, incomes, and malnutrition to design new activities. QUESTION 1 CONCLUSIONS 1. Focus country strategies and implementation of those strategies show close adherence to the Feed the Future Results Framework and its causal linkages. Very importantly, the evidence from this review also shows that country Feed the Future teams are learning and adjusting, and adapting those strategies following five years of implementation experience. Multi-year strategies approved in 2011, and implementation in FY 2012 – FY 2013, show close adherence; adjustments and addition of new interventions were introduced in FY 2014 and FY 2015 based on findings from evaluations, assessments, studies, and their interim PBS data. The majority of these changes do not represent significant departure from the Feed the Future RF, but several do include new areas of investment to those described in the Feed the Future Guide, such as the design of new activities featuring an emphasis on the participation of youth in inclusive agricultural sector growth strategies, and the application of USAID’s Resilience Policy and Program Guidance in additional ZOIs that may include, for example, basic education, conflict mitigation and resolution, family planning, and HIV/AIDS, depending on the analysis of needs. 2. Gaps in the Feed the Future RF and causal pathways at the sub-IR level are being filled by focus country programming. The absence of causal pathways to support IRs related to employment, resilience of vulnerable communities and households, and nutrition, represent significant gaps in theory development for Feed the Future to sustainably address the prevalence of poverty and stunting in children under five. Focus countries have used data from evaluations and assessments to assess what is needed going forward and to develop new activity designs, plans for strategy adjustment, and new areas of investments. Reports on these activities are providing Feed the Future with important sources of input to develop a more comprehensive RF and causal linkages. 3. USAID’s Resilience Strategy to address chronically vulnerable populations provides a solid basis for developing sub-IRs that reflect causal pathways to achieve IR 5, Increased resilience of vulnerable communities and households. 4. Youth have clearly emerged as a subpopulation group that should be included in strategies to promote inclusive agriculture sector growth. Plans to develop specific activities focused on opportunities for youth related to the agriculture sector represent new areas of investment, and also expand the definition of inclusive agriculture sector growth. 30 5. Focus countries also demonstrate adaptability in response to agro-climatic events, such as prolonged and severe droughts and flooding. The incorporation of programming to address environmental degradation and climatic fluctuations has been strengthened following the recent ravages of El Niño in many focus countries. It is clear that these areas of intervention are critical to the sustainability of inclusive agriculture sector growth and improved nutritional status of women and children. These additional interventions and other interventions to diversify risk management strategies to increase resiliency will help to preserve gains and prevent backsliding into poverty and malnutrition. QUESTION 1 RECOMMENDATIONS 1. Build out the Feed the Future Results Framework and causal linkages, especially where sub￾IRs are missing, to more accurately reflect the complexity of interventions and areas of investment that are needed to address different strata of poverty within ZOIs – particularly extremely poor and malnourished populations lacking resources. 2. Great ideas for global and national strategies often emerge from pilot activities and implementation experiences from the country and sub-regional levels. Draw on new approaches and activities designed by focus countries to address findings from recent portfolio reviews and preliminary data from the interim PBS as input for developing a more comprehensive and effective Feed the Future RF and mapping causal linkages for the second phase of the initiative. 3. Incorporate explicit strategies to address the youth bulge and high youth unemployment levels by expanding employment opportunities from value chain production linked to agro￾processing, supply of agriculture inputs, trade and markets, agriculture extension services, and the use of information and communication technology (ICT) applications. Such strategies will give youth a stake in their country’s food security and will promote more inclusive and accelerated agriculture sector growth. 4. Consider adapting elements from the RF for USAID’s Resiliency Strategy for the Feed the Future Results Framework to strengthen and develop a more effective strategy for IR 5 (Increased resilience of vulnerable households and communities), that can be applied to address conditions of extremely vulnerable populations within the ZOI, as well as in additional ZOIs established to address such populations located outside the ZOI. This approach relies on strong intra-agency involvement and coordination with local governments, private sector businesses, and civil society to address different dimensions of the problem related to root causes of specific chronically vulnerable populations. This is a programmatic approach that would also benefit from greater interagency involvement. 31 5. Re-examine the three IRs supporting improved nutrition and develop a new strategic approach to support the initiative’s objective for Improved Nutritional Status that has links to inclusive agricultural sector growth. The new strategic approach should be drawn from the 2014 USAID multisectoral nutrition strategy developed by BFS, GH, and FFP, and the new (2016) USG interagency nutrition coordination plan.30 Question 2: How has the Feed the Future initiative performed against each of its eight intermediate results outlined in the Results Framework across focus countries? Does the progress made against the eight Intermediate Results to date, coupled with changes in impact-level indicators, provide evidence that Feed the Future is on track to achieve its five-year goals of contributing to reductions of 20% in poverty and stunting? The Feed the Future initiative is, on average, performing well against each of its eight IRs outlined in the RF across focus countries. Results of IM-level indicators, coupled with changes in impact￾level (PBS) indicators, provide evidence that Feed the Future is contributing to reductions in global poverty and hunger. For 11 of the 17 countries for which data were available, there were statistically significant decreases in the prevalence of poverty and for eight of the ZOIs in 19 countries, there were statistically significant decreases in the prevalence of stunting between baseline and first interim. However, each country’s performance is also defined by a different configuration of indicators, including custom ones that were not within the scope of this analysis, which is not conducive to interpretation of simple quantitative aggregation and analysis at the initiative level. Overall performance on IM indicators at an initiative-level is moving in a positive direction. However, performance across countries and indicators varies. Country-level progress does not always translate to even progress throughout the country ZOI. Impacts of Feed the Future interventions vary within country ZOIs, with uneven progress on the ground. Feed the Future has a robust monitoring system that is structured to reflect the Feed the Future RF. All Feed the Future focus countries are required to report on the goal and high-level objectives indicators. However, for the IRs and sub-IRs, each country determines which indicators are most relevant as meaningful measures of the project and activity’s outcomes in the context of the 30 In an August 2016 communication from the BFS/MEL team, the evaluation team was informed that the nutrition side of the Feed the Future Results Framework has recently been adjusted, drawing on the USAID intra-agency nutrition strategy and the new interagency strategy referenced above. The team has not seen the revised RF based on this change. 32 country in which they operate.31 In order to assess how the Feed the Future initiative as a whole has performed against each of its eight IRs across all focus countries, the evaluation team analyzed progress of the performance monitoring indicators aggregated across all focus countries over the FY 2011 – FY 2015 timeframe. The Feed the Future performance monitoring indicators are grouped into three levels: zone of influence PBS indicators, national/regional indicators, and implementing mechanism indicators. While the PBSs are collected at the ZOI level, and national/regional indicators are collected and reported at the country level, the IM indicators are collected and reported at the activity/IM level. As such, in order to assess performance at the initiative level, the data was aggregated both across IMs within a country, and across countries to inform initiative-wide performance. The evaluation team used both levels of aggregation in analyzing the quantitative data presented in the FTFMS as well as both the quantitative and qualitative data presented in the portfolio reviews. Specifically, the evaluation team analyzed FTFMS data for the eight required (R) high-level impact indicators that all 19 focus countries are required to report on, as well as the required if applicable (RiA) and standard (S) indicators for lower levels of the RF. Additionally, Feed the Future activities use custom indicators to measure performance to further inform performance management, learning, and adapting. Given that the custom indicators are not common across all focus countries, the evaluation team did not conduct quantitative analysis on the FTFMS data for custom indicators; rather, the team incorporated custom indicator results through analysis of the portfolio reviews. The evaluation team synthesized findings from portfolio reviews to assess progress on agricultural productivity per value chain, progress in nutrition initiatives, and challenges and issues that affected both reporting and programming in each country. The evaluation team obtained all IM data from FTFMS, PBS32 data from BFS, and custom indicator data from Portfolio Reviews. Findings from this data were triangulated with findings from surveys and interviews the evaluation team conducted. Targets vs Actuals (Reported values) In order to assess performance against the eight IRs, the evaluation team analyzed how “on-target” each country was for each of the relevant indicators it reported on. The targets for IM indicators are set at the IM level; therefore, in order to compare actual performance to targets, the evaluation team first calculated how “on target” each IM’s performance was for every year per indicator. Only 31 http://www.fsnnetwork.org/sites/default/files/ftf_agriculture_guide_0.pdf 32 The PBS data made available and analyzed includes baseline values for all focus countries as well as interim values, as of 2016, for a subset of the 19 focus countries. 33 matched pairs, or instances where both a target and actual value was reported for an IM for a particular year, were considered for this analysis. In order to assess the Feed the Future initiative performance across focus countries, the percentage of targets achieved per IM was aggregated to the country level. As such, the evaluation team calculated country sums for matched pairs at the IM level, and then an average of the country sums across the years to assess how on target, on average, a country was across the years for each indicator. This analysis enabled the evaluation team to analyze and compare the extent to which focus countries were on target, on average, per indicator. For example, for the “number of children under five reached by USG-supported nutrition programs,” Haiti achieved 106.25%, 113.91%, 103.68%, and 149.86% of its target for each year from 2012 to 2015 respectively. On average, across the years, Haiti was thus over target by approximately 18% for this indicator.33 Directional Trends FTFMS data was also used for trend analysis for each focus country across all levels of the RF, to the extent that data was available. To comment at the initiative level, performance across all focus countries was aggregated per indicator and trend lines were used to analyze initiative level performance per indicator. The evaluation team analyzed the scatter plots of actual reported values, as well as a linear trend line, for all indicators per RF level. These represent the spread of, and directional trends in, reported data per indicator. The trend lines help predict what future years could look like given past performance. The above analysis was conducted for IM indicators, which report annual values when available. PBS data, on the other hand, is only collected every two to three years. For PBS indicators, confidence intervals calculated for baseline and interim values were examined to determine whether change in the indicators was statistically significant. For all PBS data used in this analysis, please refer to Annex IX: PBS Table. Below, we present an in-depth analysis on a subset of available indicators to illustrate progress against each IR.34 For clarity, we structure the presentation by the RF starting with the goal-level indicators, followed by Objective 1 indicators and the IRs supporting that objective (IRs 1-5), and then Objective 2 indicators and the IRs supporting it (IRs 5-8). While IR 5 overlaps with both objectives, it is presented only once, under Objective 1. 33 IMs are required to input deviation narratives into FTFMS whenever a target is missed or exceeded by more than 10%. For example, for Haiti, 2015 deviation narratives describe how higher than planned results were because of better than expected performance in certain programs, or because the number of children living in the neighborhood households increased. 34 The selected indicators were chosen in consultation with USAID/BFS based on their relevance and utility. 34 QUESTION 2 FINDINGS Goal: To sustainably reduce poverty and hunger in USG-assisted areas Progress toward the overall Feed the Future goals is measured through PBS data collected on the following two indicators, as outlined in the RF: 1. Prevalence of Poverty: Percent of people living on less than $1.25 (ZOI PBS) At the time of this evaluation, interim PBS data was available for 17 of the 19 focus countries.35 We received data for 17 of the 19 focus countries for this required indicator. Out of these 17 countries, 11 countries36 showed statistically significant37 progress in reducing the prevalence of poverty in their ZOI. Estimates also indicate that prevalence of poverty decreased in (northern) Kenya38 and Uganda, was unchanged in Honduras, and increased in Mozambique, Senegal, and Tajikistan, but results were not statistically significant for Honduras and Senegal, and statistical significance could not be concluded for the other results.39 It is worth noting that there is a current economic recession in Tajikistan, a factor likely to be affecting the increase in poverty rates. 2. Prevalence of stunted children under five years of age (ZOI PBS) We received data for 17 of 19 focus countries for this required indicator. Bangladesh, Cambodia,40 Ghana, Guatemala, Honduras, Kenya (HR/SA), Malawi, and Rwanda showed a statistically significant reduction in prevalence of stunting. Ethiopia, (northern) Kenya, Liberia, Tajikistan, Uganda, and Zambia also recorded positive performance; however, the results for Ethiopia and Zambia were not 35 Haiti and Mali were the two countries that the team did not have PBS data for – there were data collection issues in Haiti and a delayed start because of a coup in Mali. 36 Bangladesh, Cambodia, Ethiopia, Ghana, Guatemala, Liberia, Malawi, Nepal, Rwanda, Tanzania, and Zambia. 37 Throughout this section, the evaluation team is referring to statistical significance at p<0.01 to p<0.05 range for all indicators. The one exception to this is the prevalence of exclusive breastfeeding in Senegal, which reported significance the 90% level. 38 The baseline for the Kenya resilience ZOI included Marsabit, Isiolo, and Turkana counties. Baseline data collection did not take place in the ZOI counties of Garissa and Wajir due to security concerns. 39 Throughout this section, when we state that statistical significance could not be concluded, we are referring to one of the following reasons: Overlapping confidence intervals (CIs) or CIs at baseline not being available. 40 Based on review of the focus country portfolio reviews. It is worth noting that for Cambodia, while the PBS provides evidence of significant declines in prevalence of poverty and stunting, the portfolio review highlights concerns that there are still deep pockets of significant poverty and malnutrition within the ZOI, indicating that not all areas covered by programming have benefitted and will require further targeted programming. 35 statistically significant and statistical significance could not be concluded for the other four countries (Mozambique, Nepal, Senegal, and Tanzania). 3. Prevalence of underweight children under five years of age (ZOI PBS) We received data for 12 of the 19 focus countries for this required impact-level indicator. Out of the 12 countries, Bangladesh, Ethiopia, and Rwanda showed a statistically significant decrease in the prevalence of underweight children under five years of age. In Tajikistan and Mozambique, there was a statistically significant increase in prevalence. The current economic crisis in Tajikistan is cited as affecting nutrition outcomes and undermining local capacity to enforce health and nutrition regulations. In Mozambique, documents provided by BFS indicate there was a lack of programming in the ZOI. Although prevalence decreased in Cambodia, Honduras, Nepal, Senegal, and Uganda, and increased in Malawi and Zambia, results for Malawi, Honduras, Senegal, and Zambia were not statistically significant and statistical significance could not be concluded for the other results. First-level Objective 1: Inclusive agricultural sector growth Inclusive agricultural sector growth is measured using three indicators: Percent change in agricultural gross domestic product (GDP)41 (national/regional); Daily per capita expenditures (as a proxy for income) in USG-assisted areas (ZOI PBS); and Women’s Empowerment in Agriculture Index (ZOI PBS). For the daily per capita expenditures (as a proxy for income) in the USG-assisted areas indicator, we received data for 12 of the 19 focus countries on this required outcome indicator. Five of the countries – Bangladesh, Cambodia, Ghana, Rwanda, and Tajikistan – displayed a statistically significant positive trajectory. While an additional five countries – Malawi, Nepal, Senegal, Uganda, and Zambia – also demonstrated positive trajectories, results for Senegal and Zambia were not statistically significant and statistical significance could not be concluded for the other three country results. Honduras and Mozambique moved in a negative direction; however, the change in Honduras was not statistically significant, and statistical significance could not be concluded for Mozambique. 41 This indicator data in FTFMS was undergoing data quality checks during the time of this evaluation and therefore was not included in the analysis. 36 While the WEAI provides useful data, during the time of this evaluation, both baseline and interim WEAI data was only available for Bangladesh.42 Interim assessments for other focus countries reported on the sub-indicators that make up the WEAI, but did not report on values for the full index. These focus countries noted that “the full WEAI will be collected during the next interim survey in 2017.”43 As such, the team was unable to conduct comparative analysis on the WEAI at the initiative level. The causal pathway to measuring inclusive agricultural sector growth is informed by five IRs and eight sub-IRs supporting the IRs. Below, we present a discussion of progress against indicators for each of the five IRs: 1. Improved agriculturr productivity, 2. Expanded markets and trade, 3. Increased investment in agriculture and nutrition-related activities, 4. Increased employment opportunities in targeted value chains, and 5. Increased resilience of vulnerable communities and households. IR 5 is part of the causal pathway for both first-level objectives. IR 1: Improved agricultural productivity Gross margin (IM) is the RF indicator to assess progress on IR 1. This indicator, however, is designed to be assessed at the commodity level, which is not conducive to aggregation across commodities and focus countries for conclusions at the initiative level. Instead, for information on progress in agricultural productivity per value chain, the evaluation team conducted a systematic review of 2015 portfolio reviews, which include custom indicators found relevant by implementing partners in the focus countries. The main findings from this review are that most VCs are on target to meet FY 2016 and/or FY 2017 outcome level targets. The most common value chains across focus countries are maize and rice, with the former being on target to reach its 2016 objective in eight44 out of 10 focus countries. In Haiti, maize yields may not reach their 2016 objective due to the effect of a prolonged drought, and de-scoping of maize in the Feed the Future-North program. In Zambia, Feed the Future reached 117% of the total maize farmers that Feed the Future planned to reach by 2016. Even though 100% of farmers in the ZOI will increase crop yields, poor rainfall in 2015/16 makes it unlikely that FY 2016 targets for maize yields will be met. Rice is the second most common value chain, with yields on target to reach FY 2016 objectives in three45 out of five countries. In FY 2015, 42 USAID, “Selected indicators for the Feed the Future Zone of Influence in Bangladesh: Changes from 2011/12 Baseline to 2015 Midline.” IFPRI. May 16, 2016. 43 USAID. “Feed the Future Tajikistan 2015 Zone of Influence Interim Assessment Report.” IFPRI. Feb-March 2015; USAID. “Feed the Future Uganda 2015 Zone of Influence Interim Assessment Report.” October 2015. 44 Ethiopia, Ghana, Kenya, Nepal, Rwanda, Senegal, Tanzania, and Uganda. 45 Ghana, Haiti, and Tanzania. 37 Bangladesh and Cambodia had achieved 50% and 80% of targeted yield increase for FY 2016. Uneven rainfall in Bangladesh hampers rice harvests but Feed the Future is encouraging farmers to focus on the rain-fed season. In Cambodia, the gap in programming and current drought impacted dry season rice. Dairy and livestock value chains are also performing well and are on track to meet their targets of yield increases in Ethiopia, Ghana, Rwanda (dairy), and Mali (livestock). Horticulture is on target to meet its objectives in Honduras and Kenya, but is not on target in Cambodia and an outcome level result for FY 2015 is not available for Guatemala. All agricultural production was negatively impacted by the El Niño-linked drought in Cambodia, and a gap in programming and unfavorable economic environment in the region further depressed performance. El Niño has also negatively impacted progress on other value chains in numerous other Feed the Future focus countries, including Malawi, Honduras, Tanzania, and Zambia. Other issues negatively affecting performance across value chains include: global economic trends that affect commodity pricing; conflict and its impact on exports; macroeconomic and political instability; exchange rate depreciation’s impact on the cost of inputs; unexpected rises in crop diseases, such as black twig borer; and budget constraints affecting USAID and partners’ programming. IR 2: Expanding markets and trade Progress towards this IR is measured by three outcome indicators,46 as well as six indicators for four sub‐ IRs:47 1. “Value of incremental sales (collected at farm-level) attributed to Feed the Future implementation” (IM) During interviews in the five fieldwork countries, this indicator was identified as difficult to measure. Matched pairs for value of incremental sales are only reported for 2014 and 2015, and during this time the reported value of incremental sales aggregated across focus countries rose by 54.27% from $541,541,816.5 to 835,419,422.44. On average, over these two years, 13 of the 14 countries that reported on this indicator exceeded targets. Haiti only reported a matched pair for 2014 and in that year it achieved 16.38% of its target. Deviation narratives attribute this underachievement to the severe drought that reduced yields of all crops. 46 The three indicators are: 1) Percent change in value of intra-regional trade in targeted agricultural commodities, 2) Value of exports of targeted agricultural commodities, and 3) Value of incremental sales. 47 The Feed the Future RF does not have any indicators to measure sub-IR 2.1 “Enhanced agricultural trade.” 38 There were high levels of variation in performance between 2014 and 2015 for numerous countries on this indicator. For example, Bangladesh exceeded targets by 311.11% in 2014 but only achieved 68.39% of its target in 2015. Deviation narratives associate the 2015 dip to the slow start of agriculture value chains, as well as the conclusion of certain programming and calculation adjustments to more accurately adhere to the definitions. Similarly, Senegal’s proportion of target achieved also dropped from 292.95% to 15.1% due to rainfall deficit, and structurally lower yields across project scale-up zones and more subsistence-oriented rice farming in that zone. In the other direction, Nepal achieved 162.58% of its target in 2014 and 767.06% in 2015. This high fluctuation was due to higher yields resulting from improved technologies and practices, as well as faulty target-setting that did not factor price increases into the incremental sales target. Mali’s performance improved from 77.07% to 130.32%; deviation narratives attribute this to erroneous reporting. 2. “Value of exports of targeted agricultural commodities as a result of USG assistance” (IM) Graph 1 shows aggregate performance for this indicator moved in a positive direction, and on average, six48 out of the nine countries that reported on this indicator surpassed targets. Deviation narratives provided potential reasons for fluctuations and/or missed targets in certain countries. Malawi’s performance was impacted by government restrictions on exports and uncaptured sales through informal channels. Mozambique also witnessed a substantial drop in 2015 partially due to aflatoxin concerns with international markets. For FY 2015, Haiti reported achieving 2,847.68% of its target, which is attributed to an underestimated target and potential problems with extrapolation of the total value of exports based on a survey sample. 48 Ethiopia, Guatemala, Haiti, Mozambique, Rwanda, and Uganda. 39 Graph 1: Value of exports of targeted agricultural commodities as a result of USG assistance. Reported values and linear directional trend 2011 -2015. 0 100 200 300 400 500 600 700 800 900 2010 2011 2012 2013 2014 2015 Aggregate reported values Millions Value of exports of targeted agricultural commodities as a result of USG assistance IR 3: Increased investment in agriculture and nutrition‐related activities Progress towards this IR is measured by one standard output and two RiA outcome indicators: Number of public-private partnerships (PPPs) formed as a result of Feed the Future assistance (IM); Value of new private sector investment in the agriculture sector or food chain leveraged by Feed the Future implementation (IM) ; Number of firms (excluding farms) or civil society organizations (CSOs) engaged in agricultural and food security-related manufacturing and services now operating more profitably (at or above cost) because of USG assistance (IM). All three indicators displayed a positive trend in aggregate performance (refer to Graph 2 for a visual representation of progress in the number of PPs formed). Mali and Haiti missed targets, on average, on the number of PPPs formed; however, both countries surpassed targets on the value of new private sector investment and on the number of firms operating more profitably. While on average, a majority of countries surpassed targets on the value of new private sector investment, four49 out of eight countries that reported missed targets on the number of firms or CSOs engaged in agricultural and food security-related manufacturing and services are now operating more profitably. Deviation narratives cited incorrect targets entered in FTFMS, delays in partner collaboration, and difficulty in substantiating marketplace support for agricultural and food security-related manufacturing and services as potential reasons for stalled or negative performance in some countries. 49 Ethiopia, Ghana, Tajikistan, and Uganda. 40 Graph 2: Number of public-private partnerships formed as a result of FTF assistance - Reported values and linear directional trend 2011-2015. 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 Number of public-private partnerships formed as a result of FTF assistance 2010 2011 2012 2013 2014 2015 2016 IR 4: Increased employment opportunities in targeted value chains 1. Progress towards this IR is assessed through a single, critical, outcome indicator: Number of jobs attributed to Feed the Future implementation50 (IM). The aggregate values for this indicator showed consistent upward progress from FY 2011 to FY 2015. Half of the 14 countries that reported on this RiA indicator exceeded targets, on average, across the years. Bangladesh, Kenya, and Mozambique displayed a negative trend in performance with the number of jobs attributed to Feed the Future implementation decreasing from FY 2010 to FY 2015. Deviation narratives show that IPs found that the creation of paid jobs to meet Feed the Future standards is challenging. In some cases, job creation for a period of over one month in specific value chains is not possible, and scaling up did not take place as fast as expected in some value chains. Project delays and ambitious target-setting are also cited in deviation narratives. Additionally, the deviation narratives across countries for this indicator showed that in some cases the definition of new jobs created was misunderstood by IPs, and this led to over-reporting and unrealistic targets in some cases. One of the main sources of confusion was whether or not self￾employed individuals should be counted. 50 Based on key informant interview analysis, the evaluation team noted that this indicator was not well understood and was inconsistently measured. 41 IR 5: Increased resilience of vulnerable communities and households As mentioned above, this IR also contributes toward First-Level Objective 2: Improved nutritional status of women and children. Progress toward the IR is assessed on the basis of four indicators, the first two of which are impact-level indicators: 1. Prevalence of households with moderate or severe hunger (ZOI PBS) We received data for 11 out of 19 focus countries for this indicator. Ghana, Liberia, and Senegal displayed a statistically significant reduction in households with moderate or severe hunger. Malawi and Zambia displayed a statistically significant increase in prevalence. El Niño’s devastating effects in the Malawi ZOI may have contributed to this performance.51 Tajikistan and Uganda also displayed an increase in prevalence, while Nepal and Mozambique had decreases in prevalence; however, statistical significance could not be concluded for these results. Honduras showed no statistically significant change. Baseline data was not collected for the ZOI as a whole for Cambodia. 2. Depth of poverty: mean percent shortfall relative to the $1.25 poverty line (ZOI PBS) This indicator was established in 2014,52 and is required for countries with resilience programming or zones. We received data for nine out of 19 focus countries for this indicator. Nepal and Rwanda showed a statistically significant reduction in depth of poverty. Uganda, Malawi, and Zambia also recorded decreases in depth of poverty, but Zambia’s results were not statistically significant, and statistical significance could not be concluded for Uganda and Malawi. There was an increase in the depth of poverty in Tajikistan, Senegal, and Mozambique but statistical significance could not be concluded for these results. Baseline data was not collected for the ZOI as a whole for Cambodia. The two IM indicators (Number of USG social assistance beneficiaries participating in productive safety net (IM) and Number of vulnerable households benefiting directly from USG assistance (IM)) display a positive trend in aggregate performance. First-level Feed the Future Objective 2: Improved nutritional status (women and children)53 Progress toward improved nutritional status of women and children is measured using three impact indicators, detailed below: 1. Prevalence of wasted children under five years of age (ZOI PBS) 51 For more information on probable causes of decline, see: Feed the Future Focus Countries FY 2015 portfolio review presentations. 52 https://agrilinks.org/sites/default/files/resource/files/Indicator_Refresh_Consultation_PPT_Final.pdf. 53The first IR contributing to First-Level Objective 2 is IR 5 “Increased resilience of vulnerable communities and households,” which is considered in more detail under Objective 1 above. 42 We received data for six out of 19 focus countries for this required indicator. Bangladesh, Nepal, Malawi, and Senegal had a statistically significant decrease in prevalence of wasted children. Cambodia and Zambia also reported decreases, but Zambia’s result was not statistically significant and statistical significance could not be concluded for Cambodia. 2. Prevalence of underweight women (IM) We received data for 10 out of 19 focus countries for this required indicator. Prevalence of underweight women statistically significantly decreased in Cambodia and Bangladesh, but increased in Honduras. While results also showed decreases in prevalence in Malawi, Tajikistan, and Zambia and increases in Mozambique, Uganda, Senegal, and Nepal, Senegal’s result was not statistically significant, and statistical significance could not be concluded for the other results. The causal pathway to measuring improved nutritional status of women and children is informed by four IRs, including IR 5 (listed above): Increased resilience of vulnerable communities and households, IR6: Improved access to diverse and quality foods, IR7: Improved nutrition-related behaviors, and IR 8: Improved use of maternal and child nutrition services. A discussion of each of these IRs is presented below. IR 6: Improved access to diverse and quality foods This is the second IR contributing to Objective 2 and is measured by the following two outcome indicators: 1. Prevalence of children seven to 23 months receiving a minimum acceptable diet (MAD) (ZOI PBS) We received data for three out of 19 focus countries for this indicator. In Zambia, there was a statistically significant increase in children receiving a MAD. Bangladesh, Malawi, and Mozambique displayed reductions in MAD, but statistical significance could not be concluded. 2. Women’s Dietary Diversity: Mean number of food groups consumed by women of reproductive age (ZOI PBS) We received data for 10 out of 19 focus countries for this indicator. Honduras, (Northern) Kenya, Senegal, Uganda, and Zambia had statistically significant increases in women’s dietary diversity, while Nepal and Tajikistan showed statistically significant decreases. Graph 3 shows baseline and interim results for all statistically significant changes in women’s dietary diversity from baseline to interim. As mentioned above, prolonged economic recession in Tajikistan may have adversely affected nutritional outcomes, and in Nepal, a devastating earthquake disrupted communities where Feed the Future works, although gaps narrowed between disadvantaged and non￾disadvantaged households for measures of dietary diversity among infants and young children. 43 Malawi and Mozambique had slight increases in women’s dietary diversity but statistical significance could not be concluded. No baseline was collected for the Cambodia ZOI. Graph 3: Women’s Dietary Diversity: Baseline and Interim Statistically Significant Results 0 1 2 3 4 5 6 Honduras (northern) Nepal Senegal Tajikistan Uganda Zambia Women's Dietary Diversity ya Ken Interim Result Baseline Result IR 7: Improved nutrition related behaviors This IR is measured by a single outcome indicator: Prevalence of exclusive breastfeeding of children under six months of age. (ZOI PBS) The evaluation team received data for eight out of 19 focus countries for this indicator. Tajikistan and Mozambique displayed a statistically significant positive change, while there was a statistically significant decrease in prevalence in Senegal. Cambodia, Malawi, Nepal, and Uganda reported negative changes; however, statistical significance could not be concluded. There was no statistically reliable baseline estimate for Zambia. It is worth noting that Tajikistan has already surpassed its FY 2016 targets for this indicator. IR 8: Improved use of maternal and child nutrition services54 This IR is measured by six indicators, four of which are included in this analysis: Number of people trained in child health and nutrition through USG-supported program (IM); Number of health facilities with established capacity to manage acute under‐nutrition (IM); Number of children under five who received Vitamin A from USG‐supported programs (IM); and Number of children under five reached by USG-supported nutrition programs (IM). 54 Data was unavailable for analysis during this evaluation for the following indicators for IR 8: Prevalence of anemia among children 6-59 months (S); Anemia in women of reproductive age (RiA). 44 All four indicators displayed a positive trend in aggregate performance and overall, the portfolio reviews show that almost all focus countries are on track to meet the FY 2016 nutrition outcome and activity level results. In 2015, 1155 out of 17 countries achieved 90% or more of their targets for the number of children under five reached by a USG-supported nutrition program. Tanzania has already exceeded its FY 2016 targets for this indicator. Factors that contributed to exceeding targets include: projects ramping up as they were previously behind schedule, contributions from volunteers, and mass health screening activities. Some focus countries achieved less than 60% of their target in 2015 due to project delays, activity termination, overestimated targets, and issues with reporting and concerns about double counting, which were corrected for. QUESTION 2 CONCLUSIONS 1. Based on the evaluation team’s analysis, it can be concluded that the Feed the Future initiative is, on average, performing well against each of its eight IRs outlined in the RF across focus countries. Results of IM-level indicators, coupled with changes in impact-level (PBS) indicators, provide evidence that Feed the Future is, on the whole, contributing to reductions in global poverty and hunger. Of the statistically significant PBS data results that are available, we see that reductions in prevalence of poverty across countries have ranged from 7.46% to 35.69%, and reductions in prevalence of stunting range from 6.05% to 40.37%, indicating that the initiative is on track to achieve targets of contributing to a 20% reduction in poverty and stunting overall. However, given that this is a subset of data, the evaluation team is unable to determine the extent to which Feed the Future, as a whole, including all 19 focus countries, is on track to achieve its five-year goals of contributing to reductions of 20% in poverty and stunting as a result of the combination of the IRs that lead to the goals. This is primarily due to the nature of analysis of an initiative that is composed of 19 discrete country portfolios. Although each Feed the Future focus country reports on certain required indicators, each country’s performance is measured by a unique set of indicators, including custom indicators that best suit the Feed the Future programming in that country. As such, each country’s performance toward the overarching Feed the Future goals is defined by a different configuration of indicators, which is not particularly conducive to aggregation and analysis at the initiative level across all 19 countries. 2. Overall, performance on IM indicators is moving in a positive direction. However, performance across countries and indicators varies. Nepal, Honduras, and Rwanda have moved in a positive direction for more than 90% of the indicators they reported on. Cambodia, Tanzania, Mali, Ghana, Guatemala, Kenya, Senegal, Zambia, Liberia, Ethiopia, Uganda, and Malawi also moved in a positive direction for at least 70% of the indicators they reported on. Conversely, Mozambique, 55 Bangladesh, Ethiopia, Guatemala, Kenya, Liberia, Malawi, Nepal, Rwanda, Senegal, Tajikistan, and Zambia. 45 Haiti, and Bangladesh specifically stand out as they are moving in a negative direction on almost half (or more) of the IM indicators they reported on. Similarly, the majority56 of countries that reported values demonstrated a negative trend for the number of children under five who received Vitamin A through USG-supported programs. 3. Targets vs. actual analysis of IM indicators showed substantial variation in the percentage of targets met across countries, and also across years for the same country. The evaluation team triangulated findings from deviation narratives, portfolio reviews, and KIIs to understand these variations, and some of the most common reasons for variations can be categorized into three categories: reporting errors, performance issues, and external environmental factors affecting performance. Reporting errors were caused by confusion surrounding measurement for indicators, which led to double counting, undercounting, and other issues. In other cases, data was measured correctly but entered incorrectly into FTFMS and sometimes the mistake was only realized once the reporting season was over and FTFMS was closed so that no more adjustments could be made. Performance issues included project delays at either start-up or scale-up, inaccurate planning leading to incorrect target-setting, and budget constraints affecting implementation. Lastly, many external factors were cited as having a substantial impact on performance – most commonly, weather fluctuations, including droughts and floods; macro-economic trends impacting exchange rates, input and commodity prices and exports; and macroeconomic, political, and security concerns impacting access, programming, and immigration. The evaluation team also came across additional deviation narratives that could not be filed into any of these categories. For example, some IPs explained deviations by citing a sudden increase in people covered by programs, a bulge in volunteers, or the fact that trainees were suddenly dropped from training programs due to insufficient progress. 4. For 11 of the 17 countries, there were statistically significant decreases in the prevalence of poverty in the ZOI, and for eight of the countries, there were statistically significant decreases in the prevalence of stunting in the ZOI. 56 50% or more. 46 Table 1: Progress on PBS indicators COUNTRY PREVALANCE OF POVERTY PREVALANCE OF STUNTING Baseline Result Interim Result Difference Baseline Result Interim Result Difference Bangladesh 40.50% 34.01% -16.02% 36.90% 32.30% -12.47% Cambodia 11.70% 8.68% -25.81% 43.95% 33.70% -23.32% Ethiopia 39.90% 35.00% -12.28% 49.20% 47.10% -4.27% Ghana 22.20% 19.60% -11.71% 36.10% 29.90% -17.17% Guatemala 5.90% 4.28% -27.46% 67.40% 63.32% -6.05% Honduras 45.80% 45.80% 0.00% 38.30% 26.10% -31.85% Kenya (HR/SA) 44.70% 46.92% 4.97% 35.10% 20.93% -40.37% Kenya (northern) 61.90% 58.50% -5.49% 27.60% 26.30% -4.71% Liberia 49.40% 39.80% -19.43% 43.10% 34.30% -20.42% Malawi 66.70% 54.50% -18.29% 49.20% 42.30% -14.02% Mozambique 62.00% 66.50% 7.26% 51.60% 51.80% 0.39% Nepal 32.50% 20.90% -35.69% 45.20% 47.00% 3.98% Rwanda 67.00% 62.00% -7.46% 46.30% 39.70% -14.25% Senegal 34.30% 40.50% 18.08% 23.10% 25.80% 11.69% Tajikistan 8.80% 10.40% 18.18% 30.70% 29.30% -4.56% Tanzania 37.20% 28.20% -24.19% 34.70% 37.70% 8.65% Uganda 32.90% 32.10% -2.43% 33.00% 29.20% -11.52% Zambia 88.00% 80.90% -8.07% 45.50% 38.40% -15.60% Note: Lines that are bolded and in color indicate statistically significant change. Remaining lines are either not statistically significant or are inconclusive without further analysis. Findings from the PBS surveys do not always support the theory of change provided in the RF. For example, in Honduras, stunting in children declined from 38.3% to 26.1% and the MAD for children improved from 6.9% to 17.9%, but other nutrition and poverty indicators showed no significant progress. Similarly, in Malawi and Cambodia, there were strong declines in stunting and poverty while other indicators – such as children’s MAD and prevalence of exclusive breastfeeding under six months of age, which are supposed to reflect causal relationships according to the RF – showed no significant progress. Country-level progress does not always translate to even progress throughout the country ZOI. Impacts of Feed the Future interventions vary within country ZOIs, with uneven progress on the ground. For example, in Cambodia, it is recognized that while the PBS provides evidence of significant declines in prevalence of poverty and stunting, the portfolio review highlights concerns 47 that there are still deep pockets of significant poverty and malnutrition within the ZOI, indicating that not all areas covered by programming have benefitted and will require further targeted programming. Similarly, in Senegal, although an impact on malnutrition has been attributed, there is high variance across the ZOI. 5. Our ability to comment on progress made against the eight IRs and impact level indicators was confounded by the validity and reliability of some indicators. A review of Feed the Future data quality assessments (DQAs), deviation narratives, and KIIs highlighted these issues for certain indicators. For example, the results for the value of incremental sales in Malawi may be inflated, as this was collected from both farmers and merchants, even though it should only be collected from smallholder farms directly assisted by Feed the Future.57 Similarly, the definition of new jobs created was, at times, misunderstood by IPs, and this led to over-reporting and unrealistic targets in some cases. One of the main sources of confusion was whether or not self-employed individuals should be counted. Indicators for job growth, gross margins, and incremental sales were referenced in KIIs as particularly difficult to measure. For these indicators, the data reported is sometimes an estimate and is not necessarily accurate or entirely reliable. QUESTION 2 RECOMMENDATIONS 1. Given the significant levels of variation found in the team’s analysis of targets vs actuals, the team strongly recommends development of guidance on how to set evidence-based targets for all indicators, not only PBS indicators, and how to set evidence-based targets when no baseline data has been collected/is available. Some of these variations may be due to seasonal fluctuations – for example, missing a planting season – but this information is not readily available and so it is hard to adequately contextualize indicators. Feed the Future would benefit from creating systems that can adjust for seasonal variations or other exogenous factors, so that data collected can still be comparatively analyzed. This can be achieved through more widespread implementation and internalization of the Collaborating, Learning, and Adapting (CLA) approach across the initiative. 2. Trends emerging from the PBS survey indicate that the theory of change suggested by the Feed the Future RF may need to be revisited to incorporate learning about causal relationships between agriculture, poverty, and nutrition over the past five years.58 One particular consideration 57 USAID. Data Quality Assessments for Feed the Future Indicators. March 21, 2014. https://agrilinks.org/sites/default/files/resource/files/Consolidated_DQA_PPT.pdfhttps://agrilinks.org/sites/ default/files/resource/files/Consolidated_DQA_PPT.pdf 58 For further sources illustrating this point, see USAID. “Synthesis of Evaluations Related to the Feed the Future Learning Agenda.” 2016. See also, unpublished IITA research: P. van Asten, 2015. “Poverty and Maize 48 is the potential discrepancy in impact on direct Feed the Future beneficiaries vs. impact on the general population in the ZOI. For example, PBS data could indicate movement of indicator results in one direction for the general population in the ZOI, while IM data could point to results in the other direction.59 While both these data points could be true, this illustrates the need for systemic change analysis within the ZOI that more carefully considers spread and tipping-point effects. 3. Specific indicators that IPs identify as particularly difficult to measure (value of incremental sales, gross margins, number of jobs attributed to Feed the Future) need to be reconsidered for utility as they relate to accountability measures and/or learning measures, and adjusted to facilitate more accurate reporting and evaluation of progress. The evaluation team noted that these indicators – or their components – were used in progress reports and/or portfolio reviews. As they are so widely used, their accuracy is essential. Additionally, portfolio reviews mostly used components of the gross margin indicator. Perhaps, these components, rather than the aggregate indicator, could be included in the RF reporting structure. One potential option is to conduct a cost-effectiveness analysis of identified “difficult” indicators to determine whether it is worth adjusting that particular indicator – and how to do it – or whether it is more beneficial to discard that indicator in favor of another one (or none). 4. Although there is a copious amount of guidance for monitoring and reporting on Feed the Future indicators, there are many IM indicators that do not provide particularly meaningful insight at the aggregate level across the initiative. While many of these indicators could be useful at the country or activity level, at the aggregate level it would be more beneficial to focus on fewer indicators that lend themselves to meaningful aggregation across countries and can tell an initiative-wide story in a reliable, yet standard manner. This involves placing more emphasis on consistent data sources and timing of data collection, to allow for reliable comparability across all focus countries. However, we recognize that the priority for Feed the Future is for reliable comparability over time at the country level. – an intimate relationship.” IITA presentation at USAID-Feed the Future offices in Uganda on 4 Dec 2015, 25 PowerPoint slides, unpublished. (Contact p.vanasten@cgiar.org for a copy.) 59 An example is highlighted in the “Feed the Future Progress Report,” 2015. p. 62: “In Honduras, incomes of Feed the Future beneficiaries increased by an average of 55% since 2011, from $0.89 to $1.38 per person per day. Incomes for extremely poor families (approximately 125,000 people) nearly doubled, from an average of $0.62 to $1.21 per person per day.” However, as noted above in the conclusions section, the PBS data for Honduras shows that stunting in children declined from 38.3% to 26.1% and the prevalence of children receiving a minimally acceptable diet improved from 6.9% to 17.9% but other nutrition and poverty indicators showed no significant progress. 49 Question 3a: How have value chain approaches been applied and what have been the successes and challenges to focusing resources on strategic and limited value chains. FTFMS data suggest that Feed the Future assisted many smallholders to expand their production of staple and micronutrient-dense crops. Based on indicator data discussed above under Question 2, the overall Feed the Future program will likely meet its targets over the life of the program. There has been a positive trend in aggregate performance on the following indicators: 1) number of rural households benefiting directly from Feed the Future interventions, 2) number of vulnerable households benefiting directly from Feed the Future, 3) number of private enterprises and groups of all kinds that applied improved technologies, and 4) number of people implementing risk￾reducing practices/actions to improve resilience to climate change, and in some instances aggregate targets have been exceeded. Value chain approaches, in the opinion of the evaluation team, no doubt contributed to these results. Feed the Future is on track to achieve output targets related to value chain activity design and implementation. Synergies established through Feed the Future activities appear to have enhanced market efficiency in general and facilitated the local and regional grain trade. There are benefits from the concentration of resources on a few value chains; this approach appears to have helped focus and reinforce efforts. In general, value chains successfully engaged a wide range of private sector, civil society, and institutional actors to address constraints and stimulate strategic linkages, improve coordination, and increase efficiency. Concentrating development resources in strategic geographic areas likely increased the opportunities to collaborate. Additionally, focusing on a few value chains is conducive to monitoring progress on production￾level indicators, since it is not practical to track more than a handful of crop-specific indicators, while aggregating across crops is not meaningful for other indicators. The range of value chain activities across Feed the Future focus countries contributed to increased productivity, improved access to local and national markets, and increased agricultural sales.60 Some type of activity was put in place in every focus country to address vulnerable populations. A common characteristic of these activities was a strong focus on staple crops. Feed the Future increased staple and nutrient-dense crop and animal production (food availability and access) of smallholders and poor farmers participating in Feed the Future activities. 60 FTFMS database. 50 Employment creation through value chain approaches appears to be poorly developed, and few value chain projects directly target the landless rural poor through job creation. According to the MYS situation analysis, national, regional, and international migration is increasingly common in a number of focus countries (Bangladesh, Ethiopia, Guatemala, Northern Kenya, Haiti, Honduras, Nepal, and Tajikistan). Currently, Feed the Future offers limited guidance on employment generation. However, numerous interviewees recognized both migrants (domestic, regional, and international) and youth as increasingly important target groups for Feed the Future programming assistance. Missions and implementers have found many ways of using Feed the Future and non-Feed the Future resources to address some of the potential implications of narrow value chain selection, such as geographic pockets of the extreme poor and of poverty and vulnerability. In some countries (Ethiopia, Haiti, Guatemala, and Zimbabwe),61 Food for Peace programming has been a primary mechanism but more integrated approaches have also been tried, such as working with pastoralists in northern Kenya and Karamoja in Uganda.62 Over the course of Feed the Future, numerous creative and promising approaches have been emerging – and learning in regards to the utility as well as the necessity of this sort of flexibility is mounting within Feed the Future. FFP resources are often relied on as the primary means to address highly food insecure and disaster-prone areas. There is also a tendency to achieve greater coverage of the poor and extreme poor using FFP and nutrition resources, neither of which have strong market access and value chain components. Capacity building of participating farmer businesses to support sustainable results has not occurred in a consistent and/or strategic manner. Data from interviews, and program documentation on market linkages and partnerships, provide a clear focus on sourcing surplus but place less emphasis on building business acumen in the areas of demand analysis, price discovery, product differentiation, contract negotiations, and other basic business training. QUESTION 3a FINDINGS 61 Portfolio reviews for 2013, 2014 and 2015. 62 Portfolio review, 2014. 51 Feed the Future grew out of the international food price crisis of 2007-2008,63 along with the upsurge of humanitarian response that followed,64 and the donor consensus that stagnation in agriculture investments and growth, as well as market imperfections, were at the center of the crisis.65 There was a shift from viewing the food crisis as a food access crisis – with significant repercussions on the poor in general – to a broader crisis66 requiring a significant commitment to expanding agriculture as a means to promote rural income growth and employment.67 While the Feed the Future Guide does not explicitly point to the relationship between food prices, poverty, and social and political unrest, high-level discussions, and the literature leading up to President Obama’s announcement of the initiative, make numerous references to this. The Feed the Future Guide recognizes the challenge of the rapidly rising demand for food, particularly in poorer developing countries that currently have low levels of agriculture productivity and growing populations. The Feed the Future initiative was pledged to design and implement agriculture strategies that are consistent with the Rome Principles, which center on food security. The initiative aligns with the third Rome Principle, which says: “We support rural development, creation of employment, and more equitable income generation and distribution, to overcome poverty and increase food access.”68 Further to this aim, the Feed the Future Guide states that it sees “our role and that of other donors as catalyzing pro-poor economic growth through providing political, financial, and technical assistance.” 63 The dramatic widespread increase in food prices (primarily rice, maize and wheat) was triggered by a number of economic and political factors that disturbed world cereals markets. The escalation in prices was exacerbated by the concurrent rise in the price of oil, which raised the costs of international, regional, and local transported foods. Cooking oil, beans, and a whole series of other staple foods were affected. See, for example: Wiggins, Steve, Sharada Keats and Julia Compton. “What Caused the Food Price Spike of 2007/2008? Lessons for World Cereals Markets.” Overseas Development Institute (ODI). 2010. See also: “The Global Social Crisis. Report on the World Social Situation” United Nations. 2011. 64 See: FEWS NET reports on the unfolding crisis at www.fews.net 65 See: “The Global Social Crisis. Report on the World Social Situation.” United Nations. 2011. See also: Elliot, Kimberly and Casey Dunning. “Assessing the US Feed the Future Initiative: A New Approach to Food Security.” 2016. Washington, DC, Center for Global Development; and: Lawson, Marian L., Randy Schnepf, and Nicolas Cook. “The Obama Administration’s Feed the Future Initiative.” Congressional Research Service. 2016. 66 The food price crisis and the resultant political unrest has been repeatedly cited as a primary underlying cause of social unrest in the Middle East, West Africa, and Southern Africa as noted in the above sources and in many other documents and development and political forums. 67 The Feed the Future Guide (2010). 68 “Declaration of the World Summit on Food Security” (2009). Rome, World Summit on Food Security. 52 Defining poverty and extreme poverty: USAID recognizes that the causes and consequences of poverty are complex and that the best strategic response is to promote “pro-poor growth” with the goal of increasing their incomes.69 While the Feed the Future Guide describes its inclusive approach, the broader literature on pro-poor development provides more detail. These approaches use a poverty lens to assess the socio-economic situation of the poor, and design market-oriented value chain approaches to address their unique needs. Strategies offer lucrative and risk-reducing options that are acceptable to the poor; interventions that address the poor’s unique constraints, vulnerabilities, and opportunities; and services that are tailored to the poor and their expressed needs, and in which the returns to development are equitably distributed.70 The Feed the Future Guide does not define the poor in terms of income thresholds, nor does it distinguish between the poor and extreme poor: the Feed the Future Indicator Handbook uses the $1.25/day threshold for extreme poverty,71 as do many of the MYS documents. Food for Peace Title II Development funding is a core component of Feed the Future. During FY 2015, FFP Title II development programs were implemented in five Feed the Future focus countries.72 FFP programming tends to be oriented toward a poorer segment of the population than those Feed the Future activities funded with both Development Assistance (DA) funds and Economic Support Funds (ESF). Where FFP development activities occur in Feed the Future countries, they are part of Feed the Future, but FFP focal regions are typically characterized by a higher incidence of extreme poverty. Their target households tend to have smaller farm sizes or only limited cultivable areas surrounding their homes. These farmers tend to have few productive or personal assets and often live in remote communities with limited infrastructure. FFP and the Resilience Center provide the strategies and expertise to orient Feed the Future in the fulfillment of the third Rome Principle: “…ensure a comprehensive approach that accelerates inclusive 69 “Vision for Ending Extreme Poverty” (2015). USAID Last retrieved August 24, 2016. Retrieved from: https://www.usaid.gov/sites/default/files/documents/1870/Vision-XP_508c_1.21.16.pdf. Noted, from the document: “USAID’s interpretation of pro-poor growth focuses on increases in the incomes of the poor as opposed to the relative definition, which characterizes growth as pro-poor only if the incomes of the poor grow faster than those of the population as a whole.” 70 See, for example: Altenburg, Tilman. “Donor Approaches to Supporting Pro-Poor Value Chains” (2007). Report prepared for the Donor Committee for Enterprise Development Working Group on Linkages and Value Chains. Bonn, German Development Institute; see also: Wageningenur. https://www.wageningenur.nl/en/show/Value-chains-for-Propoor-Development-methodology.htm; and: Lopez, J Humberto (2006). “Pro-Poor Growth: A Review of What We Know (and What We Don’t). Washington, DC, World Bank, PRMPR; and: Feed the Future Tool for Targeting (2011). 71 Feed the Future Indicator Handbook (2014). 72 Congressional Budget Justification – Foreign Assistance Summary Tables Fiscal Year 2017. Table 20 – Food for Peace Title II, p. 107. 53 agricultural-led growth and improves nutrition, while also bridging humanitarian relief and sustainable development efforts.”73 The Feed the Future Guide adopts a market-oriented approach to achieving large-scale results through the promotion of small-scale agriculture, with the intention “…to build a foundation for sustainable and inclusive market-led growth.”74 For some time prior to the launching of Feed the Future, USAID and other USG agencies, as well as a number of multilateral institutions and initiatives, had been strategically investing in the private sector and the enabling environment in ways that align with and support the initiative’s poverty reduction intent (See Questions #8, #10, and #11 for more detailed discussions under each of these areas). What is potentially transformative about the Feed the Future approach is the stress on pro-poor investments throughout the value chain, which includes increased access to and efficacy of markets for small￾scale producers, and the explicit inclusion of nutrition-related activities (e.g., post-harvest preservation and processing for increased nutrient content of staple foods).75 The Feed the Future Guide defines value chains as: “…the full sequence of activities or functions required to bring a product or service from conception, through the intermediary steps of production, transformation, marketing, and delivery, to the final consumers.” This is a fairly standard definition and is consistent with what appears in the broader literature. While KII interviews indicated a diverse understanding of value chain approaches, Feed the Future activities across all focus countries indicated a fundamental understanding of the need to assess value chain nodes, identify constraints, and design and implement activities to address those constraints. a. Application of value chain approaches Variation Among Feed the Future Focus Countries Within the Feed the Future portfolio, different approaches to value chain-led poverty reduction were applied to different crops grown in different agro-ecological and economic contexts. A review of the 19 focus countries’ MYS and Strategic Reviews shows that value chain approaches are varied partly because applications are wide-ranging and strategies were designed to align with local government priorities, which are often varied, but to some degree related to agricultural growth, poverty, and malnutrition. Value chains are oriented toward: engagement with high-value external markets (e.g., mangoes in the case of Haiti, coffee and horticulture in other countries), staple and/or improved food production and markets (e.g., Mali, Ghana, and Tanzania), facilitation of regional 73 The Feed the Future Guide (2010). 74 The Feed the Future Guide (2010). 75 Ibid. 54 seed and grain trade (e.g., West and East Africa regional missions), watershed management (e.g., Haiti), food security (e.g., Cambodia), program integration (e.g., Senegal), women’s empowerment (e.g., Liberia), etc. See Annex VIII: Value Chain Summary Table for a complete value chain summary per country. Selected value chains corresponded to important local food staples and nutrient￾dense crops. For the most part, Feed the Future strategies focused on increasing smallholder production. Based on FTFMS data, value chain activities contributed to an increase in production of basic staples and nutrient-dense crops, and at the program-level targets were generally met over the life of Feed the Future. In general, Feed the Future MYSs were crafted around the theory that agriculture growth serves as an engine for economic growth and poverty reduction. A good example is the MYS for Honduras, which states that value chains for maize and bean staples were deliberately excluded because of the limited growth potential.76 Ethiopia’s MYS makes numerous references to economic growth as one of the main selection criteria for all six value chains. The Breadth of Feed the Future Value Chain Activities Feed the Future value chain activities include funding for increased on-farm productivity, access to financing mechanisms, improved market linkages, policy analysis and dialogue, regional mission resources allocated to regional seed and grain markets, and USDA’s work on trade and regulatory issues, among others. portfolio reviews cite numerous accomplishments and a steady progression toward alleviating critical constraints, as well as facilitating improvements in market services and functions. All 19 focus countries, during every reporting year, cite training conducted through producer groups and application of a diverse range of improved technologies. By 2015, Ethiopia was able to reach 50% of the livestock producers within the ZOI and increase livestock and dairy sales by 26%.77 In Malawi, the Agriculture Commodity Exchange (ACE) is an IP sub-grantee to a number of Feed the Future activities. ACE has warehouses in districts and warehouses for aggregation at the Southern and Central Region level to move produce to Blantyre and/or Lilongwe. Smallholders represent approximately 25-30% of the volume traded (see more details under Question #3b). Liberia, with a focus on reducing post-harvest losses and employment creation, supported the construction of six warehouses and 10 business hubs, which provide post￾harvest processing services.78 Peace Corps builds community storage facilities in Nepal.79 76 Maize and beans value chain work was added. In 2012, the portfolio review notes “subsistence corn and bean culture [is] difficult to change. The value chains listed in the review are horticulture, coffee, and basic crops. The MYS notes that commercial maize and bean production “is not feasible in the West (i.e., ZOI). An increase in yields of these two basic food crops was viewed as a means to free up land for more lucrative crops. 77 Ethiopia Portfolio Review, 2015 78 Liberia Portfolio Review, 2014. 79 MYS for Nepal. 55 Feed the Future’s Progress Report for 2013 notes that the Government of Ethiopia promulgated a new seed policy that targeted improving smallholder access to maize seed.80 All three regional missions in Africa have supported successful harmonization of regional seed trade regulations and procedures for implementation.81 In Central America, through numerous workshops and training of trainers, USAID provides capacity building and also disseminates information on new regulations and phyto-sanitary requirements for horticulture products imported to the United States.82 The elimination of inter-district trade permit requirements in Mozambique was also noted. Many of these findings have been analyzed and described in numerous Feed the Future evaluations. (See Questions #4, #10, and #11 for more detailed discussion of related issues). Primary Feed the Future Value Chain Approaches The common theme across Feed the Future value chain approaches is to strengthen market linkages and commercialize agriculture production. In addition, many activities focus on opportunities for significant transformation and added value (for example, livestock in Kenya and Ethiopia, coffee exports in Uganda and Ethiopia, dairy in Kenya, and horticulture for export and local supermarkets in Central America).83 In terms of identifying value chain approaches, attention was drawn to how Feed the Future activities support activities or functions required to bring agricultural products to the final consumer. Three general approaches were identified during the course of research: 1) push-pull, 2) push, and 3) inclusive growth through markets. Within these approaches, a synthesis of USAID evaluations indicates that value chain activities resulted in improved consumption patterns through greater purchasing power and increased availability of food from greater agricultural yields.84 In assessing the prevalence of the different value chain approaches, available sources did not always state explicitly which value chain approach was used, but it could often be inferred from the information provided. Push-Pull Value Chain – Based on a review of MYS documents, “push-pull” is the most common value chain approach within the Feed the Future portfolio.85 “Pull” strategies aim essentially to 80 “Growing Innovation, Harvesting Results” (2013). Feed the Future Progress Report. Washington, DC, USAID. 81 Various portfolio reviews. 82 Key informant interviews. 83 MYSs and portfolio reviews. 84 “Synthesis of Evaluations Related to the Feed the Future Learning Agenda, USAID, March 2016. Page 36 – Value Chain Investments. 85 “Pathways Out of Poverty: Applying Key Principles of the Value Chain Approach to Reach the Very Poor.” Washington, DC, USAID, MicroLinks Report 173. It should be noted that the evaluation team was not able to review all of the individual value chain project design documents across Feed the Future, and so this is not by any means an exhaustive analysis of all value chains occurring under Feed the Future. 56 create favorable market conditions and incentivize market actors to source from smallholders or households that have access to land and possess a set of characteristics deemed necessary for participation in the market. As a consequence, smallholders are “pulled” into the value chain. Smallholders receive a “push” with productivity-enhancing technologies, or services such as credit, and the opportunity to improve their market access through product aggregation and participation in farmer groups that may or may not work according to contracts with buyers or the buyer’s agent. This approach tends to focus on emergent and less-poor farmers.86 Kenya, Ghana, and Ethiopia provide examples of the “push-pull” approach, although the composition of individual interventions and local contexts in these countries are quite varied. Examples include, the Agriculture Growth Program (AGP) in Ethiopia, co-financed by the GAFSP (US Treasury), that simultaneously targets increasing agricultural productivity and production, accelerating commercialization and agro-industrial development, degradation reduction, and protection of vulnerable households from natural disasters; and the GAFSP Agricultural Productivity and Market Enhancement Project in Zambia that supports “one-stop shops” that provides agricultural inputs, has warehouse facilities for produce storage, and establishes linkages between raw material and service needs of industrial processors and smallholder producers.87 Push Value Chain – A variation of “push-pull,” this model emphasizes the “push” element, specifically through enhancement of farm productivity using new technologies and practices, as well as expansion of production, so that surplus could be a source of market income. Twenty implementing partners interviewed in field visits to five Feed the Future countries placed strong emphasis on production-focused activities, with half of them focusing inclusively on production and the other half limiting their market-focused work to connecting producers with buyers. Among the five fieldwork countries, Guatemala IPs seemed to be the most production-oriented, while those in Bangladesh were the least. The primary means by which production-focused work was delivered was through various forms of agricultural extension, including traditionally structured agricultural extension services provision; demand-driven (such as extension topics determined according to farmer demand); and farmer-led (where extension material is 86 Idem. 87 http://www.gafspfund.org/content/zambia USDA Food for Progress works in the Mozambican cashew sector, fostering buyer-seller relationships with major retailers and trading companies like Red River Foods, Inc., based in Virginia, Costco, based in Washington, and Whole Foods, based in Texas. Another Food for Progress program called MozaCaju works with all seven major cashew processors in northern Mozambique to meet industry standards for food safety and traceability and to obtain certification in organic and fair trade. 57 disseminated through peers, such as lead farmers or farmer field schools). Most IPs (over 80% of those interviewed, and a majority of those reported on in the FY 2015 Feed the Future portfolio reviews) utilized traditional agricultural extension methods, although delivery was often accomplished through a mixture of program-specific, private sector, or government extension agents, due to national extension service capacity limitations. The emphasis on the “push” element is common in FFP programs,88 and Mali is essentially an example of the “push” approach, where the idea is to first increase production and then identify markets. Similarly, Peace Corps volunteers in Ethiopia, Ghana, Kenya, and Nepal work on promotion of production through homestead gardening, farming, and small livestock rearing.89 Limited attention is given to actions aimed at market functions at higher levels of the value chain, or the potential to generate jobs and income at these higher levels. This approach is understood to be able to engage more households that fall within the “extremely poor”90 category. Inclusive Growth Through Markets – The Leading Economic Opportunities (LEO) framework defines “inclusive market systems development”91 as a value chain approach which adds horizontal linkages to the basic vertically linear value chain concept focused on market systems, in order to capture the interactions and synergies that potentially exist across different value chains.92 The Value-Added Maize Alliance in Uganda and ADVANCE in Ghana are examples of projects that take a more hands-off approach to smallholders, particularly poorer smallholders, and instead concentrate on key actors and functions higher up the chain. Other implementing partners and/or out-grower businesses work more directly with smallholder groups.93 This creates a sort of a “layered” design, where several IPs need to coordinate their activities and services related to the same beneficiary population to provide an important complement of services. Out of the 57 implementing partner interviews coded,94 36 (63%) Feed the Future-funded programs included at least one production-focused activity/component targeted at farmers (or producer groups), while 88 This finding is based on a review of FFP MYAPs. 89 From BFS-provided list of interagency support of Feed the Future. 90 “Pathways Out of Poverty: Applying Key Principles of the Value Chain Approach to Reach the Very Poor.” Washington, DC, USAID, MicroLinks Report 173. 91 For information on LEO visit https://www.microlinks.org/activities/leveraging-economic-opportunities-leo. 92 “Evaluating Systems and Systemic Change for Inclusive Market Development: Literature Review and Synthesis” LEO Report No. 3. June 2014. 93 KIIs with IPs in both countries. 94 For this analysis, implementing partners were defined as any respondent interviewed who had a defined role in a Feed the Future agricultural value chain project or activity. Partners included those who were directly funded by a USAID mission, or functioned as a local partner a prime implementer. Local partners included private sector companies, local organizations, and international organization. Implementers of policy￾focused projects were also included in the analysis. 58 33 (58%) IPs noted that that their programs included at least one activity focused on upward linkages in a value chain. Almost all IPs undertook at least one activity focused on addressing some sort of “systems” factor, which is not surprising considering that Feed the Future emphasized value chain programs that were integrated with other activities, especially those designed to address improved nutrition. Implementing partners engaged in a variety of activities along the value chain. Among the IPs interviewed, 23 (50%) discussed Feed the Future-funded activities which extend across “upper” and “lower” links in a given value chain and also included systems-focused work. Working across the spectrum for some IPs was strategic, while for others, the choice to work comprehensively was dictated by their contracts or cooperative agreements. As stated earlier, many activities focus on opportunities for significant transformation and added value. Typical Value Chain Target Populations While the evaluation team was not able to collect information on the selection criteria used for all Feed the Future value chain projects, poverty criteria are used to define all ZOIs and select target populations. A few countries, such as Mali, used a tiered selection process: they first identified high potential areas for the selected value chain crops and then further narrowed the target population based on the prevalence of small-scale producers.95 Nearly all value chain projects within the portfolio work directly or indirectly with poor small-scale producers. A review of available documents and data emerging from interviews revealed that Feed the Future value chain projects focus more on emergent and less poor farmers.96 For example, one LEO report97 indicates that the value chain activities have a strong focus on market systems approaches which require, in the majority of cases, at least some productive capacity, such as access to land.98 This is also evidenced by various project designs that employ a minimum farm size and possession of basic assets as selection criteria.99 While this does not imply Feed the Future value chain activities have not 95 Based on a review of MYSs and Strategic Reviews of all Feed the Future focus countries. 96 Emergent and less poor farmers are defined variously in the literature. But generally, “emergent farmers” refers to those who have larger than average farm sizes, participate in markets and invest in their farm enterprises without external support. Less poor farmers are distinguished from the broader group of poor farmers in that they tend to have access to adequate land and assets such that with some additional assistance they can regularly produce a surplus and engage in markets. Where an activity is based on land productivity, and thus associated with a requirement for land access as selection criteria, direct participants in production-based interventions would need to have some form of access to land. 97 “State of the Sector Report: Country-level Practices and Learning Related to Market Systems Development. Leveraging Economic Opportunities (LEO), Report No. 1.” April 2014. 98 “From Smallholders to Shareholders: A Guide to Optimizing Partnerships with the Private Sector for Smallholder Impact.” October 2014. Smallholders are defined as small-scale farmers, pastoralists, forest keepers, and fishers who manage areas varying from less than one hectare to 10 hectares. 99 Evidence was obtained through KIIs and a review of available project documents, portfolio reviews, baselines, and evaluations. 59 engaged the poor, or extreme poor, by nature of the market systems approach, a heavier emphasis is placed on a higher socio-economic tier of the population.100 Within the various ZOIs, IPs generally targeted two primary beneficiary groups – “vulnerable” beneficiaries (i.e., pregnant and lactating women and children under two, and landless, land￾insecure or ultra-poor) and “poor with assets” (i.e., smallholder farmers with a certain level of assets available – specifically land). In the five countries visited, two-thirds of respondents indicated that they engaged in activities intended to target the “poor with assets.” Finally, some IPs – especially those in Uganda and Guatemala – noted that they were implementing activities that were specifically designed to include youth. A perceived increase in emphasis on youth was noted by many IPs and was specifically discussed by USAID staff in Malawi, who noted that a youth focus was going to be written into forthcoming Feed the Future project Request for Proposals (RFPs). Pro-Poor Approaches As stated above, value chain projects work with different segments of poor populations with varying degrees of sophistication. During our research, we examined several value chain projects which use an explicitly pro-poor approach that seeks to increase the incomes of the poor.101 In taking this approach, these value chains play a key role in facilitating actions required to improve the poor’s terms of participation within the market system.102 In Guatemala, Anacafe and Funcafe form a partnership comprised of a sophisticated local smallholder coffee export association and its sister social foundation, that together provide a complement of economic and social assistance to small-scale producers, community members, and others engaged along the value chain.103 In Tanzania, Feed the Future activities facilitate market development by increasing access to finance for women and youth, promoting greater access to agricultural inputs, improving production technologies, and strengthening storage and aggregation. Food for Peace IPs, supported by FFP resources that have a specific statutory mandate under Public Law 480, Title II, start with identifying underlying household and community food security issues, 100 “Growing Innovation, Harvesting Results.” (2013). Feed the Future Progress Report. Washington, DC, USAID. 101 “Vision for Ending Extreme Poverty” (2015). USAID Last retrieved August 24, 2016. https://www.usaid.gov/sites/default/files/documents/1870/Vision-XP_508c_1.21.16.pdf. USAID’s interpretation of pro-poor growth focuses on increases in the incomes of the poor as opposed to the relative definition, which characterizes growth as pro-poor only if the incomes of the poor grow faster than those of the population as a whole. 102 “Operating Guide for the Making Markets Work for the Poor Approach” (2014). SDC. “A Synthesis of Making Markets Work for the Poor (M4P) Approach.” DfID. 103 Interview with Anacafe and Funcafe. 60 then build economic and social capacities of at each level; in some cases, these activities link to existing value chains, while in others, IPs build new value chains from the ground up.104 For example, FFP partners have, over several project cycles, been building value chains for oilseeds in Mozambique,105 and goat milk products in Guatemala. Value chain development under FFP has occurred in other commodities as well, in other focus and aligned countries, including Malawi, Zimbabwe, and Madagascar.106 FFP country programs in Guatemala, Malawi, and Zimbabwe take a longer-term farm systems approach. They work with farmers to identify strategies that will enhance the productivity of the farm, particularly around subsistence food crops, which then allows the household to produce adequate food for consumption and substitute cultivation area originally allocated to household food production for more lucrative crops. Value Chains and Nutrition The Feed the Future Guide states: “We will invest in improving nutrition throughout the value chain, including research to improve the nutrient value of staple food sources and post-harvest preservation and processing techniques that increase the nutrient content of processed staple foods.”107 A review of the MYSs for all 19 focus countries reveals that 11 countries noted that malnutrition and/or hunger were factors that determined the selection of the ZOI. Thirteen of the 19 countries’ value chains did not mention malnutrition or hunger as reasons for their selection of value chains, although vulnerability was sometimes listed as a factor in the value chain choice. In some cases, the ZOI areas where value chains work are different from those where nutrition interventions are implemented. In 2012 - 2013, SPRING conducted a “landscape analysis” of the Feed the Future program to determine the “pathways” through which agriculture activities can support improved outcomes related to nutrition.108 This analysis clarified how different agriculture interventions can contribute to the Feed the Future goal of reducing malnutrition, but in most cases these interventions need to be coupled with nutrition-specific interventions. For example, while the Cambodia HARVEST food security program exceeded its targets for value chain production, the results were quite different for home gardens focused on nutrition. The mid-term evaluation found that “home 104 KIIs with FFP Officers and IPs. 105 Mozambique MYS and portfolio reviews and an evaluation team member’s direct professional experience. 106 https://www.usaid.gov/sites/default/files/documents/1860/FoodforPeacePortfolioMay2014.pdf 107 “Feed the Future Guide” (2010), Washington, DC, USAID, BFS. 108 “A Landscape Analysis of Activities in 19 Focus Countries” (2014). Rosslyn, John Snow, Inc, SPRING. 61 gardens are viewed as a source of cash rather than additional nutrition.”109 Households that had participated in joint nutrition and home garden programming were more inclined to sell the horticultural produce from their gardens than consume it. Following on from the landscape analysis, SPRING also produced several presentations and guides on nutrition-sensitive agriculture, and offers technical assistance to Feed the Future value chain programs.110 SPRING notes that not all agriculture interventions should be expected to directly address nutrition, but where there are opportunities to make such a contribution, there are recommended best practices on how to do so, and these practices can increase the likelihood that programs promoting agriculture and nutrition linkages will achieve positive nutrition-sensitive outcomes.111 b. Successes and Challenges Expanding the ZOI and Value Chains to Include More Poor and Vulnerable Populations New value chains and geographic expansions are an example of how missions learned and adapted over recent years to bridge program gaps related to the coverage of the poor. For example, the Guatemala Mission developed an artisans value chain activity to support poor and landless indigenous women in the production and sale of traditional textiles. Also in Guatemala, Fintrac, along with local IPs, have launched two new activities for youth: one creating a youth farmer group oriented towards horticulture production and marketing, and another creating a small-scale cell phone-based communication system to disseminate agricultural messages to farmers. In order to address a more vulnerable population, Malawi added the orange flesh sweet potato (OFSP) as a value chain for both ZOIs, which will also be promoted for production for home consumption.112 In Uganda, Karamoja was added to the ZOI through a Strategy Change Memo. The area has unique challenges compared to the rest of Uganda. It is insecure due to conflict and also experiences frequent droughts. A livestock value chain was amended to the MYS because of perennial food security crises and the importance of livestock as a primary economic activity in the area, and because this was not originally selected as one of the MYS identified value chains. The Karamoja intervention is an integrated approach and includes nutrition, health, and resilience activities that 109 “Mid-Term Performance Evaluation of the Cambodia Harvest Project (Helping Address Rural Vulnerabilities and Ecosystem Stability” (2013). Washington, DC, USAID, BFS. 110 See the SPRING website: www.spring-nutrition.org 111 “Understanding and Applying Primary Pathways and Principles” (2014). Rosslyn, John Snow, Inc, SPRING and “Linking Agriculture to Nutrition: A Guide to Context Assessment tools” (2014). Rosslyn, John Snow, Inc, SPRING. 112 USAID Malawi Feed the Future “Change to Malawi’s Feed the Future Multi-Year Strategy.” October 1, 2015. 62 follow the USAID guidance on resilience. The Kenya Mission also added a pastoral area to the ZOI in order to address the recurrent shocks and resultant vulnerability of pastoralists. Similarly, Malawi amended its ZOI in 2015 to include areas of extreme poverty and high vulnerability. These additional districts included focus districts for FFP which allowed the Mission to strategically layer FFP and Feed the Future-promoted agricultural technologies and practices. These amendments to the ZOI illustrate a measure of flexibility within the Feed the Future approach, and learning in action on the part of the specific missions. Value Chains and Employment Generation The Feed the Future Results Framework and Program Guide note that value chains are tools to create employment, both on and off farms.113 However, it appears that employment or job creation is a challenge that has arisen over the life of Feed the Future. Eleven MYSs assert that value chains will generate either on-farm or off-farm jobs, or both. While most missions (17 out of 19 countries) have tried to report on the number of jobs created (indicator 4.5(2) of the Feed the Future Indicator Handbook), the reporting has not been uniform (lack of targets, reporting only for certain years) due to perceptions that it is a difficult indicator to define and/or measure in the field.114 Portfolio Reviews for four countries illustrate job creation among their results: Bangladesh reports jobs associated with shrimp hatcheries, Mozambique in cashew processing, Guatemala in coffee harvesting, and Ethiopia related to livestock sector development activities. In Ethiopia, the social safety net programs are an additional source of job creation that provide temporary employment on public works projects, productive asset transfers, training, and apprenticeships.115 Restriction on the Number of Value Chains Per Country A review of program evaluations, annual portfolio presentations, and field interview data indicated that opportunities may be missed due to restrictions on the number and selection process for value chains. Eight of the MYSs include three value chains, while Ethiopia, Kenya, and Rwanda managed to include more than four. Some of the bigger value chain IPs – in Ghana, Malawi, and Guatemala – felt that they could reach more people and be more responsive, inclusive, or impactful, if there was flexibility to work in other value chains. Feed the Future Bangladesh mentioned this issue in the FY 2015 annual portfolio review: “The portfolio in Bangladesh has a need to be broader and more expansive than a selected, small set of value chains – a strategy that just focuses on two or 113 See IR 4 of the Feed the Future Results Framework – Increased employment opportunities in targeted value chains. 114 This assertion is based on several KIIs in the field and two KIIs in Washington, DC. 115 “Food Security Country Framework for Ethiopia” (2015), Washington, DC, FFP and Berhane, Gussh, et al (2013). “Evaluation of Ethiopia’s Food Security Program: Documenting Progress in the Implementation of the Productive Safety Nets Programme and the Household Asset Building Programme.” Washington, DC, IFPRI. 63 three value chains may be too restrictive.” Feed the Future Rwanda indicated a need for flexibility with their decision to expand the list of value chains to include livestock, horticulture, Irish potato, and OFSP.116 More flexibility may also promote greater opportunities for working with the private sector. The need for flexibility in value chain selection was implicit in a key finding from the “Synthesis of Evaluations Related to the Feed the Future Learning Agenda,” which analyzed the impact of value chain interventions on income and employment and found that aligning anchor firms, financial institutions, and market actors was key to increasing incomes.”117 Reaching Significant Numbers of the Poor A review of MYSs, Country Investment Plans, and various ministerial strategies suggest that Feed the Future activities are focused on priority areas and populations. In most cases, the ZOI supports only a portion of a government’s focus on agricultural development or other policy priorities. For example, national agricultural strategies within focus countries are not limited to three or four crops or value chains. Furthermore, Feed the Future activities are implemented in a subset of the administrative units that comprise the government’s priority area (e.g., regions and districts of northern Ghana, and municipalities and districts of Guatemala’s Western Highlands). According to mission and IP key informants, the number of districts and communities where IP activities overlap and complement each other represents an even smaller subset of the Feed the Future ZOI and the priority area. Across Feed the Future programs, value chain interventions had highly variable coverage of the ZOI population, ranging from 8% of eligible farmers in Malawi and Ghana, to 67% in Ethiopia.118 Value chain IP interviewees report that their beneficiary populations represent less than 10-15% of the population. Because value chain approaches are by definition focused on agriculture production and markets, households that reside in more remote locations of the ZOI and have limited or no access to land are most likely to not participate directly (and very unlikely indirectly) in value chain activities.119 Beneficiaries in this category may, however, gain from knowledge and resource transfers within the community. In sum, there is a wide range of economic activity that 116 Feed the Future Rwanda – FY 2015 annual portfolio review presentation. 117 “Synthesis of Evaluations Related to the Feed the Future Learning Agenda, USAID, March 2016. Page 24. Key factors for increasing beneficiary income was through (1) promoting capacity of smallholder farmer organizations, (2) establishing clear commercial linkages, (3) strengthening marketing capacity, and (4) facilitating access to finance. “The key seemed to be aligning these factors with anchor firm, financial institutions and other market actors.” 118 2016 Feed the Future Portfolio Reviews. 119 “Synthesis of Evaluations Related to the Feed the Future Learning Agenda” (2016). Washington, DC, USAID, BFS. This statement is also based on a review of portfolio reviews and on the responses of nine key informants representing IPs, missions, and Washington-based USAID staff. 64 takes place within the priority areas and the ZOI, but falls outside the direct influence of Feed the Future value chains. In some cases, FFP resources and programming (which again, are also Feed the Future) are used to address food insecurity among a portion of the population excluded from direct participation by value chain selection criteria, but opportunities for FFP beneficiaries to link with Feed the Future value chain activities currently appear to remain quite limited. According to mission and FFP informants, the lack of geographic accessibility with associated product transportation costs creates a situation in which some of the districts that FFP served were too remote to profitably participate in value chains. One area for investment that could increase indirect impact on a wide range of smallholder farmers in the ZOI is improved roads for access to markets; it should be noted that Feed the Future funding is supporting road rehabilitation in Honduras and Rwanda through government-to-government agreements, while in Tanzania this occurs through a partnership with the World Bank.120 MCC has supported road rehabilitation in Ghana, Honduras and Senegal,121 and FFP implementers use food for work to rehabilitate feeder roads in Rwanda.122 QUESTION 3a CONCLUSIONS The variation in crop choices, designs, and contextual factors that impact value chain performance make it difficult to rigorously compare and contrast performance or to generalize across different value chain approaches in any systematic way. Similarly, the extraction of lessons learned or replicable successes is impossible with just the FTFMS datasets, results reports, and outputs from project evaluations to date. This important task requires a far more significant commitment of resources and a deeper dive into the designs and outcomes of specific projects and joint efforts (so-called “layered programs”). What is provided within this section is a program-level examination of what factors appear to be contributing to the overall Feed the Future objectives and what remains a challenge. 1. Value chain interventions increase agriculture production and support achievement of program targets. An increase in food production and sales expands local and regional food supplies. Thirteen out of 19 focus countries chose value chains for basic staples, including grains, roots/tubers, livestock, and fish. In addition, every focus country selected a micronutrient-dense crop or an animal/fish product. Value chains generally employed “push” or “push-pull” approaches that focused on increasing agriculture productivity, achieving higher levels of production, and 120 FY 2016 Feed the Future portfolio reviews: Honduras, Tanzania and Rwanda. 121 MYSs for Honduras, Nepal, and Senegal. 122 MYS for Rwanda. 65 marketing agricultural surplus. For the most part, they incorporated smallholders either directly or through collaboration with other implementers’ projects that focused on improved technologies and practices (e.g., climate-smart agriculture), input supply, and/or extension. Horticulture and livestock/fish value chains offer the potential for both market outlets and increased food diversity. As indicated in the introduction to this section, Feed the Future countries are on track to achieve output targets related to value chain activity design and implementation. Synergies established through Feed the Future activities appear to have enhanced market efficiency in general and facilitated the local and regional grain trade, but it is not possible to measure these outcomes with the current M&E systems in order to fully validate this observation. This is partly because there are few standard Feed the Future indicators that measure results in this space, or proxy system-level outcome indicators, and no indicators that directly link changes at higher levels of the value chain back to household level outcomes. 2. Focusing value chain resources only within the ZOI constrains development at a wider systems level to some degree. Although responses were mixed, key informants noted that value chains extend beyond ZOI borders, and sometimes there are relevant market bottlenecks and inefficiencies higher up along the value chain and thus beyond ZOI geographic boundaries. Input suppliers, livestock grazing areas, sources of critical watershed services, and the origins of natural and man-made shocks can all be located outside the ZOI and thus may be outside the area where Feed the Future can work. 3. There are benefits from the concentration of resources on a few value chains; this approach has helped focus and reinforce efforts. Based on MYSs, portfolio reviews, and a series of Feed the Future evaluations, WOG resources were largely focused on supporting consistent and complementary interventions that addressed key actors and functions along the value chains, as well as within the larger enabling environment. In general, value chains successfully engaged a wide range of private sector, civil society, and institutional actors to address constraints and stimulate strategic linkages, improve coordination, and increase efficiency. While focusing WOG resources on similar aims was seen as a benefit, many of the interviewees in the five countries visited said the restriction on the number of value chains, and the difficulties incurred when they chose to change and/or add a value chain, was too constraining and complicated the achievement of some objectives – specifically, reaching specific poor populations. It should be noted here that despite the benefits documented in this report, it is a key concern from IPs that more people could have been reached without the restriction on the number of value chains. Many missions, and some IPs themselves, managed to secure additional Feed the Future and non￾Feed the Future resources to address constraints regarding lack of agriculture financing, access to 66 inputs, links to local and national markets, and access to technology.123 In other cases, coordination among WOG actors and other donors (such as the EU with livestock in Kenya) helped close the gap. Concentrating development resources in strategic geographic areas likely increased the opportunities to collaborate, and several interviewees stressed this as a positive outcome. However, they also noted that greater flexibility in the parameters that guide the strategy designs and implementation could facilitate the introduction of complementary activities that emerged later as critical. 4. The restriction on the maximum number of value chains per country can limit opportunity for impact. Interviewees both in the field and in Washington have spoken at length on the implications of the restriction to three to four value chains. They say that more flexibility may promote greater opportunities for working with the private sector. This aspect of being able to respond to market opportunities was also highlighted in a review of evaluations on value chain interventions, which stated that the key to increasing income was critically linked to being able to align activities with anchor firms, financial institutions, and other market actors.124 The requirement to pre-select and limit the number of value chains constrains this flexibility. In general, value chain restrictions seem to have been viewed as more problematic than any downside related to the limited geographic focus of the ZOI. Despite the challenge, missions and implementers have found many ways of using Feed the Future and non-Feed the Future resources to address some of the implications, such as geographic pockets of the extreme poor and of poverty and vulnerability. In some countries (Ethiopia, Haiti, Guatemala, Zimbabwe),125 FFP programming has been a primary mechanism but more integrated approaches have also been tried, such as working with pastoralists in northern Kenya and Karamoja in Uganda.126 As the above examples indicate, over the course of Feed the Future, numerous creative and promising approaches have been emerging – and the learning of the utility as well as the necessity of this sort of flexibility is mounting within Feed the Future. USAID’s Resilience Policy offers useful guidance that mission are applying to their programming (see Question #1). These adaptive measures can provide useful lessons learned for the broader initiative. 123 Feed the Future FY 2015 portfolio reviews provide several examples of successful efforts by missions to leverage additional resources to support value chain activities. 124 “Synthesis of Evaluations Related to the Feed the Future Learning Agenda, USAID, March 2016. Page 24. Key factors for increasing beneficiary income was through (1) promoting capacity of smallholder farmer organizations, (2) establishing clear commercial linkages, (3) strengthening marketing capacity, and (4) facilitating access to finance. “The key seemed to be aligning these factors with anchor firm, financial institutions and other market actors.” 125 Portfolio reviews for 2013, 2014 and 2015. 126 Portfolio review, 2014. 67 5. Value chain approaches contribute to poverty reduction goals and reaching the poor, but are not a complete approach to either poverty reduction or to improved nutrition. The range of value chain activities across Feed the Future focus countries contributed to increased productivity, improved access to local and national markets, and increased agricultural sales.127 The level to which the poor with fewer resources benefited was unclear. Many focus countries made a limited or perhaps indirect contribution to income growth of poor rural households with limited or no access to land because by design they target households that have access to a minimum threshold of land. However, some type of activities was put in place in every focus country to address vulnerable populations. A common characteristic of these activities was a strong focus on staple crops. Feed the Future increased staple and nutrient-dense crop and animal production (food availability and access) of smallholders and poor farmers participating in Feed the Future activities. However, a clear link between vulnerable beneficiaries and how they would move into more commercial aspects of agricultural value chains was not completely evident. It should be noted that the “inclusive markets for growth” approach focused on the key aspects of market facilitation required to allow entry of the less poor, and this was evident in the approach of several focus country programs. 6. Distinguishing among the various strata and complex socioeconomic categories of “the poor” appears to be lacking in guidance as well as in value chain approaches. Growing recognition within the agricultural development community and literature, some of which has been supported through Feed the Future funding, that the rural poor is not monolithic128 emphasizes that different development approaches affect different types of poor rural households differently.129 Feed the Future research and several country programs have created typologies of the agrarian poor.130 USAID’s “Vision for Ending Extreme Poverty” states: “USAID recognizes that extreme poverty is a complex and multidimensional phenomenon; causes and consequences are not monolithic, and they vary according to region, gender, age and other variables.” 131 However, the team has not been 127 FTFMS Database. 128 TS Jayne, A Chapoto, N Sitko, M Muyanga, C Nkonde and J Chamberlin (2014). “Africa’s Changing Farm Structure and Employment Challenge.” Policy Synthesis, No. 9. Washington, DC, USAID. 129 KIIs with the Center for Resilience, FFP, missions and IPs and Gary, George; Laura Kuhl and Demese Chanyalew (2015). “External Mid-Term Performance Evaluation Report: Feed the Future Ethiopia” Washington, DC, USAID, FSB. 130 TS Jayne, A Chapoto, N Sitko, M Muyanga, C Nkonde and J Chamberlin (2014). “Africa’s Changing Farm Structure and Employment Challenge.” Policy Synthesis, No. 9. Washington, DC, USAID. And, the Uganda and Ethiopia country programs. 131 “Vision for Ending Extreme Poverty” (2015). USAID. Last retrieved August 24, 2016. https://www.usaid.gov/sites/default/files/documents/1870/Vision-XP_508c_1.21.16.pdf. 68 able to identify any Feed the Future guidance that describes a complex, multi-faceted, and stratified understanding of poverty that would support careful targeting of interventions aimed at different categories of vulnerability and poverty. The team recognizes that it is up to individual countries to tailor their programs to the situation in their country. Value chain experts interviewed say that it is critical that this socio-economic research conducted with Feed the Future resources be integrated into the thinking, vision, and strategy of Feed the Future going forward. 7. FFP resources are often relied on as the primary means to address highly food-insecure and disaster-prone areas. There is also a tendency to achieve greater coverage of the poor and extreme poor using FFP and nutrition resources, neither of which have strong market access and value chain components. This creates a disconnect between value chain economic and market access work on the one hand and FFP’s food security work or nutrition interventions on the other. Without deliberate linkages and coordinated programming, there is a chance that remote communities and the extreme poor will continue to be marginalized in the process of agriculture and rural development, which is a general problem noted by many researchers in the field of agricultural development. In the future, missions will likely apply the recommendations and approaches included in the resilience policy and program guidance. However, greater use of pro-poor approaches and best practices can ensure that strategies and program designs include productive opportunities as well as safety nets. 8. Employment creation through value chain approaches appears to be poorly developed, and few value chain projects directly target the landless rural poor through job creation. According to Feed the Future socio-economic research,132 the growing number of farms marginalized due to migration pressure and resource degradation will continually spawn youth and other underemployed or unemployed household members seeking work off the farm. Agriculture laborers are among the poorest in many countries.133 According to the MYS situation analysis, national, regional, and international migration is increasingly common in a number of focus countries (Bangladesh, Ethiopia, Guatemala, northern Kenya, Haiti, Honduras, Nepal, and Tajikistan). Both MYS and follow-on portfolio reviews note a growing concern for youth, particularly marginalized youth, who need jobs. These trends suggest that more jobs are needed in rural and small town settings to be able to absorb those seeking work off the farm. 132 TS Jayne, A Chapoto, N Sitko, M Muyanga, C Nkonde and J Chamberlin (2014). “Africa’s Changing Farm Structure and Employment Challenge.” Policy Synthesis, No 9. Washington, DC, USAID. 133 Feed the Future baseline reports for Rwanda, various individual projects in Yemen. 69 9. Currently, Feed the Future offers limited guidance on employment generation. In addition, the findings above suggest that the Indicator Handbook and Indicator Sheets have proved to be inadequate to support the IPs and missions in the application of the definitions and measurement in the field. However, numerous interviewees both in Washington and the field recognized both migrants (domestic, regional, and international) and youth opportunities as increasingly important target groups for Feed the Future programming assistance. 10. Capacity building of participating farmer businesses to support sustainable results has not occurred in a consistent and/or strategic manner. KIIs and an examination of portfolio reviews and other project documents indicate that most value chain approaches work with farmers in groups. The link to farmers is through IPs or government agricultural technicians, or lead farmers or out￾grower businesses that orient farmers on improved inputs and practices as well as on organizational skills and the formalization process. Data from interviews, and program documentation on market linkages and partnerships, provide a clear focus on sourcing surplus but place less emphasis on building business acumen in the areas of demand analysis, price discovery, product differentiation, contract negotiations, and other basic business training. These business skills are critical to empower farmers as well as enhance their independence and resilience. 11. Keeping in line with the intentions of Feed the Future as presented in the Guide and Indicator Handbook, value chain activities are expected to have a positive impact on poverty.134 The Guide notes the following strategic priority: “Investments will emphasize post-harvest and market research aimed at increased access and efficiency of markets for small-scale producers…”135 The Indicator Handbook references outcomes and impact in terms of extreme poverty. Therefore, it can be expected that Feed the Future strategies, including value chain components, will contribute to poverty reduction through deliberate and clear program pathways. While pro-poor approaches have been employed in all focus countries, there are no poverty-specific indicators or results that hold IPs accountable to poverty reduction and that would lead clearly to ultimate poverty reduction impact and orienting project design and implementation (see Question #1 for more detailed discussion of the framework and causal linkages for addressing poverty). QUESTION 3a RECOMMENDATIONS 1. Commission a study to identify what worked, where, and why. Important areas to assess are: 1) where and how value chain activities have successfully supported vulnerable populations; 2) 134 The Feed the Future Guide and Indicator Handbook define poverty as $1.25/day, the World Bank definition of extreme poverty. 135 Feed the Future Guide, 2010. 70 what group of indicators best capture standard market-level outcomes and more closely tie efforts at higher levels of the value chain to desired outcomes within the target population; and 3) which activities most effectively promoted pro-poor agriculture growth, created agriculture-based added value that is equitably shared with poor farmers, promoted viable employment opportunities for the poor, and improved the performance of local food and regional food markets. This study should also assess the effectiveness of value chains in an integrated approach to achieve the overarching goals of poverty reduction and improved nutritional status, as well as how the value chain approach should be modified to make greater contributions to poverty reduction for all poor people. 2. Using the information and findings of the studies recommended above, update the Feed the Future guidance in terms of rural poverty typologies and the types of interventions most appropriate to supporting a transition of marginalized households to increase their asset base and ability to participate in a market economy. Encourage the uptake of research findings both in Washington and the field. Be open to the potential need to 1) acknowledge that value chains, in and of themselves, may not be an appropriate means of addressing certain poor populations, and 2) adjust guidance to ensure it is understood that more flexibility in alternative approaches to address these needs is allowed, and even encouraged. Using the information from the studies and analyses above, identify existing alternative approaches and provide a guide that missions may use to develop poverty reduction programming beyond value chains. As part of this process, define the objectives, roles, and forms – and limitations – of value chain work based on an integrated and complex problem assessment, and develop explicit complementarities with other poverty-focused and/or nutrition-focused interventions. 3. Advocate that missions utilize the SPRING studies, frameworks, and guidance created with Feed the Future resources for problem assessment and program designs in order to strengthen nutrition outcomes for value chain programs. Support workshops in-country to more fully illuminate and contextualize the guidance. Support small-scale operations research on the theme of agriculture and value chain development activities and how these affect nutrition outcomes, with an aim to establish a stronger evidence base from which guidance and technical assistance can be offered to the Feed the Future community on how nutrition should best be integrated into value chain activities, or how it should occur in parallel, complementary, “layered” approaches. 4. Encourage missions and Feed the Future value chain IPs to explore “ground-up” opportunities that will add value and create jobs within small communities and other town and urban centers within the country. This may include niche markets that serve fewer farmers and working at community level to provide producers with an outlet for their smaller-scale production, in contrast to value chain approaches which focus on a larger scale of production and added value. 71 Where possible, value chain activities should emphasize designs that add value through generating employment opportunities for the rural poor. Based on a problem assessment focused on reducing poverty, and a context-specific opportunities analysis, place greater emphasis on non-agriculture job creation for people living in rural areas and small agrarian town centers. 5. Feed the Future market-oriented interventions focused on the poor should place more emphasis on building farmer and farmer groups’ business acumen in addition to increasing marketable surplus and facilitating links to new markets. This builds sustainability of market participation and reduces economic backsliding. Question 3b: How well has the initiative focused implementation and concentrated resources in appropriate (in terms of size and agro-ecology) geographic zones? Feed the Future focus countries have invested in a diverse portfolio of activities and achieved results by concentrating resources in their respective ZOIs and have learned and documented many lessons in the process. Concentrating development activities in strategic geographic areas likely increased opportunities to collaborate among implementers. Interviewees also noted that greater flexibility to include complementary activities around value chain investments, in the parameters that guide strategy designs and implementation, is required. QUESTION 3b FINDINGS Feed the Future has defined the geographic zones in which implementation and resources are concentrated as Zones of Influence.136 The ZOI was intended to be defined using administrative boundaries and be of appropriate scale so that the global Feed the Future impact level targets could be realistically achieved within the ZOI population given underlying factors.137 The concept and importance of ZOI size and agro-ecology has evolved over the course of the initiative. The 2010 Feed the Future Guide describes the initiative as working at a national level. The concept of the ZOI does not appear in the 2010 Feed the Future Guide; rather, it appears to have been developed between the initial conceptualization of Feed the Future and the approval of the Feed 136 Volume 8: Population-Based Survey Instrument for Feed the Future Zone of Influence Indicators with Revised WEAI Module, October 2012. (pg. 4) Retrieved 21 June 2016, https://www.feedthefuture.gov/sites/default/files/resource/files/ftf_vol8_populationbasedsurveyinstrument _oct2012.pdf. 137 Ibid. 72 the Future MYSs at the country level.138 Key informants from BFS suggest that it was a response meant to ensure selectivity and focus of resources due to resource limitations and policy recommendations coming out of the Presidential Policy Directive on Global Development. The term “agro-ecology” does not appear in the Feed the Future Guide, but associated concepts are discussed in several places in the document, including on page 10: “Enhance sustainability and resilience of production through a large-scale systems approach to environmental and natural resource management…We will support integrated resource management approaches, which are the best method to balance demands for resources for agriculture, people, and ecosystems.” Selection Criteria Used to Define ZOI In Feed the Future focus countries that included a sub-national geographic area of focus in their MYSs, frequently cited reasons for selection include: 1) anticipated resource levels; 2) baseline levels of food insecurity, poverty, and stunting; and 3) agricultural potential. Other criteria have also been recorded, but they are context-specific. Examples include host country government priorities (Ghana, Tanzania, and Bangladesh), ethnic diversity (Kenya), and opportunities to leverage work of other USG Agencies and donors (Mozambique, Guatemala, and Ghana). All 19 focus country strategies work within geographic areas with target populations and on specific value chains that are consistent with and guided by host country governments’ stated priorities. Field interviews confirm very strong alignment of ZOIs with host country government priorities. In several cases, governments had recently shifted focus to geographic areas and populations that had been underrepresented or marginalized by previous development efforts. In these cases, Feed the Future relocated and refocused USG development activities to ensure alignment with the host country’s priorities (Bangladesh, Ghana, Guatemala, and Honduras).139 Size of the ZOI The sizes of the ZOIs differ per focus country and have evolved over time. Six of the 19 focus countries (Kenya, Malawi, Mali, Nepal, Tanzania, and Uganda) have modified their ZOI from its original location and/or size. These modifications were triggered by portfolio reviews that incorporated new learning from evaluations or other data sources, and made adjustments to size and/or target population as a result of lessons learned. Malawi and Uganda received approval for expanding the ZOI to create new resilience areas.140 Key informants in Bangladesh felt that the ZOI 138 The evaluation team suspects that this shift was conveyed to missions through official USG cables, which were not made available for review. 139 MYS Strategy Change Memos and portfolio reviews. 140 USAID/Malawi Strategy Change Memorandum, October 1, 2015: USAID/Uganda Strategy Change Memorandum, November 17, 2015. 73 was too large, diluting resources and limiting potential impact. Given the low population coverage levels described in Question #3c., this sentiment likely resonates across focus countries. Agro-ecology of the ZOI Perceptions about agro-ecological compatibility was split among those interviewed as a part of this evaluation. Respondents in Ghana expressed concern that the agro-ecology of the ZOI was incompatible with the value chains selected. Key informants from Uganda noted that the selected ZOI was agro-ecologically appropriate for the selected value chains, while in Bangladesh the ZOI was possibly the only area of the country that possessed untapped potential for a strategy hinged on agriculturally driven growth via increased production through intensification. Concentration of Resources in the ZOI Of the USG actors included in Feed the Future, USAID Mission Feed the Future teams are the most insistent that implementers program inside of the ZOI. USAID/BFS emphasizes the need to program in the ZOI, but IPs and partners who received direct funding from USAID/Washington rather than missions report the flexibility to work outside a ZOI if BFS agrees that it is justifiable. This is much less common with USAID mission activities. In the remaining Agencies and USAID bureaus, there is little fidelity to the ZOI. Some program a portion of their resources within the ZOI, while others do not program any resources specifically within the ZOI. None of the other bureaus or Agencies limit their target population to the ZOI population. A common theme emerged from interviews in Guatemala, Malawi, and Bangladesh – that limiting work to one geographic area made it more difficult to adequately address the dual goals of Feed the Future. In these countries, the places with the highest potential to reduce poverty through agricultural value chains or improve nutrition do not always coincide with one another. If you look at the salinity and the erosion and the disaster issues, there are plenty of climate change, natural disaster issues that the south faces more than most other parts of the country. I think no one would really argue with that. But that’s not the same as women’s empowerment or poverty or health and nutrition. And so we start to look at the broader picture…It’s not that the south doesn’t need it. But since Feed the Future’s supposed to be much broader than that and even nutrition is one of the major outcomes, then it seems like to me like the south is the wrong place. – Bangladesh IP Flexibility in programming outside the ZOI varied by country depending on the interpretation of Feed the Future guidance by Country Teams. Among implementers interviewed by the evaluation team, approximately one-third stated that some of their work needed to occur partially or fully 74 outside of the ZOI in order to be successful. Mission personnel and implementers noted that programming in ways that would benefit the population within the ZOI was not equivalent to programming all activities within the ZOI. A good number of our engagements really aren't necessarily [within the] Zone of Influence. We want to make sure that the impact of this has a beneficial impact on those farms [that] are in our Zone of Influence, so it's our motivation. That's where this idea of a Zone of Influence didn't help us out very much. What we needed to look at was the seed sector, as a whole. People can criticize that or not criticize it, but you can't take a geographical subset of a nationwide system. That's some of the nuance that we've all had to deal with. – USAID/Ghana QUESTION 3b CONCLUSIONS 1. After five years of implementation, Feed the Future focus countries have invested in a diverse portfolio of activities in their respective ZOIs and have learned and documented many lessons in the process. Feed the Future has achieved results by concentrating its resources and making investments in the ZOI of any given focus country over the last five years. 2. The size of the ZOIs is not perceived to be a problem by key informants in most cases, although there were a few exceptions to this (such as in Bangladesh), even though the proportions of households reached in ZOIs remains low. 3. Concentrating development activities in strategic geographic areas likely increased the opportunities to collaborate among implementers, and several interviewees stressed this as a positive outcome. This is discussed in greater detail in Question #6. However, they also noted that greater flexibility in the parameters that guide the strategy designs and implementation is imperative. Specifically, flexibility to include complementary activities around value chain investments emerged retrospectively as critical. 4. While the logic presented by other USAID bureaus and other USG Agencies for targeting outside of the ZOI seems sound in many cases, this dilution of programmatic resources should be recognized, along with the trade-offs that arise from it. Spending outside the ZOI can lead to greater economic benefits within the ZOI; for example, investment in agro-processing facilities that source ZOI farmers’ product. With a value chain approach, the analysis of bottlenecks within the value chain determines the geographic location where Feed the Future should invest to increase favorable outcomes and impacts, and these bottlenecks are not linked to population coverage. For nutrition, policy level interventions outside of the ZOI can benefit the ZOI population, but community-based interventions outside the ZOI are far less likely to impact nutritional status 75 among the ZOI population. This may be an acceptable trade-off, however, if those resources can have a greater impact on nutrition in geographic areas outside of the ZOI. QUESTION 3b RECOMMENDATIONS 1. Feed the Future should develop an approach to capture the impact occurring through Feed the Future investments outside of the ZOI. Feed the Future will be able to convey a more accurate and robust story about its impact if it incorporates ways to capture the results of these investments in its reporting. By focusing only on the ZOI, Feed the Future is capturing only a portion of the effect of the initiative, and is missing an opportunity to tell a much larger story about national level, systemic support and change through a whole of government approach. 2. Allow for greater flexibility in strategy and project designs to allow implementers to address value chains holistically, no matter where the need lies geographically. 3. Develop regular (annual) focused and achievable WOG action plans at the country level directed at resolving key constraints in agricultural development for that country that require a multi-faceted and coordinated approach among WOG actors. These need not be formal or overly complex. A similar process could also be considered at the Washington level. Strengthen existing mechanisms that identify synergies and reinforce collaboration. These may include the sharing of planning briefs and work plans, conducting joint gap analyses, etc. In addition, provide skilled, professional technical facilitation support services to Missions to ensure that information-sharing, joint planning, and other coordination processes become more collaborative, and identify synergistic complementary joint actions. (Further discussion of this approach can be found under Evaluation Question #4.) Question 3c: Are the proportions of households being reached in each ZOI sufficient to plausibly achieve the targeted impacts? While the actual proportion of households reached with Feed the Future interventions is not documented, available data from the portfolio reviews suggest that the proportion is low. The target-setting guidance for poverty reduction did not include any calculation of coverage requirements in order to achieve the target reduction in the ZOI. The low proportion of households reached within the ZOI make it difficult to understand the degree to which ZOI-wide changes in impact-level indicators for poverty and nutrition were driven by Feed the Future interventions or broader long-term trends. 76 The effort to standardize target-setting at the impact level resulted in targets of highly variable feasibility at the country level. There is no link between the Washington-driven targets set at the intermediate result and goal level, and the beneficiary targets set at the country level through the activity design and implementation processes. Two distinct, disconnected target-setting processes took place. Emerging evidence suggests that the complete causal relationship between agriculture, poverty, and nutrition may not be adequately captured in the current Feed the Future RF and corresponding indicators. Gaps in the Feed the Future RF and causal pathways at the sub-IR level are being filled by focus country programming. The absence of causal pathways to support IRs related to employment, resilience of vulnerable communities and households, and nutrition, represent significant gaps in theory development for Feed the Future to sustainably address the prevalence of poverty and stunting in children under five. Evidence collected by Feed the Future provides a great deal of new information and learning about the complex relationships between agriculture, poverty, and nutrition, and highlights that duration of support, intensity of support, and time elapsed since support began are equally important to producing results. QUESTION 3c FINDINGS The Feed the Future Zone of Influence is defined as the geographic area in a country where Feed the Future implements activities and is described in detail in Question #3b above.141 The ZOIs selected by missions are sub-national.142 To quantify PBS indicators for the ZOI, Feed the Future M&E Guidance143 gave missions the option to use secondary data, like the Demographic and Health Survey (DHS) and the Living Standards Measurement Survey (LSMS), if appropriate, and if those data met certain criteria,144 or to conduct their own Feed the Future-specific population￾based surveys. These PBSs drew heavily from the LSMS and DHS modules.145 141 “Volume 8: Population-Based Survey Instrument for Feed the Future Zone of Influence Indicators with Revised WEAI Module.” October 2012. (pg. 4) Retrieved 21 June 2016, from https://www.feedthefuture.gov/sites/default/files/resource/files/ftf_vol8_populationbasedsurveyinstrument _oct2012.pdf. 142 2016 Feed the Future portfolio reviews. 143 “Volume 11: Guidance On the First Interim Assessment of the Feed the Future Zone of Influence Population Level Indicators, Revised October 2014.” Retrieved 4 July 2016 from https://www.feedthefuture.gov/sites/default/files/resource/files/ftf_guidanceseries_vol11_interimassessmen t_oct2014.pdf. 144 Ibid, p. 12 145 “Volume 8: Population-Based Survey Instrument for Feed the Future Zone of Influence Indicators with Revised WEAI Module, October 2012.” 77 A review of MYSs suggests that illustrative targets were established for each Feed the Future country during the MYS development and approval process, but were later revised as baseline data were collected. USAID issued target-setting guidance in 2012,146 and revised guidance in 2013,147 which attempted to refine and standardize target-setting across the Feed the Future countries. This target-setting guidance was led by USAID/BFS. To inform this target-setting, USAID documented the following assumptions:148 “From data received from the field and other sources, we calculate globally that there are currently some 151 million people living in Feed the Future geographic zones. On average, from this data we calculate that 45% of those people, or 66 million individuals, live below the international poverty line. Within the Feed the Future zones, there are some 23 million children under five in the Feed the Future target zones, of which approximately 5 million children are underweight and 9 million children stunted. Our strategy and development hypothesis aims to improve the livelihoods of as many of these people economically and nutritionally in the Feed the Future geographic zones as is possible.”149 Feed the Future impact targets were built from the top-down based on Washington guidance and rooted in mathematical modeling. Feed the Future guidance clearly articulates the rationale and methodology used to establish recommended poverty and stunting impact targets at the country level.150 USAID mission personnel expressed discontent with the impact-level target-setting process, commenting that they seemed overly ambitious, “fictitious,” “estimates of estimates,” and “politically driven.”151 The 2013 guidance provides Washington-derived targeting recommendations for some outcome indicators as well, with explanations as to why targets should be set at certain levels.152 For example, the guidance for the women’s anemia indicator advised, “Target a 30% reduction in baseline 146 “Volume 9: Target Setting for Reduction in Prevalence of Poverty, Underweight and Stunting in Feed the Future Zones of Influence.” March 2012. 147 Ibid. 148 Ibid. 149 Ibid. 150 Ibid. See also: Target Setting Tool. 2014. Retrieved 4 July 2016 from https://agrilinks.org/events/overview￾ftf-target-setting-prevalence-poverty-missions-zone-influence 151Feed the Future Focus Country Survey, Key Informant Interviews. 152“Feed the Future Guidance for Setting Targets for Zone of Influence Population-Based Indicators and for Percent Growth in Agricultural GDP indicator: Per Capita Expenditures, Exclusive Breastfeeding/Minimum Adequate Diet and Anemia guidance revised.” April 2013. USAID. 78 anemia prevalence by 2017. This will support achievement of the World Health Assembly target of 50% reduction in woman and child anemia by 2025. National-level reduction in these ranges have been achieved over a five- to six-year period in several Feed the Future focus countries (Ethiopia, Malawi, and Uganda.)”153 The 2012 Feed the Future target-setting guidance includes scenario-building for nutrition indicators that estimate the prevalence and number of stunted children in the ZOIs in order to achieve the stunting target of 20% globally. A similar calculation of number or percent of people or households that must be reached to achieve the poverty goal was not provided154 given that this kind of data does not exist for agricultural programming. The mathematical models used to provide target recommendations for the poverty goal were based on the projected change needed in GDP in order to reduce national poverty rates by the target amount.155 For 12 out of the 19 countries, Feed the Future sought to accelerate the annual rate of change in poverty reduction. For the other seven countries, the rate of change paralleled existing trends, because the trends from the latest two to three data points were extremely high. The USAID Chief Economist indicated it was not advisable or sustainable to set a higher rate of change or set five-year targets for more than 30% according to USAID’s Chief Economist. Below the global level, the evaluation team did not find any evidence of a similar type of modeling exercise for nutrition or poverty indicators to estimate the number or percentage of households that must be reached to achieve the respective impact goals. Some focus country MYSs describe the demographics of their target geographic areas, and provide illustrative coverage estimates, but these numbers were offered both illustratively (to be refined after the baseline data were available) and estimated through a process that was Washington-based, described above, with missions setting the targets. In most cases the dates of MYS approvals confirm that the two processes took place in different time periods, the country-level process having preceded the global process, with no relationship between the two. Finally, the evaluation team was not able to identify any process undertaken at the country or global level by which Feed the Future compared the actual reported number of beneficiaries reached in a Feed the Future focus country to an estimation of coverage required to plausibly meet the impact goals. 153 Ibid. 154 “Feed the Future Guidance, Volume 9.1: Target Setting for Reducing the Prevalence of Poverty.” January 2013. 155 “Feed the Future Guidance, Volume 9: Target Setting for Reduction in Prevalence of Poverty, Underweight and Stunting in Feed the Future Zones of Influence.” March 2012. P. 1. 79 USAID mission personnel156 report that the proportion of households reached with interventions is fluid from year to year as programs are adapted to address local needs, making it difficult to calculate or interpret aggregate proportions of households reached over the time period of the initiative. In the absence of this aggregate level information, focus countries report coverage as a simple calculation of number of farmers reached in a fiscal year divided by total number of farmers in the ZOI.157 Across Feed the Future programs, value chain interventions had highly variable coverage of the ZOI population, ranging from 8% of eligible farmers in Malawi and Ghana to 67% in Ethiopia.158 Coverage estimates did not disaggregate between poor and non-poor smallholders, or the extreme poor. Some households who live in Feed the Future communities, but who are not participating in Feed the Future activities, are able to gain knowledge, and sometimes access to new technologies, through a natural diffusion of information or improved varieties among neighbors (such as through sharing improved agricultural practices, or gaining access to OFSP vines and seed), but there is no mechanism in the FTFMS to capture indirect beneficiaries of Feed the Future interventions through diffusion of innovation or other organic means. Feed the Future recognizes this important requirement of their M&E system and is working to address this issue. Households within the ZOI could also be targeted with interventions other than Feed the Future value chain activities. In countries where Food for Peace programming was included in the ZOI, FFP programming complements the value chain programming by targeting vulnerable groups who may not qualify as value chain actors.159 Global health and nutrition programming specifically targets pregnant and lactating women (PLW) and children under five years of age, also complementing and expanding overall population level coverage beyond only value chain actors.160 Feed the Future places a heavy emphasis throughout guidance and strategy documents on the importance of coordination with other donors and actors in the ZOI to maximize results.161 However, Feed the Future focus countries did not systematically report on proportion of households reached by all Feed the Future programming, or those reached through efforts to coordinate other donors. 156 Feed the Future Focus Country Survey, 2016. 157 2016 Feed the Future portfolio reviews. 158 Ibid. 159 International Food Assistance: Improved Targeting Would Help Enable USAID to Reach Vulnerable Groups. General Accounting Office. 2012. Retrieved on 4 July 2016 from http://www.gao.gov/assets/650/648736.pdf 160https://www.usaid.gov/what-we-do/global-health/nutrition/role-nutrition-ending-preventable-child￾maternal-deaths 161The Feed the Future Guide: A comprehensive Approach: Partnerships and Coordination, 2010, p. 4; Feed the Future Country Multi-Year Strategy documents. 80 Similar shortcomings exist in understanding the intensity of support received by households. The initiative has not collected aggregated data to understand what proportion of households in the ZOI participated in multiple interventions (such as households receiving both agriculture and nutrition interventions). Some individual activities report this type of coverage (e.g., Masfrijol and Anacafe/Funcafe in Guatemala,162 RING in Ghana, most of the FFP programs), but it is not systematic enough to be able to understand coverage at an initiative level. Such data collection is difficult and expensive to conduct and may be impossible at the initiative level given resource constraints. However, it may be useful to collect these data in some projects to understand intensity of support at some level. For agricultural production and value chain activities, time is an equally, if not more relevant, factor contributing to poverty reduction as compared to simple coverage.163 Agricultural transformation of new technologies and practices takes considerably longer than the standard five-year project cycle.164 Feed the Future application indicators measure the extent to which farmers are willing to test new technologies and practices, as compared to sustainable adoption of them. It takes time for agriculture value chain actors to build relations and establish trust.165 The determination and implementation of regional quality standards for traded agricultural commodities, such as maize in East Africa, and consistent implementation of liberalized trade agreements that call for an elimination of trade bans, have been ongoing for several decades.166 QUESTION 3c CONCLUSIONS 1. While the actual proportion of households within ZOIs reached directly with Feed the Future interventions is not documented, available data suggest that the proportion is low in some countries. The target-setting guidance for poverty reduction did not include any calculation of coverage requirements in order to achieve the target reduction in the ZOI. 2. While the logic presented by other USAID bureaus and Agencies for targeting outside of the ZOI seems sound in many cases, this dilution of programmatic resources reduces Feed the Future’s ability to maximize coverage within the ZOI. The importance of less than maximum coverage, however, depends on the theory of change underlying the initiative, as described above. 162 KIIs with Anacafe and Funcafe in Guatemala and with Masfrijol in Guatemala and the U.S. 163Cousin, E., da Silva, J. G., & Nwanze, K. F. Principles and practice for resilience, food security, and nutrition. Huffington Post. 2013. 164USAID. Program for Research on Sustainable Intensification Strategy Overview and Vision Document, March 2016. 165 KII with COP of Ghana’s Agriculture Technology Transfer Project. 20 April 2016. 166www.fews.net, www.wfp.org and www.fao.org. 81 Programming outside the zone can lead to greater economic benefits within the ZOI, for example, through investment in agro-processing facilities that source ZOI farmers’ product. With a value chain approach, the analysis of bottlenecks within the value chain determines the geographic location where Feed the Future should invest to increase favorable outcomes and impacts, and these bottlenecks are not linked to population coverage within a ZOI. 3. The effort to standardize target-setting at the impact level resulted in targets of highly variable feasibility at the country level. The results of the interim PBSs for the 10 countries available at the time of this evaluation suggest significant variability between countries towards meeting poverty targets. This indicates that local context introduces variability in expected impact that is not captured through a standardized target-setting approach. 4. Target-setting for activity and goal levels was conducted by Washington-based teams with input from country teams. Washington-based teams developed a methodology or guidance for setting the targets for the higher levels of the Results Framework, and country teams developed the targets for goal indicators at the country level. These targets were developed with country team input for the country, the initiative-wide targets for goal indicators were based on a weighted average of country level targets. 5. Emerging evidence coming out of Feed the Future programming, evaluations, and surveys suggest that the causal pathways between agriculture, poverty, and nutrition may not be adequately captured in the current Feed the Future RF and corresponding indicators.167 Evidence collected by Feed the Future provides a great deal of new information and learning about the complex relationships between agriculture, poverty, and nutrition, as well as higher level value chain and policy interventions on households and population-based poverty and malnutrition rates. Duration of support, intensity of support, and time elapsed since support began are equally important to producing results that contribute together to achieve Feed the Future impacts. QUESTION 3c RECOMMENDATIONS 1. Targets should be built from the ground up in order for them to accurately reflect Feed the Future programming. To ensure high quality target-setting throughout the initiative, at all levels, Mission personnel should be better trained on target-setting and should be given greater autonomy with regards to target-setting. 167 These findings are discussed elsewhere in the document, under Question #2. 82 2. Once bottom-up target-setting is in place, greater scrutiny should be placed on indicator reporting that varies substantially from targets to assist any given project, if targets are not being met, to get back on track. By identifying and correcting these issues at the project level, managers can keep the program on track to reach targets at the initiative level. 3. The theory of change suggested by the Feed the Future RF should be revisited to incorporate learning about causal pathways between agriculture, poverty, and nutrition over the past five years. Feed the Future should refocus and better strategically integrate system-level and household or individual level interventions such that their combined efforts reinforce the poverty and nutrition reduction outcomes. Ensure that a single theory of change prevails from the bottom to the top of the Results Framework. The pro-poor development literature cited under Question #3a, the FFP strategy, the Resilience Guidance, and SPRING’s lessons learned documents all provide relevant guidance. 4. Find a way to capture the impact occurring through investments outside of the ZOI. With Feed the Future resources being channeled outside of the ZOI, Feed the Future will be able to convey a more accurate and robust story about its impact if it incorporates ways to capture the results of these investments in its reporting. By focusing on the ZOI, Feed the Future is missing an opportunity to tell a much larger story about national level, systemic support and change through a whole-of-government approach. As noted by many interviewees, some of the most significant results occurring in systems changes are not captured because there are no tools and metrics to do so. Question 4: How and to what extent have Feed the Future interventions, both Mission- and centrally-managed, helped build human and institutional capacities for the agricultural and nutrition/health sectors? There are numerous achievements, but evidence of success is mixed. Notable success has occurred in the policy development and reform space within the public sector at multiple levels, within the agricultural research and higher education sectors, within agricultural extension systems, the private sector (including countless farmers), within civil society, and within the broader community of Feed the Future (including USAID and IPs). The scope and breadth of activities being implemented and of actors being supported with capacity development suggests significant success within both the human and institutional dimensions; significant achievements are evident with the wider systems and policy enabling environments as well. Current approaches to M&E of capacity development at Feed the Future are focused on outputs and proxy measures that do not adequately capture changes in capacity or performance. This 83 makes assessment of success – i.e., tracking actual changes in individual and organizational performance – difficult. QUESTION 4 FINDINGS Guidance and Standards USAID has developed an approach to capacity development that is codified in a number of documents. The first is the “Human and Institutional Capacity Development Policy Paper: A Mandatory Reference for ADS Chapter 201,” dated February 15, 2009. Another is USAID’s “Human and Institutional Capacity Development Handbook,” dated October 2010. Additional guidance has been produced in the interim, including “Local Capacity Development: Suggested Approaches. An Additional Help Document for ADS 201” (New Edition Date: July 22, 2013). A number of other documents – as well as current discourse within USAID – also reveal an evolution in USAID thinking. These include the USAID Local Systems Framework (April 2014),168 the USAID approach to Collaborating, Learning, and Adapting,169 and very current thinking on what is termed “Capacity Development 2.0.”170 Capacity Development 2.0 can be summarized as an approach that emphasizes optimizing both internal and external social capital, integrates key understandings of systems theory and applies these to organizational development, and emphasizes impacts rather than tracking activities or outputs. Discussions with key personnel in USAID/BFS reveal that they are aware of and knowledgeable about this evolution in USAID’s current thinking. The approach to HICD contained in the three USAID guidance documents mentioned above explicitly recognizes, in alignment with established best practice, that HICD is not only about training; furthermore, it uses a systems model that includes the recognition that assessment should be participatory by engaging stakeholders (“clients and communities”), and should include analysis of “mission, goals, strategies, and culture.” (HICD Handbook, p. 8) There is evidence within Feed the Future, demonstrated through written guidance, briefing content, and key informant interviews, that it is understood that larger systems need to be addressed. For example, the Feed the Future Guide states: “Investing in strengthening partner country capacity to engage in…robust stakeholder consultation is a key component of our approach.” (p. 4) The numerous policy reform projects occurring under Feed the Future that are focused on addressing constraints in enabling policy environments, as well as thinking described in numerous interviews 168 USAID. “Local Systems: A Framework for Supporting Sustained Development.” April 2014. 169 https://usaidlearninglab.org/faq/collaborating-learning-and-adapting-cla 170 Root Change. “Capacity Development 2.0.” Accessed via: http://www.rootchange.org/about_us/resources/capacity_2.pdf 84 in USAID missions (including with a number of personnel involved in value chain work), also demonstrates this awareness. Content in briefing presentations used by USAID to describe the Feed the Future HICD approach further reinforces this observation,171 describing the need to understand the “enabling environment” outside the organization as well as the “external environment.” The Feed the Future Civil Society Action Plan also discusses this systems awareness – and to a degree describes building USAID capacity to execute this plan, explicitly identifying the need to develop specific guidance documents.172 There are three different definitions of capacity development provided in the USAID guidance mentioned above, which to some degree represent the process of evolution in USAID’s thinking on the subject. “Local Capacity Development: Suggested Approaches,” the most recent of these, provides a definition of capacity development that is explicitly taken from the Organisation for Economic Cooperation and Development (OECD): “Capacity” is the ability of people, organizations, and society as a whole to manage their affairs successfully. “Capacity development” is understood as the process whereby people, organizations, and society as a whole unleash, strengthen, create, adapt, and maintain capacity over time.” (p. 3) Another OECD document that supports further development of the practice area notes: “These definitions remain quite general and call for further precision in order to be operationally useful.”173 The definition in use by BFS (extracted from the HICD/ Board for International Food and Agricultural Development (BIFAD) 2015 Portfolio Review) aligns to a large degree with the OECD definition. They define capacity as: “[a] process in which people and organizations improve their performance and unleash, strengthen, create, adapt, and maintain high performance over time. Capacity development must be locally led. Partners identify performance problems, analyze causes, set goals, and develop plans (problem-driven iterative adaptation). Outsiders provide resources, connections, and expertise.” When asked if there is standard and explicit Feed the Future guidance for HICD, one key informant in BFS said there is none. This is noteworthy because capacity development is explicitly identified as a key area for strategic focus throughout the 2010 Feed the Future Guide, with numerous areas of intervention identified. Feed the Future has also produced some guidance on capacity development, but this is codified in the “M&E Guidance Series.” Volume 1 explicitly identifies “Local Capacity-Building Investments for Data Collection and Use” as a key purpose (pp. 4-5). Additionally, “Volume 5: USAID Forward/IPR and Feed the Future,” states that “Local Capacity Development is a 171 See, for example, “2011 Global Education Workshop: From Evidence to Action.” PowerPoint presented at the Global Education Workshop: From Evidence to Action. Arlington, VA; August 22-25, 2011. 172 Feed the Future. “Civil Society Action Plan.” May 2014. 173 OECD. “Perspectives Note – The Enabling Environment for Capacity Development.” January 2011. p .2 85 cross-cutting issue, in that it should be a part of USAID’s efforts across the board, not just in certain sectors or in a handful of projects. Feed the Future is proposing to achieve USAID Forward’s goals of local capacity building by working directly with host government ministries, institutions and agencies, private firms, and civil society organizations (CSOs) … Local capacity building has two main purposes: 1) To build local ministries and public institutions to effectively serve the needs within the agriculture and nutrition sectors, including implementing USAID projects; and 2) To leave behind viable local public entities, private firms, and CSOs to continue servicing the needs within the country’s agriculture and nutrition sectors, including implementing USAID projects.” There is a lack of specific and detailed Feed the Future guidance on how to approach capacity development or operationalize capacity development activities. How have Feed the Future interventions helped build human and institutional capacities? Feed the Future has developed and is implementing a wide spread of capacity development initiatives, and all available evidence suggests it is indeed being integrated thoroughly as a cross￾cutting activity. HICD activities are being operationalized at multiple levels within the program, and this includes activities designed and managed by Washington as well as activities being managed out of – or in operation at – regional and country-level missions. These diverse activities take the form of activities specifically and primarily conceived of as capacity development activities, or occur as “cross-cutting” capacity development sub-activities occurring within an activity primarily designed to achieve something else. According to a strategic assessment of the HICD portfolio conducted for Feed the Future, there are “231 projects to date…in 19 countries, involving…109 partner organizations in those countries and scores of U.S. and CGIAR consortia members.”174 The spread of these is extensive – from explicit projects focused on capacity development in certain broad technical areas – for example, agricultural research to providing a spread of activities within specific partners, such as key counterpart ministries to projects that are focused in other technical areas (such as private sector engagement and value chain focused activities) – that have specific capacity development components. Some element of capacity development was discussed in almost all interviews and documents, indicating that most, if not all, Feed the Future activities – if viewed through a broad capacity development lens – are performing capacity development to some degree. Additionally, given the explicit emphasis on “building” M&E capacity in Feed the Future guidance – along with a principle of active learning-from-experience and ongoing adaptation, as well as extensive research activities focused on strengthening the evidence base to support the work of Feed the Future – it is clear that capacity is not just being developed externally, but also within Feed the Future broadly, to include USAID and implementing partners. 174 Thomas Dichter, David Joslyn, Joyce L. Moock, and M. Diane Bellis. “Feed the Future Human and Institutional Capacity Development Strategy Review.” USAID. May 8, 2015. p. 4. 86 A useful framework for categorizing and describing this very diverse spread of capacity development activities emerges from a model referred to as an “Agricultural Innovation System.” The World Bank, the OECD, and others have produced references to this model, notably the World Bank’s voluminous “Agricultural Innovation Systems: An Investment Sourcebook,” (2012). This model presents various systems actors or sectoral nodes where capacity development may be targeted. For the purposes of this report, these nodes, synthesized from an OECD monograph on the subject,175 are: Public Sector, Agricultural Research, Higher Education, Agricultural Extension, Private Sector (including farmers), and Civil Society. It should be noted the model as presented by the World Bank identifies 13 distinct actors,176 but these can be generally distilled down to the six nodes mentioned above. The following examples emerged in the course of research, from interviews as well as document reviews, and are provided to illustrate a general spread of how Feed the Future is working to develop capacity. These are not the only examples of interventions that have been identified under each node and should be viewed only as an illustrative spread of activities that are typically included within Feed the Future capacity development. 1. Public Sector Interviews revealed that there are a variety of models used to provide capacity development to governments. While IPs frequently identified activity co-implementation as an approach to capacity development (mostly by engaging government extension workers in agricultural and nutrition extension activities), USAID missions tended to emphasize supporting alignment with and development of national policies and other priorities. IPs support co-implementation at various levels within focus countries, from parliament and key ministries, to local government bodies, to extension workers at field level. IPs also co-implement with national and local NGOs. The variety of approaches used in both models includes consultation and technical assistance, direct engagement and awareness building, and trainings. Field interviews showed that the most common form of institutional strengthening reported was technical assistance (TA). This however, took multiple forms, ranging from the provision of expertise to streamlining processes to supporting activities that establish or improve capacities. “For me, if you can support me and I deliver an activity, I don't have to get the cash. What I want is the activity to be done. The activity becomes much easier to be implemented [with 175 OECD. “Agricultural Innovation Systems: A Framework for Analysing the Role of the Government.” 26 March, 2013. p. 16-19 176 World Bank. “Agricultural Innovation Systems: An Investment Sourcebook,” 2012. p. 4. 87 TA]. Government processing funds, ours is a bit rigorous. It takes long, so if I have to hire a consultant, it can even take me two, three months. With this type of collaboration where [the IPs] can just hire and give me someone, it takes weeks, so I find it very quick. We agree on an activity, we agree what the government [will do].” – Uganda IP Similarly, through the whole-of-government approach, USDA provides TA support to the Haiti Agriculture Ministry, TA in agricultural statistics and analysis in Rwanda, and collaboration on agriculture statistics with the National Bureau of Statistics in Tanzania; and the US Forest Service provides TA on natural resources management and monitoring in Ethiopia. Although TA can help to fill vital institutional capacity gaps with just-in-time support, it is not, on its own, well-suited to facilitating sustainable capacity development at the recipient institution. In the case of the IPs interviewed for this evaluation, all but two IPs delivered TA in combination with other direct engagement, awareness building, or training activities, as illustrated by the examples below. “So [SANE], as they come [to do their needs assessment], they will be able to see if – really to analyze and find out what has been the challenges so that the DAES (District Agricultural Extension System) has and then they also have briefing sessions to make sure that all the stakeholders – because the DAES requires that they take [into account] all the stakeholders… and all programs are using that system...So that's all examples of what [SANE] is doing… we have great expectations that I think the DAES system will be[come] operational. So in the districts where SIMLESA is working, lessons which will be drawn from such a district can be extended to the other districts. So that is our expectation.” – Malawi IP “Up to now…the government [extension] monitoring system is analog, a pen and paper system. We are going to introduce an Excel-based database management system, so that the upazila-level office and district- level office will benefit through our initiative. We are going to introduce these things in 26 working upazila. After one year, we'll evaluate whether the database management system is really working. Then, we'll give the government this result, so that they can replicate it. Second, agriculture development extension staff were [CSI] not very well trained. We provided the training…we provided for them on more on a circular level, so that through this logistic support, through the capacity-building, we make sure that they're affording better service. [Our work with the] Department of Agricultural Extension, and at the upazila level….the digital monitoring system. That will help the whole country if we are successful.” 88 – Bangladesh IP Produced in 2013, the “Feed the Future Guide to Supporting Sound Policy Enabling Environments” called for interagency policy teams at post to describe planned actions to strengthen three essential components of an effective and sustainable policy system, of which two are of particular relevance from a capacity development perspective: institutional architecture and mutual accountability mechanisms.177 Mutual accountability mechanisms include the development and use of M&E systems to track performance and the formation of joint review committees to review performance. Much of the discussion under Question #11, which clearly identifies significant success and accomplishments of Feed the Future, is also – either explicitly or implicitly – relevant from a capacity development perspective. For example, to include institutional architecture in the country-specific policy reform matrices is to recognize that this is a pre-condition for effective policy, and establishes principles of capacity development directed at both public and private sector institutions. These principles include supporting or developing institutional systems and capabilities for collecting objective data, conducting evidence-based policy analysis, and conducting stakeholder consultations. The Feed the Future Knowledge-Driven Agricultural Development Project (KDAD) developed a reporting tool that tracks seven factors that promote effective policy implementation, including stakeholder participation, advocacy coalitions, and developing a strong evidence base for policy impact,178 all areas where capacity development is occurring. Among the IPs interviewed in the five fieldwork countries, close to a third (n=19) of them noted that they had incorporated some type of policy work into their activities, mostly at the national level. Although more than half of these IPs (n=12) reported that they provided TA to guide and shape policy planning and passage, policy-focused IPs (n=9) also placed a strong emphasis on supporting national-level implementation and strengthening of existing policies. Whether IPs provided TA prospectively (during policy development) or retrospectively (during policy implementation), the forms of TA provided tended to fall into two broad categories: policy analysis or applied policy research; and awareness facilitation via symposia or direct engagement with key stakeholders/decision-makers in government, the private sector or civil society. IPs are pursuing various successful (as they see it) routes to informing and influencing policy development and reform, primarily by increasing the availability of objective evidence and information. This reflects 177 “Feed the Future Guide to Supporting Sound Policy Enabling Environments.” U.S. Government Interagency. August 16, 2013. 178 Tumavick, Nancy M., and ASM Jahangir. “Final Draft: Feed the Future Report On Dialogue and Support for Policies Advancing Food Security, Agricultural Growth, And Better Nutrition.” March 30, 2016. p. 7. 89 the HICD component of the Feed the Future Monitoring, Evaluation, and Learning approach, which focuses on strengthening the quality of national datasets of line ministries and other local institutions, and through this process, developing the capacity of M&E offices within ministries and with other actors to perform this function. Numerous projects, such as the work to support rigorous data collection with the Food Planning and Monitoring Unit at the Ministry of Food in Bangladesh, are supporting the capacity of ministries to conduct rigorous monitoring and generate objective evidence that can inform policy. In Malawi, interviewees stated that they believed their advocacy efforts were largely successful because they were able to draw upon analysis performed by other policy-focused IPs funded by the Mission. Interviewees in Ghana at the Agriculture Policy Support Project (A3PSP), and in Bangladesh, both within the Food Planning Monitoring Unit (FPMU) and International Food Policy Research Institute (IFPRI), discussed similar themes related to how objective evidence was being used to support and drive the policy agenda. IPs also focus on facilitating implementation of existing policies. Based on interviews conducted for this evaluation, direct USG involvement in policy work has included the provision of technical assistance by USDA to the Government of Guatemala to facilitate compliance with the Food Safety Modernization Act (FSMA); in Bangladesh, the Food and Drug Administration (FDA) has been involved in advising the government on the implementation of national food safety policies; in Malawi, the USAID Mission advocated for adjustments to the national fertilizer subsidy program; and in Ghana and Uganda, USAID Mission officials have promoted increased involvement of the private sector in agriculture by developing opportunities and mechanisms for strengthened consultation and engagement. 2. Agricultural Research A primary mechanism by which Feed the Future is developing capacity in the agricultural research community is through scholarships that focus on training and education in agricultural science and research. Examples include activities conducted under the various Innovation Labs, as well as a number of specific projects that fall under the HICD area of responsibility of the Food Security Innovation Center in the Office of Agricultural Research and Policy, including:179 African Women in Agricultural Research and Development (AWARD); Borlaug Higher Education for Agricultural Research and Development (BHEARD); Borlaug Leadership Enhancement in Agriculture Program (Borlaug LEAP); the Borlaug US Global Food Security Fellowship Program, through USAID; Norman E. Borlaug International Agricultural Science and Technology Fellowship Program administered by 179 “Program for Human and Institutional Capacity Development.” Fact Sheet. Feed the Future. Undated. Retrieved from: https://feedthefuture.gov/sites/default/files/resource/files/ftf_factsheet_fsiccapacity_oct2014.pdf. 90 USDA; Innovation for Agricultural Training and Education (InnovATE); the Feed the Future CGIAR￾U.S. University Linkages Program; and MEAS (Modernizing Extension and Advisory Services). All of these activities focus on education and/or training provided to individuals, while several also provide additional capacity development support. For example, AWARD is a career-development program that equips top women agricultural scientists across sub-Saharan Africa by strengthening their research and leadership skills, through tailored fellowships. AWARD Fellows benefit from a two-year career development program focused on not just building science skills, but also fostering mentoring partnerships and developing leadership capacity.180 In addition, Feed the Future is building capacity by linking stakeholders to agricultural research in various ways, such as providing institutional and extension services design support, as well as conducting and sharing best practice research.181 InnovATE, managed by a consortium of U.S. universities, including Virginia Tech, Penn State, Tuskegee, and University of Florida, responds to requests from USAID missions to strengthen the full range of institutions responsible for educating agricultural professionals – including universities, technical schools, vocational schools, secondary schools, and primary schools – in such areas as curriculum reform, pedagogy, infrastructure, financing, and administration.182 In Nicaragua, Honduras, and the Democratic Republic of the Congo, InnovATE has or is conducting assessments of technical and vocational education training systems to improve youth employability in local agro-industries and reduce conflict.183 In Tajikistan, InnovATE is providing strategic support on how to make formal agricultural education and training more relevant to needs. “We have innovation networks that are designed to help share lessons learned, best practices from research. At the municipal level we do biodiversity fairs to raise awareness about the importance of biodiversity and introduce farmers to different plant varieties that they can grow. At the regional level we have universities and research centers like ICTA, the Ministry of Agriculture, NGOs, and farmers organizations to come together and say, “Okay this is a piece of research I’ve been doing, or “I’ve been using this technology.” – Guatemala IP 180 From a document provided to the research team by USAID/BFS, entitled “BFS/ARP Research and Capacity Development Programs in Ghana.” 181 “Modernizing Extension and Advisory Services (MEAS) Project.” USAID. Undated. Retrieved from: http://pdf.usaid.gov/pdf_docs/pdacu599.pdf. 182 “Program for Human and Institutional Capacity Development.” Fact Sheet. Feed the Future. Undated. Retrieved from: https://feedthefuture.gov/sites/default/files/resource/files/ftf_factsheet_fsiccapacity_oct2014.pdf. 183 Ibid. 91 Feed the Future is working with agricultural research institutes through a number of these activities, as well as through others, to establish linkages with other institutions, including U.S. universities, and with national agricultural policy systems. In addition, Feed the Future provides a variety of hard inputs, and develops capacity of agricultural research institutions and systems in the developing world to sustainably advance research that is most applicable in developing country contexts. (Many of these results are discussed in greater detail under evaluation Question #9). As one example, a number of different inputs were provided to SARI (The Savannah Agricultural Research Institute) by the Agriculture Technology Transfer (ATT) project in Ghana, including laboratory equipment and installation of a small-scale irrigation system.184 These inputs were reportedly very useful in supporting SARI to perform key functions, especially in terms of generating high quality seed in adequate volume to provide this for multiplication; however, there was also a request for additional assistance to support a general strengthening of SARI’s internal organizational management and other capacities. 3. Higher Education Much of the capacity development occurring in the agricultural research space, as described above, also bleeds into capacity development of higher education institutions. One particularly noteworthy example is the Innovative Agricultural Research Initiative (iAGRI). iAGRI is a cooperative agreement funded by USAID/Tanzania and managed by Ohio State University, in collaboration with the Sokoine University of Agriculture in Morogoro, Tanzania, and the Tanzanian Ministry of Agriculture, Food Security and Cooperatives. 185 In December 2011, iAGRI conducted a very complex assessment of the need for capacity development across the “agricultural development and food security” sector in Tanzania.186 4. Agricultural Extension Capacity development is occurring in agricultural and linked ministries, and associated institutions responsible for providing extension services in focus countries. This is targeted at developing capacity in personnel (for example, in extension agents and bureaucratic managers within ministries) to support policy analysis and formulation, and policy execution; and also to improve institutional alignment between ministries. Capacities for consultation and convening are also 184 Group interview conducted with SARI personnel in Tamale, Northern Ghana. 185 http://iagri.org/about/. 186 Crawford, Eric, Isaac Minde, Kathleen Colverson, Russell Freed, and Steven Haggblade. “Assessment of Needs for Training, Collaborative Research, and Institutional Capacity Building for Agricultural Development and Food Security in Tanzania. iAGRI Report Series, No. 1.” The Ohio State University Consortium. December, 2011. 92 targeted, and organizational systems to support this function are being developed with Feed the Future support. Local government structures – as in Guatemala, Ghana, and Senegal, for example – are also being developed, by working with municipalities or district governments and local producer groups or associations, and training/coaching local government to be more consultative and provide basic services and utilize participatory planning processes. An excellent example of how Feed the Future is developing tools that can be used to improve the performance of extension workers is the handbook entitled “Process Skills and Competency Tools: What Every Extension Worker Should Know – Core Competency Handbook,” by Murari Suvedi and Michael Kaplowitz, produced by Michigan State University under the MEAS project. MEAS has produced an extensive spread of additional knowledge support documents and technical notes, which includes guidance for managers and planners, a series on teaching and learning for extension practitioners, and various issues briefs, all of very high quality. They also provide a portal for technical content from other sources.187Capacity building through and of agriculture extension services is also supported by the whole-of-government initiative, such as the GAFSP (US Treasury) RESEPAG II program in Haiti, which provides support to both public and private extension services, training and demonstration, in 5 departments. Field interviews note that while most training as described seems to be delivered in the style of traditional extension (i.e. a trainer/expert going to the field to deliver training on agricultural technologies/practices directly to farmers; the training is typically delivered in a top-down manner by an outside trainer or extension worker and topics are determined by the IP, as dictated by their project designs/SOWs), a few IPs discussed skill development using other extension modes— demand driven (i.e., the farmer decides what kind of skill they need to develop, or peer-led (i.e., a smallholder or group of smallholders is the conduit for training information). Beyond production￾focused extension, some IPs (n=13) also discussed other types of training, such as business skills/marketing training (to position smallholders to be able establish linkages further up the value chain), such as the following example: “The biggest challenge of cultural development is people doing agriculture without a purpose. If you asked any household, there is no household with a vision, there is no household with a purpose. If you ask them why they are doing what they are doing, they will have no answer. This approach helps households to develop visions together as the household, and then develop plans on how they can meet their visions. If you must have a vision, if you must be able to attain it, then you need to make money. How do you make money? Farmers are never 187 http://www.meas-extension.org/meas-offers/technical-notes 93 entrepreneurs. They don't think about it as business. The business skills development also can be difficult…[so]are trained in entrepreneurship in addition.” – Uganda IP 5. Private Sector The private sector is being developed extensively in focus countries, and it could be argued (as noted by a key informant at BFS) that the value chain approach itself is a form of capacity development.188 Technical capabilities are being expanded within private sector entities – for example, through disseminating best practices in grain storage and certification standards, as through the work of the Ghana Grain Council supported by the Agricultural Development and Value Chain Enhancement (ADVANCE) project in Ghana.189 Micro-, small or medium enterprise (MSME) business development or business management skills development training is occurring in a wide spread of projects,190 and, as Financing Ghanaian. Agriculture Project (FINGAP) noted in Ghana, “consciousness raising” regarding potential financial products and markets for their services is occurring among financial institutions.191 Similarly, input suppliers, farmers, and agricultural enterprises of all sizes and types are being capacitated to better understand the economics and the science of agricultural productivity, illustrated by the Agricultural Innovation Partnership (AIP) Project in Malawi: “[The seed companies] have little knowledge on agronomy, on breeding, especially the germ, getting new germplasm and producing the quality seeds. We trained those seed companies, especially on agronomical aspects, like seed production and processing, marketing, and seed inspection.” – Malawi IP In addition, association development is occurring,192 as is formation and strengthening of policy advocacy networks.193 This policy advocacy is occurring extensively through the Feed the Future 188 For reference, see: USAID. “Key Elements of the Value Chain Approach- Briefing Paper.” Undated. Retrieved from: http://pdf.usaid.gov/pdf_docs/Pnadp303.pdf . Interestingly, there is no specific definition of a value chain provided in this document, but a review of this, when viewed through a sensitive capacity development lens, reveals numerous elements of the approach that either suggest or align with capacity development as broadly understood. 189 Field Interviews conducted with both ADVANCE and the Ghana Grain Council in Ghana. 190 Feed the Future. “Synthesis of Evaluations Related to the Feed the Future Learning Agenda.” March, 2016. Numerous examples of this are provided throughout this document. 191 Key Informant interview with FINGAP in Ghana. 192 Field interviews in Bangladesh and Ghana. 193 Field interviews in Ghana with Africa LEAD. 94 emphasis on consultation and explicitly engaging the private sector – a key element of the Feed the Future approach to policy development and support, as discussed elsewhere. IPs are working to increase the capacity and awareness of civil society and private sector actors to increase the number of local actors who are capable of holding governments accountable. 6. Civil Society Based on field interviews that included IPs and USAID missions, many activities said to be supporting “country ownership” are explicitly aimed at developing the capacity of key stakeholders, many of whom are outside of government. The various categories of capacity development models occurring with civil society actors, emerging from fieldwork, include: ● Maintaining consistent communication and engagement with stakeholders – including civil society actor processes of consultation to obtain input into designs or strategic plans for Feed the Future activities ● Provision of technical assistance to facilitate civil society decision-making ● Co-implementation of program activities with non-governmental counterparts ● Providing focused, customized trainings designed to strengthen non-governmental institutions key to achievement of Feed the Future goals Organizational capacity building support, especially for local private and civil society organizations, has been explicitly encouraged by USAID through its USAID Forward initiative; 10 IPs noted that USAID Forward was one of the reasons why they had engaged in or received institutional strengthening support. Approaches focused on NGOs were emphasized by a wide variety of IPs, and nearly 60% of interviewees mentioned projects that engage local civil society and private sector entities in the implementation of project activities, with some local entities engaged in co￾implementation of Feed the Future activities as the lead-implementer, such as the National Union of Coffee Agribusiness and Farm Enterprises (NUCAFE) in Uganda: “We have also had our own institutional capacity built as NUCAFE. It was one of the key objectives of this activity as a local partner. I think this is the initial part that USAID does with local partners…to improve our human resource, the number of human resource that we are working with. [They] provided some key trainings in different skills, and we have established a first-of-its kind management information system which can capture information right from the farm level, all the way to the national level.” – Uganda IP Highlighting the nexus of working with multiple categories of capacity development delineated above, Africa Lead II (supported by funding from the Office of Agricultural Research and Policy (ARP) at USAID/BFS) identified main activity areas through consultations with the African Union 95 and other African stakeholders on areas under CAADP that required greater attention. These included institutional capacity development, policy capacity development, and non-state actor participation. Towards these ends, Africa Lead provides leadership and management training; provides logistical support for training workshops and meetings; maintains a database of trainings that can be matched to local leadership and capacity development needs;194 and performs institutional capacity assessments and “internship/twinning arrangements” prioritized in consultation with missions and partners. For example, Africa Lead has a program focused on mobilizing and empowering civil society actors to form coalitions and serve as policy advocates.195 Similarly, the organizational capacity assessment (OCA) instrument used by Africa Lead covers a number of areas essential for the effective functioning of any organization. As one example, the OCA of the Ghana National Association of Farmers and Fishermen (GNAFF) addressed clarifying and systematizing organizational roles and responsibilities, and developing or reforming various policies and systems; they also provided recommendations on how to address these.196 Monitoring and Evaluation of Feed the Future HICD Activities The 2010 Feed the Future Guide states that “Investing in strengthening partner country capacity to engage in results-based planning …is a key component of our approach.” (p. 4) Numerous interviewees noted that the FTFMS, along with all the associated guidance documents, is building technical capacity across the initiative in M&E – an explicit purpose identified in the Feed the Future M&E Guidance.197 Multiple interviews revealed that this does not focus solely on host country government institutions. The spread of M&E Guidance and other documents focused on learning produced by MEL, as well as trainings and webinars led by MEL staff, are all aimed at developing capacity. These activities include training for government officials, with the main Feed the Future actors involved in this effort being USDA and IFPRI. Further, M&E technical service providers are typically required to support national and local institutions that partner with USAID to improve the quality of their M&E. Many of these various capacity development activities and results associated with FTFMS are discussed in greater detail under Question #12. Participatory Design and Management: In 2011 – 2012, interagency committees at each post identified a set of key policy reforms that would support Feed the Future objectives and developed 194 “Energizing Champions for Food Security: Reflections on What Works, Why, and What Are the Challenges.” Africa Lead. April 2013. p.1 195 From group interview discussion with Africa Lead personnel. 196 Adams, Lawrencia, and Cecilia Addae. “Organizational Capacity Assessment Report. Ghana National Association of Farmers and Fishermen (GNAFF).” December 2014. 197 Feed the Future. “M&E Guidance Series Volume 1: Monitoring and Evaluation Under Feed the Future.” Undated. pp. 3-4. 96 policy matrices that contained prioritized sets of policy actions. These priorities were identified through discussions with host country governments, regional organizations, other donors, and representatives of the private sector and civil society, reflecting the principle of participatory design of capacity development interventions.198 Given the regular engagement with key counterparts that was evident across field country portfolios (with wide variation in terms of frequency and level of participation), the evaluation team believes that participatory management approaches are, in general, being applied. This finding is discussed in more detail under Question #11. HICD Measurement Challenges The document entitled “Assessment of the Portfolio of the USAID/BFS Policy Division,” dated March 2016, makes a very compelling observation: “Monitoring systems such as the FTFMS and the country policy matrices represent efforts to track policy changes across multiple countries and reporting units. These systems record considerable progress on policy reforms related to inputs, trade, and institutional architecture, which are the three themes to which the Policy Division allocates approximately 80% of its funds. Partners are also making valuable contributions to building capacity and strengthening local institutions, yet it is difficult to tell how much progress has been made to reach a point where the institutions will function significantly better.”199 To understand this critique – which emerges elsewhere (what Dichter et. al. describe as “focusing on metrics that are inputs rather than outcomes…”),200 and is validated by a key informant in the HICD unit at BFS – it is useful to review the specific indicators from the Feed the Future monitoring indicators used to track HICD. The trends associated with these indicators are discussed in greater detail in the section that addresses Evaluation Question #2. Examples of indicators for capacity development that can be extracted from the FTFMS include: ● Number of health facilities with established capacity to manage acute under-nutrition (Outcome) ● Number of national policies supporting regionally agreed-upon policies for which a national￾level implementation action has been taken as a result of USG assistance ● Number of agricultural enabling environment policies completing the following processes/steps of development as a result of USG assistance (Output/Outcome) ● Number of farmers and others who have applied improved technologies or management practices as a result of USG assistance (Outcome) 198 USAID. “Assessment of the Portfolio of the USAID/BFS Policy Division.” March 2016. 199 Hazell, Peter, Peter Ewell, and Kristy Cook “Assessment of the Portfolio of the USAID/BFS Policy Division.” March 2016. p. 51. 200 Dichter, Thomas, David Joslyn, Joyce L. Moock, and M. Diane Bellis. “Feed the Future Human and Institutional Capacity Development Strategy Review.” USAID. May 8, 2015. p. 4. 97 ● Number of individuals who have received USG supported long-term agricultural sector productivity or food security training (Output) ● Number of members of producer organizations and community-based organizations (CBOs) receiving USG assistance (Output) ● Number of MSMEs, including farmers, receiving business development services from USG￾assisted sources (Output) The cumulative numbers of the results, like those for the illustrative indicators above, tell a story in terms of the scope and scale of Feed the Future activities occurring across the initiative, and are unquestionably important for basic accountability purposes. The figures mostly speak for themselves: millions of people trained, millions of people applying improved technologies or management practices, millions receiving USG assistance, tens of thousands of enterprises and associations receiving assistance, thousands of health facilities with capacity to manage undernutrition, etc. However, these indicators do not capture, in a meaningful way, the degree to which capacity has been developed. They do track activities and record outputs, an essential purpose related to managing performance, and although a number of these are identified as outcome-level indicators, at best they are proxies201 for capacity development – and are difficult to attribute. Further, there is no indicator in this list that is specifically designed to capture improved performance, either at human or institutional levels, and track it over time in order to demonstrate the degree to which capacity has been developed. A draft concept note provided to the evaluation team by Africa Lead states, “The USAID Feed the Future Indicator handbook (FY 2012) calls for the use of its Organizational Capacity Assessment Tool (OCA) as the primary indicator for measuring and reporting on the objective of enhanced human and institutional capacity development…The OCA is a framework tool that measures the capacity of an institution...” Referring back to the Indicator Handbook Definition Sheets, the team found: “INDICATOR TITLE: CBLD-5 Score, in percent, of combined key areas of organization capacity amongst USG direct and indirect local implementing partners. DEFINITION: The reporting of the combined key area score will represent the capacity of FTF￾assisted local organizations measured across seven key capacity areas using the Organizational 201 As used here, a proxy indicator is discussed in USAID’s TIPS Note #6, “Selecting Performance Indicators.” (2010). This document states: “Proxy indicators are linked to the result by one or more assumptions…When proxies are used, the relationship between the indicator and the result should be well-understood and clearly articulated. The more assumptions the indicator is based upon, the weaker the indicator.” 98 Capacity Assessment (OCA) tool…Operating units should record score data for each organization in their performance management plan files so changes in scores for each organization can be monitored over time.”202 The evaluation team found no evidence that this indicator is currently being used systematically by Feed the Future. Additionally, it is important to note that this issue – tracking actual improvements rather than simply tracking activities and outputs – applies to USAID more broadly. A key informant noted that M&E tools that can be used to track improvements in performance or capacity require significant amounts of time (and expense) and require a fairly sophisticated set of capacities in M&E; as a result, they may be seen by many officers at USAID as being “too difficult” to use. USAID has posted a tool – the Organizational Performance Index (OPI)203 – that is likely applicable for addressing this gap in the M&E system.204 USAID has commissioned at least three studies that examine HICD approaches as part of their scope – with one focused specifically on partner perspectives, and a fourth focused on the policy agenda. The “Synthesis of Evaluations Related to the Feed the Future Learning Agenda” (March 2016) “…identifies and chronicles evidence from 11 impact evaluations and 185 performance evaluations conducted between 2010 and 2015 across 64 countries.” Throughout the document, under each of the Learning Agenda themes, activities that either explicitly or implicitly address capacity development are discussed. The document entitled “Assessment of the Portfolio of the USAID/BFS Policy Division,” dated March 2016, is primarily an inventory of centrally managed activities, and in the evaluation team’s view, the recommendations it contains are not clearly presented and lack utility for providing structured guidance to mission-level activity managers on how to design and manage HICD activities within their portfolios (although it is understood that this is not the intended purpose of this document). A second document, the “Feed the Future Human and Institutional Capacity Development Strategy Review,” contains a number of recommendations that are meant to improve the focus and effectiveness of the portfolio and the evaluation team believes many of these have merit. However, it should be noted that many of these are outside the manageable interest of BFS, and relate to the drivers and systems of USAID at higher levels and based upon numerous interviews with field personnel implementing HICD activities that were not captured in the data used to formulate that report (as it was primarily a desk review); there are also numerous examples that would constitute success to varying degrees. 202 Feed the Future Indicator Handbook: Definition Sheets (updated October 18, 2013). p. 77. 203 DuBois, Rachel, Matt Reeves and Olga Yakimakho. “Organizational Performance Index (OPI) Handbook.” PACT. January 2015. 204 https://usaidlearninglab.org/library/organizational-performance-index-measurement-tool 99 However, given the previously discussed measurement challenges, the degree to which these diverse activities add up to significant improvements in performance at institutional or systems levels is unclear. Political Economic Challenges As Dichter et. al. write: “Overall, we conclude that the bulk of the HICD portfolio today is less effective than it could be. Undue speed, short project time-frames, spreading things too thinly, working with overly large and complex consortia, setting out multiple and often incompatible objectives, focusing on metrics that are often inputs rather than outcomes, side-stepping the consequences (low motivation to buy-in, low demand) of the many pressures on Missions, and the lack of a clear and widely shared view of institutional capacity development, are at the root of the problem. The implicit model for institutional capacity development (ICD) at USAID (and BFS) is an idealized social engineering model, rather than a less linear, more inductive view of HICD challenges, one which begins with a careful examination of complex contextual constraints and pitfalls. As a result of all these factors, quality has suffered, as have impact, sustainability, and country ownership.”205 The reader should note how many of these critiques are in fact drivers that emerge as a result of inflexible institutional imperatives and mandatory policy requirements, or limited technical capacity and lack of financial resources, rather than poor management. IPs identified a number of challenges related to policy reform (one of the key focal areas for significant amounts of capacity development support); most of these can be broadly understood to relate to political economy in host countries. These include political climate, low government and/or civil society capacity, time constraints (activities that do not have adequate time horizons to actually produce impacts), and competing agendas, and they are all largely linked to overarching political-economic dynamics and capacity constraints at multiple levels and in multiple locations within the broader system. This issue of constraints associated with political economy, and the fact that it impacts capacity development, was also identified by a key informant at BFS, and was further identified as a key challenge or success factor related to CAADP processes, as discussed in Question #8. These most commonly cited challenges from the field clearly demonstrate how issues with a larger systems context affect capacity development generally, and also clearly establish the principles at play that constrain Feed the Future’s ability to change performance within institutions and within the wider system. They also demonstrate why the literature on capacity development 205 https://usaidlearninglab.org/library/organizational-performance-index-measurement-tool 100 so consistently discusses the need for rich and rolling context analysis,206 ongoing learning and adaptation, and the need to accept extended timelines to produce intended results. Recent assessments of policy activities show evidence that progress in actual implementation has been slow and uneven.207 The same finding is documented by the joint annual report (2015) from the New Alliance for Food Security and Nutrition and Grow Africa.208 Given the complex issues that inevitably affect policy implementation, this time lag is not surprising; furthermore, it is most likely not a result of poor management of the reform process that can be attributed to failures within Feed the Future. It is, however, likely an indication of the weakness of performance-based management approaches when applied to policy reform, capacity development, and other activities that are heavily conditioned by complexity issues within open systems. Technical HICD Capacity Challenges within Feed the Future A number of data points suggest there is a strong tendency to view capacity development through the limited lens of providing training, an observation validated by a key informant at BFS. This is by no means the only approach: as noted earlier in this section, alongside discussion of training, interviewees also discussed various approaches to strengthening institutions, and examples of other activities meant to develop wider systems. These included provision of external technical assistance, improving internal management systems of local partners, delivery of organizational assessments, or in-kind provision of equipment or other material support. Twenty IPs explicitly mentioned these other activities as being aligned with or supported by training being provided. Focusing too extensively on training as a weakness in Feed the Future is noted in the HICD strategy review: “…emphasis on training that reflects a standard “fill in the gaps” approach to ICD.”209 206 The literature referenced here is extensive. Numerous documents focused on capacity development practice reinforce the theme. These include documents produced not only by USAID, such as: USAID’s “Human and Institutional Capacity Development Handbook” (October 2010); and USAID/AIDSTAR “Challenges Encountered in Capacity Building: Review of Literature and Selected Tools. Position Paper No. 10” (April 2010). Additionally, sources produced by the World Bank, UNDP, OECD, the Overseas Development Institute (ODI), and the European Centre for Development Policy Management (ECDPM) reinforce the theme. 207 See: Hazell, Peter, Peter Ewell and Kristy Cook, “Assessment of the Portfolio of the USAID/BFS Policy Division.” Feed the Future. March 2016; See also: Tumavick, Nancy M., and ASM Jahangir. “Feed the Future Report on Dialogue and Support for Policies Advancing Food Security, Agricultural Growth and Better Nutrition. Feed the Future. March 2016 (final draft). 208 “New Alliance for Food Security and Nutrition and Grow Africa Joint Annual Progress Report: 2014-2015.” p. 8. 209 Dichter, Thomas, David Joslyn, Joyce L. Moock, and M. Diane Bellis. “Feed the Future Human and Institutional Capacity Development Strategy Review.” USAID. May 8, 2015. p.4. 101 Additionally, at a meeting held in Washington focused on Innovation Labs, data was presented from a survey of Innovation Lab partners. When asked to name “three activities that convey capacity development,” the top answer – provided by 98% of respondents – was training. Some variation of training was mentioned by 56 field interviewees from IPs, while other means of institutional strengthening occurred only 46 times; at USAID, training was named 13 times while institutional strengthening was mentioned only 10 times. The evaluation synthesis report mentions training 169 times, while capacity is mentioned 104 times, and the two terms co-occur 21 times. In the 2015 HICD/BIFAD Portfolio Review, there are two pages devoted to “Agricultural Education and Training,” while one page each is focused on “Agricultural Extension,” “Agricultural Research,” and “Institutional Strengthening.” In other 2015 portfolio reviews, training is mentioned 20 times in Bangladesh, 16 times in Ghana, 22 times in Guatemala, 15 times in Malawi, nine times in Uganda, and 18 times in West Africa. It is not clear whether these discussions of training are embedded within broader, integrated capacity development interventions, but the frequency with which training is mentioned, without further nuanced discussion, is striking. As noted in the presentation made at the meeting with the Innovation Labs (slide 8), “Training is necessary, but not sufficient for institution building. Alone, it has limited impact on total institutional performance, particularly when the institution starts weak.” QUESTION 4 CONCLUSIONS 1. Given the inevitable variation that must exist – both in how capacity development activities may have been designed, and in implementation, across such a wide spread of activities – as well as the tremendous variation in enabling and/or supporting contextual factors, coupled with numerous data issues as described above, it is not possible to identify the “extent” to which capacity has been “built,” given the scope of this evaluation. There are numerous achievements, and many of these are discussed above – and this is by no means a comprehensive list. It must also be acknowledged that the evidence of success is mixed. The “Synthesis of Evaluations Related to the Feed the Future Learning Agenda” strongly supports this view. Certain very broad patterns that have been observed may serve as useful points of discussion within Feed the Future to inform thinking about areas for improvement. 2. There is an overemphasis on training and individual development evident within Feed the Future relative to broader institutional and wider systems development. This tendency is part of a broader challenge and program design risk that exists within the technical practice area of capacity development, and to some degree can be traced back to an evident lack of consistent and detailed practical guidance. 102 There appears to be a “…lack of a clear and widely shared view of institutional capacity development…”210 within Feed the Future. As a result, training is to a large degree the default approach to capacity development among program design personnel who lack a sophisticated understanding of the complex issues affecting institutional performance, and who also lack technical familiarity with the various modalities by which robust capacity development is designed and implemented. There appears to be “little nuanced understanding of ICD as a complex, iterative process; the emphasis on training reflects a standard ‘fill-in-the-gaps’ approach to ICD.”211 The need to move beyond training to ensure the success of capacity development is also widely noted throughout the literature on best practices within the field.212 3. USAID’s HICD guidance – which suffers from a lack of instructive and practical detail – clearly emphasizes the human and the institutional aspects of organizational performance and does not currently consider wider extra-organizational factors and how they influence organizational capability to function effectively within a much larger and more complex agricultural development context. Awareness of the importance of this wider context is evident in the thinking of USAID officers sitting in Washington. To a certain degree, this awareness is being integrated into capacity development interventions in the field, but this appears to be based upon individual awareness and personal experience. As a result, there is a risk that capacity development interventions will not adequately consider or strategically address this critically important systems-context dimension, which is necessary to ensure success. Furthermore, the incentives determined by the current indicators used to gauge success, as well as policy requirements and guidance provided under Feed the Future, further entrenches the risk. 4. Specific approaches to HICD at USAID – as laid out in the guidance – as well as design and implementation guidance provided under Feed the Future, are currently inadequate to support robust engagement with complexity issues conditioning organizational performance in country agricultural development spaces – especially the performance of counterpart governments as 210 Dichter, Thomas, David Joslyn, Joyce L. Moock and M. Diane Bellis. “Feed the Future Human and Institutional Capacity Development Strategy Review.” USAID. May 8, 2015. p.4. 211 Ibid. 212 See, for example: Pearson, Jenny. “Training and Beyond: Seeking Better Practices for Capacity Development. OECD Development Co-operation Working Papers No. 1.” OECD/DAC publishing. 04-Apr￾2011. See also: Hervy, Anne-Claire and Andrew Gilboy. “Good Practices in Leveraging Long-term Training for Institutional Capacity Strengthening.” Association for Public and Land-grant Universities, Knowledge Center on Higher Education for African Development. February 2014. 103 influenced by complex political economic drivers of performance. Feed the Future is addressing these issues at an operational level but needs to be done in a consistent manner. The convening and multi-stakeholder facilitation functions are potentially very powerful approaches to systems-level capacity development. Association development and multi￾stakeholder facilitated coordination processes are currently recognized as being important to improve the function of agriculture development systems – and could be integrated more consistently. 5. Current approaches to M&E of capacity development at Feed the Future are focused on activities, outputs, and proxy measures that do not adequately capture changes in capacity or performance. This makes assessment of success – i.e., tracking actual changes in individual and organizational performance – difficult. As a result, it is not possible to accurately assess the extent to which capacity has been built. QUESTION 4 RECOMMENDATIONS 1. Feed the Future should develop explicit guidance on complexity awareness and context factors that should be considered when designing capacity development activities, and an associated catalog of illustrative approaches for engaging with complex systems. 2. Feed the Future should develop guidance based on Complexity Aware Monitoring and Evaluation of capacity development activities that goes beyond the current emphasis on “counting outputs,” and that specifically recognizes principles of dynamism, emergence, action learning, and extended timelines. There are numerous other best practice resources for sensitive, adaptive, context-aware M&E213 of capacity development that can be referenced as well, including numerous documents produced by the World Bank, the OECD, the Overseas Development Institute (ODI), and the European Centre for Development Policy Management (ECDPM). 3. Feed the Future should conduct a thorough inventory of capacity development approaches being implemented, including diverse “cross-cutting” sub-activities occurring in Mission portfolios. As part of this exercise, Feed the Future should identify and compile best practices from within the 213 https://usaidlearninglab.org/library/complexity-aware-monitoring-discussion-note-brief 104 wider literature on capacity development;214 given Feed the Future resources, BFS may be one of few operating units within USAID that could assemble such a resource – which would be a useful tool for USAID more broadly. Feed the Future should explicitly identify success factors to be analyzed and/or addressed in best practice capacity development activities moving forward. 4. “Feed the Future Institutional Mapping and Assessment,” produced by Africa Lead, suggested “…in-depth capacity assessments should be conducted for all organizations for which USAID will provide program support. These assessments should … Develop a baseline of each organization’s capacity metrics that can be used … for measuring and reporting subsequent capacity improvement (institutional strengthening).”215 MEL should review available organizational assessment tools, such as the previously mentioned OPI and OCA, to issue guidance on how to develop an organizational assessment tool to produce an index that would produce a consistent, objective measure of organizational performance to track progress over time, and issue guidance for its use to track performance over time. 5. Given: (1) the clear emphasis on coordination evident in both the Feed the Future Guide and the Feed the Future Guide to Supporting Sound Policy Enabling Environments,216 as well as the emphasis on broad stakeholder consultation, and multi-donor and multi-agency coordination; (2) the fact that coordination processes are occurring as a key element of food security approaches at country level, and that the requirements for optimal success are so demanding; and (3) the recognized imperative that this task should be owned primarily by host country institutions mandated to push forward policy and ensure effective agricultural development, USAID should ensure these coordination bodies are supported through strategic capacity development interventions. Institutional capacity assessments that focus on key capacities necessary for successful engagement and coordination – such as those identified by the Food Security Collaborative217 – should be conducted in order to ensure success, and specific capacity development activities should be delivered to support these bodies. These support activities should not be conceived solely as the provision of training. 214 A body of normative and academic literature that is extensive, some of which has been supported by USAID. See, for example: http://www.developmentiscapacity.org/. 215 Burruss, Duke, Ndidi Nwuneli and Kwesi Opoku-Debrah. “Feed the Future Institutional Mapping and Assessment.” Africa LEAD. January 25, 2012. p.19. 216 “Feed the Future Guide to Supporting Sound Policy Enabling Environments.” U.S. Government Interagency. August 16, 2013. 217 Food Security Collaborative. “Building Institutional Capacity to Increase Food Security in Africa.” April 2012. 105 6. Feed the Future should design mechanisms for ensuring country projects have access to professional, high-quality, and technically proficient group process facilitation services.218 This may take the form of procuring technical facilitation services centrally that missions could buy into, or involve developing a set of tasks that could be grafted into existing activities in focus countries. As this skill set may be present to varying degrees among existing service providers – for example, it appears that Africa Lead is already performing this function to a certain degree – assessing this capacity and then merely modifying an existing agreement might perhaps suffice. Question 5: How and to what extent have Feed the Future interventions promoted gender-inclusive agricultural sector growth and improved nutritional status of women through equitable and strategic integration of women and men in agriculture and nutrition programs? There is no established common understanding of what gender-inclusivity means across the Feed the Future initiative. Given that approaches to gender-inclusivity are varied across the initiative, in general, there seems to be a fairly basic integration of men and women in agriculture and nutrition programs across all 19 focus countries. The most prominent manifestation of this is through targeting interventions that aim to reach women beneficiaries and produce sex-disaggregated output data. There are many gendered interventions across the 19 focus countries that are directly addressing nuanced constraints faced by both men and women. Almost all gendered interventions reviewed for the Feed the Future Learning Agenda Literature Review,219 regardless of their focus, were reported as successful. As agricultural interventions are targeted largely toward men and nutrition interventions toward women, nutrition interventions are generally highlighted as more gender￾inclusive than agricultural interventions. QUESTION 5 FINDINGS A large body of literature demonstrates that gender equality and women’s empowerment play a critical role in international development generally, and in the growth of the agricultural sector and 218 There are numerous resources that could be examined to structure understanding of what specifically group process facilitation entails. One very succinct document worth examining is from the University of Virginia, available here: http://www.virginia.edu/processsimplification/resources/Facilitator.pdf 219 USAID. “Feed the Future Learning Agenda Literature Review: Improved Gender Integration and Women’s Empowerment.” January 2015. See also, Meinzen-Dick, Ruth Suseela; and Quisumbing, Agnes R. 2013. 106 the reduction of poverty and hunger specifically.220 In particular, the literature highlights that there is a need to close the gender gap in access to agricultural resources, including education, financial services, and extension; to invest in labor-saving technologies that enable women to dedicate more time to more productive activities; and to facilitate women’s participation in labor markets.221 Additionally, the literature stresses that when these gaps are addressed, gender equality can lead to gains in productivity. Empowerment of women as economic, social, and political actors can lead to more representative decision-making and women’s increased control of household resources, which in turn can result in improved outcomes for the next generation.222 Building upon this evidence, the Feed the Future Strategy asserts that: 1) reducing gender inequality and recognizing the contribution of women to agriculture are critical to achieving global food security; and 2) consistent and compelling evidence shows that improvements in the status of women are associated with increases in agricultural productivity, reductions in poverty, and improvements in household nutrition.223 The Feed the Future Guide explains that these objectives are to be achieved by incorporating the following three principles in design and implementation: 1) Work with partner countries to promote gender-sensitive consultative processes; 2) Strive for gender equality in all stages of planning and implementation; and 3) Recognize the importance of considering the gender impact of all Feed the Future investments on access to and control over assets. (Feed the Future Guide – 6.1 Guidance on Gender). In accordance with the Feed the Future Strategy and the USAID Gender Equality and Female Empowerment Policy224 released in 2012, Feed the Future Gender Guidance stipulates the following three objectives: 1) Ensure women have equal access to assets, inputs, technologies; 2) Expand involvement/participation of women in decision￾making; and 3) Ensure interests of women/men are reflected in all Feed the Future policies and programs and of host countries’ Country Investment Plans. Given this normative framework, the 220 See, for example, USAID. “Feed the Future Learning Agenda Literature Review: Improved Gender Integration and Women’s Empowerment.” January 2015. See also, Meinzen-Dick, Ruth Suseela; and Quisumbing, Agnes R. 2013. 221 FAO. “The State of Food and Agriculture – Women in Agriculture (2010-11).” 2011; and The World Bank. “World Development Report 2012: Gender Equality and Development.” 2011. 222 “Women in Agriculture: Closing the Gender Gap.” In 2012 Global Food Policy Report, Chapter 4. pp. 37- 47. Washington, D.C.: International Food Policy Research Institute. 223 Congressional Research Service. “Report on The Obama Administration’s Feed the Future Initiative.” 2016. 224 The USAID Gender Equality and Female Empowerment Policy released in 2012 outlines three overarching outcomes that USAID investments should aim towards: “reduce gender disparities in access to, control over and benefit from resources, wealth, opportunities, and services…reduce gender-based violence and mitigate its harmful effects on individuals and communities, so that all people live healthy and productive lives…[and] increase capability of women and girls to realize their rights, determine their life outcomes, and influence decision-making in households, communities, and societies.” (USAID Gender Policy, 2012). 107 remainder of this section outlines the most salient findings and conclusions the team developed through triangulation of data from document review, field interviews, the FTFMS, and online surveys. At the initiative level, Feed the Future has promoted equitable and strategic integration of women and men in agriculture and nutrition programs through a number of framework and guidance documents, tools, and mandates.225 As noted above, the Feed the Future Guide, which serves as the guiding document on the Feed the Future framework, has a section dedicated to Guidance on Gender which lays out Feed the Future’s approach to gender-inclusivity (pp. 27 – 29). In addition, gender is one of the six dimensions delineated in the Feed the Future Learning Agenda, placing a significant emphasis on the importance of gender in programming (pp. 1 and 5). Building upon the centrality of gender in the guidance documents, Feed the Future promotes gender-inclusivity in various stages of implementation, largely through monitoring, evaluation, and learning approaches as outlined in the Feed the Future M&E Guidance Series Volume 6: “Measuring the Gender Impact of Feed the Future.” (March 2014). This is primarily done through three mechanisms: 1) Sex-disaggregated performance monitoring, 2) Gender-focused impact evaluations, and 3) The Women’s Empowerment in Agriculture Index (WEAI).226 As of 2014, there is a total of 33 sex-disaggregated or women-specific indicators.227 In addition to the Feed the Future Learning Agenda outlining six gender-focused questions, missions are encouraged to identify gender concerns when reviewing projects, as well as developing a learning agenda that could be used for impact evaluation.228 The Feed the Future initiative also mandates that every Feed the Future focus country conduct a gender assessment that could/should provide strategy recommendations.229 The BFS technical offices intentionally incorporate gender concerns into their portfolios as evidenced by the portfolio reviews for each division, which include a specific section dedicated to 225 These include, but are not limited to: “The Feed the Future Guide”, 2010; The Gender Integration Framework; “Increasing Feed the Future Impact through Targeted Gender Integration” Webinar, 2014; “Gender Integration Framework – Feed the Future” PowerPoint Presentation, 2014; WEAI Resource Center – Website (https://www.ifpri.org/topic/weai-resource-center), IFPRI; “Feed the Future M&E Guidance Series Volume 6: Measuring the Gender Impact of Feed the Future.” 226 “Feed the Future M&E Guidance Series Vol. 6: Measuring the Gender Impact of Feed the Future.” 2014. pp. 1 – 4. 227 For a full list of all 33 indicators, see Feed the Future M&E Guidance Series Vol. 6: Measuring the Gender Impact of FTF. 2014. p. 2. 228 “Feed the Future M&E Guidance Series Vol. 6: Measuring the Gender Impact of FTF.” 2014. p. 4. 229 “Feed the Future Guide.” May 2010. p. 20. 108 gender. The focus country portfolio review templates for 2014 and 2015 include a specific section for gender integration and empowerment in agriculture, providing information on select sex￾disaggregated indicators, analysis, and interventions to address gender gaps at scale.230 In addition, “each of the [Feed the Future] Implementation Plans [IPs] and Multi-Year Strategies [MYSs], mention women and girls or gender integration.”231 All interventions are required to collect sex￾disaggregated data on all person-specific indicators, with the intention of measuring progress of interventions across genders, as well as targeting interventions in identified weak spots. While not uniformly mandated, a number of interventions and countries also conduct impact evaluations focusing on gender-specific questions.232 Supporting the initiative’s stance that women are the primary focus of the first level objective of “inclusive agricultural sector growth,”233 Feed the Future launched the WEAI in 2012 to collect data on the roles and extent of women’s engagement in the agricultural sector in five domains: women’s role in household decision-making around agricultural production; women’s access to productive capital; women’s income and expenditures; women’s individual leadership and influence in the community; and women’s time allocation.234 The Gender Integration Framework (GIF) provides guidance for staff and implementers to look at gender-focused problems/constraints for each of the WEAI domains (as well as human capital and access to technology) and identify activities that would address those issues and reduce women’s disempowerment. The GIF also helps to identify indicators to track progress toward desired outcomes.235 Given the cross-cutting, yet sensitive, nature of gender, and the wide range of Feed the Future interventions across the 19 focus countries, this evaluation has found that the understanding of and subsequent approaches to the promotion of gender-inclusive agriculture sector growth and improved nutritional status of women is not standard across the initiative, but is instead varied, both across countries, and within countries across interventions. 230 Based on evaluation team’s review of all BFS Division – and Focus Country – Portfolio Review PowerPoint presentations. 231 Women Thrive Worldwide. “Gender in Feed the Future.” http://www.womenthrive.org/sites/default/files/images/agriculture%20genderscan.pdf. p. 1. 232 Rey, Sally. Feed the Future AgriLinks Blog: “How does Feed the Future measure gender impacts?” Accessed at: https://agrilinks.org/blog/how-does-feed-future-measure-gender-impacts. 233 Taken from the “Feed the Future M&E Guidance Series Vol. 6: Measuring the Gender Impact of FTF.” 2014. 234 Alkire, Sabina, et. Al. “The Women’s Empowerment in Agriculture Index.” IFPRI, 2012. p. 2. 235 Feed the Future AgriLinks. “Increasing Feed the Future Impacts through Targeted Gender Integration.” Accessed at: https://agrilinks.org/events/increasing-feed-future-impacts-through-targeted-gender￾integration 109 While it is difficult to neatly package and categorize the variety of interventions within specific gender-inclusivity frameworks, using the language presented in the literature review conducted for the Feed the Future Learning Agenda, most of the Feed the Future gender-focused interventions and/or sub-activities within an intervention can be described as either gendered or targeted interventions.236 “Targeting is the mechanism projects use to reach women beneficiaries, whereas gendering is the mechanism projects use to address gender disparities between women and men in the larger cultural setting in order to devise strategies to increase women’s empowerment.”237 There are Feed the Future activities that are not gender-focused by design, but can be considered gender-inclusive because they target women beneficiaries – these include trainings, seed distributions, and new technology adoption. The outputs of these targeting interventions are largely captured by the sex-disaggregated indicators and used to describe outcomes in terms of equitable integration of women and men into agriculture and nutrition programs. During field data collection in the five fieldwork countries, when asked about gender-inclusivity, most respondents first referred to the sex-disaggregated data. This approach to and understanding of gender￾inclusivity is defined by the ratio of male-to-female beneficiaries. While the FTFMS houses sex-disaggregated data for all IMs, the ability to interpret this data as a discrete data sources aggregated at the initiative level is limited. In addition to missing data for a majority of the baseline values of sex-disaggregated indicators, the sex-disaggregated data is not available for either a proportion of the total value or for the entire reported person-specific value for a number of indicators.238 A high-level, simple analysis of the sex-disaggregated data shows that the average percentage of women beneficiaries reached, for indicators counting individuals,239 236 This section recognizes that there is a distinction between interventions and sub-activities within interventions in regards to having a gender-focus. For example, either an entire intervention can be gender￾focused or sub-activities within an intervention can be gender-focused. However, this section will refer to both interventions as well as sub-activities within interventions as ‘interventions’ in order to keep the writing clear and concise. 237 “Feed the Future Learning Agenda Literature Review: Improved Gender Integration and Women’s Empowerment.” June 23, 2015. USAID. p. 7. 238 All the person-specific indicators provide the options disaggregate by male, female, or enter a value for “disaggregates not available.” There are some indicators that provide disaggregates by male and female for a portion of the total value, but not the entire value. There are also indicators that do not provide any disaggregates and only report a total value for the indicator. Specific example of these indicators included: 4.5.2(5), 4.5.2(7), 4.5.2(13), and 4.5.2(14). 239 The term “reached” is used as a proxy for the specific measure of the indicator such as “farmers who have applied improved technologies” or “individual who have received USG supported agricultural sector productivity or food security training.” 110 as a proportion of total persons reached across all 19 focus countries, was 38% in 2010, 31% in 2011, 33% in 2012, 35% in 2013, 39% in 2014, and 38% in 2015.240 However, the interpretation of these figures at the aggregated level must be nuanced given that sex-disaggregated data was not consistently, nor completely, reported across all implementing mechanisms. While sex￾disaggregated data is useful for tracking results in the number of men and women reached, it does not provide insight into what changes took place that resulted in increase of decrease of numbers of individuals reached. However, as discussed below, tools such as the WEAI can provide this contextual information. There are also Feed the Future interventions that take a gendered approach. One of the approaches mentioned most often to promote gender-inclusivity was the household approach. The household approach refers to the inclusion of all household members – women, men, and children – in intervention targeting. This aims to address gender-inclusivity through the framework of gender dynamics within the household as opposed to simply targeting women as an isolated group for activity inclusion.241 During field data collection in the five fieldwork countries, this approach was referred to in all countries at varying levels; however, it was most prominently mentioned in Uganda. A specific example of the household approach in implementation in Uganda is the “Guide to Involve Household Family Members in Development Interventions,” produced by The Agricultural Business Initiative (aBi) Trust, which was developed “in response to the gender concerns in agricultural value chains and the need for supporting households to address these concerns, including labor, access and control over resources, and sharing of proceeds from production…to ensure that gender equality is achieved at household level for the benefit of all household members, and hence livelihood improvement.”242 Other gendered intervention approaches include involving men in areas that have been seen to be traditionally women’s roles, such as care and feeding of children, which was mentioned in interviews in all five of the fieldwork countries. Similarly, designing interventions to address identified areas of gender disparities and/or supporting enabling factors, such as land rights policies and collective action communities that enable sustained gender-inclusivity, also constitute gendered intervention approaches. Examples of this include Village Savings and Loans Associations (VSLAs) to address the credit constraints faced by women. Credit constraints are cited as one of the three main constraints to women’s empowerment in agriculture in the WEAI Baseline Report, 240 This analysis only includes indicators that have female disaggregates and does not included gendered household and business types. 241 SPRING, “Linking Agriculture to Nutrition: A Guide to Context Assessment tools” (2014). p. 13. 242 “Guide to Involve Household Family Members in Development Interventions” aBi Trust, 2013. 111 2014.243 Examples of this include the SPRING activity in Ghana as well as the Food and Enterprise Development Program (FED) in Liberia.244 Additionally, there are a number of WOG examples, such as USADF grants supporting initiatives like Cooperative for the Promotion of Fishing and Commerce Activities of Isambaza (COOPPAVI) in Rwanda, a predominantly female cooperative producing highly nutritious fishmeal in the Lake Kivu region where fishing is a male-dominated industry.245 While the outputs of these types of gendered interventions are also captured to a certain degree by the sex-disaggregated data, their outcomes are multi-dimensional and better captured by tools such as the WEAI, which is designed to capture information on multiple domains and allow for context-specific heterogeneity by collecting data on both men and women in the same household.246 Although the aim of gendered interventions is also towards equitable integration, their focus is more on strategic integration of women and men into agriculture and nutrition programs rather than simple inclusion. Both in the literature and in practice, a general gender bias was noted toward working with men in the agriculture sector and women in nutrition.247 The Landscape Analysis of Activities across all 19 focus countries conducted by SPRING in 2014 found that the Feed the Future documents “clearly stated that the primary targets of Feed the Future agriculture and economic growth interventions are smallholder farmers, including women farmers, whereas direct nutrition and health activities under Feed the Future mostly target women and young children (Landscape Analysis, SPRING, p. 6).”248 This is particularly evident in identified pathways that link agriculture to improved nutrition, where three of the seven pathways are women-specific, and the remaining four are gender-neutral, focusing on agriculture and production, income, and food prices.249 243 Malapit, Hazel Jean, et. Al. “Measuring Progress toward Empowerment – Women’s Empowerment in Agriculture Index: Baseline Report.” 2014. p 1. 244 Based on review of portfolio reviews, program documents, and key informant interview analysis. 245 https://www.feedthefuture.gov/author-name/us-african-development-foundation. 246 Alkire, Sabina, et. Al. “The Women’s Empowerment in Agriculture Index.” IFPRI, 2012. p 1. 247 This is a general pattern. See, for example, “USAID SPRING Presentation: “Agriculture and Nutrition Global Learning and Evidence Exchange (AnN-GLEE),” Bangkok, Thailand. March 19 – 21, 2013. Accessed at: https://www.spring-nutrition.org/sites/default/files/2.5a-hazel.malapit.pdf. See also, Verhart, Noortje, et. Al. “Bringing agriculture and nutrition together using a gender lens.” Accessed at: https://www.kit.nl/gender/wp￾content/uploads/publications/56fe2c7688ee2_Verhart%20et%20al%20(2016)%20Food%20nutrition%20and %20gender%20(KIT%20SNV).pdf. 248 “Landscape Analysis of Activities Across 19 Focus Countries: Linking Agriculture and Nutrition.” June 2014. SPRING. USAID. 249 These include pathways: 5. Women’s own nutritional status due to workload changes, 6. Women’s ability to manage the care, feeding and health of young children given their time constraints, and 7. Women’s socio- 112 A majority of the nutrition-related interventions have been targeted towards women and include nutrition messaging, such as PROCOMIDA (Preventing Malnutrition in Children under 2 Approach) in Guatemala,250 and promotion of homestead gardening, such as Enhanced Homestead Food Production for Improved Food Security and Nutrition Project (E-HFP) in Burkina Faso251 and GAFSP’s support in Uganda to improve nutrition and food security for vulnerable groups through education for women’s groups. 252 While this women-centric approach may be strategic given that global evidence shows that children’s nutrition is linked to the mother’s overall well-being and that homestead gardens lead to improved nutrition,253 this approach is not gender-inclusive in the equitable integration of men and women. The SPRING Landscape Analysis found that there was very little information on men’s participation relating to nutrition, except for a few interventions that focused on the household approach, such as HARVEST in Cambodia and ACCESO in Honduras.254 This disproportionate targeting of nutrition interventions on women was confirmed during field work to the five fieldwork countries. However, interviewees also noted the importance of including men in nutrition interventions, such as behavior change communications, and having men present during nutrition messaging demonstrations at the household. One example of this was SHIKHA in Bangladesh, where nutrition messaging is targeted at men in the markets. Due to the cross-cutting nature of gender, many of the Feed the Future focus countries explicitly stated their intention to pursue gender-inclusive agricultural growth through the inclusion of women in value chain activities.255 One of the most common approaches to this type of inclusion was achieved by intentionally selecting certain crops because they are traditionally tended by women. Examples from fieldwork countries include the selection of the beans value chain in Uganda, the inclusion of the non-agriculture, handicrafts value chain in Guatemala, and OFSP in Malawi and Ghana. It is interesting to note that in an analysis of factors influencing the choice of Feed the Future value chains, the SPRING Landscape Analysis found that gender was the third most common factor considered, after income potential and nutrition impact, but before consideration of government/donor, food security, and number of farmers.256 economic status and ability to influence household decision making and intra-household allocation of food and other resources. 250 USAID. “Synthesis of Evaluation Related to the Feed the Future Learning Agenda.” 2016. p. 33. 251 Idem. p. 10. 252 http://www.gafspfund.org/gafspmapcountry/uga. 253 Learning Agenda and Evaluation Synthesis KDAD 254 SPRING Landscape Analysis, p. 6. 255 Based on review of MYS documents. Examples include Malawi’s MYS 2011-2015, P 7 and Kenya’ MYS 2011 – 2015, p. 12. 256 SPRING Landscape Analysis, p. 8. 113 Some of the driving factors behind the selection of specific value chain crop activities because they are women-centric include addressing gender constraints, as well as working within existing gender domains. For example, in Guatemala, the selection of handicrafts as a value chain activity was a result of the land titling constraints faced by women, which affects their ability to be formal members of farmers’ associations. Similarly, the selection of beans as a value chain crop in Uganda was a direct result of social and economic constraints faced by women to become involved in the well-established coffee and maize value chains dominated by men. While there is evidence to suggest gender-inclusivity in the selection of value chain activities, the SPRING Landscape Analysis notes that the increases in women’s incomes and successes are attributed to different types of interventions, including job creation, homestead production, business vocational training, access to loans, and women’s associations. Specifically, the study cites that women’s groups are one of the rare cases in which value chain interventions with a women’s empowerment focus were shown to result in poverty reduction outcomes. In relation to gender￾inclusive agricultural sector growth, while there is evidence to suggest increases in both men’s and women’s incomes, men’s incomes increased by more.257 One of the three main constraints to women’s empowerment in agriculture identified by the WEAI Baseline Report was their excessive workloads.258 While Feed the Future interventions aim to promote equitable and strategic integration of men and women in agriculture and nutrition programs, the baseline report explains that it is important to consider how a certain intervention may affect an individual’s time allocation given his/her other duties.259 One of the ways in which Feed the Future interventions aim to address this is through the promotion of labor-saving technologies. While many of the Feed the Future procurement documents and the MYSs mention labor-saving technologies, with the implication that women would benefit from these technologies, the SPRING Landscape Analysis found that the activity design documents did not always reflect the same ideas of reducing women’s time poverty, indicating a lack of alignment between theory and implementation.260 Another approach through which Feed the Future interventions have addressed women’s time poverty is by improving access to water resources and irrigation infrastructure as evidenced by the 257 SPRING Landscape Analysis, pp. 52 – 53. 258 Malapit, Hazel Jean, et. Al. “Measuring Progress toward Empowerment – Women’s Empowerment in Agriculture Index: Baseline Report.” 2014. p. 1. 259 Idem. p. 47. 260 SPRING Landscape Analysis, p. 9. 114 impact evaluation conducted in Pakistan.261 Given the fact that nutrition interventions are largely targeted towards women, and the close link between water accessibility and nutrition, several interviewees discussed the need for greater emphasis on WASH activities in Feed the Future programming. One example of increased focus on water accessibility is in Guatemala, where it has been identified as a major constraint and is now being incorporated into Feed the Future interventions.262 Similarly, basic hygiene and sanitation practices are targeted as core activities to reduce illnesses that contribute to malnutrition.263 QUESTION 5 CONCLUSIONS 1. There is no established common understanding of what gender-inclusivity means across the Feed the Future initiative. The equitable and strategic integration of men and women are often interpreted as discrete concepts and approached in multiple ways. Interpretations of individuals, interventions, and countries range from interpreting gender-inclusivity as number of women reached, or women’s empowerment only, to the household approach and involving men. Given that the understanding of – and approaches to – gender-inclusivity are varied across the initiative, in general, there seems to be a fairly basic integration of men and women in agriculture and nutrition programs across all 19 focus countries. The most prominent manifestation of this is through targeting interventions that aim to reach women beneficiaries and produce sex￾disaggregated output data. While Feed the Future interventions may be reaching a large number of women, this does not necessarily indicate that the interventions are designed to address gender inequality and/or women’s empowerment issues through a comprehensive gender lens. 2. There are many gendered interventions across the 19 focus countries that are directly addressing the nuanced constraints faced by both men and women; however, there is room for more. According to the Feed the Future Learning Agenda literature review, there is now an emerging consensus that gendering interventions (addressing gender disparities), rather than targeting interventions (simply including women beneficiaries) is the key to empowering women.264 Additionally, the literature review also shows that non-gendered interventions produce mixed results in relation to increasing women’s income levels. However, almost all gendered interventions 261 Taken from the Learning Agenda and Evaluation Synthesis, KDAD, p. 49. 262 Based on fieldwork key informant interviews. 263 USAID. “Honduras: Nutrition Profile” Accessed at: https://www.usaid.gov/sites/default/files/documents/1864/USAID-Honduras-Profile.pdf 264 USAID. “Feed the Future Learning Agenda Literature Review: Improved Gender Integration and Women’s Empowerment.” 2015. pp. 1-2. 115 reviewed for the study, regardless of their focus (agricultural productivity, nutrition, value chain commercialization, or leadership), were reported as successful.265 3. In the Feed the Future discourse at large, both in literature and interviews, nutrition interventions are generally highlighted as more gender-inclusive than agricultural interventions because of their focus on women. While this approach may be strategic in targeting the most accessible and potentially vulnerable beneficiaries, this does not necessarily translate to gender￾inclusivity if men and other members of the household are not included and involved. Rather than being considered gender-inclusive, or empowering, a heavy emphasis on women – which does not include men – can also be interpreted as disempowering, as cited by several fieldwork interviewees. Particularly within the framework of targeting agricultural interventions largely toward men and nutrition interventions toward women, there is a distinct possibility of reinforcing gender norms at the expense of true gender-inclusivity and women’s empowerment. Similarly, while targeting agricultural activities such as homestead gardening toward women has proven to be successful in involving women in the agricultural sector, it is essential that these types of activities are not seen as solely women’s activities, but rather, as a source of household income.266 4. The literature, as well as evaluations of Feed the Future interventions, show that lack of access to finance, inputs, technology, and markets are significant constraints for women, indicating that these constraints are tied to gender norms.267 The evaluations also showed that when women do have access to and ownership over these resources there were higher success rates in desired outcomes, such as higher adoption rates of new technology with homestead gardens (because they had ownership over that resource), more control over the income women earned from ownership of chickens, and higher success with producer organizations when they were considered business entities rather than self-help groups.268 5. The WEAI baseline report found that excessive workloads are a significant constraint to women’s empowerment. Given the gender norms associated with women’s time allocation, time poverty is an important consideration for Feed the Future interventions in general, as well as women’s involvement in value chain activities. Many of the women-focused value chain crops and 265 Feed the Future Learning Agenda Literature Review: Improved Gender Integration and Women’s Empowerment, June 23, 2015. USAID. pp. 2-3. 266 Evaluation Synthesis, KDAD, p. 51. 267 See both documents for examples listed throughout, USAID. “Feed the Future Learning Agenda Literature Review: Improved Gender Integration and Women’s Empowerment.” June 23, 2015. And USAID. “Synthesis of Evaluations Related to the Feed the Future Learning Agenda.” 2016. 268 Taken from: Synthesis of Evaluations Related to the Feed the Future Learning Agenda.” 2016. 116 activities, such as beans in Uganda, are primarily focused on production and household consumption. However, “given the constraints of time and workload, women’s involvement should be shifted to more profitable segments of the value chain process, not just production.”269 6. While most interviewees in the fieldwork countries were aware of the WEAI and existence of gendered data, including the sex-disaggregated monitoring data and gender-focused impact evaluations, the data is not being systematically used for learning purposes and/or program design. In particular, most interviewees perceived the WEAI as a complex tool but did not have a clear understanding of how to use the tool and data effectively.270 The introduction of the WEAI has increased the scope of incorporating and tracking gender considerations in Feed the Future programming. This is evidenced by the results cited in the WEAI baseline report referenced above that allow for identification of gender-specific constraints and subsequent programming. In addition, while many of the interviewees found the WEAI to be a complex tool, many survey respondents also indicated that the WEAI instructional guide was very useful, indicating increased uptake of the tool in the future. 7. Our analysis of the WEAI concludes that it is a complex tool that can be misleading if only the final index value is considered. The most informative pieces of data produced by the WEAI are embedded in considering and understanding the components and domains that constitute the index. While the WEAI was developed as a monitoring tool, it can also be extremely useful as a diagnostic tool, to identify areas in which both women and men are disempowered and target interventions accordingly. USAID recognized the utility of the WEAI as a diagnostic tool shortly after it was launched, developing the GIF in 2013 to help users assess the data. The WEAI Discussion Paper, as well as the WEAI Bangladesh household survey both show that men and women are disempowered in different areas. Therefore, it is critical to consider and understand the performance of each gender against each of the five domains (decisions about agricultural production, access to and decision making power over productive resources, control over use of income, leadership in the community, and time use). For example, the southwestern Bangladesh pilot reports that the disempowered headcount for women was 61% and for men, 59.8%. These values indicate that men are also highly disempowered. However, the “configuration of men’s deprivations in empowerment is strikingly different from women’s,”271 with lack of leadership in the community and time poverty being significant contributors to men’s disempowerment, whereas lack of control over income and decision-making are significant contributors to women’s 269 SPRING Landscape Analysis, p. 17. 270 Based on key informant interview analysis. 271 Alkire, Sabina, et. Al. “The Women’s Empowerment in Agriculture Index.” IFPRI, 2012. p. 37. 117 disempowerment. Therefore, while the index score provides a useful comparative snapshot, the WEAI’s true utility is in understanding the domains that constitute it. It was also noted that both the WEAI and GIF would be strengthened if they incorporated measures of social norms that could contextualize the understanding of the empowerment scores.272 The index score is a combination of two indices – the five domains index as well as the gender parity index. To understand what the index score is actually saying, it is important to know the magnitude of each of the two components, especially when considering change over time. For example, the Bangladesh pilot values show that 40.2% of women were less empowered, and of that group of women, the empowerment gap between them and the males in their household was 25.2%.273 If both those percentages were reversed and 25.2% of women were less empowered, with a 40.2% empowerment gap, the index score calculated would be the same. However, the programmatic implications of a smaller group of disempowered women with a higher empowerment gap are different from the reverse situation. Over time, it will be important to note to what degree each of those two percentages is changing to understand and target interventions accordingly. QUESTION 5 RECOMMENDATIONS 1. The current gender policy mandates gender inclusivity and outlines key objectives, but does not provide guidance on approaches to achieve those objectives. The Feed the Future guidance on gender included within the Feed the Future Guide should be amended to include a mandate for gendered interventions, not just targeting interventions. This includes developing a Feed the Future gender approach that outlines a set of preferred approaches to promoting gender-inclusive agricultural growth and improved nutritional status of women, while allowing for the context￾specific flexibility that is crucial for successful adoption and implementation. 2. Develop and communicate with all focus country missions a more concrete feedback loop between the mandated gender analysis, the WEAI data and the subsequent Gender Integration Framework, and project and activity design and implementation. As a critical part of this feedback loop, it is important to stress the importance of the timeline for conducting and incorporating the results of a gender assessment. Gender assessments should be done as early as possible to allow for sufficient time for learning and translation into programmatic decisions.274 BFS has developed 272 Based on KII analysis. 273 Alkire, Sabina, et. Al. “The Women’s Empowerment in Agriculture Index.” IFPRI, 2012. p. 39. 274 Women Thrive Worldwide. “Gender in Feed the Future.” http://www.womenthrive.org/sites/default/files/images/agriculture%20genderscan.pdf. p. 4. 118 a non-mandatory training of trainers course at USAID University that addresses many of these issues. Given that this training was recently launched (in April 2016), the extent that this approach fosters operational uptake at missions was unclear at the time of the evaluation. 3. Provide mandatory trainings on the WEAI and the dimensions that constitute it, its utility to the Gender Integration Framework, and how that translates to activity design and implementation. These trainings should be delivered to specific points of contact within the Mission Feed the Future team. It would be most beneficial if these points of contact include both AOR/CORs as well as M&E specialists, both within the mission as well as within IPs. 4. Encourage each Feed the Future team to have a gender point of contact at a minimum, and ideally a designated, if not full-time, gender specialist, with technical proficiency in gender. Although some missions have Mission Gender Specialists, they are not always intimately involved with Feed the Future. Given the complexity of gender inclusivity within the agricultural and nutrition landscape, it would be beneficial to have at least one gender focal person on the Feed the Future team who can ensure gender inclusivity in design, procurement, and implementation phases. Develop a specific illustrative scope of work that lays out the illustrative duties and responsibilities of gender specialists within Feed the Future programming, which would include reviewing design and implementation of gender activities and routine monitoring of partner activities to ensure full utilization of gender expertise and implementation of gender requirements in all activities. Monitoring should be aimed at learning, adaptation, and continuous improvement and not simply accountability and compliance. 5. Ensure that all activities, even those not selected because of potential for gender impact, have gender-sensitive implementation processes in place as well as rolling, gender sensitive M&E to support these implementation processes. Feed the Future should require a careful analysis of value chain crop and activity selection for those crops and activities chosen as gender-inclusive or women’s empowerment crops and activities. This analysis should consider the long-term impacts of the value chain on gender dynamics and include its effect on time allocation of both women and men. Some questions to consider include: Will it reinforce gender constraints? Will it contribute to increased time poverty? Will it aggravate gender disparities in involvement along the more profitable sections of the value chain? Question 6: How have Feed the Future interventions integrated nutrition into value chain activities? Do results differ if nutrition objectives are an integral part of the value chain work? If so, how? 119 Feed the Future’s understanding of the complex underpinnings of malnutrition has evolved substantially from the beginning of the initiative. The integration of nutrition and value chain activities in Feed the Future is complicated by the lack of a consensus understanding of what this concept means at an operational level. Many Feed the Future field staff appear to have retained the earliest versions of the understanding of this concept (value chain actors providing health sector functions), and have not benefitted from the significant body of experience and research now documented in a robust body of utilization-focused literature generated over the last five years. Nutrition integration has been attempted in many ways, with varying levels of appropriateness and success. Agriculture-nutrition integration varies in quality and has evolved over time. Early attempts at agriculture-nutrition integration in value chains reflected a much less refined understanding of the agriculture pathways to improved nutrition and how nutrition can be best influenced by value chains or other agricultural activities.275 Country teams that have evolved to implement a co-located approach, focused on the household as the unit of interest, feel that their programming more effectively addresses nutrition through agriculture and complementary support to the household.276 Programs using this complementary co-located approach seem better positioned to produce nutrition results, based on field visits to five focus countries. Little evidence exists by which to determine whether nutrition results differ if nutrition objectives are a stated objective of the value chain activity or not. Few Feed the Future value chain projects have sufficiently incorporated nutrition measures into monitoring and evaluation. The contribution of non-value chain activities toward nutrition objectives is under-recognized within the initiative. Nutrition programming outside of value chains is not fully captured in global reporting documents, including the work of Food for Peace, Peace Corps, and USDA. QUESTION 6 FINDINGS Nutrition and social and economic development are closely linked. It is estimated that stunted children earn 20% less as adults compared to non-stunted individuals.277 When applied across a population, high rates of stunting can slow economic development at rates that can be measured at the GDP level. For example, a 2012 series supported by the World Food Program entitled “The Cost of Hunger in Africa” estimated that the annual economic cost of undernutrition was equivalent 275 Based on review of MYS’, portfolio reviews, and evaluations of USAID programs. 276 This theme was reflected in Feed the Future Country Portfolio Reviews and key informant interviews in the five fieldwork countries. 277 McGregor S, Cheung YB, Cueto S, Glewwe P, Richter L, Strupp B; International Child Development Steering Group. Developmental potential in the first 5 years for children in developing countries. Lancet. 2007; 369:60– 70. 120 to 5.6% of GDP in Uganda,278 10.2% of GDP in Malawi,279 and 16.5% of GDP in Ethiopia.280 The World Bank’s Repositioning Nutrition Strategy describes hunger as a type of non-income poverty.281 These links between poverty, food security, nutritional status, and economic development underlie the Feed the Future development hypothesis. In line with this evidence, according to the 2010 Feed the Future Guide, Feed the Future aims to spur rural economic development through “coordinat[ed] and integrated agriculture and nutrition investments to maximize impact” on poverty and malnutrition. The 2010 Feed the Future Guide identified several types of activities that would be employed to improve nutrition, including the cultivation, storage, post-harvest processing, and consumption of nutrient-dense crops, improving women’s access to income, and health and nutrition behavioral messaging.282 Coordination with the Global Health Programming to provide nutrition-related health services was also mentioned.283 USAID’s Multi-sectoral Nutrition Strategy (2014 – 2025) demonstrates refined thinking about agriculture and nutrition integration. This strategy broadened the multi-sectoral nature of nutrition to include water and sanitation, education, environment, economic growth, livelihoods, and social protection, in addition to health and agriculture. This 2014 strategy included illustrative activities of value chain and nutrition integration that reflected USAID’s learning on this subject.284 The U.S. 278 The Cost of Hunger in Uganda. 2013. Retrieved on July 4, 2016 from: http://npa.ug/wp￾content/themes/npatheme/documents/Publications/Cost%20of%20Hunger%20report%20summary%20ver sion_Final%20Edition.pdf. 279 The Cost of Hunger in Malawi. May 2015. Retrieved on 4 July 2016 from: http://reliefweb.int/sites/reliefweb.int/files/resources/wfp274603.pdf. 280 The Cost of Hunger in Ethiopia. June 2013. Retrieved on 4 July 2016 from: http://www.unicef.org/ethiopia/FINAL_Ethiopia_-_COHA_Summary_Report_June_20_24pg_72dpi.pdf. 281 Ibid. 282 Pages 13, 14, 25. 283 Pages 13, 25. 284 Illustrative Activities integrating value chain/nutrition included: 1. Invest in improved agricultural techniques (more productive pest-resistant, and drought-resistant varieties) that safely and sustainably increase production and consumption of nutrient-dense foods; 2. Invest in value chains and food security activities that preferentially provide income generation for women; 3. Embed nutritional and hygiene messages in agriculture extension services along value chains and behavior change activities to increase demand for nutrient-dense foods; 4. Promote private sector partnerships that channel inputs, services, and technology to farmers and herders, and enhance food production and marketing systems to increase access to safe and nutritious foods; and 5. Reduce postharvest losses and seasonality of food insecurity to increase food availability. 121 Government Global Nutrition Coordination Plan,285 released in June 2016, represents the first whole-of-government plan to address nutrition, by “identif[ying] concrete opportunities for greater interagency communication and collaboration on human nutrition research and programming.” A review of Feed the Future Country Multi-Year Strategies and annual Portfolio Reviews demonstrates an alignment and fidelity of programming with the activities articulated in the 2010 Feed the Future Guide – that is, broadly: direct interventions, research, capacity building, and policy formulation at national and international levels. From publicly available reports it appears that the majority of the nutrition activities that occur in Feed the Future are implemented by USAID in terms of both money and scale.286 It is difficult to confirm whether this is a result of reporting bias or actual programming differences, as five of 11 USG Agencies involved in Feed the Future have not reported regularly on Feed the Future activities.287 However, the evaluation team’s field visits and document review confirm USAID’s dominance in this area of work. Peace Corps also undertakes nutrition interventions, but works at a much smaller scale.288 Agriculture-Nutrition Integration through Value Chains Building the Evidence Base “Value chain approaches can provide useful frameworks to examine the food system and the potential to achieve improved nutritional outcomes by leveraging market-based systems. However, understanding the links between value chains, the overall business environment in which they operate, and nutrition among targeted populations is complex, involving actors and activities working across agriculture, health and nutrition, and very little evidence exists on the potential or the trade￾offs involved.”289 The evidence base supporting a value chain approach to nutrition was underdeveloped at the beginning of the Feed the Future initiative and remains so today.290 The lack of a common, evidence-based foundation for programming seems to have played an important role in the Feed 285 Accessed at: https://www.usaid.gov/sites/default/files/documents/1864/nutritionCoordinationPlan_web_508.pdf 286 Hanaran, Charles. Global Food Security by the Numbers. September 2015. The Chicago Council on Foreign Affairs. p. 8. 287 Idem, p. 9. 288 Focus country MYS, FY2015 annual portfolio reviews. 289 Gelli, Aulo and Corinna Hawkes, Jason Donovan, Jody Harris, Summer L. Allen, Alan de Brauw, Spencer Henson, Nancy Johnson, James Garrett, and David Ryckembusch. “Value Chains and Nutrition: A Framework to Support the Identification, Design, and Evaluation of Interventions.” (January 30, 2015). IFPRI Discussion Paper 01413. Available at: http://ssrn.com/abstract=2564541 or http://dx.doi.org/10.2139/ssrn.2564541 p. v. 290 Idem, p. 6. 122 the Future experience with value chain-nutrition integration. The first five years of Feed the Future have been characterized by a great deal of experimentation at the field operation level – testing approaches and adjusting them over the course of the activity. The original 2010 Feed the Future Guide made a case for value chains to support nutrition improvement through a food system approach, by reducing prices, improving food safety and hygiene, and increasing accessibility to fortified staples, among others. The operationalization of the guidance, however, has resulted in value chain strategies and activities that – to the extent that they focus on nutrition – focus heavily on improving the nutritional status of the population directly engaged through that value chain, and the immediate population surrounding that value chain that resided within the ZOI.291 The initiative is not monitoring or measuring systems-level outcomes through the FTFMS, as would be expected when activities focus on fortifying staple crops or improving food safety. In 2012-13, USAID supported a process to review the evidence base on agricultural pathways to improved nutrition, which resulted in a technical series intended to inform Feed the Future on how to better integrate nutrition into agriculture programs.292 This technical series covered agriculture-nutrition integration broadly (not value chain￾nutrition integration specifically), and identified the primary pathways through which agriculture can impact nutrition that have become assimilated by Feed the Future (text box 1). It also assessed existing Feed the Future value chains against the pathways (text box 2) and made recommendations for future programming. 291 This perception was articulated by USAID key informant interviews in five field work countries and USAID/BFS. 292 “Understanding and Applying Primary Pathways and Principles.” March 2014. Accessed at: https://www.spring￾nutrition.org/sites/default/files/publications/briefs/spring_understandingpathways_brief_1.pdf Additionally, country-specific landscape analyses were developed for the 19 Feed the Future countries that evaluated their programming through the lens of the pathways identified in the aforementioned document. These were not made publicly available. See also: “Leveraging Agriculture for Nutrition Impact through the Feed the Future Initiative: A Landscape Analysis of Activities Across 19 Focus Countries.” SPRING. June 2014. Text Box 1: Learning and Adapting: Defining Pathways to Improve Nutrition through Agriculture In March 2014, USAID released Understanding and Applying Primary Pathways and Principles, produced by SPRING, a USAID-supported global nutrition program. This document included a conceptual framework outlining three primary pathways through which individual nutritional status is influenced by agricultural livelihoods: 1) food production, 2) agricultural income, and 3) women’s empowerment. 123 USAID has hosted a series of Global Learning and Evidence Exchanges (GLEEs) over the course of Feed the Future. GLEEs that focused specifically on the intersection of agriculture and nutrition (AgN-GLEE) took place in 2012-13 (Uganda, Thailand, Guatemala, and Washington). A nutrition￾specific GLEE took place in 2014 (Cambodia), and a multi-sectoral GLEE series took place in 2016 (Washington, Ghana, Tanzania, and Thailand). The purpose of these regional conferences was to disseminate new findings emerging from Feed the Future countries and global research to program implementers, government officials, and USAID Mission personnel. They were structured primarily to allow country-level staff to share at a peer￾to-peer level. The content of these regional conferences was incredibly rich and dense with new information, and all content presented was made available to the public on the Agrilinks,293 SPRING,294 and FANTA websites. USAID’s Nutrition Innovation Lab has held similar evidence-sharing events in Washington and Nepal, but they were not focused on USAID Mission personnel as the primary audience.295 Despite a rigorous effort to document and disseminate the substantial amount of technical learning that is coming out of Feed the Future, a clear theme emerged from document reviews and key informant interviews in five fieldwork countries – that uptake of this information by country teams to inform activity design or strengthen existing Feed the Future activities is low. Key informants in Washington recognize this low uptake and all but one key informant independently suggested that it reflected a general inattention to nutrition 293 https://agrilinks.org/events 294 https://www.spring-nutrition.org/events 295 http://www.nutritioninnovationlab.org/news-events/ Text Box 2: Learning and Adapting: Analyzing FTF Value Chains through the Nutrition Lens In June 2014, USAID released a landscape analysis of agriculture-nutrition integration across the 19 Feed the Future focus countries. All Feed the Future value chains were assessed for their nutrition sensitivity. For those that were classified as nutrition-sensitive, they were categorized into one of seven more specific agriculture-nutrition pathways: 1. Own production food consumption 2. Income food purchase 3. Income healthcare purchase 4. Food prices food purchase 5. Women’s time use care capacity 6. Women’s workload maternal energy use 7. Women’s control of income resource allocation All Feed the Future focus countries had value chains that explicitly or implicitly linked the value chain support to nutrition outcomes through the “production to own consumption” pathways and the “income to food purchase” pathways. Fourteen of the 19 countries had value chains that supported “nutrition through women’s income.” The remaining four pathways were much less frequently used in FTF activity design. 124 programming by field-based personnel in focus countries. The remaining key informant felt that the low uptake was more likely to be related to the program cycle, and suggested that USAID would see more uptake of this technical information as missions designed the next phase of their activities. Key informants from USAID/BFS and all five of the country teams engaged during this evaluation's fieldwork indicated that there is a widely-held perception among staff across focus countries that Feed the Future’s mandate to integrate agriculture and nutrition through value chains means that IPs should be compelled to perform basic health sector functions of health and hygiene promotion, or operate other social service activities alongside their agribusiness-focused activities. Thirty-five percent of IPs interviewed by the evaluation team cite the provision of nutrition-focused behavior change communication by value chain actors as their mechanism for value chain-nutrition integration. This orientation reflects little to none of the learning disseminated in the processes described above. When asked what was needed to ensure greater use of new learning in mission-level decisions, Washington-based key informants described a need for very high levels of in-person technical assistance to be provided to missions. Given the degree of resistance and/or skepticism noted by the evaluation team in some interviews, however, this may only address part of the problem. To some degree, a culture change is necessary whereby value chain actors with an economic growth paradigm come to appreciate the importance of nutrition within Feed the Future. USAID/BFS only has four nutrition specialists on staff keeping up with the rapidly growing evidence base for agriculture-nutrition integration. Two of these staff are focused on research, and as such are not as actively engaged in non-research mission program support. SPRING and the Nutrition Innovation Lab also support missions, but do not report that they heavily support procurement￾sensitive project design processes. Application of Value Chain-Nutrition Integration in Feed the Future Countries The evaluation team’s thematic analysis of qualitative interviews in five fieldwork countries revealed that all IPs, with the exception of those working on policy, reported that they were directed to pursue integration of nutrition into agricultural VC activities in order to pursue the Feed the Future goal of reducing stunting. Given the weak evidence base, as explained above, as well as the fact that many of these IPs lacked nutrition-specific technical expertise, the models of “integration” that emerged from this directive were diverse and in many cases opportunistic. For example, 40% of IPs interviewed by the evaluation team cited that increased income (the “income pathway” described in text box 2) was their model for nutrition integration. This pathway tended to be favored by IPs lacking nutrition capacities and/or working on upper linkages in a VC, for reasons such as those expressed by an IP from Bangladesh: 125 If you only talk about producing nutritious crops, that's not going to give them any income. No surplus, no income, even no money to buy seed for the next crop because, what they are producing, they are consuming… our home garden projects are… not a sustainable model. – USAID/Bangladesh Forty percent of IPs interviewed also identified nutrition integration through the selection of a nutrient-rich crop (the “production → own consumption pathway” described in text box 2). For the few IPs that were promoting nutrient-rich crops through value chain facilitation, they cited the connection to nutrition in terms like this IP from Bangladesh: And lentil sells at $1,000 a ton, mung bean is selling at $1,000 dollars a ton, and mustard is selling at about $700 – $800 dollars a ton [while] rice is selling at $200 a ton. So these are things which really are very valuable. If they’re passing through the household they also may eat them. – Bangladesh IP However, in most cases these nutrient-rich crops were typically not being promoted through value chain facilitation. Two-thirds of IPs who identified using this type of integration pathway were promoting intercropping of nutrition-dense crops alongside a less-nutritious value chain crop, or home gardens to boost home consumption. According to USAID/BFS personnel, Feed the Future does not consider this type of complementary activity to be value chain-nutrition integration. This represents one of many examples uncovered by the evaluation team of the lack of a common understanding across Feed the Future about what constitutes value chain-nutrition integration. While not as common as the income and production-consumption pathways, nutrition-sensitive agricultural extension was viewed as a more logical, acceptable form of nutrition integration into value chain work. The passages below illustrate how respondents discussed nutrition sensitive￾extension: In the agriculture sector, we are training our [extension workers] on nutrition. So what we are doing here is that maybe we are providing four hours of training every day. Three hours we provide on agricultural activities. One hour we provide them with training on nutrition. What they've said is that they find it very useful. So when they are doing the agricultural training, then after, they speak to nutrition. People get interested and then they go back to agricultural training.” – Bangladesh IP 126 Approximately 20% of IPs interviewed described gender targeting as their approach to nutrition integration into value chain activities. These IPs focused on crops that were dominated by women, and assumed that the income would be used to improve the nutritional status of the family. While these qualitative findings represent only a subset of the Feed the Future focus countries, the trends in the adoption of specific agriculture-nutrition pathways mirror those found in the SPRING landscape analysis from 2012-13 (see text box 2). Challenges in Value Chain-Nutrition Integration in Feed the Future Focus Countries As a part of the learning process of the first five years of Feed the Future, field-based personnel have noted significant challenges to value chain-nutrition integration. Forty-two percent of IPs (n=24) across all fieldwork countries – and especially those from Ghana – noted that it was challenging to meaningfully pursue nutrition integration into a value chain. This was especially true for IPs with a heavy upper chain or value chain productivity focus. That's one of the downsides. Sometimes I think it's like, "Oh, that's the nutrition activity. They'll take care of it." Then the other maybe more value chain- or market-focused activity won't pay as much attention to those indicators because [the nutrition-focused IP] is going to handle nutrition statistics. – USAID/Uganda Well, actually, we're not [integrating nutrition]…I mean, we made a poster on nutrition jointly with ag income or aquaculture for incomes and nutrition. So small fish and vegetables for lactating mothers, children under five. Give this to retailers. Give this to partners. But I felt like it was just a checking of the box because we're not working with the poorest of the poor. We're not working with nutrition. With the grantees, we do integrate some nutrition awareness because they are working with female farmers who come and maybe have a homestead garden and may have questions about it. But this is not our focus, so don't put pressure on me to make it my focus. – Bangladesh IP I think [nutrition integration] has been a bit of a forced effort. I know several other Chiefs of Party who are selling farm machinery. And the Mission’s coming to them and say[ing], what are you doing about nutrition? And they’re looking at them like, are you serious? We’re doing farm equipment.” – another Bangladesh IP In projects that actively pursued heavy nutrition integration, their efforts were not always acknowledged to be aligned with the Feed the Future mandate, as described by this IP in Ghana: 127 I gave an overview [at the Feed the Future partners meeting] and going through all the different technical areas, we work in agriculture, livelihoods, WASH, nutrition, governance. And there [were] several very pro-ag people in there. And I was literally ridiculed in front of the whole room, like the whole project was derided. [They told me] this is supposed to be Feed the Future and you've got WASH and you've got nutrition.” – Ghana IP The other major challenge articulated by IPs interviewed in the five fieldwork countries was that the integration pathways that predominate in the Feed the Future portfolio rely on indirect relationships that are not guaranteed, and therefore are not very reliable as a means to improve nutrition. Nearly 40% (n=21) of IPs interviewed by the evaluation team, and several other non-IP stakeholders across the fieldwork countries, said that the underlying assumptions made about the income pathway approach to integration make it an inadequate means to pursue improved nutrition outcomes, as illustrated by the following quotations: [The poverty objective] is being handled by a lot of the big agri projects that we have. But for the stunting one, this assumption that with food security, the poverty level will decrease, and this will lead to nutrition security. But it doesn't happen that way. So how can we have the same balance of projects for food security that we do with poverty, and have those that will deal with nutrition security that will deal with the stunting? – Ghana IP Feed the Future will reach its indicator of stunting and wasting and underweight in Northern Ghana, but… [because of] the Health Population and Nutrition Office projects… they’re doing WASH and nutrition… The flaw is… the crops that the ag guys have chosen... rice and maize have zero impact on – or negative impact on – nutrition.” – Ghana IP Similar skepticism existed with the production → own consumption pathway: In the case of soy and groundnuts, I can say that they sold it, and this year it’s going to be similar. The prices are so high. Why would they keep it? – Malawi IP But, you know, they just designed another project here that’s a livestock project. And, again, it’s focused on [addressing] stunting. We’re going to drop it by 10 or 15 percentage points through milk, increased consumption of milk. Which from a nutrition standpoint makes no 128 sense. Sorry. But it absolutely makes no sense. Culturally it doesn’t make sense for what people do in terms of milk consumption here. – Bangladesh IP The argument that maize and beans address nutrition, and coffee addresses income, may not be as straightforward as that. Maize and beans can also be cash crops. People grow maize and sell, people grow beans and sell, [to] get income. – USAID Uganda Agriculture-nutrition programming outside of value chains While value chains are meant to form the backbone of USAID’s engagement in ZOIs, there are myriad complementary and support activities extending beyond the value chain engagement that support nutrition and other household level needs. The 2013 Landscape Analysis296 conducted by USAID’s SPRING project outlined the three main approaches to linking agriculture and nutrition programming used by Feed the Future: integration, co-location, and retrofitting297 (see text box 3). The evaluation team found strong nutrition components in non-value chain activities supporting Feed the Future through co-location. For example, Food for Peace Multi-Year Assistance Programs (MYAPs) with nutrition components were operating under Feed the Future in Guatemala,298 Haiti,299 Mozambique,300 Malawi,301 Bangladesh,302 and Mali,303 although not always within the ZOI. USAID Global Health/Nutrition programming provided support to the ZOI populations in Bangladesh, Guatemala, and Malawi, among others.304 Peace Corps volunteers support nutrition efforts where and when they have a presence, and there are six countries where the Peace Corps has entered into direct relationships with USAID under Feed the Future.305 Peace Corps reports that in 2015 296 “Leveraging Agriculture for Nutrition Impact through the Feed the Future Initiative: A Landscape Analysis of Activities Across 19 Focus Countries.” June 2014. SPRING. 297 “Leveraging Agriculture for Nutrition Impact Through the Feed the Future Initiative: A Landscape Analysis of Activities Across 19 Focus Countries.” June 2014. SPRING. p. 4. 298 “Title II food security program, PROMASA, Save the Children, MYAP 2006-2011: Endline report.” USAID Guatemala (2011). 299 “Final Evaluation Report. The Haiti Title II Multi-year Assistance Programs (MYAP).” (2014). 300 “Food for the Hungry—Mozambique: P.L. 480 Title II Multi-year Assistance Program Final Evaluation.” (2013). 301 “Wellness and Agriculture for Life Advancement (WALA): Mid-term Evaluation Report.” USAID Malawi (2012). 302 “Save the Children Bangladesh Mid-term Review of NoboJibon Multi-year Assistance Program.” (2013). 303 “Mid-term evaluation of the Timbuktu food security initiative.” USAID Mali (2011). 304 “Leveraging Agriculture for Nutrition Impact through the Feed the Future Initiative: A Landscape Analysis of Activities Across 19 Focus Countries.” June 2014. SPRING. 305 https://www.peacecorps.gov/about/global-initiatives/feed-future/ 129 volunteers trained 38,000 caregivers in child health and nutrition, reaching 11,000 children under the age of five through nutrition programming.306 Additionally, USDA has McGovern-Dole School Feeding programs in a number of focus countries including Bangladesh,307 Cambodia,308 Honduras,309 Mali,310 Nepal,311 and Senegal.312 Additionally, significant nutrition-sensitive programming by USAID Economic Growth offices in the field is co-located or programmed in complementarity with value chain activities in most countries. Across the five fieldwork countries visited by the evaluation team, there are 10 examples of IPs that seem to be set up to deliver nutrition-focused technical assistance or health and nutrition promotion interventions to other IPs – most of whom are primarily focused on value chains. This practice was observed in all but one fieldwork country, and was particularly pronounced in Guatemala, where intra-IP coordination has been explicitly emphasized by the Guatemala Mission. In addition to the 10 IPs within the interview sample who provide this sort of technical assistance, local partners interviewed for this evaluation also discussed using other local resources to enhance their nutrition orientation, meaning that the practice could be even more widespread than the data suggests. The quotes below represent how some respondents discussed the role that co-location can play in promoting nutrition interventions, especially where a fully-integrated approach is not possible: 306 https://www.peacecorps.gov/about/global-initiatives/feed-future/ 307 http://www.fas.usda.gov/newsroom/us-wheat-helps-feed-children-bangladesh 308 http://www.fas.usda.gov/regions/cambodia 309 From BFS-provided data. 310 http://www.crs.org/media-center/news-release/food-for-education-in-mali 311 http://www.fas.usda.gov/regions/south-and-central-asia/nepal 312 From BFS-provided data. Text Box 3: Feed the Future uses three main approaches to integration of agriculture and nutrition activities. Integrated and/or flagship activities: A leading activity spearheads a Mission’s Feed the Future work. Such activities may provide both agriculture and nutrition services through an integrated delivery platform; they may also focus on providing services that are either primarily agricultural or primarily nutrition-related. Examples of this approach are found in Honduras and Cambodia. Co-locating activities: This approach involves placing multiple activities – each usually focusing on a single intervention type (e.g., health and nutrition, agriculture, or economic growth) – in one geographic area. The level of overlap in areas and target population among activities is different. In Bangladesh and Guatemala, all activities work within the same units in the ZOI. Activities in Uganda and Zambia have only partial overlap in geographic area within the ZOI. Retrofitting ongoing activities: Activities following this approach modified activities that were designed and implemented before Feed the Future’s inception by incorporating new or strengthened nutrition interventions, indicators, or geographic targeting to respond to the new Feed the Future mandates. 130 Cross-learning and engagement between the value chain activity and the [nutrition-focused] activity have helped each other in terms of getting out benefits to the farmer. And then the [integrated nutrition-agriculture IP] is kind of the epitome of a well-designed integrated agricultural nutrition activity. I mean I think it’s taken the best international practice from the design. They’re learning and adapting throughout the activity. – USAID/Uganda The only thing that maybe we're not very strong on, and the Feed the Future model helps to strengthen, is the linkage with nutrition because maybe we are also very influenced by [another international NGO]. The model is also very business-oriented. So we are quite strong in business, but not as strong in nutrition. Within the project, we actually were working with Ncoma Hospital on the nutrition bit, which is good because then we're getting expertise. – Malawi IP Homestead gardens and ruminant production, for example, is present in nearly every Feed the Future country program, and is described by Mission personnel and IPs as examples of value chain￾nutrition integration. However, some people working in Feed the Future do not consider these efforts to be value chain integration.313 Table 3 below represents the diversity of nutritious crops and frequency by which they are promoted by Feed the Future IPs interviewed by the evaluation team in the five fieldwork countries. Those with an asterisk were promoted through value chain facilitation for their income potential. The remainder of the commodities were promoted through complementary programming focused on household food and nutrition security. Table 2: Diversity of Nutritious Crops Promoted and Number of IPs Promoting Each Nutritious Crops Promoted OFSP Other Legumes* Soya Groundnuts Fruits/ Vegetables* Bio￾fortifie d Maize Animals* Inter￾cropping or Dual￾cropping Home Gardens Bangladesh (n=9) 2* 1 2* 2 1 Ghana (n=11) 2 4 1 2 1 2 2 Guatemala (n=12) 2 2 1 4 1 4 Malawi (n=16) 3 3* 2 2 1 1 313 Stated by a Technical Specialist and Senior Leadership of USAID/BFS, KII. 131 Nutritious Crops Promoted OFSP Other Legumes* Soya Groundnuts Fruits/ Vegetables* Bio￾fortifie d Maize Animals* Inter￾cropping or Dual￾cropping Home Gardens Uganda (n=9) 1 1 1 2* Implementing Partners (n=57) 6 8 6 4 8 2 10 4 5 *Indicates that at least one of the IPs counted worked with this commodity primarily for its income￾generating potential, rather than for its potential to increase household diet diversity. Although IPs provided this information as evidence of value chain-nutrition integration, this type of complementary programming at the homestead level is considered by many in USAID314 to fall short of meeting the criteria for integration because the activities have no bearing on the success or failure of the value chain crop. This represents another of many examples uncovered by the evaluation team of the lack of a common understanding across Feed the Future about what constitutes value chain-nutrition integration. Through experimentation, USAID missions have realized that the best practice targeting strategies for these approaches for poverty alleviation, commercial farming, and nutrition require different approaches, making household-level integration difficult. USAID missions and IPs report that a fundamental principle of the Feed the Future development hypothesis is that agriculture and nutrition interventions should be delivered to the same households.315 The evaluation team found a strong theme among key informants316 and within electronic survey results that ZOI populations should not be considered to be monolithic, and the ability to target subpopulations to provide them with the right combination of support to prosper is an essential approach in Feed the Future programming. This represents an important evolution of thinking from the beginning of the initiative. Emerging Connections between Agriculture and Nutrition Research emerging from USAID’s Nutrition Innovation Lab is uncovering new connections between agriculture and nutrition that were not considered at the outset of the Feed the Future initiative. As mentioned above, the literature coming out of the initiative is extensive, and fully describing it is outside the scope of this evaluation. However, a number of highly relevant examples stand out. 314 Stated by a Technical Specialist and Senior Leadership of USAID/BFS, KII. 315 Based on review of multi-year strategies and key informant interview thematic analysis. 316 Ibid. 132 Research showing the very significant impact of aflatoxin on nutritional status and its near ubiquity317 underscores the population level impact that food safety interventions can have on general population health and nutritional status. Findings from the same group on the relationship between cattle ownership, malaria, and nutrition318 also demonstrate very strong linkages between value chain work and health/nutrition. Four of 57 IPs interviewed about agriculture-nutrition integration cited linkages in areas such as post-harvest handling and storage of commodities to improve food safety. Nine of 57 cited the integration of WASH activities in value chain activities. These types of interventions to address determinants of nutrition status that are more distal to individual nutrition but more proximal to agriculture appear to be under-emphasized in current Feed the Future programming. QUESTION 6 CONCLUSIONS 1. Feed the Future’s understanding of the complex underpinnings of malnutrition has evolved substantially from the beginning of the initiative, as is evident in the development of the first USAID Multi-sectoral Nutrition Strategy and the first USG Global Nutrition Coordination Plan. These USG documents represent agricultural interventions under Feed the Future as a part of a larger and more complex USG response to malnutrition. 2. The integration of nutrition and value chain activities in Feed the Future is complicated by the lack of a consensus understanding of what this concept means at an operational level. If defined in the most narrow, literal way, “integrated approaches” would only include a very small proportion of the total work being done to improve nutritional status. Additionally, many Feed the Future field staff appear to have retained the earliest versions of the understanding of this concept (value chain actors providing health sector functions), and have not benefitted from the significant body of experience and research, now documented in a robust body of utilization-focused literature, that has been generated over the last five years. As a result, there are still many field-level skeptics of the importance of nutrition integration into agriculture or private sector work. In the absence of this common foundation for programming, nutrition integration has been attempted in many ways, with varying levels of appropriateness and success. 3. Agriculture-nutrition integration in Feed the Future programming varies in quality and has evolved over time. The first five years of Feed the Future have been a significant learning period 317 “Feed the Future Innovation Lab for Collaborative Research on Nutrition – Africa Annual Report FY 2014, October 1st, 2014- September 30th, 2015.” Retrieved on 27 June 2016 from: https://agrilinks.org/sites/default/files/resource/files/Nutrition%20Innovation%20Lab%20Africa%20Year%2 05%20Annual%20Report%20(2-2016).pdf 318 Ibid. 133 for the USG – and in particular, USAID – in how to design and implement multi-sectoral nutrition programming. This learning curve is reflected in the quality of programming during this period. Early attempts at agriculture-nutrition integration in value chains reflect a much less refined understanding of the agriculture pathways to improved nutrition and how nutrition can be best influenced by value chains or other agricultural activities.319 Initial MYSs show a heavy reliance on income generation and consumption of production to address nutrition problems.320 Country teams that have evolved to implement a co-located approach, focused on the homestead as the unit of interest, feel that their programming more effectively addresses nutrition through agriculture and complementary support to the household.321 The evaluation team agrees that programs using this complementary co-located approach seem better positioned to produce nutrition results, based on field visits to five focus countries. While the evaluation team was not able to identify any evaluations planned to empirically evaluate the effectiveness of a co-located approach as compared to a strict value chain approach, knowledgeable key informants interviewed at the USAID mission level felt that these more holistic approaches will have better results. 4. Little evidence exists by which to determine whether nutrition results differ if nutrition objectives are a stated objective of the value chain activity. Few Feed the Future value chain projects have sufficiently incorporated nutrition measures into their monitoring and evaluation activities. The evidence is not yet available to determine whether or not nutrition integration in value chain activities improves nutrition results. Additionally, many value chain activities are supported by complementary – other livelihood and health support – activities, as described above. This will make it more challenging to distinguish the effects of any one component from the effect of the entire package on a household or community (or a zone of influence). 5. The contribution of non-value chain activities toward nutrition objectives is under￾recognized within the initiative. Although detailed budget data were not available to the evaluation team, Congressional appropriation level budget data322 suggest that the majority of Feed the Future funds are spent on activities other than value chains. Nutrition programming outside of value chains is not fully captured in global reporting documents, including the work of Food for Peace, Peace Corps, USDA, and USAID Global Health. To further demonstrate this point, the 319 Based on review of MYSs, portfolio reviews, and evaluations of USAID programs. 320 See text box “Learning and Adapting: Analyzing FTF Value Chains through the Nutrition Lens” above. 321 This theme was reflected in Feed the Future country portfolio reviews and KIIs in the five fieldwork countries. 322 Lawson, M., Schnepf, R., & Cook, N. (2016). The Obama Administration’s Feed the Future Initiative. Washington, DC: Congressional Research Service. Retrieved from https://www.fas.org/sgp/crs/row/R44216.pdf 134 evaluation questions provided to this evaluation team did not include a request to evaluate community or homestead-based nutrition activities under Feed the Future that fall outside of the value chain approach. This represents a missed opportunity by the initiative to convey the complete story about Feed the Future investments and comprehensive nutrition programming. 6. Co-located programming allows Feed the Future to target beneficiary groups for maximum impact within a larger target population. It allows independent projects working together in complementarity to target beneficiaries with greater precision than would a single, integrated award. It also permits differentiated approaches for different sub-groups within the population, for example the very poor, commercially viable farmer, and non-farming populations. When well￾coordinated, the result of this layered approach is more efficient and possibly more effective programming. 7. Nutrition-sensitive agriculture is necessary but not sufficient to improve nutrition at a population level. Nutrition-sensitive agricultural interventions – including but not limited to value chains – can improve access to and availability of nutritious foods, but are less effective at addressing the utilization of those foods by the household, and cannot address adverse health conditions that affect the bio-utilization of nutrient-dense foods. QUESTION 6 RECOMMENDATIONS 1. Develop, design, and implement guidance for appropriate approaches to integrated agriculture and nutrition programming for the poorest of the poor (safety nets), commercially viable farmers, and general population levels. This guidance could be rolled out as a part of the USG annual Operational Plan (OP) process. The Global Health Bureau of USAID has included a “technical considerations” annex to OP guidance that could serve as a good model for Feed the Future. It lays out program design expectations based on best practice, and lists a priori what would constitute a “significant issue” during the open comment period. The purpose of this guidance would be to reshape agriculture-nutrition integration programming to be better aligned with global best practice and emerging evidence, and accelerate the diffusion of innovation to country teams. The evaluation team recommends that any guidance should reinforce the notion to operational level teams that nutrition-focused programming in Feed the Future extends beyond value chain activities, and includes activities that target sub-populations that are not suited to value chain inclusion. Guidance should include the importance of having community or household level interventions to support agriculture-nutrition pathways that are not addressed by the value chain activity. 2. Reconsider the directive to target value chain and nutrition interventions to the same households. Drawing on the explicit recognition in this report that Feed the Future populations are 135 diverse, use implementation guidance to provide flexibility to operating units to design programs that first and foremost respond to household needs. Designs should allow for the intervention package to change depending on the needs within sub-populations, communities, or households. 3. Differentiate value chains for nutrition (VCN) from value chains for economic growth. Define the problem statement for these two types of interventions differently. The problem statement for VCNs should be a nutrition problem, as opposed to an economic problem. The design of the value chain intervention should then be focused on solving the nutrition problem statement. IFPRI’s 2015 paper “Value Chains and Nutrition: A Framework to Support the Identification, Design, and Evaluation of Interventions” provides a good starting point for rethinking value chain design in this way.323 4. Elevate the profile of Feed the Future activities outside of the value chain interventions. The inclusion of these substantial non-value chain investments demonstrates a stronger, more comprehensive nutrition response than does the nutrition-sensitive value chain interventions alone. 5. Activities that are designed to have an impact on nutrition should measure nutrition results and intermediate outcomes as a part of good performance management. Feed the Future nutrition indicators should be revised to include intermediate outcomes that are more appropriate to the contribution that agricultural programming can make on nutrition. However, it is not appropriate for all value chain activities to be held to nutrition results. Some value chain interventions are designed specifically for economic development and aim at addressing different problems, and should not attempt to make linkages to nutrition outcomes that were not intended – nor should be – in their design. 6. Revitalize activities to promote a safe and nutritious domestic food supply in future programming in line with the USG Global Nutrition Coordination Plan. Consider using new language to differentiate direct, beneficiary-level nutrition interventions, from interventions promoting a safe and nutritious domestic food supply. Develop mechanisms to capture the impact of value-added value chain commodities on indirect beneficiaries to fully understand Feed the 323 Gelli, Aulo, Corinna Hawkes, Jason Donovan, Jody Harris, Summer L. Allen, Alan de Brauw, Spencer Henson, Nancy Johnson, James Garrett, and David Ryckembusch. “Value Chains and Nutrition: A Framework to Support the Identification, Design, and Evaluation of Interventions.” IFPRI Discussion Paper 01413. January 30, 2015. Available at: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2564541 136 Future’s impact on nutrition. Develop indicators to capture and understand how Feed the Future is contributing to improvements in food safety and nutritional quality at a system level. Question 7: What is the effectiveness of the initiative’s focus on country ownership? How well has Feed the Future fostered country ownership of the Feed the Future program in Focus Countries to support sustainability of outcomes. All 19 focus country strategies work within geographic areas and on specific value chains, and with identified target populations, that are guided by and consistent with local governments’ stated priorities. Feed the Future regional strategies align directly with priority issues and agendas established by regional bodies, and regional missions consult with country missions to ensure that these regional strategies align with locally defined government priorities. The extent to which Feed the Future has supported effective broad-based participation, an important element of ownership, is difficult to measure based on available indicators. There are robust and recurring approaches to wide participation, particularly involving civil society and the private sector; in combination with various reporting products, vigorous processes of consultation with local government and regional bodies are strong and occurring regularly. In spite of an overall positive assessment with regards to participation and consultation, it appears that there are few activities specifically and directly aimed at raising the voice of the poor, with the exception of community consultations occurring most notably under Food for Peace. A general lack of indicators to track effectiveness of these approaches makes it difficult to assess whether or not this significant effort being put forth is resulting in increased effectiveness. QUESTION 7 FINDINGS Country ownership is “the practice of partner countries taking the lead in designing and implementing clearly defined development strategies and managing their own development processes.”324 Consistent with the Paris Declaration,325 the Accra Agenda for Action,326 and the Rome Principles.327 A central tenet of the Feed the Future initiative is to focus resources to facilitate and strengthen local government ownership, increase impact, and enhance sustainability. Although there are standard principles and agreed-upon measures, that citation notes that the local context will influence the approach, form, and intensity of engagement as well as expectations for results. Feed the Future views local ownership as essential to sustainability: 324 USAID ADS Chapter 220 – Partial Revision, 03/26/2012. 325 OECD (2008). “The Paris Declaration on Aid Effectiveness: Five Principles for Smart Aid.” 326 “Accra Agenda for Action” (2008). 3rd High-Level Forum on Aid Effectiveness. 327 “Declaration of the World Summit on Food Security” (2009). World Summit on Food Security, Rome, FAO. 137 “Country ownership provides the most effective means to coordinate development efforts and achieve sustainability in reducing hunger and poverty.”328 USAID intended that each focus country’s MYS be reflective of and responsive to focus country priorities as a first step to fostering country ownership. IPs are subsequently expected to be sensitive to and promote country ownership in their projects and activities; this is seen as a key factor toward promoting sustainability of program impacts and outcomes. According to both USAID Mission and IP interviewees, “country ownership” does not only include efforts to foster a sense of ownership among focus country governments, but also among stakeholders within focus country civil societies, and country ownership in this broader sense has been supported through a variety of different approaches. Some of these approaches are intended primarily for purposes of active engagement; others are designed to bolster the capacity of key stakeholders, in order to facilitate sustainability. The various categories of country ownership-facilitating models emerging from fieldwork include: ● Alignment of program strategies to the focus country’s CIP and/or other priorities ● Maintaining consistent communication with government stakeholders throughout the lifetime of the Feed the Future initiative or project ● Engaging government stakeholders in significant consultation to obtain country-owned input into the design or strategic plan for Feed the Future projects and activities ● Provision of technical assistance to facilitate government or civil society decision-making ● Co-implementation of program activities with government or non-governmental counterparts ● Institutional strengthening through focused, customized trainings and other approaches designed to bolster key governmental and non-governmental institutions seen to be necessary to sustaining/attaining Feed the Future goals ● Direct funding/budget support to focus country governments ● In-kind academic training for government staff and others ● Community level engagement to foster local buy-in and ensure sustainability of IP activities The evaluation team analyzed the various ways in which USAID and IPs in the five fieldwork countries noted that they are facilitating country ownership, both within and outside of government. Non-government-focused modes are presented first because they were emphasized by a wider variety of implementing partners. The patterns in the data suggest that most projects 328 https://www.feedthefuture.gov 138 (nearly 60%) engage local non-government (civil society and private sector) entities in the implementation of project activities (in some cases, especially in Uganda and Guatemala, some local entities are engaged in co-implementation of Feed the Future activities as the lead￾implementer (prime) on the project.) In addition, about a third of IPs (including all five of the FFP IPs) said that they incorporate various methods of local community engagement to generate buy￾in and support for activities. IPs receive encouragement to work with and empower local entities through USAID policy requirements, such as USAID Forward and Mission-level CDCS plans, many of which were rolled out during the design or early implementation stages of Feed the Future. Qualitative data analysis from the five fieldwork countries show that while IPs also emphasized country ownership through activity co-implementation (mostly by engaging government extension workers in agricultural and nutrition extension activities), USAID missions tended to emphasize ownership through alignment with national policies and priorities (including, but not limited to, a focus country’s CIP). The nine policy-focused IPs participating in fieldwork interviews comprise the bulk of government-focused models of country-ownership promotion, including consultation, provision of technical assistance, and institutional strengthening. Although it cannot be concluded that regular communication/meetings with government stakeholders are not being maintained, it is notably under-emphasized by many IPs (not including those involved in co-implementation). Where some missions (Ghana and Uganda) strongly emphasized multi-faceted government￾focused engagement (even including direct funding), others (Guatemala) placed little emphasis on government engagement, citing political turmoil and corruption as reasons to refrain from intensive engagement. For Feed the Future, focusing resources is achieved through: 1) the selection of a limited set of focus countries, 2) concentration of effort in a locally defined geographic zone of influence, 3) coordination among donors at all levels, and 4) leveraging resources from both the donor community and the private sector. 329 To achieve this aim, Feed the Future commits to working and collaborating at all levels: global, regional, national, and local, and engages with public, private, and civil society stakeholders.330 At the time the initiative was launched, the MCC was the designated USG lead on country ownership.331 Their work was oriented through market-based solutions to food insecurity and the 329 “FTF Guide” (2010). Washington, DC, USAID, BFS. 330 “FTF Guide” (2010). Washington, DC, USAID, BFS. 331 “FTF Guide” (2010). Washington, DC, USAID, BFS; “MCC Principles into Practice: Country Ownership” www.mcc.gov 139 promotion of economic growth to reduce poverty.332 Feed the Future was meant to expand and reinforce this effort, in part, by aligning with local government priorities, strategies and programs, particularly those related to agriculture development. A primary coordination mechanism in Africa is the Comprehensive Africa Agricultural Development Plans, and Feed the Future has adopted this process as a primary mechanism for ensuring country ownership in African countries pursuing the CAADP approach.333 Globally, and including in Africa, Feed the Future planning and implementation is guided by the Country Investment Plans.334 These principles, processes, and resolutions illustrate a strong commitment to move development into the hands of sovereign and responsible governments (for more discussion of this, see Question #8, which discusses CAADP in detail, and Question #11 regarding policy reform). In the case of MCC, commitments are made directly with host country governments through the provision of grants, either as compacts or threshold programs, under the Millennium Challenge Account. Partner countries must meet rigorous eligibility requirements and, if selected, participate actively in the identification of potential project areas and the design of project activities. They are required to set up a local Millennium Challenge Account (MCA) accountable entity to manage and oversee all aspects of implementation. Monitoring of funds is rigorous and transparent, and often occurs through independent fiscal agents.335 As noted in previous discussion, IPs are supporting the government to facilitate co-implementation of Feed the Future activities, and at various levels – from parliament and within key counterpart ministries, all the way down to entities operating at field level – and this co-implementation is widely perceived to be building critical, country-owned capacity: I think that should actually be one of the key successes that we have had …because in the past when we engaged the parliamentary committee, it was just to lobby them. But with [this project], our view of engagement with the committee changed because we now started looking at them as our partners and they started looking at us as partners, such that we don't only engage them to lobby for something, but we are also interested to build their capacities. So as we've been building [our] capacity, we've also been interested in building [the committee’s] own capacity so that they can also effectively engage. – Malawi IP 332 www.mcc.gov 333 “FTF Guide” (2010). Washington, DC, USAID, BFS. 334 Ibid. 335 https://www.mcc.gov/about 140 [We] implement through district assemblies. Like, our staff wouldn't go out and do direct implementation. So we train people at the district level so they can advise and continue to operate this scheme... After about a year, the groups don't need…consistent support, so they can sustain themselves. – Ghana IP. It’s our duty also to improve where the structures are not in line, we revamp, we train. For most districts, if the members have stayed for over 10 years, they were requesting us to support them in capacity building, so we intend to incorporate that into our programming so that we are building sustainability. Because in the end, at the end of the project we're going to hand over most of these interventions to government sectors as well as the village development committees. – Malawi IP Despite efforts to engage and align with focus country governments, USAID mission and IP respondents noted that political dynamics and realities can obstruct ownership of Feed the Future. Shifting political dynamics, corruption, and general lack of interest or political will can be hard to predict or to influence. Despite USAID’s prominence in the development space, these dynamics can undermine the influence of Feed the Future activities and actors. In addition to sometimes-challenging political dynamics, focus countries (especially Ghana, Guatemala, Malawi, and Uganda) often face critical and ongoing capacity limitations in the agriculture sector, which results from chronic underfunding of national extension and research services across the focus countries. In many cases, it seems that the capacity constraints outstrip the current donor funding obligated for extension system support: [There] is the low interest of the government, if I can say so, in agriculture…In terms of the resources that they allocate to the ag sector, it's not significant maybe now, if I could say. You only have a few extensionists, for example. There is not much effort by the government to support agriculture. – Ghana IP They are too few. But we work with the Ministry of Food and Agriculture extensionists whenever we can, so we bring them to our table fields and they assist us, but it's not enough to cover all of our beneficiaries, and they stopped recruiting since a while now. – Ghana IP 141 We train extension agents but none of them last more than a year or so. They are removed or leave due to lack of or delayed salary payments. – USG Agency in Guatemala The high-level approach of engaging directly with government is consistent with grass-roots development standards regarding country ownership, as expressed in Interaction’s “Country Ownership: Moving from Rhetoric to Action,” which states that:336 “Although country ownership may look very different depending on the country context, an accumulating body of evidence points to the following universal elements that effectively support country ownership: 1. An enabling environment 2. Transparency and accountability 3. Consultation 4. Participation 5. Capacity building.” USAID Forward, launched in 2010, is an initiative to reform and modernize USAID for 21st century development. The original agenda had seven areas of reform:337 Talent Management; Rebuilding Policy Capacity; Strengthening Monitoring and Evaluation; Rebuilding Budget Management; Local Solutions; Science and Technology; and Innovation. These seven areas were subsequently consolidated into three “mutually reinforcing” themes:338 ● Deliver results on a meaningful scale through a strengthened USAID ● Promote sustainable development through high-impact partnerships and local solutions ● Identify and scale up innovative, breakthrough solutions to intractable development challenges Feed the Future reflects the USAID Forward reform agenda in its core principles of sustainability and country ownership, including an emphasis on local solutions. In a Feed the Future guidance 336 “Country Ownership: Moving from Rhetoric to Action” InterAction. 2011. 337 USAID Forward Strategic Overview Susan Reichle. Retrieved 28 June 2016 from http://www.macf.com/images/stories/pdf/ASHA/2015/Plenary_Sessions/USAID-ASHA_2015_AC_- _USAID_Forward.pdf. 338 Ibid. 142 document on local capacity development,339 Feed the Future proposes achieving USAID Forward goals through: “1. [Building] local ministries and public institutions to effectively serve the needs within the agriculture and nutrition sectors, including implementing USAID projects,” and “2. [Leaving] behind viable local public entities, private firms, and CSOs to continue servicing the needs within the country’s agriculture and nutrition sectors, including implementing USAID projects.” Activities undertaken to improve institutional capacity are described in greater detail under Question #4 in this report, as well as under Question #11. Implementation of Feed the Future coincided with USAID initiatives to also increase and build local, non-government capacity to implement Mission-funded projects. This trend has not gone unnoticed and has impressed local stakeholders in Guatemala, Uganda, and in Bangladesh, for example, as illustrated below: This is the first time, on the Feed the Future initiative award, that [a big] national NGO has gotten the contract as the prime implementer. This is one of the three initiatives of the global Feed the Future and of the important Feed the Future activities for… food security. – Bangladesh IP What I must say … is that for the first time in many USAID projects, that we have had our direct engagement as a local organization with USAID. No go-between, if I can put it that way, and moving forward, what we see is this kind of ownership, our local ownership, which is already inbuilt. What I normally share with my colleagues is that if you are looking for sustainability, it doesn't come at the end of the project. It comes right at the beginning. – Uganda IP USAID missions and IPs alike acknowledged the great need for and the importance of capacity building to facilitate country ownership and more importantly, sustainability, whether through a mechanism like co-implementation, or through more discrete means like training, technical assistance, and institutional strengthening. It's not only the consultation process, I would say. The fact that a mission or USAID itself had this USAID Forward, that allowance called Local Solutions, that allowed us to directly engage 339 VOLUME 5: USAID Forward/IPR and Feed the Future March 2014. 2014. Page 1. Retrieved 28 June 2016 from: https://www.feedthefuture.gov/sites/default/files/resource/files/ftf_guidanceseries_vol5_localcapacitydevel opment_march2014.pdf. 143 with government. That has been the driving force because we use some of the funding mechanism. That's what we did. It allowed us to really work closely with them and even in better ways than we used to do. – USAID/Ghana USAID Mission personnel and some staff of IPs also noted the importance of engaging the private sector to step in if and when governments are not able to provide services – and these are another set of key stakeholders to consider in assessing country ownership. Private sector engagement was acknowledged as important across fieldwork countries, with particular emphasis in Ghana and Uganda. Although the private sector is recognized by USAID as an important actor for building and sustaining country ownership, respondents from Bangladesh, Malawi, and Guatemala, in particular, noted that the current state and structure of the private sector presents problems that prevent it from being well-positioned to support and sustain Feed the Future goals. Problems cited by respondents include too little regulation, no quality control (Bangladesh), the dominance of a few firms (Bangladesh, Malawi, Guatemala), and a high level of risk aversion that requires public-sector incentivization (Malawi and Ghana). USAID’s primary measure of local solutions is mission program funds implemented through local systems, tracked on an annual basis.340 USAID as an Agency established a goal to increase direct obligations from a baseline of 9.7% (FY 2010) to 30% by FY 2015.341 As of FY 2014 (the most recent published data), the Agency had reached 16.9% direct obligation to local institutions.342 FY 2014 data were not available in disaggregated format, but the Agency’s 2013 USAID Forward Progress Report343 shows only 1.5% of Agriculture funding obligated to local institutions. Key informant interviews with mission personnel in the five fieldwork countries frequently involved discussions on the challenges of local solutions reform. While the local solutions reform agenda has added levels of complexity to USAID award administration due to the need for risk assessment vetting and accounting oversight, Agency personnel view the local solutions agenda as important, appropriate, and worth the additional effort needed for implementation. 340 https://www.usaid.gov/usaidforward/usaid-forward-draft/frequently-asked-questions 341 2015 USAID Forward Results: Strengthening Local Capacity Data Table. USAID. Retrieved 29 June 2016 from https://www.usaid.gov/usaidforward 342 USAID Strengthen Local Systems Data Set. Retrieved 28 June 2016 at https://www.usaid.gov/data/dataset/d45f05be-50de-4840-83a1- 8ebb8e93f25fhttps://www.usaid.gov/data/dataset/d45f05be-50de-4840-83a1-8ebb8e93f25f 343 USAID Forward Progress Report 2013. Page 22. Retrieved 28 June 2016 from: https://www.usaid.gov/sites/default/files/documents/1868/2013-usaid-forward￾report.pdfhttps://www.usaid.gov/sites/default/files/documents/1868/2013-usaid-forward-report.pdf 144 Providing support that is deemed relevant and thus useful to host country institutions is another vehicle for ensuring ownership. Feed the Future funds nutrition policy research and advocacy work to generate useful evidence to inform policy formulation. FANTA II has helped Guatemala’s high￾level food security and nutrition coordination body identify key underlying factors of malnutrition, priority themes, and geographic areas, trained numerous government and IP development agents, and established a web-based accountability platform to monitor resource allocation and key outcome indicators.344 IFPRI is currently conducting a similar research into nutrition and food policy in Bangladesh.345 Additionally, the U.S. Army Corps of Engineers, under an interagency agreement with USAID and the U.S. Embassy carried out a technical review of rainwater harvesting systems in Honduras in order to write technical construction guides for building water reservoirs for agricultural purposes. This was done in close coordination with the Honduras Ministry of Agriculture346 . Guatemala is an excellent example of how Feed the Future can support layered poverty reduction and environmental initiatives as well as work in a manner consistent with locally identified priorities. The Mission recently added a new program, Nexus Locales, to address capacity limitations at various levels of government and civil society. The program works with municipalities and a wide range of governmental and non-governmental bodies that are responsible for or advocate for agriculture, water, watershed management, indigenous rights, etc. As described in interviews, the program works with 344 Based on interviews with government and IP KIs in Guatemala. IP KIs working directly with the poor noted that the training and orientation was helpful. They could apply the guidance and support consistent messaging at different levels and across IPs. It was noted, however, that some of the nutrition messages were not practical for some extreme and remote households. 345 Field interview data. 346 From BFS provided interagency spreadsheet data. Providing support to local non-profit organizations: In Guatemala, two local organizations – the Guatemalan Exporters Association and Guatemalan National Coffee Association – are now USAID’s signature Feed the Future partners. The organizations are directly managing grants that cumulatively represent over $60 million in USAID contributions, cost-sharing, and funds leveraged from the private sector, and will benefit more than 32,000 households. This five-year project will improve household access to food by expanding and diversifying rural income, and contribute to improving the nutritional status of families – especially children under five years old facing chronic malnutrition. Source: USAID Forward Progress Report 2013. Retrieved 28 June 2016 from https://www.usaid.gov/sites/default/files/documents/1868/2013- usaid-forward￾report.pdfhttps://www.usaid.gov/sites/default/files/documents/1868/ 2013-usaid-forward-report.pdf 145 these representative bodies that advocate at all levels of government. The Nexus Locales project focuses on municipal action plans, transparency, and public accountability. Feed the Future supports the Development Foundation of Guatemala – FUNDESA, a local foundation dedicated to conservation and responsible management of multipurpose natural resources, which has implemented creative whole watershed approaches to improve sustainable and equitable access to water, forest products, and land for smallholder agriculture. USDA works with the Honduran and Guatemalan governments to support compliance with key U.S. policies affecting the agriculture sector and export of agricultural products to the United States, specifically the Central American Free Trade Agreement (CAFTA), supported through the Office of US Trade Representative, and the Food Safety Modernization Act.347 Coffee and horticulture value chains are central to both countries’ growth and poverty alleviation strategies.348 In Ghana, the Resiliency in Northern Ghana (RING) project works through local government structures in seven northern districts.349 The main objective is to provide integrated food security assistance to vulnerable households. RING is an add-on project, established after the design of the MYS and in recognition that certain poor households and communities were unable to participate effectively in value chain activities. This is another illustration of how Feed the Future has been able to reflect, learn, and evolve its program focus and content to better address the extreme poor. Engagement and ownership is not consistent across all countries. In Guatemala, civil servant turnover is frequent and extraordinarily high at every level; when this occurs in key offices responsible for implementation of elements of the Feed the Future country strategy, it indicates a certain lack of commitment – and thus ownership – on the part of the government. Turnover is also a problem in Bangladesh, but an agreement was reached that led to a commitment to not rotate personnel through key offices engaged with Feed 347http://www.caftadr.net/papascope_files/PAPA%20PROGRAM%20DESCRIPTION%20Annex%20A%20%20 April%2018-07%20REC%20FINAL.pdf 348 KII with USDA/Guatemala and evidence found on www.usda.gov under Food for Progress. 349http://www.ghana.gov.gh/index.php/media-center/regional-news/1706-districts-sign-ring￾implementation-project-n-r 350https://www.usaid.gov/sites/default/files/documents/1869/USAID%20Forward%20Fact%20Sheet%20Apri l%202016.pdf Providing support to local private sector companies: By investing seed money in a local startup in Kenya that is using new technology to convert waste into bio-gas, Feed the Future is not only supporting local development solutions, but also investing in an organization that stands to create jobs and become self-sustaining through its profits in the long run.350 146 the Future.351 Non-governmental bodies are, in many countries where staff rotation is an issue, the main source of institutional memory, a weakness in the infrastructure for government-led implementation as well as an indicator of less than committed ownership. In Ghana and Malawi, host country officials (at National Agricultural Research Service (NARS) and Ministry of Agriculture (MOA), respectively) noted that Feed the Future Technical Advisors to the MOA, and IPs working within agriculture, are viewed as another “project” of USAID, rather than as assistance provided to support the government in achieving its own development priorities. QUESTION 7 CONCLUSIONS 1. USAID’s primary measure of local solutions is Mission program funds implemented through local systems, tracked on an annual basis.352 The FY 2015 Agency goal is for 30% of these funds to be obligated to local institutions. FY 2014 data were not available in disaggregated format, but the Agency’s 2013 USAID Forward Progress Report353 shows only 1.5% of Agriculture funding obligated to local institutions, indicating that Feed the Future is not significantly contributing to this goal. However, these indicator statistics do not adequately reflect the level of effort that has gone into this reform agenda from USAID and other Feed the Future agencies. As documented in a Government Accountability Office (GAO) report on this subject, USAID has spent a great deal of effort establishing mechanisms for evaluating the capacity of local institutions to manage USG resources, developing new procurement mechanisms to manage and mitigate risk and ensure accountability, and provide technical assistance to organizations to help address weaknesses prior to receipt of USG resources.354 These actions, while reported during field interviews as requiring significant time and resources, are not tracked or measured by current metrics.355 Additionally, there is no current emphasis placed on the quality or sustainability of the work performed, outside of general performance management processes.356 Therefore, it is difficult to understand whether or not the significant effort being put forth to fund local institutions – as well as interventions that do not involve direct funding – is resulting in any increase in effectiveness or sustainability of results.357 351 Key informant interview. 352 https://www.usaid.gov/usaidforward/usaid-forward-draft/frequently-asked-questions 353 USAID Forward Progress Report 2013. Page 22. Retrieved 28 June 2016 from: https://www.usaid.gov/sites/default/files/documents/1868/2013-usaid-forward￾report.pdfhttps://www.usaid.gov/sites/default/files/documents/1868/2013-usaid-forward-report.pdf. 354 Foreign Aid: USAID Has Increased Funding to Partner Country Organizations but Could Better Track Progress. 2014. GAO. Retrieved on 6 July 2016 from http://www.gao.gov/assets/670/662596.pdf. 355 Ibid. 356 Ibid. 357 Ibid. 147 2. The evaluation team conducted a comprehensive review of MYSs and strategic reviews and analysis of these that determined that all 19 focus country strategies work within geographic areas with target populations and on specific value chains that are consistent with and guided by local governments’ stated priorities (See Question# 3 for more discussion on this). Field interviews confirm very strong alignment with host country government priorities. In several cases, governments had shifted their priorities to geographic areas and populations that had been underrepresented or marginalized by previous development efforts, as specifically mentioned by interviewees in Bangladesh. In accordance with these changes, Feed the Future relocated and refocused USG development activities (Bangladesh, Ghana, Guatemala, Honduras, Senegal).358 Regional strategies align directly with locally defined priority issues, which include agendas set by regional bodies such as Common Market for Eastern and Southern Africa (COMESA), Southern African Development Community (SADC), Economic Community of West African States (ECOWAS), and Council of Ministers of Economy of Central America (COMECO). The regional missions consult with the affiliated country missions to ensure that regional strategies align with MYSs, and thus local government priorities. 3. An element of ownership that has evolved under Feed the Future is that of participation. The extent to which Feed the Future has supported effective broad-based participation beyond partner government officials, and large commercial actors or CSOs, is difficult to measure based on the available indicator evidence, but field interviews describe robust and recurring approaches to wide participation, particularly from civil society and the private sector. It appears, however, based upon available evidence, that there are few initiatives or program interventions that are specifically and directly aimed at raising the voice of the poor, with the exception of community consultations (most notably under FFP). The specific purpose and approach of these are not entirely clear, specifically as they may relate to raising the voice of the poor – particularly the ultra-poor and landless – who are typically socially as well as economically marginalized. “Community consultation,” therefore, runs the risk of perpetuating the marginalization of this group. To date, it appears little attention has been given to the facilitation of drawing out views, priorities, suggestions, and feedback directly from this significant sub-category of the beneficiary population. QUESTION 7 RECOMMENDATIONS 1. Feed the Future should develop tools and metrics that are capable of capturing the significant results being achieved that relate to country ownership but that are not currently captured by indicators within the FTFMS. A number of metrics currently available could be useful, 358 MYS Strategy change memos and portfolio reviews. 148 including the numbers of civil society or private sector actors who receive USG assistance – although it should be noted that counting outputs does not necessarily reflect the complexities of “ownership,” and any measure of these outputs would need to be bolstered by robust qualitative data providing explanation of how engagement resulted in actual ownership. To a degree, the ultimate measure of the success of this country ownership approach would be sustainable improvements in the overall capacity of systems actors. This recommendation – to develop sensitive tools and metrics to capture systems changes – is discussed in recommendations provided for Question #3 and Question #4. 2. Ensuring local capacity to provide strong, technically capable facilitation of consultative processes is also indicated as a means to strengthen country ownership. This recommendation is discussed in greater detail under Question #4. 3. ICTs are increasingly introduced as mechanisms to reach the poor with messages and market information. Farmer organizations could provide a mechanism to articulate a collective voice for poor and marginal producers. The International Maize and Wheat Improvement Center (CIMMYT),359 partly funded by Feed the Future, has been developing ICT-based information systems and tools to enhance collaborative planning and implementation at all levels.360 They have initiated these activities in Bangladesh, Nepal, and Guatemala.361 There is potential to tailor these tools to facilitate the inclusion of poor smallholders and rural households as well as create useful tools for farm management tailored to the poor.362 More detailed discussion on several of the key ownership principles discussed above can be found in other sections of this report: i.e., capacity building (Question #4), multilateral institutions (Question #8), private sector engagement (Question #10), and policy enabling environment (Question #11). Question 8: What has been Feed the Future’s contribution to influencing and leveraging multi-lateral institutions and initiatives, specifically: i) the G8 and the New Alliance; ii) the Comprehensive Africa Agriculture Development Program 359 CIMMYT has similar activities to those funded by Feed the Future in Nepal, Honduras, Haiti and several African countries. 360 Interview with Buenamilpa project in Guatemala and CIMMYT in Mexico. 361 Interview with CIMMYT in Mexico. 362 Interview with Buenamilpa project in Guatemala and CIMMYT in Mexico. 149 (CAADP); iii) the Global Agriculture and Food Security Program (GAFSP); and, iv) the Consultative Group on International Agricultural Research (CGIAR)? G8 and New Alliance A series of international agreements and declarations, championed and strongly supported by the United States, highlighted global food security issues and mobilized increased commitments to address food security and nutrition needs. U.S. leadership helped establish the New Alliance, which created a framework for leveraging private sector and national government support. Feed the Future includes mechanisms and actions that provide support to the New Alliance. Self-reported results for the New Alliance indicate significant progress in improving the agricultural policy environment and promoting increased private investment. There is mixed data on how effective the New Alliance and Grow Africa have been in achieving their objectives, and whether those objectives have an overall positive or negative impact on smallholder farmers. CAADP The Comprehensive Africa Agricultural Development Programme is an African-owned process with strong regional engagement that aligns well with Feed the Future development priorities and has been heavily supported by Feed the Future at multiple levels. CAADP has mixed success across countries as a result of domestic political economic issues, but it is solidifying principles of regional multilateralism and gaining momentum over time. CAADP is now well established within African agricultural policy processes, at continental, regional, and (although unevenly) at national level. GAFSP Feed the Future was successful from 2010 to 2014 in leveraging other donors to contribute to the Global Agriculture and Food Security Program, and in providing leadership. CGIAR Feed the Future, through BFS, has exerted strong leadership in the Consultative Group on International Agricultural Research during a period of major reforms. G8 and the New Alliance for Food Security and Nutrition A global financial recession caused world food prices for staple commodities to increase dramatically in 2007–2008, which exacerbated food insecurity for more than 870 million people in 150 the developing world.363 Facing a situation of increasing global instability, the G-8364 Summit in July 2009, at L’Aquila, Italy, initiated actions that led to an international donor commitment of $22 billion for worldwide food security. Out of these efforts came the Global Agriculture and Food Security Program, established in 2010.365 The U.S. commitment to this effort led to the establishment of Feed the Future, which provided a framework for USG efforts to leverage international donor support, encourage national government commitment, and increase private investment to promote agricultural development.366 Within the G8, the United States led a process in 2012 that brought African countries, private corporations, and the donor community together to promote improved agriculture sector policy, increased investment, and greater agriculture productivity. Under this partnership, the New Alliance for Food Security and Nutrition seeks to improve smallholder farmer incomes and increase food security.367 G8 AND NEW ALLIANCE – FINDINGS G8 contributions for food security and nutrition increased from 2010 – 2014. The OECD reported an increase of 17% in ODA368 from 2011 to 2014.369 This increase coincided with a positive trend for bilateral ODA for agriculture, forestry, and fisheries in 14 out of 19 Feed the Future focus countries.370 The White House used its leadership under its presidency of the G8 in 2012 to help create a mechanism – the New Alliance – to meet a reported shortfall in agriculture investment in Africa.371 The U.S. State Department provided assistance in developing the G8 New Alliance policy framework.372 The New Alliance brings together the strategic interests and capacities of different 363 “Assessing the Feed the Future Initiative” – The Center for Global Development. March 2016. 364 The G8 is comprised of the following countries: United States, Canada, France, Germany, Italy, Japan, Russia and the United Kingdom. In 2014, Russia was removed from the group for the annexation of Crimea. 365 USAID, Bureau for Food Security. (2015). 2015 Achieving Impact: Leadership and Partnership to Feed the Future. Washington, DC. 366 The Obama Administration’s Feed the Future Initiative. Lawson, M. L.; Schnepf, R.; Cook, N. Congressional Research Service 7-5700 R44216. 367 FACT Sheet: The New Alliance for Food Security and Nutrition. White House press release. June 18 2013. 368 Official Development Assistance (ODA) is defined as government aid designed to promote the economic development and welfare of developing countries. 369 OECD (2016), "Data warehouse," OECD Stat (database). DOI: http://dx.doi.org/10.1787/data-00900-en (Accessed on July 3, 2016) 370 OECD (2016), Geographical Distribution of Financial Flows to Developing Countries 2016: Disbursements, Commitments, Country Indicators, OECD Publishing, Paris. DOI: http://dx.doi.org/10.1787/fin_flows_dev￾2016-en-fr 371 FACT Sheet: The New Alliance for Food Security and Nutrition. White House press release. June 18 2013. 372 BFS provided interagency spreadsheet data 151 stakeholders in the effort to “facilitate inclusive, agriculture-led growth in Africa and address key constraints to private investment and increased smallholder productivity and market access.”373 The strategic focus of the G8 was on helping food-insecure and low-income countries increase agriculture productivity and smallholder farmer income, through donor fulfillment of L’Aquila financial pledges in support of country plans, promotion of an improved enabling environment, and increased private investment.374 The OECD reported that even with the increased contributions, there is a 50% shortfall in funding needed to meet food security and nutrition needs.375 USAID indicated that increased national government investment and private capital from local and international private sector is needed to provide the capital, innovation, and access to markets required to spur agricultural development.376 As the mechanism for delivering on the USG’s L’Aquila commitments, all Feed the Future investments are contributions to the New Alliance in New Alliance countries. Partners within the New Alliance, including Feed the Future, are committed to specific policy reforms and investments in order to accelerate the implementation of CAADP country food strategies. Feed the Future has played a critical role in the New Alliance, acting as the donor lead from headquarters in 2012 and 2014, as well as the donor lead in several countries, working directly with partner governments to facilitate accountability of the commitments in the Cooperation Frameworks. In addition, Feed the Future includes specific mechanisms to support the New Alliance through support of “enabling actions” such as the New Alliance Information and Communications Technology Extension Challenge Fund and the Global Action Network for Agricultural Index Insurance, technical assistance to the African Union to take on coordination, mission engagement in focus countries, and regional mission programs.377 Furthermore, in an effort to achieve country-driven development, Feed the Future has provided support to transition full leadership of the New Alliance 373 New Alliance for Food Security and Nutrition: Progress Report. (2014). https://new￾alliance.org/sites/default/files/resources/New%20Alliance%20Progress%20Report%202014_Print%20Friend ly%20Version.pdf. 374 USAID, Bureau for Food Security. (2015). 2015 Achieving Impact: Leadership and Partnership to Feed the Future. Washington, DC. 375 OECD (2015), “The Grow Africa partnership,” in Development Co-operation Report 2015: Making Partnerships Effective Coalitions for Action, OECD Publishing, Paris. DOI: http://dx.doi.org/10.1787/dcr-2015- 22-en 376 USAID, Bureau for Food Security. (2015). 2015 Achieving Impact: Leadership and Partnership to Feed the Future. Washington, DC. 377 Summary of Feed the Future Focus Country Annual Portfolio Review, 2012 – 2015 and Online Survey of Feed the Future field staff. 152 to the African Union. This technical support is critical to the initiative’s ability to support policy change and increased private investment, in coordination with Grow Africa. Governments across the New Alliance countries have demonstrated commitment to policy reforms in agriculture, with 33% of policy commitments reported as completed and 59% making progress as of June 2015.378 While the New Alliance has demonstrated significant progress expanding partnership from three to 10 African countries, catalyzing private sector investment in African agriculture, challenges remain to create enabling environments for the private sector. In its 2015 Progress Report, the New Alliance addressed constraints faced by companies within the partnership in operations and investments, including limited access to finances, lack of availability of agricultural inputs, infrastructure problems (i.e. electricity and roads in rural areas), lack of market access, unskilled workers, and lengthy bureaucratic processes.379 G8 AND NEW ALLIANCE – CONCLUSIONS 1. A series of international agreements and declarations, championed and strongly supported by the United States, highlighted the global recognition of the issue of food insecurity and increased commitments to address food security and nutrition needs. This process to address food insecurity also indicated that ODA contributions were insufficient to cover the development needs of low-income, food deficit countries.380 As of July 2015, 93% of funds committed at L’Aquila had been disbursed.381 2. U.S. leadership helped establish the New Alliance, which created a framework for leveraging private sector and national government support that can cover the shortfall for countries in Africa. Feed the Future includes mechanisms and actions that support the New Alliance.382 378 New Alliance for Food Security and Nutrition and Grow Africa: Joint Annual Progress Report; 2014 – 2015. https://www.new￾alliance.org/sites/default/files/resources/New%20Alliance%20Progress%20Report%202014-2015_0.pdf 379 New Alliance for Food Security and Nutrition and Grow Africa: Joint Annual Progress Report; 2014 – 2015. https://www.new￾alliance.org/sites/default/files/resources/New%20Alliance%20Progress%20Report%202014-2015_0.pdf. 380 The Obama Administration’s Feed the Future Initiative.” Lawson, M. L.; Schnepf, R.; Cook, N. Congressional Research Service 7-5700 R44216. 381 “Tracking the L'Aquila Food Security Initiative Pledge and Related Funding. July 2015 Update: Commitments and Disbursements.” Available here: http://www.bmz.de/g7/includes/Downloadarchiv/2015_G8_tracking_table_AFSI.pdf. 382The New Alliance for Food Security and Nutrition and Grow Africa Joint Annual Progress Report. 2014 – 2015. Retrieved June 30, 2016, from https://new-alliance.org/resource/2014-2015-new-alliance-progress￾report. 153 3. Self-reported results for the New Alliance indicate significant progress in improving the agricultural policy environment and promoting increased private investment in the sector. However, there are mixed signals on how effective the New Alliance and Grow Africa are in achieving their objectives and whether those objectives have an overall positive or negative impact on smallholder farmers.383 G8 AND NEW ALLIANCE – RECOMMENDATIONS 1. USAID should execute an independent evaluation of how effective the New Alliance and Grow Africa initiatives have been in improving the local enabling environment, leveraging private sector resources, and promoting increased investment by national governments in food security and nutrition for Feed the Future focus countries. 2. As part of the assessment process of the impact of Feed the Future in promoting both the New Alliance and Grow Africa, there should be a country-by-country evaluation of the overall impact on the agriculture and food security sector. Focus should be on the impact of national government policy changes to promote private investment and the results of private investments in terms of promoting smallholder farmer incomes and improved nutrition. Comprehensive Africa Agriculture Development Programme (CAADP) CAADP FINDINGS The Comprehensive Africa Agriculture Development Programme provides a framework for coordinated programming from local to continental levels based on evidence-based analysis and learning, stakeholder consultation and negotiation, and compacts between country governments, regional actors, and development partners.384 CAADP aims to eliminate hunger and reduce poverty through agriculture through two target approaches: 1) Achieve 6% annual growth in agricultural productivity by 2015, and 2) Increase the allocation of national budgets to at least 10% in the agricultural sector. In addition to these two targets, CAADP also has four stated “pillars,” which include: extending the area under sustainable land and water management; improving rural 383 “The New Alliance for Food Security and Nutrition: A Turning Point?” Brache, I. EU Report. 2015. See also: De Schutter, Olivier. “The New Alliance for Food Security in Africa.” Directorate-General for External Policies, Policy Department, of the European Commission. Available at: http://www.europarl.europa.eu/RegData/etudes/STUD/2015/535010/EXPO_STU(2015)535010_EN.pdf. 384 Irene Annor-Frempong. “Human and Institutional Capacity Development Strategy Review: African Stakeholder Perspectives.” Feed the Future. May 15, 2015. p. 9. 154 infrastructure and trade-related capacities for market access; increasing food supply and reducing hunger; and agricultural research, technology dissemination, and adoption.385 In 2003, the African Union (AU) Summit adopted CAADP386 as an integral part of the New Partnership for Africa’s Development (NEPAD).387 At the second meeting of the CAADP Partnership Platform, the AU called for the establishment of a multi-donor trust fund (MDTF) to be administered by the World Bank; the AU and key donors felt that the World Bank had the capacity to manage the fund and provide technical leadership.388 The 2007 – 2008 food price crisis added momentum to CAADP,389 and delegates at the 2009 G8 summit at L’Aquila pledged more than $20 billion in aid390 to support “sustainable agriculture development”; furthermore, the L’Aquila Declaration stated that the aid would be used to “support the implementation of country and regional agricultural strategies and plans through country-led coordination processes.”391 Increased U.S. support to agricultural development in Africa as a result of the launch of the Feed the Future program was then linked to implementation of CAADP.392 NEPAD is an AU strategic policy framework for pan-African socio-economic development, “spearheaded by African leaders,”393 to address critical challenges facing the continent. NEPAD provides an organizational framework for African countries to control their development agendas, to work more closely together, and to cooperate more effectively with international partners; it also facilitates and coordinates the development of continent-wide programs and projects, 385 Kimenyi, Mwangi S, Brandon Routman, and Andrew Westbury. “CAADP at 10: Progress Toward Agricultural Prosperity.” Africa Growth Initiative at Brookings. June 12, 2016. https://www.brookings.edu/wp￾content/uploads/2016/06/12-CAADP.pdf. 386https://www.ifpri.org/blog/caadp%E2%80%99s-10-year-report-card-evaluation-premier-program￾investment-agriculture 387 http://www.nepad-caadp.net/about-us 388 Gerrard, Chris, Gem Argwings-Kodhek, Ali Marouani, and Godfrey Mudimu. “Independent Evaluation of CAADP Multi-Donor Trust Fund (MDTF).” Volume 1: Main Report. April 9, 2016. p. 1. 389 Poulton, Colin, Kassahun Berhanu, Blessings Chinsinga, Brian Cooksey, Frederick Golooba-Mutebi, and Augustin Loada “The Comprehensive Africa Agriculture Development Programme (CAADP): Political Incentives, Value Added and Ways Forward. Working Paper 077.” Future Agricultures. February 2014. pp. 2- 3. 390 Marc J. Cohen. “U.S. Aid to Agriculture: Shifting Focus from Production to Sustainable Food Security.” The Penn State Journal of Law & International Affairs, Vol. 3, Issue 2.2015. Available at: http://elibrary.law.psu.edu/jlia/vol3/iss2/8. 391 Poulton, Colin, Kassahun Berhanu, Blessings Chinsinga, Brian Cooksey, Frederick Golooba-Mutebi, and Augustin Loada “The Comprehensive Africa Agriculture Development Programme (CAADP): Political Incentives, Value Added and Ways Forward. Working Paper 077.” Future Agricultures. February 2014. p.3. 392 Idem. 393 http://www.nepad-caadp.net/about-us 155 mobilizes resources, and engages the global community, regional economic communities (RECs), and member states in the implementation of these projects.394 While “inclusive development” is stated as the purpose of CAADP, there has been significant criticism of its overall effectiveness, with various NGO analyses asserting that the process falls short.395 Nonetheless, according to IFPRI, “Investment in the African agricultural sector was largely in a state of decline until the early 2000s, followed by Africa’s “decade of growth.”396 A major source of this growth, according to IFPRI, is CAADP. 397 Multi-Donor Trust Fund The World Bank administers several multi-donor trust funds that were established to provide financial support directly to CAADP, including GAFSP and a CAADP-specific MDTF.398 Both the relevance and the effectiveness of the MDTF have been highly dependent upon the relevance and effectiveness of CAADP.399 The MDTF and associated Child Trust Fund (CTF) projects can point to a number of achievements.400 As of December 2015, $65.2 million have been contributed to the MDTF by six donors: the United States, the European Union, the United Kingdom, the Netherlands, Ireland, and France.401 The World Bank established CTF grants for the AU Commission, the NEPAD Planning and Coordinating Agency (NPCA), and four RECs to accelerate regional and country-level CAADP processes, and for 394 Ibid. 395 See, for example: Marc J. Cohen. “U.S. Aid to Agriculture: Shifting Focus from Production to Sustainable Food Security.” The Penn State Journal of Law & International Affairs, Vol. 3, Issue 2.2015. Available at: http://elibrary.law.psu.edu/jlia/vol3/iss2/8; see also: Crola, Jean-Denis and Saya Saulière. “Aid Coordination and Alignment- Myth or Reality? The Case of the ECOWAS Regional Agriculture Policy.” Oxfam. 8 September 2011; and: Muñoz, Eric and Emmanuel Tumusiime. “Promise and Potential: Inclusive, Sustainable Development for Small-Scale Food Producers through the Feed the Future Initiative.” Oxfam America. Undated; see also: Howell, John, and Mark Curtis “Fair shares: is CAADP working?” ActionAid. May 2013. 396https://www.ifpri.org/blog/caadp%E2%80%99s-10-year-report-card-evaluation-premier-program￾investment-agriculture 397 Idem. 398“Building Institutional Capacity to Increase Food Security in Africa.” Food Security Collaborative. April 2012. p. 2. 399Gerrard, Chris, Gem Argwings-Kodhek, Ali Marouani, and Godfrey Mudimu. “Independent Evaluation of CAADP Multi-Donor Trust Fund (MDTF).” Volume 1: Main Report.April 9, 2016. p. 3. 400Idem. 401Poulton, Colin, KassahunBerhanu, Blessings Chinsinga, Brian Cooksey, Frederick Golooba-Mutebi, and Augustin Loada “The Comprehensive Africa Agriculture Development Programme (CAADP): Political Incentives, Value Added and Ways Forward. Working Paper 077.” Future Agricultures. February 2014. p.3. 156 the Council of Ministers of West and Central African States (CMA/WCA),402 as well as a Technical Assistance Fund to support CAADP processes.403 “These donor decisions to link aid for agriculture to implementation of CAADP catalyzed the CAADP process in many additional African countries.”404 One of the key assumptions underlying the MDTF was that establishing an alliance of donors supporting the MDTF would foster a more unified/aligned approach to the development of CAADP institutions and processes.405 The MDTF is recognized to have considerably increased coordination at different levels, particularly among African lead institutions, between sectors, and among MDTF contributing development partners.406 Contributing donors have often had different implicit objectives for the MDTF, which has created some tensions.407 Some donors seemed to be more interested in leveraging their own contributions, others in institutional capacity building, and others in achieving country-level results. As a result, some contributing donors have been more willing than others to continue to contribute in a consistent manner based upon perceived successes and achievements.408 The MDTF has increased the ability of lead institutions to drive the CAADP process at international, continental, and regional levels.409 CTFs in particular have had a major impact on institutional capacity.410 This improved capacity has enabled continental and regional institutions to increase their ownership and stewardship of the process, although CAADP ownership at national level is often a different story.411 Donor Engagement 402 Ibid. 403 Ibid. 404 Ibid. 405 Gerrard, Chris, Gem Argwings-Kodhek, Ali Marouani, and Godfrey Mudimu. “Independent Evaluation of CAADP Multi-Donor Trust Fund (MDTF).” Volume 1: Main Report.April 9, 2016. p. 9-10 406 “Independent Assessment of the CAADP Multi-Donor Trust Fund.” Discussion paper no. 158. ECDPM, ESRF, and LARES. February 2014. p. vii. 407 Gerrard, Chris, Gem Argwings-Kodhek, Ali Marouani, and Godfrey Mudimu. “Independent Evaluation of CAADP Multi-Donor Trust Fund (MDTF).” Volume 1: Main Report.April 9, 2016. p. 9-10 408 Idem. 409 “Independent Assessment of the CAADP Multi-Donor Trust Fund.” Discussion paper no. 158. ECDPM, ESRF, and LARES. February 2014. p. vi. 410 Ibid. 411 Ibid. 157 The CAADP approach requires that African institutions at continental and regional level run the process.412 Progress at the national level is mixed, with CAADP too often remaining a process with low levels of ownership and financial sustainability, to a large extent dependent on national political economy dynamics.413 Reactions of governments have varied from enthusiasm to reluctance.414 Domestic political incentives to support smallholder agriculture remain weak in many African countries and, as a result, national ownership of the CAADP process has often also been weak.415 In situations of weak commitment, donors have found themselves not so much supporting a national process as pushing it.416 Developing clearly defined and consistent policy frameworks is viewed as a measure of the success of CAADP in guiding countries toward developing transparent and inclusive agriculture and food security strategies.417 CAADP compacts undergo external technical review and provide a framework for identifying investment requirements; thus, these compacts generally meet Feed the Future requirements.418 USAID has worked with the African Union on these compacts, as they form the basis for investments in agricultural development, including those under Feed the Future.419 CAADP country compacts are developed via a process that facilitates stakeholder consultation; evidence￾based analysis; development of investment goals, programs, and alliances; and an iterative process of assessment, learning, and improvement.420 Forty-two countries had finalized and adopted or 412 Poulton, Colin, Kassahun Berhanu, Blessings Chinsinga, Brian Cooksey, Frederick Golooba-Mutebi and Augustin Loada. “The Comprehensive Africa Agriculture Development Programme (CAADP): Political Incentives, Value Added and Ways Forward.” Working paper 077. Future Agricultures. February 2014. p. 14. 413 “Independent Assessment of the CAADP Multi-Donor Trust Fund.” Discussion paper no. 158. ECDPM, ESRF, and LARES. February 2014. p. vii. 414 Kolavalli, Shashidhara and Regina Birner. “The Comprehensive Africa Agriculture Development Programme as a Collective Institution.” Discussion note # 005. IFPRI/Ghana Strategy Support Program, paper presented at the Transforming Agriculture Conference, 8-9 November 2012. 415 Poulton, Colin, Kassahun Berhanu, Blessings Chinsinga, Brian Cooksey, Frederick Golooba-Mutebi and Augustin Loada. “The Comprehensive Africa Agriculture Development Programme (CAADP): Political Incentives, Value Added and Ways Forward.” Working paper 077. Future Agricultures. February 2014. p. 14. 416Idem. 417 USAID. “Institutional Architecture for Food Security Policy Change: Cross-Country Study.” March 2015. P.11, 418 Lawson, Marian L., Randy Schnepf, and Nicolas Cook. “The Obama Administration’s Feed the Future Initiative.” Congressional Research Service. January 29, 2016. pp. 7-8. 419 Ibid. 420 Ibid. 158 were developing CAADP compacts as of late 2015.421 It is widely believed that CAADP has served a useful role in bringing donors together.422 Results of the CAADP Approach CAADP uses a regional framework to stimulate domestic dialogue and improve political and economic governance and, as a result, mutual accountability between external partners and African countries has been steadily developing – as demonstrated through the development of the African Peer Review Mechanism.423 The CAADP framework offers common indicators for peer monitoring of progress towards targets.424 When rated for mutual accountability by USAID, African countries performed better than the rest of the world, with 73 percent of African countries scoring strong in mutual accountability, compared to 40% in non-African countries.425 Through CAADP, Feed the Future works with the African regional economic communities: SADC, ECOWAS, and COMESA. Many individual Feed the Future MYSs at country level align with CAADP compacts.426 Feed the Future teams in Africa have contributed to the development of the host country policy priorities linked with the implementation of CAADP.427 The 2013 Feed the Future Policy Guide requires a system for mutual accountability, and Joint Sector Reviews are being implemented in Africa as part of the CAADP.428 Posts reported that the CAADP process and associated planning processes have raised awareness about the need for clear national policies and more effective coordination mechanisms.429 In an independent assessment of CAADP, country focal points and CAADP partners felt that the enabling environment for agricultural policies and programs in their country or region had 421 Ibid. 422 “Independent Assessment of the CAADP Multi-Donor Trust Fund.” Discussion paper no. 158. ECDPM, ESRF, and LARES. February 2014. p. vi. 423 Idem. 424 Irene Annor-Frempong. “Human and Institutional Capacity Development Strategy Review: African Stakeholder Perspectives.” Feed the Future. May 15, 2015. p. 8. 425 USAID. “Institutional Architecture for Food Security Policy Change: Cross-Country Study.” March 2015. p.27 426 Lawson, Marian L., Randy Schnepf, and Nicolas Cook. “The Obama Administration’s Feed the Future Initiative.” Congressional Research Service. January 29, 2016. p. 11. 427 Tumavick, Nancy M., and ASM Jahangir. “Final Draft: Feed the Future Report on Dialogue and Support for Policies Advancing Food Security, Agricultural Growth, and Better Nutrition.” Feed the Future. March 30, 2016. pp. 1-3. 428 Ibid. 429 Ibid. 159 improved much or some in relation to the indicators in the MDTF Results Framework.430 These respondents attributed improvements in the enabling environment to more inclusive policy￾making processes, training for the CAADP country teams, technical assistance, capacity building and information sharing, the preparation of the National Agricultural Investment Plans (NAIPs), the independent technical reviews, and in some countries, the Joint Sector Reviews.431 These improvements were attributed to the support of development partners, followed by the contributions of the RECs, non-state actors, and the African Union Commission (AUC) and the NPCA.432 Feed the Future has engaged in and supported a number of these actors and processes, regionally and nationally. In addition, there has been growing awareness of the important role that the agricultural sector can and should play in economic development. 433 CAADP CONCLUSIONS 1. CAADP is a primary mechanism that FTF supports as a component of advancement of its country-ownership principle. It supports and enhances consultation, policy reform, evidence-based planning, peer review, and accountability within the agriculture sector. Resources and support provided to CAADP under Feed the Future have advanced Feed the Future objectives. 2. Feed the Future has facilitated the process for better coordination between sectors, among donors, and with non-state actors in the agricultural sector. Through Feed the Future’s support, CAADP’s awareness-raising role has put agriculture development forward as a key area of concern for African economic growth and food security, and provided a vehicle for dialogue on agricultural issues at continental, regional, and national levels. 3. CAADP has mixed success across countries as a result of domestic political economic issues, but, with support from Feed the Future, it is solidifying principles of regional multilateralism which are gaining momentum over time. This process of regional multi-lateral integration meets the objective of FTF and opens the political space to make it increasingly difficult for African countries to manifest challenging political and development behaviors internally. CAADP RECOMMENDATIONS 1. Feed the Future should continue to provide support to CAADP, both through the MDTF and through supporting CAADP processes and institutions at regional and country level. 430 Gerrard, Chris, Gem Argwings-Kodhek, Ali Marouani, and Godfrey Mudimu. “Independent Evaluation of CAADP Multi-Donor Trust Fund (MDTF).” Volume 1: Main Report. April 9, 2016. p. 5. 431 Ibid. 432 Ibid. 433 Ibid. 160 Global Agriculture and Food Security Program (GAFSP) The Global Agriculture and Food Security Program is a multilateral fund to assist in the implementation of pledges made by the G8 at the L’Aquila Summit in July 2009, and set up in response to a request from the G20 in Pittsburgh in September 2009.434 The objective of GAFSP is to improve food and nutrition security in low income countries by investing in the food security and agricultural development investment plans that have already been prepared by countries, such as the CAADP investment plans in Africa. GAFSP provides financing to scale up agricultural and food security assistance on a coordinated basis in response to demonstrated commitment to results by countries. Channeling multiple sources of donor financing through a common mechanism aims to reduce fragmentation and recipient country transaction costs and improve alignment of donor programs. GAFSP is governed by a Steering Committee made up of voting representatives of contributing donors, and representatives from low income countries, and non-voting members from civil society and other stakeholders.435 Funds may be contributed through either a public sector or a private sector window. A Coordinating Unit in the World Bank provides administrative, technical, coordination, and communication support. Once individual projects are approved, they are administered through one of several supervising multilateral entities – the World Bank, International Finance Cooperation (IFC), IFAD, United Nations Food and Agriculture Organization (FAO), or the regional development banks. While Feed the Future, through the U.S. Treasury, is the lead donor to GAFSP, it aims to leverage funding from other donors by committing to contribute one dollar for every two dollars contributed by others.436 As of August 2015, about $1.5 billion had been pledged, involving a total of 11 donors, with the United States committing 38% of the total.437 Nearly $1.4 billion has been received to date and investments have been financed in 30 countries through the public sector window (total $1.02 billion), and in 23 countries through the private sector window (total $0.22 billion). GAFSP investments leverage additional resources in recipient countries, through host government, bilateral donor, and private sector co-financing.438 434 The description in this section is taken from www.gafspfund.org 435 Governance and funding windows are described in www.gafspfund.org 436 KII interviews with BFS and US Treasury. 437 Information on funding in this section is taken from www.gafspfund.org 438 Review of project documents for projects funded by GAFSP, available from www.gafspfund.org 161 GAFSP FINDINGS GAFSP funding is focused on smallholder agricultural productivity, resilience and adaptation to climate change, access to assets, linking to markets, and nutrition programs, all areas that are highly consistent with Feed the Future priorities for enhancing food security.439 With a target of 12 million projected beneficiaries for the existing portfolio,440 GAFSP greatly expands the scale and reach of Feed the Future programs. Interviews with the U.S. Treasury and the GAFSP Coordination Unit indicate that the U.S. Treasury, in collaboration with BFS, has been strongly proactive in providing the vision for GAFSP and attracting additional donors to largely meet the objective of leveraging two dollars for every one dollar contributed by Feed the Future. Even so, contributions slowed notably from 2015, especially from other donors, as other donor priorities surfaced and the urgency of the 2008 global food crisis faded. After regular annual calls for proposals from 2010 – 2013, there was no call for proposals in 2014 or 2015. A call is proposed for the end of 2016, if current negotiations for additional contributions with two major non-US donors are successful.441 The first tranche of GAFSP projects approved became operational in 2012 and 2013. The M&E systems for these projects are starting to yield evidence of a substantial number of beneficiaries reached and successful outcomes. For example, in Rwanda, the GAFSP project promoting improving land husbandry for climate resilience has already reached 209,251 beneficiaries (50% women), and yields of maize, beans, and potato in treated areas are 30%, 167%, and 219%, respectively, above the national average.442 GAFSP has committed to rigorous impact assessment of the 30% of its investments provided by Feed the Future.443 At present, impact assessments are being carried out in: Bangladesh, Haiti, Liberia, Nepal, Rwanda, Uganda, and Mongolia, delivered through partnerships between the concerned governments and specialist technical agencies, currently including the Development Impact Evaluation group (DIME) of the World Bank, Makerere University, and Imperial College.444 The evaluation team found that GAFSP has largely been seen by missions as a separate activity to mainstream Feed the Future investments. The FY 2014 WOG portfolio reviews frequently drew attention to the operation of GAFSP projects in the focus country under review that were not considered by the missions in these countries to be part of the Feed the Future portfolio. Although 439 www.gafspfund.org and the Feed the Future Guide (2010). 440 GAFSP Coordination Unit, 2016. 441 Interview with U.S. Treasury and the GAFSP Coordination Unit. 442 GAFSP Coordination Unit, 2016. 443 https://www.treasury.gov/press-center/press-releases/Pages/tg1083.aspx 444 Communication from the GAFSP Coordination Unit, World Bank. 162 GAFSP is clearly a multilateral activity, the similarity of objectives and activities funded by Feed the Future in its ZOIs to those funded by GAFSP suggests unrealized benefits from greater sharing of results and good practices as part of the wider monitoring, evaluation, and learning agenda. GAFSP CONCLUSIONS 1. Feed the Future, led by the U.S. Treasury, was successful from 2010 to 2014 in leveraging other donors to contribute to GAFSP, and in providing leadership to the Fund. However, the slowdown in contributions is indicative of GAFSP’s substantial sustainability challenge in maintaining donor interest. One way to sustain engagement of donors is to fully and rigorously complete the ongoing impact assessment of GAFSP investments and communicate the results to donors. 2. GAFSP operates in most of the Feed the Future focus countries, but despite similarity of objectives and activities funded, has operated largely independently of other Feed the Future investments in those same countries. There is much potential to build synergies between Feed the Future country investments and GAFSP projects, especially through knowledge-sharing. GAFSP RECOMMENDATIONS 1. The GAFSP Coordination Unit should redouble efforts to assemble emerging evidence from M&E systems, including rigorous impact assessments, on the success (or otherwise) of GAFSP investments, and mount a strong effort to communicate these results to existing and potential donors. 2. Country missions and the BFS/MEL learning group should integrate lessons from GAFSP projects to share knowledge on good practices in implementing programs to address food security. Consultative Group on International Agricultural Research (CGIAR)445 Given the high costs of research and the potential for research outputs to “spillover” across countries, international agricultural research is central to the Feed the Future agenda of achieving food security at global, national, and local levels.446 The CGIAR is the leading multinational effort to implement agricultural, food, and nutritional research in support of Feed the Future objectives. The CGIAR system, made up of 15 autonomous international research centers, mostly located across Asia, Africa, and Latin America, committed to far-reaching reforms in 2010.447 These reforms 445 Interview with Executive Secretary of the Fund, and members of Fund Council. 446 Alston et al., 2006. 447 See CGIAR (2011) for the description of the CGIAR and reforms implemented. 163 had three central elements. First, the CGIAR centers would commit to a common system-wide Strategy and Results Framework around four objectives – reduce rural poverty, improve food security, improve nutrition and health, and sustainably manage natural resources. Second, centers would collaborate around 15 cross-cutting and globally strategic CGIAR Research Programs (CRPs), each embracing multiple centers and external partners, to achieve the system-level objectives. Third, donors would establish a CGIAR Fund governed by a Fund Council, and would commit to long-term pooled funding of the CRP portfolio through the Fund. Two funding windows were established to pool funding – Window 1 contributions to system-level CGIAR efforts including funding to the entire CRP portfolio and Window 2 for contributions to specific CRPs. In addition, donors could contribute, via Window 3, to specific centers, usually through project-based activities linked to a CRP, or they could continue to contribute bilateral project funds to centers outside of the CGIAR Fund. In the spirit of the reform, donors were expected to channel a substantial share of their funds through Window 1 and Window 2. CGIAR FINDINGS The Strategy and Results Framework (SRF) was agreed in early 2011 and the Fund was established in that year.448 Currently, there are 64 donors to this fund.449 Key informant interviews confirmed that the United States, through Feed the Future/BFS, played a very active role in the reform process, including the preparation of the SRF. It was also the largest donor, accounting for about one￾quarter of contributions to the CGIAR Fund. The Feed the Future research strategy, also finalized in 2011, linked Feed the Future funding of research in the CGIAR to the reform process. The strategy committed Feed the Future to provide “continued US leadership…focused on a more output- and impact-oriented structuring of the CGIAR global research programs.” (p. 42)450 Feed the Future contributions to the CGIAR Fund were focused on the most relevant CRPs for the Feed the Future agenda largely through Window 2 – mainly the commodity CRPs linked to Feed the Future priority value chains (rice, wheat, maize, dryland cereals, roots-tubers-bananas, legumes, livestock-fish, and CRPs on policies, institutions and markets, and on agriculture, health and nutrition).451 Overall, Feed the Future contributions to the Fund through Window 1/Window 2 increased to $31 million in 2014.452 As the largest donor, Feed the Future was important in contributing to an overall doubling in funding to the CGIAR from 2010 to 2015.453 However, Feed 448 CGIAR (2011). 449 Wadsworth, 2016. 450 Feed the Future, 2011. 451 http://annualreports.cgiar.org/finance/ 452 These and other funding data in this paragraph are based on data provided by BFS/ARP. 453 Wadsworth, 2016. 164 the Future has taken a very cautious approach to pooled funding of the CRPs – among the 10 major donors, it has the second lowest share of Window 1/Window 2 funding to total contributions to the Fund, at around 20% during 2013-2015. The great bulk of Feed the Future funding to the CGIAR Fund continues to be channeled to individual centers through Window 3. Some Window 3 funds are provided by Missions who have little incentive to fund global research, but no overall breakout of Mission funding to the CGIAR Fund was available to the evaluation team. Some Mission funding to the CGIAR is also provided bilaterally to CGIAR centers, but some of this may be for development rather than research activities. Overall, the CGIAR Fund fell 38% short on Window 1/Window 2 contributions, relative to approved budgets, over the period 2011-15.454 Feed the Future, through BFS/ARP, has also provided strong leadership in the Fund Council, according to interviews with fellow members of the Council. As expressed by one member, “the U.S. has been actively engaged, led by the Bureau’s Chief Scientist, with support from a number of others. Their contributions are invariably constructive, well informed, and well intentioned.” The United States has been especially influential in maintaining the World Bank as both a donor and as the Chair of the Fund Council.455 Even so, the United States has encountered limits to its influence in the CGIAR Fund. According to BFS/ARP, some CRPs were unable to clearly articulate priorities and strategically allocate funds across the participating centers. As a result, in 2015, Feed the Future, along with at least one other major donor, moved away from programmatic funding and back to funding a limited number of CRPs. Feed the Future reallocated about $10 million from Window 2 to Window 3. This was also intended to send a signal to the CGIAR that Window 1/Window 2 funds had to be used strategically to realize the results agenda articulated in the SRF.456 A recent “meta-review” of evaluations of the 15 CRPs by the Independent Evaluation Arrangement (IEA) of the CGIAR reinforced the perceived lack of priority-setting and identified the CRPs’ wide use of the “easy way” of allocating funds across centers according to a formula, rather than strategically according to priorities, science quality, and results.457 At the same time, the IEA review also highlighted the high potential of the CRP programmatic approach to build synergies and add value to the CGIAR research effort. CGIAR CONCLUSIONS 454 Ibid. 455 Interviews with Fund Council members. 456 Interview with BFS. See also discussion under Question #9. 457 IEA, 2016. 165 1. Feed the Future, through BFS, as a member of the Fund Council, has exerted strong leadership in the CGIAR during a period of major reforms. These reforms, especially the commitment to channel a significant share of funding through the CGIAR Fund to strategically important CGIAR programs, rather than individual centers or projects, have only partially succeeded. The U.S. has taken a cautious approach to programmatic funding and, as weaknesses of some CGIAR research programs became apparent, has partially reversed its commitment to programmatic funding. 2. The CGIAR programs are now entering a second five-year phase of funding at a time when the outlook for the reforms is particularly uncertain. Given the continuing need for international agricultural research to meet the objectives of Feed the Future, even stronger BFS leadership in terms of vision and funding will be critical for consolidating the reform agenda. The evaluation team concludes that the time is propitious for Feed the Future, through BFS, to redouble efforts to salvage key objectives of the CGIAR reform process, especially the idea of pooled funding of strategically selected and well prepared CRPs. In collaboration with other major donors, it would need to send a clear signal that only those CRPs that meet high standards on priority-setting, quality, and results would be funded. In some cases, this may lead to some CRPs that are central to the Feed the Future research strategy and priority value chains not being funded, at least initially, until they can meet those standards. CGIAR RECOMMENDATIONS 1. Feed the Future should maintain a comprehensive and transparent database of annual funding from both BFS and the Missions to each of the CGIAR Fund windows, and to bilateral projects within the CGIAR that are outside of the Fund. The database should also disaggregate contributions that are for research activities and those that are for development activities. 2. Feed the Future should lead a new round of dialogue with other major donors, the CGIAR centers, and major stakeholders to recommit to the aspirations of the reform agenda. 3. Feed the Future should channel the bulk of its support to research in the CGIAR through Window 1 and Window 2 to those CRPs that meet high standards in terms of strategic focus, science quality, and results. These standards should be clearly articulated and conveyed to the CRPs. Conversely, Feed the Future should minimize funding to CRPs, including Window 3 funding, that do not meet the standards. Question 9: The effectiveness, relevance, and collaboration in Feed the Future’s implementation of its research strategy. How well are Feed the Future-supported 166 research activities designed to address major global challenges and spur agricultural development across initiative Focus Countries? Feed the Future has developed a highly relevant research portfolio that strikes a good balance between research on global challenges and providing relevant research findings to focus country programs. Research programs address the major value chains targeted in focus countries. External evaluations of specific research programs have generally been positive, although the evidence base for assessing utilization of research findings is incomplete. Feed the Future has supported wide￾ranging partnerships on both research and funding, including with the private sector. These partnerships have leveraged substantial additional resources. Uptake of improved varieties in Africa has been slow, but this is being tackled through scaling programs and closer integration with private seed companies. The heavy investment in legumes appears to be appropriate given their role in human nutrition, soil fertility, and animal feed. The research portfolio is being implemented through whole-of-government actors with a diverse set of partners in the CGIAR, U.S. universities, and the private sector, and all are subject to regular external evaluations. Collaboration and integration of research activities at Mission level and with implementing partners is a continuing challenge. Smallholder productivity growth is critical to the success of the Feed the Future agenda and investment in research and development (R&D) is critical to achieving this.458 Research is also essential for realizing Feed the Future objectives of sustainable intensification, improved nutrition, and resilience to climate change. Accordingly, one pillar of the Feed the Future Guide commits Feed the Future to increasing its investment in R&D and enhancing coordination and collaboration in its implementation with other partners. To guide the scaling up of investment in R&D, BFS finalized a detailed research strategy in 2011 that was implemented through seven research programs grouped into three types.459 Group 1 related to major research programs on food that included three programs: 1. Climate resilient cereals 2. Advanced approaches to combat pests and diseases 3. Productive legumes Group 2 consisted of cross-cutting programs, again with three programs: 458 World Bank, 2007. CGIAR Consortium, 2011. 459 Feed the Future, 2011 and https://feedthefuture.gov/article/feed-future-food-security-innovation-center 167 1. Market and policy research and support 2. Nutritious and safe foods 3. HICD related to research, education and extension Group 3 consisted of the sustainable intensification program designed to integrate research findings from all major research programs into priority production systems of West Africa, East Africa, and Southern Africa, as well as South Asia. These systems were selected based on high populations of extreme poor, high incidence of stunting, and the comparative advantage of Feed the Future-supported research organizations. Most of the ZOIs are also located in these systems. These research programs were implemented through support to the CGIAR and to U.S. universities through ILs, and awards to the private sector, USDA, and other providers. All were designed to leverage collaboration and resource mobilization from other sources.460 The CGIAR reforms and the funding challenges have already been described in Question #8 above. For this evaluation question, all funding windows were considered, including bilateral funding that is often provided through missions. In addition, in 2015 BFS launched a portfolio of 10 scaling projects designed to accelerate adoption of technologies developed through Feed the Future and related investments. All but one of these were implemented through the CGIAR.461 The 24 Feed the Future Innovation Labs resulted from a revamping of the Collaborative Research Support Programs (CRSPs) following a Board for International Food and Agricultural Development) review in 2012.462 Up until this time, CRSPs used a competitive model whereby a lead university (the awardee) served as a management entity to administer competitive sub-awards which implement specific research program activities. After 2012, CRSPs were rebid over time, often to new lead universities, and were either renamed Feed the Future ILs to better align them with Feed the Future, or they were closed. Many of these Innovation Labs are organized around a value chain and include discovery research on globally relevant problems, with a strong applied research component designed to support Feed the Future focus countries. At the same time, 14 new ILs were added to the portfolio. Many of these were narrowly defined in terms of a specific problem of global importance to be addressed in five-year time frames, particularly tolerance to abiotic stresses for climate resilience in major food staples, such as drought and heat tolerance in beans, drought tolerance and fungal resistance in cowpeas, and vaccine development for Rift Valley Fever. Ten of 460 KII interviews with BFS/ARP. 461 Communication from BFS/ARP. 462 BIFAD (2012). 168 the new ILs use a consortium model in which research partners are identified in advance of award, in the competitive application process. For example, the Climate Resilient Beans Innovation Lab is implemented by a consortium led by Pennsylvania State University and includes several other U.S. universities, USDA, the International Center for Tropical Agriculture (a CGIAR center), and national research programs in partner countries. Feed the Future support to R&D in the private sector is relatively small and much of the agenda is related to incorporation of specific traits – especially heat, drought and low nitrogen tolerance, and insect resistance – through transgenic technology. Support to USDA has also been modest and centered around specific discovery research projects in which USDA has the human and infrastructural capacity to perform the research.463 Figure 2 Trends in USAID/BFS Funding of Agricultural Research ($ US) Source: Data provided by BFS QUESTION 9 FINDINGS Relevance Feed the Future has a well-developed and well-conceptualized research strategy. It was widely discussed with stakeholders, including those in focus countries, and priorities for global programs were further informed by overlapping these with geospatial data on poverty and stunting. 463 USDA’s National Institute of Food and Agricultural also received a sizable one-time Feed the Future grant to competitively bid research on legumes that was implemented by U.S. universities. 169 The resulting research portfolio is generally highly relevant at both the global and focus country level, with a good balance between discovery research on global challenges and applied research targeting focus countries. At the global level, priorities around discovery science in genomics, major pests and diseases, drought and heat stress, and nutritional traits are consistent with other global assessments of needed research into key issues.464 The choice of products/value chains for applied research corresponds well with the priority value chains in focus countries except for export crops, where Feed the Future has little comparative advantage relative to other research providers.465 All the staple crops (which include rice, maize, and wheat), legumes, livestock, and fish, as well as nutrition and policy, are being addressed through the portfolio of Feed the Future research programs. Examples include the USDA-implemented efforts under the Norman Borlaug Commemorative Research Initiative, including the Grain Legumes project and a goat genomics project.466 Feed the Future has elected to fund only the most important CGIAR Research Programs in terms of the Feed the Future research strategy and priority value chains.467 However, evaluations by the IEA of the CGIAR have highlighted the need for stronger priority-setting within these programs to make them more relevant to today’s specific challenges and to better fit the CGIAR’s comparative advantages.468 An issue that is raised by several of the IEA evaluations of the CRPs is that the CGIAR has often gone beyond its comparative advantage in producing international public goods by engaging in very downstream delivery and development activities, such as extension and seed production.469 Many of these activities are financed by missions in focus countries that are looking for a reputable partner with good knowledge of the technology to achieve quick impacts in the ZOIs. In addition, since 2015, BFS has financed 10 scaling projects, mostly in seed but also in vaccines, to accelerate uptake of technology and expand the number of Feed the Future beneficiaries. The Feed the Future evaluation team did not see evidence of research on delivery mechanisms that could inform the choice of the most cost-effective and equitable approach for specific contexts, but an initial evaluation of scaling efforts in Zambia suggests that more attention needs to be given to demand-side issues in producer acceptance and profitability of new technologies.470 464 CGIAR Consortium, 2011; Godfray et al., 2010; World Bank, 2007. 465 Feed the Future is co-funding a project on coffee rust at Texas A&M University. 466http://blogs.usda.gov/2016/02/04/special-bean-team-tackling-world-hunger-with-science/#more-62953 and https://blogs.usda.gov/2012/03/27/usda-science-in-support-of-feed-the-future-and-global-food￾security/#more-39198 https://blogs.usda.gov/2012/03/27/usda-science-in-support-of-feed-the-future￾and-global-food-security/#more-39198 467 For more information, see discussion in Question #8. 468 IEA (2016). 469 Ibid. 470 USAID (2016). 170 The Innovation Labs also address most of the priority value chains in staples, legumes, livestock, and fish, as well as irrigation, sustainable intensification, and nutrition. Moreover, new or revamped Innovation Labs with a strong applied research focus have carefully developed priorities through participatory processes in the focus countries.471 New Innovation Labs have also been added for discovery type research, often in the same value chains as existing Innovation Labs. For example, new ILs were initiated for climate resilient sorghum and for climate resilient millet, adding two ILs in addition to the longstanding IL on sorghum and millet. Additionally, there are U.S. universities that partner in some of these efforts although they may not be leading ILs specifically. The implementation of the research programs on specific value chains or themes through the CGIAR, the Innovation Labs, USDA, the private sector, and other partners, such as the International Fertilizer Development Center (IFDC), obviously requires close coordination to avoid duplication and gaps. Many Innovation Labs and USDA emphasize discovery research that can feed into the more applied research of the CGIAR or national partners. Where ILs and CRPs are engaged in applied research on the same value chain or theme, there has been some demarcation based on geographic specialization. Effectiveness The evaluation team reviewed effectiveness largely by drawing on the many evaluations of specific research programs that have recently been completed. In the CGIAR, all CRPs were evaluated externally in 2015.472 Overall, the CRPs funded by Feed the Future were rated as effective, and in some cases, strongly so (such as the Global Rice Science Partnership, and MAIZE), although with only three to five years of implementation it was too early to reach definitive conclusions. These evaluations did note some common weaknesses in implementing a value chain approach, scaling up, and development of adequate monitoring systems. Instability in funding, especially from Window 1/Window 2 funds, was widely noted as undermining investment in long-term discovery￾type research. The Innovation Labs are subject to five-year external evaluations. Programs may be renewed for a further final five years pending a good evaluation, continued relevance to Agency priorities and availability of funds. Again, the majority of the evaluations have been positive, and where serious problems were identified, BFS has rebid or closed the Innovation Lab. Rebidding has resulted in a change of the university serving as the management entity and/or as well as new partnerships. 471 Interviews with ILs and survey of ILs conducted as part of this evaluation. 472 IEA (2016). 171 Both of the projects under sustainable intensification, Cereal Systems Intensification in South Asia (CSISA) and Africa RISING (Research in Sustainable Intensification for the Next Generation) have been recently evaluated. CSISA, which builds on decades of research on rice-wheat systems in South Asia, was evaluated as being “likely to be one of the most productive investments in the agricultural R&D portfolio of USAID.”473 The 2016 evaluation of Africa RISING, a relatively new project, endorsed its continuation after a slow start, but recommended, among other things, a much stronger focus on farming systems, including the role of livestock, and on nutrition.474 Finally, a recent evaluation of policy research sponsored by Feed the Future was very positive on the range and quality of research but found little effort to carefully target decision-makers and track outcomes.475 Effectiveness can also be assessed through the outcomes of the research portfolio. BFS compiled an impressive database, in collaboration with their research implementing partners, of thousands of research outputs, especially new breeds of crops, but also management practices, institutional innovations in value chains, appropriate nutritional interventions, and policies. Similarly, the release of open datasets from Famine Early Warning System (FEWsNet), implemented by an interagency team including USDA and USGS, and the launch of www.usaid.gov/developer, provide datasets and tools not previously available.476 The real question of effectiveness is the uptake of these research findings – by applied research partners (in the case of discovery research, via CGIAR or national systems), and farmers, policy-makers or other users in the case of applied research. Obviously, tracking uptake is a major task and for many products it is still too early to be assessing uptake. However, the evaluation found no evidence that BFS has developed a comprehensive system for tracking uptake, although the CGIAR CRPs do report on outcomes annually. The very incomplete evidence available suggests continuing outcomes and impacts of Feed the Future’s investment in research. Impacts are most evident in genetic improvement of major staples and many of these have focused on tolerance to stresses, such as flood-tolerant rice in South Asia and drought-tolerant maize in Africa, that are critical to improving the food security of the poorest farmers that are most vulnerable to climate change.477 Some of the discovery type research is also feeding new traits for stress tolerance and disease resistance into national systems. Other research 473 Feed the Future, 2015 CSISA Phase II Evaluation Report, p. 9. 474 Feed the Future, 2016. 475 Hazell et al., 2016. 476http://blogs.usda.gov/2013/05/08/why-open-data-matters-g-8-and-african-nations-increase-open￾data-for-food-security/#more-45501 477 Emerick et al., 2016. Fisher et al., 2015. 172 has emphasized nutritional outcomes, especially OFSP and bio-fortified beans in Africa, and adoption of new legume and fish breeds. The fact that the researchers who led the work on OFSP, much of it supported by Feed the Future, won the 2016 World Food Prize, attests to the success of this research.478 There have also been a number of success stories on integrated pest management, including biological control in cowpeas.479 Systems research, especially CSISA, is registering significant outcomes such as the uptake of early rice varieties to facilitate timely wheat sowing by some 500,000 farmers in the Indo-Gangetic Plain.480 Despite these successes, feedback from missions on the effectiveness of the research programs has been mixed. As many as half of respondents in the evaluation team’s survey of missions were not aware of specific research programs, and nearly 40% of respondents to the survey – and many recent country portfolio reviews – say that research is not well integrated into the overall Feed the Future program.481 Yet, in another survey question on sources of technology such as crop varieties, missions overwhelmingly identified the CGIAR (12 of 14 responses) and U.S. universities/national systems (10 of 14 responses) as their major source. Field visits also found that most missions have provided additional financing for some research activities in their country, through support to ILs or to particular CGIAR centers. At the same time, in interviews with research entities, scientists often found it hard to engage missions in research since they are guided by incentives to achieve impacts in the short term. Indeed, an external review of the Ethiopia Feed the Future portfolio found that missions and IPs could benefit by drawing more on the expertise of the various research entities.482 According to the survey of ILs, some were also appreciated by BFS and the missions for their expertise in helping design interventions, especially the Nutrition, Legumes and Food Security Policy Innovation Labs. Effectiveness and sustainability of the Feed the Future investment in research greatly depend on research capacity at the national level. Recent reviews of research capacity in Africa have noted very uneven progress in developing national capacity.483 Human and Institutional Capacity for research, extension, and higher education is an integral part of the Feed the Future research strategy. Feed the Future has made impressive contributions to HICD in research systems through 478 http://www.worldfoodprize.org/. 479 Interview with Innovation Labs. 480 Feed the Future, 2015. 481 See for example, Feed the Future Country Portfolio Reviews for Cambodia FY 2015; Ethiopia – FY 2013 and FY 2014; Honduras – FY 2015; Kenya – FY 2012; Nepal – FY 2012; Uganda – FY 2014; and Zambia – FY 2012. 482 Feed the Future Ethiopia, 2015. 483 Lynam, et al., 2016. 173 graduate training. The Innovation Labs, for example, have supported more than 2,000 graduate students since 2006, mostly from partner countries, and contribute to the development of higher education curriculum in focus countries (for example, the new M.S. program in aquaculture at Kwame Nkrumah University of Science and Technology in Ghana, and an M.S. program in dietetics in Malawi).484 However, the evaluation team found little evidence of investment in broader institutional capacity development of national research systems (both public and private). By nature, these investments are long-term and may not be conducive to the timeframe used in Feed the Future mission programming. Finally, effectiveness also relates to the quality of overall BFS/ARP support to the research portfolio. Improvement of BFS scientific, coordination, and management capacity was one of the major recommendations of the BIFAD review of the Collaborative Research Support Programs in 2012.485 Interviews consistently praised the quality of BFS support to research programs of the Food Security Innovation Center, established in 2012. Collaboration Feed the Future research involves wide-ranging and expanding research partnerships that aim to leverage additional resources and specialist skills to better address Feed the Future objectives. Based on surveys and interviews with 12 ILs, collaboration and communication across them has generally improved since the 2012 BIFAD evaluation. There are also good examples of excellent collaboration between the CGIAR and the Innovation Labs working on the same value chain. The Legumes Innovation Lab recently hosted a global conference on legume research involving the major players in the Feed the Future legumes program, and two of the ILs and the CGIAR research program on legumes have partnered to support graduate training in legumes through scholarships. The Food Security Policy Innovation Lab employs an innovative model of collaboration between a U.S. university (Michigan State University), the CGIAR (IFPRI), and an African university (University of Pretoria) that is reported by interviewees to be working well. However, based on the Innovation Lab survey and interviews, respondents perceive that there is still a need for improved collaboration between the Innovation Labs and the CGIAR where they work on the same value chain or theme. The CGIAR is widely viewed as looking at collaboration as a funding mechanism rather than a partnership where each party contributes complementary resources to achieve a common objective. 484 Interviews with ILs and data provided by BFS/ARP. 485 BIFAD (2012). 174 At the WOG level, the Norman Borlaug Commemorative Research Initiative serves as an umbrella agreement that covers the USAID-USDA research partnerships – both the National Institute for Food and Agriculture competitive grants (using USAID funding) and the Agricultural Research Service (which brings its in-kind capacity with USAID funding to implement activities). USDA also brings in its own wide-ranging partnerships to this research, many of them advanced research institutes from OECD countries and the large emerging countries. Feed the Future also demonstrates partnerships with the private sector, especially in research on transgenics that has potentially high rewards, such as pest resistance or improved nutritional content, but also high risks in terms of the transactions costs of undergoing regulatory approval and seeking public acceptance. Most of these partnerships also involve CGIAR centers and U.S. universities. Likewise, evaluations of CGIAR programs have noted expanding partnerships with the private sector on discovery research (for example, with Monsanto on water efficient maize for Africa) as well as with seed companies (for example, with rice seed companies in South Asia), with machinery service providers (in CSISA), and agro-processors (such as potato processors in the Andean zone) on technology delivery and value chain development.486 Finally, BFS has increased partnership with other major donors to support research that is relevant to the focus countries, especially the Bill and Melinda Gates Foundation. A Memorandum of Understanding has been signed between BFS and BMGF, and annual meetings are held to review and map out areas of joint funding.487 At the international level, Feed the Future along with the Biotechnology and Biological Sciences Research Council (BBSRC) from the United Kingdom led the formation of the International Wheat Yield Partnership (IWYP). This consortium has attracted funding from agencies from Canada, Australia, France, and India, the Syngenta Foundation, and at least three multinational seed companies. USDA's National Institute for Food and Agriculture (NIFA) also partnered with the IWYP by aligning a competitive call with the goals and principles of the consortium.488 QUESTION 9 CONCLUSIONS 1. The evaluation team concludes that Feed the Future has developed a highly relevant portfolio of programs that strikes a good balance of research on global challenges as well as providing relevant research findings to focus country programs. Discovery research has been designed to feed into applied research at the country level or to the CGIAR. The seven Food Security 486 IEA (2016). For potatoes see http://pdf.usaid.gov/pdf_docs/PA00JW45.pdf. 487 Interview with BMGF. 488http://iwyp.org/ http://iwyp.org/ 175 Innovation Center (FSIC) research programs also address the major value chains targeted in Feed the Future focus countries. 2. The heavy investment in legumes appears to be appropriate given their role in human nutrition, soil fertility and, increasingly, animal feed. However, their role in human nutrition needs to be rationalized given rapidly expanding livestock consumption in many Feed the Future focus countries and the low share of protein provided by legumes in diets. At the same time, an assessment today might analyze the need for more investment in rice (the most important food staple and a crop without an Innovation Lab), livestock (important not only for nutrition but also livelihoods and many joint products), natural resources management for sustainable production, and food safety and nutrition (both high on the Feed the Future overall agenda). 3. External evaluations of specific research programs have generally been positive although the evidence base for assessing effectiveness in terms of utilization of research findings is very incomplete. The best evidence is on crop genetic improvement that has continued to be highly effective in terms of outputs. Uptake of improved varieties in Africa has been slow, but this is now being tackled through scaling programs and closer integration with private seed companies. Other research areas, such as systems research, also need attention in Africa to make them effective in integrating a wide range of research findings at the farm and community level. 4. The research portfolio is being implemented through a diverse set of partners in the CGIAR, Innovation Labs, WOG, and the private sector, all subject to regular external evaluations. However, the number of partners and awards has increased the overall complexity of the research programs. The rationale for some new ILs on climate resilience that address the same crop as some of the existing ILs is not clear. 5. Collaboration and integration of research activities at mission level and with IPs is a continuing challenge to both the research entities and the missions. The many ILs, each with multiple IPs, together with several CGIAR research programs, each with multiple centers, are being requested to concentrate efforts on 19 focus countries. Conceptually, country coordination should be done by the NARS rather than the missions. However, many NARS do not have the capacity to pro-actively set the national agenda and engage the relevant international research partners in implementing that agenda. At the same time, given that research spillovers within countries and from country to country are often large, BFS could relax guidance on concentrating research activities in the specific Feed the Future focus countries. 6. Feed the Future has supported wide-ranging and impressive partnerships on both the research and funding side, including growing links to the private sector. These expanding 176 partnerships have leveraged substantial additional resources for the Feed the Future research agenda, including resources at U.S. universities and USDA. However, the large number and complexity of partnerships bring their own management challenges. In particular, the integration of Innovation Labs and CGIAR, the two major implementing instruments for Feed the future research, remains a work in progress. QUESTION 9 RECOMMENDATIONS 1. At nearly five years since its launch, now is a good time to revisit the research strategy in terms of emerging lessons and gaps in the portfolio. BFS should update its research strategy to incorporate emerging priorities and lessons learned, paying particular attention to criteria for establishing new Innovation Labs, funding CGIAR CRPs, strengthening complementarity between the CGIAR and Innovation Labs, and finding ways to better engage and coordinate with the Feed the Future focus countries. An alternative approach that might achieve the same objective without expanding the number of ILs would be to ensure the effectiveness and transparency of the management entities at the existing ILs, and channel additional funds for climate resilient crops through them. 2. As part of its revamped strategy, BFS/ARP should develop its own Results Framework to transparently map resources to both programs and research entities, and track effectiveness in achieving outcomes relevant to the success of the overall Feed the Future effort. 3. At the country level, Feed the Future should give priority and incentives to building national research capacity as part of its research and HICD activities. This support should go beyond human capacity development to finding ways to support institutional capacity development, including the ability of national systems to effectively set the research agenda, better coordinate Feed the Future research activities in that country, and engage relevant international research partners in implementing that agenda. Question 10: How well has the initiative leveraged private sector participation to support agricultural and nutritional outcomes? The Feed the Future initiative has effectively leveraged private sector participation in support of agricultural and nutritional outcomes. Looking at Feed the Future output data on private sector participation to support agricultural and nutritional outcomes, the most cross-cutting and comprehensive data were for indicators that measured private sector relationships (number of PPPs), commitment (private sector investment), market access (incremental sales), engagement (private enterprises assisted), and feasibility (agricultural loans). Additionally, these indicators are 177 consistent with the findings gathered in USAID’s instrumental 2011 study on the value of leveraging the private sector, which found that the most important attributes for measuring impact include increased reach, improved efficiency, increased effectiveness, and increased sustainability. Together, these indicators provide a valid quantitative measure and support the finding that the Feed the Future initiative has effectively leveraged private sector participation, signaling increased commercial growth and value chain development. QUESTION 10 FINDINGS The whole-of-government effort to address the private sector involves 11 USG entities, of which six are directly involved in field activities: USAID, U.S. Department of Agriculture, Millennium Challenge Corporation, U.S. Department of the Treasury, Peace Corps; and U.S. African Development Foundation.489 Within USAID, the Office of Market and Partnership Innovation (MPI); Feed the Future Innovation Labs led by U.S. universities; the Center for Transformational Partnerships (CTP) within the U.S. Global Development Lab, which leads the Agency’s overall efforts to strengthen private participation in international development;490 and the Partnering to Accelerate Entrepreneurship (PACE) Initiative work to support private sector participation through partnerships, financing mechanisms for agricultural value chains, political risk insurance to encourage investment, SMEs lending facilities, and innovation challenges.491 The Bureau for Food Security leads the Feed the Future initiative and aims to integrate activities, many of which are led by USAID field missions, which promote market-led innovation, private sector engagement, increased private investment, greater access to finance, and improved agriculture risk management options.492 USAID missions in 19 Feed the Future focus countries design and implement activities promoting private sector engagement with smallholder farmers and actors across the targeted value chains in order to support agricultural and nutritional outcomes. Feed the Future also supports efforts under the New Alliance for Food Security and Nutrition, aimed at bringing African governments, donors, and the private sector together to increase investment in agriculture in Africa. A quantitative assessment of how well Feed the Future has leveraged private sector participation was based on an analysis of the FTFMS database for five indicators: (1) number of public-private partnerships – where a partnership was defined as “a collaboration between public and private sectors that involves a commitment of resources from both sides and jointly addresses business 489 USAID, Bureau for Food Security. (2015). 2015 Achieving Impact: Leadership and Partnership to Feed the Future. Washington, DC: USAID. 490 USAID Lab Year in Review – 2015. 491 Feed the Future Guide, May 2010. Stakeholders, p. iii and Range of Private Sector Engagement, p. 6. 492 Congressional Budget Justification 2017, Foreign Operations, Appendix 2. 178 and development objectives … (and) is designed to create shared value,”493 (2) private sector investment, (3) incremental sales, (4) private enterprises assisted, and (5) agricultural loans provided. This approach was based on a previous USAID study focused on assessing the value of private sector contributions to USAID activities in which the most important outcome identified was increased commercial growth and value chain development.494 Expanded relationships with the private sector, combined with increased investment, growing sales of agricultural products, and increased financing, provide a measure of how well Feed the Future has leveraged participation of the private sector. A qualitative analysis using a focus country online survey, key informant interviews, and a review of Feed the Future documentation was also used to assess private sector participation. An online survey conducted with 13 focus countries identified four important areas in which the private sector was contributing to Mission-level Feed the Future activities: (1) provision of agricultural inputs, (2) training of smallholder farmers, (3) buyers of farm production, and (4) financing agriculture production. The same survey also indicated that “scaling up” activities to meet targets was underway. One concern the evaluation team noted was the lack of an adequate mechanism to measure system-wide impacts.495 Using KIIs to triangulate information, four themes emerged regarding private sector participation in Feed the Future activities: (1) “Consciousness-raising and changing of mindsets” created by initiative activities is providing the private sector, smallholder farmers, and national governments a new understanding of how to work together; (2) Local and national private sector entities are mobilizing to engage with smallholder farmers with a focus on sustainable relationships;496 (3) Large-scale commercial farming is bringing both opportunity and challenges to policymakers and international development entities, and indicates a commitment to the sector; and (4) As more market opportunities arise, private sector-led financing becomes more feasible with smallholder farmers.497 The following quotes capture the overall tenor of the KII responses: “The private sector is mobilized … we call this a private sector-led implementation approach.” 493 What is a Partnership? USAID Feed the Future. See https://feedthefuture.gov/private-sector-engagement￾hub/about-partnering%20. Last accessed on July 5, 2016. 494 Understanding Private Sector Value: How USAID Measures the Value of Its Partnerships. August 2011. See https://www.usaid.gov/sites/default/files/documents/1880/Understanding%20Private%20Sector%20Value_ Assessment%20Report_Final.pdf. Last accessed on July 6, 2016. 495 Feed the Future Focus Country Survey – conducted online in April and May 2016. 496 The definition used for sustainability is the likelihood and ability that a development outcome will prevail or continue beyond the scope of a donor or aid-sponsored project. 497 Private Sector References from Key Informant Interviews. June 27, 2016. 179 – Bangladesh “We are bringing the market closer … a large-scale commercial farm is working with our growers to create massive economic change.” – Ghana “Mission selected major private-sector associations (as activity implementers) because they are well placed to link farmer cooperatives to export markets.” – Guatemala “We engaged the local private sector to commercialize seed production and processing for soybeans, drought tolerant maize, and orange-fleshed sweet potato.” – Malawi “We are looking at the private sector because of obstacles to farmers getting the technologies since they are not accessible.” and “We are not very good at capturing the vast majority of the benefits that (our activity) creates.” – Uganda These types of responses affirming private participation in Feed the Future activities were consistent throughout the key informant interviews. During the period FY 2012 through FY 2015, the initiative established 6,493 PPPs, leveraged almost $600 million of private investment, facilitated more than $1 billion in incremental sales, assisted more than 300,000 food security private enterprises, and supported the provision of $1.4 billion in agricultural loans. Malawi, Zambia, and Senegal led in the number of PPPs established. Ethiopia, Kenya, and Uganda registered the greatest incremental sales. Ethiopia and Kenya also led in the amount of private sector investment achieved, while Uganda, Malawi, and Senegal led in the number of private sector enterprises assisted. The final category – agriculture loans – had Kenya as the top leader in this category with over 60% of all loan disbursements under the Feed the Future initiative.498 An analysis of private investment leveraged by Feed the Future activities indicates an increase in the number of large-scale PPPs primarily focused on improving production. Thirty-five large-scale PPPs (i.e., partnerships valued at $1.5 million or more) support agriculture and food security. Fifteen PPPs targeted cash crops (coffee, cocoa, and horticulture), 12 focused on food staples (maize, wheat, rice, potato, and sorghum), seven provided support for 498 FTFMS Database – Dexis Data Pull – June 25, 2016. 180 increased productivity and income without targeting a specific commodity type, and one focused on dairy.499 Only 4% of all the PPPs registered in the FTFMS database are clearly identified as supporting nutrition, indicating a strong focus of partnerships on agricultural outcomes. The table below provides a summary of five Feed the Future indicators assessed to determine the level of private sector participation in the activities under the initiative.500 Year-to-year outputs have increased for both the total number of PPPs and the value of private investment leveraged as a result of Feed the Future assistance. The column to the far right provides total USAID funding by country for the period FY 2012 through FY 2015. Table 3: Private Sector Participation FY 2012 – FY 2015 Private Sector Participation501 Partnerships; Investment; Incremental Sales; Micro, Small and Medium Enterprises Assisted; Agricultural Loans; and Total FTF Allocation FY 2012 – FY 2015 Focus Country Total # PPPs 4.5.2 (12) Private Sector Investmen t $ Millions 4.5.2 (38) Incrementa l Sales $ Millions 4.5.2 (23) Private Enterprises Assisted 4.5.2 (11) Agricultural Loans $ Millions 4.5.2 (29) USAID Funding $ Millions502 Bangladesh - $21.7 $280 7,997 $4.2 $249 Cambodia 22 $8.8 $27.5 2,829 $2.7 $49 Ethiopia 203 $125.8 $208.2 19,980 $177.5 $262 Ghana 150 $18.8 $17.4 10,300 $40.2 $256 Guatemala 197 $10.6 $19.3 4,762 $10.5 $84 499 USAID Public-Private Partnership Search Tool. Center for Transformational Partnerships. Retrieved July 5, 2016, from https://partnerships.usaid.gov/. 500 “Synthesis of Evaluations Related to the Feed the Future Learning Agenda.” March 2016 Report. Page 24: “Four types of interventions stood out as successfully generating income: developing the capacity of small-holder organizations; establishment of clear commercial linkages; building marketing organization management; and facilitating access to finance. The key seemed to be aligning these factors with anchor firms, financial institutions and other market actors.” 501 FTFMS Database – Dexis Data Pull – June 25, 2016. 502 Lawson, M. L., Schnepf, R., & Cook, N. (January 29, 2016). “The Obama Administration's Feed the Future Initiative.” Congressional Research Service. Funding information is from FY 2010 – FY 2015 as indicated in the source. 181 Private Sector Participation501 Partnerships; Investment; Incremental Sales; Micro, Small and Medium Enterprises Assisted; Agricultural Loans; and Total FTF Allocation FY 2012 – FY 2015 Focus Country Total # PPPs 4.5.2 (12) Private Sector Investmen t $ Millions 4.5.2 (38) Incrementa l Sales $ Millions 4.5.2 (23) Private Enterprises Assisted 4.5.2 (11) Agricultural Loans $ Millions 4.5.2 (29) USAID Funding $ Millions503 Haiti 16 $5.5 $5.4 4,128 $100.2 $164 Kenya 244 $211.2 $64 22,950 $850.7 $246 Liberia 3,507504 $5.5 $6.7 7,347 $0.39 $70 Malawi 169 $2 -$1 46,697 $19.3 $89 Mali 177 $19.1 $14.1 5,303 $17.7 $145 Mozambiqu e 191 $42 0 15,814 $12 $122 Nepal 0 $2.3 $85 11,074 $8.5 $57 Rwanda 17 $46.4 $51.6 5.830 $2.3 $197 Senegal 326 $15.5 $2.92 41,669 $86.3 $131 Tajikistan 0 $3.4 $0.9 820 $1.3 $58 Tanzania 156 $8.3 $9 6,838 $6.6 $327 Uganda 377 $9.1 $151 66,630 $23.2 $196 Zambia 661 $23.9 $44 19,694 $4.4 $65 Total 6,493 $588.6 $1,065.3 302,642 $1,385.8 $2,853 Several additional USG agencies and departments are also contributing to activities in the focus countries. The USDA provided support for food security through 48 active Food for Progress programs in 17 Feed the Future focus countries, established 433 PPPs,505 and since FY 2011, 503 Lawson, M. L., Schnepf, R., & Cook, N. (January 29, 2016). “The Obama Administration's Feed the Future Initiative.” Congressional Research Service. Funding information is from FY 2010 – FY 2015 as indicated in the source. 504 In the course of implementing the FY 2015 work plan, more farming groups and MSMEs reached out to FED and expressed interest to be included in the program as well as willingness to share in the investment cost. FED decided to absorb all of them, resulting in actual achievement higher by 14% than target. 505 PPPs by country: Mozambique (173), Mali (116), Malawi (98), Tanzania (16), Liberia (9) and Guatemala (21) 182 invested almost $650 million.506 The U.S. Treasury Department has provided $1.4 billion in funding to the GAFSP, managed by the World Bank. Of this funding, $622 million has financed grant programs in 16 Feed the Future focus countries with Bangladesh, Ethiopia, and Rwanda each receiving $50 million, followed by Kenya, Liberia, and Nepal with $45 million.507 Another $120 million has been invested in enterprises in 11 focus countries through GAFSP’s private investment mechanism to finance smallholder farmers, cooperatives, and improved production and market access. Since 2011, the Department of Treasury has also financed $118 million in agriculture development activities by the International Food and Agriculture Development Program. MCC has signed 32 compacts with 26 countries totaling $9.56 billion of obligated funds. Through MCC’s model of development to promote host government accountability, these compacts support country-led programs for agriculture and food security, which include elements of the private sector such as irrigation, roads, ports, post-harvest infrastructure, property rights and land policy, productivity and business training, agriculture finance, and institutional and policy reform. Twenty￾two of these compacts have included substantial food security-related investments, which account for roughly half of MCC’s obligated funds, with additional agricultural-related programing in Niger expected to be obligated before the end of FY 2017.508 The Grow Africa Partnership and the New Alliance are important private sector initiatives of the African Union, with support from crucial partners including NEPAD, the World Economic Forum, and the G8. The United States supports this important alliance through Feed the Future. The 2014- 2015 New Alliance and Grow Africa Progress Report indicates that this initiative has leveraged private sector investment totaling $1.8 billion to date against private investment commitments of $10.2 billion. This investment is reported to have provided either jobs or market access to more than 8.7 million smallholders in 2014, including providing at least $300 million from increased sale of agricultural production. Also, more than 21,000 jobs were reported to have been created by African-owned companies. During this same period, donors were reported to have contributed approximately $3 billion committed as part of Country Cooperation Agreements.509 506 Lawson, M. L., Schnepf, R., & Cook, N. (January 29, 2016). “The Obama Administration's Feed the Future Initiative.” Congressional Research Service. 507 Global Agriculture and Food Security Program – Recipient Countries website. See http://www.gafspfund.org/content/recipient-countries. Last accessed July 6, 2016. 508 Lawson, M. L., Schnepf, R., & Cook, N. (January 29, 2016). “The Obama Administration's Feed the Future Initiative.” Congressional Research Service. 509 “The New Alliance for Food Security and Nutrition and Grow Africa Joint Annual Progress Report 2014- 2015.” October 2015. 183 The impact of activities being implemented by USG Feed the Future partners, USAID missions, BFS, and the US Global Development Lab programs through CTP appear to be having a systemic impact that is, according to KIIs, increasing the participation of the private sector. Feed the Future Uganda is initiating a four-year assessment of technology uptake in embedded value chains (i.e., value chains which play a functional role in the effectiveness of another) to be conducted by the Massachusetts Institute of Technology (MIT). The study will focus on three levels: (1) Individual Actors: Entities within the supply chain (including farmers) to map business models, behaviors and decision-making; (2) Supply Chains: Relationships and flows of material, finance, and information among individual actors with a focus on creating models for social networking, risk mitigation, marketing strategy and profitability; and (3) Market System: Interactions of embedded supply chains in context through the construction of a system level model that includes individual supply chains in various contexts and circumstances. The objective is to identify the factors that are most critical for creating systems that effectively disseminate technology, promote adoption, and create sustainable systems for continued application. One key informant noted “We are helping to create enterprise models that, we believe, have wide-ranging, positive impacts but we can’t really gauge the scope of those impacts. This study will give us insight into the systemic impact we are having.”510 Key informant interviews also indicate coordination is taking place within USAID and at the inter￾agency level, but there appears to be room for improvement. For example, a recent evaluation of the MPI-funded Partnership for Innovation (P4I) in Kenya and Mozambique highlighted the importance of engaging the private sector but cited one example of a failure to effectively coordinate between USAID Washington and the field. USAID/Kenya Feed the Future developed its own private sector innovation mechanism – the Kenya Feed the Future Innovation Engine (KFIE) program – a $22 million, five-year activity with essentially the same objectives as P4I. According to the evaluation report: “[The Mission] was surprised to learn that the P4I was authorized for Kenya given that KFIE was already approved. There was little subsequent involvement of the Mission in P4I activities in Kenya.”511 Since the mid-term evaluation, KFIE and P4I participated in a technical workshop on investor readiness in June 2015 in Kenya; KFIE was also invited to participate in person and virtually in the Agriculture Innovation Investment Summit that was co-organized by P4I and held in Washington, D.C. in June 2016. P4I partnerships are not targeted to the Kenya Mission as the KFIE mechanism provide a partnership and innovation acceleration platform for agriculture. However, other USAID Missions that do not have similar mechanisms have utilized P4I to form 510 Key Informant Interviews – Uganda. 511 “Feed the Future Partnering for Innovation Mid-term Evaluation Report.” November 2015. p. 9. 184 agriculture partnerships in the past year (including Zambia, Nigeria, Benin, and Nigeria) and a second round of partnerships in Mozambique. During a key informant interview, MPI staff were asked about how they work with other USG Feed the Future entities. They responded that there was some coordination but there was not a sense of a strategic focus on investing, or rather helping private entities that have operations in Feed the Future countries improve product sourcing through PPPs. Increasing the reach, commitment, and sustainability of private sector engagement was identified in both the literature and KIIs as an important factor in promoting value chain and commercial growth. An analysis of 52 distinct key informant data sources, which included 17 Washington-based respondents and 35 field-based respondents, found that the most common form of coordination was participation in meetings usually held once a month. There were varying responses to the effectiveness of these meetings, but none of the U.S. agency respondents specifically referenced having a formalized role or specific objective with regard to Feed the Future collaboration. Within USAID, some coordination was reported but again, no specifics were given on how it is working other than to cite information￾sharing.512 It was stated that direct interaction with missions tends to go through Country Support Officers (CSOs) in BFS. Lastly, both KIIs and an analysis of Feed the Future annual portfolio reviews by mission from FY 2012 through FY 2015 found references to the need for better coordination between programs in USAID Washington and the field to leverage private sector involvement.513 The 2017 Congressional Budget Justification for Foreign Operations, Appendix 2, provides a comprehensive description of the MPI Office’s role, responsibilities, and activities in terms of enhanced private sector engagement with a focus on increasing private sector investment through partnerships, improving access to finance, and providing greater access to risk management mechanisms. The potential for the MPI Office to play a greater role in Feed the Future coordination appears inherent in this description. MPI is also involved in expanding the market for agriculture technologies aimed at increasing productivity and decreasing production costs and in the adoption of new technologies. The Office is also determining how best to enhance public policy dialogue and improve agriculture and nutrition-enabling environments. Finally, MPI’s engagement with missions to provide technical support to focus country programs provides an important interface with the field. MPI appears to be in a unique position in terms of its potential to engage interagency 512 Dexis Analysis of Key Informant Interviews – Feed the Future Inter-Agency and Intra-Agency Collaboration and Coordination. 513 USAID Feed the Future Portfolio Reviews. See Cambodia FY 2015; Ethiopia – FY 2013 and FY 2014; Honduras – FY 2015; Kenya – FY 2012; Nepal – FY 2012; Uganda – FY 2014; and Zambia – FY 2012. 185 stakeholders on private sector engagement as well as play a defining role in collaborating with Feed the Future focus country programs. QUESTION 10 CONCLUSIONS 1. The Feed the Future initiative has effectively leveraged private sector participation in support of agricultural and nutritional outcomes. Both quantitative and qualitative data indicate significant increases in private sector relationships (number of PPPs), commitment (private sector investment), market access (incremental sales), engagement (private enterprises assisted), and feasibility (agricultural loans). These combined outputs signal increasing commercial growth and value chain development, which was previously identified by USAID as a key indicator of the value of private sector engagement.514 A better understanding of causal relationships and independent verification of output data would provide the initiative with a roadmap for understanding impacts and replicating successful models. 2. While important activities are happening at all levels of the initiative, there is an opportunity through stronger coordination to create synergy and leverage greater impact. In terms of leveraging private participation in Feed the Future activities, the Office of Market and Partnership Innovation appears to be in a unique position to strengthen the coordination process within USAID and across the interagency entities. An assessment could define how coordination could be strengthened for private sector engagement. 3. A low percentage of private sector partnerships through Feed the Future reported to specifically addressing nutrition, while most of these partnerships promoted improved production. QUESTION 10 RECOMMENDATIONS 1. Design and execute analytical studies focused on better understanding of Feed the Future causal relationships and outcomes. A key objective of this work should be to understand how different aspects of increased private engagement can positively impact the entire market system in a focus country, including spillover effects. A better understanding of system-wide impact should guide program design, set priorities for leveraging private engagement, and create an evidence￾based framework for the next phase of the initiative. 514 “Understanding Private Sector Value: How USAID Measures the Value of Its Partnerships.” August 2011. See: https://www.usaid.gov/sites/default/files/documents/1880/Understanding%20Private%20Sector%20Value_ Assessment%20Report_Final.pdf. Last accessed on July 6, 2016. 186 2. Put in place the resource levels and bureaucratic structures needed to more effectively coordinate private sector engagement within USAID and the Interagency entities. An assessment across the initiative should be conducted to better understand how and where coordination to leverage private participation can be more effectively implemented. The assessment should include the following: (1) Identify an appropriate entity within the Bureau for Food Security that will take the lead with the USG Interagency to establish a private sector engagement coordination system and determine the demand-driven needs of potential corporate partners across Feed the Future countries; (2) Analyze and align staff and resources required to effectively implement the complex activities required to execute initiative-wide private engagement coordination; (3) Recommend coordination mechanisms within USAID, and among other USG Feed the Future partners, to proactively identify critical opportunities, provide a “call to action” forum to support direct, effective engagement to leverage greater coordination and impact, and provide sufficient staff support for follow-up; and (4) recommend the re-design and implementation of an incentive mechanism, such as the former Global Development Alliance (GDA) Incentive Fund, to spur ideas and visions that could promote a broader approach that benefits multinational corporations, increases market sales for small holders, and is sustainable long after Feed the Future programming ends. Question 11a: How well is Feed the Future identifying and promoting policy reform at the national and regional level? Feed the Future has done very well in identifying an initial set of crucial policies and regulatory changes in each country and region, on a prioritized basis, needed to achieve the initiative’s goals, while aligning with country-owned plans. The coordinated approach among interagency policy specialists led to agreements with governments for the finalization of Feed the Future policy agendas. There is a dynamic and ongoing process of identifying additional policies and policy adjustments needed since the initial identification of policy agendas in 2012. These processes ensure that policy agendas are relevant to current needs and country specific dynamics. The process used across focus countries and regional missions for the initial identification of country￾specific policy matrices modeled the use of inclusive stakeholder participation, and the importance of building inclusive consultative mechanisms within which policies are considered, decided, and implemented. QUESTION 11a FINDINGS The initial identification of Feed the Future policy reforms in 2012 was based on a coordinated interagency process, in collaboration with Feed the Future policy teams at post, to identify and rank-order policies to develop the policy agenda. A highly consultative and inclusive process was used that included Mission Feed the Future team members, representatives from participating USG 187 agencies, host country governments, and other donors, as well as representatives from local private sector and civil society organizations, and local and multi-lateral institutions in each focus country. Feed the Future Deputy Coordinators for development and diplomacy required posts to submit policy matrices identifying crucial policy changes required, on a rank-order basis of priority, to maximize the impact of USG investments in food security to achieve Feed the Future goals.515 The policy matrix requires a description of each policy and its specific objectives, its relationship to the CIP, planned measures and actions, indicators and targets related to timeframes, USG support mechanisms, and key local partners. The selection of policies to promote Feed the Future nutrition goals must support country strategies that are found in country health sector plans, or multi￾sectoral nutrition plans. The Feed the Future Deputy Coordinators also called for each post to form country Feed the Future policy teams to develop a work plan for implementing the policies. The work plan requires policy teams to describe their planned actions to strengthen what are described as the three essential components of an effective and sustainable policy system: a substantive policy agenda (reflected in each policy matrix), institutional architecture (IA), and mutual accountability mechanisms. Mutual accountability mechanisms include the development and use of M&E systems to track performance of policy initiatives, and the formation of joint review committees to review and discuss performance. Members of the Feed the Future Interagency Sub-Committee on Policy traveled to each post to support the development of the country policy matrix. Seven priority areas related to food security were outlined based on the examination of policy matrices submitted by each post. They include: the enabling environment to promote private sector investment; agricultural inputs; agricultural trade; land natural resources tenure, rights, and governance; nutrition; resilience and risk management; and institutional architecture of public and private sector institutions involved in policy decisions and policy implementation. According to the Interagency Guide, IA refers to capacity for specific policy-related activities, “including collecting and analyzing data, proposing, implementing, and enforcing policy. It also includes the process of consultation between partner governments and stakeholders from the private sector, civil society organizations, think tanks, and development partners.” IA is also one of the essential components of effective and sustainable policy systems.516 This same interagency sub-committee completed a Feed the Future policy guide in 2013517 to serve as a companion to the policy matrices and to guide effective practices to support capacity development and implementation of policies. The Feed the 515 Idem, p. 4. 516 IA was included as a policy area based on a large body of evidence demonstrating that lack of attention to these factors will stall or halt policy implementation, and undermine the sustainability of policies that have been implemented. 517 USG Interagency, Feed the Future Interagency Guide to Support Sound Enabling Environments, 2013. 188 Future Policy Guide describes the formation of the initiative’s policy reform agenda and approach.518 A review of focus country policy matrices shows the alignment of policy agendas with country CIPs. In African countries, the agendas are based on country CAADP investment plans. In Central America and Asian focus countries, Feed the Future policy agendas support CIPs that were based on existing plans to increase food security and agricultural development. Policies related to Feed the Future nutrition goals support country health sector strategies or multi-sector nutrition strategies. These agreements, along with targets and timelines for policy implementation, were documented in the country policy matrix. Regional program policy agendas were identified following the same inclusive process. Regional Mission Feed the Future teams and their interagency partners collaborated with their existing regional partners and consulted with focus countries served by the Regional Mission.519 The finalization of regional policy agendas support the agendas of RECs520 and are based on constraints and issues that are best served by a regional approach; for example, country policies and regulations that impede cross-border and intra-regional trade.521 Since the initial identification of policy agendas in 2012, new policy needs have been identified. During FY 2015, BFS’ Policy Division sponsored several workshops to support policy discussions. For example, a series of nutrition policy workshops were held to identify and reframe policy priorities. Participants involved in the African Union Malabo Declaration Implementation Strategy and Road Map achieved consensus on seven areas of institutional reform needed.522 Priority plans for FY 2016 include policy research on structural changes in agricultural systems, as well as evidence-based development of agricultural risk management and resiliency policies.523 518 USG Interagency, Feed the Future Interagency Guide to Support Sound Enabling Environments, 2013, p. 2. 519 MYS documents from each regional mission describe the consultative process used to develop the regional Feed the Future program and the identification of a policy agenda. 520 Regional program MYS documents and the policy matrix for each regional program. 521 An example to illustrate these last two points is based on USAID/East Africa’s MYS. “Support the EAC, COMESA, and their agencies to accelerate their own agendas to promote free trade and improve the enabling environment for regional trade and market access. Support will be focused on harmonizing policies and regulations to de-bottleneck trade in the targeted staple commodities, and to get them implemented and enforced at the national level.” USAID/EA MYS, 2011, p. 12. 522 These examples are from the BFS/ARP Policy Division Portfolio Review Presentation covering FY 2015. 523 Ibid. 189 Additional areas of policy have been identified through a significant body of applied research and analysis conducted by research consortia and their local partners funded by USAID/BFS.524A key contribution of these consortia is their work on strengthening and increasing the evidence base required for evidence-based policy debates, and to inform effective policy design to support the initiative’s goals and objectives.525 Ongoing applied policy research conducted by these consortia have identified areas of policy change, as well as new policies and regulations – which are needed now. An important example is the policy research conducted through the Feed the Future Food Security Policy Innovation Lab, showing that the rapid growth of mid-size farms in some countries in East Africa is affecting smallholder farmers, migration, and employment.526 A policy concept paper based on their research findings, commissioned by the BFS Policy Division, was reported to be currently underway as part of a larger effort to examine policy needs emerging from structural changes in agricultural economies.527 Finally, there is also applied policy research specifically designed to inform the Feed the Future initiative globally. For example, BFS/MPI sponsors research on policies to improve the enabling environment for private sector investment. The U.S. Department of Commerce conducts applied research on mechanisms to promote private sector access to finance.528 QUESTION 11a CONCLUSIONS 1. Feed the Future has done very well in identifying an initial set of crucial policies areas, on a prioritized basis, needed to support Feed the Future investments and achieve the initiative’s goals, while at the same time, aligning agendas to support identified policy agendas detailed in country￾owned plans. The coordinated approach among interagency policy specialists, who drew on ongoing policy work and updated analysis as the basis for extensive consultations and discussions with governments and a wide range of stakeholders, led to a rank-ordering of the most pressing policy change needs in each focus country, and in Regional Missions. This approach led to the development of agreements with governments for the finalization of Feed the Future policy agendas. 524 These Feed the Future-funded consortia include the Innovation Lab for Food Security Policy Innovation Lab comprising Michigan State University, IFPRI, and the University of Pretoria, and the CGIAR Research Program on Policies, Institutions and Markets comprising multiple research centers involved with a variety of policy research topics with overall leadership from IFPRI, 525 MSU, IFPRI, University of Pretoria Annual Report of the Innovation Lab for Food Security Policy: October 2014-September 2015. 526 T.S. Jayne, A. Chapoto, N. Sitko, M. Muyanga, C. Nkonde and J. Chamberlin. “Africa’s Changing Farm Structure and Employment Challenge,” Policy Synthesis, Number 91. April, 2014. 527 Interview with BFS/ARP Policy Division, May 2016. 528 Interview with BFS/MPI, June 2016; interview with US Department of Commerce, June 2016. 190 2. There is a dynamic and ongoing process of identifying additional policies and policy adjustments needed from several policy-related activities, since the initial identification of policy agendas in 2012. One source of identification of additional policies and needs for adjustment is based on trends and specific issues emerging from discussions between the Feed the Future Interagency Sub-committee on Policy and Posts, based on closely tracking and analyzing implementation progress from annual policy reporting. The second source of emerging issues and new policies that are needed following three to four years of Feed the Future implementation is the applied policy research conducted on key topics on an ongoing basis by consortia funded by USAID/BFS and the CGIAR Fund Council. These processes are important for ensuring that policy agendas are relevant to current needs and country specific dynamics. 3. The process used across focus countries and regional missions for the initial identification of country-specific policy matrices modeled one of the key tenets of Feed the Future for building an effective and sustainable policy system: specifically, the use of inclusive stakeholder participation, and the importance of building inclusive consultative and stakeholder mechanisms within which policies are considered, decided, and implemented. QUESTION 11a RECOMMENDATIONS529 1. Continue the focus on identifying trends, issues, and policies and regulations that will contribute to achieving Feed the Future goals and objectives at this mature stage of Feed the Future implementation. Conducting assessments of impacts to date at the outcome level will also be of critical importance to providing evidence for policy changes required. Impact assessments currently being designed should not only include methods for testing hypotheses on the benefits of specific reforms and combinations of inter-related reforms in different policy areas, but also to identify unanticipated adverse effects on smallholder agriculture, income and poverty levels, and nutritional outcomes of both male and females. 2. The evaluation team concurs with recommendations from the 2016 assessment of the Policy Division portfolio (refer to Evaluation Question #4), but wishes to specifically highlight one of the assessment’s findings and recommendations on the need to focus more attention on the gender dimensions of agricultural policies given the primacy that Feed the Future places on the inclusion of women. One of their recommendations is to “develop guidance for partners to ensure 529 Many excellent recommendations are found in the recent assessments completed in 2016 by Hazell et al., and Tumavick Jahagi. The evaluation team elected not to cover the same ground in our set of recommendations with the exception of our concurrence with the need to focus more on the gender dimensions of agricultural policies. 191 attention to gender issues by requiring that they develop objectives and actions for reducing gender disparities in their respective agricultural policy area.” The authors make a strong case for analyzing how policy changes affect men and women, noting that agriculture nutrition policies are not gender neutral.530 Question 11b: How well is Feed the Future implementing policy reform? Initiative-wide, there has been considerable success in the adoption of policy and regulatory reforms, major actions, and measures completed to support policy implementation, but there is slow and uneven progress in the actual implementation of those reforms. Applied policy research conducted under Feed the Future represents a very important contribution to the adoption and implementation of policy and regulatory reforms by deepening the evidence base. Additional attention is required to increase the human and institutional capacity of government partners and civil society organizations to use evidence-based data, assess policies and regulations, and to engage in inclusive and effective policy dialogue. QUESTION 11b FINDINGS USAID missions in focus countries and USAID regional missions lead the process of promoting or supporting policy implementation with host governments with policy dialogue and institutional support from other USG agencies, depending on the country and the type of reform. Data from annual portfolio reviews, country policy matrices, and interviews with agency Feed the Future representatives and field interviews provide examples. For example, the USTR is most often involved in supporting or leading reforms in the area of trade policies and regulations. The FDA has been involved in advising the Government of Bangladesh on the implementation of national food safety policies. In focus countries with MCC negotiated compacts, policy support is provided which complements and strengthens efforts of USAID-funded Feed the Future programming. USDA is the primary contributor supporting USAID regional mission policy work, and in Guatemala, USDA provided assistance to the Government of Guatemala to facilitate compliance with the U.S. Food Safety Modernization Act which resulted in increased exports to the United States of certain agricultural products. Other USG agencies that are most often involved in policy support include the U.S. Department of State and the U.S. Department of the Treasury. Bilateral donors, the World Bank, and other multilateral institutions provide technical and institutional support for host government country policy agendas. Policy research to provide evidence for governments on the need for policy implementation is conducted by Feed the Future research mechanisms and local 530 Hazell, Ewell and Cook, Assessment of the Portfolio of the USAID/BFS Policy Division, March 2016. 192 institutions. USAID supports implementation through a variety of Washington-based activities and mission specific policy activities. A review of focus country policy matrices shows that policy implementation support at the sub￾national level frequently involves assistance complementary to national level policy reforms, such as technical support to local governments to review and understand policy changes that will, for example, create new economic incentives at a provincial, district or local level, and support to enable them to implement new regulations. Kenya represents a case in which the Mission has increased its involvement to promote policy implementation in the agriculture sector at both the national and regional level to support country political reforms requiring devolution of certain government authorities.531 At the local level, Feed the Future IPs are engaged in raising awareness and developing the capacity of civil society and private sector organizations to hold governments accountable, and more importantly, to engage with their governments as advocates for policy and regulatory changes that are beneficial for the people and businesses they represent. In Ghana and Uganda there is a particular emphasis on engaging private sector value chain actors, including farmers, traders, and processors. In the countries where interviews were conducted, the civil-society/private sector￾focused work seems to fall into two categories – providing technical assistance to build capacity in policy analysis/interpretation or advocacy strategy generation, and awareness-raising through mobilization efforts, information campaigns, and media training. To address how well Feed the Future is implementing policy reforms at the national and regional levels, the evaluation team drew primarily on the analysis of policy reform progress conducted by Tumavick and Jahangi, based on data from the Matrix Reporting Tool from each focus country and regional mission,532 coupled with data on policy progress included in FY 2015 Portfolio Review presentations. The Matrix Reporting Tool covers policies under each of the seven policy areas based on the policy matrices developed in 2012. It incorporates those policy actions required for implementation, and target achievements expected by FY 2017. With five years of Feed the Future focusing on policies required to support the initiative’s goals and objectives, focus countries show good progress in actions taken to promote and support implementation of policy and regulatory reforms in all seven policy areas, as defined by the Feed 531 Kenya Annual Portfolio Review presentation FY 2015. 532 Tumavick and Jahangi, Feed the Future Report on Dialogue and Support for Policies Advancing Food Security, Agricultural Growth and Better Nutrition, March 2016 final). 193 the Future Policy Guide, and in many areas, adoption of reforms has occurred. But recent assessments show evidence that actual implementation has been slow and uneven.533 USAID/BFS’s Policy Division acknowledges the continued need for support and has plans to support technical networks to assist country and regional implementation of CAADP agreements, as well as provide increased support for strengthening local policy systems to implement and sustain policy changes.534 Initiative-wide, there are also impressive achievements in implementation that can be cited from country reporting. The areas of greatest progress in actual implementation of policy reforms are related to agricultural inputs and trade. 535 A review of portfolio review presentations in FY 2015 show reporting of successful policy implementation across all policy areas. For example, in the area of Nutrition, the Bangladesh 2013 Food Safety Act was enacted in 2015 as the Food Safety Law, and the Food Safety Authority was established for enforcement of the law. In the area of resiliency and risk management policy, the Government of Ethiopia Council of Ministers approved and has begun implementing the National Policy and Strategy on Disaster Risk Management; reporting from Ethiopia on the policy matrix stated that this has helped to create an accountable structure to ensure a holistic response and coordination among relevant government authorities. In the area of agricultural trade, Zambia succeeded in removing the export ban on maize and is progressing in removing tariff and non-tariff barriers to comply with regional trade agreements. Policies enacted by the Government of Rwanda in 2013 transitioned fertilizer procurement and the distribution supply chain from the government to the private sector have – to date – led to the issuance of licenses for eight private importers to import and sell fertilizer throughout the country. All three regional missions in Africa have supported harmonizing regional seed trade regulations and procedures for implementation. The USAID Central America and Mexico Regional Mission reports success in removing non-tariff barriers to movement of agricultural goods across country borders in the region, resulting in reduced wait times and a reduction in illegal payments. USAID/East Africa reports a decrease in tariffs, lifting of some export bans for some agricultural products, and progress in the removal of other trade restrictions. 533 Hazell, et. al 2016; Tumavick and Jahangi 2016. The New Alliance/Grow Africa Annual Report also notes that across 10 African countries with New Alliance agreements, there is more progress in implementing policy reforms related to agricultural inputs compared to all other areas of policy reforms. New Alliance for Food Security and Nutrition and Grow Africa Joint Annual Progress Report 2014-2015. Pg. 7. This point was also noted by MCC (KII with MCC, June 2016.) 534 USAID/BFS/ARP Policy Division FY 2015 portfolio review presentation 535 Tumavick and Jahangi (2016). 194 A new component was added to the Matrix Reporting Tool for policy reporting in FY 2015, which asked posts to report on factors that either contributed to effective implementation of policies in each policy area where implementation has been achieved or on barriers to effective implementation for those policies where progress has been very slow (see Table 5). This component of the reporting tool is essentially a drop-down menu listing seven contributing factors that promote or constrain policy implementation, which posts can check off as appropriate. There is also a category called “other” for posts to document other factors related to policy implementation progress not included in the menu. The menu was designed by a team from Michigan State University, the International Food Policy Research Institute (IFPRI), and the University of Pretoria based on a review of the extensive literature on policy reform processes and implementation and their analysis of key drivers of policy change in the domains of agenda setting, design, adoption, and implementation. 536 Table 4: Contributing Factors and Barriers to Effective Policy Implementation (From: Tumavick and Jahangi, “Feed the Future Report on Dialogue and Support for Policies Advancing Food Security, Agricultural Growth and Better Nutrition.” March 2016.) Factors Contributing to Effective Implementation Barriers to Effective Implementation Fiscal resources Fiscal constraints Inclusivity/stakeholder participation Lack of inclusivity/stakeholder participation Powerful advocacy coalitions Powerful veto players Reliable policy framework Unpredictable policy framework Pressing problem/crisis event Lack of urgency/crisis event International/regional pressures or examples International/regional pressures or examples Overall, the two factors most commonly cited as contributors to effective implementation, across 18 focus countries reporting in FY 2015, are the use of procedures to increase inclusive stakeholder participation and a strong base of evidence on the potential impact of reforms. The most 536 D. Resnik, S. Babu, S. Haggblade, S. Hendricks, D. Mather. Conceptualizing Drivers of Policy Change in Agriculture, Nutrition, and Food Security – The Kaleidoscope Model, IFPRI Discussion Paper 01414, January 2015. This document discusses the research and analytical process that underpins the menu created for the Matrix Reporting Tool. 195 commonly reported constraint to effective policy implementation was the lack of inclusive stakeholder participation. 537 Matrix reporting from focus countries shows some commonalities based on policy areas. For example, the most frequently cited barrier to implementing policy and regulatory reforms under the area of agricultural inputs was powerful veto players. In the area of agricultural trade, the most common barrier to effective implementation of policies reported was fiscal constraints. However, reporting countries included other key factors constraining implementation of agricultural trade reforms in the “other” category, including, for example, lack of political will, and low government capacity to implement reforms. The East Africa Mission reported that “Progress [has been] made but powerful veto players remain ready to revert positive progress.”538 Fiscal constraints were also the most common barrier cited in the policy area of improving the enabling environment for private sector investment. Information from several key informants, evaluations, and other sources indicate that the most important factor impeding private sector investment is the lack of access to local sources of finance by both local, U.S., and multinational businesses.539 Seven countries reported slow progress of reforms in the financial and banking sector that would increase access to finance. Among the reasons cited in the “other” category were vested interests in the status quo, highly regulated regimes, and complications arising from the need to reform inter-related policies and regulations that reach beyond the agriculture sector in that country. Hazell et. al (2016) concluded that the policy process leading to implementation will require continued support to build the human and institutional capacity of national and local governments. However, missing from the dropdown list of factors that contribute, or conversely, represent barriers to effective policy implementation, is human and institutional capacity. Human and institutional capacity development is included as one of the key elements required for strengthening the IA for policy formation and implementation. This constraint is mentioned by posts under the “other” category, but we do not have systematically reported data from policy matrices across focus countries and regional missions on the importance of HICD as a contributing factor to policy implementation. An overview of findings from an IA assessment noted in the BFS/Policy Division in their FY 2015 portfolio review presentation states that, in order of importance, the greatest constraints to effective policy change are: 1) the absence of administrative 537 Tumavick and Jahangi (2016). 538 Tumavick and Jahangi cite this comment in their 2016 assessment report, p. 34. 539 Interview with OPIC, May 2016, Interview with U.S. Department of Commerce, June 2016. Midterm Evaluation of Ethiopia’s 2015. George Gray, Laura Kuhl and Demese Chanyalew, Feed the Future Ethiopia: External Midterm Performance Evaluation Report, 2015. This evaluation was conducted through USAID/Ethiopia’s Agriculture Knowledge, Learning, Documentation and Policy Project. 196 and technical capacity for policy implementation; 2) major capacity gaps across both private sector and civil society; specifically, the limited constructive, constituent-based engagement on policy; and 3) weak mechanisms for inter-ministerial coordination for approved multi-sector food security strategy. The fourth point mentioned is that despite growing political commitment to evidence￾based analysis, the practice of evidence-based policy-making is still quite limited. Hazell, et. al., also noted that there is not a large body of evidence to date on the impacts of Feed the Future-supported policy reforms that have been implemented. They make important recommendations regarding the type of impact assessments that should be executed now to determine the effect of policy changes at the immediate outcome level, as well as recommendations for collaborative planning between Rutgers University and IFPRI to design quasi￾experimental methods for future impact evaluations. The evaluation team discussed the importance of these types of evaluations for identifying future policy change needs; however, this information is also critical to provide evidence that can encourage governments to sustain policies, to adjust policies as needed, and to continue to implement policies that have not moved past the implementation stage. QUESTION 11b CONCLUSIONS 1. Initiative-wide, there has been considerable success reported to date in adoption of policy and regulatory reforms, major actions, and measures completed to support policy implementation, and to a lesser degree, the actual implementation of those reforms. While findings based on mission policy matrix reporting show that among the seven contributing factors associated with effective policy implementation, the most frequently reported are inclusive stakeholder participation and an evidence base on the effects of policy, the fact that there are no other significant reporting trends based on the other five factors indicates that other contributors are likely to be based on issues related to government capacity. As noted by a recent (FY 2015) assessment of IA, the absence of administrative and technical capacity for policy implementation is a major barrier to effective policy change. Capacity gaps in the private sector and civil society limit constituent engagement on policy with their governments. A continuation of efforts to address these weaknesses is critical to enacting policy reforms that will support Feed the Future goals. 2. The finding that there has been slow and uneven progress in policy implementation should also be tempered by the findings related to complexity of country-specific situations and starting points and major climatic (i.e., the effects of El Niño) and political events. These situations were reported by posts in the Matrix Reporting Tool. As a corollary conclusion, expectations for achieving policy implementation for reducing the prevalence of poverty and stunting according to the initiative’s overall targets within Feed the Future’s five-year program horizon are likely too high. 197 3. The applied policy research conducted under the Feed the Future initiative represents a very important contribution to efforts promoting the adoption and implementation of policy and regulatory reforms by deepening the evidence base. This conclusion is based on one of the key findings from the Tumavick and Jahangi (2016) assessment showing that a strong base of evidence on the potential impact of policies is one of the two crucial factors that promoted effective implementation of policies across bilateral and regional missions reporting against their policy matrix.540 Designing evaluations on the impact of policies that have been implemented to date, at the outcome level, will perhaps provide more credible evidence to governments, encouraging them to sustain those policy changes, and to make greater progress in implementing other key policy changes. QUESTION 11b RECOMMENDATIONS 1. Based on findings on factors contributing to the effectiveness of policy implementation from policy matrix reporting and the recent assessment of IA, we recommend an emphasis on efforts at the post level to develop and strengthen human and institutional capacity for effective policy change, including country mechanisms for obtaining inclusive stakeholder participation. Provide more focused guidance and capacity development on the effective use of those mechanisms to inform policy content, promote adoption, and achieve broad stakeholder support for policy implementation. Increase efforts to develop the capacity of policy constituents at the local level, namely, CSOs and the private sector, to advocate for policies and regulations that will increase the benefits of policy change. Couple these efforts with equally strong efforts to effectively develop both the human and institutional capacity to foster country ability, in both government and non-government institutions, to take the leading role in facilitating these processes. The evaluation team acknowledges that focused efforts related to these issues are being taken. 2. Feed the Future should track progress in building IA in focus countries, add indicators/reporting elements on IA to the policy matrix reporting tool to help assess the outcome of these efforts,541 and conduct evaluations to understand how IA capacity development interventions can be strengthened initiative-wide. 540 Interview with BFS/ARP/Policy Division, May 2016. 541 The lack of outcome indicators to assess progress in IA for government institutions that are assisted was one of the findings of the Hazell et. al assessment. This is referenced in the team’s evaluation of Feed the Future’s HICD work under Evaluation Question #4 in this report. 198 3. We concur with the recommendation made by Hazel et al542 on the importance of conducting evaluations of policy impact to date to determine if they work as intended at the immediate outcome level. 4. Consider designing a case study or evaluation of multi-agency efforts to promote policy and regulatory reforms meant to support inclusive agricultural sector growth in a given country to examine the combined effect and complementarity of interventions from each USG agency (e.g., USAID, MCC, Department of State, USDA) and to determine if these multi-agency policy efforts could better leverage each other. Question 11c: What kinds of regional policy interventions have been the most effective at contributing to bilateral impacts and why? Evidence from FY 2015 reporting mechanisms (FTFMS indicators, the policy matrix reporting tool, and annual portfolio reviews) can only provide an early and incomplete view on the contribution of regional policies for agricultural trade and agricultural inputs. Given the lag between success in reaching regional policy agreements, and the formation of national policies to implement those agreements, followed by adoption and actual implementation, it is too early to assess the impact of regional policies in focus countries within the five-year timeframe of the initiative. QUESTION 11c FINDINGS There is scant data to date showing the effectiveness of different kinds of regional policy interventions in contributing to bilateral impacts. There are several explanations for this lack of data, including the poor quality of trade data in African countries and lack of evaluations that could identify best solutions and pathways toward trade liberalization.543 To date, few impact evaluations have been done to test the assumptions on the benefits of regional agricultural trade or on the effect of policy work across all seven policy areas, largely because of the longer timeframe required to see impact. Once individual member state representatives in regional institutions agree on reforms or new laws, they develop regulations and guides to help governments implement these agreements. Country governments then have to tailor or customize the basic agreements to country-specific circumstances before they begin the process of promoting adoption and implementation. Another reason for the lack of data on bilateral impacts is described by USAID/East Africa in their FY 2015 Portfolio Review. Their presentation states that benefits from region-wide policy and regulatory reforms promoted through regional mission efforts and their 542 These recommendations are discussed as one of our findings. 543 Hazell et. al, 2016. 199 partners depend ultimately on focus country governments taking concerted actions to support/promote reform, and moving from signing agreements to actual implementation. Some data from various reporting mechanisms provide evidence that focus countries have begun implementing policies adopted through agreements finalized on a regional level. These successes are related to agricultural trade and agriculture inputs, and data is in the form of projections of benefits from implementation, and preliminary data on those benefits. Sources for this include policy matrix reports from focus countries,544 FY 2015 portfolio reviews (to a lesser degree), reporting on FTFMS policy indicators on the number of policies implemented on the national level from regional agreements, as well as from responses to the online survey for focus countries designed as a part of this evaluation. As one example, USAID/West Africa’s FY 2015 Portfolio Review presentation reports significant progress in the number of West African countries that have adopted the harmonized ECOWAS standards for fortified vegetable oil (12 of 15 countries) and fortified wheat flour (14 of 15 countries). The most frequently cited contributions of regional policy reforms to bilateral programs from the online survey (reported by eight respondents representing six missions) are the regional harmonization of grades and standards for key staple foods (eight responses), the reduction of tariffs and fees for moving staple foods across borders (eight responses), and streamlining customs practices for clearing agricultural goods across borders (six responses). These responses are consistent with policy matrix reporting by regional missions and by focus countries. For instance, Tanzania reported reductions in tariff and non-tariff barriers, and the elimination of some export bans related to agriculture trade policies. The reasons cited for implementation include pressure on the government to implement regional agreements, strong evidence on the benefits of these reforms, and sustained stakeholder engagement in reducing some of the tariff and non-tariff barriers. While there are no data on impacts, USAID/Central American and Mexico Regional Mission (USAID/ECAM) reported a country level impact from the establishment of science-based harmonization of Sanitary and Phytosanitary Standards (SPS). This prompted the Government of Honduras to enact two laws on meat and poultry inspection. As a result of enacting the law on poultry inspection, egg exports from Honduras to the United States generated $2 million in 2015. 544 Data from the Policy Matrix Tool is drawn from the synthesis of those reports, captured in the Tumavick and Jahangi report. 200 A question was added to the matrix reporting tool for FY 2015 about the effects of El Niño on trade policy. Policy matrix reporting from regional missions and focus countries cited concerns that prolonged drought from the 2014 – 2015 El Niño effects will erase years of work to promote intra￾regional trade because of serious declines in agricultural productivity. FY 2015 reports forecast that El Nino effects will likely cause governments to either overturn policies they implemented to promote intra-regional trade, or halt the adoption and implementation of trade policies based on regional agreements made by Member States. For example, USAID/South Africa reported projections that trade bans for key food crops are likely to be reinstated by several countries throughout the southern region of Africa. Declines in agricultural productivity in countries hit hardest by drought will reduce the amount of key food crops for storage and for trade. QUESTION 11c CONCLUSIONS 1. Evidence from FY 2015 reporting mechanisms (FTFMS indicators, the Policy Matrix Reporting Tool, and annual portfolio reviews) can only provide an early and incomplete view on the contribution of regional policies for agricultural trade and agricultural inputs. Limited responses to questions on the online survey related to the contribution of regional policies in focus countries provide insufficient information to triangulate with data from these various other reporting mechanisms across focus countries. 2. Given the lag between success in reaching regional policy agreements by Member States in each region, and the formation of national policies to implement those agreements, followed by adoption and actual implementation, it is too early to assess the impact of regional policies in focus countries within the five-year time frame of the initiative. It is also too early to assess the impact of El Niño on the willingness of countries to implement, and in some cases to sustain, policies based on regional agreements, in those countries most severely affected by adverse climate effects. QUESTION 11c RECOMMENDATIONS 1. Design evaluations to examine the effect of implementing regional policies on agricultural inputs and intra-regional trade in focus countries, and to test assumptions on the benefits of increased regional trade in basic food commodities to food security, increased income, and sales prices in local markets. 2. Consider developing regional policies to support and strengthen country policies and actions to increase resiliency and reduce the effects of global climate change on food security and hunger. Invest in policy research on how to introduce flexibility in regional trade policies to adjust to adverse climatic effects in support of countries that are hit particularly hard, and to help governments withstand pressures for reinstituting tariffs and laws that impede regional trade. 201 Question 12a: How well have Feed the Future MEL approaches achieved accountability for commitments Feed the Future has made? BFS/MEL has developed a comprehensive and robust approach to promote accountability for USG commitments. Systems and processes that coordinate annual reviews of Feed the Future portfolio performance and the reporting of performance data used for annual progress reports from USAID and other USG agencies have been institutionalized across all operating units where the initiative is implemented. Guidance, data collection and analysis tools, and training to support the MEL approach are valued and appreciated by users and are critical to support the system of accountability. The evaluation team found that the M&E Guidance Series and training and webinars led by BFS/MEL staff for the current set of Feed the Future indicators are valued by stakeholders. In 2010, President Obama committed a total of $3.5 billion of USG funds over three years to the interagency Feed the Future initiative, representing the U.S. contribution to supporting the UN MDGs of reducing the high rates of poverty and hunger around the world by 50%. The USG committed itself to reducing the prevalence of poverty and hunger by an average of 20% in targeted ZOIs in 19 focus countries. Named as the lead agency to coordinate and implement Feed the Future, USAID formed the Bureau for Food Security to fulfill this coordination function for the initiative. The Bureau was also charged with the responsibility of developing a robust system of accountability and learning that promotes the achievement of the U.S. commitment. The Bureau’s Office of Strategic Planning and Performance Management (SPPM) developed the initial approach in 2010, which aimed to increase accountability and facilitate learning based on the generation of empirical evidence on the results and impacts of food security programming.545 One of the core functions of SPPM is to oversee performance management issues for Feed the Future and for BFS, including the process of conducting annual Portfolio Reviews. SPPM also advises BFS on issues and supports compliance with USAID performance management policies. A Monitoring, Learning, and Evaluation Team (MEL) exists within SPPM to provide leadership and guidance on monitoring, reporting, and evaluation of Feed the Future, and to maintain and support a robust system for accountability and learning. QUESTION 12a FINDINGS The SPPM MEL Team has faced the enormous challenge of developing and coordinating an accountability approach that covers the complexity and breadth of Feed the Future programs: 545 Increased Accountability and Learning Based on the Generation of Empirical Evidence on the Results and Impacts of Food Security Programming is the SPSS Office Monitoring and Evaluation Unit Strategic Objective presented in its annual portfolio review presentation for FY 2011. 202 activities implemented across 19 focus countries, five regional missions, and aligned country programs in selected missions. The MEL team developed a comprehensive approach that includes the coordination of performance reporting across USAID operating units in Washington and the field, and across five additional USG agencies that directly contribute to field operations; Feed the Future indicators data requirements are based on an assessment of these agencies’ interventions.546 Elements of the overall accountability approach include: the collection and reporting of annual performance monitoring data; the PBSs conducted in country ZOIs to collect impact and outcome level data on Feed the Future objectives and goals; impact evaluations; annual Portfolio Reviews of the Feed the Future program from each focus country, regional mission, and BFS Office; producing annual progress reports for Congress and other major stakeholders in Washington; as well as the posting of publicly available baseline data and final reports from PBSs and performance and impact evaluations. The Annual Progress Report is the main reporting tool used to communicate progress. The Feed the Future Progress Scorecard, produced in 2012 and 2013, and developed to promote accountability for how the initiative is doing business differently to meet targets for reducing poverty and undernutrition, also drew on FTFMS and PBS data. The scorecard was dropped as a reporting tool in FY 2014 because it duplicated reporting in the annual progress reports.547 A major component of the accountability system requires USAID and contributing USG agencies implementing Feed the Future activities to collect and report data on a set of indicators. Those indicators were developed through an interagency process to measure progress toward achievement of the Feed the Future goals and each level of result, based on the Feed the Future Results Framework (described in the introduction of this evaluation report). These indicators were designed so that performance toward each result can be aggregated across country and regional programs, for analysis and presentation in annual reports to Congress. The annual progress reports from each year (to date, covering 2012 through 2015) are publicly available on the Feed the Future website. Of the total 53 Feed the Future indicators, eight are required in reporting provided by each focus country, and 21 are “required as applicable” for all Feed the Future programs. Of these 21 indicators, nine are used for WOG reporting requirements that incorporate all USG agencies (including USAID) implementing programming aligned with Feed the Future. A total of 17 Feed the 546 Participating USG agencies that report on these countries as part of the WOG approach include USAID, USDA, Peace Corps, MCC, USADF, and GAFSP (Treasury). 547 Communication from BFS/MEL, August 2016. 203 Future indicators are based on population-based data collected through baseline and interim surveys in the ZOI in each focus country, and 23 of the indicators are collected through USAID’s IMs.548 The remaining 24 indicators are referred to as “standard” indicators, which are optional. They represent “best practices” in tracking project and activity-level progress in areas of key interest to the Feed the Future strategy. Data reporting began in 2011. Achievement of results is determined based upon meeting pre-set targets developed by USAID and its partners, both in focus and aligned countries and in regional Missions. Data are entered directly into the Feed the Future Monitoring System, a web-based WOG database developed to collect and house data for the initiative’s indicators. The BFS MEL team employs data from the FTFMS as well as from PBSs from focus countries to produce annual progress reports against Feed the Future goals and objectives on an initiative level (globally), and by region. Feed the Future annual progress reports, PBS baseline and interim data, performance evaluation, and impact evaluation baseline and final reports are publicly available on USAID’s Development Experience Clearinghouse, and are also posted on the publicly accessible Feed the Future website. One element of the overall accountability system is conducted internally through annual interagency portfolio reviews held in Washington. Presentations from each focus and aligned country and regional mission provide data on progress toward meeting the targets for goal and outcome-level indicators and for indicators that reflect the specific Feed the Future program, plans for activities that will be implemented to reach targets, the results of evaluations, as well as key issues and challenges and plans for their resolution. The data incorporates WOG reporting per indicator from each USG agency that participates in focus country and regional Feed the Future programs.549 Portfolio review slide decks include interagency contributions, with Peace Corps and USDA being the most common agencies reporting out on indicators. Interviews conducted in Washington with six agencies revealed that agencies prioritize attendance at portfolio reviews for those countries where they are involved in initiative activities.550 A summary document of agreed￾upon follow-up actions from each portfolio review is prepared by BFS. These may include official Strategy Change approval documents if Operating Units (OUs) have made an evidence-based case documenting the need to change strategies to achieve Feed the Future goals. Some of the follow￾up actions entail the provision of support from MEL or other BFS technical offices to help resolve 548 Implementing mechanism indicators refers to those required as applicable and standard i.e. optional indicators used by USAID’s IPs that implement Feed the Future programs, projects and activities. 549 Portfolio review presentations do not cover Feed the Future activities from every USG agency. 550 FTF agency coordinators interviewed in Washington said they cannot attend every country portfolio review due to limited staff that are involved in the initiative. 204 issues discussed during the review. BFS follows up with each focus country and regional mission to ensure that actions have been completed. One of the most important functions of BFS/MEL in promoting accountability is the support they provide to focus and aligned countries, regional missions, participating USG agencies, and the technical offices in BFS to collect, analyze, and report on performance monitoring and PBS data, as well as other major sources of data produced by the initiative through impact and performance evaluations.551 MEL developed an M&E Guidance Series, and the Handbook of Feed the Future Indicators, a document that provides a definition for each indicator and detailed guidance on the methods that should be used to collect and analyze the data for that indicator. Volume 1 of the M&E Guidance Series provides an overview of the M&E system, which can be seen as a companion piece to the Feed the Future Guide produced in 2010 for the development of country and regional level Feed the Future programming (discussed in detail in Question #1.) Subsequent volumes provided guidance on collecting baseline data, sampling procedures for population-based surveys, and designing and conducting impact evaluations. As requests from the field for additional guidance on specific performance monitoring topics were made, MEL developed additional volumes and data collection tools. Examples include the Agricultural Indicators Guide and volumes providing guidance for measuring local capacity development, the Gender Implementation Framework for interpreting gender-related data, and guides for measuring natural resources management and climate change resiliency. In 2011, the MEL team engaged IFPRI to develop and pilot the Women’s Empowerment in Agriculture Index, which measures progress in the empowerment and inclusion of women in the agriculture sector within country ZOIs, along with guidance and tools to collect and analyze data required by the Index (discussed in detail in Question #5.) In 2015, MEL developed an evaluation design template and guidance (Volumes 12 and 12a), and an Introduction to Feed the Future Resources for new Feed the Future agencies and personnel. Guidance was also developed for conducting data quality assessments of Feed the Future data to support USG and Government Performance and Results Act (GPRA) requirements for accountability, and to be sure that presentations on performance are based on credible, valid data.552 To ensure accountability for compliance with U.S. Environmental Protection Agency (EPA) regulations and USAID policy on environmental impacts, BFS worked with Agency specialists to develop explicit protocols, including BFS Environmental Compliance Guidance and templates for 551 Data providing evidence on support to Feed the Future implementers was drawn from interviews with BFS/MEL in April and in May 2016, a review of BFS/SPPM annual portfolio review presentations, and reviews of the M&E Guidance Series and tools to support data collection, surveys and evaluations. 552 BFS guidance for conducting DQAs is based on USAID’s policy and approach to be in compliance with GPRA and its requirements to establish the means to verify and validate performance data. The 2010 GPRA Modernization Act emphasizes requirements to ensure the accuracy and reliability of data. 205 Environmental Mitigation and Monitoring, templates and guidance to conduct an Initial Environmental Examination, and an Environmental Compliance Tracking System. A Bureau Environmental Officer (BEO)553 joined BFS to work with the Bureau and MEL to oversee compliance with all environmental procedures and policy requirements. To accompany this body of guidance and tools, MEL offers numerous training courses, workshops, and webinars, as well as direct technical assistance from one of its implementing mechanisms or from MEL staff. Courses are also available online. New courses, and updated and revised versions of existing courses, have been offered on a continual basis from FY 2011 to FY 2015. For example, according to the STTA FY 2014 portfolio review, MEL reported that to date over 200 stakeholders were trained through 10 regional M&E workshops, in addition to providing ongoing virtual and direct technical assistance for M&E capacity development. In FY 2015, MEL introduced three new courses: WEIA/GIF training, advanced topics in agriculture, and an online course in nutrition￾sensitive agriculture. BFS/CSI also provides WEIA/GIF training. In 2016, MEL introduced a new Feed the Future Performance Monitoring course for USG agencies and USAID implementing partners that will be held at USAID University and rolled-out regionally. An example of technical assistance is the support MEL and other BFS technical offices provide to country and regional programs to address action items emerging from portfolio reviews. To obtain perspectives from the field, the evaluation team used responses from an online survey sent to USAID regional missions and missions representing focus and aligned countries, and data from interviews in Washington and the five focus countries the team visited. The online survey included several questions on the utility of the MEL system for promoting accountability, the helpfulness of MEL guidance related to Feed the Future indicators, and how well the indicators help countries to report on the objectives included in their Feed the Future Results Framework.554 The majority of responses to the online survey (81%), representing seven focus country missions, replied that the Feed the Future standard indicator framework and FTFMS reporting are either 553 USAID policy requires the placement of a BEO in each Bureau in Washington, DC. 554 The findings from the online survey must be tempered by the fact that only 13 of the 19 focus countries and three of the five regional missions responded to the survey. Focus countries that did complete the survey did not respond to all of the questions included in the survey. For example, just seven of the 13 focus countries responded to question on how well the Feed the Future indicator framework and FTFMS promote accountability. 206 helpful or very helpful for promoting and achieving Feed the Future commitments.555 Responses from each of the three regional missions that completed the survey replied that the standard indicator framework and FTFMS reporting are helpful. In a corollary finding, the majority of responses (representing eight of the 13 focus country missions) replied that the indicators under the Feed the Future framework and handbook provide useful indicators to measure most of their objectives (62.5%), followed by replies that the framework provides useful indicators to measure all of their key objectives (31.3%). The online survey also asked respondents to rate how helpful they found the PBS approach for monitoring impacts and outcomes to promote achievements and accountability for Feed the Future commitments. The majority of respondents (68.8%) replied that the PBS approach is either helpful (31.3%) or very helpful (37.5%). An additional question was included in the survey to determine how helpful BFS/MEL has been in clarifying requirements and supporting countries on how to collect and analyze data that serve as the foundation for accountability. Respondents were asked to rate the degree of helpfulness of documents from the M&E Guidance Series, other guidance documents, and technical support and training from MEL staff and BFS MEL implementing mechanisms. The majority of respondents, representing 13 focus countries, replied that the Feed the Future Handbook of Indicator Definitions (68.75%) and the Feed the Future Agricultural Indicators Guide (68.75%) were very helpful for clarifying how to collect and analyze data. An additional 31% of the respondents said that both the Feed the Future Indicator Handbook and the Feed the Future Agriculture Indicator Handbook were helpful. Respondents from the three regional missions all indicated that the Feed the Future Indicator Handbook and the Feed the Future Agriculture Indicator Handbook were very helpful. The majority of respondents from focus countries (68.75%) also replied that the WEAI Instructional Guide was helpful.556 With respect to guidance for the PBS, the majority of respondents (68.75%) replied that the Volume 8 Guide for the Population Based Survey Instrument for Feed the Future ZOI Indicators, with a Revised WEAI Module, was either helpful or very helpful, and (68.75%) also replied that the Volume 11 Guidance on the Interim Assessment of the Feed the Future ZOI Population Level Indicators was helpful. This guide also includes the Interim PBS tool. An additional 6% responded that this Volume was very helpful. The analysis of data from interviews with Mission staff and implementing partners and third party M&E support contractors from five of the focus countries provides some corroboration of the 555 Nine of 16 respondents, or 56%, replied that the Feed the Future indicator framework and FTFMS reporting are helpful for promoting achievement of commitments, and four of the 16 respondents, or 25%, replied these are very helpful. 556 6.25% replied that the WEAI Instructional Guide is very helpful. 207 survey findings about the utility of M&E guidance documents. Although the interview data isn’t specific regarding which guidance pieces are most helpful, the thematic analysis of opinions about the Feed the Future M&E system indicates that in general, that guidance is seen as beneficial to their work. With respect to training and webinars led by staff from BFS/MEL, 74.75% of survey respondents said these sources of support were either helpful or very helpful in clarifying how to collect and analyze data.557 Respondents from the three regional missions all replied that training and webinars led by BFS/MEL are helpful. Fewer respondents (43.75%) said that support from MEL implementing mechanisms was either helpful or very helpful.558 Other survey responses, as well as interview data from the five focus countries, suggest that custom indicators are used more than the standard indicators to inform project management and decision￾making, and to provide greater understanding of the results of activity implementation. One of the comments from an interview with an implementing partner in Uganda is illustrative: “There are 12 additional indicators that we collect because we think they help us to see what's on the ground.” When asked about the indicators they used for managing their projects, IPs often referenced the use of custom indicators rather than the Feed the Future standard indicators. Of the 53 implementing partners interviewed, 12 (22.64%) responded that they use Feed the Future indicators for management. Of those interviewed, one IP said that the set of indicators contributes to helping them in their decision-making processes; when examining progress toward achieving indicator targets, it helps them pinpoint areas where improvement is needed in order to reach targets. This is in fact the way performance-based management is intended to work. BFS has always recognized the need for custom indicators and the importance of their role. The recently updated Feed the Future Indicator Handbook (2016) encourages operating units “to design and use custom indicators as a way to better capture progress toward objectives and outcomes that aren’t fully covered by the standard indicators, and have the option to upload these indicators in FTFMS and report on them in the PPR [Performance Plan and Report].”559 But because these custom indicators are developed to obtain data for country-specific Feed the Future programming, they cannot be aggregated across reporting units for use in annual progress reports. In contrast, the standardized indicators used for reporting, both required and required if applicable, 557 43.75% of the respondents said BFS/MEL led training and webinars were helpful and 31% said very helpful. 558 25% responded that support from MEL implementing mechanisms were helpful, and 18.75% responded that this support was very helpful. 559 Feed the Future Indicator Handbook, 2016, p 5. 208 must be standardized to a range of activities across reporting units in order to be aggregated more broadly at an initiative level. While the respective roles of standardized and custom indicators have long been understood and acknowledged by USAID, the efforts and resource requirements to collect data for both types of indicators create a tension that is not unique to the Feed the Future initiative. It is therefore not surprising that many of the comments from the field mentioned the burden of collecting data for indicators, and that those comments were directed toward the standardized indicators. There was a mixed response from both Mission staff and IPs on whether there are too many Feed the Future indicators. A single question was asked in the online survey on this topic, and slightly under half of the respondents (eight focus countries or 43.85%) responded that there are too many. Of the 26 Mission staff and 53 IPs interviewed in the five fieldwork countries, a little over one-third of the Mission staff (34.6%) and IPs (33.9%) said that they thought the Feed the Future indicators are burdensome. One of the key factors that seems to render the Feed the Future M&E approach burdensome is the sheer number of indicators that are required and required as applicable. IPs with less comprehensive programs – such as focused TA programs, policy-focused IPs, or single￾activity local partners – have less burdensome reporting requirements. Comprehensive “flagship” programs with many indicators seemed more apt to describe the program as burdensome. Part of the burden comes from the fact that IPs are also collecting data on custom indicators, thus increasing the total number of indicators and effort required for data collection and analysis and reporting. One of the common themes that emerged from the analysis of field data, from slightly more than one-third of Mission staff (34.6%) and slightly more than one-third of the IPs interviewed (37.73%), is that there is too much emphasis on indicators that require counting things – as one IP expressed it, too many of the indicators require “chasing numbers.”560 The main reason given was that without contextual information, this type of indicator does not provide, in and of itself, very useful information; and yet considerable effort is required to collect data on these indicators and to get an accurate value. Over time, one can calculate percentage increases and decreases. Nonetheless, unless such indicators can be expressed as a percentage (with a known or estimated “n”), or correlated with like indicators on a particular issue for an analysis that would permit a more comprehensive picture of a result, the indicators have limited utility for management and learning 560 This issue does not refer to ZOI coverage indicators, which do have targets based on calculations of the numbers of people (for example) that need to be reached to reduce the prevalence of poverty and malnutrition. 209 purposes, at least as perceived at field level. Their main value is for aggregation across OUs for annual progress reports and for the analysis of trends. Fortunately, MEL has been engaged in ongoing efforts to examine indicators collected through IMs to determine how useful they are and to make adjustments based on the accumulation of feedback. Five indicators requiring counts were dropped through the “Indicator Refresh” exercise conducted by MEL in FY 2014. This exercise was designed to get systematic feedback and opinions from the field on which Feed the Future indicators could be dropped, which should be adjusted to make it easier to collect data and calculate values or to provide a more meaningful measure, and which indicators are essential. MEL provided options for different new indicators and suggested adjustments for comment by respondents. Eleven indicators, nine of which require counts, will be dropped at the end of FY 2016. These indicators are scheduled to be archived as a result of an exercise conducted by the U.S. Department of State Office of Foreign Assistance, in collaboration with USAID and other Department of State Offices. The purpose of that review was to update the list of Foreign Assistance Standard Indicators to include only those which had a proven internal learning or external reporting utility. This exercise resulted in a reduction of indicators applicable Agency-wide, and eliminated some indicators which are likely not useful to Feed the Future stakeholders in Washington in terms of accountability or understanding what the initiative is achieving. However, this recent reduction of indicators was undercut to some extent by the incorporation of new indicators, presented in the updated Handbook of Indicators. It includes new nutrition indicators, based on the new multisectoral nutrition strategy.561 Nonetheless, the total number of required if applicable indicators reported through IMs decreased from 33 to 28. Although the majority of respondents from focus countries who replied to the online survey said the Feed the Future standard indicators are useful for measuring most or all of the objectives in their RFs, this same set of respondents also replied that important indicators are missing from the standard framework. Interview data from fieldwork countries corroborate these responses. The evaluation team identified requests and proposals for the type of indicators that are needed from open-ended comments included with online survey responses, from within FY 2014/2015 Portfolio Review presentations, and from interviews in Washington and the field. One common theme that emerged was the need for indicators to measure long-term systemic changes. For example, USAID/Bangladesh noted the tension between immediate, short-term results and long-term systemic change, and the need to engage further with the BFS M&E team on indicators and methods for reporting on systemic change.562 BFS/MEL acknowledges the need for this type of 561 The augmentation of the Feed the Future RF is mentioned in Evaluation Questions #1 and #4. 562 USAID/Bangladesh Portfolio Review Presentation FY 2015. 210 indicator and methods to measure systemic change, and they are in the process of development.563 USAID/Uganda has commissioned research to develop indicators to measure systems level changes. If successful, this approach could be piloted and adopted for use by other interested actors, or folded into the work that MEL is currently doing. Other indicators mentioned included more meaningful measures of resilience to measure IR 5 (Increased resilience of vulnerable communities and households) than are currently used. Interviews with BFS/MEL indicated such measures have been developed and are in use by some resilience-focused activities; they will be included in the next set of indicators produced for USG food security efforts. Other requests were expressed for indicators and methods to measure the resiliency of agricultural systems to economic and environmental shocks, marketing systems changes based on increases in private sector involvement and investment, and the sustainability of Feed the Future results. Regional missions have called for data collection methods and indicators that reflect the contribution of their work to the region and to focus countries. For example, USAID East Africa expressed the need for methods to capture the prevalence of informal as well as formal regional trade, as a great deal of trade is still done informally. USAID/West Africa highlighted the need for standard indicators for trade flow and trade facilitation.564 The USAID Regional Mission for Central America and Mexico has developed several custom indicators that represent the contribution of their programs, which augment the standard indicators required for regional missions, including: “the number of regional buyer alliances established to link small producers and small enterprises resulting in sustainable, market-driven agreements,” and “the number of small producer units participating in regional and export trade.” MEL obtains considerable feedback every year from annual Portfolio Review presentations, surveys, discussions that take place during training, and other organized events, as well as through direct communication from the field. One key source of feedback is from stakeholders participating in the annual FTFMS webinars, which include an indicator session, with Q&A, and a chat. Much of this feedback assists the MEL team in its ongoing efforts to improve the interpretation of indicators and their definitions. The team has expressed plans to review the entire indicator structure, as well as the Results Framework, for the next iteration of Feed the Future. Further changes in indicators 563 Interview with BFS/MEL, May 2016. 564 This finding is based on a review of annual portfolio review presentations from FY 2013 through FY 2015. The two examples cited are from FY 2015. 211 will be based on that review and input from a range of stakeholders, as well as from the feedback captured during the FTFMS webinars. The key indicators used for accountability of the USG commitment to decrease the prevalence of poverty and stunting in children and to meet the Feed the Future objectives of increased inclusive agriculture sector growth and improved nutrition status of women and children have, very importantly, remained stable. Efforts continue to ensure that indicators measure progress towards the achievement of results at lower levels, in support of Feed the Future goals, and that these measures are valid and meaningful to capture changes going forward. QUESTION 12a CONCLUSIONS 1. BFS/MEL has developed a comprehensive and robust approach to promote accountability for USG commitments. Systems and processes that coordinate annual reviews of Feed the Future portfolio performance, and the reporting of performance data used for annual progress reports from USAID and other USG agencies, have been institutionalized across all operating units where the initiative is implemented. Guidance, data collection and analysis tools, and training to support the MEL approach are valued and appreciated by users and are critical to support the system of accountability. 2. Since 2011, the MEL team has continually assessed, adjusted, and updated their accountability approach and systems through additional guidance, training, and workshops to support the accurate collection, analysis, and reporting of data, and has made changes to the set of Feed the Future indicators. While the key indicators used to measure Feed the Future goals and objectives have remained stable, others have been added, dropped or adjusted to make improvements to the system based on formal and informal feedback sought by the team, and through systematic efforts such as those described in the findings. With the evolution and adjustments of activities and strategies that have occurred during five years of implementation, the MEL team is to be commended for working with many stakeholders to design and test new indicators that can measure important changes. 3. Feed the Future is at a mature stage of implementation and thus can anticipate progress toward achieving key outcomes related to Intermediate Results, as well as higher level results at the objective and goal levels. Agricultural and trading systems that are the markers of agricultural transformation are developing beyond the confines of the ZOI. Accordingly, it is not surprising that project managers and implementers are calling for additional “important and useful” indicators to measure the effects of implementing Feed the Future that the current set of indicators do not capture. 212 4. Indications are that the resource requirements for collecting and reporting data, based on the total number of Feed the Future and custom indicators collected by any one focus country through its implementing mechanisms, represents a burden; yet both types of indicators are collected to satisfy reporting requirements for accountability needs as well as to support project management and a more comprehensive understanding of what is happening on the ground. In addition to the cost of collecting data for indicators, there are the resource requirements for data analysis, developing performance narratives, internal discussions of performance, setting up systems for internal DQAs, and conducting DQAs in the field. The MEL team understands the tension between these needs and the resource requirements. Ongoing efforts to assess indicators and to drop, adjust or replace as needed, signal to the field that they are being heard, as well as ensuring that the overall set of FTFMS indicators does not create a greater burden than is strictly necessary. 5. The M&E Guidance Series and training and webinars led by BFS/MEL staff for the current set of Feed the Future indicators are valued by stakeholders. A focus on increasing training opportunities for implementing partners to increase the quality and validity of the data they collect will be necessary on an ongoing basis, in addition to the DQAs that are routinely conducted to meet USG requirements. More capacity development in data collection methodologies is also needed for local partners at the country and regional level to obtain quality data. Translating the indicator handbooks and guides and training materials into other major languages, principally Spanish and French, would extend the value of these materials to a wider range of Feed the Future implementers across focus countries, and further support the explicit capacity development purpose for which they are intended. QUESTION 12a RECOMMENDATIONS 1. USAID/BFS should consider leading a second “Indicator Refresh” exercise designed to capture utilization-focused feedback from stakeholders that Feed the Future reports to, focusing on eliminating FTF indicators that do not provide useful information for understanding field-level results. This process could yield further reductions in the number of standard indicators collected by IMs, while at the same time make room for new indicators to measure gaps that have been identified. 2. BFS should continue – and increase – training and webinar opportunities for IPs to maximize their knowledge and understanding of correct procedures for data collection and analysis – and their use beyond reporting. The recently designed Feed the Future Performance Monitoring course, held in August 2016, provides a new opportunity for IPs. USAID missions in focus countries with third party M&E contracts should be resourced adequately and required (where they are not already doing so) to provide training to implementing partners, local partner organizations, 213 Mission Feed the Future staff, and staff from other USG agencies implementing Feed the Future activities in those countries, to augment training opportunities offered by BFS/MEL. Question 12b: Has the Feed the Future MEL system supported improved programming and how? There are many sources of learning promoted under the MEL approach, including the production of data from BFS-supported assessments, studies and evaluations, and mechanisms to support access to information, such as posting information on Agrilinks, MEL-sponsored learning events, and blog conversations. MEL has broadened its learning approach to capture data produced in the field through reviews of evaluations produced in-country and through annual portfolio review presentations. Agrilinks is an important resource for accessing learning and guidance that is relevant to its users for learning in order to increase the effectiveness of current focus country programs and to design new activities. FY 2014 and FY 2015 portfolio review presentations show that important sources of field-level learning consist of evidence generated by recent evaluations, special studies, assessments, and surveys conducted in individual countries. FTFMS indicator data has different uses among stakeholders in Washington and in the field, and utility for different users varies by the type of indicator. FTFMS data has been valuable for learning at the initiative level in BFS, through aggregation of indicator data at the goal, objective, and IR levels to support assessments of trends across operating units and within specific areas of results, and to compare results between countries and in different contexts. Field interviews with IPs from the five focus countries visited by the evaluation team suggest that data from FTFMS IM indicators have less utility for learning and application than the custom indicators they design. QUESTION 12b FINDINGS Another primary function of the BFS/MEL Team is to lead efforts that support learning for improved programming to achieve USG objectives of reducing the prevalence of poverty and stunting among children. Two of the important components of the MEL system are the Learning Agenda, and impact evaluations conducted on Learning Agenda topics. Other components of the MEL system include the Agrilinks website, learning events, a blog series, support to OUs for conducting high￾quality performance evaluations, and most recently, requests for inclusion in annual portfolio reviews of information on learning used to inform future programming. MEL developed the Learning Agenda for the purpose of expanding and improving the knowledge base for effective food security programming to achieve Feed the Future goals. The Agenda is 214 based on the causal linkages between levels of results in the Feed the Future RF, and is divided into six general categories: (1) Improved Agricultural Productivity, (2) Improved Research & Development, (3) Expanded Markets, Value Chains and Increased Investment, (4) Improved Nutrition and Dietary Quality, (5) Improved Gender Integration and Women’s Empowerment, and (6) Improved Resilience of Vulnerable Populations. Each category includes strategic questions, developed through an interagency process, that are intended to provide evidence on “which interventions have the greatest impact in a given context, which interventions are most cost￾effective, and what combination and/or sequence of interventions/investments have the greatest impact on the multiple objectives of improving agricultural growth, reducing poverty, and reducing malnutrition.”565 Important learning products supported by MEL include extensive literature review documents on each Learning Agenda category that cover what has been learned to date and current best practices, impact and performance evaluations, and blog posts on different topics. A recent addition to the set of learning products is the evaluation synthesis report commissioned by BFS/MEL through its KDAD contract. This report is based on KDAD’s analysis of 196 program evaluations, and their synthesis of key findings from those evaluations, organized around the six areas of the Learning Agenda.566 MEL also supports learning by providing support to the field for the development of evaluation scopes of work and evaluation designs that will maximize the utility of ongoing evaluations for learning.567 For example, in FY 2016, MEL produced a new volume in its M&E Series that provides an evaluation design template and accompanying guidance for third party and USAID evaluators to meet professional standards for high quality evaluations and follow the requirements of USAID’s Evaluation Policy.568 BFS/MEL found it necessary to cut back on the number of planned impact evaluations to test causal linkages between results specified in the Feed the Future RF, although these were certainly intended to contribute to the development of important learning within the initiative.569 565 Feed the Future Learning Agenda. 2011. 566 KDAD, Synthesis of Evaluations Related to the Feed the Future Learning Agenda, March 2016. 567 Interview with BFS/MEL, May 2016. 568 Volume 12: Feed the Future Evaluation Template and Volume 12a: Supplemental Guidance on the Feed the Future Evaluation Design Template. 569 FY 2015 BFS/MEL annual portfolio review presentation. Among the reasons cited are the large costs involved in conducting impact evaluations, the long-time horizon required for planning, and the reluctance of USAID missions in focus countries to expend resources to support the requirements of management oversight and local level support needed for impact evaluations. 215 Learning products are publicly available on the Feed the Future website, the DEC, and on Agrilinks, the Feed the Future website developed by MEL’s Knowledge Management contract to promote learning and discussion among stakeholders and food security professionals worldwide. BFS￾supported applied research, evaluations, assessments from MPI, and ARP’s Research and Policy Divisions are also posted in these locations. In addition to posting the documents described above, MEL sponsors learning events and webinars to promote active discussion on specific Feed the Future topics. Discussion topics include recent findings and documents that have been posted and disseminated on Agrilinks and through other mechanisms. Agrilinks provides a blog series that encourages Feed the Future stakeholders to share experiences and lessons learned. Currently, the blog series includes Knowledge Management Insights, It’s All About M&E, Urban Food Security, and the Readers Corner, where bloggers post recommended reading and highlight key points of learning from them. Learning events germane to food security that are not specifically sponsored by BFS are also posted on Agrilinks. One of the challenges the MEL team faces is to capture what stakeholders are learning on an ongoing basis and whether – and how – they are applying what has been learned to improve programming.570 In an effort to more directly capture this information from the field, MEL added a specific slide on learning to the standardized portfolio review slide decks for FY 2014, which asks country and regional Feed the Future teams to identify findings from surveys, studies, and evaluations conducted in-house. They also ask for examples of ways in which findings from these sources were used for management, adjustments to programs activities, or future program design. This slide was standardized for the FY 2015 portfolio review slide decks and has been incorporated as a new element of MEL’s learning approach. MEL is also trying to foster more discussion on learning and adjusting as part of these annual reviews. The MEL Knowledge Management team attends each review to document these discussions and lessons learned, and follows up with each country team to obtain additional information. To support the dissemination of current lessons related to the Learning Agenda and emerging topics from Feed the Future implementation, Agrilinks gives prominence to new evaluations and assessment reports produced with initiative support – from USAID and from other participating USG agencies – and incorporates them in the library resources section of the website. For example, Agrilinks posts Peace Corps evaluations of its activities in focus country ZOIs and MCC shares its baseline data on activities they fund.571 One example is the evaluation of the Master Farmers pilot 570 Interview with MEL Team, May 2016. 571 Interviews with Peace Corps and MCC, June, 2016. 216 project conducted by the Office of Overseas Programming and Training Support (OPATS), that aimed to promote adoption of improved agricultural practices and inputs in remote, rural communities not covered by agricultural extension services. The evaluation report is posted on Agrilinks and USAID’s DEC.572 To obtain opinions from the field on how Feed the Future has supported improved programming, the evaluation team included questions in the online survey for focus countries and regional missions on key components of the approach to promote learning, and the use of evidence-based data in their programming to achieve initiative goals. The survey included questions on the use of data from routine performance monitoring activities, learning products from the BFS Learning Agenda, and the Agrilinks knowledge management website which also provides online learning opportunities, and host peer-to-peer discussions. While 40% of the respondents said “I don’t know” in response to the question on how helpful the development and application of the Feed the Future Learning Agenda has been in promoting achievements and accountability for Feed the Future commitments (eight focus countries responded), a more specific question – that included each category from the Learning Agenda – yielded fewer “I don’t know” responses – and more useful feedback for MEL. Nonetheless, 40% of the responses stated the Learning Agenda was “helpful,” 13.3% said it was “somewhat helpful,” and one respondent said it was “not helpful at all.” Two questions included on the online survey about the use of Agrilinks indirectly covered the usefulness of the Feed the Future Learning Agenda. Responses – based on 11 respondents representing eight focus countries – showed that Agrilinks is an important source of learning used to inform and improve programming. Respondents were asked to check off the different ways their USAID team uses Agrilinks. Responses showed that Agrilinks is most often used to access resources useful for the design of new Feed the Future activities (11), and to learn about recent innovations that may be useful/applicable to their program (10). Other responses included to learn about upcoming training activities (10) and to learn about upcoming events (8). When asked what resources on Agrilinks the Mission Feed the Future Team finds valuable, 11 respondents said the most valuable links were to webinars (11) and how to implement cross-cutting activities, such as climate-smart agriculture, gender, nutrition sensitive agriculture, etc. (10). Other valuable resources 572 Peace Corps Knowledge Management and Learning Unit. “Process Evaluation of the Peace Corps/Senegal Master Farmers Program.” October 2014. Also see Feed the Future Press Release, June 28, 2013, Peace Corps Master Farmer Program Boosts Food Security and Economic Growth in Senegal. 217 included reports on recent research findings (9), and recent evaluations related to Feed the Future topics (6). The analysis of interview data from the field country visits with respect to IP use of FTFMS indicators and indicator data indicates that just 20% of the 53 implementing partners interviewed use these data for management and learning purposes (as discussed in the previous section); some of these respondents provided specific examples of how they use data from FTFMS indicators (also as discussed in the previous section). For example: This data is extremely useful to our day-to-day work. Even though as far as USAID is concerned it is a requirement, it is more important to us because it helps us in decision￾making processes…We also use this information during our annual planning process because throughout the year, the data that we have been collecting and reporting to USAID we are able to [use for] a learning event for the annual planning for the second year. – Uganda IP Interview data from IPs also shows that their custom indicators provide an important source of information for learning about the effectiveness of interventions. Based on interviews, some stakeholders are using CLA approaches to facilitate learning and are commissioning special studies, evaluations, or assessments to learn about and better manage their programs. This finding from field interview data is reinforced by a review of annual portfolio review presentations from each of the 19 focus countries. Beginning in FY 2014, Portfolio Reviews provide ample evidence that focus countries are learning and applying evidence from mid-term performance evaluations of their Feed the Future portfolio, impact evaluations, assessments, recent studies, and early findings from mid-term PBS surveys. The presentations provide evidence of applying learning from these sources to adapt their strategies, design new projects, and establish additional elements within their country level Results Frameworks. As referenced earlier, countries began reporting on learning and adaptation in FY 2014 in response to a new requirement from MEL, which requested countries to include this information in their Portfolio Review presentations. The MEL Knowledge Management team attends every Portfolio Review session to document key learning and to support the initiative’s learning approach. Strategy Change Memos that approve focus country requests for changes in value chains and ZOI coverage also provide evidence of learning and adaptation to improve programming at the country level. Examples of learning to support approval of strategy changes related to value chains include evidence presented by focus countries showing that current value chains do not adequately contribute to poverty reduction goals, or that there is low market demand for the value chain product. (Strategy Change Memos are discussed in greater detail under Question #1). 218 QUESTION 12b CONCLUSIONS 1. There are many sources of learning promoted under the MEL approach. These sources comprise the production of data from BFS-supported assessments, studies and evaluations, and mechanisms to support access to information, such as posting reports from these sources on Agrilinks, as well as from sources outside of BFS from stakeholder participants initiative-wide. MEL￾sponsored learning events and blog conversations are additional sources of learning. More recently, MEL has broadened its learning approach to capture data produced in the field through reviews of evaluations produced in-country and through annual Portfolio Review presentations. 2. A review of FY 2014 and FY 2015 portfolio review presentations shows that one of the most important sources of field-level learning under the MEL approach, that is used to strengthen the effectiveness of focus country strategies and activities and to design new activities, is evidence generated by recent evaluations, special studies, assessments, and surveys conducted in individual countries. 3. Agrilinks is an important resource for accessing learning and guidance that is relevant to its users for obtaining learning support in order to increase the effectiveness of current focus country programs and to design new activities. The utility and value of Agrilinks is strengthened by providing a variety of pathways for learning. 4. The team decided not to use data from online survey responses on the utility of the Feed the Future Learning Agenda. The team concluded that responses may not be entirely valid because the question was too general to obtain an adequate response – as evidenced by 40% of the respondents who replied “I don’t know.” A differently worded survey question, which specified products produced related to each of the Learning Agenda categories, may have yielded different and more useful results for understanding the utility of the learning agenda for improvement programming. 5. Plans for conducting additional evaluations on Learning Agenda topics, the development of actionable findings related to the Learning Agenda, and other emerging issues from performance evaluations will be an important source of ongoing learning to support improved programming. 6. FTFMS indicator data has different uses among stakeholders in Washington and in the field, and utility for different users varies by the type of indicator. FTFMS data has been valuable for learning at the initiative level in BFS, through aggregation of indicator data at the goal, objective, and IR levels to support assessments of trends across operating units and within specific areas of 219 results, and to compare results between countries and in different contexts. Evidence from portfolio reviews demonstrates that data from the PBS indicators generated through the recent interim surveys has prompted USAID mission staff to conduct assessments and evaluations, and deeper examination of data collected through other mechanisms, to understand population level results. Findings are being used to design future activities. Field interviews with IPs from the five focus countries visited by the evaluation team suggest that data from FTFMS IM indicators have less utility for learning and application than the custom indicators they design that are specific to that context, although some IPs provided concrete examples that illustrate how data from IM indicators are used. The tension between FTFMS standard and custom indicators continues, based on their differing utility at the initiative and field level, and based on the resources required to collect both types of indicators (discussed further in findings under 12.a.) QUESTION 12b RECOMMENDATIONS 1. Continue the practice that essentially treats each focus country, aligned country, and regional mission as an independent learning lab. The capture plan should include future evaluations and impact assessments commissioned by central offices of USG agencies that will be conducted on a more global level, on certain topics related to the learning agenda such as the effectiveness and impact of policy and regulatory changes. This supports the obvious next step, already discussed by the MEL Knowledge Management team, to synthesize and develop actionable findings that can be applied as appropriate to Feed the Future focus countries and regional missions. 2. To augment the follow-up already conducted by MEL/Knowledge Management, following their documentation of lessons learned and their application during annual portfolio reviews, identify and aggregate emerging evidence of effective use of learning related to specific learning topics across countries. Design evaluations to assess the effectiveness of new activities and strategies implemented in those countries for improving results that support Feed the Future objectives and goals. 3. Given the importance of Agrilinks as a learning resource, conduct a user interface assessment to determine if the organization of Agrilinks can be optimized to support greater accessibility to resources and improve their use.