Performance Evaluation Report Feed the Future Innovation Lab for Collaborative Research on Assets and Market Access (AMA IL) March 23, 2018 PERFORMANCE EVALUATION REPORT FEED THE FUTURE INNOVATION LAB FOR COLLABORATIVE RESEARCH ON ASSETS AND MARKET ACCESS This publication was produced at the request of the United States Agency for International Development Developed under Contract/Task Order Number: AID-OAA-1-15-00024/AID-OAA-T0- 16-000008. It was prepared independently by: Scott Rankin, Team Leader Ahmed Sidahmed, Technical Specialist Tamara Duggleby, Technical Specialist Contractor: Program Evaluation for Effectiveness and Learning (PEEL) ME&A 1020 19th Street NW, Suite 875 Washington, DC 20036 Tel: 240-762-6296 www.MEandAHQ.com Cover page: Shop selling drought tolerant maize seed developed by CIMMYT and used within Tanzania DT Seeds Project, Singida Town, Tanzania. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS EXECUTIVE SUMMARY i Evaluation Purpose i Project Background i Evaluation Design, Methods, and Limitations i Main Findings, Conclusions, and Recommendations i 1.0 EVALUATION PURPOSE AND QUESTIONS 1 1.1 Evaluation Purpose 1 1.2 Evaluation Questions 1 2.0 PROGRAM BACKGROUND 2 2.1 Overview 2 2.2 Request For Assistance, UC Davis Proposal,Theory of Change 3 2.3 Theory of Change 6 3.0 EVALUATION METHODS AND LIMITATIONS 7 3.1 Methodological Context 7 3.2 Selection of Sites and Projects for Evaluation 8 3.3 Further Considerations 10 3.4 Qualitative Research and Analysis 11 3.5 Quantitative Research and Analysis 12 3.6 Evaluation Limitations 13 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 14 4.1 Research Quality 14 4.1.1. Have the research projects developed scientifically valid and robust conclusions and professional-level outputs? In what ways did the portfolio of projects make progress and contribute toward the stated research objectives in the RFA? How did it diverge from objectives in the RFA? 14 4.1.2 How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement? 20 4.1.3 To what extent has work from previous iterations of the Lab been effectively incorporated and built upon? 24 4.2 Outreach And Dissemination 27 4.2.1 To what extent has the AMA IL and its research projects effectively analyzed, synthesized, and distilled research results into actionable information or recommendations? 28 4.2.2 What are examples of successful outreach aimed at achieving impact (or of adoption of recommendations)? 30 4.2.3 In what ways could research results be better communicated with different stakeholders, including USAID Missions, IPs, public and private sector partners, including technical and non-technical stakeholders? 31 4.2.4 In what ways have different partners (especially USAID Missions and the private sector) been engaged in the research process? What opportunities are there to increase this engagement? 31 4.3 Policy 34 4.3.1 In what ways was policy research relevant to current concerns, objectives, and needs of private and public sector stakeholders, USAID Missions, and BFS? 35 4.3.2 Which, if any, innovations, or research findings are most likely actionable? 38 4.3.3 What characteristics of these models or innovations make them more policy relevant and actionable? 39 4.3.4 Is there any evidence of changes to policy, programming, private sector products, or approaches because of research findings? In what ways, if any, did the project monitor these changes? 40 4.4 Capacity Building 43 4.4.1 How well have the projects and consortium of researchers identified and addressed academic and technical capacity needs of host country stakeholders? To what extent has this contributed to capacity building, broadly defined, in the host Country? 43 4.4.2 In what ways did the consortium of researchers and projects support the participation of the private sector? 45 4.4.3 What opportunities are there to better support inclusion of the private sector in market viable development solutions? 45 4.5 Program Management 48 4.5.1 How did the management entity effectively communicate and coordinate with research partners to achieve the objectives of the RFA? 48 4.6 Future Directions 50 ANNEXES 53 Annex A: Supported AMA Research Projects 2011-2017 54 Annex B: Summary of AMA-IL Research Project Output 58 Annex C: Unpacking the RFA, UC Davis Proposal, and Program Theory of Change 62 Annex D: Comparative Analysis of Index Insurance Products for Agricultural Production 71 Annex E: Lubango Drought Tolerant Maize Seed Value Chain 75 Annex F: AMA IL Evaluation Expression of Interest 78 Annex G: Evaluation Questions (as agreed to and detailed in Evaluation Protocol) 88 Annex H: Key Informant Interview and Focus Group Discussion Guides 91 Annex I: AMA IL Evaluation Travel and Scheduling Master Plan (August-September 2017 104 Annex J: Ghana: “Disseminating Innovative Resources and Technologies to Smallholder Farmers” (DIRTS) 109 Annex K: Lender Uptake and Use of the Insured Credit Product in Northern Ghana 111 Annex L: Potential for Integrated Programming and Learning—AMA in Tanzania 114 Annex M: AMA IL Capacity Building Within Host County Academic Community 116 Annex N: Evaluation Work Plan 118 Annex O: Outcome Measures Used to Frame and Guide Evaluation Approach 121 Annex P: Summary Description of Data Sources, Collection, and Analysis Methods Against Evaluation Questions 133 Annex Q: Agricultural Adviser Technical Overview of AMA IL Work Around Improved Seeds 141 ABSTRACT The United States Agency for International Development (USAID) funded the Assets and Market Access Innovation Lab (AMA IL) to advance knowledge and understanding of development approaches and technologies in order to increase rural households’ ability to acquire, protect, and effectively utilize productive assets. This evaluation assessed AMA IL’s overall program performance across five themes: research quality; outreach and dissemination; policy; capacity building; program management; and future directions. The evaluation focused on generating evidence to guide programmatic and policy decisions. The Evaluation Team (ET) used a mixed methods approach that included site visits to Ghana and Tanzania; 77 key informant interviews with key stakeholder groups, and nine focus group discussions— eight with community members and one with extension agents in Ghana. The results show that AMA IL supported high-quality and relevant research for strengthening the resilience of smallholder farming households across diverse topics, countries, and sectors. The ET found that the management entity is well positioned to support the research and to oversee a program as dynamic as AMA IL, and systematically build a valuable evidence base around new approaches and technologies to increase rural household resilience. However, the outreach and dissemination components, as well as the capacity building of local researchers and institutions, were lagging. The ET recommends that AMA IL further synthesize and consolidate lessons learned on the new technologies it supports across countries and develop a strategy for local research capacity building and awareness raising among stakeholders towards broad adoption and policy impact. ACRONYMS AA Associate Awards ADVANCE Agricultural Development and Value Chain Enhancement (Feed the Future￾funded activity in Ghana) AEA Agriculture Extension Agent AECT Africa Center for Economic Transformation Africa RISING Africa Research in to Sustainable Intensification for a New Generation AMA IL Agriculture Technology Transfer Project (Feed the Future-funded activity in Ghana) ATT Assets Technology and Transfer BASIS Broadening Access and Strengthening Input Markets and the Collaborative Research Support Program BFS Bureau for Food Security CBM Community-Based Marketers CEA Community Extension Agent CGIAR Consultative Group on International Agricultural Research CIMMYT International Maize and Wheat Improvement Center COP Chief of Party CRSP Collaborative Research Support Program DFID (UK) Department for International Development DIRTS Disseminating Innovative Resources and Technologies to Smallholders DT Drought Tolerant ET Evaluation Team EQ Evaluation Question EUA Exploring Underinvestment in Agriculture FIN-GAP Financing Agriculture in Ghana (Feed the Future-funded) FGD Focus Group Discussion GAN Global Action Network to Advance Index Insurance GAIP Ghana Agricultural Insurance Pool GCC Global Climate Change GLEE Global Learning and Evidence Exchange GSSP Ghana Strategy Support Program IBLI Index-Based Livestock Insurance ICED International Centre for Evaluation and Development ICT Information and Communications Technology IFPRI International Food Policy Research Institute IITA International Institute for Tropical Agriculture IL Innovation Lab ILRI International Livestock Research Institute IP Implementing Partner IPA Innovations for Partner Action IRB Institutional Review Board IRDP Institute of Rural Development Planning KII Key Informant Interview KLIP Kenya Livestock Improvement Program KM Knowledge Management M&E Monitoring and Evaluation ME Management Entity MOFA Minister of Food and Agriculture MOU Memorandum of Understanding NAFAKA NAFAKA Tanzania Staples Value Chain (Feed the Future-funded) NGO Non-Governmental Organization PEEL Program Evaluation for Effectiveness and Learning PI Principal Investigator RCT Randomized Controlled Trial RFA Request for Applications SARI Savanna Agricultural Research Institute (Ghana) SIMLESA Sustainable Intensification of Maize and Legume Systems for Food Security in Eastern and Southern Africa (Australian-funded) SMS Short Message Service ToC Theory of Change UAP UAP Insurance UC Davis University of California, Davis UDS University of Development Studies UK United Kingdom U.S. United States USAID United States Agency for International Development USG United States Government WFP World Food Program ZOI Zone of Influence i EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of the external performance evaluation of the Assets and Market Access Innovation Lab (AMA IL) was to provide empirical evidence of how the program effectively enhances its goals. Goals include: 1) advancing the knowledge and understanding of approaches and technologies to increase rural households’ ability to acquire, protect, and effectively utilize productive assets; and 2) improving the United States Agency for International Development (USAID) ability to understand the successes and failures of various tested and implemented development approaches designed to improve farmer livelihoods and productivity and spur agriculture growth. The evaluation intended to inform the development of future research agendas and include recommendations for future program directions. Specifically, the evaluation sought to determine: 1. The overall quality of the research program, including its success in meeting outreach and capacity building goals, and the opportunity for research outputs to enhance rural households’ ability to acquire, protect, and effectively utilize productive assets; 2. The effectiveness of the management model employed by the Innovation Lab in designing and implementing its research program; and 3. Priorities and suggestions for future research that might be undertaken by AMA IL. The audience includes USAID Missions, the Bureau for Food Security (BFS), implementing partner (IP) institutions, and stakeholders like government partners where USAID works, relevant private sector partners, academia, and other donors and development stakeholders. PROJECT BACKGROUND AMA IL is funded by USAID/BFS. It is implemented by a consortium led by the University of California Davis (UC Davis) as the management entity (ME). The program has projects in: Bangladesh, Burkina Faso, Dominican Republic, Ghana, Haiti, India, Kenya, Malawi, Mexico, Mozambique, Nepal, Senegal, Tanzania, and Uganda. The target population of AMA IL is agricultural sector actors not fully taking advantage of agricultural and financial technologies to reduce the yield gap and maximize their income potential. EVALUATION DESIGN, METHODS, AND LIMITATIONS The Evaluation Team (ET) used a mixed methods approach, including qualitative and quantitative data collection and analysis. Qualitative data were collected through 77 key informants. Sixty-six of these were with in-country key stakeholders (38 in Ghana and 28 in Tanzania) and 11 were with U.S.-based informants [6 people within the ME and 5 U.S.-based Principal Investigators (PIs) and researchers]. Qualitative data from across different stakeholder typologies were triangulated to help reach higher￾level program findings. Quantitative data were extracted from information in program documents and reports to assess research quality, program outputs, and outcomes where appropriate. The ET reviewed relevant program documents and made site visits to Ghana and Tanzania. Gender-disaggregated approaches were utilized whenever possible. MAIN FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The evaluation’s findings, conclusions, and recommendations are summarized in the table below: ii Findings Conclusions Recommendations Evaluation Domain 1 – Research Quality As of mid-September 2017, this iteration of AMA IL had produced a combined total of 179 papers (33 academic papers, 36 working papers, and 110 policy briefs). This number will certainly rise significantly as many research activities reach completion. The documents were of high quality and AMA IL’s program research portfolio has responded to the identified research issues coherently, creatively, and in a scientifically rigorous way. The portfolio has been assembled strategically to support a variety of different, yet complementary, approaches around its key areas of focus—notably, index insurance, drought tolerant seeds, agricultural extension, and the application of satellite technologies as tools for risk minimization. Individual research projects touched upon multiple issues and different permutations of issues. The quality of research undertaken through this iteration of AMA IL is high. AMA IL’s research is scientifically rigorous and transparent and is systematically building a valuable evidence base around several new, innovative technologies thought to be well￾suited to improving rural households’ outcomes ability to acquire, protect, and effectively utilize productive assets. Overall, it is recommended that USAID support a further iteration of the AMA IL to advance progress in its key areas of innovation. Knowledge areas of future iterations of AMA IL deserving of further inquiry include: • Index insurance capacity to leverage credit access for smallholders; • Potential to reduce index insurance cost through coupling with other risk management approaches; • Development of minimum standards for index insurance to help ensure reliability and relevance; • Consideration of social protection transfer schemes within the overall picture of resilience building; and • Investigation of optimal risk management portfolios as smallholders accumulate resources. (Further examples are cited in Section 4.6 – Future Directions.) AMA IL’s high-quality research often appears undermined by insufficient attention and strategy applied to stakeholder engagement, outreach, and dissemination. Of the research activities reviewed in depth by the ET, none were able to persuasively describe their stakeholder landscape clearly. In particular, there was an absence of engagement of the senior officials who would be central in facilitating the use of research results for policy consideration and impact. This is thought to undervalue the importance of cultivating relationships and building the capacity of senior officials to comprehend and advocate on a technology’s behalf. Although AMA IL has worked for the past two years to bolster its outreach and dissemination, more needs to be done to strengthen the overall communications capacity of the program. While research quality is high, the ET has concerns that its potential value and contribution are prone to being undermined due to insufficient consideration of the stakeholder landscape, within which the research occurs. Any future iteration of AMA IL should be clearer in its articulation of approaches and expected outcomes of dissemination efforts. Achieving “development impact” is a long-term process and investment that might not yield results for years to come. The ET recommends that AMA IL develops a considered and strategic approach that aims to attain buy-in and build ownership of stakeholders, both in￾country but also externally, through different communities of practice partnered with AMA IL in the positioning of policy impact. Maximizing the potential contribution of satellite weather technology to production and rural resilience is an area where AMA IL is positioning to be a thought leader. In particular, efforts to reduce basic risk within index insurance products and better ensure access for smallholder farmers to the best available weather data AMA IL proactively works to keep itself at the cutting￾edge of technological change, utilizing satellite technologies and Information and Communications Technology (ICT), including consideration of how technology can best be utilized to support smallholder farming. This is a very important contribution given the rapid change of technology and the limited (or non￾AMA IL should deliberately work to promote itself as a “go to” research body for researchers interested in technological change and its potential to address constraints that affect resilience within poor farming communities. Intensified and strategic partnerships with existing AMA IL private sector partnerships—such as that with Ignitia, a Swedish weather satellite company iii Findings Conclusions Recommendations are knowledge areas that are greatly valued from those participating in randomized controlled trials (RCTs). existent) capacity of developing world governments to understand its potential. invested in AMA IL’s areas of focus—should also be explored. The ET observed considerable variability in the degree to which gender issues were integrated and potential “gender learning” maximized within different research approaches. While some research demonstrates high gender focus, for the most part, gender was addressed in an ad hoc manner within the overall AMA IL research approach. This reflects the lack of priority given to the issue. Given that women and their dependents face different and multiple vulnerabilities and are known to be disproportionately affected by household-level shocks, this feels like a lost opportunity. Cost was cited as justification for not adding a stronger gender dimension to research activities. While women were included and active in all of the research projects reviewed by the ET, there was no consistent effort made to ensure the depth of gender￾disaggregated analysis suggested within the Request for Applications (RFA). In the areas visited by the ET, women and their dependents faced different and multiple vulnerabilities and were commonly among the poorest cohort within the community—especially female-headed households. While there are likely additional costs involved in expanding a research approach to better understand gendered impact, given the centrality of gender issues to entrenched poverty, this additional investment would be justified. Gendered approaches remain relevant and should be more explicitly demanded of grantees in any future phase, even if they entail higher cost. Greater emphasis should be placed on analysis and strategy for ensuring that gender aspects and differences are fully understood and optimized through the research process. Examples of issues relevant to any research context that should be mainstreamed within AMA IL’s scope include: a) workload factors related to adoption of new technologies; b) household budget management and decision-making; c) gender issues as they relate to accessing credit and other financial services; and d) women’s mobility and how that affects awareness and adoption. Evaluation Domain 2 – Outreach and Dissemination The AMA IL ME utilizes a range of approaches to support its outreach and dissemination objectives. The ME’s appointment of both an Outreach Specialist and Strategic Communications Manager has significantly enhanced the capacity for proactivity in the outreach and dissemination space, as seen through the greater presence of the program in webinars, regional meetings, and efforts to engage policymakers in Washington, D.C., as well as through more targeted and systematic outreach towards both USAID and the World Bank, participation within Global Learning and Evidence Exchanges (GLEEs), and strategic identification of regional knowledge management (KM) partners such as the International Centre for Evaluation and Development (ICED). AMA IL’s steps to enhance outreach and dissemination capacity have been both timely and strategic given that much of the research portfolio is now moving towards completion and final generation of results. This investment has facilitated more strategic and proactive identification of audiences and opportunities relevant to AMA IL core areas of interest at the regional and global levels. While the enhanced capacity for outreach and dissemination has already achieved considerable results in terms of web presence, outreach, and “big picture” policy impact positioning, the opportunity exists to provide clearer direction and more support to researchers to better manage policy positioning within their national research contexts. AMA IL should develop guidelines that support researchers to better identify and engage their stakeholder landscapes. These guidelines should be complemented by direct support by the ME during research to help ensure that findings are disseminated to key audiences, especially policymakers. Both the RFA and UC Davis were explicit in stating the importance of research being positioned to achieve policy traction; however, there is ambiguity within both documents as to whether this was intended to occur at While the importance of “generalizable” research within ILs is noted as central to their modus operandi, this perspective appears to contribute to an underplaying of effort by researchers to achieve policy traction within The ME and researchers should work collaboratively to achieve policy traction. Together, they should adopt approaches that better reflect the reality that policy traction is generally most readily achieved within the iv Findings Conclusions Recommendations the national level (where research was undertaken), higher levels, or “either/or.” This ambiguity is reflected in an irregular and uneven approach to outreach and dissemination within the countries where research occurs. While all researchers intend to disseminate final results locally, the ET saw only limited strategic identification and/or engagement of policy influencers at the national level during research. the environment that the research inevitably carries most relevance in—the country where it occurred. Adoption of research findings within the country of research provides important evidence of research relevance and can be used as leverage in terms of engaging other potential knowledge users—particularly in neighboring countries that share common traits. country of research. This is due to contextual relevance, local awareness of approaches, capacity building that has occurred, and accessibility of policy influencers. Insufficient local stakeholder mapping was observed across all activities reviewed in depth. Local partners are primarily restricted to those on the ground that are mandatorily required for permissions and technical research support. In both countries, senior government officials expressed the opinion that more strategic engagement of the Government would be valuable in terms of helping the Government understand the research purpose, approach, and findings. This situation relates in large part to structural issues, notably researchers not being sufficiently present in country to develop the types of relationships necessary to position research for policy impact. At the same time, the team at UC Davis is too distant and too stretched to appreciate the full texture of the research context at the local level, the relationships needing to be cultivated, and other opportunities arising through the research period. Absence of a detailed stakeholder map or road map for achieving policy dialogue—at the national level or beyond—is seen to be a weakness within the research protocols of most AMA IL projects. Furthermore, formal adoption of research within its national research context is viewed by the ET as an acutely important platform for broader promotion of the validity of research to other similar contexts. While there is a perspective that senior officials should not be engaged until technologies are adequately tested, it is the perspective of the ET that there is more to gain than to lose by providing policymakers with an opportunity to observe and participate in the research journey—especially given that the research should, by definition, be of acute relevance and interest to local policymakers. Policy positioning should be a core aspect of research work. The ET recommends as a matter of process that each research activity, with support of the ME, should: • Outline a detailed description of the stakeholder landscape at the national level, including a plan, roles, and responsibilities for the engagement of different actors on that landscape; • Work with the Government to ensure participation of relevant senior representatives both from sectoral ministries and from agricultural research bodies, and nominate a contact point for the research team to communicate through and strategize with in relation to knowledge dissemination; and • Pre-identify a realistic number of key points within the research cycle where formal information sharing effort occurs, ensuring key stakeholders have the opportunity to provide input into the research approach. Engagement of U.S. Missions appears to be limited and affected by structural issues related to the PI only rarely being in country. This relationship is likely also impacted by the immediate workload of Mission staff. Another challenge is that some Missions, such as Tanzania, host a very large number of ILs, putting even more pressure on strategic dissemination and management of IL-generated knowledge. Engagement of Missions in ILs is a complex challenge, affected by the dynamic, heavy, and ever-shifting workload common to most Missions. Generally, the different parties have the best of intentions, but given the heavy demands of PI time when in country these are thwarted by the practical day-to-day realities of workload and finite windows of opportunity. AMA IL outreach staff should aim to reach agreement with Missions on a contact point and an agreed model of and frequency for communication of updates. USAID Washington staff responsible for award management should help facilitate Mission-focused outreach and dissemination strategy given its more developed understanding of Mission structures, dynamics, and personalities. v Findings Conclusions Recommendations In both Ghana and Tanzania, various Feed the Future￾funded development programs closely intersect the subject matter of AMA IL research and often program within those technical areas. In Ghana, the Agricultural Development and Value Chain Enhancement (ADVANCE) project is actively working to promote index insurance, including systems for the sale of insurance policies across the same geographic area as both AMA IL-supported research activities. In Tanzania, NAFAKA Tanzania Staples Value Chain (NAFAKA) is undertaking programming that is relevant to each and every aspect of AMA IL research in the country. During an interview, the NAFAKA Chief of Party (COP) expressed a clear interest in better understanding the research approach and what is being learned—and avoiding duplication of effort through his own program. Similarly, discussions with various public and private stakeholders signaled that research results could be better communicated and would be more easily received if their communication occurred through partnership and close collaboration between different stakeholder groups—USAID Missions, public sector service and research agencies, private financial institutions, and development programs. AMA IL should work with selected Missions (in countries with a heavy IL presence) to identify a workable approach that facilitates a clear and dynamic information flow between different USAID-supported actors in country. Such an approach should generate knowledge of importance to Missions, development partners, and researchers themselves. It should also enhance opportunities for adoption. It is also feasible that Feed the Future-funded development projects could be AMA IL research collaborators, providing cost-efficient locations from which research could occur. Such an approach within AMA IL would require slight adjustments in expectations that accommodate the interests and desired outcomes of different parties. Evaluation Domain 3 – Policy AMA IL has worked hard to ensure the relevance of its research portfolio to both the global and national agricultural research agenda as it relates to better understanding resilience and barriers to broad-based growth. All research investigated during this evaluation reflected issues of national concern, even though these needs were not always articulated in existing national policy. The research portfolio is also aligned with and supportive of Feed the Future objectives with all research able to be seen and understood through the lens of the Feed the Future results framework. AMA IL systems for screening proposals seem effective in ensuring a portfolio that is relevant and broadly in alignment with the policies of host countries and Feed the Future. The context-relevant nature of the research being generated in the Feed the Future Zone of Influence (ZOI) positions AMA IL to have significant policy impact in these countries and regions, although as described above, it is felt that the overall policy engagement approach lacks sufficient proactivity and strategy to maximize such opportunities. There is also opportunity for research to better leverage the support of other local actors. AMA IL should be strategically clearer on how it intends to influence policy, both within and beyond countries of research. While robust evidence is invaluable in supporting influencing efforts, cultivating relationships with policymakers and supporting them to be present in the research process is an important aspect needing greater attention. The capacity of the private sector to strengthen policy objectives appears variable. In Ghana, both research activities partner with Ghana Agricultural Insurance Pool (GAIP), which is widely regarded to be weak and faltering. In Tanzania, collaboration with UAP Insurance (UAP) through the drought tolerant (DT) seeds Given the cutting-edge nature of index insurance and its constant evolution as a product of relevance to smallholder resilience, if policy traction is to be achieved there is a huge need for ongoing capacity development among potential stakeholders interested in the area. One significant challenge is that index insurance enjoys More strategic identification and inclusion of private sector entities that have the potential to strengthen national capacity in AMA IL areas of interest is needed. This is especially important in those domains that are truly innovative and cutting-edge, and where very little existing capacity is present, such as index insurance and vi Findings Conclusions Recommendations research has been an important factor in strengthening policy impact potential, given UAP’s enthusiasm to carry forward the “insured seed" product promoted by the research. little support or ownership from many governments at this point in time. This makes it even more important that private sector actors are supported to develop their interest and capacity in the model. the application of satellite data technology towards development outcomes. AMA IL is a key actor within the Global Action Network to Advance Index Insurance (GAN)—an important vehicle for progressing the understanding and policy potential of index insurance. AMA IL research provides important raw material and evidence from which GAN is able to further advance thinking and discuss key issues and options for the application of index insurance. Given its global profile, GAN is well placed for the KM of research findings, and to support adaptation and adoption beyond the country of research. Through its contribution to GAN and, more generally, its role as a thought leader in relation to index insurance, AMA IL is well-positioned to influence higher￾level thinking on the potential and applicability of index insurance to building the resilience of smallholders (see p. 27 and Annex D for analysis of different index insurance products trialed). The potential for policy impact correlates to the quality of KM and dissemination efforts. AMA IL should : • Continue to promote development of competitive index insurance schemes that ensure agriculture insuring companies the legislative and policy frameworks, financial and technical capacities, and skilled human resources to develop and market viable products directly to local dealers and farmers; • Support the development of minimum standards and government quality assurance measures on index insurance contracts to guide global product development and roll out; and • Consider how to better integrate the potential of satellite technologies within agricultural policy, including improved weather forecasting and updates in satellite imagery maps. Evaluation Domain 4 – Capacity Building Data provided by AMA IL highlight that relatively few host country students have received support through the program. To date, training has mostly benefited U.S. and European students. For example, 44 percent of the 12 long-term trainees are U.S. citizens and 33 percent are European citizens. U.S. citizens also received 44 percent of the 117 trainings, compared to just 12 percent being African students despite the overwhelming majority of AMA IL research occurring in countries of Africa (see Section 4.1.1 and Annex E). Of the formal capacity building opportunities offered through AMA IL, more than 50 percent of students are from the U.S. and just 10 percent are from Africa. Supporting the development of local capacity within the public sector is critical. Given that the overwhelming majority of research is located in Africa, more emphasis should be placed on strategies that support greater inclusion and support of African students within the program. While it is difficult for the program to directly do this in a tangible form, creative solutions should be sought through better partnering with local institutions and enticement for Award and Sub-Award institutions to connect African students with AMA IL research. Much of the capacity building occurring through AMA IL relates to a practical need for day-to-day research implementation capacity. This has especially been the case in new technological areas such as index insurance, weather-based systems, information technology (IT), and DT seeds, where little or no existing capacity was While capacity building is always important, it becomes more important in fields that are truly innovative, since very little (or no) core capacity exists. Given the innovative nature of much that AMA IL focuses upon, and the fact that the public sector is commonly weak and/or financially challenged, private sector capacity AMA IL needs to strategically identify and more actively include private sector partners as a string to the bow of strengthening national capacity in AMA IL areas of interest, notably those domains that are truly innovative and cutting edge, and where very little existing capacity is present within government—such as index insurance vii Findings Conclusions Recommendations available. While skills have been advanced in these areas, the capacity to sustain adoption of these new interventions at both public and private sector levels looks problematic given that there has been little systemic change. development is critical. Weaknesses within the private sector have the potential to undermine innovative approaches, as might be the case with GAIP in Ghana which enjoys little confidence across the research and development communities. and the application of satellite data. It should also provide more support to developing local capacity within the public sector, especially in equipping them to be advocates of innovation, notably in relation to index insurance understanding and adoption. Agricultural extension departments commonly perform an important function within AMA IL research, and also offer an ongoing mechanism for knowledge extension. In Ghana, an aspect of Disseminating Innovative Resources and Technologies to Smallholders (DIRTS) research has been investment in the capacity development of community extension agents (CEAs). Communities value and utilize the capacity of CEAs, in large part generated through DIRTS-funded training. However, it currently appears that the CEA model will cease to operate when DIRTS research is completed, given the lack of awareness and support for the model at the national level. AMA IL interest in community-based extension (notably through Ghana DIRTS) is an important “capacity investment” showing potential but is also one which will require a focused advocacy effort to progress and consolidate within policy. More generally, the important role played by extension departments in the dissemination of AMA IL-generated knowledge is highly significant—and an important point of capacity investment. However, the critically important research-enabling role played by extension workers in the achievement of research outcomes appears to be undervalued in terms of adoption potential. AMA IL should continue to develop evidence related to the contribution made by informed extension agents (government or community-based) in knowledge transfer, dissemination, and adoption. An overhaul of the global agricultural extension system approach is clearly well beyond the means of AMA IL, but it is an important area of study nonetheless, given that extension workers are central to many of the research outcomes being achieved. AMA IL already has its toe in the water of this subject, recognizing its centrality to dissemination and adoption of new technologies. Evaluation Domain 5 – Program Management Over the course of implementation, the UC Davis-based AMA IL ME has taken steps to strengthen its capacity for effective coordination and communication with all stakeholders, including research partners. These changes have resulted in strengthened capacity for oversight of most aspects of the program’s theory of change, including the capacity to engage and support research partners. The ME team composition appears to be finely tuned to the complex needs that exist in supporting a diverse research program across 15 countries, and multiple regional and global fora. This fine-tuning is based in clarity of role and function, as well as the specific skills of different staff to fulfill their designated function. The ME’s role in overseeing and guiding implementation, and in ensuring effective KM that facilitates research findings being accessible to all who are interested and stand to benefit needs to be underpinned by appropriately skilled staff. Future RFAs should explicitly reflect this importance and insist on adequate staffing within the ME alongside appropriate accountability measures that help ensure USAID return on investment. Currently, local-level outreach and dissemination is the responsibility of the PI and his/her research team. Based on observations of the ET, insufficient priority is placed on this aspect of research—in large part because outreach and communications are neither a strength nor an interest of some researchers. While strengthening of ME capacity over the past two years has been significant, it is the opinion of the ET that there remains room for greater outreach and communications support to be directed towards ensuring strong KM at the national level, as described above. Strategic agreements need to be reached between researchers and the ME on how outreach and dissemination will occur and contribute to program objectives. Clear articulation of roles and responsibilities within that approach is also required. viii Findings Conclusions Recommendations Evaluation Domain 6 – Future Directions A dynamic of AMA IL over its iterations has been a deliberate effort to bring increasing clarity and definition to its knowledge areas by testing and honing understanding of the potential of certain approaches. Much of this honing has occurred through adjustments being made to bundles of technologies, recognizing that no one technology is likely to work in isolation. The efficacy of this approach can be seen in the subtle evolution of strategy within different research approaches. Index-Based Livestock Insurance (IBLI) and DIRTS are both examples of research efforts that are systematically generating and fine-tuning approaches that facilitate willingness among smallholders to take risk. Similarly, the Credit and Insurance research in Ghana has identified an important opportunity in utilizing insurance to open up lines of credit to smallholders who previously had little to do with the formal banking system. Over the course of implementation, ME capacity to astutely manage the mass of knowledge being generated with nuance has advanced considerably. This allows for more strategic KM and dissemination of the advances in understanding that are being achieved. Better targeted research and more dynamic capacity for dissemination positions AMA IL to achieve greater policy traction as its evidence base solidifies. This strategy of honing and fine-tuning is especially logical for AMA IL given that very little prior research exists in its areas of focus, and that there is subsequently little core understanding of their potential among policymakers. This makes it even more imperative that potential “solutions” are evolved and well-tested if they are to achieve traction at the policy level. Further systematic evolution of the AMA IL program portfolio will allow already advanced technologies to be further fine-tuned and “proven,” which, in turn, should better position them to be more actionable at the policy level. Different, yet related research activities also present the potential for the program whole to be greater than the sum of its parts since learning across different research activities can commonly be assessed and triangulated to reach significant conclusions regarding where the research agenda needs to head next. The following are some areas of research that would further the understanding of how best to strengthen the resilience of agriculture-reliant households. • Furthering understanding of the potential for index insurance to be used as a tool to help leverage banks in a way that encourages them to extend lines of credit, at reduced interest rates, to smallholder farmers. • Deeper investigation of what can be done to reduce the cost of insurance, including coupling insurance with other risk-management instruments/technologies, options for the underwriting of insurance, and study of whether they increase farmer investments. • Study of how the optimal risk management portfolio changes over time as a poor farmer begins to accumulate the resources necessary to rely less on insurance and more on the cheaper options of savings and contingent credit. • Further exploration of the concept of minimum standards for index insurance, including assembly of mechanisms, tools, and procedures that can be used to ensure that insurance products marketed and sold to smallholder farmers are of sufficient quality. • Further investigation of the potential for insurance products to enable farmers to protect assets, such as livestock, rather than wait for disbursements to replace them after they are lost. • How to optimally design subsidy schemes so that they offer beneficiaries the strongest incentives for investment and income growth. 1 1.0 EVALUATION PURPOSE AND QUESTIONS 1.1 EVALUATION PURPOSE This is an external performance evaluation of the Feed the Future Innovation Lab for Collaborative Research on Assets and Market Access (AMA IL). The AMA IL program is funded by the United States Agency for International Development/Bureau for Food Security (USAID/BFS) and implemented by a consortium of partners led by the University of California Davis (UC Davis) as a management entity (ME). AMA IL started in April 2012 and is due to conclude in September 2018. The total project funding for AMA IL is $25,000,000. The evaluation of AMA IL was conducted between July and October 2017, by a team assembled by ME&A under the Program Evaluation for Effectiveness and Learning (PEEL). The Evaluation Team (ET) included three core members—Mr. Scott Rankin (Team Leader and Evaluation Specialist), Dr. Ahmed Sidahmed (Agriculture Specialist), and Ms. Tamara Duggleby (Financial Services Specialist). The team was supported by Mark Kwamei Offei in Ghana, and by Festo Ngulu in Tanzania. The purpose of the evaluation was to assess the overall effectiveness of the AMA IL program, and to identify how the program has supported continuous improvement for USAID’s work in better understanding the successes and failures of various tested and implemented development approaches designed to improve farmer livelihoods and productivity, and spur agriculture growth. Specifically, the evaluation sought to determine: • • • The overall quality of the research program, including its success in meeting outreach and capacity building goals, and the opportunity for research outputs to enhance the ability of rural households to acquire, protect, and effectively utilize productive assets; The effectiveness of the management model employed by the Innovation Lab (IL) in designing and implementing its research program; and Priorities and suggestions for future research that might be undertaken by AMA IL. The audience for this evaluation includes USAID Missions, BFS, implementing partner (IP) institutions, relevant private sector partners, and other donors and development partners. USAID looks to AMA IL and its research portfolio to shed light both on applicable policy and programming recommendations and institutional innovations. The evaluation will also provide information and recommendations for a future research agenda, and suggestions for future program directions for BFS and Missions to inform country- and institutional-level project effectiveness. 1.2 EVALUATION QUESTIONS Evaluation questions (EQs) are presented in the Expression of Interest (EOI) (see Annex F). Following an iterative process between the ET and USAID, the initial EQs were modified and finalized in the evaluation protocol, which was approved by USAID in July of 2017. Below are the six EQs addressed in this evaluation and the final full set of questions and sub-questions is presented in Annex G. 1. Research Quality 1.1 Have the research projects developed scientifically valid and robust conclusions and professional-level outputs? In what ways did the portfolio of projects make progress and contribute toward the stated research objectives in the Request for Applications (RFA)? How did it diverge from the objectives in the RFA? 2 1.2 How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement? 1.3 To what extent has work from previous iterations of the lab been effectively incorporated and built upon? 2. Outreach & Dissemination 2.1 To what extent has the AMA IL and its research projects effectively analyzed, synthesized, and distilled research results into actionable information or recommendations? 2.2 In what ways have different partners (especially USAID Missions and the private sector) been engaged in the research process? What opportunities are there to increase this engagement? 3. Policy 3.1 In what ways was policy research relevant to the current concerns, objectives, and needs of private and public-sector stakeholders, USAID Missions, and BFS? 3.2 Which, if any, innovations, or research findings are most likely actionable and policy￾relevant? 3.3 What characteristics of these models or innovations make them more policy-relevant and actionable? 3.4 Is there any evidence of changes to policy, programming, private sector products, or approaches as a result of research findings? 4. Capacity Building 4.1 How well have the projects and consortium of researchers identified and addressed academic and technical capacity needs of host country stakeholders? 4.2 In what ways did the consortium of researchers and projects support the participation of the private sector? 4.3 What opportunities are there to better support the inclusion of the private sector in market-viable development solutions? 5. Program Management 5.1 How did the ME effectively communicate and coordinate with research partners to achieve the objectives of the RFA? 6. Future Directions 6.1 What SPECIFIC technical areas and research topics merit new or continuing research investment particularly with regard to the new Global Food Security Strategy? 2.0 PROGRAM BACKGROUND 2.1 OVERVIEW AMA IL is one of 24 ILs currently under implementation across the United States (U.S.). The program builds on previous work undertaken through the Broadening Access and Strengthening Input Markets and the Collaborative Research Support Program (BASIS), which was first initiated at the University of Wisconsin in 1996. In the 2011 AMA IL RFA, USAID expressed interest in “better understanding the root causes of persistent poverty and food insecurity, and how to effectively increase the capacity of poorer households to engage in and benefit from agriculture-led growth.” Accordingly, the RFA stated, “The AMA IL will contribute to USAID’s strategic objectives by conducting rigorous policy- and programming-relevant research in defined areas of inquiry: • • By building capacity of host country institutions and maximizing training of host country scientists By achieving impact through the development, testing, and adoption of innovative approaches 3 • Through the shaping of development discourse.” 1 AMA IL assembled a portfolio of projects that focused on key topics designed to bridge the gap between what is possible given currently available technologies and the realistic experiences of most developing country agriculturalists. The main Leader Core Award for AMA IL, “Program in Assets and Market Access,” (2012-2018) has also been supported with three Associate Awards (AA) that include: 1. AA #1: “Advancing Index Insurance by Closing the Gap Between Knowledge and Implementation” (2014-2019), designed to support the responsible scaling of index insurance for agricultural development, food security, and resilience applications in developing country contexts. 2. AA #2: “Achieving Development Impact With Complementary Stress-Resistant Seed & Financial Technologies” (2015-2019), designed to support the development of drought tolerant (DT) maize varieties in collaboration with the International Maize and Wheat Improvement Center (CIMMYT), while also ascertaining if their rate of adoption and impacts can be enhanced by bundling them with a financial technology (index insurance) designed to pay out in the face of severe drought events under which even DT seeds fail. 3. AA #3: “Feed the Future Evaluating the Effectiveness of Programs That Enhance the Economic Resilience of Vulnerable Populations” (2016-2020), allowing expanded understanding of the nature of resilience through focus on strategies to address vulnerability and unprotected risk. AMA IL’s target population consists of agricultural sector actors who are not fully taking advantage of agricultural and financial technologies to reduce the yield gap and maximize their income potential. AMA IL implemented projects in 15 countries, including: Bangladesh, Burkina Faso, Dominican Republic, Ethiopia, Ghana, Haiti, India, Kenya, Malawi, Mexico, Mozambique, Nepal, Senegal, Tanzania, and Uganda. 2.2 RFA, UC DAVIS PROPOSAL, AND THEORY OF CHANGE The RFA released in late 2011 outlined a broad framework within which it expected AMA IL to operate, making it clear that “AMA IL was not expected to cover each topic comprehensively, or even to necessarily have a research activity in each of the areas mentioned.” Instead, the areas of inquiry outlined in the RFA were to be regarded as illustrative of USAID’s interests, setting parameters within which the Recipient (UC Davis) was expected to develop and model a coherent program of high quality research. 2 Those parameters included expectations that AMA IL would: • • • Mobilize U.S. university expertise in support of USAID’s strategic objectives around food security, agricultural development, and rural resiliency; Generate innovations that support inclusive agriculture-led economic growth through enhanced access to markets, improved access to financial and risk management services, increased technology adoption and climate change adaptation, and increased resiliency of both men and women in vulnerable households and communities; Conduct rigorous policy- and programming-relevant research in defined areas of inquiry: Inclusive Market Access; Risk Management and Resilience; and Rural and Agricultural Finance; and 1 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 7. 2 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 14. 4 • Ensure gender and impact evaluation as important cross-cutting themes needing to be recurrent throughout the portfolio regardless of specific substantive issues addressed. More importantly, USAID stated explicitly that while its program statement was being written broadly enough to allow some intellectual flexibility in the research program, it preferred AMA IL focus its research portfolio along two or three themes in order to build a critical mass of work in these areas. Table 1 presents a summary of the program’s evolution from the RFA to the detail of what UC Davis specified in their successful proposal. A more detailed analysis of each RFA, the UC Davis proposal, and the program Theory of Change (ToC) is attached in Annex C. Table 1: Comparing RFA and UC Davis Proposal Areas of Focus Proposed in the RFA Detailed in UC Davis Proposal to Serve as Management Entity Inclusive Market Access and Engagement • • • • • • • • • Explore the role of social protection in enhancing the poor’s capacity to engage in economic opportunity. Identify methods by which agricultural-led economic growth programming can complement efforts to build the resiliency of the poor by enhancing the entry points for the rural poor, both men and women, in value chain activities. Research ideas for promoting greater benefits for value chains traditionally done by women, and greater participation of women in higher￾value value chains. Better understand constraints to improved outcomes in labor markets, especially issues of market integration or interaction and the most relevant policy and programmatic interventions in non-labor markets. Consider the value and opportunity costs of unpaid family labor, especially that of women and children. • • • Target value chains at women and small-scale farmers. “Cargo Net” programs that transfer assets and lift resource-poor households over critical thresholds to participate in agricultural growth. The impact of larger-scale farming on poor households via the Labor Market. Risk Management and Resilience Further develop insurance instruments that help farmers and other rural households better manage risk and improve productivity and incomes. Explore the role of government in setting policy and providing safety net programs that do not distort markets or displace traditional coping mechanisms. Support appropriate public goods and institutions that enable private sector provision of risk management and mitigation mechanisms. Expand engagement on risk management by: - Adding pilots that complement Feed the Future implementation programs in various areas (i.e., value chain programming); - Facilitating Mission integration of insurance into Feed the Future and Global Climate Change (GCC) programming through • • Progress work commenced under the previous AMA Collaborative Research Support Program (CRSP), with the aim of consolidating a critical mass of learning on the use of innovative insurance contracts that transfer correlated risk out of small farming systems. Research focused on: - - - Intelligently designing index contracts that minimize basis risk, protect against price fluctuation and crowd in agricultural credit supply; Exploring gender-smart targeting of index contracts; Creating the science to cost￾effectively scale-up index contracts without undercutting 5 Areas of Focus Proposed in the RFA Detailed in UC Davis Proposal to Serve as Management Entity outreach and technical support; and - Designing and testing the scale up of insurance products to broader areas (i.e., beyond the original areas for which micro data were available for product design), for example, by utilizing agro-ecological mapping and crop modeling. their insurance and development value; and - Pricing insurance to incentivize climate change adaptation investment. Rural and Agricultural Finance • • • • • • • • • • • • Explore and develop innovations that promote productive linkages between rural financial sector deepening and development objectives. Explore lending practices and procedures that would incentivize clients to make investments that build their resiliency to climate and other shocks. Enhance access and capacity to weather and climate data. • • • • • • • Prioritize research and impact evaluations that explore small farm finance options: - Self-finance and savings (including savings-secured lending); - Self-collateralizing loans, including those for tree crops; - Index insurance and joint liability credit; and - Smart subsidies. Gender Investigate how gender relations will affect the achievement of sustainable results under the CRSP. Investigate how proposed results will affect the relative status of men and women. Consider the different roles of men and women, but also the relationship and balance between them and the institutional structures that support them. Impact Evaluation Contribute to USAID’s (and the development community’s) body of evidence on the effectiveness of various approaches to development challenges relevant to food security, poverty reduction, and agricultural development, especially where evidence is comparative. Develop and test innovations as input for USAID food security programming design and refinement. Support efforts to refine USAID’s monitoring and evaluation (M&E) methodology relative to food security. Pilot project and impact evaluations that build on programs conceived and implemented by USAID Missions or other development agencies. Fundamental research that explores the nature of constraints and problems that require further understanding before their solutions can be intelligently designed. Capacity Building Design research activities to maximize long￾term degree training for host-country students, as well as shorter-term training for researchers and practitioners. Incorporate a variety of approaches for building capacity including degree programs, distance learning, web-boards, or other communications technologies and communities of practice that link researchers, policymakers, and development practitioners struggling with similar topics. Ensure balance in access to training and capacity building opportunities in terms of the Multi-faceted approach recognizing financial constraint to broad-based training of host country academics within U.S. universities. Ensure all principal investigators (PIs) initiate capacity building plans. Empower host country PIs as key partners in research, while also working to ensure institutional capacity is strengthened. Work to ensure opportunities for women students. 6 Areas of Focus Proposed in the RFA Detailed in UC Davis Proposal to Serve as Management Entity gender of trainees and participants. Achieving Development Impact • Achieve development impact through efforts that: - Promote adoption of policy and/or programming recommendations that research activities generate; - Ensure policy/programming relevance; - Promote adoption of recommendations and subsequent development impact; and - Engage policymakers, USAID Mission staff, private sector representatives, and other stakeholders as appropriate—even in early stages of the research design and implementation. Oversee a three-step process linking research to long-term development impact: • Obtain research results; • Distill research results into recommendations for policy and programming; and • Enhance the probability that recommendations are adopted. Steps proposed to ensure dissemination: • Ensure funds for translation of documentation; • Appoint dedicated outreach person; • Support “evidence” summits, webinars, and other strategies for dissemination; • Ensure research grant solicitation clearly delineates researcher obligations to engage in these tasks; and • Develop strong and clear linkages to USAID in particular, as well as other key entities. 2.3 THEORY OF CHANGE In response, UC Davis details the specific approach proposed within the vast landscape laid out within the RFA, identifying the key topics that it planned to place emphasis on. Four realms were proposed to help provide balance and ensure AMA IL achieves development impact: • • • • Generation of New Development Knowledge needed to advance the Feed the Future agenda through rigorous, competitive selection of projects from the very best development scholars; Design and Rigorous Evaluation of Pilot Projects that explore the complementarities between agricultural and financial technologies in promoting the participation of women and poor farmers in the growth process; Effective Outreach and Communication about both project design and project results with USAID Missions and policymakers abroad as well as with Washington-based staff of USAID and other donors and multilaterals; and Training of the Next Generation of host country scientists needed to sustain creative economic development policy. To help guide the overall program implementation and achieve its expected results, a ToC has been developed that delineates the logic of how the AMA IL is going to better understand and respond to the root causes of persistent poverty and food insecurity (see Table 2 and more details in Annex C, Figure 1). 7 Table 2: Summary of AMA IL Theory of Change ToC Indicative Activities ToC Indicative Outputs ToC Indicative Outcomes Research should: Research will: Research aims to: Engage stakeholders to identify relevant research questions Assemble a suite of accessible policy￾oriented publications Improve understanding of key issues Be focused and policy relevant Deliver high-quality peer-reviewed academic publications and presentations Ensure evidence-based research to guide policy Be innovative and utilize pilot interventions Provide policy and program recommendations and other materials responsive to stakeholder needs Deliver financial innovations ready for adoption Integrate project portfolio around key themes to create a coherent, yet innovative body of work Disseminate learning and results strategically Collaborate and contribute to networks and a community of practice through which knowledge can be disseminated Extract and synthesize key lessons learned Bridge gaps between research and policymaking Ensure implementation, government, and all other partners have increased awareness of evidence-based program options 3.0 EVALUATION METHODS AND LIMITATIONS 3.1 METHODOLOGICAL CONTEXT The evaluation used a mixed methods approach that included qualitative and quantitative data collection and analysis. The approach aimed to ensure sufficient flexibility to collect data from multiple sources to appropriately address the EQs. Qualitative data methods included document reviews, focus group discussions (FGDs), and key informant interviews (KIIs) with stakeholders. Quantitative data were obtained through a review and synthesis of existing program M&E data reports. The mixed methods approach also allowed the ET to identify and understand different points of view within the stakeholder landscape that helped assess AMA IL’s achievements, including the quality and relevance of the research they support. The different stakeholders also helped assess and determine the degree to which research has been strategically positioned to build capacity, achieve impact, and affect different policy landscapes, as well as the global and national contexts within which research occurs. The decision regarding evaluation methodology was significantly guided by the EQs and how best to manage evaluation of a vast and complex program supporting 27 research initiatives across 15 countries, within the context of a time constrained evaluation, and with field work limited to just two countries. This context required a strategic approach aimed at identifying the best possible sample of activities based on area of focus, size, ongoing relevance, and logistical considerations. Central to the AMA IL approach is the ME team based at UC Davis, which provides leadership, oversight, coordination, and horizontal linkages across the research portfolio. This team holds responsibility for ensuring that all research activities are focused and relevant to ongoing development 8 debates within AMA IL’s research focus areas. They are also responsible for ensuring that the overall research approach is suitably positioned to provide practicable recommendations for policy reforms, institutional innovations, and/or development programming relative to these debates. 3.2 SELECTION OF SITES AND PROJECTS FOR EVALUATION 3.2.1 Desk Review and Site Selection The ET started the evaluation process with a thorough desk review of AMA IL key documents to prepare an evaluation concept note and the evaluation protocol. Given the limited time available for the evaluation and the vast implementation landscape of the overall program (including the need to consider headline results, achievements, and direction set by earlier iterations), the desk review focused on identifying the best practicable approach given limitations of the overall evaluation context. The ET therefore used an approach that balanced the need for an evidence base that could cite primary data to support the findings that could be reasonably extrapolated to the broader AMA IL program. This process resulted in the desk review focusing on review of headline AMA IL documents such as the RFA, UC Davis proposal, Leader and AA documents, and the program periodic reports. The documents reviewed included: • • • The RFA, ME proposal, associate agreements, and other background data to understand the program’s “bigger picture” and overall context; Each individual research activity’s library of calls for proposals, progress reporting, training reports, stakeholder engagement reports, formal presentations, policy briefs, working papers, published academic papers, media coverage, and miscellaneous other outputs; and The ME’s array of documents aimed at collation, synthesis, and promotion of the collective findings of the program. The desk review was followed by rapid shortlisting of potential countries to be visited based on summary consideration of the overall program portfolio, and the degree to which participating countries offered an opportunity for exposure to a broad cross-section of the AMA IL operating landscape. In this process, the ET looked at the evaluability of different research activities in a country, logistical considerations, and the feasibility that sufficient depth of review could be achieved within the limited amount of time allotted for field work before a decision was made in consultation with USAID/BFS. The following factors were used to provisionally identify a shortlist of projects for potential focus within the evaluation: • • • • • Degree to which individual research efforts are coherent and reflective of the overall AMA IL approach and intent; Opportunities that exist for comparative evaluation between similar but different research approaches; Degree to which the activity actively collaborates with other stakeholders of significance; Options that exist for lighter touch review of other research efforts occurring in the vicinity; and Logistical considerations (ensuring field work was realistic within the time frame available). Based on these criteria, Ghana and Tanzania were selected for field work given that they each have multiple AMA IL-supported activities. To help ensure the opportunity of an unfettered line of sight from field-level research to positioning for recommendation and/or policy impact, the ToC was used as guidance to the evaluation approach since its generalized, indicative outputs and outcomes are easily applied across contexts. The evaluation process involved consultation with participating smallholder farmers, local governance mechanisms and actors, U.S. and local research institutions, other actors of significance [private sector, bilateral, and/or non-governmental organization (NGO) programs], and 9 those capable of using generated knowledge to initiate practicable recommendations for policy reform, institutional innovations and development programming—both at the national level and beyond. 3.2.2 Selection of Projects for Evaluation Implicit within the desk review was the need to decide whether to go narrow and deep into analysis of a few selected projects or to look more broadly at a breadth of the program. The final decision in consultation with USAID was to take a middle road. This allowed for deep investigation of some of AMA IL’s more interesting and diverse projects, while also offering the opportunity to look briefly at other AMA IL-supported projects in the same geographical area. Both Ghana and Tanzania have multiple, sizeable, multi-faceted Feed the Future-funded programs focused on Feed the Future Zones of Influence (ZOI), with there being an overlap between AMA IL research work and some Feed the Future￾funded projects, both geographically and substantively. This background provided an interesting backdrop in terms of the emphasis placed in the RFA on researchers engaging their partner landscape and disseminating results strategically. Based on this overall assessment, the ET proposed in-depth review of two projects (one in each country)—Disseminating Innovative Resources and Technologies (DIRTS) in Ghana and Achieving Development Impact with Complementary Stress Resistant Seed and Financial Technologies Project (referred to in document as Tanzania DT Seeds)—and a lighter touch review of other AMA IL-supported projects occurring in these countries. The two projects selected for in-depth evaluation both have an integrated approach, introducing improved production technologies in areas that experience damaging climatic events (mostly drought), and utilizing index insurance as an option for helping to remove barriers to technology adoption. Both have been structured to measure the impact of agricultural index insurance upon farmer uptake and adoption of improved production technologies. Both projects use similar benefit packages with at least one different variable that has been tested (type of benefit extended, way in which delivered or priced) and similar randomized controlled trial (RCT) methodologies. It was thought that this scenario would allow depth of insight into performance around the two key themes that AMA IL is most heavily invested in, and shed insight on the program’s ability to investigate and understand linkages and synergies between themes. The ET also undertook lighter touch review of other AMA IL-supported research occurring in the two target countries. These were selected based on their relevance and potential complementarity to the research projects being reviewed in depth. Logistical considerations were considered in terms of the final selection of activities for lighter touch review. Table 3: Summary of Research Activities to be Focused on by ET Country In-Depth Review Lighter Touch Investigation Ghana Disseminating Innovative Resources and Technologies (DIRTS) to Smallholders in Ghana (referred to in document as Ghana DIRTS) Promoting Adoption of Improved Production Technologies Among Smallholders in Ghana via Coupled Credit and Insurance Contract (referred to in document as Ghana Credit and Insurance) Tanzania Achieving Development Impact with Complementary Stress Resistant Seed and Financial Technologies Project (referred to in document as Tanzania DT Seeds; also implemented in Mozambique) Evaluating the Effect of Site-Specific Soil Information on Farmer Input Choices and the Relationship Between Poverty and Soil Quality (referred to as Tanzania Site Specific Soil) Communication, Search and Mobile Phones: Telephone Directory Intervention in Tanzania (referred to in document as Tanzania Mobile Phones) 10 3.3 FURTHER CONSIDERATIONS In conducting the evaluation, the ET took into considerations the following: 1. Research Quality: The underlying assumption is that research quality under AMA IL will be high, given that it is undertaken within the quality framework of the U.S. university system that typically adheres to high research quality standards. Although the RFA did not clearly articulate indicators of research quality, it provided details of expected outputs that imply and necessitate high-quality research, including: • • • Strategic identification of stakeholders and partnering of host country universities and research institutions; The capacity to structure research in such a way that it ensures a sound evidence base for promotion of policy/programming recommendations and institutional innovations and/or development programming relative to these debates;3 and High-quality and systematic oversight of the research portfolio to ensure quality standards are being met while also ensuring strong knowledge management (KM), outreach, and dissemination. While research quality can be assessed in many ways, in this report it will be discussed within the framework of the activities and outputs detailed in the ToC previously outlined in Table 2. 2. Outreach and Dissemination: The importance of high-quality outreach and dissemination was highlighted in the RFA as being critical to the achievement of development impact that USAID increasingly seeks from its ILs. As can be seen from the RFA, “achieving development impact” is explicitly linked to effectiveness in dissemination of results and outreach and inclusion of key stakeholders, stating that development impact will be aided by efforts that: • • • Ensure policy/programming relevance; Promote adoption of policy and/or programming recommendations that research activities generate; and Engage policymakers, USAID Mission staff, private sector representatives, and other stakeholders as appropriate—even in early stages of the research design and implementation. In this evaluation, consideration of outreach and dissemination is assessed based on the program’s successes or failures to realize the following outputs: • • • • User-friendly website to store and make all resources publicly available; Meetings with stakeholders to explain results; Policy and program recommendations and other materials responsive to stakeholder needs; and Variety of events to disseminate results and insights, including seminars, webinars, and conferences. 3. Policy: Given the RFA provides no specific detail on where it sees policy traction being achieved, it is the interpretation of the ET that USAID reasonably sees the possibility for AMA IL research to impact policy both at the country of research level and more broadly through regional and global fora. Progress will also be achieved through engagement of the broader development community, notably within USAID itself. Therefore, policy will be discussed in relation to the achievements/failures to realize the following ToC outputs: • • Suite of accessible policy-oriented publications; and Policy and program recommendations and other materials responsive to stakeholder needs. 3 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 10. 11 4. Capacity Building: The RFA clearly asked for a commitment to capacity building, stating that applications must include a strategy to maximize long-term degree training opportunities for developing-country nationals, as well as placing emphasis on other capacity building and institutional strengthening approaches. UC Davis itself proposed a multi-faceted approach to capacity building, given financial constraints to broad-based training of host country academics within U.S. universities. PIs were charged with development of capacity building appropriate to their research context. UC Davis also proposed empowering host country PIs as key partners in research, while also working to ensure local institutional capacity strengthening. The program ToC includes “capacity development through in-country partnerships” as an output. 3.4 QUALITATIVE RESEARCH AND ANALYSIS In addition to document review, FGDs and KIIs were used as main sources for collecting qualitative data. A total of nine FGDs were conducted at the community level with local community members within the research context—including comparison groups, when feasible. In Ghana, the ET held five FGDs. Four of them (two males, two females) involved treatment groups from the DIRTS research. The fifth was with Community Extension Agents (CEAs) supported by the DIRTS project. In Tanzania, the ET conducted four mixed FGDs: three were with treatment groups of the DT Seeds Project and the fourth was with a group representing all different treatment groups within the Site-Specific Soil research. The ET conducted KIIs with key actors, including: farmers involved in research trials; local government partners; civil society partners; private sector partners; academic institutions—including PIs; national government representatives; USAID Missions; Feed the Future-funded programs; and AMA IL Program Management (Table 4). In total, the ET conducted interviews with 77 key informants. Sixty-six of these were with in-country key stakeholders (38 in Ghana and 28 in Tanzania) and 11 were with U.S.-based informants (6 people within the ME and 5 U.S. based PIs and researchers). The KIIs in the two countries visited were conducted face-to-face, as was the ME team interview. Remote conferencing took place with U.S.-based lead researchers via teleconference. Efforts were made to consult with those on the research landscape capable of affecting policy in order to assess their awareness, understanding, and level of engagement with research activities. Table 4: Number of Respondents by Data Collection Method by Country Data Collection Method USA Ghana Tanzania Remote TOTAL Key Informant Type Farmers involved in research trials 8 8 16 Local government partners 2 5 7 Civil society partners 8 2 10 Private sector partners 7 4 11 Academic institutions—including PIs 3 5 5 13 National government representatives 2 1 3 USAID Missions’ staff 4 2 6 Feed the Future-funded programs 4 1 5 AMA IL program management 6 6 TOTAL 6 38 28 5 77 12 Data Collection Method USA Ghana Tanzania Remote TOTAL FGDs (# people) 5 (100 people) 4 (72 people) 9 (172 people) Six different interview guides were prepared to provide direction and structure for interviewing different stakeholder groups, based around the indicative outcomes measures prepared as part of the evaluation protocol. Prior to each KII or FGD, questions were tailored from the relevant guide to the specific context of the interviewee(s) as detailed in Annex H. Assessment of the degree that gender issues have been incorporated within research approaches was interlaced within the evaluation approach. At the community level, gender-disaggregated approaches were utilized whenever possible. During field work, the data collection approach and structure of each KII or FGD was guided by the relevant interview instruments (Annex H). However, each interview meeting required its own nuance and tailored areas of focus and inquiry, due to: • • • The highly varied subject matter, stakeholder landscapes, and contexts of different AMA IL￾supported research activities; Different informants having highly varied understanding of the AMA IL “whole,” with most focused primarily on their own unique point of engagement and area of interest, with little interest or knowledge of the grander ambitions of AMA IL; and Different research activities being at different stages of implementation. To address the data variability, the ET established a systematic approach for analyzing and interpreting the data by the key EQs. The analysis process started with the ET meeting on a daily basis during the field work to review their data and organizing them by EQ. At this stage, the ET discussed and summarized individual interviews and considered how they contributed to answering the EQs and to the bigger program performance management framework picture. These daily meetings were also used to assess outstanding information needs and identify opportunities and strategies for collecting that additional data through upcoming interviews. The next level of analysis was conducted on a weekly basis when more extensive team meetings were held. Efforts were made to triangulate key qualitative themes emerging across interviews of different stakeholder types. As required, strategies were developed for how these findings could be further explored/tested in subsequent KIIs and FGDs, and the ET worked with the ME to locate key data still missing from the analysis. Through this process of qualitative analysis, the team was able to build an evidence base and more generally work towards a consensus in the key findings and “take away” from that week’s work. In the post field work phase, team members were allocated time to review and cross-check field work data with program documentation, with the benefit of having field observations to reflect upon. Each team member was asked to assemble key findings, with a clear expectation from the Team Leader that these be triangulated as much as possible and backed up by specific evidence drawn from specific meetings. Team members then critiqued each other’s findings, which contributed to a robust process of synthesis that led to broad agreement on the findings detailed below. 3.5 QUANTITATIVE RESEARCH AND ANALYSIS To assess research quality, the ET reviewed research documentation, requiring consideration of approaches and outputs, including conclusions based on quantitative data. Annual reports, annual work plans, performance indicators, surveys, and policy briefs were also used to collect secondary data. The ET made every effort to ensure that qualitative approaches and findings were underpinned by analysis of available quantitative data in AMA IL reports, which was used to triangulate results. However, 13 the degree to which quantitative approaches could be applied to the evaluation was restricted by the limited time available to the overall evaluation. 3.6 EVALUATION LIMITATIONS The primary limitation of the evaluation was the limited time available to undertake desk review (5 days) and field work (22 days) for a program active across 15 countries. As to be expected of a research program, AMA IL has developed a vast amount of documentation across 27 research activities, including: calls for proposals; work plans; various forms of progress reporting; training reports; stakeholder engagement reports; formal presentations; policy briefs; working papers; published academic papers; media coverage; and miscellaneous other outputs. Collectively, these outputs quite literally amount to tens of thousands of pages of relevant data. Inevitably, time limitations meant that many aspects of the program could only be viewed quickly. Field work itself was time consuming. Travel to, within, and between the two selected countries, was slow, especially given the remote nature of research sites. While every effort was made to maximize the time available, there were inevitably days that involved more travel than work. Details of the ET travel schedule are described in Annex J. The ET was only able to visit two countries—Ghana and Tanzania—while AMA IL has supported research across 15 countries. This approach may have engendered a degree of bias given that field work focused only on activities in two sub-Saharan African countries, despite the fact that the program also operates in Asia, Latin America, and the Caribbean. This limitation was overcome to a degree by a deliberate effort to review documentation of a cross-section of research types and geographic regions. The ET was challenged by the fact that while AMA IL engages an extraordinarily diverse group of different stakeholders, these stakeholders do not have a shared understanding of the “program whole,” its purpose, or its theory of change. Mostly, farmers, local researchers, local partners, host governments, and development actors have very little interest in the program beyond their unique area of engagement. This meant that very few informants could offer more than a small portion of the type of evidence required to reach robust program-level conclusions. Thus, the ET had to invest a substantial amount of time piecing evidence together in order to triangulate findings. A related challenge was considerable variability in performance and approach from research activity to research activity. To address these challenges, the ET regularly took time to consider qualitative data with a view to extracting elements capable of contributing to higher-level findings. In the report, this is reflected by efforts to strategically provide illustrative examples and evidence of each stakeholder type within each key finding domain. Despite the various limitations described above, every effort was made to achieve understanding of the “program as a whole.” While findings and conclusions emerged largely from both the field-level analysis in Tanzania and Ghana, and meetings with the ME in Davis, every effort was made to extrapolate, verify, and frame these findings and conclusions in a way that aimed to capture overall program-level performance. Efforts were also made to cross-reference findings through review of documents and literature related to the activities in countries not visited by the ET. 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 4.1 RESEARCH QUALITY 4.1.1 Have the research projects developed scientifically valid and robust conclusions and professional-level outputs? In what ways did the portfolio of projects make progress and 14 contribute toward the stated research objectives in the RFA? How did it diverge from objectives in the RFA? FINDINGS The ET found that UC Davis implements the AMA IL program based on five principles aimed at ensuring high-quality research: 1. i Open competition to select the highest quality research projects to create a portfolio that mplements the vision of the AMA IL; 2. Engage USAID Missions with dedicated outreach resources to identify innovative, policy-relevant pilot activities that complement their strategic goals and programming under Feed the Future; 3. Monitor research to ensure sound technical progress, achievement of impact, and contribution to program and Agency goals; 4. Commitment to innovative, high-quality research that generates policy-relevant results, with lessons drawn out and communicated for maximum utility by policymakers and programmers; and 5. World-class leadership operating under a Director with the global reputation, knowledge, and commitment needed to mobilize and mold the highest quality research program.4 The selection process of primary research projects involves reviews of all proposals by an outside panel comprised of experts in the field, a USAID representative, and the Director and Assistant Director of the ME. Proposals are scored based on their: 1) technical merit; 2) applicability to AMA IL areas of interest; 3) plan for collaboration and capacity building; 4) plans for dissemination and outreach; and 5) potential to achieve development impact. Research Quality Assurance Based on the documents review and synthesis of results, the ET found that the AMA IL program research portfolio has responded coherently, creatively, and in a scientifically rigorous way to the identified research issues. For the most part, this judgement is based on the use of advanced research methodologies using RCT. From 2011 to the time of this evaluation, AMA IL has generated a diverse research portfolio that responds innovatively to the research topics posed in the 2011 AMA IL RFA, including bundling different permutations of resilience-focused approaches across a diversity of socio￾economic, political, and agro-economic contexts. Researchers engaged extensively with relevant stakeholders in the design of their proposals. Successful proposals are closely aligned with the AMA IL’s areas of focus and appear to be policy-relevant—at the national and higher levels. In addition, the AMA IL Board of Directors ensures that proposals are reviewed and assessed by the world’s leading research experts in AMA IL’s focus areas. The ET found that documentation related to proposal selection processes suggests both detailed analysis of each individual proposal by a highly qualified Board of Directors, to ensure research quality. There is also evidence that the selection process takes into consideration the program potential of the collective group of proposals to be accepted. The process also considers how they potentially relate to and complement each other. This process, in theory, should help foster program synergies and avoid overly similar proposals being supported. Where appropriate, researchers were asked to consult with previously supported researchers to help ensure that their new research was fully informed by prior related efforts. Interviews with PIs revealed that there were significant modifications undertaken in the “negotiation process,” resulting in proposals being improved during an iterative selection process. 4 UC-D proposal to serve as ME to the AMA IL CRSP, p. 2. 15 Research quality within the portfolio is high, involving 27 highly qualified PIs based within leading U.S. universities and working across 15 countries. The PIs’ research capacities range from the leading subject matter experts in the world to younger researchers developing their subject matter niche. The U.S. researchers collaborated with local research institutions that have been targeted for their interest and capacity to support the specific research topic and tailoring the research to local relevance. For example, in Tanzania, the “Communication, Search, and Mobile Phones” research partners with the Dodoma-based Institute of Rural Development Planning (IRDP), which brought existing interest in the subject to the research, as well as established relationships with key stakeholders. One of these relationships was with the Tanzanian Communications Regulatory Authority, which issued IRDP a license to use unstructured supplementary service data (USSD) technology—the protocol used by global system for mobile (GSM) cellular telephones to communicate with the service provider's computers. This ensured unrestricted access of the target group to the technology being investigated in the research, thus avoiding research being compromised by bureaucratic issues. The overall research portfolio has been assembled strategically to support a variety of different, yet complementary, approaches around its key areas of focus—notably, index insurance, drought resistant seeds, agricultural extension, and the application of satellite technologies as a tool for risk minimization. Individual research projects touch upon multiple issues and different permutations of issues. Furthermore, the process of data triangulation across different research approaches allows AMA IL to logically and strategically build off earlier program iterations into new areas of inquiry. The synthesis and triangulation of portfolio findings help bring greater clarity to the evidence base established by AMA IL, especially on overcoming barriers to technology adoption among smallholders. The different research approaches within the portfolio help position AMA IL as an important laboratory for index insurance given its ability to triangulate and compare approaches across different research combinations, locations, and contexts. This potential to achieve increasing clarity around quite specific key challenges should logically expand as the evidence base is strengthened by more related research activities reaching completion. Technical Committee meetings are another aspect of ensuring research quality. These are held annually, bringing together PIs and a limited number of strategically identified guests from host countries and key stakeholders from the organizations best positioned to utilize the knowledge being generated, notably USAID and the World Bank. Efforts are made to make sure these meetings are dynamic and productive by ensuring a fast-paced, interactive approach that allows each researcher one hour to present findings, with another researcher acting as chair and lead discussant. However, these meetings are also limited to less than two days per year, which restricts the degree to which they can be used as meaningful opportunities to delve deeply into program issues and direction. While the 2012 meetings were held in Rome, all subsequent meetings have been held in the U.S., alternating between Davis, CA and Washington, D.C. Another element of research quality is the degree to which the program is effective in its strategic outreach and dissemination to key audiences. In its bid to become ME, UC Davis stated that, “knowledge by itself is of little use if it does not spread beyond the research community.”5 Recognizing weakness in this area, the AMA IL ME has worked over the past two years to bolster its outreach and dissemination capacity, resulting in more proactive and dynamic approaches to KM and dissemination, especially through regional and U.S.-based channels. The importance of this effort to strengthen overall communications capacity was observed in the field, where high-quality research often appears to be 5 UC-D proposal to serve as ME to the AMA IL CRSP, p. 15. 16 undermined by insufficient attention and strategy being applied to stakeholder engagement, outreach, and dissemination. This issue will be discussed in more detail in section 4.2 Outreach and Dissemination. Much of the AMA IL-supported research is designed to contribute knowledge across each of the three project areas (risk management and resilience, rural and agriculture finance, and inclusive market access and engagement). This is done through bundling different technologies and approaches to advance the program’s vision of relevant and holistic approaches that enhance the resilience of vulnerable communities. AMA IL’s most common form of bundling is combining index insurance alongside improved access to higher-quality inputs and capacity building to holistically address smallholders’ needs. In almost all cases, this pairing is augmented by additional context-appropriate strategies like application of information and communications technology (ICT), attention to supply chain constraints, support to local extension capacity, and/or improved access to improved weather data. It is this nuanced variance that positions the program to advance knowledge and understanding around its key themes in an efficient and forward-looking manner. AMA IL uses scientifically rigorous methods with the aim of developing a credible evidence base. Good examples of high-quality research advancing knowledge across the three project areas are the Ghana DIRTS and the Tanzania: Achieving Development Impact with Complementary Seeds and Financial Technologies (Tanzania DT Seeds) projects. Through a DIRTS-sponsored randomized evaluation of 3,240 households in 162 farming communities in northern Ghana (divided into four treatment groups), researchers are testing the impact of agricultural extension and improved input access and delivery when farmers have access to the risk reducing instrument of index insurance. While the research involved regular surveying of 3,240 households, its impact was far greater given that interested farmers were encouraged to participate in the community-based extension program irrespective of whether they were within a treatment group or not. In a focus group in Afaylili Village, one farmer proudly stated that, “even though I’m not in any of the (treatment) groups, I sit in on the information sessions and I’m getting better results than my neighbor who is in the group!” When pressed on what he had done to improve practice, it became clear that this farmer had been active in a community extension group, and had applied learning related to organic fertilizers, resulting in a significant yield increase and behavior change (see Text Box 1 and Annex K for a more detailed description of the Ghana DIRTS program). Illustrative examples of general research quality and approach of AMA IL research are described in Text Boxes 1 and 2, below and in Annex Q. Text Box 1: Illustrative Example 1 – Ghana: “Disseminating Innovative Resources and Technologies to Smallholder Farmers” (DIRTS) This project builds off an earlier research program—Exploring Underinvestment in Agriculture (EUA)—that identified reluctance to take risk as the major inhibitor of agricultural production among smallholders. It is a multi-donor activity enjoying support from Massachusetts Institute of Technology Agriculture Adoption Initiative, the United Kingdom (UK) Department for International Development—Economic and Social Research Growth Program, and the Development Innovation Venture of USAID, as well as support from AMA IL. The research is led by Yale University, in partnership with the Savanna Agricultural Research Institute (SARI) in northern Ghana and the International Food Policy Research Institute (IFPRI) in Ghana. On the ground implementation is undertaken by the NGO Innovations for Poverty Action (IPA), based in Tamale, the main town in northern Ghana. This composition of donors and partners provides the research an acutely relevant cross-section of agencies that bring an important set of different skills to the research, while also being steeped in the practice of high-quality research themselves. IPA’s “on-the ground” role also allows for research to be managed in a manner that permits close attention to research fidelity while also offering capacity for troubleshooting and timely modification, as required. While the research on the ground appears to be of the highest quality, the ET observed that the project has missed opportunities for not being more proactive and deliberate in identifying champions and sharing findings, information, and progress with key stakeholders—particularly at the national level. For example, an interview with a Senior Technical Advisor to the Ministry of Food and Agriculture (MOFA), who is a strong advocate of 17 the need for agricultural insurance focused on smallholders revealed that he had no knowledge of the DIRTS program. However, this senior official also noted that there were inherent challenges in engaging MOFA around innovative approaches, given the revolving door at the Ministry, which has seen five Ministers in five years. Text Box 2: Illustrative Example 2 – Tanzania: Achieving Development Impact with Complementary Seeds and Financial Technologies (also implemented in Mozambique) This research program (still underway) has been commissioned to study the impact of DT maize seed varieties being scaled up across three distinct rural production areas in Tanzania. A baseline survey has been undertaken covering 1,796 households in Tanzania (and 501 in Mozambique, where the research is also occurring). It occurs in partnership with CIMMYT through implementation of a multi-year, spatially diversified RCT of two proven, complementary, stress-resistant technologies: DT maize seeds and index insurance fine-tuned to cover the high stress conditions when DT maize fails. Partnering with CIMMYT in this context was a strategic move given their outstanding record of developing seed-improved gene pools and agronomic practices with proven positive impact on crop farming systems. A three-armed RCT is being implemented with different treatments looking at combinations of DT seed access and index insurance. A yield recall survey was used to guide research direction and provide a baseline. The approach also utilizes satellite imagery and builds off knowledge generated through an earlier AMA IL-supported intervention in Tanzania—Developing a Satellite-Based Index to Predict Crop Yields in Smallholder Agriculture. While this latter mentioned project was not successfully completed due to the withdrawal of an IP, it was notable to see the learning generated from this failed project being utilized to advance the overall direction of the AMA IL program. The program worked in close partnership with two agricultural research institutes, from which it receives on￾the-ground technical and logistical support. It also supports three national seed companies to enhance distribution of improved seeds and consider the potential of bundled index insurance as a viable product for Tanzania. One of the three companies supported through the Tanzania research is IFFA Seed Company Ltd., based in Arusha, Tanzania. Based on FGDs, the ET learned that farmers valued the improved seed made available by the project (through IFFA), understood the benefit of insured seed, and were willing to purchase it even at the expense of reducing land under cultivation. The interviews revealed that seed availability and farmer uncertainty of how and where seed could be accessed moving forward— especially insured seed—were a major challenge. The recommendations from this research include that the research parameters should better address accessibility barriers based on attributes such as access to markets and access to associated inputs. The program has also been effective in both engaging and motivating private sector actor UAP Insurance (UAP) to better understand and integrate smallholder-focused insurance products into its business model. While the program has not yet been proactive in its engagement of the Ministry of Agriculture at the national level, this is more easily understood than in the DIRTS activity given that the research is only at its mid-point now. The program has also worked to cultivate a national-level presence and capacity through its partnering with UAP. For its part, UAP is clear that, while it needs to engage key stakeholders in government soon, it still needs to further develop its products. Gender Issues The ET observed that although gender dimensions were included in the UC Davis proposal to USAID, in reality gendered understanding in research appears to have been given little priority, including no gender considerations applied within the proposal selection process. Findings show that while women were active in all of the research projects reviewed by the ET, the degree to which gender issues were integrated and potential “gender learning” maximized within different research approaches varied considerably. This is seen as a lost opportunity given the 18 established fact that women are more vulnerable to shocks than men and the highly gendered nature of many of AMA IL’s focus area. The ET noted that the scope exists for gender to be more deliberately considered and structured into the overall working of AMA IL. It was also observed that in the absence of a committed and deliberate approach to bring clarity within all research to gender issues, gendered research is limited and continues to occur in an ad hoc manner, dependent upon the perspective of individual researchers and research teams. In the DIRTS case, although community extension meetings are mostly gender disaggregated for cultural reasons within the project activities, gendered analysis of findings has not been conducted. For example, the 2016 Annual Report of DIRTS noted that research findings differed between male and female, yet little analysis of this important finding was conducted. Cost was cited by the ME as justification for not adding a stronger gender dimension to research activities, and for the focus implied within the UC Davis proposal being wound back. While there are likely additional costs involved in expanding a research approach, the ET believes that this additional investment would be justified given the centrality of gender issues to entrenched poverty. Research Outputs Researchers are also required to submit periodic reports to the UC Davis team, including annual work plans, quarterly and annual reports, trip reports, training reports, workshop reports, and the development of performance indicators. Reports are reviewed by the ME which commonly seeks clarification and re-drafting of reports to ensure clarity and rigor. The reporting process also involves a financial management element, with the ME using narrative reporting to cross-check against financial obligations. An additional dimension of this reporting and documentation process is to identify program synergies and linkages and allow for higher-level program findings to be extracted in support of outreach and dissemination efforts. Beyond the standard reporting, researchers publish academic papers, prepare working papers and policy briefs, and undertake presentations with key audiences. A full list of research outputs is contained in Annex B. Table 4, below, provides a summary of AMA IL project outputs. A review by the ET of such documentation suggests they are prepared to a very high level of quality and bring depth of insight and clarity to complex issues. During field work, the ET observed the two projects reviewed in depth had achieved publication of outputs in important publications, the Journal of the International Association of Agricultural Economists and the Ecological Society of America. Table 5: Summary of AMA IL Research Project Outputs (as September 15, 2017) Outputs Number Published Academic Papers 33 Working Papers 36 Briefs/Policy Documents 110 Presentations 87 Stakeholder Meetings 83 External Media Coverage 86 Miscellaneous Other Outputs 22 19 Divergence from Objectives in the RFA Overall, the AMA IL’s portfolio of research aligns with the broad objectives of the RFA as outlined above in Table 5. In cases where it could be argued that there is divergence, such as the Tanzania Telephone Directory Intervention, this can be justified as responding to an emerging issue of increased interest and opportunity to leverage widespread ownership of mobile phones as a tool supportive of the roll out of new technologies. In this sense, divergence can be seen as acceptable if it helps fill a knowledge gap. This also fits with a UC Davis mantra that if AMA IL “does not generate a few surprises in terms of unforeseen, innovative ideas, then it will not have done its job of engaging the very best researchers.” 4.1.2 How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement? FINDINGS As described earlier, one of the AMA IL research program strengths is that many of the individual research activities actually cut across the three areas of risk management and resilience, rural and agriculture finance, and inclusive market access and engagement. For example, the Ghana credit and insurance activity includes index insurance, improved access to credit, and a focus on women as elements. This combination of elements avoids the trap of issues and approaches being considered in isolation and offers insight to different permutations of approaches that respond to the complex, and often context-specific, challenges faced by poor farming households. AMA IL works to advance understanding around key resilience-related issues, such as the potential of DT seeds as a strategy for building resilience through improved awareness and improved market access. Improved seed varieties (for various characteristics such as pest resistance, drought tolerance, high yield) are long established, having been developed over the years through global, regional, and national level collaboration between developed/developing country universities, the Consultative Group on International Agricultural Research (CGIAR) system, and others. However, AMA IL sheds important new light on their potential by recognizing that, when developing new seed varieties and agronomic technologies, researchers have often focused solely on solutions to the challenges of reduced crop production and productivity without considering: 1) the ability of the farmers to access the seeds and the accompanied technology and inputs; and 2) the risk to the farmers from using new seeds at high costs and uncertain sustainability options. Accordingly, AMA researchers are now working to address these issues through a wide number of research activities across a diversity of contexts and locations. Illustrative examples are presented below. Access to Improved Drought Tolerant Seed Varieties A doctoral research in Tanzania focused on the strategic understanding that the DT maize technology “is a weather-contingent technology based on the detection of the response of DT maize to precipitation using high resolution and high frequency weather data.” Hence, AMA IL researchers have focused on investigating associated measures that enhance the confidence of the traditionally risk averse dryland farmers, while working to increase the rate of replacement of the obsolete/less productive seed varieties. The ET found that some very interesting results have emerged from this research, including: • According to Michael Carter (UC Davis and CIMMYT October 30, 2015) “stress-tolerant seed and insurance technologies at first glance substitute for each other: Both try to stabilize farmer income in the wake of adverse climatic events. The DT seeds are tolerant of drought but are not drought proof. On the other hand, the most probable solution, insurance, can be expensive, especially when applied at varied drought events (relatively frequent, moderate drought events).” 20 • • According to Carter and Giannini (July 2017), use of improved seeds and inputs (e.g., fertilizers) drops remarkably during drought years compared to normal years. For example, households using improved seeds in Mozambique dropped to 21 percent during their 2016 drought, and only 8 percent used some chemical fertilizers. Such information is significant and demonstrates farmers’ capacity to be strategic in application of scarce financial resources. Adoption of the associated measures must be stratified according to the growing cycle (see Figure 1, below) where the risk of weather changes and the dynamics of droughts (onset during the growing cycle, severity, persistence) vary and, in some regions, are very difficult to predict or interpret by the non-experienced technicians or farmers. As reported by KIIs from UC Davis, the impact of DT seeds on stabilizing yield is significant during modest drought conditions, where it can support plant resilience during establishment and flowering. However, the research findings show that this impact is reduced to zero if the droughts are severe, thus making the DT seeds not any better than other rainfed dependent seeds. Figure 1: Strengths of DT Maize Source: Key Informant UC Davis, August 2017 Risk Management and Resilience The approach and direction of AMA IL in relation to risk management and resilience is premised on the recognition that risk management is economically costly in low-income agricultural economies, and that this is a core factor explaining low levels of adoption by smallholder farmers of the risk mitigation technologies introduced. The ET observed that AMA IL has emerged as a global leader in research related to index insurance as a method for promoting adoption of new technologies among cohorts that have historically resisted such technologies. This is a uniquely important space needing to be filled given the rapid evolution of satellite technologies, notably remote sensing and automated weather measurement, the critically important data from which index insurance derives. Keeping abreast of technological change and responding to complex emerging opportunities are critical in further advancing and consolidating index insurance utilization by farmers, including opportunities to better address issues of basis risk that undermine the product. Globally, the subject of index insurance is new and innovative, poorly understood, rapidly evolving, and rarely encapsulated in policy or strategy with sufficient rigor and commitment, even though it is emerging as an increasingly interesting “resilience product.” AMA IL is therefore making an important 21 contribution to the knowledge base both through its research and dissemination efforts, and through its leading role in development of the Global Action Network (GAN) to Advance Index Insurance. GAN is a community of practice operating in partnership with the Impact Insurance Facility of the International Labor Organization in Geneva, which also serves as the Secretariat for GAN. As a mechanism for the financing of index insurance-related research, AMA IL is a resource of critical importance to GAN, and central to its capacity to: • • • Establish and coordinate a community of experts to serve as an action network that discusses key issues around agricultural insurance; Build the capacity of practitioners and governments where index insurance-related research is occurring, including support to country strategy and work plan development, collaboration with existing initiatives, and conducting training for practitioners; and Promote responsible scaling of agricultural insurance to the broader insurance community by re￾packaging and disseminating lessons into knowledge products, tools, and training modules. Through its breadth of index insurance-related research, AMA IL is in the process of assembling a critical mass of different smallholder-focused insurance products. The different entry points and bundling with related technologies help evolve and bring more textured understanding to the capacity for index insurance to contribute to smallholders’ resilience. The ET noted that within AMA IL focus countries, there were similar yet uniquely varied approaches to index insurance research that offered significant learning opportunities. These include: • • • • • Potential for index insurance to be used as a safety net that encourages banks to extend lines of credit, at reduced interest rates, to smallholder farmers (Ghana Credit and Insurance). Approaches reliant on more traditional “door-to-door” direct marketing of index insurance to smallholders, which require highly developed capacity within the insuring institution—in terms of technical skills, marketing, and reach (Ghana DIRTS). Evidence that clear communication of how an index insurance product works is central to marketing. Without confidence and clear understanding, cash poor farmers will not be convinced to participate (Ghana DIRTS). Indications that while governments are generally interested in the concept of index insurance, they commonly have very limited understanding of it. Strategic longer-term engagement and awareness raising within government of an index insurance product can build government ownership [Kenya Index-Based Livestock Insurance (IBLI)]. Clear indications that index insurance products become more attractive when bundled with other contextually-relevant risk-reducing technologies (improved seeds, extension support, etc.). Removing Barriers to Broadly-Based Growth AMA IL’s work around reducing barriers to broadly-based growth is informed by the recognition that, even when poor farmers are better equipped to take risks, there is often a long and complex list of other barriers that commonly hinder their participation in higher-yielding, and higher-returning agricultural opportunities. These barriers mostly stem from a low income and/or asset base but are further compounded by issues such as a lack of awareness and understanding of technology and access to technical assistance or a lack of knowledge of and/or access to markets. Another barrier is the isolated nature of many poor communities, which positions them beyond the easy and cost-effective reach of input suppliers and extension services. AMA IL’s work in this area is multifaceted and commonly occurs in conjunction with work focused on enhancing resilience—notably through index insurance. For example, the Ghana DIRTS program discussed earlier focuses on agricultural extension, improved systems for timely distribution of higher 22 quality agro inputs, access to higher quality weather forecasts, and index insurance. While these issues are often addressed within development programs, within DIRTS different treatments are systematically and scientifically compared across a broad range of community types, with a clear focus placed on data gathering and management. This has facilitated learning about entrenched barriers to growth including the following: • • • • • With structured training and support, CEAs are capable of providing important information and technical support around improved crop technologies. By being community based, CEAs give confidence to farmer-to-farmer adoption of new technologies. They also have the potential to work closely with and complement government extension efforts. Higher quality weather data distributed by short message service (SMS) is valued by farmers and has the capacity to positively influence crop management decision-making. Furthermore, farmers have demonstrated through trials that they share weather information within their community, thus overcoming difficulties posed by low levels of literacy within some households. When underpinned by index insurance, farmers have proven more willing to adopt newer technologies and invest more in production systems, with evidence of increased tractor use and fertilizer application apparent in some research activities. With only modest program support, input suppliers can be encouraged to engage and provide better service to poorer, more remote communities that were previously thought unprofitable. However, even when awareness is raised and new technologies are made more readily available, adoption of higher-cost new technologies is often still beyond the reach of many—in the case of DIRTS, 80 percent of households made no major change to their farming system due to affordability constraints. As identified by the PIs, the most significant learning to emerge from DIRTS is that there is no evidence that new technologies and varieties are increasing household profitability. DIRTS research concluded that low yield, low input production systems work as well as anything for smallholders reliant on rainfed agriculture. In Tanzania, the “Evaluating the Effect of Site-Specific Soil Information on Farmer Input Choices and the Relationship Between Poverty and Soil Quality in Tanzania” (Tanzania Site-Specific Soils) uses a rapid, on-farm soil diagnostic kit—SoilDoc—combined with ICT to provide farmer-specific management recommendations. This approach responds to an earlier finding that government fertilizer recommendations are incorrect in 80 percent of cases. This research aims to test the hypothesis underpinning SoilDoc—that farmers will apply productive inputs more effectively and increase yields in response to improved information about soil quality. Using an RCT approach, different treatments were used to compare and consider the degree to which farmers would act on site-specific fertilizer recommendations. The different treatments involved provision of site-specific recommendations and vouchers to buy inputs. While tailored application of fertilizers would represent a 20 percent saving over the less accurate government recommendation, farmer enthusiasm to act on the recommendations has proven to be low, due in large part to people being unable to afford the inputs even when they could be applied more effectively and efficiently. As with DIRTS, these findings again highlight input affordability as a fundamental barrier to adoption that overrides the raising of awareness and other mechanisms aimed at promoting adoption. Rural and Agricultural Finance Both the experience of Ghana DIRTS and Tanzania Site-Specific Soils highlight that, even with high￾quality extension and technical advice, farmers are still commonly unable to act on advice due to their household financial situation. Within Tanzania DT Seeds, proximity to market had significant impact on adoption. Accordingly, AMA IL has invested in research that works to identify strategies that can help 23 better connect smallholders to the financial resources and technical understanding necessary to safely invest in higher quality inputs. This research is systematically looking at bundling interventions to identify the best permutations of insurance, subsidy, and finance that are effective in accelerating agricultural investment and growth. The Ghana Credit and Insurance project tackles this phenomenon by looking at ways of opening up more affordable lines of credit for smallholders. Under this project, index insurance was introduced as part of an insurance contingent loan product available to smallholder farmers through local banks, to be used by farmers for purchase of improved inputs for crop production. This is a different insurance product from crop index insurance, in that product design is focused upon a different combination of stakeholders and incentives to participate, as well as different risk sharing mechanisms. The program is also a good example of having structured opportunities within its approach that allow for clear line of sight on gendered differentiation of the impacts of index insurance on technology adoption. This index insurance contingent loan product was offered based upon two options for indemnity payment in the case of weather-related unforeseen crop losses—one where the indemnity was payable to the farmer/borrower, and one where it was payable to the lender. While the “indemnity payable to bank” option was more successful in inducing bank participation in the program, with the introduction of this insurance contingent loan product, farmers were 53 percent more likely to apply for loans and banks increased their credit approval rate by as much as 24 percent. Importantly, credit agreements were dependent upon borrowers working with extension agents to develop a budget as the basis for the loan, with credit being provided in the form of a voucher to purchase agreed inputs from a local input provider. Findings of the ET show that product design and implementation for the insurance contingent loan product helped build the capacity of a number of rural banks to appraise, extend, and better manage risks inherent in lending to smallholder farmers, with two participating banks interviewed considering further adaptation and extension of the product to small farmers. Project participation also increased the capacity of banks to work with local input providers to mitigate credit use and repayment risk. See Annex L Lender Experience with Insured Credit in Ghana for more detail. See Annex D for a comparative analysis of Index Insurance products. 4.1.3 To what extent has work from previous iterations of the Lab been effectively incorporated and built upon? FINDINGS BASIS, the first iteration of the AMA IL was established by USAID at the University of Wisconsin Madison in 1996. Since its conception, the Lab’s mandate has remained relatively stable and focused on supporting poor and smallholder farmers in the developing world to manage risk, adopt productive technologies, and participate more actively in the marketplace. The RFA speaks of the previous iteration having: 1) fostered particularly robust engagement on the topic of risk and vulnerability; and 2) advanced USAID’s understanding of when, where, and how to use index insurance in developing country contexts to support development objectives. As described earlier, the current iteration of the AMA IL research portfolio has focused on multi￾faceted investigation of risk and vulnerability, with emphasis on the role of index insurance and how products can better overcome barriers to economic growth. One aspect of the progression from the earlier iteration is that of technological advancement. Progress, evolution, and the improved affordability of satellite weather data, all offer ongoing opportunities to further sharpen and focus index insurance products, and address historical weaknesses related to basis risk, making it an important area of inquiry. Several research activities such as Kenya IBLI and Ghana DIRTS were funded through the previous iteration and had new research programs approved on the basis of the progress achieved and the 24 opportunity to further advance understanding. For example, IBLI was originally conceived as a research program looking at the feasibility of an asset protection insurance program for pastoralists in northern Kenya. Inspired by earlier work that not only indicated the presence of a poverty trap in the region, but also signaled large gains toward a “productive” safety net that would break the slide of families into indigence, the program undertook the ambitious task of devising an implementable insurance contract for households in this isolated and infrastructure-deficient region. The core knowledge base assembled over the life of the research effort is now being adapted to different contexts across livestock value chains of sub-Saharan Africa. In a significant achievement for AMA IL, the Kenyan government recently launched and funded Kenya Livestock Improvement Program (KLIP). The Oromia region of Ethiopia is also now introducing a livestock insurance scheme based on learning generated through IBLI. CONCLUSIONS FOR EQ1 The AMA IL research quality is high based on its relevance to the focus areas of inquiry, research methodology and practice, and the quality of outputs in robust evidence-base and actionable recommendations. AMA IL research is scientifically rigorous and transparent and is systematically building a valuable evidence base around several new technologies thought to be well-suited to improving rural households’ outcomes ability to acquire, protect, and effectively use productive assets. This evidence base is particularly valuable given that technologies such as index insurance and application of satellite technologies are relatively new to development discourse, and there is little historical data available to guide thinking on their potential value. Also, there is only limited awareness or capacity present in the development community. Another contributing factor to research quality is that the overall program is underpinned by a rigorous system for management and oversight from research solicitation through research monitoring to results synthesis. This process is multi-faceted and logically applied to monitor both performance and opportunities for higher-level learning and knowledge sharing. For example, DIRTS research concluded that low yield, low input production systems work as well as anything for smallholders reliant on rainfed agriculture in northern Ghana. This finding has significant implications for Feed the Future programming in northern Ghana—and beyond—given the current focus of both Agricultural Development and Value Chain Enhancement (ADVANCE) and Assets Technology and Transfer (ATT) on adoption of new technologies. While research quality is high, the ET has concerns that its potential value and contribution are prone to being undermined by insufficient consideration of the stakeholder landscape within which the research occurs. Of the research activities reviewed in depth by the ET, none were able to persuasively describe their stakeholder landscape, key decision-makers, or a holistic dissemination approach. In particular, there was an absence of engagement of the senior officials who would be central in facilitating the use of research results for policy consideration and impact. While researchers desire to be thorough and have their results clearly determined before engaging higher levels of government, this is thought to undervalue the importance of cultivating relationships and building the capacity of senior officials to comprehend and advocate on a technology’s behalf. This subject will be discussed in more detail in the section on Outreach and Dissemination. While only a trial at this point, work undertaken through the Ghana Credit and Insurance research deserves further investigation and expansion as a way of both opening up lines of more affordable credit and also helping develop better smallholder index insurance products. As with the crop index insurance product, more work remains to be done in Ghana at the policy level in securing endorsement of index insured loans as a policy for extending more production credit to small farmers. 25 Despite improved risk mitigation options and removal of other barriers, the reality of many poor farmers globally is that they are extraordinarily cash poor, making investment in new technologies problematic even when the will is there. This reality needs to be central to research conceptualization, otherwise research can quickly become relevant only to middle class farming households. The degree to which gender is deliberately and proactively built into research approaches is another area that could be strengthened, given that the many complex, additional layers of vulnerability faced by women (especially female-headed households) are well documented. While important gender analysis is occurring, it is not always explicitly stated in research proposals and approaches, nor is it a focus of dissemination efforts. This quite possibly relates to gender not being highlighted in UC Davis’ ME application nor being given priority in research grant solicitation or scoring of research proposals. This runs the risk of a missed opportunity in terms of considering and working to build a gender￾disaggregated evidence base that helps ensure policy recommendations are sufficiently nuanced to best address the needs of women and men. Gender analysis should not occur as an afterthought. It must be proactively structured from the outset to ensure best possible outcomes. While both outreach and deepened gender analysis require investment of time and resources, the ET feels that these investments can be easily justified given the RFA emphasis on ensuring research approaches that are well-positioned to achieve development impact. RECOMMENDATIONS FOR EQ1 AMA IL research is occurring across a highly innovative landscape where little preceding data is available. Both index insurance and improved varieties present relatively new frontiers for research, while also being subject areas that are rapidly evolving given the fast pace of technological advancement in these areas. Given that ever-greater clarity is being achieved within the research portfolio regarding the various technologies the program focuses on, opportunities exist for future directions to be increasingly specific and targeted, further fine-tuning knowledge developed during earlier iterations of the Lab. Overall, it is recommended that a further iteration of the AMA IL be supported by USAID to further advance research and progress in its key areas of innovation. This will allow further focus on workable permutations of innovations that penetrate the core challenge of low levels of adoption of technologies by smallholders, such as strategies to further reduce basis risk of index insurance products to eliminate obstacles to technology adoption. The combining of credit and insurance as a model for opening up smallholder access to formal lines of credit also offers an important line of inquiry requiring ongoing attention. Close monitoring of technical advances, especially in relation to satellite weather technology, is another area with the potential to be bundled into and add value to a range of research needs. While Section 6.0 of this report—Future Directions—puts forward some ideas regarding how this focus might be achieved, there are many different (and better informed) audiences that can help shape the future research portfolio, including BFS, GAN, and multilateral agencies such as the World Bank and the International Fund for Agricultural Development (IFAD), to name but a few. The private sector also has capacity to contribute important perspective to the future research agenda. Agencies such as the Swedish satellite weather company Ignitia, pan African insurers such as UAP, and seed producers are invested in AMA IL’s areas of focus, and can make a valuable contribution to the direction of future research. However, opportunities exist to more strategically position AMA IL work by bolstering the process through which research is identified and carried out. These include: • More deliberate up-front consideration of the policy environment that research will occur within, and identification of key people on that landscape who can provide policy-positioning advice. 26 • • • More analysis undertaken, and greater strategy applied to ensuring research approaches are sufficient to ensure gender aspects and difference are fully understood and optimized through the research process. This could be achieved by stipulating the need for a gender analysis component within the call for proposals, and by it being a criterion measured in the process of assessing research proposals. More emphasis in AMA IL research upon having a strategy for and placing importance on understanding the gendered nature of poverty in every context that it undertakes research. For example, understanding gender issues and how they pertain to index insurance access and policyholder risk can shape strategic considerations, future research needs, and policy advice and enhance chances of adoption of index insurance and other risk management products. Greater emphasis being placed on ensuring a well-focused gender lens adequately captures gender dynamics of the research landscape, through deeper analysis of issues such as: o o o o o Workload factors and gender roles related to adoption of new technologies; Household budget management and decision-making; Gender issues as they relate to accessing credit and other financial services; Women’s mobility and how that affects awareness and adoption; and Impact of low levels of dominant language literacy (including financial literacy) on technology adoption. 4.2 OUTREACH AND DISSEMINATION The importance of high-quality outreach and dissemination was highlighted in the RFA as being critical to achieving the development impact that USAID increasingly seeks from its ILs. As seen in the RFA, “achieving development impact” is explicitly linked to effective dissemination of results and outreach, and by including key stakeholders. The RFA states that development impact will be aided by efforts that: • • • Ensure policy/programming relevance; Promote adoption of policy and/or programming recommendations that research activities generate; and Engage policymakers, USAID Mission staff, private sector representatives, and other stakeholders as appropriate—even in early stages of the research design and implementation. While these linkages are valid, the ET acknowledges that “development impact” is only very rarely achieved quickly and needs to be understood as a longer-term process and investment that may not yield results for years. Having said that, positioning for policy impact is of critical importance and requires a considered and strategic approach that aims to attain buy-in and build ownership of stakeholders, both in-country and externally. Given the above, consideration of outreach and dissemination was assessed based on its successes or failures to realize the following outputs: • • • • User-friendly website to store and make all resources publicly available; Meetings with stakeholders to explain results; Policy and program recommendations and other materials responsive to stakeholder needs; and Variety of events to disseminate results and insights, including seminars, webinars, and conferences. Outreach for the purpose of promoting opportunities for adoption is a complex and nuanced aspect of research work. It is the experience of the ET that “successful outreach” is generally hard won and built off strategic and determined engagement of key stakeholders to that research. While the language of “outreach” is recurrent across the AMA IL portfolio, the ET observed that actual outreach practice is 27 undermined by a range of structural issues that are not easily overcome, such as the limited time spent in country by U.S.-based researchers. The ET observed that the standard operating environment of a typical Ministry of Agriculture involves a multitude of different “aid and development” entities positioning and lobbying to adopt their desired approach and policy agenda. It is also common within Ministries that senior positions are a revolving door, posing relationship management difficulties. Similarly, USAID Missions are prone to rapid staff turnover, presenting similar difficulties. This busy and shifting environment within agriculture Ministries presents challenges for a U.S.-based research mechanism whose primary clients and delivery agents are U.S.-based universities. For example, the research PIs are on the ground each year only briefly (mostly for no more than two to three weeks). It is therefore unrealistic that they be the main driver of outreach and policy positioning, even though they are generally the person best able to represent the research. This scenario is a key factor for why policy dialogue within AMA IL-supported research is so often left until after research is completed, and (in the opinion of the ET) misses opportunities of creating awareness, interest, and ownership from policymakers throughout the research process. Although researchers in all cases have local partners, their relationships are more geared towards research implementation, management, and oversight. While there are exceptions, local-level research partners are commonly not comfortable, confident, or capable of assuming the role of relationship management and outreach activities with policymakers, except in cases where a deliberate effort has been made to identify a specific partner capable of assuming that role. For example, within the Ghana DIRTS program a strategic partnership with IFPRI was agreed upon to ensure the support of a longstanding, high-profile agriculture expert based in Accra, someone well-known to senior government officials. While in this particular case there has still been only limited engagement of senior government officials in relation to the DIRTS research, in principle this model has the potential to resolve some of the challenges described above. In recognition of the challenge of achieving successful outreach, the AMA IL ME in UC Davis has taken significant steps over the past years to bolster its capacity to better support outreach and to be more proactive in getting program learning before those who have the greatest capacity to utilize it. The approach adopted by the UC Davis-based communications team involves a considered blend of different approaches. These are described below. While the ME has in-house communications expertise, it needs to be noted that given their focus on engagement of U.S.-based actors (at USAID and World Bank, for example), they are already stretched thin. This is exacerbated by the fact that AMA undertakes research in 15 different countries. More importantly, they are not sufficiently familiar with the specific content and stakeholder context to take on a lead outreach role supporting individual research activities (though they are well placed to provide strategic advice on outreach and dissemination as well as linkages to the regional and global-level mechanisms that AMA IL is trying to influence and maintain relationships with). Considering these challenges, the importance of research efforts having a strategic partnership capable of facilitating reliable and regular stakeholder management becomes even more critical. While an aspect of the ME’s communications ambition is to better engage at the national level, it appears more realistic and achievable that its team facilitate learning events at the regional and global levels around AMA IL’s areas of specific interest. An important guiding document for AMA IL outreach and dissemination is its Communications Strategy, which clearly and logically identifies different audiences and describes the inherent challenges of researching innovation. This is an important milestone for AMA IL and positions it well for a more coherent and aligned approach to outreach and dissemination moving forward. 28 4.2.1 To what extent has the AMA IL and its research projects effectively analyzed, synthesized, and distilled research results into actionable information or recommendations? FINDINGS Publications and analyses being generated through AMA IL are of a high standard and provide faithful description of what was planned, and the results achieved, as documented through the following forms: • • • • • • Periodic project- or subject-specific policy briefs, focusing on strategies/issues and disseminated to a wide range of practitioners as tools for program decision-making; Press releases and external media coverage on effective interventions; Twice-yearly newsletters on specific AMA IL research projects and their accomplishments; AMA IL website—providing access to policy briefs, reports, and links to timely research papers; AMA IL Annual Reports, covering Lab accomplishments at all program levels; and AAs received to advance the state of practice (for example, the “Advancing Index Insurance” Update). At this point in time, a total of 33 academic papers and 110 policy briefs have emerged from AMA IL￾supported research. It is reasonable to expect that the number of academic papers will rise significantly as research activities conclude, given full-time maturity is a prerequisite for publishing final results. However, of the different activities reviewed in depth by the ET, all had yet to collate or publish final results. In the case of Tanzania DT Seeds, this is explained by research still having two years left to run. Significant effort has gone in to enhancing the AMA IL website as a central storage point for all information ready to be publicly available. Furthermore, the AMA IL website is aligned with Agrilinks, allowing Agrilinks users to be easily directed to the website. There has also been significant recent effort to improve the AMA IL website’s accessibility, including development of a more user-friendly site structure that allows ease of flow from multiple entry points. This effort has seen a rise in website traffic over the past 12 months, with the average number of “page views having more than doubled in the twelve months since the website was updated, from 39 per day to 83.”6 Examples of enhanced communication approaches (in addition to the website) include: • • • • • • Supporting researchers to improve communications for potential and strategic application of policy briefs; Supporting researchers more generally to develop communications plans; Coordinating targeted opportunities to engage strategic audiences, such as an index insurance “Ted Talk” aimed at senior USAID staff (which also drew in USAID discussants, including their Senior Economist); Holding webinars around key issues of interest to AMA IL; Setting up of an AMA IL YouTube channel (https://www.youtube.com/channel/UCAW5KXJpbGFwvq0htSRw8dA); and Revamping the monthly newsletter. Importantly, the program adopts a campaign approach around key areas of interest and aims to promote these areas of interest through multiple, related media tailored to specific audiences’ needs, interests, and opportunities. For example, a multi-faceted campaign was initiated to raise awareness of the benefits 6 Data provided by Alex Russell, AMA IL Communications and Web Marketing Specialist; September 29, 2017. 29 and challenges of using index insurance as a way to support small-scale farmers facing a range of climate￾related threats. While the above represents strong performance at a global level, the ET observed that, beyond these universal levels, there were insufficient arrangements or allocations in place for regular and evolving communication and policy dialogue at the national or private sector levels. This was reflected in the absence of detailed stakeholder mapping and the absence of the clear development of a road map for policy dialogue within individual projects. Engaging senior policymakers and provoking their interest and ownership of the research and support for the development of polices reflective of and supportive of results surfaced as one of the major challenges for the overall program. While both USAID and UC Davis view measurement of policy traction at the national level as only one aspect of development impact, it is felt by the ET to be (at least initially) the most likely point of impact, and also an important precedent from which further policy influencing work could occur. This is not to suggest that policymakers, senior officials, the private sector, and other key stakeholders have not been engaged. However, this engagement is often felt to be less than a priority and subsequently lacks strategy and a clear sense of purpose. Thus, the extent that results have been distilled and made available to be presented in the form of innovations ready for dissemination (actionable) has been undermined by the absence of a focused outreach and dissemination strategy. 4.2.2 What are examples of successful outreach aimed at achieving impact (or of adoption of recommendations)? FINDINGS The AMA IL outreach works at multiple levels focused around key issues, campaigns that draw together different learning, and at the individual project level. The ET found that recent examples of outreach at each of these levels include: • • • Issue-based—deepening understanding of a specific approach: o o GAN Peer Learning Platform for Policymakers on Index Insurance; and Promoting Resilience Through Index Insurance: The Way Forward; Washington DC, April 21, 2017. Campaign-based—considering bigger issues being uncovered through AMA IL research: o o Evidence to Action: Towards Data-Informed Policy and Practice in: ▪ ▪ East Africa; Nairobi, May 2017, and West Africa; Accra, July 2017; Mind the Gap: Exploring the Disparities Between Smallholder Farmer Practice and Potential; Washington DC, November 2016. Project-based—aimed at building awareness and a local constituency of support for research findings: o o Enhancing Smallholder Productivity in Kenya: Evidence from a Randomized Controlled Trial of New Seed Varieties; Nairobi, February 2017; and System of Rice Intensification in Haiti: Agronomic Effects & Household Impacts; Port Au Prince, December 2016. Long-standing and highly-regarded research initiatives such as IBLI demonstrate the effectiveness of strategic outreach and thoughtful dissemination of results, while also highlighting the importance of viewing the potential for development impact of research in a longer timeframe. While IBLI’s longevity and status give it a platform to communicate from and assist it in opening policymaker doors, the basic tenets of that approach can be applied to all AMA IL research in terms of being clear on the stakeholder landscape within which research occurs, enlisting of strategic partnerships, such as that with the 30 International Livestock Research Institute (ILRI), and strategically working to identify key players within that landscape for regular, tailored engagement. While the Communications Team in Davis, CA is supportive of project-based communications planning, it is beyond their capacity to systematically fulfill this role across all AMA IL research activities. As described above, outreach and dissemination are therefore largely dictated and directed by researchers who have varying levels of commitment and place uneven priority on the task. Even though dissemination is clearly understood as important, it seems to be often planned in an ad hoc and opportunistic manner and without a comprehensive strategy, as demonstrated by the absence of systematic awareness raising with senior policymakers and strategists. Evidence supporting this point of view could be readily seen in both Ghana and Tanzania, where very little national-level engagement has been undertaken. In Ghana, both projects were unable to name senior-level officials within the Ministry of Food and Agriculture officials in Accra with whom they had maintained any consistent form of dialogue. Even within the northern zone of Ghana, engagement primarily occurred at the sub-provincial level. The ET was directed by the research team to District officials as evidence of their engagement of government officials. This situation was more or less mirrored in Tanzania. One explanation offered for this situation is that ILs are focused on assembly of generalizable research that is not only relevant to the country where research is carried out, but also able to speak persuasively to the global dialogue about ways to promote food security and reduce rural poverty. The ET supports this position but feels that it is realistic for different levels of dissemination to occur simultaneously. Furthermore, the ET sees adoption as most likely achieved in the context of the country where research has proven effective; early adoption can be a powerful tool in subsequent research and dissemination efforts. 4.2.3 In what ways could research results be better communicated with different stakeholders, including USAID Missions, IPs, public and private sector partners, including technical and non-technical stakeholders? FINDINGS While it is accepted that clear results can only be shared at research end, it is vital that key people be clearly identified and at least given the opportunity to follow research progress and direction. Such an approach can foster ownership, which in turn can facilitate policy positioning. While communication of research results is offered in situ to local institutions at village and district levels, and with the local private sector, communication is limited and less structured with: • • • • Senior government officials and institutions; USAID Missions; USAID country projects (notably Feed the Future-funded); and Private sector peak bodies. While each of these target groups presents challenges regarding easy access on a regular basis, the energized communications capacity now available within the ME needs to continue to work to set expectations and a standard for outreach and dissemination at the country level. 4.2.4 In what ways have different partners (especially USAID Missions and the private sector) been engaged in the research process? What opportunities are there to increase this engagement? 31 FINDINGS Predictably, there appears to be considerable variability across the program in terms of communication with and engagement of different key stakeholder groups, including USAID Missions and the private sector. One dynamic that is common and worth mentioning is that most research occurs in a country context where Missions oversee a complex array of agriculture and livelihoods activities, most of which occur under the banner of Feed the Future. This is significant on multiple fronts. Mission staff are often extremely busy in support of the many large Feed the Future development efforts under implementation, including undertaking efforts to leverage host government institutions in support of projects’ objectives. Commonly, these larger Feed the Future projects have (and should have) considerable crossover and shared areas of interest with IL research, both geographically and technically. There is therefore significant opportunity for information sharing, shared learning, and mutually advantageous relationships. However, in both Ghana and Tanzania, there was at best only limited coordination between researchers and Feed the Future-funded programs, and also limited proactivity at the Mission level to ensure lines of communications were established and maintained. In Ghana, multiple U.S.-funded activities work with Ghana Agricultural Insurance Pool (GAIP) in one form or another, yet there was no point or mechanism through which these different institutions came together to consider their approaches, experience, and ideas in relation to this faltering, yet very important, institution. Similarly, the ATT and ADVANCE (both Feed the Future-funded) activities were unfamiliar with Ghana DIRTS and the Ghana Credit and Insurance, including the fact that DIRTS was responsible for the CEAs their staff were encountering in their target areas. In the case of ADVANCE, they supported a similarly conceived Community Marketer model, and had spoken with GAIP regarding the potential for these women marketers to sell index insurance policies. It is hard to understand why these different activities were not in close communication. A similar situation was observed in Tanzania, where the long running NAFAKA Tanzania Staples Value Chain program (NAFAKA) claimed to have no awareness of the various AMA IL research activities, despite many of them occurring in common geographic areas of shared technical interest (DT seeds, agricultural insurance, soil improvement, application of ICT). Within this scenario, the Mission in Dar es Salaam is very busy, and they acknowledged that they have not taken action to broker this relationship. Even though AMA IL has evidence that they had in fact approached NAFAKA to brief them on their program, the overall context highlights the limitations of the current outreach approach, and the lost opportunities that come through the lack of a systematic approach to engagement (see Annex M “Potential for Integrated Programming and Learning—AMA in Tanzania Program”). Importantly, UC Davis and researchers were able to cite evidence that approaches had been made and information shared with each of these programs. This seeming confusion is likely explained by a variety of factors related to staff movements, general busyness, and general information overload within the hectic nature of program implementation. However, it is felt that opportunities still exist in relation to better communication and coordination when a clear engagement strategy is put in place. In terms of researcher engagement of the private sector, this appears to primarily occur at the field level as research explores how to improve market performance and linkages. While local private sector engagement is important and understandable, higher-level private sector engagement would also seem strategic and mutually beneficial. For example, in Ghana, the Ghana Seed Producers Association (peak body) was not engaged despite both research activities having an interest in ensuring reliable supply of improved seed varieties. However, there were notable exceptions where private sector engagement has been more profound, with consideration of macro issues front of mind. For example, through the Tanzania DT Seeds Project: 32 1) three domestic seed producers are central, invested, and highly supportive of the research approach; and 2) UAP (a pan African insurance provider based in Kenya also involved in IBLI) has been supported to consider how it might better engage smallholder farmers, and is now looking at insurance product development well beyond the confines of the DT Seeds Project. At the local level, partnership with local seed companies is proving successful in channeling insured seeds to local distributers and farmers. However, sustainability remains an issue in the absence of policy backup that influence quality, price, availability, and accessibility. In Ghana, the Credit and Insurance program’s partnering of the Association of Rural Banks has helped positioning for effective dissemination of findings within the group and allowed the important rural banking structure to develop awareness and capacity related to assessing the potential of new financial instruments. At the global level, efforts have been made (notably through GAN) to globalize key policy issues such as steps towards setting minimum quality standards for index insurance contracts. This ongoing effort has increased emphasis on the importance of better addressing basis risk and can be reasonably credited, at least in part, to the AMA IL’s work in this area. A related contributing factor is the AMA IL Director’s article based on “poverty traps” research that has helped shape discussion around poverty dynamics and how to design interventions that address poverty effectively, thus increasing the prospects of more focus on risk and resilience. Traction on this topic can be seen through the Global Food Security Research Strategy recently released by USAID that draws on AMA IL research in this area.7 CONCLUSIONS FOR EQ2 Some major challenges faced by AMA IL include the need to: 1) complement generated innovations with strategic thinking and further approaches that facilitate broad-based understanding of the dynamics around which the many facets of an innovation works (e.g., how financial technologies work if bundled with seed technologies); 2) better disseminate understanding of spillover effects from the interaction between the various technologies and innovations (DT Seeds, Index Insured DT seeds, formal and informal insurance, improved versus recycled seeds); and 3) understand, identify, and disseminate potential strategies that can build from the impact of trained farmers (sustainability of the CEAs, spillover effect of the contact farmers). The opportunity to progress each of these challenges towards success (development impact) is, in the opinion of the ET, determined by the degree that outreach and dissemination of findings and recommendations is effective and actioned. For example, both research activities occurring in Ghana are producing results consistent with the Government of Ghana’s direction and objectives, yet little awareness exists among senior government officials of the research, let alone dialogue on how the findings inform and could potentially shape policy. While AMA IL’s focus was regarded by the ET as universally relevant to context, the ET is concerned that insufficient emphasis is demanded of proposals to be clear and concise in their proposed outreach and dissemination focus and approach. An observation of the ET was that beyond universal levels, there were insufficient arrangements or allocations for communication and policy dialogue to take place at the national or private sector levels. Absence of a detailed stakeholder or road map for achieving policy dialogue is seen to be a weakness within the research protocols of most projects. While this can be partly explained by the various structural challenges described above, solutions need to be sought to ensure the regular engagement of senior policymakers, with the aim of provoking their interest and ownership to help ensure that AMA IL’s high-quality research is achieving traction with those who can carry its policy. 7 U.S. Global Food Security Research Strategy, p. 20. 33 Holding one-to-one and multi-stakeholder fora to present and discuss results coupled by intensive communication efforts (media, extension) would enhance the knowledge and understanding of key officials and policy influencers and the private sector of research results and the options. Inclusion of higher-level government officials and academics in such sessions would also be strategic. RECOMMENDATIONS FOR EQ2 While each context is, of course, different, each research activity moving forward should be expected, as a matter of process, to have: • • • • • Outlined a detailed description of the stakeholder landscape, including a plan and roles and responsibilities for engagement of different actors on that landscape (this should be updated at the mid-point of the research, given the inevitability of significant change within that landscape); Ensured that at least one senior official within the host government is identified at all times to act as a contact point and champion for strategic thinking and dissemination of research findings; Identified a realistic number of key points within the research cycle where a formal information sharing event will occur, presenting an opportunity for different relevant parties to be aware of, and have input on the research approach; Explored opportunities for strategic partnerships that allow for the type and frequency of engagement of senior officials that simply cannot be maintained by a PI, given that the PIs are U.S.-based and fully employed within their institution; and Communicated with Missions and reached agreement on a contact point and model, and frequency for the communication of updates (in close cooperation with the team at UC Davis). At the program level, the UC Davis-based team should make contact with each relevant Mission (or at least those with multiple activities) to outline the work of AMA IL, its direction, its activities on the ground in that particular country, and other AMA IL-supported activities that might be of interest based on review of the respective USAID country strategy. Through this dialogue, clear agreement should be reached on a suitable contact point within the Mission and a preferred mode of delivery for program and activity updates. Support to the facilitation of relationships with other U.S. investments should also be sought. To ensure different options and dissemination styles for different audiences and learning types, each research institution should continue to strengthen research results dissemination through various means. For example: 1. Organize and conduct more frequent “peer learning” events addressing innovations, findings, and “ways forward” for specific innovations that have been field tested by AMA IL and its research partners. An efficient and logical entry point for such approaches appears to be at the regional level, and through regional practitioners, as was successfully undertaken by the Lab in 2017 with a “government policymakers” session in Kenya involving regulators from various African countries in discussions of index insurance policy. Another example is a recent AMA IL “peer learning” session held with insurance carriers in Africa aimed at discussing “client value.” 2. Increase use of “webinars” involving technical and non-technical stakeholders in discussions of innovations and learnings in specific areas, such as opportunities for introduction and scaling of index insurance and bundling with improved production technologies. 3. Develop and implement a more deliberate strategy for communications and collaboration with key stakeholder groupings within national policy contexts. For example, in Ghana foster understanding between the Bank of Ghana, MOFA, GAIP, the National Insurance Commission, and relevant stakeholders to promote learning and potential mutual benefits of putting in place policies and program strategies aimed at increased access to credit for smallholders and improved risk aggregation with banks and other value chain stakeholders. 34 While unlikely to have impact at the policy level, it is important to note that outreach, dissemination, and communications approaches are also required at the local level where research is occurring in order to ensure awareness, political support, and a level of ownership of the research activities occurring. Examples of local outreach needing to occur include: • • • • Sensitization of sub-national leadership; Provincial and district agricultural officials; Community leadership (tribal/ethnic leaders); and Other development actors active in target areas. 4.3 POLICY The UC Davis proposed a three-step process linking research to long-term development impact through obtaining policy-relevant research results, distilling research results into recommendations for policy and programming, and then undertaking actions that enhance the probability that recommendations are adopted. ET findings are presented from the perspective that achieving policy impact is complex and often requires extensive time to come to fruition. For this reason, the ET talks more about policy potential, policy positioning, traction with policymakers, and evidence that a strategy and road map towards policy impact is evident. 4.3.1 In what ways was policy research relevant to current concerns, objectives, and needs of private and public-sector stakeholders, USAID Missions, and BFS? FINDINGS AMA IL has worked to ensure the relevance of its research portfolio to both the global and national agricultural research agenda relative to a better understanding of resilience and barriers to broad-based growth. All research topics investigated during this evaluation were reflective of issues of national concern, even though these needs were not always articulated in existing national policy. This context relates to AMA IL areas of focus that are truly innovative with little or no pre-existing knowledge or capacity present within the national context for subjects such as agricultural insurance or satellite weather technology as tools of relevance to smallholders. These subjects are therefore commonly not yet visible within existing policy, thus making the achievement of policy traction even more complex and challenging. Agricultural insurance as a tool for smallholders is a good example of a pressing issue and need that is much discussed, but not commonly reflected or prominent, within existing national policy and strategies. Such a scenario highlights the importance of AMA IL and of each research project structured and strategic in how it intends to engage and affect the policy realm. Overall, the AMA IL research portfolio is aligned and supportive of Feed the Future objectives with all research able to be seen and understood through the lens of the Feed the Future results framework: 1. Objective One: Inclusive and Sustainable Agricultural-Led Economic Growth 2. Objective Two: Strengthened Resilience Among People and Systems 3. Objective Three: A Well-Nourished Population, Especially Women and Children To ensure alignment of AMA IL research with different key agendas (Feed the Future, globally identified needs, national agriculture plans, etc.), measures were put in place by the ME for screening research proposals, including a rating system that deducted points from proposals that were not regarded as having global policy relevance or alignment. Similarly, while AMA IL will consider research proposals outside Feed the Future focus countries, it gives priority to Feed the Future countries through its proposal ratings system. 35 While research focus was seen as universally relevant to both the Lab mandate and local context, there is only limited emphasis placed on proposals needing to explicitly demonstrate how they link to specific host government, or USAID strategies. An observation of the ET was that beyond the universal level, there were insufficient arrangements or allocations for communication and policy dialogue in place at the national or private sector levels. A result of this, a context of underdeveloped strategy for affecting policy could be seen in Ghana and Tanzania in terms of challenges being faced identifying and engaging senior policymakers and provoking their interest and ownership in developing polices supportive of scaling of results. It is also noted that AMA IL research is commonly co-financed by other interested donors, including collaboration with CGIAR system entities and national research institutions. This approach could, in theory, enhance opportunities for policy engagement and traction, given the different types and depths of relationships maintained by different research “investors.” However, there was no clear evidence uncovered by the ET to demonstrate any proactive use of different partners’ networks as entry points to engage policymakers. An interesting example of the potential for a multi-donor approach can be seen within the Ghana DIRTS program. One partner in this multi-donor activity is IFPRI. IFPRI is more generally supported in Ghana through the multi-donor Ghana Strategy Support Program (GSSP), which receives core funding from USAID to support its stated functions as “a research, communication, and capacity-strengthening program to build the capabilities of researchers, administrators, policymakers, and members of civil society in Ghana to develop and implement agricultural and rural development strategies, with core funding from the USAID in Ghana.” Given this mandate and clear cross-over with the objectives of AMA IL, IFPRI appears to be well-positioned to support policy-influencing opportunities emerging from the implementation of DIRTS. Despite this context, there appeared to have been only limited strategic engagement of senior Ministry of Food and Agriculture officials in Accra during the DIRTS research process (which is now drawing to a close). This is despite the ET hearing directly from such officials of their interest in agricultural insurance-related research and expressing frustration at not having the opportunity to more closely follow progress. The ET found that despite the DIRTS research being set up in such a way that it had a potential “policy influencer” on board, there was still seemingly little strategy applied to how the influencing would occur. Overall, there is room for more strategic policy positioning of research efforts at the national level. This should be possible, given that researchers have, for the most part, proven successful in securing support from country-based international and national research partners, civil society, the private sector, and district- and village-level administrators in day-to-day research efforts implementation. This demonstrates that they are investing time in stakeholder engagement to support implementation that could, with a modest amount of additional effort, also be geared toward laying groundwork for more specifically policy-focused engagement. While private sector needs and priorities are reflected and recognized in research, they are not highly visible in most project protocols. When they are, it is primarily at the micro and local level and occurs through implementation rather than strategy framing. However, there are notable exceptions: 1. A fundamental aspect of the DT Seeds Project in Tanzania, occurring in collaboration with CIMMYT, is to support national seed production capacity in Tanzania working in very close partnership with three national seed producers along the seed value chain (who each ensure quality seed production, promote certified seeds with village focused distributers, and support training of Extension Agents in use of these seeds), with the added innovation of bundling a seed￾based index insurance product in to the cost of the seed. These seed producers are sizeable entities with the potential impact being progression of a national private sector seed production capacity. Importantly, these companies also feel a strong degree of ownership and potential benefit in this particular research process. 36 2. Ghana’s national agriculture insurer, GAIP, is a central actor within the research of both the Ghana DIRTS program and the Credit and Insurance program that works with rural banks in Ghana. GAIP is governed by 17 insurance companies and is endorsed by the National Insurance Scheme. Importantly, all agricultural stakeholders are in agreement that Ghana needs an agricultural insurance scheme capable of providing products to smallholders; however, there are reservations as to whether GAIP has the capacity or is the right vehicle to effectively carry the concept forward. GAIP also faces problems of survival due to inadequate funding and staffing, limited levels of active support for it and the model within the Ghanaian government, and its inadequate business model which limits outreach and promotion of products. This has resulted in limited market penetration. Despite this context (which is in most respects beyond the control of research programs), GAIP has benefited significantly from both the knowledge generated through AMA IL research and the capacity the two research activities have helped foster among GAIP staff. Central to the AMA IL approach is production of high-quality policy-oriented publications, in support of policy and program recommendations prepared by researchers, involving published academic papers, working papers, and policy briefs. In addition, 87 formal presentations have been made to key audiences and 83 stakeholder meetings convened.8 A breakdown of these figures by specific research project is attached at Annex B. While these outputs are globally accessible via the AMA IL website, there appear to be no formal mechanisms to ensure these findings are proactively and strategically put in front of policymakers, Missions, and others with potential of critiquing and/or progressing their findings. This restricts the degree to which the policy and program recommendations of the various outputs are reflective of consultation and input of policymakers and influencers within the country or regional context, which in turn creates distance in terms of active discussion around their viability and adoption. Based on meetings with both Ghana and Tanzania USAID Missions and various Feed the Future-funded projects, it appears there is also a lack of cohesive communication between researchers, the AMA ME, and USAID Missions, undermining the potential for Missions to support the policy outcomes emerging from different research. Within this dynamic is a need for USAID Washington to assist in facilitating improved communication and ensuring the right information is shared strategically within Missions. The factors underpinning this are complex, though mostly relate to an interplay related to the workloads of Missions, the limited time spent in-country by research PIs, and the somewhat peripheral nature of the research to Mission “core business” when compared to the larger multi-million-dollar development programs that require immediate, day-to-day attention of Mission staff. It is also significant to note the complex management challenge posed for Missions by the breadth of IL outputs. For example, three quarters of all ILs have activities on the ground in Tanzania, making it very difficult for this Mission to keep abreast of all that is being researched, let alone take action in support of each effort. One step towards bringing greater cohesion to Mission-IL relations is that, moving forward, all Feed the Future￾funded activities (including IL research) will require Mission concurrence before commencing. Given this current context, the onus of responsibility for engagement of Missions requires the IL and USAID Washington work strategically to facilitate linkages that make sense and add value at the country level. Because PIs are only briefly on the ground each year, it is for the most part unrealistic that they be the sole driver of outreach and policy positioning—especially during the research process. This is a key factor as to why policy dialogue is so often left until after research is completed, missing opportunities of provoking early interest by policymakers. In most cases, national-level partners are neither comfortable nor confident to assume the role of relationship management (with the exception being 8 Details provided to ET by AMA IL team in Davis. 37 efforts where a deliberate effort has been made to identify a partner capable of assuming that role). And given that AMA IL ME in Davis is less familiar with the content and stakeholder context, they too are not immediately well positioned to take on that role. As discussed above, the ME has taken steps over recent years to bolster its capacity to better support policy influencing through being more proactive in getting program learning before those who have the greatest capacity to use it. While an aspect of this strategy is to support better engagement at the national level, it is more realistic for UC Davis-based staff to facilitate learning events at the U.S, regional, and global levels around AMA IL’s areas of specific interest. While this approach is both relevant and reflective of practical realities, it also fails to adequately address the core issue of how best to ensure strategic engagement of policymakers in country. 4.3.2 Which, if any, innovations, or research findings are most likely actionable? FINDINGS Globally, the subject of index insurance is innovative, poorly understood, rapidly evolving, and rarely encapsulated in policy or strategy with sufficient rigor and commitment, even though it is emerging as an increasingly interesting “resilience product” given climate change and ever-developing satellite technologies. This context contributes to a policy void despite strong enthusiasm for index insurance. This gap is based on uncertainty and the absence of systems for the marketing and roll out of index insurance products. Different index insurance products are therefore seen generally as innovations with the potential of being carried forward in policy. However, they require even more intensive advocacy than that needed to support the adoption of better understood approaches. The other element of policy positioning is targeted further research. Generally speaking, there is strong interest to consolidate and further develop agricultural insurance products suitable to smallholders. However, it equally appears that policymakers have very few ideas regarding how to bring that to fruition—underlining concerns expressed above at the lack of strategy applied to “taking policymakers on the research journey.” Both in Ghana and Tanzania, government representatives noted that national ownership would be built for index insurance if it was integrated within National Agricultural Investment Plans, and that such an approach would assist in making index insurance innovation policy relevant and actionable. For example, with deeper engagement of national authorities, there is potential for further adaptation of index insurance-backed loan products aimed at enabling small farmers to access credit at lower interest rates, as has been the case in northern Ghana through the Credits and Insurance research work. A senior advisor to the Minister of Agriculture in Ghana thought that such a strategy could be well supported for inclusion in a National Agriculture Investment plan. Trials through Tanzania DT Seeds of bundling insurance into the sale of DT seeds is another simple, yet well-received innovation. Given this particular research project still has two years to run, it should be allowed to run its course for clear findings to be reached. However, early signs are that a market exists for insured seed, and that it is effective in giving farmers confidence to pay more for new seed varieties that offer a degree of protection against climate risk. Achievements such as these are managed by the ME at the global and regional levels through mechanisms such as the GAN, and regional fora such as USAID Global Learning and Evidence Exchange (GLEE) events that AMA IL is now targeting. It also is being enabled by the ME’s newly established strategic alliance with International Centre for Evaluation and Development (ICED), an Africa-based organization focused on regional and continent-wide outreach events. While learning achieved in relation to the potential of CEAs through Ghana DIRTS research is theoretically relevant, needed, and actionable, the process of integrating them within the existing extension system is both politically (there are many unemployed agriculture graduates) and fiscally 38 difficult. Despite this, the Ghana DIRTS research highlights the efficacy of community-based extension services in raising awareness and understanding of new technologies. It is therefore important that further research be undertaken in relation to how to ensure farmers ongoing access to such knowledge and support. Delivery through DIRTS of higher-quality, “hyper-local,” SMS-delivered weather information provided by Swedish private sector company, Ignitia, is another research output valued by farmers and regarded as an important tool from which farmers can make better informed decisions in relation to their agricultural processes. The broader rollout of this mechanism is immediately possible, and its potential value is reinforced by DIRTS research, which demonstrates the degree that such information is shared within a community. Advances in satellite technology are also likely to make such mechanisms even more valuable as they become more advanced, accurate, and location-specific. Rural banks in Ghana have shown interest in directly accessing and utilizing satellite weather data, which could also be used to help strengthen GAIP’s hand as it tries to consolidate its position as a strong agricultural insurer. 4.3.3 What characteristics of these models or innovations make them more policy￾relevant and actionable? FINDINGS To achieve traction for formulating new ideas, it is important that key stakeholder groups have broad￾based awareness and understanding of different research being undertaken. The ET believes that a characteristic of policy relevance and actionability is inevitably its “relevance to context,” and that policy traction will most readily and realistically be achieved within the policy context of the country where research has occurred. This is not to suggest that efforts should not be made at the regional and global levels to seed research findings within key bodies. However, policy take up within country of research should not be underestimated as an important point of leverage for broader, externally-focused policy engagement. While “local impact” is important evidence of research efficacy, it is senior people on the policy landscape—both at national level and beyond—that need to be convinced that knowledge is policy￾relevant and actionable. Relevance to context and responsiveness to needs identified in national strategic planning documents also offer points of leverage from which researchers and the AMA IL team can more easily engage policymakers. At the same time, it needs to be acknowledged that the innovative nature of much of the research places AMA IL ahead of the curve, and without an obvious constituency to engage. However, even in this context, stakeholder mapping and a clear understanding of how and who takes policy decisions is critical in terms of strategic relationship management and effective, efficient, and purposeful engagement of the policy landscape. The important contributing and enabling role of local extension services to research outcomes is a common theme across most AMA IL research. While this has been helpful in building local capacity, it is also a fragile aspect, given how common it is globally for extension services to be underfunded, under resourced, and prone to focusing their efforts where per diems and resources (motorbikes, fuel, etc.) are available. This is a threat to capacity building investments and also to good practice emerging from a research process being sustained in the absence of higher-level ownership of and investment in the innovation at senior levels of government. The important and growing role of ICT and mobile phone technology is another aspect present in much of the research being undertaken, either as a tool for dissemination of information or as a financial services tool. Placing greater research emphasis on understanding the potential of ICT in general, and mobile phone technology in particular, is a potentially interesting future direction for AMA IL research, and consistent with its mandate. 39 Overall, in order to be actionable, the models/innovations developed by AMA IL need government policy support and private sector investment support. For example: • • • • • Index insurance schemes need policy support and consideration in agricultural strategy and sector investment planning [e.g., integrated within the Comprehensive Africa Agriculture Development Program (CAADP) process in all of the African countries]; Index insured, locally produced DT Seeds value chains need support within government policy (quality control, price, competition against introduced seeds); The important role of extension support in promoting new and innovative approaches such as index insurance and satellite weather information needs recognition and action; Private sector capacity to invest in supportive tools such as soil testing; and Diverse and competitive index insurance providers with government policy support and adequately trained human resources. 4.3.4 Is there any evidence of changes to policy, programming, private sector products, or approaches because of research findings? In what ways, if any, did the project monitor these changes? FINDINGS While AMA IL research has had obvious impact on district and village community-level practices, this has not always been progressed towards national or broader policy change. The experience of the ET is that the concept of bottom-up policy-demand creation is not well suited to hierarchical governance systems, such as Ghana and Tanzania—notwithstanding the theoretical decentralization focus of both countries. Understandably, the research approach is focused on generating results that are based on very good and reliable data collected under strictly controlled environments. This is to be commended. However, more rigorous policy dialogue at all levels, including the national and Mission levels, is required. Areas where there is evidence of significant change include: 1. In northern Ghana, work undertaken through the Credit and Insurance project in partnership with the Association of Northern Banks has contributed to some banks changing their agricultural lending policy from one of risk aversion in terms of lending to smallholders to one of promoting lending options to this market segment using an index insurance backed loan, where the indemnity is held by the bank. Most significant in this scenario was the fact that net lending rates offered by these banks (some, not all) were reduced when index insurance was bundled into the lending agreement, compared to previous rates where no insurance was included. Also significant was that overall lending to smallholders increased 50 percent year-on-year, both in numbers of loans and value of loans. While this scenario sits in relative isolation in northern Ghana, it is understood and being monitored within the broader Association of Northern Banks and is worthy of further investigation and promotion. 2. A significant weakness within GAIP has been its pricing of index insurance products, which was originally priced at 10 percent of indemnity in the product offered through Ghana DIRTS. GAIP is now showing greater capacity to adapt to context and has been induced to reduce the index insured premiums to 5 percent in many contexts, and to as low as 3 percent in the case of the northern rural banks described above. Given climate change and more extreme weather conditions, such insurance rates will be significantly more attractive to smallholders and thus offer the organization much needed opportunity to achieve the market traction it currently lacks—especially if issues of basis risk can be more adequately resolved. Within this discussion is an urgent need to develop agreed minimum standards to guide index insurance products forward, for which AMA is planning future research, in partnership with GAN. 40 3. The Ghana DIRTS hypothesis is that smallholders can enhance both yield and productivity when they are supported with knowledge, technical support, improved market access, and risk mitigation technologies (improved seeds, index insurance, weather data). Testing has thus occurred to determine whether access to improved-yield agricultural inputs and agricultural extension advice, when provided individually or in combination, leads to more intensive cultivation and increased earnings among farmers when bundled with and underpinned by rainfall insurance. While research results have been disappointing in terms of farmers increasing incomes, it has resulted in some significant findings all the same: • i • f • The concept of community-based extension has proven effective in transferring knowledge and encouraging farmers to use new technologies. However, this is not resulting in greater ncomes, with the cost of inputs and labor offsetting improved-yield benefits of new technologies. Access to information does not equate to adoption in cash poor farming communities, with only 20 percent of DIRTS participating farmers thought to have made major changes to their arming system in terms of purchase of higher yielding inputs, due primarily to their prohibitive cost. Weather forecasting SMS data have been used by farmers and facilitated adaptation of planting practices that are more weather considerate and risk aware. 4. The premise of the Tanzania DT Seeds research is that index insurance in isolation is not cost effective, and that bundling DT technology with index insurance enhances use of stress resistance technologies that subsequently increase farm production and purchasing power of vulnerable farmers and, as a result, improve nutrition and human development. While still under implementation, the model of integrating insurance within the price of the improved seed product is highly appreciated and supported by participating farmers, and well-suited to promotion of an easy and affordable path forward for promoting adoption of new varieties. In terms of broader program impact and traction towards policy impact, a review of research documentation alongside input from the AMA IL team in Davis, highlights the following projects where encouraging progress is being made: 1. Kenya: Index-based livestock insurance. Over a ten-year period, IBLI has created mechanisms that trigger payments based on remote sensing indicators forecasting forage scarcity and livestock mortality. IBLI was developed as a commercial product but was adopted in 2015 by the government of Kenya under KLIP and is now providing IBLI contracts to targeted households through the KLIP. 2. Ethiopia: IBLI has also impacted across borders. ILRI is currently scaling IBLI in the Somali and Afar regions of Ethiopia through a partnership with the World Food Program (WFP). There is considerable interest in the adoption of this product in other countries in East Africa. 3. Burkina Faso: After positive results through the pilot study in relation to area-yield based index insurance for cotton farmers, AMA IL is now working with private sector partners Allianz, Ecobank, Planet Guarantee, and Sofitex to transfer skills and responsibilities for implementation that will allow scaling up of the intervention in 2018 season. 4. Uganda: AMA IL-supported research in Mozambique has provided compelling evidence that a temporary voucher subsidy not only improved productivity, but also put voucher users on a transformational path, shifting them from near-subsistence farmers to farmers selling more of their output on the markets. This innovation influenced the World Bank to roll out a similar approach in partnership with the Government of Uganda. 41 5. Nepal: Through research undertaken in the evaluation of the Welfare Impacts of a Livestock Transfer Program in Nepal, Heifer International is now working with the research team using poverty trap theory to see how best to design asset transfer size and structure. In terms of process to monitor adoption, the team in UC Davis keeps in constant contact with all research activities. They have a sophisticated system in place for review of reporting and for follow up, and they maintain a constantly updated spreadsheet of all major outputs and events relevant to policy traction. CONCLUSIONS FOR EQ3 The context-relevant nature of the research being generated in Feed the Future ZOI positions AMA IL to have significant policy impact in these countries and regions. However, the overall policy engagement approach lacks sufficient proactivity and strategy to maximize such opportunities. In particular, the ET sees a need for more strategic and considered strategy during the research process to inform and sensitize policymakers of the research direction, intended research destination, how that relates to the current policy framework, and how it might enhance that policy framework in the future. Such approaches would better position research for both policy adoption and impact, as well as lay the ground for scaling and promotion in regional and global fora. Engagement of USAID and other stakeholders strengthened over the course of this phase of AMA IL, although challenges remain in terms of meaningful engagement of Missions, with a view to affecting country and regional strategy. More regular and strategic dialogue and presentation of findings in Washington, DC appears to be contributing to stronger relationships with key decision-makers, as evidenced by the interest of USAID’s Senior Economist to participate in AMA IL events. RECOMMENDATIONS FOR EQ3 Given the quality of the papers and policy briefs it prepares, AMA IL is well armed with evidence capable of affecting policy and strategic thinking. While robust evidence is invaluable in support of influencing efforts, cultivating relationships with policymakers and supporting them to be present in the research process requires further attention. Policy influencing also commonly benefits from a multi-stakeholder approach that enlists other advocates in support of policy outcomes. Tanzania DT Seeds and CIMMYT, DIRTS and IFPRI in Ghana, and IBLI and ILRI are examples of such an approach. While each policy context is different, each research activity, as a matter of process, should: • • • • • Outline a detailed description of its stakeholder landscape, including identification of key policy influencers at the national level and beyond; Work to identify at least one senior official within the host government and national agricultural research arena to act as a contact point and champion for strategic thinking and dissemination of research findings; Aim to ensure policy influencers are aware of research progress and provided opportunities to engage and be active in learning events; Explore opportunities for strategic partnerships that allow for multi-pronged engagement of key stakeholders, through consolidation of mechanisms such as the alliance with ICED in Africa; and Work with the ME (supported by USAID Washington) to ensure strategic communication with Missions, including reaching agreement with Missions on a contact point, and an agreed model and frequency for communication of updates (in close cooperation with the team at Davis). This process mirrors and complements the recommendations related to outreach and dissemination described above, and should be considered as one overall outreach, dissemination, and policy-influencing approach. 42 At the individual activity level, the evaluation observes opportunities for AMA IL to be engaged in the following policy domains: 1. Continue in Ghana and Tanzania to support development of competitive index insurance schemes that ensure agriculture insuring companies supportive legislative and policy frameworks, the financial and technical capacities, and the skills and human resources to develop and market feasible products directly to local dealers and farmers; 2. Support development of minimum standards and government quality assurance measures on index insurance contracts to guide global product development and roll out (logically in partnership with the GAN); 3. Support policy recognition of the important role of the private sector in the development, branding, certification, and distribution of high-quality and DT seed varieties; 4. Consider how to better include the potential of satellite technologies within agricultural policy, including improved weather forecasting and updates in satellite imagery maps; 5. Continue development of a compelling evidence base; and 6. Consider how best to consolidate and ensure availability of effective extension services to smallholders. 4.4 CAPACITY BUILDING The RFA clearly stated a commitment to capacity building, noting that applications must include a strategy to maximize long-term degree training opportunities for developing-country nationals, and placing emphasis on other capacity building and institutional strengthening approaches. UC Davis itself proposed a multi-faceted approach to capacity building, which took into considerations the financial constraints to broad-based training of host country academics within U.S. universities. PIs were charged with developing capacity building appropriate to their research context. UC Davis also proposed empowering host country PIs as key partners in research, while also working to ensure local institutional capacity strengthening. The program ToC includes “capacity development through in-country partnerships” as an output. 4.4.1 How well have the projects and consortium of researchers identified and addressed academic and technical capacity needs of host country stakeholders? To what extent has this contributed to capacity building, broadly defined, in the host country? FINDINGS Capacity building through in-country partnerships was emphasized as a key output within the design of all AMA IL research projects and was also factored into research proposal assessment. This could be readily seen in the projects examined by the ET in Ghana and Tanzania, with capacity building occurring across each of the three areas of AMA IL innovation: • • • Removing barriers to broadly-based growth in agricultural production—including strengthening capacity of extension services, input suppliers, farmer production capacity, insurance providers, lenders of insured loans, and academics; Instruments for improved risk management—including strengthening capacity of extension services, farmers, insurance providers, and academics; and Extension of agricultural finance to small farmers—including strengthening capacity of insurance providers, rural banks (as lenders), and academics. 43 From both the dimensions of awareness and technical capacity, there was and remains a practical need for capacity development in new technological areas such as index insurance, satellite weather technology, ICT application in support of smallholders, and DT seed technologies. However, there was limited evidence of active efforts at “identification” of academic and technical capacity needs within research proposals. Based on data provided by AMA IL, relatively few host county students have received support through the program—the training targets mostly U.S. and European students. For example, 44 percent of the 12 long-term trainees (B.Sc., MS, and PhD) are U.S. citizens and 33 percent are European citizens. U.S. citizens also received 44 percent of the 117 Trainings (degree or no-degree), compared to just 12 percent being African students, despite the overwhelming majority of AMA IL research occurring in countries in Africa. While this is a complex issue, there is a belief within the ME and U.S. institutions that creative solutions can be sought through better partnering of local institutions and enticement for Award and Sub-Award institutions to connect African students with Lab research. More broadly speaking, evaluation findings indicate that significant capacity was built among local partner entities in designing projects that explore and examine opportunities around these innovations, as well as providing training and oversight for local enumerators to collect baseline, midline, and end-line survey and digital data. Locally engaged researchers noted that their own capacity was enhanced by exposure to the high-quality research methodology design and execution being undertaken by AMA IL, in areas ranging from introduction of improved inputs and related extension, to combining improved production technologies and crop index insurance, and coupling of index insurance and credits for small farmer purchase of improved inputs (see Annex N Capacity Building Within Host Universities). One approach employed by AMA IL to increase the capacity of individuals and host country research institutions has been facilitation of short courses created around impact evaluation methodology and how to engage with evidence from these evaluations, as well as training for IPs on project-related technical activities, such as introduction of index insurance and drought resistant seed varieties. Mentoring of students with interest in AMA focus areas also occurs through students working directly with researchers on a project. For example, researchers at Columbia University have been working with a Sokoine University student, as part of research associated with her degree, to determine optimal sampling time for fertilizer recommendations and response of maize to different levels of nitrogen fertilizers, under the AMA IL-supported SoilDoc project in Tanzania. Extension Services Capacity development can be seen within participating national agricultural departments, and specifically within extension services that are consistently playing a key role within different research approaches. In one innovative example in Ghana, CEAs have been trained and established with help from Ghana MOFA to explore the potential role of community-based extension capacity in overcoming some of the known barriers to farmer acceptance of improved production technologies and achieving agricultural growth. Collectively, this approach has produced the spin-off of improved capacity of farmers to adopt some of the production, risk management, and resiliency practices being researched. While farmer capacity development has been significant, their ability to sustain adoption of new approaches is often dependent upon the continued availability of inputs and supply lines established through the research, as well as affordable pricing of inputs and insurance products and the ongoing presence of community-level extension services. Given these systems were set up purely in support of the research and are often reliant on subsidies and incentives provided to input traders, some are not likely to be sustained afterwards. Examples here are the CEAs and community-based marketing agents established under Ghana DIRTS project, which appear unlikely to be sustained in a meaningful way without uptake support from MOFA and/or project-based support. This context strengthens the argument for greater AMA IL 44 engagement of the development landscape in general, and other Feed the Future-funded programs in particular since they could potentially address such gaps. Index Insurance Crop index insurance as a risk management tool for farmers against weather-induced crop losses is a very new field of inquiry, and one where national-level capacity is generally very low or nonexistent. Researchers note that this context necessitates considerable awareness raising and capacity development to ensure capacity deficits do not undermine research findings, and also to build awareness and a constituency in support of the approach. The DIRTS research approach has also enhanced community capacity, through: greater community awareness of new technologies and options for agriculture; capacity building of government extension workers; awareness raising in relation to index insurance; and strengthened local supply lines and linkages between communities and agricultural input traders. While the DIRTS project made gains in terms of introducing an index insurance product to smallholder farmers, lack of field capacity at the insurance provider (GAIP) to strategically market and price the insurance product to reach and serve small farmers was a major factor in the relatively low levels of farmer product intake. GAIP currently only maintains six staff to serve the whole northern Ghana region. Interviews with two of these staff in Tamale highlighted the importance of training undertaken by the various donor-funded projects to develop GAIP staff capacity. At the field level, DIRTS had supported an additional layer of “capacity” for the insurance product through its team of Community￾Based Marketers (CBMs). However, ET interviewing of project field staff indicated that CBMs did not develop sufficient capacity to effectively articulate the index insurance model to farmers, and that these marketers will unlikely continue to promote the insurance products. The simplicity of integrating insurance into the marketing of DT seeds ensured the Tanzania DT Seeds Project a less complex capacity building challenge. This appears to have allowed project staff to work intensively with and develop capacity of their insurance partner UAP, a private provider that appears to be well equipped to quickly appreciate and engage the concept. Building on that experience, UAP is now seeking to develop its own weather index insurance product aimed at smallholders. The Ghana Credit and Index Insurance Project worked with GAIP and with a variety of established rural banks operating in the region, to bring about design, marketing, and extension of an insured loan product for small farmers seeking to purchase inputs. Discussions held with the PI and managers of two participating banks (Garu and Bangamarigu Banks in Upper East Region), signaled that participating banks had developed awareness and capacity, leading to increased lending to smallholders (individuals, as part of groups). Since research end, Garu Bank has developed an adapted insured loan product offered at a reduced insurance premium rate and has used it to continue lending to many of the small farmer borrowers it served under the project (see Annex L Lender Uptake of Insured Credit in Ghana). 4.4.2 In what ways did the consortium of researchers and projects support the participation of the private sector? FINDINGS Common to most AMA IL research is an effort to support and strengthen the capacity of the private sector to better respond to the complex needs of smallholder farming communities. As part of both the DIRTS Project in Ghana and the DT Seeds Project in Tanzania, AMA IL put in place linkages that could strengthen the future capacity of private inputs dealers to develop more effective supply lines to local farmers, as suppliers of seeds, fertilizer, and other products. DIRTS program field staff identified and involved local dealers in target areas as suppliers of inputs to participating farmers, with project￾subsidized input transport to villages. Some of the participating suppliers and village-based distributors have continued to supply inputs to farmer groups after DIRTS. 45 In Tanzania, the DT Seeds Project worked in close collaboration with three medium-sized, nationally￾owned local seed producers to promote quality seed production and distribution of CIMMYT-developed certified DT seeds to commercial and village distributors and support the training of farmers by project supported local extension agents. This project also worked very closely with UAP as a partner identified as capable of and interested in blazing a trail in Tanzania in relation to index insurance. A meeting with a UAP representative highlighted the organization’s interest in progressing insurance products for smallholders, and also the important role played by the project in helping this insurer understand the model, its potential, and the longer-term business proposition of providing a low-cost insurance product to thousands of smallholder farming families in Tanzania (potentially of relevance more broadly in sub￾Saharan Africa, given UAP is a pan African organization). Notably, UAP had previously partnered with AMA IL research through IBLI in Kenya. The ET’s discussions with UAP’s Tanzania agent indicated that this company is building on that experience and is now seeking to develop its own weather index insurance products aimed at smallholders, with the “initial” endorsement of Tanzania Insurance Regulatory Authority (the insurance company regulator) now received. Their aim is to achieve the “volume” necessary to launch a successful product, by working through farmers’ associations and community leaders to roll it out. 4.4.3 What opportunities are there to better support inclusion of the private sector in market viable development solutions? FINDINGS Given the cutting-edge nature of index insurance, and its constant evolution as a product of relevance to smallholder resilience, there is huge need for ongoing capacity development among those potential stakeholders interested in the area. One significant challenge is that index insurance enjoys little support or ownership from many governments at this point in time. This makes it even more important that private sector actors be supported to develop their interest and capacity in the model. There is a clear opportunity in both Ghana and Tanzania to support private insurance providers to: • • • Develop their understanding and skills in relation to product development and marketing; Secure regulatory approval of one or more weather-indexed insurance products aimed at the smallholder market; and Assist in development of the necessary standards for regulating these products. There is likewise an opportunity to provide assistance to interested private insurers to establish working relationships with mobile network operators in order to track farmers, verify who is insured and who encounters crop losses, and verify that payouts are made. In the banking sector, building on the experience of the Rural Banks in Ghana and commercial banks in countries like Kenya and Nigeria with insured loans, there are opportunities to provide technical assistance to commercial banks and insurers in developing, testing, and rolling out insured loan products targeting small farmers as borrowers. As part of the same assistance package, private financial institutions and insurers could be assisted to negotiate premium rates as part of memoranda of understanding (MOUs) defining the product and terms. A third opportunity arises to better support inclusion of the private sector in developing market solutions through provision of technical assistance to banks and other financial institutions in developing ways to aggregate and manage risk involved in extending and servicing loans to small farmers. For example, linking a financial institution with a private input supplier in order to enable small farmers to get credit for their inputs’ needs (aggregated as a group). 46 CONCLUSIONS FOR EQ4 Across all of the AMA IL research activities that were reviewed in depth, there was a cross-section of different capacity building approaches and outcomes. At a formal level, students and staff within academic institutions and agricultural research institutes have enjoyed both formal and informal capacity building opportunities. Capacity has been built across all three areas of AMA IL interventions. However, it is noticed that within formal capacity building opportunities offered through AMA IL, more than 50 percent of benefitting students are from the U.S. and just 10 percent are from Africa. UC Davis acknowledges this weakness, and, while they see its cause to be in large part a systemic weakness, believe that inroads can be made through taking advantage of new post graduate programs and consortia that offer economics training on the continent itself. In Ghana and in Tanzania, AMA IL research programs have taken steps to identify and address the academic and technical capacity needs of host country stakeholders along their chosen value chains, with tangible results in terms of broadly building capacity within key partner institutions. At the same time, the capacity to sustain adoption of some new interventions at the public sector and private sector levels is problematic. This is particularly the case with the capacity of public agricultural extension services to introduce and sustain small farmer adoption of improved agricultural technologies and inputs. There are also limitations related to the capacity of insurance providers to market, provide, and service insurance products focused upon small farmers, as well as their lenders. Given the innovative nature of much that AMA IL focuses upon, inclusion of the private sector in program thinking and capacity development is critical. This is especially the case given both the financial and technical limitations of the public sector in relation to AMA IL-researched technologies. The RFA itself highlights the need to enable private sector provision of risk management and mitigation mechanisms. AMA IL engagement of the private sector can be seen in different forms: • Stronger market linkages were developed between input suppliers and farmers, by allowing input suppliers to access improved inputs they had not previously carried and to make them accessible to small farmers they would otherwise not likely have been able to reach and serve; • Insurance providers were assisted to develop, market, and provide crop insurance and, in the case of the Ghana credit and insurance project, loan insurance enabling small farmers to access insurance contingent credits via local banks; and • Participating commercial banks serving rural communities in the project target areas were introduced to index insured credit as a tool for managing repayment risk and extending short￾term inputs credits to small farmers, a market segment that they did not reach before. This helped a number of banks to develop the capacity to make and service those loans. RECOMMENDATIONS FOR EQ4 There are opportunities to more specifically and deliberately ensure that host country students are supported and benefit from the different research activities supported by AMA IL. Continued support to strategically identified private sector entities that strengthen the national capacity in AMA IL areas of interest is also important. This is especially important in those domains that are truly innovative and cutting-edge, and where very little public sector capacity is present. Supporting the development of the local capacity within the public sector is critical. This is especially true in the need to equip them to be advocates of innovation, notably in relation to index insurance understanding and adoption. More generally, ensuring that extension support to poor rural communities is available remains an important challenge, which requires resolution. AMA IL interest in community￾based extension is an important “capacity investment.” While the model shows potential, it also requires a focused advocacy effort if it is to be adopted within national agricultural policy approaches. 47 Field observations suggest the following actions need to be considered in order to accelerate institutional capacity building: • Developing more effective linkages between public extension services and local agents, such as CEAs, to introduce improved agricultural technologies to small farmers, and to assist and monitor application and adoption. • • • • • • Coordinating public and private sector partners to improve delivery mechanisms for scaling-up innovations that are successfully introduced. This can be done through agricultural extension, inputs in supply chain development, crop insurance, and index-insured loans to small farmers. Developing effective strategies to educate small farmers about the use and added value of improved inputs and weather indexed crop insurance. Strengthening established insurance providers’ capacity to develop, market, and service weather￾indexed insurance products aimed at the smallholder farmer market. Developing and introducing improved satellite-based weather prediction technologies and facilitating their adoption at the local level. Involving private seed companies and inputs suppliers in the process of strengthening the local supply line for improved inputs and improving mechanisms for delivering those inputs to small farmers at the community level. Bringing together inputs suppliers, farmers, insurers, financial institutions serving rural areas, and agricultural products processors and buyers to develop more effective ways to aggregate risk at points in the value chain where stakeholders have the capacity and the interest. 4.5 PROGRAM MANAGEMENT UC Davis’ proposal to act as the AMA IL ME included a description of the proposed management approach, detailing: ME structure and roles; structure and purpose of advisory bodies; processes for competition and selection of primary research projects; systems for research oversight and monitoring; program synthesis and outreach approaches; and plans for monitoring and reporting on overall program performance. 4.5.1 How did the management entity effectively communicate and coordinate with research partners to achieve the objectives of the RFA? FINDINGS Relationships with researchers begin when the call for proposals is initiated. An iterative process to strengthen proposals and ensure alignment with the overarching objectives of the AMA IL then follows. The AMA IL Board of Directors considers all proposals. This occurs through the bringing together of high-level experts on the AMA IL subject matter, as well as a representative from USAID. Proposal assessment factors in and scores against the following elements: • • • • • • Technical merit of proposal (25 points); Broader applicability and synthesis (5 points); Collaboration and capacity building (20 points); Policy integration and outreach (20 points); Contribution to USAID objectives and initiatives (15 points); and Development impact and impact indicators (15 points). This selection process appears rigorous and allows the individual merits of each proposal to be considered. The process also considers the degree to which a proposal builds off earlier learning, and the degree to which the collective body of research “covers off” AMA IL’s different areas of interest. Up to this point, the applicants’ primary points of connection with AMA IL are the Director and Assistant Director. 48 Through this process, a relationship between the ME and the researcher and their host institution is created. An understanding of administrative and reporting processes is demanded. The AMA IL Account Manager will support setting up a research agreement with the researcher and host institution and will be responsible for financial management and auditing of acquittals. Once an agreement is signed, researchers assume the lead role in setting up the research program. This includes delineation of roles with, and between, partners. The ME monitors this process at a distance and through periodic reporting and financial acquittal. Once research has begun, researchers are required to submit a range of reports to the ME including annual work plans, quarterly and annual reports, trip reports, training reports, and workshop reports. They also need to develop and submit performance indicators. Reporting is reviewed by the ME in Davis, which ensures clarity and rigor by seeking clarification and new drafts of reports. The reporting process also involves a financial management element that includes narrative reporting crosschecked against financial acquittals. This reporting and documentation process intends to identify program synergies and linkages and allows for extraction of higher-level program findings. This process ensures regular communication between researchers and the ME, which in turn facilitates up-to-date understanding of issues, challenges, and opportunities that arise through the research. Management meets weekly to flag these issues, challenges, and opportunities for action, and to draw in the participation and contribution of the Outreach Specialist and Strategic Communications Manager. Another important program element is the annual Technical Committee meeting, which brings together PIs, a limited number of strategically identified guests from host countries, and key stakeholders from organizations best positioned to use the knowledge generated, notably USAID and the World Bank. Established two years ago, the Outreach Specialist and Strategic Communications Manager positions provide new and additional capacity to AMA IL. The ET observed that these appointments have been highly significant, ensuring more dynamic and opportunistic capacity to leverage and better use knowledge generated in the field. This is especially true in terms of getting knowledge on to the web though websites, webinars, policy briefs, etc., and in front of key people at events in Washington, DC and regionally. There is room for similar outreach and communications support to ensure strong KM. Currently, local￾level outreach and dissemination is the responsibility of the PI and his/her research team. CONCLUSIONS FOR EQ5 Generally strong systems exist for oversight of the research continuum from calls for proposals through to dissemination of results and advocacy for policy uptake within regional and global fora. Over the course of implementation, the ME has taken steps to strengthen its capacity for management and effective coordination and communication with all stakeholders, including research partners. The current team contains a diverse and complementary skill set that has helped ensure a finely tuned capacity for oversight of most aspects of the program’s ToC, including the capacity to engage and support research partners. The benefits of the new team composition can be seen in a more dynamic approach to outreach and dissemination evident over the past 18 months, through strengthened AMA IL profile in Washington, DC and participation in strategically identified events, such as GLEEs. Strategic alliances with other entities committed to development of a strong evidence base for change can also be observed, notably the strong partnership developed with ICED. These all help ensure AMA IL research is reaching key audiences beyond the country where research occurred. The major remaining management challenge for AMA IL is to ensure a system better equipped to manage the knowledge generated within individual research activities within the countries where they are carried out. The ET believes that this will be where policy traction will most easily be achieved, and 49 from which other opportunities can build. Genuine opportunity exists to affect policy across a number of research areas, yet these opportunities are at risk of not being realized due to insufficient policy positioning and stakeholder engagement. The ME has an important role to play in ensuring this weakness is addressed. RECOMMENDATIONS FOR EQ5 Strategic agreements need to be reached between researchers and management to ensure a thorough and multi-faceted approach to outreach and dissemination will occur that will more reliably contribute to program objectives and position research within different policy contexts. Clear articulation of roles and responsibilities within this approach is also required, especially given the ME’s strengthened capacity to support KM and promotion. This recommendation aligns with those, detailed above, that relate to outreach and dissemination and policy. Responding actions should consider all recommendations in an integrated manner. 4.6 FUTURE DIRECTIONS: WHAT SPECIFIC TECHNICAL AREAS AND RESEARCH TOPICS MERIT NEW OR CONTINUING RESEARCH INVESTMENT? Based on the findings of this evaluation, it is the conclusion of the ET that AMA IL occupies a very important position within the agricultural research landscape. Its focus on innovative solutions to the longstanding challenges of encouraging smallholder farmers to adopt proven technologies is critical. In particular, AMA IL has assumed an important knowledge leadership role related to two innovations with great potential to support the rural poor—index insurance and the application of satellite weather technologies to their specific context. These two technologies are often poorly understood, have little existing national capacity, and, currently, are barely reflected in the policy frameworks of most Feed the Future countries, or in the investment portfolio of donors who support rural economic growth through agriculture. This is, in itself, a key factor in the ET recommending a further iteration of the AMA IL. A diverse range of research and KM must occur so that the potential of these innovative technologies is better understood and that their inclusion in the policy framework of target countries can be facilitated. It is also critical to consider how index insurance and satellite technologies might interact with other longer-standing approaches like extension, micro-finance, ICT, and improved seed varieties. The importance of the role of private sector engagement in moving innovative research into programmed action needs to be fully recognized and supported through all AMA IL research. While this is occurring, a more proactive and deliberate process of scoping and engagement of the private sector will help further consolidate understanding of how best these new technologies can be marketed and made available to smallholders. To achieve this, AMA IL needs to ensure greater clarity and strategy within research protocols to encourage the active interest of senior policymakers and provide a sense of ownership in the countries where research is hosted. AMA IL also needs to partner more formally with the private sector as a key stakeholder in the research arena. By developing scalable innovations for governments and the private sector to adopt and implement, a tool kit of adaptable models can be assembled by AMA IL. Broader adoption can stem from these tools. The progress achieved through IBLI demonstrates what can happen when a logical flow of relevant, high￾quality research is advanced through strategic stakeholder engagement, and findings are presented in forms that allow host governments to observe, understand, and adopt—and also for neighboring countries to monitor, adapt, and adopt themselves. While resource-intensive, AMA IL should consider leading national innovation workshops that bring researchers and other Feed the Future activities together to consider their development landscape and how it relates to AMA IL’s areas of focus. This approach would allow for synergies and collaboration opportunities for Feed the Future development activities to be aired and leveraged. 50 Specific Technical Areas of Interest Through its breadth of index insurance-related research, AMA IL is making progress towards assembling a critical mass of knowledge related to different smallholder focused-insurance products. Using different entry points and bundling with related technologies helps evolve the capacity for index insurance to contribute to smallholder resilience and brings more textured understanding. With this comes relevance and greater ease in engaging those who shape policy. Research done during this current AMA IL iteration points to a need for additional research aimed at consolidating understanding of how best to strengthen index insurance to genuinely support the smallholder-farming context. The following are some examples: • • • • • • Work to date suggests significant potential for index insurance to be used as a tool to help leverage banks and encourage them to extend lines of credit, at reduced interest rates, to smallholder farmers. Further research should be done to develop and test lending models and the potential that index insurance has to encourage rural banks to lend to smallholders. Additional research is needed to investigate options for reducing insurance costs, possibly through coupling of insurance with other risk-management instruments or technologies. It could also look at options for underwriting insurance, including studying whether this would increase farmer investments. Another new area of research could explore how the optimal risk management portfolio changes over time as poor farmers accumulate resources and rely less on insurance and more on less expensive savings and contingent credit. In the longer term, this process could lead to needing insurance only for the most catastrophic losses that could not be mitigated any other way. Further exploration of minimum standards for index insurance is required. This includes assembling mechanisms, tools, and procedures to ensure that the insurance products marketed and sold to smallholder farmers meet minimum standards so that farmers are not worse off for having bought them. How, or if, insurance products can be applied to enable farmers to protect assets such as livestock when they are alive, instead of waiting for disbursements to replace them after they are lost, should also be researched. While many countries offer farmers a variety of subsidy and social protection transfer schemes, there remains a lot to learn regarding how to best design these schemes, so that they offer beneficiaries the strongest incentives for investment and income growth. More research is necessary to better understand the reluctance smallholders have to seeking credit. It is well-known that many smallholders choose not to use credit, even when it is readily available. In conjunction, there should be research that unpacks perceived and actual risks involved when micro￾lenders advance credit to smallholders. Agricultural extension services played an important role in most of the research initiatives undertaken during this iteration of AMA IL. There is evidence that when extension services deliver knowledge and information in a structured and pedagogically sound way, it positively impacts farmer decision-making and willingness to try new technologies. The community-based extension model tested within the Ghana DIRTS program offers an interesting line of study, which has potential to address many of the known weaknesses in current public sector-led extension delivery. New extension approaches have the potential to create better understanding and to address behavioral factors that appear to affect smallholder decision-making. Such research could also feasibly affect government willingness to more robustly finance extension services. There is also opportunity to further develop and evaluate new tools, especially ICT-based tools (notably smart phones), to overcome the costs of delivering reliable information to small-scale farmers who are dispersed over wide geographic areas. These tools include interactive e-extension and simulation games 51 that allow farmers to increase understanding of conservation practices or potentially beneficial but complex financial instruments like index insurance. There are also opportunities to better understand the way that information flows across existing social networks. Within Ghana DIRTS, higher-quality satellite weather data were shared far more widely than anticipated. This demonstrates a willingness to share important information across social networks. This work could help develop programs that facilitate the greatest information dispersion with more cost￾effective spillovers—an especially important issue where extension systems are not accessible. While the importance of gendered understanding of AMA IL’s various issues of focus is acknowledged, the degree to which individual research activities have increased the understanding of gender dimensions is patchy. Moving forward, a more deliberate focus and a more explicit expectation of significant level of effort should be put on understanding gender dimensions of research and should be built in to the program approach. 52 ANNEXES 53 ANNEX A: SUPPORTED AMA RESEARCH PROJECTS 2011-2017 54 The table below shows the research projects that have been awarded by the AMA Lab and the research themes that they address. Research Theme Addressed USAID Mission Request Project & U.S.- Based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance 1. BANGLADESH Building Resilience and Assets for Food Security in Bangladesh, Elisabeth Sadoulet, University of California, Berkeley 2. NEPAL Feasibility Study of Agricultural Insurance in Nepal, Michael Carter, University of California Davis 3. HAITI A Quasi￾Experimental “Post-Mortem” Study of a Discontinued Insurance Product in Haiti, Emily Breza, Columbia University 4. TANZANIA Developing a Satellite-Based Index to Predict Crop Yields in Smallholder Agriculture in Tanzania, Michael 55 Research Theme Addressed USAID Mission Request Project & U.S.- Based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance Carter, University of California Davis 5. INDIA Selling Formal Insurance to the Informally Insured in India, Mushfiq Mobarak, Yale University 6. DOMINICAN REPUBLIC USAID Climate Resilience and Index Insurance Program for Small Farmers in the Dominican Republic, Michael Carter, University of California, Davis 7. GHANA Disseminating Innovative Resources and Technologies to Smallholders in Ghana, Chris Udry, Yale University 8. GHANA Promoting Adoption of Improved Production Technologies Among Smallholders in Ghana via Coupled Credit 56 Research Theme Addressed USAID Mission Request Project & U.S.- Based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance and Insurance Contract, Mario Miranda, The Ohio State University 9. MOZAMBIQUE Health, Education, and Economic Interventions for Orphans and Vulnerable Children in Mozambique, Dean Yang, University of Michigan 10. UGANDA Building Market Linkages for Smallholder Farmers in Uganda, Craig McIntosh, University of California, San Diego 11. TANZANIA Communication, Search and Mobile Phones: A Telephone Directory Intervention in Tanzania, Brian Dillon, University of Washington 12. TANZANIA Evaluating the Socio-Economic 57 Research Theme Addressed USAID Mission Request Project & U.S.- Based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance Impacts of Western Seed’s Hybrid Maize Program in Kenya, developing a Satellite-based Index to Predict Crop Yields in Smallholder Agriculture in Tanzania, Michael Carter, University of California, Davis 13. TANZANIA A Multiple Interventions Approach to Increasing Technology Adoption with a View Towards Scaling-Up, Aprajit Mahajan, University of California, Berkeley 14. BURKINA FASO Demand and Supply Constraints to Improved Sorghum Technology Adoption and Their Gender￾Differentiated Effects in Burkina Faso, Andrew Dillon, Michigan State University 58 Research Theme Addressed USAID Mission Request Project & U.S.- Based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance 15. KENYA Tailoring Contract Farming to Smallholders in Kenya: Experimental Evidence on Enrollment Impact, Insurance Provision, and Communication Technologies, Lorenzo Casaburi, Stanford University 16. HAITI Household-Level Impacts of System of Rice Intensification (SRI) in Haiti: An SRI Intervention with Training, Insured Credit, and Coordination by Irrigation Bloc, Travis Lybbert, University of California, Davis 59 Research Theme Addressed USAID Mission Request Project & U.S.- Based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance 17. TANZANIA Evaluating the Effect of Site￾Specific Soil Information on Farmer Input Choices and the Relationship Between Poverty and Soil Quality in Tanzania, Cheryl Palm, Columbia University 18. MALAWI Smart Subsidies to Promote Peer Monitoring of Conservation Agriculture Compliance in Malawi, Andrew Bell, New York University 19. TANZANIA Rural Livelihoods and Institutional Reform in Small￾Scale Fisheries in Tanzania, Yaniv Stopnitzky, University of San Francisco 20. MOZAMBIQUE and 6. TANZANIA Achieving Development Impact with Complementary 60 Research Theme Addressed USAID Mission Request Project & U.S.- Based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance Stress-Resistant Seed & Financial Technologies in Mozambique & Tanzania, Michael Carter, University of California, Davis 61 ANNEX B: SUMMARY OF AMA-IL RESEARCH PROJECT OUTPUT 62 Summary of AMA-IL Research Project Output (as September 15, 2017) Project Title Published Academic Papers Working Papers Briefs/Policy Documents Presentations Stakeholder Meetings External Media Coverage Misc. Other Outputs Smart Subsidies to Promote Peer Monitoring of Conservation Agriculture Compliance in Malawi 6 1 1 1 3 1 1 A Quasi-Experimental “Postmortem” Study of a Discontinued Insurance Product in Haiti 0 2 1 1 4 0 0 USAID Climate Resilience and Index Insurance Program for Small Farmers in the Dominican Republic 1 0 2 1 1 0 0 Evaluating the Socio￾Economics Impacts of Western Seed’s Hybrid Maize Program 0 0 7 7 6 3 1 Feasibility Study on Agricultural Index Insurance in Nepal 0 1 4 0 4 0 2 Developing a Satellite￾Based Index to Predict Crop Yields in Smallholder Agriculture in Tanzania 1 3 2 2 1 0 0 Demand and Supply Constraints to Improved Sorghum Technology Adoption and Their 1 0 3 3 8 3 1 63 Project Title Published Academic Papers Working Papers Briefs/Policy Documents Presentations Stakeholder Meetings External Media Coverage Misc. Other Outputs Gender-Differentiated Effects in Burkina Faso Communication, Search, and Mobile Phones: A Telephone Directory Intervention in Tanzania 1 0 2 1 3 1 2 Disseminating Innovative Resources and Technologies to Smallholders in Ghana 0 0 12 0 10 4 1 Household-Level Impacts of Systems of Rice Intensification (SLR) in Haiti: An SRI Intervention with Training, Insured Credit, and Coordination by Irrigation Block 0 2 3 2 2 5 1 Evaluation of the Welfare Impacts of a Livestock Transfer Program in Nepal 0 3 5 4 3 4 1 A Multiple Interventions Approach to Increasing Technology Adoption with a View Towards Scaling-Up: Evidence from Mexico (MITA) 0 0 7 7 2 4 1 Building Market Linkages for Smallholder Farmers in Uganda 0 0 3 3 2 6 0 64 Project Title Published Academic Papers Working Papers Briefs/Policy Documents Presentations Stakeholder Meetings External Media Coverage Misc. Other Outputs Promoting Adoption of Improved Production Technologies Among Smallholders in Ghana via Coupled Credit and Insurance Contracts 0 4 4 4 6 4 1 Selling Formal Insurance to the Informally Insured in India 1 2 3 1 3 0 3 Index-Based Livestock Insurance in East Africa 17 12 28 42 4 33 2 Evaluating the Effects of Site-Specific Soil Information on Farmer Input Choices and the Relationship Between Poverty and Soil Quality in Tanzania 2 0 3 0 3 8 1 Building Resilience and Assets for Food Security in Bangladesh 0 0 3 1 4 0 0 Tailoring Contract Farming to Smallholders in Kenya: Experimental Evidence on Enrollment Impact, Insurance Provision, and Communication Technologies 0 3 3 2 5 0 2 Complementarities of Training, Technology, and Credit in Smallholder 3 3 3 4 6 7 2 65 Project Title Published Academic Papers Working Papers Briefs/Policy Documents Presentations Stakeholder Meetings External Media Coverage Misc. Other Outputs Agriculture: Impact, Sustainability, and Policy for Scaling-up in Senegal and Uganda Rural Livelihoods and Institutional Reform in Small-Scale Fisheries in Tanzania 0 0 4 1 1 0 0 Health, Education, and Economic Interventions for Orphans and Vulnerable Children in Mozambique 0 0 2 0 0 2 0 Bundling Innovative Risk Management Technologies to Improve Nutritional Outcomes of Vulnerable Agricultural Households 0 0 5 0 2 1 0 33 36 110 87 83 86 22 66 ANNEX C: UNPACKING THE RFA, UC DAVIS PROPOSAL, AND PROGRAM THEORY OF CHANGE 67 Through its problem analysis, the RFA provided a broad framework within which it expected AMA IL to operate, while also making it clear that “AMA IL was not expected to cover each topic comprehensively, or even to necessarily have a research activity in each of the areas mentioned.” Instead, the areas of inquiry outlined (in the RFA) were to be regarded as illustrative of USAID’s interests, setting parameters within which the Recipient (UC Davis) was expected to craft a coherent program of high-quality research activities.9 Those parameters included expectations that AMA IL would: • • • o o o • Mobilize U.S. university expertise in support of USAID strategic objectives around food security, agricultural development, and rural resiliency, through generation and dissemination of knowledge, and the promotion of recommendations on policies, programming, and practices that will improve rural households’ ability to acquire, protect, and effectively utilize productive assets. Generate innovations that support inclusive agriculture-led economic growth through enhanced access to markets, improved access to financial and risk management services, increased technology adoption and climate change adaptation, and increased resiliency of both men and women in vulnerable households and communities. Conduct rigorous policy and programming relevant research in defined areas of inquiry: Inclusive Market Access; Risk Management and Resilience; and Rural and Agricultural Finance. Ensure gender and impact evaluation as important cross-cutting themes needing to be recurrent throughout the portfolio regardless of specific substantive issues addressed. Significantly, USAID stated explicitly that while its program statement was being written broadly enough to allow some intellectual flexibility in the research program, it preferred AMA IL focus its research portfolio along two or three themes in order to build a critical mass of work in these areas. Within the three defined areas articulated in the RFA, the following guidance was provided regarding areas of interest to USAID that required further research: • • o Inclusive Market Access and Engagement o o o o o The role of social protection in enhancing the poor’s capacity to engage in economic opportunity; Methods by which agricultural-led economic growth programming can complement efforts to build the resiliency of the poor by enhancing the entry points for the rural poor, both men and women, in value chain activities; Ideas for promoting greater benefits for value chains traditionally done by women, and greater participation of women in higher-value value chains; Better understanding of constraints to improved outcomes in labor markets, especially issues of factor market integration or interaction and the most relevant policy and programmatic interventions in non-labor markets; and Consideration of the value and opportunity costs of unpaid family labor, especially that of women and children.10 Risk Management and Resilience Further development of insurance instruments that help farmers and other rural households 9 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 14. 10 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 10-11. 68 better manage risk and improve their productivity and incomes; o o o Exploration of the role of government in setting policy and providing safety net programs that do not distort markets or displace traditional coping mechanisms; Supporting appropriate public goods and institutions that enable private sector provision of risk management and mitigation mechanisms; and Expanding engagement on risk management by: ▪ ▪ ▪ Adding pilots that complement Feed the Future implementation programs in various areas (i.e., value chain programming); Facilitating Mission integration of insurance into Feed the Future and GCC programming through outreach and technical support; and Designing and testing the scale up of insurance products to broader areas (i.e., beyond the original areas for which micro data were available for product design), for example, by utilizing agro-ecological mapping and crop modeling.11 • Rural and Agricultural Finance o o o Exploration and development of innovations that promote productive linkages between rural financial sector deepening and development objectives is encouraged; Exploration of lending practices and procedures that would incentivize clients to make investments that build their resiliency to climate and other shocks; and Providing access to weather and climate data.12 In addition, two cross-cutting issues were identified as needing to be recurrent throughout the portfolio regardless of specific substantive issues addressed. These were elaborated as follows: • Gender Given the important role of women in agriculture throughout the developing world, the RFA emphasized better understanding of constraints faced by women, including their access to and utilization of productive assets, as vital. The RFA proceeded to make clear that it “places a high priority on research that explicitly examines gender issues relative to all the substantive issues” described in the RFA, making clear that even research activities that do not explicitly focus on gender issues should still use gender as one lens of analysis and build gender considerations into research methodology. Subsequently, the RFA went on to state that AMA IL should consider: o o o How gender relations will affect the achievement of sustainable results under the CRSP; and How proposed results will affect the relative status of men and women. Taking into account not only the different roles of men and women, but also the relationship and balance between them and the institutional structures that support them.13 • Impact Evaluation The RFA stated that there is substantial demand in USAID for empirical research that documents the impact of development interventions, especially where research can comparatively document various approaches or combinations of approaches currently or potentially used by Feed the Future development partners. The RFA went on to request that AMA IL contribute to the Agency’s objectives under Feed the Future by: 11 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 11-12. 12 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 12-13. 13 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 13. 69 o o o Contributing to the Agency’s (and development community’s body of evidence on effectiveness of various approaches to development challenges relative to food security, poverty reduction, and agricultural development, especially where this evidence is comparative; Developing and testing innovations, as input for USAID food security programming design and refinement; and Supporting efforts to refine USAID’s M&E methodology relative to food security. 14 In addition, the RFA stated that each research activity conduct capacity building, training, and institutional strengthening to enhance skills and expertise among host country scientists, universities, research institutions, and/or NGOs with which it works, by: • • • Designing research activities to maximize long-term degree training for host-country students, as well as shorter-term training for researchers and practitioners; Incorporating a variety of approaches for building capacity including degree programs, distance learning, web-boards, or other communications technologies and communities of practice that link researchers, policymakers, and development practitioners struggling with similar topics; and Ensuring balance in access to training and capacity building opportunities in terms of the sex of trainees and other participants. The RFA also stated as critical that AMA IL strive to achieve development impact15 from the adoption of policy and/or programming recommendations that the research activities generate. All activities were expected to benefit a wide audience of users, including developing country policymakers and technical specialists, development practitioners from NGOs, other donors and consultants, and USAID staff and projects. To ensure policy/programming relevance, adoption of recommendations, and subsequent development impact, the researchers were encouraged to engage with policymakers, USAID Mission staff, private sector representatives, and other stakeholders as appropriate even in early stages of the research design and implementation. UC Davis Proposal UC Davis’s response to the RFA is detailed in its successful technical proposal to serve as the ME. This is a key document in that it details with nuance what UC Davis saw as an appropriate and viable approach within the vast landscape laid out within the RFA. Appropriately, UC Davis emphasized that “priorities for a university-based research consortium must necessarily be dynamic and evolve over time,”16 while also acknowledging the importance of responsiveness to the input of USAID and other key stakeholders. To mediate this tension between policy responsiveness and intellectual creativity, UC Davis proposed mechanisms for both: • • Pilot project and impact evaluations that build on programs conceived and implemented by USAID Missions or other development agencies; and Basic research projects that may include impact evaluations as well as more basic or fundamental research that explores the nature of constraints and problems that require further understanding before their solutions can be intelligently designed.17 The proposal also emphasized the unique opportunity for learning within the mechanism of a university- 14 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 13-14. 15 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 14. 16 UC-D proposal to serve as management entity to the AMA IL CRSP, p. i. 17 UC Davis proposal to serve as management entity to the AMA IL CRSP, p. i. 70 based research consortium. “Unlike consultancy arrangements where the consultant provides answers for questions already on the agenda, university research at its best will look at the development problem in new ways and help reshape the agenda itself” while also acknowledging at the same time the responsibility of AMA IL “to speak effectively to the practical needs of development practitioners.” 18 The BASIS approach is predicated on two theoretically grounded understandings. The first is that poverty dynamics and chronic poverty are best studied through the analysis of assets (the resources that people have to produce a livelihood) rather than through the analysis of income or other livelihood outcomes. The second is that there may exist a critical minimum asset threshold, and that individuals whose assets fall below that level become mired in chronic poverty, unable to escape from that position over time.19 Four realms were proposed to help provide balance and ensure AMA IL achieve development impact: • • i • • Generation of New Development Knowledge needed to advance the Feed the Future agenda through rigorous, competitive selection of projects from the very best development scholars; Design and Rigorous Evaluation of Pilot Projects that explore the complementarities between agricultural and financial technologies in promoting the participation of women and poor farmers n the growth process; Effective Outreach and Communication about both project design and project results with USAID Missions and policymakers abroad, as well as with Washington-based staff of USAID and other donors and multilaterals; and Training of the Next Generation of host country scientists needed to sustain creative economic development policy. In response to the research areas stipulated in the RFA, UC Davis proposed the following: • • Instruments for Risk Management and Resilience o o Progress work commenced under the previous AMA CRSP, with the aim of consolidating a critical mass of learning on the use of innovative insurance contracts that transfer correlated risk out of small farming systems, including bringing attention to opportunities within recent technological advances in remote sensing and automated weather measurement. Research focused on: ▪ ▪ ▪ ▪ Intelligently designing index contracts (making them “demand worthy” that minimize basis risk, protect against price fluctuation, and crowd in agricultural credit supply); Testing the development impacts of index insurance, including exploring gender￾smart targeting of index contracts; Creating the science to cost-effectively scale-up index contracts without undercutting their insurance and development value; and Pricing insurance to incentivize climate change adaptation investment. Inclusive Market Access and Engagement o o o o Improving smallholder access to markets and technologies and aiming to overcome perceptions that smallholders are “too small” for engagement to be beneficial to markets. Targeting value chains at women and small-scale farmers. “Cargo net” programs that transfer assets and lift resource-poor households over critical thresholds to participate in agricultural growth. Considering the potential impacts of larger-scale farming on poor households via the Labor 18 UC Davis proposal to serve as management entity to the AMA IL CRSP, p. 2. 19 UC Davis proposal to serve as management entity to the AMA IL CRSP, p. 2. 71 Market. • Rural and Agricultural Finance o Prioritizing research and impact evaluations that explore small farm finance options: ▪ ▪ ▪ ▪ Self-finance and savings, including savings-secured lending; Self-collateralizing loans, including those for tree crops; Index insurance and joint liability credit; and Smart subsidies. Measuring and achieving development impact In response to the RFA expectation that AMA IL “achieve development impact,” UC Davis focused on the “gaping lack of knowledge about which development programs are effective” as stated by the Center for Global Development in 2006. AMA IL observes an “evidence gap” within Feed the Future programming, which reflects the complexity of obtaining credible impact estimates for complex, real world, and scalable programs. By engaging the highest quality researchers, AMA IL sees itself as able to help close this evidence gap and subsequently contribute to development impact. However, it is acknowledged that knowledge by itself is of little use, and that dissemination to the development community is critical. The approach proposed involves three intermediate steps aiming to link social science research to long-term development impact: • • • Obtaining research results; Distilling research results into recommendations for policy and programming; and Enhancing the probability that recommendations are adopted. Steps proposed to ensure these steps are followed included: • Ensuring funds for translation of documentation; • A dedicated Outreach person, to carry knowledge forward; • “Evidence” summits, webinars, and other strategies for dissemination; • Research grant solicitation clearly delineating researcher obligations to engage in these tasks; and • Strong and clear linkages to USAID in particular, as well as other key entities. Training and Capacity Building AMA IL stated an aim to facilitate a research and discovery process that contributed to the training and capacity of individuals who are committed to solving, and positioned to help solve, long-term development problems, with training of developing country students particularly encouraged. Training of women students and partnering with women PIs was stated as a priority. While it is hoped that participating institutions will facilitate study of host country students at the PI’s institution, it is understood that other complementary capacity building approaches are also needed. These “complementary approaches” were best seen as being undertaken by the researchers themselves. Furthermore, research grant solicitations stipulated that applicants for AMA research grants would be required to devise a training and capacity building program. Focus was also placed on development of a deep intellectual relationship with host country PIs, where they are empowered equals within the research process. Support for younger, junior researchers at the host country level was also encouraged. This collective approach was described as an enabling environment for “institutional capacity building.” By establishing sub-contract relationships with their partner institutions, AMA grants will contribute to the institutional competence and experience and help empower these institutions to manage money and contracts in the future. Gender 72 The emphasis placed in the RFA on gender aware research was not reflected in the UC Davis proposal to serve as ME. While gender and women were identified in some of the proposed research areas, gender was not mainstreamed within then overall approach. Nor was a need for gendered analysis of findings mentioned in any form within the research grant solicitation template. AMA IL has been supported with three AAs: 1. AA #1: “Advancing Index Insurance by Closing the Gap Between Knowledge and Implementation” (2014-2019), designed to support the responsible scaling of index insurance for agricultural development, food security, and resilience applications in developing country contexts. 2. AA #2: “Achieving Development Impact with Complementary Stress-Resistant Seed & Financial Technologies” (2015-2019), designed to support the development of DT maize varieties in collaboration with CIMMYT, while also ascertaining if their rate of adoption and impacts can be enhanced by bundling them with a financial technology (index insurance) designed to pay out in the face of severe drought events under which even DT seeds fail. 3. AA #3: “Feed the Future Evaluating the Effectiveness of Programs that Enhance the Economic Resilience of Vulnerable Populations” (2016-2020), allowing expanded understanding of the nature of resilience, through focus on strategies to address vulnerability and unprotected risk. AMA IL Theory of Change Since its initiation in 1996, the AMA IL has been deeply investigating multiple mechanisms that encourage resilience and self-sufficiency through broad-based agricultural-led growth. Over that time, the Lab has evolved beyond a theoretical understanding of risk to a point where it is now constructing and trialing interventions that test the theories developed earlier. To help guide the overall program, a ToC has been developed that delineates the logic of how the Lab is going to better understand and respond to the root causes of persistent poverty and food insecurity (see Figure 1). 73 Figure 1: AMA IL Theory of Change The premise of the AMA IL ToC is that unprotected risk can lead farmers to pursue strategies that leave them entrenched in poverty. Shocks have the capacity to reduce farmers’ asset base, impact family food security, and contribute to ongoing intergenerational poverty. Smallholder farmers, who are risk averse, are often reluctant to adopt potentially profitable technologies. Reducing financial constraints and improving insurance models will give farmers and pastoralists’ confidence and make it easier for them to take advantage of available technological improvements with the potential for higher yields. Strengthening markets and increasing market access will open opportunities for small scale producers— especially women—to increase their profits from production to end-market retail. Addressing this complex scenario holistically requires input from a cross-section of stakeholders. Maintaining gains in the long-term will require integrated, flexible systems that cultivate resilience and self-reliance. Undertaking high-quality research on well-established, risk-reducing instruments will create an evidence base and help develop mechanisms, which have the potential to improve risk management in agriculture by helping farmers protect their assets and make prudent investments toward a future free of the need for emergency aid. The work of the AMA IL aims to progress the ToC by both assembling and disseminating important knowledge generated at the field level. In theory, the knowledge assembly is dynamic and progresses on a continuum—it positively impacts local communities and continues upwards to national policymakers and out to other actors active and interested in the issues AMA IL focuses on. As stipulated in the RFA, 74 outlined in UC Davis’ proposal, and detailed as an outcome within the ToC, this knowledge requires management aimed at promoting adoption of recommendations emerging from the research, with a view to affecting policy, while also feeding into USAID’s future programming decisions. Knowledge should also be strategically assembled to contribute to the thinking of other knowledge generating and disseminating communities of practice.20 Given the above ambition, stakeholder mapping and engagement, outreach, and the strategic dissemination and communication of results to different audiences is of critical important to AMA IL work. A wide range of stakeholders, each with different and nuanced information needs, are potential beneficiaries of the program, and all have the potential to contribute to the program’s success. When strategically engaged by both the program and its researchers, different stakeholders become more aware and enjoy a more sophisticated understanding of the potential of innovative solutions to complex, long-standing problems. This leaves them better placed to support, and be supported, to take holistic action and complementary actions—and to develop a sense of ownership of what it is AMA IL is aiming to achieve. If implemented effectively, AMA IL will improve the understanding of issues that restrict smallholder farmers from progressing out of poverty. It will provide innovative approaches, and a multi-faceted, coherent evidence base from which policymakers can be strategically engaged. It will also foster collaborative relationships and networks capable of advancing knowledge into practice and policy. 20 RFA, Assets and Market Access Collaborative Research Support Program (RFA-OAA-12-000001), issued October 11, 2011, p. 14. 75 ANNEX D: COMPARATIVE ANALYSIS OF INDEX INSURANCE PRODUCTS FOR AGRICULTURAL PRODUCTION 76 In response to the AMA IL Terms of Reference, UC Davis adapted and tested several index insurance products, as potential means for increasing agricultural production, incomes, and resiliency at the smallholder farmer level. These included index insurance products designed to: ● ● Improve Risk Management and Resilience—focusing on the issues of risk management within low income agricultural economies and communities. Increase Access to Rural and Agricultural Finance—improving access to financial services for smallholder farmers, and better understanding and addressing risks perceived when providing capital access to them. The three index insurance products tested under the AMA IL projects evaluated in Ghana and Tanzania included the following: ● ● ● Crop index insurance offered to small farmers in northern Ghana to reduce their risk of loss in the case of unforeseen drought incidence and encourage their acquisition and use of project introduced improved maize seeds. Insurance was offered in collaboration with Ghana’s national agricultural insurance provider (GAIP). Bundling of an improved DT seed product with an index insurance product for smallholders in Tanzania, with the insurance premium rolled into the cost of the seed, offered in collaboration with private insurers and seed producers. Access for smallholders in northern Ghana to index insurance contingent agricultural production credit from local banks, for financing their seasonal production costs. While each insurance product was designed to reduce risk and ultimately increase production and resilience at the small farmer level, each was employed using a different approach to risk reduction, with different incentives offered to stakeholders in the insurance transaction and different delivery mechanisms for insurance coverage and inputs delivery. In the following sections, each of the three index insurance products is briefly examined, with observations based on the research framework specified by USAID for these AMA IL evaluations. Ghana DIRTS Project Crop Index Insurance Ghana’s national agriculture insurance provider (GAIP) was a key IP for both the Ghana DIRTS program, and the Credit and Insurance program working with rural banks in Ghana. GAIP is, as noted above, governed by 17 Insurance companies, and is endorsed by the National Insurance Scheme. While all agricultural stakeholders for the DIRTS project interviewed by the ET were clear that Ghana needed an agricultural insurance scheme capable of providing products to smallholders, the evaluation findings of the ET signaled that GAIP may not have the capacity to effectively carry the crop index insurance product forward, at both the marketing level and the level of efficiently servicing payout claims. While GAIP partnered with the DIRTS project to develop and market the index insurance product to small farmers in Northern Ghana, its ability to market and service the product has been undermined by several inherent weaknesses at GAIP, which acts monopolistically in the absence of private agricultural insurance competitors. During the initial marketing of index insurance to smallholders under the DIRTS project, GAIP was charged with taking the lead with participating farmers. For reasons including limited staff and outreach in the region, GAIP was unable to reach the marketing scale projected by the project for the product marketing to small farmers. As a result, the DIRTS project staff wound up doing much of the product marketing with small farmers in the target region. ET field interviews with GAIP staff indicated that it also has a limited business model, which in turn has limited outreach and promotion of insurance products for crop as well as loan insurance in Ghana. GAIP also faces problems of survival due to inadequate funding as well as staffing. Among the other lessons 77 learned under the roll out of this crop index insurance product was the critical importance of IP selection, orientation, and capacity building. Interviews with the Ministry of Agriculture signaled that there are limited levels of active support for GAIP and the index insurance product within the Ghanaian government. This is a policy-level issue which must be addressed early in project design and implementation. ET field findings also indicated that the failure of the DIRTS project to directly link farmer adoption of the project introduced improved seed technology with the farmer’s purchase of insurance could have been a factor in lower take-up rates at small farmer level, for the index insurance product—a research project design-level issue. Under the AMA IL DT Seeds and Index Insurance Project in Tanzania, the insurance take-up rates among participating small farmers appeared to be higher. This could have been in part attributed to the bundling of the insurance product premium with the cost to farmer of improved seeds introduced. Tanzania Drought Tolerant Seeds and Index Insurance Project. Project-introduced trials through bundling index insurance into the sale of DT seeds in Tanzania appear to have been a simple, yet well-received innovation among small farmers. Given this particular research project still has two years to run, it should be allowed to run its course. However, early signs are that a market exists for insured seed, and that it is effective in giving farmers confidence to pay more for new seed varieties that offer a degree of protection against climate risk. This research project has also been supporting three medium-sized national seed companies to enhance local distribution of improved seeds, and has effectively partnered with a private insurance company, UAP Insurance. This has allowed UAP to successfully integrate smallholder-focused insurance products into its business model, and market and provide them to smallholder farmers. The more private sector approach taken in design and implementation of this index insurance product in Tanzania appears to have made significant accomplishments at the level of local institutional capacity building. As a result of participating in project-related seed development and production, a local seed company (Lumbargo Seeds) has developed and is marketing to farmers improved and locally-tested seed varieties. UAP, which is a branch of a private insurance provider serving much of east and southern Africa, is now in the process of developing and seeking to market its own crop index insurance product, targeted at smallholder farmers in Tanzania with the objective of serving them as part of producer groups or associations. Ghana “Combining Credit with Index Insurance” Research Project Under this project, index insurance was introduced as part of an insurance contingent loan product available to smallholder farmers through local banks, to be used by farmers for purchase of improved inputs for crop production. This is a different insurance product from crop index insurance, in that product design is focused upon a different combination of stakeholders and incentives to participate, as well as different risk sharing mechanisms. Previous research on risk management interventions among smallholder farmers has signaled that even with improved access to technology-based risk mitigation options, the reality of many poor farmers globally is that they are extraordinarily cash poor, making investment in new technologies problematic even when the will is there. The Ghana “Combining Credit with Index Insurance” project tackled this phenomenon by looking at ways of opening up more affordable lines of credit for smallholders through local banks, using insurance contingent loans to small farmers to mitigate non-payment risks for both the participating bank and borrowing farmer. This index insurance contingent loan product was offered based upon two options for indemnity payment in the case of weather-related unforeseen crop losses—one where the indemnity was payable 78 to the farmer/borrower and one where it was payable to the lender. While the “indemnity payable to bank” option was more successful in inducing bank participation in the program, with the introduction of this insurance contingent loan product, farmers were 53 percent more likely to apply for loans, and banks increased their credit approval rate by as much as 24 percent. Importantly, credit agreements were dependent upon borrowers working with extension agents to develop a budget as the basis for the loan, with credit being provided in the form of a voucher to purchase agreed inputs from a local input provider. Findings of the ET signaled that product design and implementation for the insurance contingent loan product helped to build the capacity of number of rural banks to appraise, extend, and better manage risks inherent in lending to smallholder farmers, with two participating banks interviewed considering further adaptation and extension of the product to small farmers. Project participation also increased the capacity of banks to work with local input providers to mitigate credit use and repayment risk. While only a trial at this point in time, this product deserves further investigation and expansion as a way of both opening up lines of more affordable credit and also helping develop better smallholder index insurance products. As with the crop index insurance product, more work remains to be done in Ghana at the policy level in securing endorsement of index insured loans as a policy for extending more production credit to small farmers. 79 ANNEX E: LUBANGO DROUGHT TOLERANT MAIZE SEED VALUE CHAIN 80 The Producer Mr. Doe was the son of a seed retailer. Mr. Doe started his own business by producing Tomatoes in 2008 then shifted to DT Maize in 2009. At present, he is producing DT seeds from 500 acres and the plans for expansion to 800 acres are underway. His company is one of 10 owned by Tanzanian nationals (Local). Most of the seeds that reaches farmers throughout Tanzania is from three (Giant) companies (Monsanto, SIDCO). Lubango, a high performing drought tolerant maize (DTM) variety, is the trade mark seed produced by the Company. It means “blessed” in local Sukuma language. Lubango was first produced by IFFA Seed in 2015 and is already replacing traditional seeds on farms across Tanzania as a result of the company’s hands-on, targeted marketing approach. A smart locally-adapted marketing initiative is followed by this innovative company that combines demonstration/training with advocacy. At present, there are demonstration plots covering 10 out of Tanzania’s 12 districts. Promotion is handled by a dedicated promotion manager (CiMMYT.org, Features May 2016) who underscores the fact that, “Lubango was created with the smallholder farmer in mind as a drought tolerant, affordable, high-yielding, and great tasting.” The training and capacity aspects of promotion include ensuring farmers are equipped with smart agronomic practices (spacing, weeding, etc.). The company uses also audio-visual tools to promote the seed among the farmers’ communities, such as a very popular film that provides farmers with full details of the life cycle of the seed from planting to harvest. For the seed to be distributed to dealers and retailers, it must be tested and certified by Tanzania Official Seed Certification Institute (TOSCI). The seeds are then packed into 2-kilogram sealed bags with the certification sticker prominently shown in each bag. The challenge facing IFFA is the fact that most farmers are not willing to buy insured seeds. According to Mr. Doe there are enough seeds, but the demand is low. IFFA provides training to the village seed dealers directly or through the extension agents, who, in turn, train the farmers. 81 Lubango seeds are sold in competitive prices. They are sold to the retailers either without Index Insurance at 8,000 Tsh or with Index Insurance at 8,500 Tshs, prices significantly lower that seeds sold by the large producers (12,500 to 13,000 Tsh/2kg). Training is provided to Village Retailers and Farmers through CEAs (themselves trained by District Extension Departments). The ET visited a town (normal) retailer who, being a retired extension agent, also provides extension messages (about improved seeds, index insurance, and inputs) to the village dealers. The retailer sells several seed varieties (Lubango, DK 8,030 and DK 9,089, etc.). 82 ANNEX F: AMA IL EVALUATION EXPRESSION OF INTEREST 83 PEEL TASK ORDER EXPRESSION OF INTEREST – PERFORMANCE EVALUATION I. BACKGROUND INFORMATION A) Identifying Information 1. Project/Activity Title: Feed the Future Innovation Lab for Collaborative Research on Assets and Market Access (AMA Innovation Lab (IL or Lab) 2. Award Number: AID-OAA-L-12-00001 3. Award Dates: 04/20/2012 – 09/30/17 4. Project/Activity Funding: $25,000,000 5. Implementing Organization(s): University of California Davis 6: Project/Activity AOR: Kelley Cormier B) Development Context 1. Problem or Opportunity Addressed by the Project/Activity Being Evaluated While many organizations in the international agricultural development sector focus on technology and production, (e.g., improved seeds, fertilizer, etc.), there has been less of an emphasis on social scientific research that aims to better understand the constraints to the adoption of improved agricultural technologies that lead to agriculture-led growth. Per the original RFP, USAID was “particularly interested in better understanding the root causes of persistent poverty and food insecurity, and how to effectively increase the capacity of poorer households to engage in and benefit from agriculture-led growth.” A deeper understanding of the root constraints that inhibit greater performance were a focus of this project. The project re-assessed and further dissected the capacity and capability constraints, socio￾cultural factors and behaviors, and incentive structures that either limit or maximize smallholder farmer potential and choice. The AMA Innovation Lab (Innovation Lab or “Lab”) assembled a portfolio of projects that focused on key topics designed to bridge the gap between what is possible given currently available technologies and the realistic experiences of most developing country agriculturalists. 2. Target Areas and Groups The target populations are agricultural sector actors who are not fully taking advantage of agricultural and financial technologies to reduce the yield gap and maximize their income potential. Projects were realized in the following countries: Bangladesh, Burkina Faso, Dominican Republic, Ghana, Haiti, India, Kenya, Malawi, Mexico, Mozambique, Nepal, Senegal, Tanzania, and Uganda. C) Intended Results of the Project/Activity Being Evaluated 84 The AMA IL was designed to conduct applied research on the elements that shape rural households’ resiliency, food security, and participation in agriculture-led economic growth. The research was expected to inform, develop, and test innovative interventions that would enhance rural households’ ability to acquire, protect, and effectively utilize productive assets, as well as their ability to effectively engage in markets. The AMA IL was created to, “… mobilize U.S. university expertise to support USAID in achieving its goals and strategic objectives around food security, agricultural development and rural resiliency.” It aimed to do this through “the generation and dissemination of knowledge, and the promotion of recommendations on policies, programming and practices that will improve rural households’ ability to acquire, protect, and effectively utilize productive assets” as stated in the RFP’s scope of work. The AMA IL aimed to achieve impact through various social scientific research projects to better understand the opportunities and limitations of current agriculture development programming and create new theories of change for improved productivity and uptake of technology. The intended ways to advance these findings were to provide reflections in the areas of policy integration and outreach, through collaboration and capacity building, and by providing support to Mission objectives and initiatives through rigorous impact evaluations of development programs and policy interventions). Furthermore, cross-cutting themes such as gender and impact evaluation rigor were expected contributions from the grantee to accomplish the following: a. Contributing to the Agency’s (and development community’s) body of evidence on effectiveness of various approaches to development challenges relative to food security, poverty reduction, and agricultural development, especially where this evidence is comparative; b. Developing and testing innovations, as input for USAID food security programming design and refinement; and c. Supporting efforts to refine USAID’s M&E methodology relative to food security. In addition to the requirement for high-quality research activities the grantee was expected to: • • • • Elucidate research findings and discoveries that may lead to more success development impacts; Disseminate activities that highlight potential development impacts through policy or programming recommendations from the research activities generated; Promote the adoption of recommendations and engagement with policymakers; and Conduct capacity building training and institutional strengthening. D) Approach and Implementation The Innovation Lab for Assets and Market Access has been supported with three primary awards: Main Leader Core Award, “Program in Assets and Market Access” (2012-2017): 1. Associate Award #1: “Advancing Index Insurance by Closing the Gap Between Knowledge and Implementation” (2014-2019); 2. Associate Award #2: “Achieving Development Impact with Complementary Stress-Resistant Seed & Financial Technologies” (2015-2019); and 3. Associate Award #3: “FTF Evaluating the Effectiveness of Programs that Enhance the Economic Resilience of Vulnerable Populations” (2016-2020) In response to the RFA for the core award, the UC Davis team specifically proposed to develop research projects in three areas. The three areas and a brief rationale for each are: 1. Instruments for Risk Management and Resilience: Risk is economically costly in low-income agricultural economies, prompting self-insurance strategies that keeps small farmers poor as 85 they eschew remunerative, but risk opportunities. Making matters worse, self-insurance only partially protects small farm households against the damaging drops in consumption that can irreversibly damage the long-term physical and cognitive development of young children. Recent technological advances in remote sensing and automated weather measurement open the door to risk management products that can transfer the correlated or covariant risk out of the small farm systems. Ongoing projects have offered an impressive array of innovations and insights around risk management mechanisms, but there is still much to learn about index insurance and its potential welfare impacts. 2. Removing Barriers to Broadly-Based Growth: Even when farmers are able to bear the risks associated with new, higher-returning agricultural opportunities, a series of barriers often block the participation of small farmers. Previous research indicates that low incomes or low assets may put small farmers in a position where they simply cannot successfully engage with programs designed to enhance productivity. 3. Rural & Agricultural Finance: Providing an avenue of upward mobility to small-scale farmers may not only require risk mitigation and connections to markets, it may also require access to financial resources that allows farmers to make the investments necessary to experience a discrete jump in farm productivity and income. There is a range of alternative mechanisms that hold the promise of providing capital access to small farmers, especially when the risks for lenders and borrowers are reduced when credit is interlinked with a risk management mechanism. This is an area wide open to systematic experimentation and testing to see what combinations of insurance, subsidy and finance are most effective in accelerating agricultural investment and growth. Through sub-awards to research partners, the Main Leader Core Award built a global network to undertake research related to the various research themes identified through sub-grant research collaborations. Annex 1 describes projects supported under each theme, including project partner details. Together, this suite of projects forms the main body of AMA IL research work. In addition, the Innovation Lab received the following Associate Awards which provided additional funding intended to allow the lab to further contribute to its core mission: Associate Award #1 on “Advancing Index Insurance by Closing the Gap Between Knowledge and implementation” was designed to “… to support the responsible scaling of index insurance for agricultural development, food security and resilience applications in developing country contexts.” The approach was for the recipient to support a number of activities aimed at improving communication, coordination, and collaboration across the many ongoing efforts to develop appropriate products and expand the penetration of insurance markets in these countries. The RFP called for the Recipient to support the development of a community of practice, a Global Action Network to Advance Index Insurance (GAN).” In response to this RFA, the AMA Lab proposed to sub-contract with the micro insurance Innovation Facility (housed at the Social Finance Program of the International Labor Organization in Geneva) to serve as the Secretariat for the Global Action Network (GAN) to promote index insurance. It was proposed that the Secretariat focus on three main activities: 1. Establish and coordinate a community of experts that will serve as an action network that discusses key issues around agricultural insurance; 2. Build capacity of practitioners and governments in three focus countries; coordinate country strategies and work plans; explore collaborations with existing initiatives; develop and conduct training for practitioners; and 86 3. Promote responsible scaling of agricultural insurance to the broader insurance community by repackaging and disseminating lessons into knowledge products, tools, and training modules. In addition, the AMA Lab proposed to carry out at least one major research project to study the efficacy of a newly launched index insurance program launched with the guidance and support of the GAN which resulted in the third AA below. Associate Award #2 on “Achieving Development Impact with Complementary Stress-Resistant Seed & Financial Technologies” was designed to support the development of drought tolerant maize varieties under the Drought Tolerant Maize for Africa (DTMA) project. As USAID moved to encourage the International Maize and Wheat Improvement Center (CIMMYT) to promote the roll out and adoption of the new DT varieties across a variety of African countries, it also approached the AMA Lab to implement a large-scale randomized controlled trial (RCT) to evaluate the impact of DT seeds. The AMA proposal, developed in collaboration with CIMMYT built on earlier Lab work and proposed to not only investigate the adoption impact of the seeds alone, but also to see if their rate of adoption and impacts can be enhanced by bundling them with a financial technology (index insurance) designed to pay off in the face of severe drought events under which even DT seeds fail. Utilizing research methods developed in an earlier Lab project, it was decided to implement the RCT in Tanzania and Mozambique. Associate Award #3, “FTF Evaluating the Effectiveness of Programs That Enhance the Economic Resilience of Vulnerable Populations” was recently awarded in 2016 and will run to 2020. This AA will focus on studying strategies to address vulnerability and unprotected risk. As USAID increases efforts to better understand mechanisms to enhance resilience and enable growth, several USAID Missions have approached the AMA Lab requesting assistance in generating evidence regarding impacts of USAID-supported interventions. In response, the AMA Lab proposed to undertake rigorous evaluation of three Mission-supported resilience interventions: 1) Comprehensive Services for Orphans and Vulnerable Children in Mozambique; 2) Risk Reduction for Vulnerable Dairy Farmers in the Dominican Republic; and 3) Insuring Rice Farmers in Nepal. While the theme of this new AA #3 complements and builds on previous work, the nascent stage of this award makes it not viable to include fully in this evaluation although comments as to the follow up and inclusion in the portfolio are surely welcome. Lastly, the AMA Lab aimed to use a variety of mechanisms to ensure that new knowledge was communicated to national and international development policy and programming communities. This included the hiring of dedicated strategic communications and outreach staff, the development of events on amassed research topics and a variety of written and other outputs to communicate what the research has found to both technical and non-technical audiences. In responding to RFA for the Core Award, the UC Davis aimed to address “at least three intermediate steps to link social science research to long-term development impact: 1) obtaining research results, 2) distilling research results into recommendations for policy and programming, and 3) enhancing the probability that recommendations are adopted.” II. EVALUATION RATIONALE A. Evaluation Purpose The purpose of this external performance evaluation is to provide empirical evidence as to the effectiveness of this project overall and to identify how it supported continuous improvement for USAID’s work in better understanding the successes and failures of various tested and implemented development approaches designed to improve farmer livelihoods and productivity, and spur agriculture growth. This evaluation will also provide information and recommendations for a future research agenda and suggestions for future program directions for BFS and Missions to inform country level and institutional level project effectiveness. 87 Specifically, this evaluation seeks to determine: 1. The overall quality of the research program, including its success in meeting outreach and capacity building goals, and the opportunity for research outputs to enhance rural households’ ability to acquire, protect, and effectively utilize productive assets; 2. The effectiveness of the management model employed by the Innovation Lab in designing and implementing its research program; and 3. Priorities and suggestions for future research that might be undertaken by the Innovation Lab. B. Audience and Intended Uses The audiences for this evaluation are USAID Missions, the Bureau of Food Security, implementing partners, and other stakeholders, such as government partners where USAID works, relevant private sector partners, academia, and other donors and development stakeholders. C. Evaluation Criteria and Questions Research Program Section D above lists the three general themes in which the Innovation Lab has developed a research program. Appendix A categorizes each of the individual projects funded by the Lab in terms of their relationship to each theme. The evaluation should cover the work of the Lab under each theme using the following criteria and indicative questions and generate robust evidence-based findings, conclusions, and actionable recommendations. 1. Research Quality 1.1. Have the research projects developed scientifically valid and robust conclusions and professional-level outputs? In what ways did the portfolio of projects make progress and contribute toward the stated research objectives in the RFA? How did it diverge from objectives in RFA? 1.2. How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement and made progress toward technical and policy impacts? 1.3. To what extent has work from previous iterations of the lab been effectively incorporated and built upon? 2. Outreach & Dissemination 2.1. To what extent has the AMA IL and its research projects effectively analyzed, synthesized, and distilled research results into actionable information or recommendations? In what ways could research results be better communicated with different stakeholders, including USAID Missions, implementing partners, public and private sector partners, including technical and non-technical stakeholders? 2.2. In what ways have different partners (especially USAID Missions and the private sector) been engaged in the research process? What opportunities are there to increase this engagement? 3. Policy 3.1. In what ways was policy research demand-driven—addressing current concerns, objectives, and needs of private and public sector stakeholders, USAID Missions, and BFS? 3.2. Which, if any, new models or innovations are most likely actionable and policy relevant? What characteristics of these new models or innovations make them more policy relevant and actionable? Is there any evidence of policy change as a result of research findings? In what ways, if any, did the project monitor this uptake? 88 4. Capacity Building 4.1. How well have the projects and consortium of researchers identified and addressed academic and technical capacity needs of host country stakeholders? To what extent has this contributed to capacity building in the host country? 4.2. In what ways did the consortium of researchers and projects support the participation of the private sector? What opportunities are there to better support the inclusion of the private sector in market viable development solutions? 5. Program Management 5.1. How did the management entity effectively communicate and coordinate with research partners to achieve the objectives of the RFA? 6. Future Directions 6.1. What specific technical areas and research topics, if any, merit new or continuing research investment, particularly with regard to the new Global Food Security Strategy? III. TIMEFRAME & TRAVEL A. Timeframe The evaluation will occur over approximately 6 months. (From approximately Jan. - June 2017.) B. Travel Travel to the field will be undertaken in 1-2 countries where activities are implemented. One to two domestic trips will be taken to meet with consortium/university partners based within the U.S. IV. DELIVERABLES & DESIGN A) Deliverables All Deliverables as listed in section F. 7 of the PEEL-TO contract. 1. Concept Note 2. Evaluation Plan – (at least 2 revisions, pending USAID approval), prior to data collection field visits 3. Mission Out-briefs – short presentation of country-wide findings for USAID Missions for each country where site visits were conducted following both phases of data collection 4. Presentation of findings – following the data collection phase, short presentation of initial findings/results for USAID and IP (utilizing web-based interface) 5. Evaluation report (at least 2 revisions, pending USAID approval), draft due date to be determined within work plan approval. FINAL DRAFT due three weeks after comment period. Report outline should include evidence-based findings and short-term and longer term actionable recommendations relating to the evaluation questions and activity implementation. The final evaluation report should sufficiently address all comments and corrections provided to the draft report and be 508 compliant and uploaded to the DEC. 6. A USAID template will be provided. The evaluation team may include other topics that are deemed relevant and are evidence-based. V. TEAM COMPOSITION The technical qualifications of evaluation team members must be matched with the technical areas of focus of the Assets & Market Access Innovation Lab. Team members must have the expertise necessary to evaluate the Assets & Market Access Innovation Lab and to fully address the evaluation questions. 89 The team will also have: a) a demonstrated capacity to conduct independent program evaluation; b) an understanding of USAID’s foreign assistance goals and its particular objectives related to collaborative research, agricultural development, and food security; and c) the ability to analyze issues and formulate concrete recommendations orally and in writing. Experience in developing countries and work with smallholder farmers is required. The Technical Team leader and 1-2 team members must be experts in a field related to agricultural economics in international development with specific expertise in applied economics research, including strong quantitative and qualitative skills. Preference will be given to candidates that have experience in risk and resilience, behavioral economics, poverty dynamics, socio-cultural behavior change, international agriculture development, development finance, marketing, and other related areas that match the focus of the project. The evaluation team must include at least one member serving as the evaluation expert with a strong background in evaluation methodology, conducting mixed methods evaluations, and a minimum of 15 years managing and/or evaluating multifaceted international development research and/or university￾based programs with demonstrated experience and knowledge of qualitative research methods. VI. SUGGESTED LOE Task/Deliverable Team Lead (days) Team Member Team Member Conference Call/Desk Review 6 6 6 Evaluation Plan & revisions 8 6 6 Data collection, including travel 22 20 20 Presentation of findings 3 3 3 Draft Report 10 8 8 Integrate comments and Revisions of Final Report 6 4 4 Total 55 47 47 90 Annex 1 - AMA Research Projects The table below shows the research projects that have been awarded by the AMA Lab and the research themes that they address: Research Theme Addressed USAID Mission Request Project & U.S.- based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance Building Resilience and Assets for Food Security in Bangladesh, Elisabeth Sadoulet, University of California, Berkeley Feasibility Study of Agricultural Insurance in Nepal, Michael Carter, University of California Davis A Quasi￾Experimental “Post-Mortem” Study of a Discontinued Insurance Product in Haiti, Emily Breza, Columbia University Developing a Satellite-based Index to Predict Crop Yields in Smallholder Agriculture in Tanzania, Michael Carter, University of California Davis Selling Formal Insurance to the Informally Insured in India, 91 Research Theme Addressed USAID Mission Request Project & U.S.- based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance Mushfiq Mobarak, Yale University USAID Climate Resilience and Index Insurance Program for Small Farmers in the Dominican Republic, Michael Carter, University of California, Davis Disseminating Innovative Resources and Technologies to Smallholders in Ghana, Chris Udry, Yale University Promoting Adoption of Improved Production Technologies Among Smallholders in Ghana via Coupled Credit and Insurance Contract, Mario Miranda, The Ohio State University Health, Education, and Economic Interventions for Orphans and Vulnerable Children in Mozambique, Dean Yang, University of Michigan 92 Research Theme Addressed USAID Mission Request Project & U.S.- based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance Building Market Linkages for Smallholder Farmers in Uganda, Craig McIntosh, University of California, San Diego Communication, Search and Mobile Phones: A Telephone Directory Intervention in Tanzania, Brian Dillon, University of Washington Evaluating the Socio-Economic Impacts of Western Seed’s Hybrid Maize Program in Kenya, Developing a Satellite-based Index to Predict Crop Yields in Smallholder Agriculture in Tanzania, Michael Carter, University of California, Davis A Multiple Interventions Approach to Increasing Technology Adoption with a View Towards Scaling-Up, Aprajit Mahajan, University of 93 Research Theme Addressed USAID Mission Request Project & U.S.- based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance California, Berkeley Demand and Supply Constraints to Improved Sorghum Technology Adoption and their Gender￾Differentiated Effects in Burkina Faso, Andrew Dillon, Michigan State University Tailoring Contract Farming to Smallholders in Kenya: Experimental Evidence on Enrollment Impact, Insurance Provision, and Communication Technologies, Lorenzo Casaburi, Stanford University Household-Level Impacts of System of Rice Intensification (SRI) in Haiti: An SRI intervention with training, insured credit, and coordination by irrigation bloc, Travis Lybbert, University of California, Davis 94 Research Theme Addressed USAID Mission Request Project & U.S.- based PI Risk & Resilience Broadly-Based Growth Rural & Ag Finance Evaluating the Effect of Site￾Specific Soil Information on Farmer Input Choices and the Relationship Between Poverty and Soil Quality in Tanzania, Cheryl Palm, Columbia University Smart Subsidies to Promote Peer Monitoring of Conservation Agriculture Compliance in Malawi, Andrew Bell, New York University Rural Livelihoods and Institutional Reform in Small￾Scale Fisheries in Tanzania, Yaniv Stopnitzky, University of San Francisco Achieving Development Impact with Complementary Stress-resistant Seed & Financial Technologies in Mozambique & Tanzania, Michael Carter, University of California, Davis 95 ANNEX G: EVALUATION QUESTIONS (AS AGREED TO AND DETAILED IN EVALUATION PROTOCOL) 96 The evaluation research questions that follow are based upon the questions outlined in the Expression of Interest, though with some modification, separation, and amalgamation occurring when it is felt that brings greater clarity and/or efficiency to management of the questions. 1. Research Quality 1.4 Have the research projects developed scientifically valid and robust conclusions and professional-level outputs? In what ways did the portfolio of projects make progress and contribute toward the stated research objectives in the RFA? How did it diverge from objectives in RFA? 1.5 How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement? 1.6 To what extent has work from previous iterations of the lab been effectively incorporated and built upon? 2. Outreach & Dissemination 2.1 To what extent has the AMA IL and its research projects effectively analyzed, synthesized, and distilled research results into actionable information or recommendations? What are examples of successful outreach aimed at achieving impact (or of adoption of recommendations)? In what ways could research results be better communicated with different stakeholders, including USAID Missions, implementing partners, public and private sector partners, including technical and non-technical stakeholders? 2.2 In what ways have different partners (especially USAID Missions and the private sector) been engaged in the research process? What opportunities are there to increase this engagement? 3. Policy 3.1 In what ways was policy research relevant to current concerns, objectives, and needs of private and public sector stakeholders, USAID Missions, and BFS? 3.2 Which, if any, innovations, or research findings are most likely actionable and policy relevant? 3.3 What characteristics of these models or innovations make them more policy relevant and actionable? 3.4 Is there any evidence of changes to policy, programming, private sector products, or approaches as a result of research findings? In what ways, if any, did the project monitor these changes? 4. Capacity Building 4.1 How well have the projects and consortium of researchers identified and addressed academic and technical capacity needs of host country stakeholders? To what extent has this contributed to capacity building, broadly defined, in the host country? 4.2 In what ways did the consortium of researchers and projects support the participation of the private sector? 4.3 What opportunities are there to better support the inclusion of the private sector in market viable development solutions? 5. Program Management 5.1 How did the management entity effectively communicate and coordinate with research partners to achieve the objectives of the RFA? 6. Future Directions 6.1 What SPECIFIC technical areas and research topics merit new or continuing research investment particularly with regard to the new Global Food Security Strategy? 97 ANNEX H: KEY INFORMANT INTERVIEW AND FOCUS GROUP DISCUSSION GUIDES 98 The following six guides were developed to provide a framework for KIIs and FGDs undertaken with key audiences. The six broad audiences were: 1. AMA IL Management Team in UC Davis 2. AMA IL Implementing Partners on the ground 3. AMA IL Supported Researchers (including sub-grant recipients) 4. U.S. Missions, Other Feed the Future Programs 5. Community Leaders and Community Participants (also used as guide for FGDs) 6. Agricultural Extension Workers Ahead of each individual meeting, the ET adapted the question guide in order to tailor it to the specific context of the interviewee. 1. Key Informant Interviews—AMA IL Management Team Used with – core staff working as part of the AMA IL team based at UC Davis Start Up Please describe how this iteration of AMA IL aligns with the RFA. Do you believe that the current programming of AMA IL builds logically off earlier iterations? Describe the process for identifying research to be supported by AMA IL. How do you ensure the relevance of AMA IL-supported research to the context of smallholder farming communities where the research is occurring? Partnerships What was the approach to mobilizing partnerships in support of AMA IL research? Was sufficient effort put in to ensure the full landscape of relevant stakeholders was aware of and engaged regarding the proposed research? Are development partners sufficiently active in the program and contributing to research discussion? What have been the strengths/weaknesses and challenges/successes faced by the program and/or individual researchers in terms of maintaining effective partnerships? Are you satisfied that there is sufficient national ownership of the research program? What indicators exist of strong national ownership? What opportunities are there to improve partnerships? Are there other potential partnerships that could strengthen the research and potential for dissemination and adoption? Research Approach and Quality To what extent were: 1) current host government priorities and recommendations; and 2) Feed the Future priorities reflected in decisions related to the specific research areas supported? How are research needs identified and are there ways that needs identification and development can be improved? Are you satisfied that sufficient protocols exist for documentation of research results? Is an inventory of results maintained? If so, (and if necessary) are these accessible by different partners? What processes are in place for validation of results? How common is it for results to be published? Are social scientists participating/active in validation processes, e.g., health, nutrition, gender? 99 Are you satisfied that research projects are (or are on track to) being effectively analyzed, synthesized, and distilled into actionable information or recommendations? How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement? What is the value add of the AMA IL Management Unit to the work of individual researchers? How does the AMA IL Management Unit pull such a vast array of research into a coherent program that facilitates synergies and linkages between research efforts and contexts? Inclusion and Capacity Building To what extent is capacity building promoted and supported through the research program? • • • • • National partners Development partners Farmer-Based Organizations Farmers Private sector/Input dealers How well have the projects and consortium of researchers identified and addressed academic and technical capacity needs of host country stakeholders? To what extent has this contributed to capacity building, broadly defined, in the host country? Given the market focus of AMA IL, have private sector actors been adequately included and supported in the research? Have there been changes over time in levels of participation in the research by partners—good and bad? What factors are key to maintaining stakeholder enthusiasm for research? Have efforts been made to ensure women feel included within the program approach? What mechanisms exist to monitor the degree that research approaches are gendered? Outreach and Dissemination Describe the range of strategies in place for dissemination of AMA IL results—both within AMA IL and external of AMA IL? Is it suitable for achieving traction and impact with key stakeholders? Are there examples of results dissemination leading to technology uptake by external parties? Given low levels of literacy within many farming communities, what strategies have been employed to ensure effective communication with farmers? Do you have a specific strategy for proactively promoting results with potential new partners or relevant organizations? Do you monitor researchers to ensure they have a line of communication and/or collaboration with key stakeholders as laid out in the research plan? • • • USAID Missions in the region and how the research is fitting into the overall Mission priorities? Other relevant U.S. programs (esp. Feed the Future) in the country and region? Other relevant programs? 100 Research Monitoring Overall, are you satisfied that the program is on track to provide results that are relevant to the needs of small landholders? What have been the standout findings of the research? To what degree can AMA IL claim programmatic attribution for the achievement of its supported researchers? (esp. in multi-donor contexts) Are systems in place to measure adoption (or potential for adoption) in communities? Is uptake in line with your expectations? If so, why is this and if not, why not? Do roles exist for local actors to contribute to performance monitoring and assessment? Are M&E and data management systems effective in sharing information between partners? Are they effective in fine-tuning approaches? Are market issues, challenges, and realities adequately considered in the research approach? Is there any evidence of changes to policy, programming, private sector products, or approaches as a result of research findings? In what ways, if any, did the project monitor these changes? What are the characteristics common to research that successfully gain traction in relation to policy and recommendations? Management Is the current AMA IL program structure appropriate for optimum achievement of program objectives? What changes have been made to the program management approach over time? What are the major challenges faced in managing such a diverse program? What changes would further strengthen the program moving forward? Moving Forward What SPECIFIC technical areas and research topics merit new or continuing research investment? What changes would you suggest for any new AMA IL iteration to best progress from the work of this current iteration? 101 2. Key Informant Interviews – AMA IL Implementing Partners Used with—interviewing of senior staff of agencies partnering with AMA IL researchers at country level, e.g., host government, donors, private sector, NGO partners Start Up Please describe the context of your engagement with AMA IL and your role in setting the approach and focus of the research you are participating in. What is your specific role in the research activity being undertaken? (If active prior to this iteration) Do you believe that the current programming of AMA IL builds logically off earlier iterations? What was your involvement in the process of identifying research areas to be supported by the program? How would you describe the relevance of this research to the context of smallholder farmers where the research is occurring? Do you regard yourself to be an active/equal partner in the research? Partnerships What was the approach to mobilizing partnerships in support of this research? Was sufficient effort put into ensuring the full landscape of relevant stakeholders was aware and engaged regarding the proposed research? Are development partners sufficiently active in the program and contributing to research discussion? What have been the strengths/weaknesses and challenges/successes in terms of maintaining effective partnerships? Are you satisfied that there is sufficient national ownership of the research program? What indicators exist of strong national ownership? What opportunities are there to improve partnerships? Are there other potential partnerships that could strengthen the research and potential for dissemination and adoption? Research Approach and Quality To what extent were: 1) current host government priorities and recommendations; and 2) Feed the Future priorities reflected in decisions related to the specific research areas supported? Are you satisfied that sufficient protocols exist for documentation of research results? Is an inventory of results maintained? If so, (and if necessary) are these accessible by different partners? What processes are in place for validation of results? How common is it for results to be published? Are social scientists participating/active in validation processes, e.g., health, nutrition, gender? Are you satisfied that research projects are (or are on track to) being effectively analyzed, synthesized, and distilled into actionable information or recommendations? What is the approach employed in terms of ensuring farmers feel an active role in the research process and validating efficacy of trials? How effectively do the different partnerships contribute toward a rigorous process and quality research outcomes? 102 How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement? Inclusion and Capacity Building To what extent are capacity building elements promoted throughout the research program? • • • • • National partners Development partners Farmer-Based Organizations Farmers Private sector/Input dealers How well have the projects and consortium of researchers identified and addressed academic and technical capacity needs of host country stakeholders? To what extent has this contributed to capacity building, broadly defined, in the host country? Given the market focus of AMA IL, have private sector actors been adequately included and supported? Have there been changes over time in levels of participation in the research—good and bad? Have efforts been made to ensure women feel included within the program approach? Outreach and Dissemination What strategy exists for dissemination of AMA IL results within AMA IL and external of AMA IL? Is it suitable for achieving traction and impact with key stakeholders? Are there examples of results dissemination leading to technology uptake by external parties? Given low levels of literacy within many farming communities, what strategies have been employed to ensure effective communication with farmers? Do you have a specific strategy for proactively promoting results with potential new partners or relevant organizations? Do you have a line of communication and/or collaboration with? • • • USAID Missions in the region? Other relevant U.S. programs (esp. Feed the Future) in the country and region? Other relevant programs? Research Monitoring Overall, are you satisfied that the program is on track to provide results that are relevant to the needs of small landholders? What have been the standout findings of the research? To what degree can AMA IL claim programmatic attribution for these achievements? (esp. in multi-donor contexts) Are levels of uptake in communities in line with your expectations? If so, why is this and if not, why not? Do roles exist for local actors to contribute to performance monitoring and assessment? Are M&E and data management systems effective in sharing information between partners? Are they effective in fine-tuning approaches? Examples? Are market issues, challenges, and realities adequately considered in the research approach? 103 Management Is the current AMA IL program structure appropriate for optimum achievement of program objectives? What changes would strengthen the program moving forward? Moving Forward In your opinion, what SPECIFIC technical areas and research topics merit new or continuing research investment? 104 3. Key Informant Interviews—AMA IL Supported Researchers (including sub-grant recipients) Used with—researchers supported through AMA IL grants, including national-level researchers Start Up Can you describe how you first became involved in AMA IL? What was the process involved in identifying the research area you work on? How would you describe the relevance of your research area to the needs of smallholder farmers resident in the area where research is occurring? (If active prior to this iteration) Do you believe that your research benefits from or builds logically off the work of earlier iterations of AMA IL? What were the roles of different parties in assessing your proposal to be supported by the program? Did the review process strengthen the research? Did it help identify stakeholders and/or strengthen pathways for dissemination? Partnerships What was your approach to mobilizing partnerships in support of this research? Are you satisfied that the full landscape of relevant stakeholders is aware and engaged regarding the proposed research? Has the AMA IL Management Team been supportive of stakeholder identification and engagement? Are development partners active in the program and contributing to research discussion? What have been the strengths/weaknesses and challenges/successes in terms of maintaining effective partnerships? What could be done to better leverage partnerships? By you? By the program? Are you satisfied that there is sufficient national ownership of the research program? What indicators exist of strong national ownership? What opportunities are there to improve partnerships? Are there other potential partnerships that could strengthen the research and potential for dissemination and adoption? Research Approach and Quality To what extent were: 1) current host government priorities and recommendations; and 2) Feed the Future priorities reflected in decisions related to the specific research areas supported? Are you satisfied that sufficient protocols exist for documentation of research results? Is an inventory of results maintained? If so, (and if necessary) are these accessible by different partners? What processes are in place for validation of results? Have you had/will you have results published? Are social scientists participating/active in validation processes, e.g., health, nutrition, gender? Are you satisfied that your research is capable of being distilled into actionable information or recommendations? What is the approach employed in terms of ensuring farmers feel an active role in the research process and validating efficacy of trials? How effectively do the different partnerships contribute toward a rigorous process and quality research outcomes? 105 How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement? Inclusion and Capacity Building To what extent are capacity building elements promoted through the research program? • • • • • National partners Development partners Farmer-Based Organizations Farmers Private sector/Input dealers How well has the researcher/research team identified and addressed academic and technical capacity needs of host country stakeholders? To what extent has this contributed to capacity building, broadly defined, in the host country? Describe examples of capacity building impact? Given the market focus of AMA IL, have private sector actors being adequately included and supported? Have there been changes over time in levels of participation in the research—good and bad? Have efforts been made to ensure women feel included within the program approach? What mechanisms exist to monitor the degree that research approaches are gendered? Outreach and Dissemination Describe the range of strategies in place for dissemination of AMA IL results—both within AMA IL and external of AMA IL? Is it suitable for achieving traction and impact with key stakeholders? Are there examples of results dissemination leading to technology uptake by external parties? Given low levels of literacy within many farming communities, what strategies have been employed to ensure effective communication with farmers? Do you have a specific strategy for promoting results with potential new partners or other relevant organizations? Do you benefit directly from other AMA IL research? How do you work to ensure a line of communication and/or collaboration with? • • • USAID Missions in the region? Other relevant U.S. programs (esp. Feed the Future) in the country and region? Other relevant programs? Research Monitoring Overall, are you satisfied that the program is on track to provide results that are relevant to the needs of small landholders? What have been the standout findings of the research? To what degree can AMA IL claim programmatic attribution for the achievement of its supported researchers? (esp. in multi-donor contexts) Are systems in place to measure adoption (or potential for adoption) in communities? Is uptake in line with your expectations? If so, why is this and if not, why not? Do roles exist for local actors to contribute to performance monitoring and assessment? 106 Are M&E and data management systems effective in sharing information between partners? Do they contribute to fine-tuning of approaches? Are market issues, challenges, and realities adequately considered in the research approach? Is there any evidence of changes to policy, programming, private sector products, or approaches as a result of research findings? In what ways, if any, did the project monitor these changes? What do you see as the characteristics common to research that successfully gain traction in relation to policy and recommendations? Management Is the current AMA IL program structure appropriate for optimum achievement of program objectives? How could program-level management better support your work and outcomes? What are the major challenges faced in managing such a diverse research program? Do you see AMA IL work as coherent and connected? What changes would further strengthen the program moving forward? Moving Forward What SPECIFIC technical areas and research topics merit new or continuing research investment? What changes would you suggest for any new AMA IL iteration to best progress from the work of this current iteration? 107 4. Key Informant Interviews—U.S. Missions, Other Feed the Future Programs Used with—U.S. Missions and other Feed the Future programs operating in research areas who would/should potentially stand to benefit from the research, and who have potential to adopt and consolidate learning Start Up Please describe the context and level of your engagement with AMA IL and the different research being undertaken in your country of responsibility/the wider region. Do you regard this research as being relevant to needs you observe in your country of responsibility? Does it complement your work in any way? Have researchers and/or AMA IL program management been proactive in engaging with you? Do they provide information to you in an accessible form? Is it in a form that makes it easy for to use and/or forward to other potential stakeholders of interest? What are your observations in terms of issues and needs related to: • • • Risk management and resilience? Rural and agriculture finance? Inclusive market access and engagement? Partnerships Are the researchers and/or AMA IL Program Management active in the overall Feed the Future Community of practice in this country? If not, what ideas do you have as to how they could better contribute to Feed the Future thinking and programming in this country? Do you believe they are engaging relevant stakeholders and that these stakeholders are aware of and engaged regarding the proposed research? Is there sufficient national ownership of the research program? What indicators exist of strong national ownership? What opportunities are there to improve partnerships? Are there other potential partnerships that could strengthen the research and potential for dissemination and adoption? Research Approach and Quality Does the research approach reflect: 1) current host government priorities and recommendations; and 2) Feed the Future priorities? Do you have any observations regarding research quality? Do you know of results being published? How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement? Inclusion and Capacity Building Do you believe capacity building elements are being promoted throughout the research program? • • • • National partners Development partners Farmer-Based Organizations Farmers 108 • Private sector/Input dealers Given the market focus of AMA IL, have private sector actors been adequately included and supported? What is your experience of the opportunities and challenges of private sector engagement in this country? Do you perceive that that the needs of women are addressed within the program approach? Outreach and Dissemination What strategies have you encountered regarding dissemination of AMA IL results? Is the approach, in your opinion, suitable for achieving traction and impact with key stakeholders? Are you aware of results dissemination leading to technology uptake by external parties? Have you utilized research results in any form? Research Monitoring Overall, are you satisfied that AMA IL research is on track to provide results that are relevant to the needs of small landholders? Are M&E and data management systems effective in sharing information between partners? Do you observe fine-tuning and adaptation approaches? Examples? Are market issues, challenges, and realities adequately considered in the research approach? Management Is the current AMA IL program structure appropriate for optimum achievement of program objectives? What changes would strengthen the program moving forward? Moving Forward In your opinion, what SPECIFIC technical areas and research topics merit new or continuing research investment? 109 5. Key Informant Interviews—Community Leaders and Community Participants (also used as guide for FGDs) Used at the community level, to determine research practice at the community level, and degree to which local stakeholders have been engaged and sustainability issues considered Start Up Describe the research being undertaken in this village. What new technologies have you adopted on your farm BEFORE AMA IL? How did you first become aware of the research? Why did you decide to become involved in the project? What did you hope to get from the program? Who are the people you have most contact with in relation to the trial? What do you hope to get out of the research? Do you see any risks to you and your family from participation in the research? Research Approach How was the research purpose explained to you? Do you feel like you understand the research purpose? Have you had an opportunity to provide your perspective on the issue that the research is trying to address? Do you feel like your issues/thoughts/concerns have been reflected in the research approach? Do you have a person you can easily contact to raise issues/ask questions in relation to the research? Have you been provided any inputs because of your involvement in the research? Was access to inputs a major incentive for your participation? Communication Are you updated as scheduled regarding the research findings? Is this information easy to understand? Is it provided in an accessible form? Do you have any recommendations on how the results could be better communicated to you? Is there interest in the broader community? Do people not participating in the trial get access to the information? Technical Questions Technical questions will be developed in relation to the specific research approach. Adoption Do you believe the research findings are of relevance and helpful to you? To your community? Do you plan to adapt your practice and planning to the technology? Do you think other people in the community are interested to take up the technology? Will you continue to use the technology if inputs are no longer provided free of charge? Have new private sector actors started to provide services to your community? 110 • • • DT seeds Insurance options Financial Services Recommendations What would make you value the research more? What other research is required to help people like you? 111 6. Key Informant Interviews—Extension Workers (also used as guide for FGDs) Used with—both Agricultural Extension Agents, Women in Agriculture Agents, and CEAs (in Ghana) to determine research practice at the community level, and degree to which local stakeholders have been engaged and sustainability issues considered. Start Up Describe the research being undertaken in this village. Describe your role in participating communities. What support and/or training have you been provided in relation to the research purpose and approach? Research Approach How was the research purpose explained to you? Do you feel like you understand the research purpose? Have you had an opportunity to provide your perspective on the issue that the research is trying to address? Do you feel like your issues/thoughts/concerns have been reflected in the research approach? Do you have a person you can easily contact to raise issues/ask questions in relation to the research? Do you have the necessary knowledge and/or resources necessary to fulfill your role in support of this research? Communication Are you updated regularly regarding the research findings? Is this information easy to understand? Is it provided in an accessible form? Do you have any recommendations on how the results could be better communicated to you? Technical Questions Technical questions will be developed in relation to the specific research approach in that area. Extension Are you provided resources through the research activity that help facilitate the extension work? Do you believe the research findings should and/or can be mainstreamed within the extension approach? Do you think other people in the community are interested to take up the technology? Do you think smallholder farmers will continue to use the technology if inputs are no longer provided free of charge? Have new private sector actors started to provide services to related to the research area? • • • DT seeds Insurance options Financial services Has your involvement in the research forced you to reduce your effort in? • Other geographic areas 112 • Other substantive areas of extension Recommendations What would make the research more valuable to this area? What other issues require research in this area to help strengthen resilience of smallholder farming communities? 113 ANNEX I: AMA IL EVALUATION TRAVEL AND SCHEDULING MASTER PLAN (AUGUST-SEPTEMBER 2017) 114 Date Place Activities Tuesday, August 8th Arrive Davis Wednesday, August 9th Davis Meetings with AMA IL team 0830 • Introductions 0900 • Research Program Selection Approaches 1100 • Project Management and Reporting 1400 • Communications and Outreach 1600 • Capacity Building 1900 • Dinner, Michael Carter Thursday, August 10th Davis Meetings with AMA IL Team 0830 • Relationships with USAID and Others 1000 • Introduction and overview of Field Visit Projects 1200 • Ghana Projects 1400 • Drought Tolerant Maize Project 1530 • Future Directions Friday, August 11th Davis Day off Saturday, August 12th Depart Davis 1410 Depart Sacramento for Accra Sunday, August 13th Arrive Accra 2000 In transit; arrive Accra 2200 Monday, August 14th Accra 0930 • IFPRI 1200 • Chief of Party, Financing Agriculture in Ghana (FIN-GAP) 1400 • PI DIRTS (by Skype) 1600 • Assistant PI, Promoting Adoption of Improved Production Technologies Among Smallholders in Ghana via Couple Credit and Insurance Contracting Tuesday, August 15th Accra 1000 • Director Ghana Agricultural Insurance Program 1400 • Technical Advisor to Minister of Food and Agriculture Wednesday, Fly Accra 0930 115 Date Place Activities August 16th to Tamale • Fly Accra to Tamale 1130 • Chief Scientist, Africa Research in to Sustainable Intensification for a New Generation (Africa RISING) West Africa project 1400 • Meeting with team at Innovations for Poverty Action Thursday, August 17th AM Visit participating DIRTS community—Afaylili village • Meeting with Village elders • FGD with men receiving program treatments • FGD with women receiving program treatments PM Visit participating DIRTS community—Ghani village • Meeting with Village elders • FGD with men receiving program treatments • FGD with women receiving program treatments Friday, August 18th AM • FGD with DIRTS-supported CEAs PM • Managing Director Wunzuyu Agro Chemicals and Deputy Director Department of Food and Agriculture, Mion District Saturday, August 19th Travel by road to Upper East 0800 • Travel Tamale to Bolgatanga, Upper East 1500 • Director, BEFFSA Rural Bank 1900 • Project Manager, Association of Rural Banks Sunday, August 20th Travel by road Upper east to Tamale 1000 • Credit Officer and General Manager, Bangmarigu Rural Bank • Travel Bolgatanga to Tamale 1600 • Researcher, Savanna Agricultural Research Institute Monday, August 21st Fly Tamale to Accra (afternoon) 0830 • Former Director Food and Agriculture, Savelugu District 1000 • N.Y., Director Food and Agriculture, Sagnarigu District o Manager Information Systems o Women in Agricultural Development Officer 1130 • Deputy Chief of Party, Agriculture Technology Transfer project (USAID-funded) 1400 116 Date Place Activities • Deputy Chief of Party, ADVANCE project (USAID￾funded) o Agricultural Financing Officer 1530 • Marketing Officers, GAIP Northern Zone Tuesday, August 22nd Depart Accra 2045 1100 • IPA Country Director 1500 • Agriculture Team Leader, USAID Mission 2045 • Depart Accra for Nairobi/Dar es Salaam/Arusha Wednesday, August 23rd Arrive Arusha 1245 • Arrival in Arusha 1400 • Introductions to DT Seeds team; CIMMYT 1530 • Meet IFFA Seed Company Director, and Visit facility near Arusha 1700 • Meet with International Institute for Tropical Agriculture (IITA) Tanzania Country Representative Thursday, August 24th Singida 0600 • Drive to Singida 1100 • Meet with Director Singida DAICO and Agriculture Department Representatives 1400 • Meeting with Singida Agro dealer Friday, August 25th Singida 8 – 12 pm 1 – 4 pm 0800 • Visit Survey community 1, Merya Village (insured seed) 1300 • Visit Survey community 2, Ndugwila Village (uninsured seed) Saturday, August 26th Dodoma (need to confirm with Brian Dillon) 0600 • Drive to Dodoma 1400 • Meeting at Institute of Rural Development Planning • Meeting with private sector participants Sunday, August 27th Dodoma Day Off 117 Date Place Activities Monday, August 28th Morogoro Travel Dodoma to Morogoro PM • Dean Department of Soil and Geological Sciences, Sokoine University of Agriculture • Co PI and Professor Department of Soil and Geological Sciences, Sokoine University of Agriculture • Meeting with grantees at SUA Tuesday, August 29th Morogoro Ward/Village visit to Site Specific Soils—Mtamba Village, Kiseme Ward, Morogoro • Meet with Village Council • FGD with 8 program participants (2 from each treatment) • Meeting with Ward Extension Officer Wednesday, August 30th Morogoro Village visits to DT Seeds Project AM Mangai Village—Insured seed • Meet with Village Council • FGD with 8 program participants (2 from each treatment) • Meeting with Ward Extension Officer PM Melela Village, Melela Ward—Uninsured seed • Meet with Village Council • FGD with program participants • Meeting with Ward Extension Officer Thursday, August 31st Dar Drive Morogoro to Dar 1500 • PM: Meeting with Mission Friday, September 1st Dar 1100 • Meeting with COP NAFAKA 1300 • UAP Insurance Team debrief Report preparation Saturday, September 2nd Dar Departure of team Sunday, September 3rd Home Arrive home 118 ANNEX J: GHANA: “DISSEMINATING INNOVATIVE RESOURCES AND TECHNOLOGIES TO SMALLHOLDER FARMERS” (DIRTS) 119 This project builds off an earlier research program—Exploring Underinvestment in Agriculture (EUA)— that identified reluctance to take risk as the major inhibitor of agricultural production among smallholders. It is a multi-donor activity enjoying support from Massachusetts Institute of Technology Agriculture Adoption Initiative, the UK Department for International Development—Economic and Social Research Growth Program, and the Development Innovation Venture of USAID, as well as support from AMA IL. The research is led by Yale University, in partnership with the Savanna Agricultural Research Institute in northern Ghana, and IFPRI in Ghana. On the ground implementation is undertaken by NGO Innovations for Poverty Action (IPA), based in Tamale, the main town in northern Ghana. This composition of donors and partners provides the research an acutely relevant cross-section of agencies that bring an important set of different skills to the research, while all also being steeped in the practice of high-quality research themselves. IPA’s “on-the ground” role also allows for research to be managed in a manner that allows for close attention to detail, while also offering capacity for trouble shooting and timely modification as required. The DIRTS research context aimed to bring focus to options to support poor farmers who face chronically low yields and cultivate primarily maize. The DIRTS research sample is 3,240 households across 162 farming communities, with households randomized into four treatment groups: 1,000 receiving crop index insurance and intensified extension; 600 receiving insurance and improved inputs; 600 receiving insurance, extension, and inputs; and 600 receiving extension only. While the number of households closely studied is 3,240, many more were active and stood to benefit from the approach, particularly the community-based extension service the research facilitated and efforts to improve input supply lines. Detailed surveying occurred throughout the research process, including both an annual household survey and a knowledge attitudes and practice survey. In total, more than 12,000 questions were asked of the four treatment groups annually. The research approach has also enhanced community capacity, through: greater community awareness of new technologies and options for agriculture; capacity building of government extension workers; awareness raising in relation to index insurance; and strengthened local supply lines and linkages between communities and agricultural input traders. While the research on the ground appears to be of the highest quality, the ET felt that the project could have been more proactive and deliberate in sharing findings, information, and progress with key stakeholders—particularly at the national level. An example of the opportunities presented by systematic engagement of senior officials was highlighted during an interview with the Senior Technical Advisor to MOFA, who is a strong advocate of the need for agricultural insurance focused on smallholders, yet who had no knowledge of the DIRTS program. This senior official noted the challenges inherent in engaging MOFA around innovative approaches, given the revolving door at the Ministry, which has seen five Ministers in five years. 120 ANNEX K: LENDER UPTAKE AND USE OF THE INSURED CREDIT PRODUCT IN NORTHERN GHANA 121 Using the product-related education provided to the Rural Banks by the AMA research program, the Garu Rural Bank participated as the lender of insured loans to small farmers in the Credit Combined with Index Insurance Project. Since project end, this bank has subsequently used that experience to develop and extend an ADAPTED insurance backed credit product to smallholder farmers. The bank’s Marketing Director (and former Credit Manager) indicated to the ET that they believe this will serve as a better basis for SCALING UP insurance backed loans to small farmers. This bank plans to continue making index insured loans to smallholder farmers, backed by GAIP provided insurance. However, the bank will do so at a lower insurance premium rate of 3 percent, which was negotiated with GAIP, as part of a direct MOU between Garu Bank and GAIP. The bank’s General Manager estimated that 90 percent of the borrowers they have been serving with the adapted product participated in the AMA IL research project. Since negotiating the reduced insurance premium rate with GAIP, this Rural Bank has made insured loans to 1,924 farmers, of which 844 have been male borrowers and 1,080 female borrowers. These borrowers have made up 138 groups and have been producing on 3,134 hectares (average 1.5 hectares per farmer). These have been “mezzo” insured loans, where the indemnity is paid to the lender in the case of crop loss, which in turn uses the indemnity to pay off the balance on the farmer loan. This is how the adapted insured loan product works: • • • • This Rural Bank provides the financial education and introduction of their product to smallholder farmers, as part of local groups. Credit is applied for at the Rural Bank by a group of farmers from the same community. Loans are made to and repaid by INDIVIDUAL farmers within the group. Members of the group in turn provide a guarantee for each other’s loan repayments, as each borrowing farmer is repaying into a GROUP ACCOUNT (basis for the joint and several repayment guarantee). Market building among small farmers, brought about by participation in the AMA IL research program, has created a wider market for this Rural Bank for extending the index insurance backed loan to small farmers. Managers at another rural bank interviewed by the ET, Bangmarigu in Upper East Region, indicated that they would continue offering the insurance-backed loan to farmers, IF they could negotiate a lower than 10 percent premium rate with GAIP based upon a direct MOU with them, and if GAIP provides the bank with better access to information on satellite-based weather projections, on which GAIP does analysis on indexes. Step-by-Step Process for Insured Loan Extension to Small Farmers at Garu Rural Bank in Ghana: • • • • • • The Bank provides the prospective borrowers with financial education on the use of the loan product and repayment. The Bank’s Board approves the loan amount needed to satisfy the input needs of all of farmers in applying group. The Bank issues a coupon (voucher) to the group, the leader of which in turn takes it to the inputs provider and gets the seeds and fertilizer members need. The supplier’s voucher is sent to the bank, which checks it for accuracy against the “budget” established by the group for their input needs. The bank then approves the invoice and disburses the total payment to the supplier. During the growing season (about 12 months in the case of maize), each individual farmer repays his loan in quarterly payments. 122 • At the harvest of the crops, an “aggregator” comes to the growers’ location, collects the crop and pays the producers the “fair market price” by check, which is deposited to the farmers’ joint account at the Bank. 123 ANNEX L: POTENTIAL FOR INTEGRATED PROGRAMMING AND LEARNING—AMA IN TANZANIA 124 The development landscape in Tanzania has long been challenging, complicated, and crowded as the global development community works to better harness the country’s human and physical potential to reduce deeply entrenched rates of poverty. As is common with development actors in Tanzania, AMA IL supports a portfolio of different activities all aimed at better understanding the many elements that shape rural households’ resiliency, food security, and participation in agriculture-led economic growth. Collectively, these different efforts cut across many of Tanzania’s most pressing development challenges and opportunities. AMA IL-supported research includes detailed focus on specific topics such as DT seeds, index insurance, soil improvement, satellite weather information, private sector mobilization, and ICT. Despite this breadth, there is also a very clear and common thread to the whole portfolio. Each effort in its own way is working to promote understanding and enhanced application of the many different technologies now available in the marketplace and of demonstrated relevance to the complex needs of smallholder farmers in Tanzania. In turn, each AMA investment also carries contemporary, cutting edge knowledge and understanding that is of relevance and potential importance to the other. For example, while the “Drought Tolerant Maize and Index Insurance Bundling” project ostensibly focuses on two technologies, its approach is heavily invested in strengthening private sector seed producers and is moving towards a need for far greater consideration of ICT and mobile phone technologies for awareness raising and registration of insured seed products. Improved farmer access to higher quality weather information and prediction, as well as better site-specific assessment of nutrient depletion all have the potential to further enhance farmer willingness to adopt newer technologies. Similarly, while the “Evaluating the Effect of Site-Specific Soil Information on Farmer Input Choices” project has identified highly significant divergence between government fertilizer recommendations and the nutrient needs of specific areas, its methodology also sheds very important light on the capacity and/or willingness of poor farming households to adopt proven technologies even when advice is specifically tailored to their unique context. Such findings are of immediate value and relevance to all of the many actors working to build evidence as a strategy for increasing demand for higher quality inputs. Enhancing consumer choice, better understanding market behavior and investigating how mobile phone technology is utilized in poor rural communities underpins the “Communication, Search, and Mobile Phones: A Telephone Directory Intervention in Tanzania” project. This focused research has the potential to add enormous value to all of the programs across Tanzania thinking through and trialing the potential application of mobile phone technology in information dissemination, product registration, and financial services. While all of the above are AMA IL research areas, they are all also subjects of acute interest to NAFAKA, one of USAID’s major Feed the Future investments in Tanzania. They would also potentially be of interest to many of the 11 other Innovation Labs currently active in Tanzania. Such a context raises questions regarding the potential for more strategic, dynamic, and mutually supportive KM across Feed the Future investments in a crowded and ever-evolving implementation environment such as Tanzania. 125 ANNEX M: AMA IL CAPACITY BUILDING WITHIN HOST COUNTY ACADEMIC COMMUNITY 126 One of the three goals of the AMA IL focus on capacity building is increasing the capacity of host country researchers to conduct rigorous research on agricultural development and related fields. The Lab has sought to increase the capacity of individuals and host country research institutions by conducting short courses created around impact evaluation methodology and how to engage with evidence from these evaluations, as well as training for IPs on project-related technical activities, such as introduction of index insurance and drought resistant seed varieties. AMA IL research activities have also developed and utilized the talent of a number of local researchers, including long-term students at U.S. universities who can provide substantive assistance in implementation of specific research activities, and go on to enhance their host country’s capacity to continue to conduct research in these areas after Lab projects end. Working with local research partners in 15 countries, AMA IL conducted a number of capacity building activities, at various levels in the research process. Examples include the following: • • • • A partnership between researchers at Stanford University and Maseno University in Kenya included as part of partner institution capacity building short courses on research design and introduced a variety of impact evaluation strategies and analysis techniques, which the local institution could incorporate into ongoing training of graduate students. Mentoring of students directly related to the core work of the project is done when they are working directly with researchers on a project. For example, researchers at Columbia University have been working with a Sokoine University student, as part of research associated with her degree, to determine the best sampling time for fertilizer recommendations and the response of maize to different levels of nitrogen fertilizers, under the Lab-supported SoilDoc project in Tanzania. The Lab has on a number of occasions brought together researchers, local private sector partners, and policymakers to share research results and assist their introduction into participants’ development and planning work. An example is the 2017 conference, “Enhancing Smallholder Productivity in Kenya: Evidence from a Randomized Control Trial of New Seed Varieties,” which was conducted by researchers at the Lab and the Tegemeo Institute of Agricultural Policy and Development. Lab staff and researchers have conducted numerous one-on-one and small group meetings with host country stakeholders, to allow for open discussion about challenges and potential solutions among institutions and organizations directly engaged in relevant product and policy development. For example, over a two-year period Lab staff met with relevant decision-makers in the Government, including the Ministry of Agriculture and Development and the Insurance Board, to discuss the benefits, limitations, and related factors involved in introducing agricultural insurance, and index insurance specifically. 127 ANNEX N: EVALUATION WORK PLAN 128 Activities Responsible Person(s) July August September October 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 TASK 1. Develop Evaluation Design Prepare Concept Paper Evaluation team X Prepare Evaluation Protocol Evaluation team X X TASK 2: Data Collection Desk Review Evaluation team X X Undertake field work Evaluation team X X X X TASK 3: Data Analysis Summary data review Evaluation team X X X X Full data review Evaluation team X X TASK 4: Report Writing Prepare draft report Evaluation team X X Redrafting and final report Evaluation team X X 129 Deliverables Timeline Kick Off May 17, 2017 (when data were received from AMA-IL) Concept Paper To USAID by June 17, 2017 Performance Evaluation Protocol To USAID by July 17, 2017 Field work 22 days (including travel) through August 2017 Draft Assessment Report To USAID by September 20, 2017 Final Evaluation Report: To USAID by October 27, 2107 130 ANNEX O: OUTCOME MEASURES USED TO FRAME AND GUIDE EVALUATION APPROACH 131 Evaluation Questions Indicative Outcomes Indicative Outcome Measures Principal Data Sources Data Collection Methods 1. Research 1.1.1 Have the research projects developed scientifically valid and robust conclusions and professional-level outputs? • AMA IL-supported research reflects and responds to challenges articulated by poor farming communities. • An inventory exists of on￾station and on-farm research data. • Trials are clearly documented and well described in the research inventory and adhere to best research practices. • High-quality research is prepared and published. • Strategic communication of results raises awareness of successes and failures of innovations among key audiences. • Research results are well documented, with an inventory of results maintained. • On-farm experiments are evaluated statistically and economically when costs differ markedly among treatments. • Technologies and/or combinations of technologies were found to have scientifically robust conclusions. • Researchers express satisfaction at being able to utilize program￾wide data, as relevant, within their own research. • Research results are on course to be adopted and replicated. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers Data collection methods a. Compilation of reporting data b.–h. KIIs i. Compile all working papers 1.1.2 In what ways did the portfolio of projects make progress and contribute toward the stated research objectives in the RFA? • Capacity of host country institutions and scientists enhanced. • Innovative solutions to complex problems faced by smallholder farmers have been identified. • Broader development community engaged and adopting new technologies. • Number of local partner institutions and scientists participating in activities • New approaches are being applied by government and development partners that reflect AMA IL-generated knowledge. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers j. RFA 132 Evaluation Questions Indicative Outcomes Indicative Outcome Measures Principal Data Sources Data Collection Methods Data collection methods a. Compilation of reporting data b.–h. KIIs i. Compile all working papers j. Review of RFA 1.1.3 How did it diverge from objectives in RFA? • Comparative analysis of findings against RFA • Comparative analysis of findings against RFA Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers j. RFA Data collection methods a. Compilation of reporting data b.–h. KIIs i. Compile all working papers j. Review of RFA 1.2 How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement? • • • Strategies for better integrating weather risk into risk management planning are identified and are accessible to smallholder farmers. New options exist to better link smallholders to markets and value chains. Viable options are identified that open up access for rural • • • National structures and mechanisms exist for assessment and broadcast of weather risk. Smallholder farmers newly adopt financial services. Smallholder farmers report improved access to new technologies. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers j. Smallholder farmers 133 Evaluation Questions Indicative Outcomes Indicative Outcome Measures Principal Data Sources Data Collection Methods communities to financial • services. • • Viable options are identified for the interlinking of financial services. Enhanced awareness in terms of identifying weaknesses in approaches previously thought to be “successful.” Private sector entities have incentive to provide services to smallholder farmers. Data collection methods a. Compilation of reporting data b.–g. KIIs i. Compile all working papers j. FGDs 1.3 To what extent has work from previous iterations of the lab been effectively incorporated and built upon? • Longitudinal studies demonstrate continuity of learning between different iterations of AMA IL. • Lines of inquiry commenced in earlier iterations are now practiced and/or reflected in policy. Data sources a. Review of headline data from earlier iterations of AMA IL b. Program and project reporting c. Program management d. Researchers e. Academic partners f. Working papers g. RFA Data collection methods a.–b. Compilation of reporting data c.–e. KIIs f. Compile all working papers g. Review of RFA 2. Outreach and Dissemination 2.1.1 To what extent has the AMA IL and its research projects effectively analyzed, synthesized, and distilled research results into actionable • • Research results are available and tailored in forms accessible to key audiences. Clear recommendations are relevant and feasible to local contexts. • • • Policymakers have access to and value research results. Recommendations meet the needs of policymaking institutions. Policy and recommendations have been prepared that reflect Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners 134 Evaluation Questions Indicative Outcomes Indicative Outcome Measures Principal Data Sources Data Collection Methods information or recommendations? research findings and strong communication of researchers. g. Academic partners h. Missions (U.S. and other) i. Working papers Data collection methods a. Compilation of reporting data b.–h. KIIs i. Compile all working papers 2.1.2 What are examples of successful outreach aimed at achieving impact (or of adoption of recommendations)? • Stakeholder maps and communications approaches identify and effectively engage stakeholders of relevance. • Alignment and complementarity of contribution and approaches of different stakeholders. • Stakeholders across particular landscapes express satisfaction and inclusion in research efforts. • Holistic and complementary approaches are adopted by multiple stakeholders. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers j. Smallholder farmers Data collection methods a. Compilation of reporting data b.–g. KIIs i. Compile all working papers j. FGDs 2.1.3 In what ways could research results be better communicated with different stakeholders, including USAID Missions, IPs, and public and private sector partners, including • Communication approaches at both the program and research activity levels are effective in facilitating knowledge and resource sharing between sites. • • Communications strategies are documented and in place. Partners inside and external to AMA IL express ease at accessing program results. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners 135 Evaluation Questions Indicative Outcomes Indicative Outcome Measures Principal Data Sources Data Collection Methods technical and non￾technical stakeholders? h. Missions (U.S. and other) i. Working papers j. Smallholder farmers Data collection methods a. Compilation of reporting data b.–g. KIIs i. Compile all working papers j. FGDs 2.2 In what ways have different partners (especially USAID Missions and the private sector) been engaged in the research process? What opportunities are there to increase this engagement? • Tailored and nuanced approaches are developed by the program and researchers aimed at ensuring efficient and effective engagement of key stakeholders. • Communications strategies are documented and in place. • Partners inside and external to AMA IL express ease at accessing program results. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers Data collection methods a. Compilation of reporting data b.–h. KIIs i. Compile all working papers 3. Policy 3.1 In what ways was policy • research relevant to current concerns, objectives, and needs of private and public sector stakeholders, USAID Missions, and BFS? Research proposals clearly articulate how they respond to concerns, objectives, and needs of private and public sector stakeholders, USAID Missions, and BFS. • Private and public sector stakeholders, USAID Missions, and BFS express satisfaction that research reflects their priority information needs. • Private and public sector stakeholders, USAID Missions, and BFS actively utilize research. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners 136 Evaluation Questions Indicative Outcomes Indicative Outcome Measures Principal Data Sources Data Collection Methods h. Missions (U.S. and other) i. BFS j. Working papers k. Smallholder farmers Data collection methods a. Compilation of reporting data b.–i. KIIs j. Compile all working papers k. FGDs 3.2 Which, if any, innovations or research findings are most likely actionable and policy relevant? • Breadth of different research is deemed by researchers to be policy-relevant and actionable. • Policymakers have awareness and are utilizing research to develop new policy and action. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers Data collection methods a. Compilation of reporting data b.–g. KIIs i. Compile all working papers 3.3 What characteristics of these models or innovations make them more policy-relevant and actionable? • Research initiatives have undertaken measures to disaggregate different aspects of insurance with aim of understanding their individual strengths and weaknesses. • Policymakers have sophisticated appreciation of research findings and are able to break down research findings into actionable components. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners 137 Evaluation Questions Indicative Outcomes Indicative Outcome Measures Principal Data Sources Data Collection Methods g. Academic partners h. Missions (U.S. and other) i. Working papers Data collection methods a. Compilation of reporting data b.–g. KIIs i. Compile all working papers 3.4 Is there any evidence of changes to policy, programming, private sector products, or approaches as a result of research findings? In what ways, if any, did the project monitor these changes? • Mutually reinforcing and aligned action is being taken by public and private sector partners. • Reliable systems are capable of capturing granular impacts of research findings. • Directly attributable actions exist that can be attributed to AMA IL research. • Program is able to report clearly on changes attributable to program-supported research. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers Data collection methods a. Compilation of reporting data b.–g. KIIs i. Compile all working papers 4. Capacity Building 4.1 How well have the projects and consortium of researchers identified and addressed the academic and technical capacity needs of host country stakeholders? To what extent has this contributed • • • Baseline capacity assessments are available and guide capacity building. Local partners express satisfaction at skills being developed through AMA IL. Research capacity is being improved (at all levels) and • Detailed capacity assessment plans are available at the country level. • Researchers within universities and the National Agricultural Research System (NARS) have developed advanced skills and are being published. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners 138 Evaluation Questions Indicative Outcomes Indicative Outcome Measures Principal Data Sources Data Collection Methods to capacity building, broadly defined, in the host country? developed in subject areas relevant to improving the lives of poor farmers. • Public partners express confidence that the national capacity in AMA IL areas of focus has strengthened and is being utilized. h. Missions (U.S. and other) i. Working papers Data collection methods a. Compilation of reporting data b.–g. KIIs i. Compile all working papers 4.2 In what ways did the consortium of researchers and projects support the participation of the private sector in research activities? • Private sector mapping and engagement has been undertaken. • Private sector partners are well informed, contribute to research effort, and feel valued in relationship. • Private sector partners have confidence to utilize and apply research. • Private sector engagement strategies are articulated and available. • Private sector partners express satisfaction that they have been effectively engaged and heard in the research process. • Private sector believes their knowledge and perspective is reflected in the research approach. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers Data collection methods a. Compilation of reporting data b.–g. KIIs i. Compile all working papers 4.3 What opportunities are there to better support the inclusion of the private sector in market-viable development solutions? • Dialogue exists with private sector to identify opportunities for collaborative approaches that overcome known constraints that inhibit smallholders from more active market engagement. • Active collaboration with the private sector is occurring. • Private sector is introducing new approaches that are valued and utilized by smallholder farmers. Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers 139 Evaluation Questions Indicative Outcomes Indicative Outcome Measures Principal Data Sources Data Collection Methods Data collection methods a. Compilation of reporting data b.–g. KIIs i. Compile all working papers 5. Program Management 5.1 How did the ME effectively communicate and coordinate with research partners to achieve the objectives of the RFA? • Mechanisms are in place for frequent communication of program strategy and achievements. • Mutual understanding of project goals enhances opportunities for collaboration. • Researchers express consistent and aligned understanding of the overall program purpose and strategy. • Collaboration BETWEEN researchers is occurring. Data sources a. Program and project reporting, including Technical Committee Meetings b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. RFA Data collection methods a. Compilation of reporting data b.–g. KIIs h. Review of RFA; Technical Committee Meetings 6. Future Directions 6.1 What SPECIFIC technical areas and research topics merit new or continuing research investment? • To be determined • To be determined Data sources a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. BFS 140 Evaluation Questions Indicative Outcomes Indicative Outcome Measures Principal Data Sources Data Collection Methods j. Working papers k. Smallholder farmers Data collection methods a. Compilation of reporting data b.–i. KIIs j. Compile all working papers k. FGDs 141 ANNEX P: SUMMARY DESCRIPTION OF DATA SOURCES, COLLECTION, AND ANALYSIS METHODS AGAINST EVALUATION QUESTIONS 142 Evaluation Questions Data Sources Data Collection Method(s) Data Analysis Method(s) 1. Research Quality 1.1 Have the research projects developed scientifically valid and robust conclusions and professional-level outputs? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers a. Compilation of reporting data b.–h. KIIs i. Compile all working papers • Reading and analysis of existing reporting and research • Qualitative analysis of KII results • Quantitative analysis of existing surveys and research reports 1.1 In what ways did the portfolio of projects make progress and contribute toward the stated research objectives in the RFA? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers j. RFA a. Compilation of reporting data b.–g. KIIs i. Compile all working papers j. Review of RFA • Reading and analysis of RFA, existing reporting and research • Qualitative analysis of KII results • Quantitative analysis of existing surveys and research reports 1.1 How did it diverge from objectives in RFA? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) a. Compilation of reporting data b.–g. KIIs i. Compile all working papers j. Review of RFA • Reading and analysis of existing reporting and research • Qualitative analysis of KII results 143 Evaluation Questions Data Sources Data Collection Method(s) Data Analysis Method(s) i. Working papers j. RFA 1.2 How has the research advanced the work around risk management and resilience, rural and agriculture finance, and inclusive market access and engagement? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers j. Smallholder farmers a. Compilation of reporting data b.–g. KIIs i. Compile all working papers j. FGDs • Reading and analysis of existing reporting and research • Qualitative analysis of KII results • Quantitative analysis of existing surveys and research reports 1.3 To what extent has work from previous iterations of the lab been effectively incorporated and built upon? a. Review of headline data from earlier iterations of AMA IL b. Program and project reporting c. Program management d. Researchers e. Academic partners f. Working papers g. RFA a.–b. Compilation of reporting data c.–e. KIIs f. Compile all working papers g. Review of RFA • Reading and analysis of existing reporting and research • Qualitative analysis of KII results 2. Outreach and Dissemination 2.1 To what extent has the AMA IL and its research projects effectively analyzed, synthesized, and distilled research results into actionable information or recommendations? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) a. Compilation of reporting data b.–h. KIIs i. Compile all working papers • Reading and analysis of existing reporting and research • Qualitative analysis of KII results 144 Evaluation Questions Data Sources Data Collection Method(s) Data Analysis Method(s) i. Working papers 2.1 What are examples of successful outreach aimed at achieving impact (or of adoption of recommendations)? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers j. Smallholder farmers a. Compilation of reporting data b.–g. KIIs i. Compile all working papers j. FGDs • Reading and analysis of existing reporting and research • Qualitative analysis of KII and FGD results 2.1 In what ways could research results be better communicated with different stakeholders, including USAID Missions, IPs, and public and private sector partners, including technical and non￾technical stakeholders? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers a. Compilation of reporting data b.–g. KIIs i. Compile all working papers • Reading and analysis of existing reporting and research • Qualitative analysis of KII and FGD results 2.2 In what ways have different partners (especially USAID Missions and the private sector) been engaged in the research process? What opportunities are there to increase this engagement? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers a. Compilation of reporting data b.–h. KIIs i. Compile all working papers • Reading and analysis of existing reporting and research • Qualitative analysis of KII results 3. Policy 145 Evaluation Questions Data Sources Data Collection Method(s) Data Analysis Method(s) 3.1 In what ways was policy research relevant to the current concerns, objectives, and needs of private and public sector stakeholders, USAID Missions, and BFS? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. BFS j. Working papers a. Compilation of reporting data b.–i. KIIs j. Compile all working papers • Reading and analysis of existing reporting and research • Qualitative analysis of KII results 3.2 Which, if any, innovations or research findings are most likely actionable and policy-relevant? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers a. Compilation of reporting data b.–g. KIIs i. Compile all working papers • Reading and analysis of existing reporting and research • Qualitative analysis of KII results • Quantitative analysis of existing surveys and research reports 3.3 What characteristics of these models or innovations make them more policy￾relevant and actionable? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers a. Compilation of reporting data b.–g. KIIs i. Compile all working papers • Reading and analysis of existing reporting and research • Qualitative analysis of KII results • Quantitative analysis of existing surveys and research reports 146 Evaluation Questions Data Sources Data Collection Method(s) Data Analysis Method(s) 3.4 Is there any evidence of changes to policy, programming, private sector products, or approaches as a result of research findings? In what ways, if any, did the project monitor these changes? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers a. Compilation of reporting data b.–g. KIIs i. Compile all working papers • Reading and analysis of existing reporting and research • Qualitative analysis of KII results 4. Capacity Building 4.1 How well have the projects and consortium of researchers identified and addressed the academic and technical capacity needs of host country stakeholders? To what extent has this contributed to capacity building, broadly defined, in the host country? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers a. Compilation of reporting data b.–g. KIIs i. Compile all working papers • Reading and analysis of existing reporting • Qualitative analysis of KII results 4.2 In what ways did the consortium of researchers and projects support the participation of the private sector in research activities? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers a. Compilation of reporting data b.–g. KIIs i. Compile all working papers • Reading and analysis of existing reporting and research • Qualitative analysis of KII results 4.3 What opportunities are there to better support the inclusion of the private a. Program and project reporting a. Compilation of reporting data • Reading and analysis of existing reporting 147 Evaluation Questions Data Sources Data Collection Method(s) Data Analysis Method(s) sector in market-viable development solutions? b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. Working papers b.–g. KIIs i. Compile all working papers • Qualitative analysis of KII results 5. Program Management 5.1 How did the ME effectively communicate and coordinate with research partners to achieve the objectives of the RFA? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. RFA a. Compilation of reporting data b.–g. KIIs h. Review of RFA • Reading and analysis of existing reporting and research • Qualitative analysis of KII results 6. Future Directions 6.1 What SPECIFIC technical areas and research topics merit new or continuing research investment? a. Program and project reporting b. Program management c. Researchers d. Development partners e. Government partners f. Private sector partners g. Academic partners h. Missions (U.S. and other) i. BFS j. Working papers k. Smallholder farmers a. Compilation of reporting data b.–i. KIIs j. Compile all working papers k. FGDs • Reading and analysis of existing reporting and research • Qualitative analysis of KII results 148 ANNEX Q: AGRICULTURAL ADVISER TECHNICAL OVERVIEW OF AMA IL WORK AROUND IMPROVED SEEDS 149 The improved seed varieties (for various characteristics such as pest resistance, drought tolerance, high yield) were developed over the years through global-, regional-, and national-level collaboration between developed/developing country universities, the CGIAR system, and others. Although the impact on increasing production and productivity was significant, the impact on access of the small farmers (who statically cultivate most of the global land) to food, and the impact on poverty reduction or economic growth remains a challenge in most, if not all of the developing countries. When developing new seed varieties and agronomic technologies the researchers often focused on solutions to the challenges of reduced crop production and productivity but without considering: 1) the ability of the farmers to access the seeds and the accompanied technology and inputs; and 2) the risk to the farmers from using new seeds at high costs and uncertain sustainability options. The research results mostly failed to offer investment opportunities for lifting the poor out of poverty, which is a major objective of the USAID’s Feed the Future initiative. Classic examples of such failures are portrayed in the poor impact on poverty from results achieved from several improved critical crop commodities such as maize, sorghum, millet, rice, ground nuts, legumes, vegetables, and cash crops such as oil seeds and cotton in Malawi (Annex 1). 21 This was mainly because of lack of/poor adoption of the small farmers who prefer to recycle their own seeds than venturing into purchasing expensive high yielding variety (HYV) and DT seeds. For example, in order to fulfil the Feed the Future objectives of finding global solutions, the AMA IL researchers (e.g., Laura Paul) focused on the strategic understanding that the DT maize technology “is a weather-contingent technology based on the detection of the response of DT maize to precipitation using high resolution and high frequency weather data.” Hence, AMA IL researchers focused on investigating associated measures that enhance the confidence of the traditionally risk averse dryland farmers and increase the rate of replacement of the obsolete/less productive seed varieties. Some very interesting results emerged. Examples: • • • According to Michael Carter (UC Davis and CIMMYT October 30, 2015) “stress-tolerant seed and insurance technologies at first glance substitute for each other: Both try to stabilize farmer income in the wake of adverse climatic events. But if we look more closely, they may complement each other. The DT seeds are tolerant of drought but are not drought proof. On the other hand, the most probable solution—insurance—is expensive, especially when applied at varied drought events (relatively frequent, moderate drought events).” According to Carter & Giannini (July 2017), use of improved seeds and inputs (e.g., fertilizers) drops remarkably during drought years compared to normal years. For example, households using improved seeds in Mozambique dropped to 21 percent during 2016 drought and only 8 percent used some chemical fertilizers Adoption of the associated measures must be stratified according to the growing cycle (see figure) where the risk of weather changes varies, and the dynamics of droughts (onset during the growing cycle, severity, persistence) vary and in some regions are very difficult to predict, or interpret, by the non-experienced technicians or farmers. This understanding was used by AMA IL and CIMMYT when proposing the Index Insurance Contract for Tanzania (ref. Tanzania Insurance Design v 1.0 161108.pdf. Source Michael Carter) based on the climate-based triggering events that occur in the insurance zone (early season rain deficit, end of season yield shortage, or a total yield shortfall); and was reflected in the development of DT Maize Index 21 Past research results produced seeds with several favorable characterizes but did very little to impact on rural and small farmers’ poverty reduction. 150 Insurance Policy (Box 1). As reported by Laura Paul (UC Davis August 2017) the impact of DT seeds on stabilizing yield is significant during modest droughts conditions. This impact is reduced with severity of droughts until it reaches zero impact (do no better than other seeds) under severe and protracted droughts. • Farmers’ confidence and adoption will depend on their ability to access reliable climate forecasts, possibly through rural banks, extension agents, or directly through trained farmers (information gathered by ET August 2018). Figure 1: Periods of Benefit of Drought Tolerant Maize Source: Laura Paul UC Davis August 2017 Assessment of Research Quality Some details upon which the ET based its assessment of the research quality (of the three projects summarized in Box 1, Source: Desk reviews and field Mission): 1. Research Outreach: varied levels of research outreach—Ghanaian and Tanzanian villages participating in the research are a mix of remote and marginal to near-the-main-road connected villages. 2. Research support: The researchers succeeded in developing an agent-based network through which the producer reaches the farmers through dedicated, trained agents. Successful examples: o o The Lubango Seed value chain in Singida (see below: Case study—Lubango DTM Seed Value Chain); and Faro Seeds developed by CYMMIT, etc. In the project progress report, there is mention of partnership with Meru Agro and Suba Agro technology. 151 3. Research was well designed and implemented with the support of CIMMYT which is active in Tanzania. Also, there was significant support from the local research organizations and the local governments (District, Village). The pairing with CIMMYT was a smart essential move because of the IARC’s outstanding record of developing seed improved gene pools and agronomic practices with proven positive impact on crop farming systems (yield, quality, nutrients composition, tolerance to droughts, short growing seasons, etc.). 4. Farmers made aware of the benefits from improved and insured seeds…however: Based on the interviews held by the ET it was obvious that the farmers understand the benefit from insured seeds and are willing to purchase even at the expense of reducing land area. However, because they most times do not know if seeds will be available, they recycle seeds. Therefore, research parameters need to include accessibility attributes (access to markets, access to associated inputs). 5. Research was focused on one commodity: A further observation by the ET was the fact that the research was focused on one commodity (maize) while the farmers normally diversify crops (sorghum, rice, sunflower, fruits) and sometimes use supplemental irrigation. 6. Varied effectiveness of extension: Despite extension messages use of fertilizers is very low and, except for those who keep livestock, use of manure is also very limited. In addition, in the communities where index insurance was introduced and the price of the DT maize seed was bundled into the seed price (e.g., prices of Lubango seeds), farmers opted to recycle their own seeds. 7. The research did not go far to achieve the planned AMA IL outputs (as in the ToC) in the areas of policy, strategy, outreach, and wide-scale dissemination. The partnership with CIMMYT that has already increased demand for Lubango seeds through creation of partnership with a locally innovative seed company (IFFA) did not lead to breakthrough results such as creation of demand for scaling up or for diversification (other seed companies, other DT seeds than maize). Box 1: DT Maize Index Insurance Policy 1. The contract is between Hollard Insurance and the seed company. The two participating seed companies are Klein Karoo and Phoenix. The two DT varieties of seed for which the insurance is offered are: PRIS 601 offered by Klein Karoo and ZM offered by Phoenix. 2. The insured unit is a 1-kilogram seed packet. 3. The price of the insurance is 20 percent of the retail cost of the seed packet. The retail price of the two varieties for the 2016 planting season are 2.53 USD per kilo for PRIS 601 and 0.89 USD per kilo for ZM 523. The price of insurance for a one kilo pack of PRIS 601 is thus 0.51 USD. The price of insurance for a one kilo pack of ZM 523 is 0.18 USD. 4. The seed company will remit the premium to Hollard by the end of the sales period and will provide a list of the number of seed packets insured in each of the insurance zones in Machete and Nhamatanda districts. 5. If the insurance is triggered (see item #7 below), Hollard will remit the 2.53 USD per kilo of insured PRIS 601 and 0.89 USD per kilo of insured ZM 523. 6. The seed company, in turn, will deliver to insured farmers one kilogram of replacement seed for each insured kilogram of seed at the beginning of the 2017 planting season. (If necessary, the research team can support the delivery of the replacement seeds during the pilot phase.) 7. The insurance is triggered if any of the following three conditions hold: 152 a. Condition 1: Rainfall during the establishment period is less than 90 mm in the contract zone. Appendix A provides details of the calculation of the establishment rainfall index. During the pilot phase, the UC Davis/CIMMYT research team will announce the value of the establishment rainfall index on February 15th via SMS messages sent to insured farmers. b. Condition 2: Predicted end of season average maize yield in the contract zone is less than 65 percent of average. Appendix B provides details of the calculation of the end of season average maize yield index. During the pilot phase, the UC Davis/CIMMYT research team will announce the value of this index on May 1st via SMS messages sent to insured farmers. c. Condition 3: A farmer-requested audit reveals that average maize yields are below the trigger (65 percent of average). Appendix C provides details of audit implementation. During the pilot phase, the UC Davis/CIMMYT research team will pay the cost of the audit. (Source: Michael Carter) Investing in Innovations (III) The Dilemma of Not-So-Impressive Impacts of Agricultural Research on Poverty Reduction22 The Problem Model: Agriculture Development Research results (public goods) do not reach the poor for whom such research results are produced. In other words, these results mostly do not offer investment opportunities for lifting the poor out of poverty. However, these research results are credited for the significant increase in agriculture production and productivity and are credited for putting food on the tables of hundreds of millions of people throughout the developing world. How come? 1. Example for Northern Nigeria: CGIAR Scientists (IITA, ICRISAT, etc.) contributed remarkably to the improvement of sorghum and millet verities (production, production, water use efficiency, disease resistance). The contact farmers made significant contributions to the adoption of the research results… and achieved significant improvement to their livelihoods. But the majority of the poor in the villages became landless, subsistent wage-labor in the newly acquired fields of the emerging wealthy farmers. Source: Ahmed E Sidahmed, IFAD Senior Technical Adviser, 1992-2005, observations from field visits, support to design, and implementation of investment projects targeting the rural poor. 22 Prepared by Dr. Ahmed E Sidahmed for the Food and Agriculture Organization of the United Nations (FAO) Investment Center (TCI) Investment Day, July 2017, Rome, Italy. 153 2. Examples from Malawi Poverty Indices by Household Headship Following Research-Generated Productivity Gains Commodity Male-Headed Female-Headed Poverty Extreme Poverty Poverty Extreme Poverty Before research 36.6 19.4 52.3 28.0 After research on: Maize 33.1 17.0 46.6 24.0 Roots/Tubers 36.5 19.3 52.1 27.9 Groundnuts 36.4 19.3 52.1 27.7 Other legumes 36.2 19.1 51.3 27.4 Cotton 36.4 19.3 52.3 28.0 Vegetables 35.7 18.8 50.4 27.0 Rice 36.5 19.4 52.1 28.0 Sorghum/Millet 36.4 19.3 52,2 27.9 Oilseed 36.6 19.4 52.3 28.0 The results point to some of the limitations encountered when using agricultural research to reduce poverty. Because Malawian smallholders (and those in other sub-Saharan African countries) have small landholdings, depend on off-farm income, and face multiple constraints, many will be hard-pressed to directly benefit from agricultural research. The poverty problems faced by a large proportion of Malawi’s smallholder population require a broader rural research and extension strategy, in combination with policy reforms, and instruments to enhance smallholders’ meager asset bases. Note: In order to more accurately measure the impact of research on smallholders, however, labor and commodity market effects need to be considered in the overall research impact assessment. Source: Alwang, P.B. Siegel/Agricultural Economics 29 (2003) Department of Agricultural and Applied Economics, Virginia Tech, Blacksburg, VA 24061, USA Social Protection Unit, The World Bank, Washington, DC 20433, USA Note: “poverty” refers to the headcount (proportion) of households below the upper poverty line, and “extreme poverty” is the head-count below a lower line.