External Evaluation Team Report on the Feed the Future Innovation Lab for Collaborative Research on Grain Legumes August 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Jeffrey Alwang (Chair), Steven Temple, James Myers, Jessica Fanzo, and Eric Welch. Cover photo: MASFRIJOL includes a mobile educational facility used to educate families about benefits of bean consumption, and how to produce and prepare beans. Photo courtesy of Jeffrey Alwang. External Evaluation Team Report on the Feed the Future Innovation Lab for Collaborative Research on Grain Legumes August 2016 The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS ACRONYMS ............................................................................................................................. i LIST OF TABLES .......................................................................................................................iv LIST OF FIGURES .....................................................................................................................iv EXECUTIVE SUMMARY.............................................................................................................v A. OVERVIEW OF THE PROGRAM AND ITS PROJECTS .............................................................. 1 B. EVALUATION QUESTIONS AND EVIDENCE........................................................................... 3 Research Program ............................................................................................................................3 EQ 1. LIL research outputs...................................................................................................................3 EQ 2. Metrics on quality of research outputs....................................................................................11 EQ 3. Research-related challenges faced by the Legume IL ..............................................................17 EQ 4. Research balance between global and domestic priorities......................................................20 EQ 5. Training and technical capacity-strengthening achievements.................................................23 Research: Lessons learned and recommendations ...........................................................................26 Program Structure and Management .............................................................................................28 EQ 1. Facilitation of communication, coordination, and implementation ........................................28 EQ 2. Effectiveness of Management Organization model.................................................................31 Program structure and management: Lessons learned & recommendations...................................34 Institutional Capacity Collaboration ...............................................................................................35 EQ 1. Partnerships and collaborations ..............................................................................................35 Institutional capacity collaboration: Lessons learned & recommendations .....................................39 Program Future ..............................................................................................................................40 EQ 1. Changes in implementation to better achieve objectives........................................................40 EQ 2. Technical research future.........................................................................................................42 Program future: Lessons learned & recommendations.....................................................................46 C. MANAGEMENT OFFICE RESPONSE TO EXTERNAL EVALUATION..........................................47 APPENDICES ..........................................................................................................................53 Appendix A: Scope of Work ............................................................................................................54 Appendix B: Evaluation Plan...........................................................................................................70 A. FTF Project Evaluation Design .......................................................................................................71 B. Data Collection and Management Plan.........................................................................................97 C. Descriptions of Research Projects...............................................................................................105 Appendix C: Travel Itinerary .........................................................................................................112 Appendix D: List of Materials Reviewed .......................................................................................114 Appendix E: Photos ......................................................................................................................115 i ACRONYMS AGRA Alliance for a Green Revolution in Africa APHIS Animal and Plant Health Inspection Service ARS Agricultural Research Service (of the USDA) BCMV/BCMNV Bean Common Mosaic Virus and Bean Common Mosaic Necrosis Virus BFS Bureau for Food Security (of USAID) BMGF Bill and Melinda Gates Foundation CCRP Collaborative Crop Research Program CGIAR/CG Consultative Group on International Agricultural Research CIAT International Center for Tropical Agriculture CRP Consortium Research Program (of the CGIAR) CRSP Collaborative Research Support Project DEPI Dynamic Environmental Phenotyping Imager DFATD Department of Foreign Affairs, Trade and Development (Canada) EET External Evaluation Team (of the LIL) EQ FtF Evaluation question Feed the Future FY Fiscal year HC Host country IARC International Agricultural Research Center IARM Institute of Agriculture Research of Mozambique ICARDA International Center for Agricultural Research in Dry Areas ICRISAT International Center for Agricultural Research in the Semi￾Arid Tropics ii ICT Information and Communications Technology ICTA Instituto de Ciencia y Tecnologia Agricola (Guatemala) IITA International Institute of Tropical Agriculture IL Innovation laboratory INERA Institut de l’Environment et des Recherches Agricole (Burkina Faso) IP Innovation platform IPM Integrated pest management ISRA Institut Senegalais de Recherches Agricole ISU Iowa State University KSU Kansas State University LIL Feed the Future Innovation Lab for Collaborative Research on Grain Legumes (Legume Innovation Lab) LUANAR Lilongwe University of Agriculture and Natural Resources, Malawi MAS Marker-assisted selection MO Management office MSU Michigan State University NARL National Agricultural Research Laboratories (Uganda) NARS National Agricultural Research System NDSU North Dakota State University NGO Non-Governmental Organization NSS National Seed Service (Haiti) PABRA Pan-African Bean Research Alliance PAGLWCC The 2016 PanAfrican Grain Legume and World Cowpea Conference iii PCCMCA Programa Cooperativo Centroamericano para el Mejoramiento de Cultivos y Animales PCIT Phytosanitary Certificate Issuance & Tracking System PI Principal Investigator QTL Quantitative trait locus (or loci) R4D Research for development RFA Request for Applications SARI Savanna Agricultural Research Institute (Ghana) SAWBO Scientific Animations without Borders SDC Swiss Agency for Development and Cooperation SNP Single Nucleotide Polymorphism SO Strategic Objective SSTP Scaling Seeds and Technologies Partnership SUA Sokoine University of Agriculture, Tanzania TL Tropical Legume Program TMAC Technical Management Advisory Committee (of the LIL) UCR University of California, Riverside UIUC University of Illinois, Urbana-Champagne UNZA University of Zambia UPR University of Puerto Rico US United States USAID United States Agency for International Development USDA United States Department of Agriculture WUSL Washington University of Saint Louis iv LIST OF TABLES Table 1. Legume Innovation Lab project descriptions. ............................................................................4 Table 2. Bibliometric evaluation of the scientific journals in which Legume Innovation Lab scientists published from 2013 – 2015.......................................................................................................14 Table 3. Long-term training table for the Legume Innovation Lab, cumulative through 2015. ....24 Table 4. NGO engagement to extend research technologies to a broader in-country audience. .............................................................................................................................................................................38 LIST OF FIGURES Figure 1. Dry bean yields from 1961 - 2014 in Eastern Africa and US. ..............................................7 Figure 2. Climbing bean greenhouse trials. ............................................................................................ 115 Figure 3. Soil fertility research on beans in a farmer's field. .............................................................. 116 Figure 4. Sampling of bean diseases. ........................................................................................................ 117 v EXECUTIVE SUMMARY The Feed the Future Innovation Lab for Collaborative Research on Grain Legumes, branded as the “Legume Innovation Lab (LIL), began in 1980 as the Bean/Cowpea Collaborative Research Support Program (CRSP) (1980–2007). Following a re-competition of the Management Entity in 2006, USAID awarded the Dry Grain Pulses CRSP to Michigan State University (MSU) for the period FY 2007-2012. The current 4.5 year LIL contact (April 1, 2013 – September 29, 2017) with MSU, an extension of the DGP CRSP, is a Cooperative Agreement- Leader with Associates with an authorization of $ 25,500,000. The program supports ten multi-disciplinary collaborative research and institutional capacity strengthening subcontracted projects working in 13 Feed the Future countries in Africa and Central America and the Caribbean involving scientists at 10 US universities, 3 USDA/ARS research centers, and 23 developing country national agriculture research systems and universities. This evaluation, conducted February-July 2016, follows a USAID-approved evaluation plan. It involved a review of background literature and project documents; a week-long series of interviews with more than 75 project stakeholders at the 2016 Pan-African Grain Legume and World Cowpea Conference (PAGLWCC) in Zambia; site visits to the Management Office (MO) and LIL projects in Guatemala and Uganda; and remote interviews with various stakeholders. Interviewees included all project PIs and selected collaborating project scientists, LIL-supported graduate students, other scientists and policy makers with an interest in grain legume research, host-country partners and government officials, USAID Mission staff, and industry stakeholders. The evaluation focuses on the ten projects comprising the LIL research portfolio, the performance of the MO, and USAID Mission supported Associate Awards managed by the LIL MO. Four targets of interest were identified: achievements in research; program structure and management; institutional capacity and collaborations; and the program future. The main findings of the evaluation are: § The LIL is producing a sufficient quantity of high-quality scientific output that can over time contribute to higher level impacts on the program goals. These outputs take the form of germplasm and variety releases which contribute to productivity increases; information about constraints to increased productivity, barriers to market expansion, consumer preferences and how they affect bean consumption, and the role of legumes in nutritious diets; better-trained legume scientists; and increased capacity for legume research in the US and participating host countries. The mix of research in the LIL is weighted toward genomics and breeding efforts, but social sciences and nutrition research is gaining in prominence. Legume research impacts may be hampered as much by problems with seed systems, poor market development, and consumer perceptions about beans as by genetic limitations of traditional varieties. § Training targets are being met and an appropriate mix of grain legume scientists and technicians are being trained. § US industry and consumers receive tangible but difficult to quantify benefits from the LIL. A chief benefit is the contribution of LIL funding to maintaining a productive grain vi legume research complex; other impacts are also important. § The Management Entity model with a MO, whereby an administrative structure is set up to coordinate ten heterogeneous projects and communicate with external audiences, is largely effective. Some of this effectiveness is due to the stability of the LIL’s MO and an active Technical Management Advisory Committee (TMAC), but, given the complexity of the LIL a management office is probably necessary. While its costs are not negligible, the evaluators conclude that USAID is receiving good value for its money. § The MO has promoted communications across projects, but some steps might be taken to improve this communication. Research efficiency will be gained by sharing findings from area to area. This sharing could be improved through increased and better quality communication from the MO. The evaluation suggests that the MO should take a larger role in communicating the importance of research findings to relevant audiences. § Many stakeholders noted that LIL research is not meeting expectations in terms of reducing a long-standing gap between experiment station yields and national averages. Numerous factors contribute to this yield gap, and LIL research is investigating many of these factors, but policy makers and development assistance programmers are not incorporating research findings into their actions, often because they are unaware of the findings. § The LIL is collaborating with the appropriate partners. These include national agricultural system researchers, regional universities, international agricultural research centers and their organizations, USDA/ARS scientists, other funding agencies and development actors, and non-governmental organizations. The collaboration varies by country and research partner, but this evaluation finds evidence of good collaboration. § The LIL is contributing to USAID Mission goals, but this contribution is uneven. When LIL scientists travel to host-countries, they generally meet with USAID Mission staff, but communications outside these brief and irregular meetings are limited. The MO has a role in efforts to improve communications between the LIL and USAID Missions, but it is important to create incentives for project leaders to communicate more effectively with USAID and other audiences. Recommendations with respect to the program future are: § Maintenance of research balance between genomics/breeding and socio-economic factors inhibiting bean consumption and diffusion of productivity gains. The External Evaluation Team recommends that multi-disciplinarity continues to be integrated into research teams, as appropriate. § Continued research on human nutritional and health consequences of increased bean consumption and obstacles to this consumption. § Invite project proposals for warm season tropically adapted legume species such as pigeon pea, tepary bean, and lablab that are known to offer promise in the harsh environments anticipated for some legume-producing regions with the onset of climate change. § Increased regionalization, including tighter linkages between Central America and African research projects. vii § Clearer communication from USAID and the MO about responsibilities for achieving impact, the role of agricultural research in contributing to impact, and how projects should adjust to facilitate achievement of program goals. § A priority-setting exercise to clearly identify research gaps and obstacles to increased impact. § Continued support for the Management Entity-MO model and the role of the MO in planning and guiding research projects. 1 A. OVERVIEW OF THE PROGRAM AND ITS PROJECTS The Feed the Future Innovation Lab for Collaborative Research on Grain Legumes (Legume Innovation Lab; LIL) began as the Bean/Cowpea Collaborative Research Support Program (CRSP) (1980–2007). Following a recompetition of the Management Entity, USAID awarded the Dry Grain Pulses CRSP (2007-2012) to Michigan State University. The current Legume Innovation Lab, a 4.5-year extension (April 1, 2013 – September 29, 2017) of the previous program, is a Cooperative Agreement- Leader with Associates with an authorization of $25,500,000. The LIL research program carries some research themes from the initial five-year phase of the program (FY 2007-2012), but also began new efforts. The LIL Technical Application (FY 2013- 2017) proposed to refocus and redirect research on priority areas with high promise for significant technological gains and the potential for widespread impacts on legume value chains in developing countries in accord with strategic Feed the Future research priorities. Important structural considerations for the evaluation of the Legume Innovation Lab program for FY 2013-2017 include: a. Partial change in research project structure in which seven of the projects are continuations of prior projects with significant revision and redirection as recommended by the Technical Management Advisory Committee (TMAC) and requested by USAID to conform to its Feed the Future Research Strategy for grain legumes. Three new projects were started as part of the 2013 LIL funding. For the seven continuing projects, the Lead US PIs remained in place but collaborating HC and US PIs changed in numerous cases. b. A focus on priority legume-based cropping systems and countries/regions aligned with Feed the Future (FtF) priorities. c. An effort to broaden the expertise to include a new generation of US and partner country scientists; greater embedded multidisciplinarity in some projects including the addition of social and medical scientists; enhanced engagement of USDA/ARS bean scientists, and improved integration with Consultative Group on International Agricultural Research (CGIAR) partners as a means of achieving broader global impacts. d. Finally, specific innovations for the new LIL program were: § Phenometrics for identifying new physiological and biochemical traits to enhance photosynthesis and assimilate transport to reproductive structures, and to improve the effectiveness of the plant to acquire water and nutrients from the soil under agroecological conditions typical of smallholder, resource-poor farms; § Identification of Single Nucleotide Polymorphism (SNP) markers for Qualitative Trait Loci (QTLs) of important physiological, and disease and pest resistance traits utilizing association mapping; 2 § Introgression of genes for drought tolerance and disease and insect resistance traits from other legume species (e.g., Phaseolus acutifolius, P. coccineus); § Development of dynamic, virtual, interactive internet-based maps of insect pest population changes and crop damage in West Africa; § Development of decision tools for low-literacy farmers to improve Integrated Pest Management (IPM) and sustainable integrated soil fertility management of legume￾based cropping systems in Eastern Africa; § Identification of appropriate governance structures to improve legume value chain functions and to increase market access for smallholder grain legume farmers; and § Increased understanding of the role of grain legumes in diets to improve nutritional status overall, and more specifically, for improved microbiome bacterial ecology and gut health and function (nutrient absorption). Much of the diet and nutrition related work is developing more effective education and food interventions to improve nutrition for infants and children, and women of childbearing age. This evaluation of the LIL by the External Evaluation Team (EET) follows a USAID-approved evaluation plan (see Appendix A for the EET scope of work and Appendix B for the approved evaluation plan). The focus is on the 10 projects comprising the scientific research (Table 1), the performance of the Management Office (MO), and USAID Mission supported Associate Awards. 3 B. EVALUATION QUESTIONS AND EVIDENCE Four targets of interest were identified for this evaluation: achievements in research; program structure and management; institutional capacity and collaborations; and the program future. The evaluation is structured around these areas of interest and specific questions, approved by USAID, and raised during preparation of the evaluation plan. Evaluation questions (EQ) are also listed in Appendix A. Research Program EQ 1. LIL research outputs What contributions have the LIL research activities made to developing research outputs that could contribute to higher level impacts on the program goals of: a. substantively increasing grain legume productivity through sustainable intensification of smallholder farm systems, b. increasing the availability of affordable grain in domestic markets, c. increasing consumption of legumes by the poor, and d. improving nutrition security in developing countries? Findings LIL-supported research should eventually lead to downstream impacts, but the program itself produces outputs. These outputs are new technologies, and information about human behavior, institutions, and constraints to achieving program goals. New technologies are embodied in new forms of equipment or instruments, seed, etc. or disembodied in the sense that new management practices raise productivity without purchase of new inputs1 . Several projects produce information as an output: information on what works and does not work in obtaining the program objectives. Outputs also include human capital formation (formal degree and informal training), capacity building, and institution strengthening, which are covered elsewhere in the report. Research outputs: legume productivity and sustainable intensification: Much of the LIL research portfolio is geared toward producing new technologies, mainly new grain legume varieties; six of the ten projects are part of strategic objective 1 (Advancing the productivity frontier). Five of these are focused on plant breeding and genetic improvements while research of the sixth (SO1.B1) focuses on integrated pest management technologies and overcoming other production constraints. The SO1 projects have produced impressive quantities of new varietal releases (discussed below) and these releases are necessary for the achievement of the first strategic objective. The EET was not able to obtain credible estimates of area planted to these new technologies, and discussions with stakeholders indicated that widespread diffusion is 1 See Kaldor, Nicholas (1957), "A Model of Economic Growth," The Economic Journal 67 (268): 591–624, for a discussion of embodied and disembodied technical change. 4 constrained in all project countries by insufficient development and functioning of sustainable warm-season grain legume2 seed systems. Several non-germplasm technologies are also being developed in the LIL. For example, SO1.A2 is developing new tools to rapidly and efficiently phenotype plants in the field for photosynthesis-related traits that may have application to evaluating biotic and abiotic stresses and yield. Traditional techniques for evaluating abiotic stresses and yield require large-scale field trials repeated in time and space to accurately assess these traits. The new technologies have the promise to produce useful data faster without requiring extensive field work. The project has produced and distributed in Zambia a prototype photosynthetic sensor unit that is currently being tested in the field (it has also been thoroughly tested in the US). This output has the promise to improve breeding by allowing more efficient selection for complex traits, leading to more productive cultivars released at a faster pace. US and Host Country (HC) breeders are just beginning to work with the instruments provided by the SO1.A2 project; as such, the project has generated significant outputs in the form of instrumentation, with data collection just beginning. Table 1. Legume Innovation Lab project descriptions. Project No. Project name Countries Lead Univ. Principal Investigator Associated Institutions S01.A1 Genetic Improvement of Guatemalan Climbing Beans for Efficient Production in the Highlands Guatemala NDSU Dr. Juan Osorio ICTA Guatemala S01.A2 Improving Photosynthesis in Grain Legumes with New Plant Phenotyping Technologies Zambia MSU Dr. David Kramer UNZA, ZARI Zambia UCR S01.A3 Improving Genetic Yield Potential of Andean Beans with Increased Resistances to Drought and Major Foliar Diseases and Enhanced Biological Nitrogen Fixation Uganda, Zambia MSU Dr. James Kelly NaCCRI-Uganda ZARI-Zambia USDA-ARS U. of Nebraska S01.A4 Development and implementation of robust molecular markers and genetic improvement of common and tepary beans to increase grain legume production in Central America and Haiti Guatemala, Haiti, Honduras UPR Dr. James Beaver EAP Zamorano￾Honduras ICTA￾Guatemala NSS-Haiti USDA-ARS NDSU S01.A5 Genetic improvement of cowpea to overcome drought and biotic constraints to grain productivity Burkina Faso, Ghana, Senegal UCR Dr. Philip Roberts INERA-B. Faso SARI-Ghana ISRA, CERRAS￾Senegal USDA-ARS 2 The terms “grain legumes” and “warm season grain legumes” in this context, refer mainly to common bean (Phaseolus vulgaris) and cowpea (Vigna unguiculata), but may include other Phaseolus and Vigna spp. These terms are used interchangeably in this report. 5 S01.B1 IPM-omics: Scalable and sustainable solutions for pest management of insect pests of cowpea in Africa Benin, Burkina Faso, Niger, Ghana UIUC Dr. Barry Pittendrigh IITA-Benin INERA-B. Faso INRAN-Niger CSIR-SARI Ghana INRAB-Benin S02.1 Farmer Decision Making Strategies for Improved Soil Fertility Management in Maize-Bean Production Systems Mozambique Uganda ISU Dr. Robert Mazur Makerere U.- Uganda NARL-Uganda IIAM￾Mozambique U. of Hawaii UIUC S02.2 Enhancing Value-Chain Performance through Improved Understanding of Consumer Behavior and Decision￾Making Malawi, Tanzania, Zambia KSU Dr. Vincent Amanor￾Boadu UNZA Zambia LUANAR Malawi SUA Tanzania S03.1. Legumes, Environmental Enteropathy, the Microbiome and Child Growth in Malawi Malawi WUSL Dr. Mark Manary U. of Malawi S04.1 Impact Assessment of Dry Grain Pulses CRSP Investments in Research, Institutional Capacity Building and Technology Dissemination for Improved Program Efforts All LIL countries MSU Dr. Mywish Maredia Other LIL projects Project SO2.1 is addressing the legume productivity goal by developing tools to diagnose soil fertility and enhance farmer decision making about alternative soil management practices. The output in this case is a disembodied technology (management information) that will potentially complement varietal technologies produced by other LIL projects. The main premise of this research is that alleviating soil-related productivity constraints requires not simply increasing access to fertilizers or use of other soil amendments, but also enhancing capability of farmers to diagnose problems and experiment to overcome them. The project is gathering information on soil characteristics, their geographic distribution, and their relationship to yield on farmer fields as a part of this exercise. No decision tool outputs have yet been produced and a challenge the project will face is making the resulting tools (which are more than one year off) available to farmers. Unfortunately, the project duration will not allow comprehensive testing of these tools against alternatives, so information (an output) on their efficacy will not be forthcoming. An additional information output being produced by this project is evidence from testing of innovation platforms (IP), which have been used widely in Africa for communication, dissemination and scaling up of agricultural innovations3 . This information will be valuable to development practitioners who wish to scale up successful interventions. However, rigorous testing of the IP concept is beyond the scope of this project. Project SO4.1 is examining constraints to increased productivity through the lens of access to 3 See http://r4d.dfid.gov.uk/pdf/outputs/WaterfoodCP/Brief6.pdf for a description of this tool. 6 quality seed and producer behavior. The output of the seed-system research is information on what types of institutional arrangements work in different environments. Information produced from SO4.1 research in Central America is being used by the MASFRIJOL associate award which is relying on community-based seed banks (basically selected farmer fields) to provide high-quality seed. SO4.1 is also involved in examining behavioral constraints to the purchase of improved legume seeds. Their innovative willingness to pay experiments will provide information (the output) about farmer characteristics affecting willingness to pay. These experiments have already been conducted in Central America and are being completed in Tanzania; a cross-country assessment of findings might provide broader lessons for the LIL and other stakeholders. The program focus on productivity enhancement is appropriate, because on-farm productivity gains are a major constraint to achieving development impacts of legumes. However, a majority of project scientists and other stakeholders noted that despite evidence of high quality variety releases and success in addressing biotic and abiotic constraints, downstream impacts have not emerged. This reality represents an opportunity to increase the development impacts of grain legume through steps identified elsewhere in this report. Several interviewees noted that the ‘yield gap’ between results on-station and those in farmer fields has not closed substantially despite relatively productive up-stream research. Figure 1 shows that dry bean yields in East Africa have remained around 700 kg ha-1 for the past 53 years while those in the US increased steadily to about 2,000 kg ha-1 . Stagnant yield may be attributed to declining soil productivity, plantings on less favored land, other biotic constraints, barriers to diffusion of research outputs, under-developed seed systems, and other abiotic factors associated with climate change. Thus, the long-term development impact of the LIL is being constrained by an array of factors, some of which are not addressed by breeding programs.4 The EET heard sentiment that cowpea5 research received less emphasis in the LIL in this funding cycle, with greater emphasis on upstream research. This change in emphasis might pose the risk that cowpea research in West Africa loses its relevance and critical mass. Research needs for cowpeas in Southern Africa are different from West Africa and the breadth of research needed to meet these diverse needs may be threatened. The EET did find that of the ten projects composing the LIL, five have a full or partial focus on cowpea. 4 Detailed analyses and data discussing the relationships between seed systems, variety release and acceptance, for legume species in many African countries, are available in studies conducted by C.G. Rubyogo, and funded under the AGRA-SSTP program. See Rubyogo et al. (2016) Integrated seed systems delivering on the promise: experiences from Tropical Legumes II. In Seven seasons of learning and engaging smallholder farmers in the drought-prone areas of sub-Saharan Africa and South Asia through Tropical Legumes, Monyo ES and Varshney (eds) RK 167-179 ICRISAT, Panthancheru India 5 Cowpea refers to Vigna unguiculata that may be consumed as a dry mature grain, or as a high moisture shell-out (blackeye or crowder pea). 7 Figure 1. Dry bean yields from 1961 - 2014 in Eastern Africa and US. Figure illustrates the “yield gap” between the developed and developing worlds. Dry bean refers to Phaseolus vulgaris L. where the mature grain is consumed. Data from FAOSTAT (http://faostat3.fao.org/). Research outputs: increased availability of legume grains in domestic markets: All LIL projects addressing productivity constraints indirectly contribute to this outcome because increased productivity (lower per-unit cost of production) at the farm level can stimulate market supply leading to increased availability. However, productivity gains may be necessary, but are not sufficient for this outcome; many factors can hinder market supply even with higher productivity at the farm level. Thus, the LIL has invested in value-chain studies that analyze how market-related deficiencies might constrain legume supplies. Projects SO2.1 and SO2.2 are both examining conditions in legume value chains affecting access to and performance of output markets. The outputs of these projects are information about steps, policies and institutional change to overcome market obstacles. Project SO2.1 is analyzing data on markets and prices received by farmers and engaging stakeholders in IPs to help them understand how to improve bean value chains. This participatory research is testing a number of techniques to increase stakeholder awareness and empower them to make better marketing decisions. As such, it is both a development and applied research project and the ultimate output will be information on what works and does not work. To date, the tangible outputs are reports; none of the reported scholarly accomplishments relate directly to legume markets and constraints. Project SO2.2 is directly identifying market-related constraints; one of its stated objectives is to conduct analyses of marketing systems in Malawi, Tanzania and Zambia, with an aim at identifying gaps in the value chains. The project then plans to undertake capacity building to overcome these gaps; thus this project also contains a development component, which should increase its impact in target countries. Project PIs are using secondary data to evaluate marketing constraints; as of this review, this analysis was still under way. The analysis should produce the useful output of enhanced knowledge of dry bean market systems in each country. This output has direct implications for investments in market enhancement which will be necessary to achieve the supply objective. Its output is not likely to be an international public good, except if a comprehensive cross-country synthesis analysis is undertaken. In FY 2015, the y = -0.847x + 709.23 R² = 0.0365 y = 12.699x + 1319.8 R² = 0.78103 0 500 1,000 1,500 2,000 2,500 1961 1963 1965 1967 1969 1971 1973 1975 1977 1979 1981 1983 1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013 Kg Ha-1 Eastern Africa USA 8 project facilitated workshops on entrepreneurial activities and strategic alliances in the dry bean value chain. The project also conducted choice experiments to quantify urban consumer preferences for dry beans. This exercise identified relatively low consumer demand for legumes in Zambia, Malawi and Tanzania as an obstacle to dry bean market development. This information output could be used to inform breeders and enhance the efficiency of breeding research. In Zambia, where SO1.A2 and SO1.A3 projects are also being conducted, dry bean breeders have used the information to help guide their research. The SO2 projects are progressing well, but the EET notes that, unlike embodied technologies, where distribution and diffusion are barriers to achievement of impacts, the impact of information-producing research will depend on the form of the output (i.e. how the information is packaged). If the LIL is to have a long-term impact on increased availability of legumes in markets (also increased consumption of legumes by the poor), it is essential that these outputs be appropriately packaged so that the information becomes available to policy and development programming decision-makers. Research outputs: increased consumption of legumes by the poor: As in in the case of market availability (above), all LIL projects addressing productivity and market constraints indirectly contribute to this outcome. New productivity-enhancing technologies, if adopted by resource poor producers, may increase household-level availability and consumption. More supply to markets will lower warm-season grain legume prices, allowing increased consumption by the poor. Across the LIL, projects are working in geographical areas largely occupied by smallholder resource-poor producers where technology outputs should directly increase household food security and grain legume consumption in diets. Project SO1.A1 reflects a recent adjustment to the LIL research portfolio that will directly contribute to this goal. This project is working on genetic improvement of climbing dry beans varieties (Bolonillo) that fit particularly well into the farming systems of poor producers in the Guatemalan Western Highlands, the target region of Feed the Future. It is noteworthy that little internationally-supported breeding research had been done on Meso-American climbing beans prior to LIL recognizing their yield potential and importance to highland agriculture systems and food security of the region (Figure 2). While the output of this project (new varieties with disease resistance and improved agronomic traits) has been delayed due to unforeseen factors, it is likely to emerge in this project year. Fortunately, the climbing bean varieties will be directly inserted into the dry bean seed production and outreach components of the MASFRIJOL associate award in Guatemala and will quickly have impacts among the poor in the Highlands. Projects working with disembodied, but productivity-enhancing technologies (such as SO2.1) are also working with poor producers. The only project with a direct focus on consumption is SO2.2 which conducted choice experiments to understand consumer preferences for different dry bean attributes. The analysis was not stratified by socioeconomic status so that specific inferences about poor consumers could not be made. As stated above, the main output of this exercise is information and the analysis (first completed in 2015) might be nuanced enough to provide information about preferences of the poor. Research outputs: improving nutrition security: A single and relatively new project, S03.1 (Enhancing 9 nutrition), is focused on providing outputs for this program goal. This research is designed to provide evidence on how legumes could affect infant nutritional status and growth through discrete mechanisms involving microbiome ecology and gut function and health. The project started activities in FY 2014 and appears to be making steady progress toward its objectives. The randomized control infant feeding trials underway in Malawi are well-designed, rigorous and have the potential to provide evidence on how specific foods interact with the bacterial flora in the microbiome and their impacts on reducing inflammation of the gut, nutrient absorption, and child nutritional status and growth. Project outputs will include legume recipes (increased legume consumption), recipe testing for consumer acceptance, and information about the effects of increased consumption on child anthropometry, environmental enteric dysfunction6 and characteristics of the gut microbiome. Other LIL projects address important, indirect pathways that could improve household food security and diets through legume availability, income generation and overcoming obstacles to increased consumption, but do not have direct, explicit pathways aiming to impact nutritional status. An exception to this statement is the MASFRIJOL associate award in the Guatemalan highlands where nutrition education is a major component. MASFRIJOL includes intensive measures to promote dry bean consumption through consumer (particularly women’s) training and education (Cover Photo). The training is intended to move under-five children’s diets from mainly maize consumption toward more diversity and foods with higher nutrient density such as beans. The output from MASFRIJOL is educated consumers. The EET judges this training to be high quality and adequate. Enhanced impact: LIL project impact pathways are complex. As a result of this complexity, the pathway from research outputs to impacts for all projects requires fulfillment of several conditions, most of which are beyond the scope of a pure research project. The LIL made a conscious decision to institutionalize impact assessment through project SO4.1, focused on improving project outcomes and impacts. This SO4.1 PI’s main function is to integrate a culture of impact assessment (and awareness about obstacles to obtaining impacts) in other projects, document impacts and facilitate accountability and learning for the MO. A major output of this project was to work with other LIL projects to create project-specific impact pathways. It was successful in doing so, with all projects except SO3.1 having developed impact pathways. With some exceptions, LIL scientists found the process of developing an impact pathway at project inception to be a good one. Those with a favorable perception thought the exercise helped them visualize how their research fits into development change and to understand constraints to realization of impacts. A minority thought the exercise was not useful and only represented an additional administrative hurdle. Even among those who found the exercise useful, most felt that once the exercise was completed, the impact pathways did not continue to provide insights into actions needed to improve impacts. That is, the impact pathways were a useful descriptive tool, but did not affect subsequent planning. There is not a lot of evidence 6 Environmental enteropathy is a pervasive chronic subclinical inflammatory condition among children that arises when complementary foods are introduced, places them at high risk for stunting, malabsorption, and poor oral vaccine efficacy. 10 that the exercise affected the path of research beyond first-year planning. Project SO1.B1 had research activities in subsequent years that referred to actions identified in the impact pathways, but evidence was not found in other project annual work plans of substantial changes in activities as a result of the impact pathway exercise. SO4.1 produced other outputs related to its goals. Information baselines were created for SO1.A1 and SO1.B1. Measurement at baseline is a valuable input into subsequent analysis of impacts; for these projects, the main utility of the baseline survey is to gain information about conditions faced in their target population. For SO1.B1, the project required information on the suitability of biopesticides in the Benin context. A problem was observed with the long gestation and analysis timelines associated with a high-quality baseline survey. These periods are exacerbated because some research partners (such as technicians at NARS) are not equipped to conduct household surveys. The survey then becomes a separate activity requiring new research partners or extensive training. The information from the baseline survey in Guatemala (relevant for both SO1.A1 and the MASFRIJOL associate award, and possibly SO1.A4) is not yet available. Thus, delays in producing the output affect the operations of the projects. Ways are needed of streamlining the baseline surveys and facilitating quick release of relevant information. One possible means of enhancing the relevance of the baseline surveys is by a careful prioritization of baseline hypotheses and analytics. The research project could interact with the SO4.1 team to identify critical time-sensitive information and the baseline analysis could have be structured with this information in mind. Other outputs of SO4.1 include information about the efficacy of different models of seed systems in Central America, the efficacy of the Scientific Animations without Borders (SAWBO) animations in West Africa, and the effectiveness of cowpea seed-producing Farmer Associations in Burkina Faso. All these outputs are potentially important inputs into achieving program goals, and project PIs need to learn from them. These outputs are steps in understanding how to alleviate obstacles along the impact pathways. Several LIL scientists remain skeptical about the need for impact assessment and this skepticism dampened their enthusiasm for learning from activities such as the creation of project impact pathways. Recognizing that USAID does not consider scientists to be directly responsible for impacts, the EET speculates that skepticism about impact assessment reflects uncertain incentives. Since scientists are not responsible for achieving impacts, and USAID invests more than $800 million annually for agricultural development projects to achieve them, scientists might wonder about the utility of impact assessment or impact promotion. The EET detects a certain cognitive dissidence related to responsibilities of SO4.1 and the other projects. The LIL has invested heavily in a specific impact-promoting project. This project has cross-program responsibilities to build a consciousness about impact assessment among project scientists. In the same context, USAID is sending the message that projects are not on the hook for impacts, so it might be natural for scientists to be confused about their responsibilities. If impacts matter to the LIL, then scientists should be provided the proper messages and incentives. Conclusions The program as a whole is producing adequate outputs, but the EET notes that the bulk of program research is addressing legume productivity, and developing new technologies to 11 address productivity constraints. The research mix of the LIL program reflects a judgment about research priorities. That judgment is: the comparative advantage of dry grain legume agricultural research with respect to achieving the four goals named above is in breeding research and genomics. While this might be true, the EET encountered substantial evidence of broad-based problems in seed systems, input access, technical management advice, access to markets and others. While these areas might best be addressed through development programming (i.e. not through research), the EET feels that additional research in these areas is needed and should be a high priority. In any case, the yield gap persists and the LIL has produced several outputs in addition to germplasm that can be used to address it. The two broad types of outputs in the LIL take remarkably different forms and require different packaging/promotion to contribute to longer-term program goals. Embodied technologies are associated with impact pathways requiring investments in seed systems, diffusion mechanisms and other infrastructure to promote widespread adoption. The output of the research (e.g. the variety) can be directly introduced to projects and other development activities and national agricultural research institutions and other players are well-versed in these processes. Disembodied technologies, once tested in a rigorous fashion, need other types of promotion before we see wide-spread uptake. The information—e.g. what works and doesn’t work in pest management—needs to be made available to users. Broader information outputs such as information on the efficacy of IPs, best practices in seed systems, and constraints to increased warm-season grain legume production should be packaged according to the information-gathering practices of intended recipients. The EET feels that the LIL could improve its packaging of outputs and getting them to the appropriate audience. This theme is more fully developed later in this evaluation report. EQ 2. Metrics on quality of research outputs To what extent did the project generate robust and quality research outputs using disciplinary-appropriate metrics? Are the outputs relevant for a research-for￾development project (e.g., did they generate new breeding lines of relevance to FtF countries)? Were the projects undertaken using the right tools/technologies and were they well executed? Findings The LIL has generally been productive in generating outputs appropriate to its funding level. Outputs have taken the form of improved dry bean and cowpea varieties and germplasm, published research in refereed journals, communication to peers at professional meetings, and outreach to stakeholders through extension materials. Other technological outputs include the SAWBO animated videos project and the photosynthetic sensor unit of SO1.A2. Generation of research outputs: A bibliographic analysis of LIL written and oral outputs revealed 51 papers published in peer refereed journals (Table 2). Most journals where LIL researchers publish are indexed by Google Scholar and Web of Science, although a few are not. Most 12 papers seem directly related to LIL-supported work although some were more tangentially related. It is difficult to draw an absolute line as what is and what is not LIL-supported research, particularly for breeding programs where international objectives and activities are intertwined with domestic ones, or where LIL sponsored graduates students conducted the research. The majority of papers (49) were outputs under SO1 (advancing the productivity frontier). Approximately 20 of these papers were contributions from genetics and plant breeding with a number published in the highest quality journals in the crop sciences (PLoS One, Theoretical and Applied Genetics, BMC Genomics, G3, Plant Genome, and Crop Science). One paper was published in the very high profile journal, Nature Genetics. This paper7 included five US PIs as authors and used LIL-generated germplasm in part for the research (association mapping studies). The paper announced the first whole genome sequence of common bean8 , but went well beyond in examining the genetic architecture of the species and implications for genetics and breeding. The published papers showed a mix of authorship involved in the LIL, with many including US and HC personnel, and graduate students. This authorship mix is indicative of a truly collaborative program. In addition to peer refereed articles, many peer-reviewed papers were published, including six book chapters, and 69 reports, conference papers, presentations, working papers, manuals, etc. Some of these peer-reviewed reports (Annual Report of the Bean Improvement Cooperative9 ) and oral presentations at regional meetings (such as PCCMCA) are important means of distributing results widely and in a timely manner to the legume research community. Graduate student training is well represented with 17 theses defended during the time period considered for this evaluation. Publications on the socio-economics side were sparsely represented, although it is still early in the program to be too judgmental about numbers. The potential quality of those papers appeared to be high. Much of the socio-economics research is focused on the development end of the research for development (R4D) continuum and use of solid and tried-and-true methods will help ensure that the findings could be used to inform policies and programs. Research on obstacles to increased dry bean consumption and constraints in the value chain system (SO2.2), farmer decision making (SO2.1) and the appropriateness of alternative models of seed systems (SO4.1) will provide information on how to increase productivity, broaden consumption and increase nutritional status. As noted above, this information output should be packaged for delivery to appropriate audiences; while refereed publications are clearly desirable, dissemination in formats useful to development practitioners should be explored by the MO. The EET noted the absence of recent soils publications from project SO2.1, whose principal “draw” for farmer involvement is focused on the skilled use of soil amendments. As above, this project’s efforts are relatively recent and directed at applied research where communication to 7 Schmutz et al., 2014, “A reference genome for common bean and genome-wide analysis of dual domestications” Nature Genetics 46:707–713. 8 Common bean (Phaseolus vulgaris) includes both dry and vegetable (fresh, or shell-out and snap) forms of the species. 9 See http://bic.css.msu.edu/Reports.cfm for an archive of annual reports. 13 peers and stakeholders may be achieved more efficiently through means other than refereed journal publications. Relevance of outputs: Most variety releases by LIL researchers are published as peer-refereed papers in Journal of Plant Registrations (Table 2). Other venues for variety release publication may be Annual Report of the Bean Improvement Cooperative and regional journals (such as Journal for Agriculture of the University of Puerto Rico). LIL researchers were highly productive with eight variety or germplasm release notices published in Journal of Plant Registrations during the past two years. An additional dry bean release and five cowpea releases were reported in other venues for an impressive total of 14 releases over the prior two years. Six of the 14 are intended for US markets; the rest are targeted to HCs. Variety development typically takes a decade or more from cross to seed increase, so varieties being released now were initiated under a previous CRSP, and those being initiated now will be finished after the current funding cycle of the LIL has ended. It was not possible to obtain information on seed dissemination and adoption rates of these new varieties from information available to the EET. 14 Table 2. Bibliometric evaluation of the scientific journals in which Legume Innovation Lab scientists published from 2013 – 2015. Web of Science Google Scholar Journal Number of papers listed Total citations (journal) Journal Impact Factor h5- indexy h5- medianx Archives of Virology 1 8,773 2.390 38 56 ARPN J. Agric. and Biol. Science 1 --z -- -- -- Arthropod-Plant Interactions 1 469 1.462 15 19 BioControl 1 1,615 1.693 BMC Genomics 1 25,105 3.986 80 97 Crop Protection 1 5,156 1.493 30 41 Crop Science 6 15,246 1.575 34 50 Entomology Society of Canada Bulletin 1 -- -- -- -- Food and Bioprocess Technology 1 3,406 2.691 47 59 Food Policy 1 2,551 1.799 42 68 Frontiers in Plant Genetics and Genomics 1 -- -- -- -- Frontiers in Plant Science 2 4,004 3.948 36 43 G3: Genes/ Genomes/ Genetics 1 1,674 3.198 30 40 International J. Advanced Research 1 -- -- -- -- Intl. J. Edu. Devel. Using Info. Comm. Tech. 1 -- -- 15 19 Intl J. Voluntary and Nonprofit Org. 1 -- -- 23 30 J. Agric. Food Info. 1 -- -- -- -- Journal of Agricultural Economics 1 971 1.278 24 40 J. Agric. Univ. of Puerto Rico 2 -- -- -- -- Journal of Applied Biosciences 2 -- -- 5 6 Journal of Plant Registrations 8 295 0.640 11 14 J. Amer. Soc. Hort. Sci. 1 5,001 1.276 18 21 Molecular Breeding 3 3,796 2.246 36 47 Nature Genetics 1 85,481 29.352 182 268 Plant Biotechnology Reports 1 354 1.188 17 27 Plant Genome 3 578 3.933 22 69 PLoS ONE 3 332,716 3.234 161 210 Tanzania Journal of Agricultural Science 1 -- -- -- -- Theoretical and Applied Genetics 2 20,524 3.790 51 64 Total number of papers 51 Total number of journals 29 z Not indexed by Web of Science and/or Google Scholar. y h-index is the largest number h such that at least h articles in that publication were cited at least h times each during the last five years. x h-median is the median of the citation counts that meets the criterion of the h-index. 15 Three programs that are still too new to have substantial outputs, but have promise are SO2.1 with its innovation platforms, SO1.A2 which is developing photosynthetic sensing instrumentation and SO3.1 addressing human nutrition effects of beans (detailed above B. I. 1.). SO2.1 is still in a data-gathering phase, primarily on soils and plant nutrition. An objective of this project by 2017 is to develop effective methodologies for media and information dissemination about their findings. SO1.A2 has two main technological platforms: Dynamic Environmental Phenotyping Imager (DEPI) and PhotosynQ. The former is a stationary growth chamber facility rigged with sensors to assess various photosynthetic parameters. Environmental conditions can be simulated in these chambers and plant responses can be monitored. Because of substantial infrastructural requirements, this system will be most accessible to researchers in developed countries. PhotosynQ is a platform (www.photosynq.org) to bring sophisticated phenotyping and analytical tools to farmers, researchers, extension agents and entrepreneurs. The PhotosynQ platform includes a plant sensor, called the MultispeQ, which makes a series of useful plant and environmental measurements, yet is inexpensive, easy to use, wirelessly connected, and easily transportable. MultispeQ provides measurements of environmental conditions (light intensity and quality, temperature, humidity, CO2 levels, time and position), leaf chlorophyll and other pigments, soil and leaf respiration and assimilation, and a series of useful plant phenotypes, including both standard and novel probes of photosynthetic performance. The PhotosynQ platform has been tested in Malawi through a McKnight Foundation project. An updated model of MulispeQ was displayed at the PAGLWCC in Zambia and units competitively awarded to grain legume researchers in Kenya, Botswana and Uganda to explore new applications in grain legume research. Units have been tested in the US and are supposed to be generally available in the US beginning in July 2016. The SAWBO application was developed by SO1.B1 in response to a need for low-cost information and communications (ICT)-based information transfer in diverse languages around the world. The effectiveness of ICT-based messages and the SAWBO animations has been documented for many outcomes but there is still an open question about the specific types of information that can be transferred through such mechanisms. While a study by SO4.1 indicated that for some outcomes, SAWBO is more effective than traditional extension methods, but did not examine the effectiveness of SAWBO in communicating highly technical messages. SO1.B1 plans to test this application to see if it is successful in inducing behavior change among low-education farmers; preliminary evidence suggests reason for optimism. Appropriate tools/technologies and execution: The common bean breeding and genetics community has traditionally been an early adopter of new research technologies. The main limitation to adoption has been the high cost of new genomics research technologies; more research dollars have generally been available to the major commodities compared to “minor” crops such as common bean and other grain legumes. In recent years, the bean community has been able to leverage funds from multiple federal funding sources to sequence the common bean genome and develop additional tools for marker assisted selection, as well as generating new knowledge about various traits in common bean. Several papers listed in Table 2 are outputs from this work. These projects have also generated populations useful for answering additional research questions including several that have been posed within LIL-funded projects. These new 16 technologies help LIL researchers remain on the cutting edge of research, and to produce new varieties in a timely manner. The cowpea research community has fewer members, but it too has effectively acquired and uses genomics technologies. The UC-Riverside program has been instrumental in making this happen. Some scientists expressed the opinion that the LIL emphasizes genomics and other advanced breeding techniques at the expense of field-based plant breeding. Interviewees noted potential for a disconnect between genomics research, much of which is conducted in US laboratories, and needs of NARS scientists, who are mainly engaged in varietal development. This potential, however, did not seem to be evident in EET field visits. Site visits confirmed that NARS breeders are capable of using marker-assisted selection (MAS) tools. For genetic traits that present difficulty in uniform field testing conditions the MAS techniques offer certainty and time-saving virtues. There is some uncertainty if those advantages will hold true for more complex genomics tools (association mapping) and genetic traits being studied today. None of the scientists and other stakeholders the EET interviewed thought that more upstream research is needed at the expense of downstream work. Most affirm the importance of upstream efforts, but thought that complementary downstream interventions would be most helpful in generating long-term impacts10. The yield gap is only partially due to genetic factors. The IPM work embodied in SO1.B1 also appears to be progressive and innovative, and has already had positive outcomes. The project is using a variety of classical biological control (release of parasitic insects to control cowpea pests) and combining the gradual release of the bio-controls with community education. This work is as advanced in IPM technologies and practices as many of those used in the sister IPM IL. PhotosynQ (SO1.A2) is also very cutting edge, and its application to common bean sets this crop well ahead of others in terms of developing applications. On the social science side, SAWBO (SO1.B1) represents a new approach to information dissemination. The IP concept being utilized by SO2.1 has been around for a number of years with application in Eastern Africa first occurring in about 2005 with the Lake Kivu Pilot Learning Site project11. IPs present a convenient way to organize research across the value chain, and their application in Uganda and Mozambique is certainly appropriate. SO4.1 (Impact assessment) engages with contemporary tools appropriate to the discipline. 10 Here is an area where it is important to separate research needs from development programming needs. The LIL physical scientists recognize that actions (i.e. development programming) are needed to help ensure that breeding efforts lead to impacts and close the yield gap. There is less recognition by these scientists of specific research needs at the downstream end of the impact pathways. If the LIL is not on the hook for development impacts there is a remaining open question about whether research is needed on these downstream portions. During a general session among project participants held in Zambia, much of the discussion revolved around actions needed to close the yield gap, not questions about what actions would be most effective. The latter is a research question. 11Adekunle, A, AO Fatunbi, R Buruchara and S Nyamwaro (2013). Integrated Agricultural Research for Development: from Concept to Practice. Forum for Agricultural Research in Africa (FARA). 17 Conclusions Overall, the LIL has been a productive research program particularly with regard to the more upstream outputs. Mature LIL projects have a steady rate of output, and the newer programs seem poised to contribute in the next couple of years. What stands out in particular is the number of students trained in a relatively short period of time as demonstrated by the number of theses. The rate of variety release is also impressive. The research balance of the LIL has led to a mix of research techniques that are cutting edge and other that are more tried and true. The research technologies being used are appropriate for the problems and the settings. EQ 3. Research-related challenges faced by the Legume IL What research-related challenges has the Legume IL faced during research design and implementation? In what ways have those challenges been addressed? What opportunities are there to improve research outputs moving forward? Findings Identified constraints to conducting research in the LIL involve policy and regulations, institutional infrastructure, human resources and environmental variability. Infrastructure and human resources can be anticipated and addressed by careful planning whereas the unpredictability of the environment in its effect on field work cannot always be anticipated. Likewise, regulatory policies are usually beyond the control of the LIL. Research related challenges and how addressed: Through many years of experience, the LIL has evolved a structure that deals well with policy and regulatory constraints, and the EET saw few examples of where regulatory issues delayed research. SO1.A4 noted a delay in transfer of germplasm from Guatemala to the US because of phytosanitary requirements, but this example is not unique to the LIL since any individual bringing germplasm into the US must deal with the same regulatory hurdles. With the current system in place (Phytosanitary Certificate Issuance & Tracking System—PCIT--administered through USDA-Animal Plant and Health Inspection Service-APHIS), there is a substantial learning curve and requirements such as a face-to-face interview with USDA personnel will allow a US researcher to obtain import permits and phytosanitary certificates. While requiring extra effort, experience and careful planning with sufficient lead time can minimize the delay, and the MO can share information to overcome such constraints. More difficult is where changes in a country’s policies present new barriers. An example of this is that SO3.1 encountered delays in importing a vehicle into Malawi for field work because of new import restrictions and taxes implemented on vehicles. SO4.1 is particularly dependent on cooperative relationships with other projects and faces challenges in contracting and fund transfers to host countries for cooperative research. Nearly every institution can identify infrastructure needs. However, few examples of infrastructure constraints that slowed or delayed LIL research were identified in the evaluation process. The relatively new SO1.A1 project experienced delays during the first year in implementing some activities due to problems with the storage of germplasm at the Instituto de Ciencia y Tecnolgia Agricola (ICTA)–in Chimaltenango, Guatemala. Because of less than optimum seed storage facilities and seed that was in poor condition, the collection had to be 18 grown out to multiply and produce better quality seed. Long-term seed storage is a challenge in many developing countries and represents a case where the capacity building grants (discussed below) can help. The LIL has also overcome capacity constraints by assigning research tasks according to comparative advantage (e.g. molecular technology mainly done in the US). Quite often it is a HC graduate student receiving training in the newest technologies and conducting research that is published jointly with US and HC authors (see also discussion of authorship under B. I. 2.). One concern voiced by a small minority of interviewees was that a HC graduate student trained in the latest technologies may lack the infrastructural support upon returning to the HC to conduct such research. Students who return with training in the newest technologies may lack the skill to do their job using traditional approaches. The EET saw no evidence of this happening; returning HC personnel were competent in their jobs, and in some cases had access to the technologies in which they were trained. US PIs are aware of these possibilities and technical training at the graduate level is complemented with instruction and practice in more traditional breeding approaches. Limited human capacity in developing countries is an ongoing constraint to conducting research. The LIL has emphasized training of HC scientists and this training probably represents its most tangible output. The LIL has also faced challenges in supervision of in-country research. An example is the SO2.1 project Mozambique where the research site (Gurue) is so far from the capital city that regular supervision is impossible. This distance may have compromised the research project as field activities were conducted without the input of leading scientists (e.g. PIs). SO2.2 noted that additional training of the student recruited to conduct research in Zambia was required because the methodology used was new and relatively unknown. The training was provided by the US Principal Investigator (PI) and research proceeded, but the process did delay data acquisition. SO1.A3 delayed collection of disease samples because collectors needed training in preservation techniques. Again training was provided and research proceeded. The unpredictability of the environment is the constraint that most impacts field research, and is particularly important in developing countries. Climate change is leading to more extreme and unpredictable weather fluctuations, further impacting field research. The best way to deal with the random nature of the environment is redundancy, where experiments are repeated at several locations and/or seasons. This approach requires additional resources, further constraining what an individual project can do. The EET encountered multiple examples of environmental constraints. Several projects indicated that research was delayed or modified because environmental conditions for a trial were not conducive for the problem being studied. For example, SO1.A3 had high rainfall interfere in Uganda with their intended drought evaluations. Zambian scientists noted difficulty in screening for anthracnose because of dry weather conditions not conducive to the disease. SO1.A4 indicated that drought in two successive seasons reduced yields thereby reducing seed supplies for breeding and seed increases necessary for cultivar dissemination. SO1.A5, working in Senegal, had difficulties evaluating an aphid resistant panel two years in a row because of unfavorable weather. In aphid screening in California, uneven infestation in plots was also observed. Other technical problems encountered by projects included insufficient seed supplies, difficulty in getting wild accessions to germinate and problems in synchronizing crossing or evaluating in 19 the field with day-length sensitive materials. In the tropics and subtropics, projects face a myriad of challenges that often extend beyond what researchers face in temperate production regions. SO2.1 provides an example of this where the project is focused on soil and fertility factors, but diseases and pests can substantially alter experimental results. The EET observed an increase in productivity in Ugandan farmer fields in plots which had fertilizer inputs over the control plots with no added fertility (Figure 3). However, even in the most productive plots, evidence was found of diseases such as common bacterial blight, halo blight, common bean rust, anthracnose, angular leafspot, and floury leafspot (Figure 4). Given the right environmental conditions, these problems would affect productivity. EET observations provided a snapshot in time at flowering and early pod set, but, based on a collective 60+ years of experience working with common beans in the tropics, it is the EET’s judgment that disease progression was likely to be substantial and that a disease management protocol from a plant pathologist would be beneficial to the research. Opportunities to improve research outputs: LIL has encouraged efforts to overcome constraints to widespread adoption of improved legume technologies. Examples include the SAWBO efforts, the SO4.1 seed system research, and investigation of consumer preferences and market constraints to wider adoption of improved legume varieties (SO2.2). Research findings from these projects should be shared with other projects to facilitate program-wide measures to encourage adoption and increased legume consumption. All interviewees stated that the institutional capacity strengthening funds made available by the MO are effective at overcoming research constraints. Scientists expressed appreciation of the transparency of allocation of capacity building funds, and no-one claimed to have been overlooked. The capacity building funds were not evenly distributed; four projects received more than $90,000 during FY2014 and FY2015, four received less than $40,000 and one (SO1.A2) received none. In terms of overcoming environmental constraints, replication of the field testing programs in time and space has been the best means of mitigating variability (weather, disease/insect pressures) and ensuring steady progress. This replication typically leads to increased research costs that must be budgeted in advance. New technologies being employed by the LIL have the potential to increase efficiency in breeding. Marker assisted selection for disease resistance and other relatively simply inherited traits where phenotyping in the field is labor intensive is one example. A second example is represented by the PhotosynQ instrumentation to generate useful data that may assist in breeding for tolerance to abiotic stresses and increased yield. These technologies, however, have their own limitations. MAS, for example, allows selection of markers linked to traits of interest in the lab, but phenotypic evaluation of those traits in the field is still required to ensure that the trait and marker have not separated through genetic recombination. Likewise, with PhotosynQ, traits associated with yield can be measured and selected, but yield per se must be evaluated under field conditions to ensure that indirect selection has actually led to gains in yield. Conclusions While the LIL projects have experienced some research challenges, none of the challenges are severe or atypical of similar international collaborative agricultural research efforts. Challenges 20 have been overcome through short-term adjustments to annual work plans based on project learning. The MO has helped address broader challenges (e.g. by adjusting the program portfolio and by making the “institutional capacity strengthening funding” available). EQ 4. Research balance between global and domestic priorities Given the dual benefit mandate of Title XII authorized programs, such as the Legume IL, is there a sufficient balance between research efforts directed towards priorities of the domestic bean/cowpea stakeholder community and the bean/cowpea research priorities required to advance global food security goals, particularly in FTF countries? Findings While LIL research mainly focuses on production constraints faced by developing country producers, the EET identified important benefit flows to US stakeholders. These benefits result from factors including leveraged support for high-quality research and training at US universities, germplasm access, research-related spillovers, and increased access to international markets. US industry stakeholders expressed appreciation for LIL support for warm-season grain legume research and the international focus of the research is understood. One industry stakeholder indicated substantive engagement with LIL scientists and noted that sharing of research results in better-informed business practices. LIL funding often leverages domestic funding and the combination helps maintain interest of high-quality US scientists in warm-season grain legume research. Industry representatives recognize the difficulty of maintaining an adequate cohort of US scientists without the complementary funding provided by USAID through the LIL or similar mechanisms. The representatives did not express any degree of disapproval that US scientists are engaged in international-focused research and instead spoke of how the LIL facilitates better warm-season grain legume research. In particular, US scientists are releasing promising varieties, many of which were developed through activities funded by the LIL (see above). This appreciation is particularly apparent for basic laboratory efforts in genomics and breeding. Interviewees identified two research-related benefits from the LIL. First, US scientists have access to and knowledge of a wider pool of genetic material as a result of the LIL. While most of this material would be available without the LIL, the LIL reduces the costs of obtaining it and facilitates personalized information flow about traits and genetic sources. Second, the spatial variability of the LIL network facilitates efficient on-station research. Research can be conducted more rapidly by exploiting geographical differences in timing of growing seasons and worldwide differences in biotic and abiotic stresses. Further, multi-country testing of particular nurseries increases the probability that timely results will be obtained, even in the face of one lost nursery or the absence of required test conditions at one or more sites. Access to the international grain legume research network (via LIL and the CGIAR centers) thus lowers the cost (in money and time) of conducting research on US problems only in the US. Industry stakeholders identified an important market-development benefit of the LIL. As approximately 40 % of US dry bean production is destined for overseas markets, export 21 markets are important to the domestic industry. While LIL partner countries are not among the largest recipients of total US dry-bean exports (Mexico, Canada, the UK, Angola and the Dominican Republic are, in order, the largest importers), Haiti absorbs about 8.2% of US pinto bean exports (the 4th largest importer) and Guatemala and Haiti, the 3rd and 4th largest importers of US black bean exports respectively, absorb 4.5 and 2.5% of the global value. As an example of the market-development potential of LIL research and engagement, Angolan imports of US dry-beans were close to zero as late as 2000. Since then, the country has converted into a major importer of US beans. Angola was an active participant as late as 2012 in the Dry Grain Pulses CRSP, and this increase in imports is at least partly due to CRSP engagement with the Angolan bean community. As industry stakeholders note, market development is a tricky endeavor, but they clearly appreciate the networking opportunities fostered by the LIL. US handlers and shippers are proud of their product quality (compared to international competitors), and that advantage is displayed and discussed in LIL circles. Industry stakeholders, university administration, and LIL scientists noted that LIL funding enables high-level graduate training that benefits the academic community as a whole. Spillovers to broader society are obvious. LIL-supported scientists voice a commitment to graduate training and appreciate the LIL support for graduate training. Scientists noted that the institutional linkage between LIL research and graduate training stimulated their own interest in the LIL. International research also promotes recruitment of high-quality students, both US and from HCs. The IL-graduate student linkage was viewed favorably by all scientists who expressed an opinion. Beyond benefits to the university, trained professionals benefit countries where graduates eventually reside. While the majority of LIL-funded students are from and return to developing countries, eight out of 73 graduate students being trained by the LIL are from the US. It is impossible to know how many of these graduate students benefit US stakeholders over the course of their professional lives. LIL-supported research also produces, or has the potential to produce, direct domestic benefits. Research on dietary diversity and nutrition-related consequences of increased legume consumption, examined in project SO3.1, has important implications for the domestic industry, which may benefit from expanded markets, and domestic consumers. While the research is in early stages and mainly in low income settings, there is the potential for impacts in the US12. The interviews with industry indicated that research on consumer uptake and preferences for warm-season grain legumes is important to better understand the needs of their customers. Research on cooking times and color retention of dry bean varieties conducted as a part of SO1.A3 is also of interest to the US dry bean industry, which seeks ways to market and expand US demand. Some of the early work of LIL at Michigan State on color retention provided important technical data in improving consumer acceptance (i.e. black beans). Industry 12Industry was particularly appreciative of USAID funding of nutrition research because if positive findings come from this work, industry can use this information to promote dry beans without conflict of interest becoming a major issue. This was given as a reason why particular companies do not directly fund dry bean research. 22 representatives recognize the potential benefits of shorter cooking time and the emergence of “convenience” bean foods. The SO1.A2 project, with LIL funding (highly leveraged funds) is creating low-cost instruments for measuring different indicators of plant health and growth. These tools, being tested in Zambia, have potential for wide-spread commercial application in the US. With ongoing climate-related challenges, many US growers are interested in low-cost applications for water and fertilizer management that will allow them to retain yields and crop quality while improving their competitive edge. Information on these things is being produced by the LIL. Planning and priorities for the SO1.A2 project would seem to benefit from the experiences and challenges faced by the SO2.1 project in Uganda. While there is little quantitative evidence of realized domestic benefits, the SAWBO suite of animations has potential for widespread use in the US. SAWBO was developed by SO1.B1 in response to a need for low-cost information and communications (ICT)-based information transfer to overcome weak public agricultural extension systems in West Africa. The SAWBO application was conceived as a means of diffusing IPM information in West Africa, but has been extended to inform asthmatic patients in the US about proper use of metered dose inhalers. Evidence of the effectiveness of the animations among West African farmers was obtained through a randomized control trial conducted with the assistance of the SO4.1 project. The effectiveness of ICT-based reminders in developed countries has been documented for objectives such as regular use of medications, smoking cessation, and savings13. However, one of the obstacles to wider use of text messaging is its limited capacity to convey information. SAWBO presents an alternative with widespread potential applications. Conclusions The EET finds substantial, although difficult to quantify, evidence of domestic benefits from LIL research. No stakeholder with whom the EET had conversations suggested that balance was an issue. Part of the explanation for these benefits is the complementarity between US- and HC￾focused research. Research tools are somewhat universal and even if biotic or abiotic constraints to domestic and international producers are different, spillovers are natural. As scientists in the US focus on addressing constraints in developing countries, skills and insights are developed that have benefits to domestic stakeholders. A second part of the explanation is the density of the warm season grain legume research networks in the US. The major players are well-known to each other and the LIL provides important interdisciplinary networking opportunities (breeders with pest specialists, physiologists with breeders/agronomists) with synergistic effects across the network. A third 13 Larochelle, C., J. Alwang and E. Travis (2016) “Did you really get the message? Using text reminders to stimulate adoption of agricultural technologies,” paper accepted for presentation at the 2016 Agricultural and Economics Association annual meeting, Boston, MA, August 3. This paper provides a comprehensive overview of this literature. 23 component of the explanation is an intellectually engaged MO which helps, through its own networking and communications efforts, insure that research is structured in a way to foster domestic benefits. EQ 5. Training and technical capacity-strengthening achievements Has the Legume IL met its academic training and technical capacity strengthening targets? Are the appropriate type and number of people being targeted for the right kind of training? In what ways has the program identified and addressed academic and technical capacity needs? In what ways could this be improved? Findings Information on degree training programs is presented in Table 3. These and short-term trainings developed during the current LIL cycle are deemed by most people interviewed by the EET (students, PIs, collaborators, and stakeholders) to be adequate. LIL training targets are being met. Training is strongly encouraged in the program and the MO has taken steps to expand access to and resources for training. PABRA stakeholders mentioned the contribution of LIL in training for important NARS positions. The EET found examples where talented HC legume scientists, trained for advanced degrees during earlier CRSPs, now hold important leadership positions in national agricultural institutions. There is additional evidence of CRSP￾trained students filling positions in regional programs and within the CGIAR. During the PAGLWCC in Zambia, the EET observed interaction and a strong sense of “community” among NARS scientists currently studying in US and regional universities with the support of the LIL. And graduates of earlier CRSPs were identified who are currently serving as NARS collaborators on LIL projects. Type and number of people being trained: The training numbers in the LIL are impressive and the EET found consistency in the disciplinary mix of student training with project objectives. The interdisciplinary strategies employed in several projects are appropriate given the objectives. The depth and breadth of training is extensive, if uneven in some disciplines and some geographic areas. For example, the EET site visit to SO 2.1 (Uganda), where the project uses soil amendment practices to increase dry bean yields and “scale-up” interest in the economic potential of planting a dry bean seed crop, identified a shortage of soil science and plant pathology expertise at the field level. Pathology expertise is needed due to the importance of seed quality and disease transmission by infected seeds. The absence of well-trained seed technologists resonated with multiple PI and HC collaborator comments during the group discussion held at the Zambia conference. These discussions and those with individuals highlighted the importance of quality seed production and distribution as a limiting factor in getting promising new germplasm out to farmers, and stakeholders identified the lack of technicians as a constraint to this process. It is common for a US PI specialized in soil science, breeding or other disciplines to train graduate students in their respective discipline rather than seed systems or seed pathology. In a few cases, a gap was noted between more “upstream” training NARS scientists receive at US universities compared to the broader role they play after returning to their own national program. The EET did not hear this often, but US PIs might be made more aware of the unique needs of HC trainees. The MO has a role 24 in communicating these needs to PIs. Table 3. Long-term training table for the Legume Innovation Lab, cumulative through 2015. LIL Long term training (2015) Degree: PhD MS BS Total Men 12 29 13 54 Women 14 18 14 46 Total 26 47 27 100 Location of University US Africa Latin America European Men 17 26 11 0 Women 12 28 5 1 Total 29 54 16 1 Nationality African Central America US Total Men 35 14 5 54 Women 35 8 3 46 Total 70 22 8 100 Funding from LIL Full Partial Men 11 43 Women 10 36 Total 21 79 LIL training of women has been outstanding. For short-term trainings, 3,215 men and 3,042 women have been trained in Central America and sub-Saharan Africa, a near perfect balance. As for long-term training, the LIL has fully or partially funded 46 female students and 54 male students in university degree programs. Some interesting examples of training of women were done for the Malawi project in which six female nurses were trained to improve their research skills in nutrition. Another example is the SO1.B1 project which targeted village women for short-term training and subsequent employment based on the finding that women are better able than men to prepare IPM packets. The EET concludes that the LIL training has a good gender balance. Identification of academic and technical capacity needs and mechanisms for its improvement: Individual projects all use the annual planning process and consultations with HC partners to identify training needs. The EET review encountered considerable flexibility on the part of the projects and the LIL program in adjusting training needs. Although resources for training under the LIL are relatively generous, and made more so by the advent of the Legumes Scholar program, training resources are, by definition, scarce and their use must be prioritized. The EET found no evidence that this prioritization was not correctly undertaken and has no suggestions for improved identification of training needs. In some countries, projects might encounter the 25 problem of lack of trained expertise in certain disciplines (e.g. plant pathology in Uganda); the projects themselves need to identify such gaps and prioritize the training program. The EET found evidence of need for increased training of technicians, and the LIL has responded by partnering with CG/PABRA and with regional institutions like EAP-Zamorano, Makerere University, and Sokoine University to create short-courses on important themes. These regional institutions have seen their technical capacity increase markedly and, with LIL support, are now full partners in regional training efforts. The EET also heard examples of opportunities for graduate thesis research training for future NARS scientists at CG centers (especially when supported by Borlaug funding14). These opportunities are particularly relevant in disciplines where the tropical climate provides conditions relevant for preparation in applied field research and extension. Both the International Institute of Tropical Agriculture (IITA) and the International Center for Tropical Agriculture (CIAT) have hosted LIL MS and PhD thesis research candidates. Anecdotally, there is concern in the international legume community and the LIL about continued institutional support for quality dry bean research and extension education at EAP￾Zamorano in Honduras. The incumbent faculty member has played a key role in Central American dry bean research and extension for decades, and any loss of capacity will be felt throughout the region. The LIL, partnered with the Feed the Future Innovation Lab for Collaborative Research on Peanut Productivity & Mycotoxin Control to establish the Legume Scholars Program. From an initial pool of 248 applications, five were awarded to outstanding students. Individuals were placed in programs at US institutions with field research conducted at CG centers. The final five were chosen on a basis of commodity diversity and gender balance. Conclusions Overall, the EET finds that training targets are being met. As noted above, strengthening of regional institutions has opened even greater opportunities for training at different levels. The LIL is exemplary in its gender balance of training at every level. Gaps in seed technology training were identified by PIs and collaborators and these gaps were evident in site visits. Both short term and MS training are justified to deal with this critical need. There is also a need for additional non-degree technician training. Institutional configurations for agricultural technology transfer vary among HCs, but as outputs of LIL projects are used in agricultural development projects we anticipate that additional technicians will be needed to fill gaps in seed systems and other parts of the value chain. 14 The US Borlaug Fellows in Global Food Security, funded by USAID and administered by Purdue University, supports graduate students interested in developing a component of their graduate research in a developing country setting and in collaboration with a mentor from an International Agricultural Research Center (IARC), or a qualifying National Agricultural Research System (NARS) unit. 26 Research: Lessons learned and recommendations Lessons learned The LIL research program covers broad topics and has research sites in diverse countries. The LIL has attracted a strong and diverse group of US and HC scientists. Research under the program ranges from upstream cutting edge genomics to applied research on consumer preferences and measures to improve access to legume seed and reduce post-harvest losses. Recent changes to the research portfolio have broadened the focus to include high-potential (but also high-risk) avenues of inquiry such as plant photosynthesis and effects of legume consumption on child nutritional status. This breadth is exactly what a research program should aspire to—a portfolio of relatively low-risk activities mixed with a few high-return “blue sky” research activities. The portfolio is enhanced by a mix of activities seeking to address two issues of growing importance to the legume community: the persistence of a large yield gap and the slow growth in consumption of high-quality legume-based products. The program has adequate rates of publications and other measures of output, with longer￾standing projects, as expected, having more outputs compared to the newer ones. The program has recruited a good cohort of motivated scientists. Collaboration is also apparent in the large number of joint publications. US domestic benefits from the LIL research program are an important part of meeting Title XII goals. These benefits, although difficult to quantify, are substantial and were clearly identified during the EET evaluation. Degree training is progressing according to plans, with most of the advanced degree training occurring at US universities. The ability to work closely with graduate students is a clear benefit to PIs and forms an important motivation for participation in the LIL. Non-degree training is widespread, with increased movement toward regionalization of this training. A clear gender balance was noted in degree and non-degree training. Recommendations The EET sees a clear need for tighter integration of upstream breeding and genomics research with downstream research on factors further down the impact pathway. The LIL, as it is currently structured, supports research on the two ends of the R4D continuum, but these efforts are conducted in separate projects. This separation hampers cross-project learning. More integration could be achieved by inviting projects to produce joint work plans when they work in the same countries, or by designing RFAs with a stated preference that multiple disciplines be represented within a project as appropriate. The EET recommends more regionalization of research and a stronger integration between research and projects in Africa with those in the Western Hemisphere. This has occurred in previous legume-focused CRSPs for germplasm evaluation, exchange, and some breeding. This could be further extended (in other disciplines, in technician-level capacity building, and in shared experiences emerging from technology transfer activities). Some lessons, however, do not directly transfer across regions. Evidence from seed systems research in Africa, for 27 example, shows that the model of community seed systems employed in Central America may not be appropriate in the African context15. Lessons from comparisons across contexts are important for development planning. The EET recommends that when household surveys are conducted (e.g. baseline surveys), analysis priorities should be structured to provide critical information to projects as quickly as possible. Generally, USAID Missions are not impressed by the slow turn-around associated with most research efforts. The LIL could do a better job showing the applied value of research and quicker turn-around of high-priority findings would help. In order to show impacts on FtF targets, the EET suggests that the LIL better document and justify poverty impacts of research. Most of the impacts of legume research on the poor come through markets and falling food prices, and it is important to document how new legume varieties and production techniques contribute to increased availability to and consumption of grain legumes by the poor. The LIL should continue to evaluate alternative models of outreach, such as animations and use of Innovation Platforms. As USAID seeks to scale up successes, low-cost and effective means of dissemination are required. As many projects are working with modified approaches to outreach, the LIL might undertake a program-wide evaluation of alternative measures. While nutrition-specific research across the LIL is limited, it would be useful to engage more with researchers who lead research on improving dietary diversity, quality and consumer behavior. Industry stakeholders indicated that globally, legume consumption has not increased despite widespread perception that consumption is healthy for consumers. More research is needed on the positive health effects of legume-based diets and why consumer preferences remain unchanged. Of the few LIL projects that do focus on nutrition, there is some knowledge exchange but there have been no deeper, substantive collaborations to expand the nutrition portfolio. This is an area the LIL can expand upon. The EET learned, for example, that the amount of resistant starch present in common bean seed may affect traits such as cooking time, digestibility, and human nutrition (see further discussion under Program Future). The EET feels that further collaborative research between SO3 and SO1 would lead to more effective breeding programs. The agronomic research is substantive but the uptake by consumers will be key to the longer term benefits of these efforts. Understanding what consumers prefer (cooking, taste, price), what they have access to, and how legumes can improve diets (in the context of poor access to animal source foods) and ultimately nutritional outcomes (including child growth, iron deficiency anemia, zinc deficiency etc.), are key. 15 Compare findings from Rubyogo, et al. (2016) cited above with those from Maredia, M., Reyes, B. A., DeYoung, D. 2014. "An assessment of the Bean Seed Distribution Models Implemented under the Bean Technology Dissemination Project: Results of key informant interviews and surveys conducted in Guatemala, Honduras and Nicaragua.” Final Regional Project Workshop of the BTD Project, Guatemala City, March 20. 28 Program Structure and Management EQ 1. Facilitation of communication, coordination, and implementation How effectively has the Legume IL MO facilitated communication, coordination and implementation amongst sub-awardees and stakeholders to carry out the research and training activities in the US, Latin America and Africa? In what ways could communications and coordination be improved as a way to advance project goals and objectives, create a shared research agenda, meet reporting requirements and encourage additional relevant partners? Findings Facilitation of communication, coordination and implementation: The EET found universal satisfaction among participants and stakeholders for the MSU Management Office, and especially for the long service of Dr. Irvin Widders. The expertise of Dr. Widders is complemented by the Deputy Director, Dr. Cynthia Donovan, who brings a strong social science background to the management team. CG and TMAC interviewees were complementary of the work of the MO, as were representatives of HC institutions. Respondents cited the eagerness of the MO to interact in planning, identify opportunities, provide guidance, and work to attract quality scientists to participate in the LIL. The EET found evidence that the MO is proactive in encouraging LIL projects to interact and partner with HC government and non-government organizations. In its most recent evaluation/report, the LIL TMAC lauded the MO for pushing project PIs and NARS collaborators to network regionally. Specifically, the TMAC endorsed the MO’s effort in facilitating project leveraging of funding from the Bill and Melinda Gates Foundation (BMGF) and the McKnight Foundation for the cowpea IPM effort in West Africa. PIs, HC collaborators, and stakeholders described the MO’s efforts in communicating opportunities and coordination as very good. The EET also saw evidence that the MO has moved to strengthen intercontinental linkages in breeding and plant pathology research. Most LIL PIs endorsed the way that the MO is facilitating flows of germplasm (finished varieties and international sources of parental genetic materials) across projects. One PI felt that NARS scientists are being inundated with excessive numbers of nurseries and lines to evaluate, and objected to resources being used to evaluate grain types with attributes not close to local accepted preferences. The EET considered this to be a serious issue, but further questioning of project PIs and HC scientists suggested the problem might be due to limited human resources of a relatively new HC and not typical of other, more mature projects. The EET visit to the MO reinforced messages from interviews in Zambia and site visits to Guatemala and Uganda: researchers reported that the amount and content of communication from the MO is excellent. Most noted that communiques are not too detailed and cover the critical needs well. A few HC scientists thought more MO communications should be translated into the local language for better comprehension. There would clearly be “trade-offs” in cost and speed of delivery. 29 The MO is well organized in the way it deals quickly and effectively with logistics issues (passports, visas, travel, accounting, reporting). The EET observed a sense of teamwork and confidence among the individual members of the MO staff, and at all levels. The size and institutional priority of international agricultural programs at Michigan State is clearly a positive factor for quality MO structure and function. A MO challenge noted in a previous EET review, and recently by the TMAC, is the aging and retirements of US and NARS international legume research leaders. There have also been changes to HC affiliations resulting from FtF country alignments and other factors. Parallel to this challenge has been the amplification of the LIL research portfolio to include novel projects and new disciplines. This expansion involved introducing new players into the established legume research network. In that regard, interviewees cited synergies from the growth of the “LIL Community of Projects”, and the positive role of the MO in supporting these changes. A TMAC member expressed the perception that HC Collaborators are treated and viewed as full PIs in terms of planning, evaluation, and writing/reporting. This treatment helps build the Community of Projects. There is also vocal support for the impact that the small institutional capacity strengthening grants have had on project outputs. As noted, the EET found that some projects working in the same country were not communicating as effectively as possible. Several SO1 projects are working in countries where SO2 and SO4 projects are working, yet coordination could be improved. For example, project SO2.1 is focused on soil fertility, while SO1.A3 develops superior varieties, both for Uganda. It was evident from the EET site visit to Uganda that seed quality in SO2.1 trials would benefit greatly from input from pathologists and breeders. Knowledge of changes in biotic and abiotic stressors, gained by SO2.1 researchers from on-farm trials, would be valuable to breeders in determining trait priorities. Other examples of this disconnect were found and is an area where more intervention and communication from the MO may be required. This is an area where the Director and Deputy Director should combine their expertise to promote cross-discipline sharing. Steps for improved communications and coordination: EET interviews and site visits produced several suggestions for improved communications and coordination. Some suggested that the MO might require projects to work with HC collaborators to further integrate germplasm from US, regional, and international sources. Another area where MO leadership might be productive is in facilitating a cross-project discussion of training needs. Training prioritization currently occurs in project silos and cross-project communications might help achieve economies, particularly when countries or regions lack a particular expertise. Given the genetic challenges of some important “ongoing” disease and insect problems, and combined with new demands in drought, heat, root health and infertile soils, it would be valuable for the MO to lead a focused, program-wide discussion on how NARS plant breeders can work most effectively with LIL PIs and partners to prioritize factors and agree on trait combinations and testing responsibilities. HC breeders must retain a balance of near- and long￾term breeding goals in the face of multiple challenges. An effective regional distribution of those responsibilities might be achieved through enhanced communication. Additional communication might be used to build cross-project learning. As noted elsewhere in 30 this report, the LIL as a whole is multidisciplinary, but several projects tend to be focused on single outputs (e.g. germplasm for increased productivity; information about effective seed systems). Lessons from research on community seed systems in Central America and Tanzania, lessons about the effectiveness of innovation platforms (SO2.1) in Uganda and Mozambique, and information on consumer preferences and grain legume value chains (SO2.2) in Zambia, Malawi and Tanzania can be used to formulate impact-increasing steps across the LIL. In particular, breeders need information about consumer preferences and closer integration with the SO2 projects would provide benefits. Lessons about the effectiveness of ICT in improving farmer knowledge (SO1.B1) have broad implications as does information on the effectiveness of farmer field school training. Evidence of impacts of the release of beneficial insects in West Africa (also SO1.B1) can also provide LIL￾wide lessons. These lessons are not being widely disseminated across projects, and the MO might consider taking actions to stimulate cross-program learning. Better communication across projects might help without radical changes to program structure. The “Spotlights and Success Stories” series highlighted on the LIL web-site is inadequate for information sharing and serves as more of an advertisement vehicle. US and HC personnel routinely visit the Mission jointly when the US PI comes to visit the HC, but LIL projects are generally not at the front of the minds of Mission staff. In the EET’s visit to the USAID Mission in Uganda, the agricultural program officer was aware of the HC PI for the breeding program and was familiar with breeding efforts, but none of the USAID Mission personnel seemed to know about SO2.1 visits even though those visits were described in annual technical reports for SO2.1. These findings illustrate a disconnect between USAID – Washington and the Missions. The latter have short term objectives and experience rapid staff turn-over. It is often difficult to find common interests between the LIL projects and the USAID Mission. Best possibilities for success are like MASFRIJOL, where an off-the-shelf technology is readily available and fits into Mission goals and strategies. However, the LIL could do a better job in communicating its research findings and their relevance to country operations to USAID Mission staff. US PIs view Mission clearances and visits as administrative hurdles and do not generally communicate outside of scheduled visits; incentives to enhance these communications might be explored. One mechanism might be to tie institutional capacity strengthening funds to communications with Mission staff. USAID Missions need short and focused communications, and there is room for improvement in targeted messaging to the Missions. A publication series that highlights research findings and their operational significance is recommended. The structure and content of the LIL web-site are insufficient and the EET recommends that steps be taken to improve it. Currently, many of the hyperlinks on the site are not functioning, and much of the information is outdated. Steps should be taken to liven the presentation and improve the relevance of content. A final area where communication might be enhanced is in the realm of impact: regular communications about how impact pathways might be used to alter project activities (such as by broadening participation in meetings) would clearly enhance the utility of the pathway exercise. Project scientists need to be made aware that they “are not on the hook for impacts” but are expected to be entrepreneurial in building an environment in the HCs where impacts are more likely to result from their research outputs. 31 Conclusions Findings relative to the MO and its communication were generally quite supportive. MO communications have supported efforts at regionalization and intercontinental germplasm exchanges. Projects would benefit from increased information sharing, but regular communications are facilitating a productive interaction among LIL projects with shared interest in genetic resources improvement. More production of user-friendly research briefs might facilitate communication within the IL and to external audiences. EQ 2. Effectiveness of Management Organization model In what ways did the model of having one lead university serve as the MO to manage a global program of multiple interdisciplinary projects contribute to effective implementation and program objectives? In what ways could this model be improved to better meet the Legume IL’s objectives? What strengths or synergies did this model encourage, e.g., among the individual projects to ensure complementarity of effort? For example, did the MO use research outputs to shape its outreach or did the MO serve in its role of bringing information to people's attention as appropriate across the projects? Findings Lead university model and management: The review of supporting documents and interviews with stakeholders conveyed a favorable impression of the Management Entity MO-led program administration model and, more precisely, of a university-led management model. Project scientists were unanimous in their support of the various services the MO provides and most recognize that the MO model reduces their own transactions costs. Reduced transactions costs facilitate specialization and increased research productivity. Specific contributions of the MO model, highlighted during scientist interviews, were enhanced networking across projects, communications regarding short-term funding opportunities, and linkages created with the legume scholars program. The legumes scholar program is a good example of something that might not exist in the absence of a large laboratory approach embodied in the current model. The MO model provides efficiency gains in communications with USAID Missions. Several PIs highlighted the benefits of MO communications in coordinating travel and obtaining the appropriate clearances. The MO model facilitates institutional memory. The MO model has, in the case of the LIL, led to stable leadership that learns from successes and failures. The EET found broad appreciation for MO knowledge of the national and international legume community and the corresponding science. Part of the strength of the LIL is its individual leadership. An MO-model with lesser leadership may not achieve many of the benefits noted here. An effective TMAC might overcome weak management, but would not replace it. The MO, embedded in a large US land-grant university has access to infrastructure, including financial management skills, to manage large programs. The large size of a program such as the LIL helps ensure continued access to infrastructural resources such as office space, ICT infrastructure, and accounting and legal support. The size of the integrated program makes it 32 important to university administration, and this importance is reflected in administrative assistance it receives. In addition to economies of size, the MO model helps insulate subcontractors from accounting related-liability, a feature that makes the MO model more attractive to university and other subcontractors. The flexible management system facilitates interactions with less-developed country institutions. The interactions between the MO and these institutions help build developing-country capacity to manage similar programs. MO model and LIL objectives. Part of the benefit of the MO model comes from its ability to identify new opportunities, research themes, and researchers. Examples are: § Expansion of research in Central America (SO1.A1) to include climbing beans which are under-researched yet represent an important component of the CA and Southern Mexico production system; § Expanding the reach of SO1.4 to include Central American and African dry beans; § Involvement of Dr. David Kramer who is developing a technology which, if as cost￾effective as suggested, would provide enormous benefits in developing countries and the US. This technology has potential as a tool for developing “climate smart” production systems to respond to climate changes; § Expansion to include diet and nutrition-related research on recommendation from the TMAC (SO3 project); and § Initiation of the Legume Scholars program and the institutional capacity strengthening grants Some interviewees noted the high cost associated with maintenance of the MO model. The EET visit to the MO helped dispel some of this concern—the MO is a Spartan (no pun intended) operation with modest facilities and seemingly low overhead. The idea that costs are too high depends on judgments about the per-unit cost of salaried personnel, other operational expenditures, and about whether the resources (personnel and otherwise) are necessary for efficient functioning of the program. The EET saw no evidence of waste. Interviews with USAID Mission staff indicated that they see the MO model as an effective means of streamlining communications. This staff expressed concern that a large number of projects with overlapping geographical foci would lead to excessive requests for country clearances and other services; the MO plays a critical role as a buffer between scientists and the HC development community. The EET has no recommendations about how the MO model might be improved based on its interviews and evaluations of the evidence. However, evidence of the benefits of flexibility might be used to infer stronger MO guidance during second-stage bidding of future projects under a new incarnation of the LIL. Many ILs follow a two-stage process whereby the MO is bid in the first stage and in a second stage a project RFA is released. The observations about the MO strength in the LIL argue strongly that the MO should be expected to play a major role in the design and selection of second-stage projects. 33 Synergies from the MO-model: The LIL program has multiple objectives and program complexity argues for MO-type leadership, especially given the potential spillovers across projects (see above for discussions of spillovers and examples of cross-disciplinary exchanges stimulated by the MO). The MO facilitated cross-project fertilization through its encouragement that SO1.B1 physical scientists engage with SO2.2 social scientists to share experiences about using ICT to facilitate knowledge transfer. The SO1 project in Zambia was encouraged to dialog with the SO2 project to gain knowledge about consumer preferences for beans. These preferences are now being considered by the breeding program. There are other areas where communications could be improved (noted above), but the role of the MO is central in cross-project information sharing. Evidence of cross-project synergism stimulated by the MO is reflected in the connections forged between the impact assessment project (SO4.1) and others. SO4.1 has supported projects SO1.A1 in Guatemala (establishing baseline survey) and SO1.A3 in Guatemala (evaluation of technology dissemination models) and Haiti (effectiveness of seed system model). It helped evaluate the effectiveness of the SAWBO application for SO1.B1 in Burkina Faso. Most recently, upon request from the MO, SO4.1 began a study on the long-term sustainability of a community-based seed system in Burkina Faso. While the MO is not directly responsible for all these synergies, its identification of project needs, networking and encouragement of cross-project communications contributes to them. The importance of academic leadership is underscored by the complexity of the project impact pathways discussed elsewhere in this report. Researchers tend to be specialized in their approach. This specialization increases productivity, but also creates a need for coordination, information sharing, and active effort to identify and fill gaps along the impact pathways. An important academic role of the MO is, as noted elsewhere, in packaging research findings to make them accessible for development programming (such as USAID Missions). This packaging requires communication about how research findings can be incorporated into such programming and highlighting existing gaps along impact pathways. An important synergy created with the MO model is in maintaining links with the US industry. MO-supported ties to the dry-bean industry leads to two important benefits. First, contact between industry and university researchers/extension will help align industry needs with the research program. A program outside of these institutional lines of communication may dampen the voices of an important stakeholder group—US industry representatives. Second, a large program facilitates generation of political support for the program. These synergies underscore the importance of having a MO that is responsive to industry groups across the country. The Michigan Dry Bean Commission voiced strong support. Conversations with similar stakeholders from other states showed support for research they were familiar with via exchanges with their respective university scientists, but less knowledge or opinion of the overarching LIL/FtF objectives. The EET is clear in noting that the model of a MO-led program combined with a strong and engaged TMAC provides several benefits. The TMAC is an active part of the LIL, and competes for scarce resources in the mind of a few scientists (who are nonetheless appreciative of TMAC inputs). The EET believes that if the MO were not characterized by a strong leadership team 34 with extensive knowledge of legume science and the industry, and if MO stability had not been such a hallmark of this IL, the contribution of the TMAC would need to be greater. Conclusions The MO-led model provides important benefits from its size and visibility. The cost of the MO appears to be relatively modest compared to advantages it brings. In the case of the LIL, the MO-led model is successful largely due to dedicated leadership with deep familiarity with the legume industry, the research community, and other stakeholders, and a finger on the pulse of emerging issues. Without such dedicated and intellectual leadership, many benefits of the MO￾led model will diminish. The MO model, thus, depends critically on the leadership, both administrative and intellectual, of the lead PI. The TMAC complements the MO leadership by providing external inputs, a regional perspective, and intensive technical guidance. The combination of strong MO leadership and an engaged and technically competent TMAC in some senses creates redundancy. But the MO model requires such redundancy in the event that MO leadership is weaker. Program structure and management: Lessons learned & recommendations Lessons learned The MO is an efficient model for providing support for the LIL projects. The program is respected and valued by HC institutions. Due to the diffuse nature of the LIL research portfolio, the MO faces some challenges in being the sole source of communications, particularly with external audiences. The composition and size of the TMAC should be evaluated as the LIL evolves. The current TMAC has played an important role in restructuring the LIL and in adding new projects and lines of research. Today’s TMAC, which includes leaders from the LIL PI group, NARS, CG centers, regional PABRA, international research experts in critical disciplines (e.g., nutrition and soil science) and the US stakeholder community, should be well-placed to offer guidance to the MO on critical LIL needs named earlier in the Research section of this report, and in future delivery phases. Recommendations The EET finds that the MO model provides substantial benefits and any change to it should be considered carefully. The TMAC should be structured so that more disciplines in the program are represented. The current TMAC does not have a socio-economist and because the program has a number of projects with heavy socio-economic content, work plan review is a challenge. While the TMAC has high-quality scientists, the economics field is changing rapidly and expert review is necessary to ensure that the projects are evolving with the science. Of course, external peer review of project proposals is done with disciplinary expertise, but a number of critical decisions are made about focus and methods during the duration of the program. Communications with internal and external audiences should be strengthened through an improved web-site. Several links in the current web-site do not function and much of the information is outdated. The “Spotlights and Success Stories” should be expanded to include 35 research results; decisions could be made about an internal organ versus a series targeted at external audiences. Incentives should be provided for individual PIs to communicate with external audiences and make sure their messages are heard. The optimal size of project funding and participation should be explored. Some projects have finances that are spread very thin due to a large number of participants. This makes supervision and coordination difficult for the PIs. It also means that the project might not be taken seriously by the USAID Mission, and researchers/participants with low levels of funding are likely to be less committed to project success. Institutional Capacity Collaboration EQ 1. Partnerships and collaborations What types of partnerships were established to address the different aspects of the activity, including any partnerships with USAID Missions, CGIAR centers and NGOs? What types of productive collaborations were established with other stakeholders, such as host country government and academic institutions, local NGOs, other FTF Innovation Labs and other relevant USAID activities in target countries? Other relevant international research programs working in the target countries (e.g., PABRA (CIAT) and IITA)? What type of opportunities are there for other partnerships and collaborations to further the Legume IL’s objectives? Findings In general, in agricultural research the boundary between formal partnerships and collaborations is blurry. For a partnership, a memorandum of understanding (MOU) or another form of written contract may be established, whereas collaborations are often based on informal verbal agreements, and often in the context of a working group. A formal partnership may facilitate multiple forms of collaboration as the relationship among parties matures. For the purposes of the discussion that follows, we are not distinguishing between legal and non-binding partnerships. Partnerships: A strong emphasis of the LIL spearheaded by the MO has been to cultivate and engage USAID Missions in LIL research efforts with the goal of obtaining Mission support to leverage LIL research findings to create impacts. The main success story has been with the MASFRIJOL Associate Award where the Mission in Guatemala supports dissemination efforts of several dry bean cultivars. MASFRIJOL has done an excellent job coordinating with USAID staff and MASFRIJOL is benefiting from long-standing LIL/CRSP/USAID investments in the region. The communications between LIL and other country and regional USAID Missions are uneven and it is the judgment of the MO that enhanced communication might strengthen the partnerships. It is difficult to generalize about how to enhance LIL-USAID Mission partnerships. The most successful LIL partnership (Guatemala-MASFRIJOL) was idiosyncratic. A Mission staff member who was knowledgeable of and impressed by CRSP and DGP research and training activities while on assignment in an African country mission, became interested in engaging the LIL when 36 in Guatemala, a country where beans are a strategic value chain in the FtF country development strategy. The staff person, who appreciated the importance of beans in the livelihood systems and diets of the poor in Western Guatemala, saw opportunity for a partnership in development. The broad lesson is that engagement is important—USAID staff should regularly be invited to planning meetings and encouraged to participate in field visits of project PIs. The primary responsibility for such actions is with project PIs, but the MO has a responsibility to create incentives for and facilitate such communications. The LIL has been strategic in seeking partnerships with CGIAR centers and NGOs, but the extent and depth of these partnerships vary by project. The MO has attempted to complement and not duplicate efforts with other partners. TMAC has used its position as a somewhat independent body to examine the “big picture” on the ground and identify gaps and opportunities, and has taken the lead, where appropriate, in aligning priorities and identifying potential partners. The LIL addresses elements important from the standpoint of the CG centers, particularly through its contributions to capacity building. LIL funding for research is relatively small compared to the CG centers, and opportunities need to be sought to leverage LIL research with CG resources. An example of this leveraging is in the IITA involvement in West Africa with the SO1.B1 project where research is being conducted to develop and deploy biological controls and biopesticides in Benin, Ghana, Niger, and Burkina Faso. The team has leveraged funds (to IITA) from a BMGF Seed Grant. It is working with NGO partners to extend the work to smallholders. Another fruitful collaboration is between SO1.A5 and IITA in Nigeria to conduct a genetic analysis of aphid resistance in cowpea. A second example is the collaboration of SO1.A4 scientists with CIAT in Central America and Dr. Juan Carlos Rosas at EAP-Zamorano in Honduras. The productivity of these programs in developing and releasing dry bean varieties adapted to the region has, in part, benefited from long-term interactions with CIAT. NGOs with global perspectives include the BMGF, the McKnight Foundation, and the Kirkhouse Trust. The BMGF funds similar types of research activities, and there is overlap between FtF and BMGF country priorities and overlapping interests in some crops. The BMGF has a memorandum of understanding with USAID Bureau of Food Security (BFS--Office of Agricultural Research and Policy) to coordinate investments and avoid competition in funding. BMGF provides support to the Association for the Green Revolution in Africa (AGRA), which partners with USAID to fund the Scaling Seeds and Technologies Partnership (SSTP) to incentivize commercial development of seed companies. These seed companies can take the technologies developed by NARS and other breeding programs and scale up production to levels that will widely disseminate improved cultivars. Efforts are focused largely on maize and groundnuts, but the program does have a grain legume focus. Where this program intersects with the LIL is primarily for cowpea production in West Africa, with S01.A5 participants involved in these efforts. Collaborations: Collaborations with CG centers extend to germplasm testing and selection. For breeding programs, this is a very important activity. It provides access to materials that may be developed directly into cultivars or used in crosses within the HC breeding program. Most 37 nurseries accept elite germplasm from LIL projects. National programs develop a selection strategy and the CG centers often serve as facilitators. IITA has also provided germplasm for exchange with the LIL cowpea program. The flow of cowpea germplasm has been altered in recent years; in the past there was a much more direct flow between UCR, IITA, and the various NARS. Now Ghana gets germplasm from IITA but the flow has been reduced and nurseries are not sent regularly, partly due to reduced funds at IITA. The Pan-African Bean Research Alliance (PABRA) is the main organizational structure through which CIAT coordinates research in Eastern and Southern Africa. Main donors supporting PABRA are DFATD (Canada), SDC (Switzerland) and USAID. It has activities in breeding, integrated crop management, nutrition, seed systems, gender and value chains. LIL researchers are involved in some of these activities, but by far, the breeding area is the main point of contact for LIL breeding programs. PABRA provides the main conduit for variety and germplasm trialing throughout the region. The strength of interaction fostered by PABRA is impressive. SO1.A3 working in Uganda and Zambia has a good working relationship with PABRA, where products from LIL feed into PABRA trials and thereby to other countries in Africa. Trials provide the means to acquire productivity and adaptation data. Depending on the site, the presence of particular diseases and pests allows screening for specific constraints. Repeated trials across the region buffer against the possible loss of individual trials in any season or location. Some projects in the LIL engage in research with NGOs. These are primarily the projects with a strong outreach and/or social science component. They are working with those NGOs on the ground in a country that have specific capabilities that complement the LIL project. For breeding programs this may take the form of seed dissemination. LIL projects have engaged with local NGOs in all regions and where appropriate to specific projects. These range from groups focusing on grain production for market sales, to increasing farmer knowledge of crop management, to production of improved seed and post-harvest operations, to rural value chains. Many have a strong emphasis on gender equity and some engage in participatory research. Details can be found in Table 4. Some of these partnerships have led to institutionalization of various activities. For example, Dr. Rosas (S01.A4) has leveraged funding for participatory research from the Norwegian Development fund in Honduras directed to the farmer-driven community research groups. He has helped release more than 20 locally adapted varieties- an activity that is now self-sustaining. Similar efforts facilitated by Dr. Rosas are present in Nicaragua and Costa Rica. NGO-related farmer groups have been important since the 1990s in Honduras when the extension service was eliminated. 38 Table 4. NGO engagement to extend research technologies to a broader in-country audience. Country LIL Project NGO Areas of collaboration Ghana SO1.A5 CARE International Women farmers, crop management. Ghana SO1.B1 Centre for Learning and Community Development Extending SAWBO to larger community. Guatemala SO1.A4 Western Highlands Integrated Program supported by Asociación Guatemalteca de Exportadores and the Asociación Nacional del Café, others Seed dissemination. Niger SO1.B1 MercyCorps, and Sahel Bio and Hilfswerk der Evangelischen Kirchen Schweiz (HEKS-EPER) Training in producing and using IPM technologies by women, Senegal SO1.A5 Réseau des Organizations Paysannes et Pastorales du Sénégal Farmer organization-training in seed production and postharvest care. Uganda SO1.A3 Community Enterprise Development Organization, Integrated Seed Sector Development–Uganda, CARE, ADRA, SHUPO, SASAKAWA Global 2000; Nyakatozi Growers Cooperative Union, and Appropriate Technology (Uganda) Seed production, not specified. Zambia SO1.A3 Kusefya pa Ngw’ena, Shangila Seed Growers Association, and Participatory Village Development in Isolated Areas Women’s farmer groups, seed and grain production for market sales to elevate income and reduce poverty. Project SO3.1 has long-standing collaborations with the University of Malawi School of Medicine and with government field health officers and technicians under the Ministry of Health. Building human and institutional capacities has been key to this project, and much of this is probably due to the longstanding relationships that the PI has had in the country. Opportunities for partnerships and collaborations: There are opportunities for LIL projects working in Africa to connect with other activities under the PABRA umbrella. In particular, CIAT’s seed 39 distribution models would be an important resource for LIL to tap into to promote more widespread adoption of improved varieties. CIAT also has links to private seed companies in the region which would be invaluable for establishing sustainable seed systems and experience training seed technicians. The LIL complements the CG centers through degree training. US and HC universities provide the basis for training many of the regions’ HC scientists. The LIL has a clear comparative advantage in degree training and in some forms of capacity building for NARS staff, but the CG centers are able to offer research team support and high-quality research facilities for students. There is opportunity for deeper linkages between LIL and CG Centers in thesis research and in leveraging resources for technical training. There appears to be less interaction between the LIL and McKnight Foundation and the Kirkhouse Trust on an administrative level, but individual researchers do interact. McKnight mainly operates through the Collaborative Crop Research Program (CCRP) (also partially funded by the BMGF Foundation). It has projects in some of the same countries and crops as the LIL in Eastern and Southern Africa. There is little apparent coordination in research although there are examples where technologies developed under the former Bean/Cowpea CRSP have been supported for further development. The Kirkhouse Trust supports a West African Cowpea Consortium and the African Bean Consortium in countries where LIL projects are also engaged. As with McKnight, there appears to be little interaction or coordination with the LIL. Technology transfer is a challenge in many contexts. In the dry grain legume context, several actors are working in Africa to strengthen seed systems. The partnership between AGRA and USAID is providing $47 million to develop private sector seed production systems. LIL research could provide important inputs into this effort. At present, LIL is engaged in this effort principally in West Africa with cowpea, but expanding efforts in eastern and southern Africa with common bean presents an opportunity. While coordination with the BMGF seems adequate, there are opportunities for a more strategic approach with McKnight Foundation and Kirkhouse Trust. Enhanced communications could allow better coordination with programs that support similar research efforts. Institutional capacity collaboration: Lessons learned & recommendations In general, the LIL does a good job coordinating activities with NGOs and CG centers. The synergies arising from collaboration via PABRA provide the LIL with greater reach of its outputs and greater potential for impact. What stand out as areas for improvement are enhancing partnerships and collaborations with other FtF programs, major NGOs such as McKnight and Kirkhouse Trust, and with the USAID Missions. 40 Program Future EQ 1. Changes in implementation to better achieve objectives What types of changes in implementation would more effectively achieve outcomes in the following components: research (i.e., design, implementation, communications, stakeholder involvement), capacity development (i.e., student recruitment and selection, content, location) and/or institutional collaboration? The LIL has been effective in achieving its goals. In part this is attributable to the stability of the MO and project leaders. But, as noted in earlier sections of this report, there have been a number of changes and initiatives in the current cycle of funding that have directly contributed to these successes. TMAC evaluation and MO actions have helped foster a dynamic and agile portfolio of project activities. The LIL has moved vigorously to modify or replace some projects, while maintaining those that were most productive. A strong consensus emerging from this evaluation is that the MO has gathered the best US and international legume research minds into this “community”, and has adjusted projects and HC alignments in ways that have produced positive results. The new projects combine interesting upstream and applied research with the potential for near-term outcomes. Research: The EET believes that research productivity could be enhanced by stronger linkages across projects within and across countries. Some research would benefit from interaction with scientists from other LIL projects in the region. In a few project areas (varietal development, value chains, and IPs) this might be further extended across hemispheres (as has already been initiated for some breeding work). The production of international public goods in the social sciences would be facilitated by increased comparison of results across heterogeneous contexts. Regionalization should involve the International Agricultural Research Centers (IARCs), other stakeholders, and premier regional universities. A 2019/2020 assessment of legume genomics, MAS technologies, and phenometrics contributions to LIL varietal improvement and increased on-farm yield potentials, with projected timelines, would be appropriate, given the emphasis on legume breeding. As stated in an earlier section, a simpler/faster form of baseline surveying, would contribute more positively to the project impact planning activity. There appears to be great opportunity for inter-project fertilization between social science and breeding projects, and especially in places where each has interest in expediting diffusion of new varieties and/or technology. Capacity development: Capacity development under the LIL has been successful. The strongest support for this conclusion comes from comments from the PIs and HC collaborators. The LIL projects that date back to the CRSP era have stable research teams, while the newer projects are rapidly reaching full strength. The EET sees a regional LIL capacity building opportunity in combination with the legume IARCs and regional universities. These entities continue to offer cost-effective and quality thesis options in a tropical research environment. A LIL/CIAT/IITA “cross-project summit”, with key 41 regional institutions like UPR, EAP-Zamorano, Makerere University, and Sokoine University of Agriculture could produce multinational and multi-institutional teams to develop and deliver degree and non-degree, technical training programs on topics such as grain legume seed systems, legume innovation platforms, climate smart legume systems, and grain legume value chains. These regional teams could also be a valuable short-term resource to fill project human resource disciplinary gaps while students are receiving graduate training designed to fill that project necessity. The MO/USAID could also offer “Mini-Associate Awards” to regional institutions as an incentive to offer non-degree courses on specific grain legume topics of interest to LIL projects the region. Institutional collaboration: The LIL is respected for effort and success in building institutional collaboration. The EET heard this from and about the TMAC. The EET wonders if the TMAC might be restructured to play an even stronger role in advising LIL projects like those dealing with innovation platforms, value chains, and dissemination of SAWBO technology. The EET sees a need for greater collaboration between the LIL and USAID Missions, but increased collaboration will likely occur on a country-by-country basis. In countries where warm-season grain legumes form an important part of smallholder production systems or the diets of the poor, FtF programs are likely to contain such a component. In such cases, such as is occurring in Guatemala, there is a natural link between past and present LIL research and development programming. In other cases, where FtF programming does not have a specific “warm-season grain legume focus”, it is necessary that the LIL communicate regularly about the implications of its research for in-country operations. The EET finds that the LIL is working in appropriate countries—where dry grain legumes are important and research is needed— however, because USAID funding is limited, dry grain legumes are not a focal crop in most the FtF countries. As FtF programming evolves, however, a robust stream of research outputs and a strong local capacity in dry grain legume research may facilitate programming. The relatively low cost of the LIL research in these countries might be viewed as an insurance contingency for future programming needs. Increased regular and effective communication about this research is important, especially when key Mission decisions are being made. Several of the projects produce information as outputs and the impact of this research will depend on how the information is packaged. The long-term impact of the LIL on legume availability and increased consumption by the poor depends on how well this information is presented to decision makers. Important outputs are available, but it is important that they be packaged and delivered to appropriate audiences. The LIL should facilitate sharing of information and experiences with other ILs. For example, in impact assessment, the LIL provides important lessons. Training events on impact assessment methods could be coordinated to develop consensus among practitioners or assist them in learning about experiences. Comprehensive ex-post impact assessment is beyond the capability (given resources) of many ILs, but impact assessment practices and principles could be used to improve their efficiency. The Standing Panel on Impact Assessment (SPIA) of the CGIAR, which largely focuses on methods and applications for ex-post assessment of impacts of agricultural research, could be a partner in such an effort. 42 Another example where cross-IL sharing might be beneficial is in conducting systems research. Often (as in Guatemala), legumes form a relatively modest proportion of the livelihood portfolio and impacts of alternative management of legume crops should be evaluated in terms of impacts on the entire livelihood system. For example, in a maize-bean system, impacts on maize yields should be considered. The EET was surprised by the lack of overt coordination across ILs and how little information is shared among the USAID-supported labs. Through inquiries regarding studies on seed systems, we were surprised to learn of significant and ongoing research (partnered with USAID/BFS) that seems to not be known or used by the numerous LIL researchers who commented on this barrier to technology transfer. EQ 2. Technical research future Based on this program and the general knowledge generated across the legume research community, what specific technical areas within grain legumes merit new or continuing research investment (e.g., breeding investments vs economic analysis vs nutrition research) or geographic considerations (e.g., what is value of having Central American and African researchers in one program vs two separate investments?) Technical areas The EET concludes that the balance of technical areas is adequate for this stage of program development. Genomics and bioinformatics have potential to increase breeding efficiency. A technology already being used is MAS although the application of SNPs to the process is relatively new. Because SNPs are numerous and well distributed throughout the genome, this approach is expected to be particularly fruitful. Breeders have typically used bi-parental populations for genetic analysis but are now assembling genotype panels for association mapping approaches, and for multi-parent mapping populations. The Andean Diversity Panel has been used by some of the projects in the LIL and can be extended to study other traits. A disconnect observed by the EET is that breeding projects have worked on one to a few constraints at a time, and while success has been achieved for those individual traits, the gains have not necessarily been consolidated over time. Project priorities may change to address a new set of constraints and the progress from the initial work may be lost. The countries in which USAID has funded breeding research has shifted. For example, breeding research in Eastern and Southern Africa under the Bean/Cowpea CRSP incorporated disease resistances to pathogens such as BCMV/BCMNV, common bacterial blight and angular leafspot, into elite breeding lines of several different seed types. It did not appear to the EET that these materials had been shared beyond the original breeding program even while these constraints were the subject of breeding activities in the more recent LIL supported projects. Ideally, programs throughout the region should capitalize on these materials to fix gains in disease resistance, while adding additional traits (including additional disease resistances, quality and productivity, etc.). It should be noted that the situation in Africa for dry beans is complicated by the need to work in different seed classes and for adaptation to varying agroecological zones, not to mention new races of the pathogens arising. It should, however, be possible to create a germplasm base that is uniformly resistant to key pathogens. A number of crops have traits that have almost universally been incorporated into contemporary germplasm. For example, 43 essentially all modern snap bean cultivars carry the same resistance gene to BCMV/BCMNV16. This trait has been fixed in snap bean such that breeders do not have screening programs for this virus disease, and only test their materials at the end of the breeding process to verify that they do retain the resistance gene. New approaches are needed to stabilize previously achieved gains and to bring additional multiple beneficial traits together in the same genetic package. Depending on intended growing environment, the ideal dry bean cultivar might include resistance to rust, angular leaf spot, halo blight, common blight, brown spot, anthracnose, BCMV/BCMNV, white mold and root rot combined with heat and drought tolerance, rapid cooking time and high yields. MAS has worked best for transfer of single genes and would provide an efficient means of combining several of the resistances mentioned above, but would not necessarily address those which are inherited as quantitative traits. QTL analysis has provided insight into genetic control of quantitative traits and in some cases has provided markers useful for MAS, but new techniques are needed to efficiently manipulate quantitative traits. Another area where genomics has significant potential is in identification of candidate genes for particular traits. Having the common bean genome in hand facilitates the process of mapping traits to chromosomes and associating underlying DNA sequences to gene expression. Gene discovery can be coupled to gene modification via genome editing technologies (although there is still the hurdle of requiring an efficient regeneration system for common bean). Additional upstream technologies with application to genetic improvement are metabolomics and proteomics. USAID will want to continue investing in genomics and bioinformatics related technologies. High throughput phenotyping is an area of interest to the breeding world. PhotosynQ is an example of a technology that has the potential to collect data rapidly and accurately on large plant samples. Other robotics-based technologies are being developed and some have application to screening traits of relevance to tropical production. One of the last frontiers in breeding research is with roots. The below ground portion of the plant is especially difficult to phenotype, yet vigorous, healthy roots are essential for high productivity and resilience. Root rot organisms are a major constraint to productivity and the symbiotic relationships between legumes and Rhizobia spp. bacteria are important. Measuring root traits is a slow and tedious process and advances in sensor technology may open a window into making root studies less onerous. While the LIL is focused exclusively on common bean and cowpea, other grain legumes grown in the Africa and Central America and Caribbean regions merit increased research. It would also be productive to explore thoroughly the agronomic potential and the dietary acceptability of a few selected legume species that are well-known for characteristics deemed important for Climate Smart Agriculture, such as pigeon pea, lablab, and the tepary bean (Phaseolus acutifolius, already an interesting part of the work in project SO1.A4). The LIL should explore future 16 We are not advocating incorporation of this particular resistance into African adapted dry beans as the trait is particularly vulnerable to the temperature insensitive necrosis inducing strains of BCMNV endemic in Africa. 44 partnerships with ICRISAT for pigeon pea and ICARDA for cool season legumes. The EET is convinced that the LIL should continue and strengthen the engagement of multiple disciplines in its research projects. The suggestion is to incorporate multiple disciplines within each project, rather than have separate economics and nutrition projects. Examples from site visits were found where a given LIL project would benefit from asking professional help from another LIL project on a particular limiting factor. The new LIL should incorporate mechanisms to share information and expertise across projects. Several themes are emerging for investigation involving socio-economic research. These include evaluation of seed system constraints and rigorous analysis of models for improved and viable seed systems. This analysis would include institutional analysis of legal and regulatory constraints to private sector involvement in seed systems. Similarly, further analysis is needed of constraints in input and output marketing systems and transmission of prices and information on consumer preferences up and down the value chain. At the farm and household levels, research is needed to understand behavior and factors affecting production, marketing and consumption of warm-season grain legumes. Farm system-level analysis of varieties and management alternatives is also needed. Currently, most LIL projects examine impacts on warm-season grain legume productivity and cost of production, without considering the role of the crop in the household livelihood system. The LIL has made good headway in experimenting in dissemination and additional evaluation is needed of the effectiveness of alternative dissemination models including use of ICTs for training. A combined research program to synthesize what works and does not work in seed provision, dissemination, and knowledge sharing would benefit development decision makers. The LIL provides a good case study for how to integrate impact assessment into a comprehensive agricultural research program. The Impact Pathway exercise led by the SO4.1 appears to have made project PIs more aware of the steps they need to take to increase impacts of their own research. However, USAID no longer holds research projects to be responsible for achieving impacts and there is currently a disconnect between a project designed to enhance impact awareness and assess impacts (SO4.1) and a program where the scientists are not responsible for impact. USAID should provide clear guidance to the MO and project PIs about its conception about the role of agricultural research projects in producing impacts. The impact assessment project should be assigned a more proactive role in identifying constraints to impact and making recommendations to USAID and the MO about steps to enhance the income-generation potential of the program. Moving forward, it would be beneficial to see more substantive integration of nutrition into LIL projects. There is a heightened awareness of the need to better understand the linkages between agriculture and nutrition, and to decipher ways in which agriculture can contribute to improved nutrition through certain commodities. The LIL should in the future, conduct research to understand pathways toward improved dietary diversity and nutritional status with legumes as a central component. Legume production-income pathways and their impacts on consumption are also important. There is an increasing demand for foods that are higher in nutritional value such as animal source foods. However, it is clear that the planet will suffer serious environmental constraints to meet that demand. Legumes are an important source of 45 protein and nutrients that could serve as an alternative to animal source foods. A specific example of a potentially important linkage between nutrition and breeding was noted by the EET. The central hypothesis of the SO3.1 project – that dry bean flour fed as a porridge to infants and young children can improve gut health and overall nutritional status – remains to be proven, but already it is posing questions that may impact research and breeding efforts in other research projects. SO1.A3 found that lower resistance-starch levels are correlated with decreased cooking time, but the nutrition group specifically selected high resistant-starch dry beans for their feeding trials because it is believed that resistant starch is important in improving gut biome health17. If findings by both groups are verified, then it may be that breeding varieties for high and low resistance, intended for different uses, may be necessary. To our knowledge the two groups have not discussed these issues, and this deficiency illustrates the importance of integrating nutrition-specific research into the LIL. What would future areas of research in the area of diets and nutrition be? The evaluators argue that more can be done to understand consumer demand and preferences of legumes, improvements in “nutrition sensitive value chains”, better information on cooking, preparation and recipes for complementary foods for young children. Women are the predominant producers of legume crops in many countries; they are also responsible for food preparation and feeding. This combination of responsibilities provides an opportunity for LIL to reach and empower women through nutrition tools and knowledge related to benefits of legume consumption. Women’s opinions about legume cooking time, appearance and other consumption attributes should inform the entire LIL research program. Evidence has shown that by focusing on women, the biggest impacts can be felt on child health. Conclusions The EET has heard concern about the long-standing “Yield Gap” and lagging yield gains. Global legume production and consumption have not grown commensurate with the potential role in a nutritious diet. A comprehensive strategy must be articulated that leads toward larger impacts. While LIL scientists are themselves not responsible for producing impacts, the program as a whole might be better designed to gain information on obstacles to increased impact. This EET evaluation finds that projects should be designed with knowledge of factors affecting the “Yield Gap”. In the near term, there are a few opportunities to further exploit the “low hanging fruit” of successful efforts by CRSP/LIL and other projects, such as the Central American Bean Technology Dissemination project, MASFRIJOL and the Lake Kivu Innovation Platforms. This is not to criticize nor eliminate valuable “upstream” work in progress, but to place increased awareness and strategize downstream effects and reasonable timelines for outcomes. One step would be to analyze links between upstream research and the status of 17 The nutrition group was advised by Colorado State University researchers that the navy bean market class is associated with high levels of resistant-starch. As a result, the nutrition group sought to locally source such dry beans in Malawi, but encountered some difficulty in finding them because they are not a commonly grown market class in the country. 46 NARS and NGO delivery systems. This would inform projects at an early stage if new research and training will be needed to deliver the project outcomes. There is urgency in addressing and resolving inadequacies in host country seed delivery systems. Several useful LIL technologies (varieties, soil amendments and some aspects of value chain enhancement) are hostage to this dilemma. Some of these efforts are currently challenged more by limitations of extension delivery than by research information. Problems must be addressed at the level they appear, and by the entire LIL community. There is also an existing community of experienced professionals, ongoing projects, and documented studies that should be used to address the LIL’s seed systems concerns. Program future: Lessons learned & recommendations The EET has heard concern from scientists about lagging warm-season grain legume yield gains throughout the developing world. This lag is not entirely due to absence of high-quality germplasm. In Guatemala, for example, rapid diffusion of 25-year old dry bean technologies combined with farmer training has led to impressive yield growth for farmers (the MASFRIJOL Associate Award). The contribution of LIL research to overall program goals could be enhanced by evaluating knowledge gaps with respect to these constraints. The future research program should be designed to conduct research that fills the largest knowledge gaps; an impact priority setting ought to be conducted to understand what kinds, other than genomics, of research most likely will contribute to impact. Although the gaps themselves will be filled by operational actions (not the research program), it is important to fill the most pressing information gaps. Technical areas of research are spelled out above, but the overall sense of the EET is that there needs to be additional balance in research across the impact pathway continuum, increased integration of social and nutritional sciences in the breeding programs, and more learning across study geographies and disciplines. Ultimately, the responsibility for creating integrated multi￾disciplinary projects is in the hands of the PIs who design the proposal. However, PIs can be unaware of the strengths other disciplines might bring to a research project. The RFA process might stress the need for proposed projects to be multi-disciplinary, but the acquired knowledge and broad perspective of a model like the MO can be exploited to “nudge” the right teams together. Once the program is in place, the MO model is suited for bringing multiple disciplines together. It can do so by broadening the participation in the project (e.g. encouraging an entomologist to join a social science project) and by encouraging information sharing across projects (e.g. encouraging dialog between social scientists studying preferences for variety attributes and breeders who are breeding for attributes). The MO should promote knowledge sharing across LIL (and other) projects and communicate LIL research to the ultimate users of LIL-generated information. 47 C. MANAGEMENT OFFICE RESPONSE TO EXTERNAL EVALUATION September 2016 Introduction The Management Office (MO) of the Feed the Future Innovation Lab for Collaborative Research on Grain Legumes (Legume Innovation Lab) at Michigan State University would like first to thank the External Evaluation Team (EET) for their review of the program, including visits to project sites and institutions and attending the PanAfrican Global Legume and World Cowpea Conference in Zambia to gather information and insights for this document. We also thank the Office of Agricultural Research and Policy, Bureau for Food Security in USAID for supporting this review and for their continuing support of grain legume research through this program. We read this report with great interest and recognize that it has valuable content for improving research and administration in a future new program. The report is also a confirmation that the Legume Innovation Lab has supported an appropriate set of research priorities and effectively administered a large complex multidisciplinary program across Sub-Saharan Africa, Latin America, and the United States. Although there are detailed aspects of the EET’s observations and recommendations that may need clarification, the greatest value of the report, and our input here, lies in the global perspective and recommendations concerning future research and development directions, as well as guidance for USAID on the management of a new program. Our brief response will focus on the broader issues and address six key topics: 1) importance of strong technical leadership and management; 2) multi-disciplinarity in a future Innovation Lab; 3) key thematic priorities and new frontiers for research and development in a new Innovation Lab; and 4) communications for coordination and outreach; 5) role of impact and adoption research, and 6) capacity strengthening and training. The last section pulls together the key elements in a look to the future. 1. Importance of technical leadership and management The Legume Innovation Lab was designed with a management office (MO) based at the management entity, Michigan State University (MSU), to provide both technical and administrative leadership to the program. In addition, a Technical Management Advisory Committee (TMAC) was established to advise the MO and USAID on program direction and performance, and on administration. The EET Report provided strong support for this model in the context of USAID’s investments in global research and training under Feed the Future (FTF). The members of the EET found that the MSU model for the MO and TMAC was highly effective, stating “While its costs are not negligible, the evaluators conclude that USAID is receiving good value for its money”. The EET supports the management entity structure was based on four key points: 1) the MO provides USAID with a focal point of contact for a range of projects; 2) the MO leadership and 48 TMAC comprise a center of technical expertise on grain legumes; 3) the MO enables support of and communication with a worldwide network of grain legume researchers; and 4) it reduces transaction costs for researchers by taking on administrative responsibilities related to centralized compliance measures, standardized contracting, and coordinated communications with USAID, both in Washington and at the Missions. We thank the EET for recognizing the strong institutional support, facilities, and support provided by the MO at MSU that are critical to efficient contracting and coordination across the collaborating institutions. The combination of experienced leadership with technical expertise, a strong committed institution, and an active TMAC, ensures that the MSU MO effectively delivers on its administrative responsibilities. 2. Multi-disciplinarity in the Legume Innovation Lab The EET report highlights the strategic contributions that the range of research disciplines and projects offer to the quality of work and outputs of the Legume Innovation Lab. Throughout the report, there is recognition of the value of multidisciplinary teams collaborating to achieve the objectives of the various projects, and a call for additions to teams where a new disciplinary approach would be beneficial. In keeping with the EET recommendations, we recognize the need for a continued research balance, along with the need to seek new projects; for example, but not limited to, warm season tropically adapted legume species. As the EET noted, the MO and the TMAC have always sought to be agile in program design, looking for opportunities to be more effective, while also opening the door to new and innovative research over the range of disciplines. The ME strongly endorses one of the key recommendations of the EET that the Legume Innovation Lab continue to include nutrition research, and to broaden it to include research on consumer behavior, dietary diversity and other related areas, with links to plant breeding, farm management, and value chain research. We will discuss the justification for nutrition research further below, but are convinced that to be effective in achieving changes in food security and nutrition, integrated multidisciplinary approaches, including nutrition, are necessary, and build on the previous research of the Legume Innovation Lab and its predecessors. Nutrition research is also complementary to the research foci of the CGIAR on grain legumes, which lacks capacity in this strategic area. Geographical diversity of the Legume Innovation Lab also plays a part in identifying solutions and addressing global issues, and, as the EET recommends, ensuring linkages across the regions, continents and projects adds value to the investments. 3. Key thematic priorities for the future In this section we will build on the EET’s assessment of key priorities for a future Innovation Lab. There are seven main areas in which continued or expanded investments will build on existing research to develop technologies that respond to the needs and demands of farmers, consumers, and other stakeholders in grain legume value chains in FTF developing countries. Those main research areas are 1) closing the yield gap; 2) plant breeding and genomics; 3) integrated pest management (IPM); 4) sustainable seed systems; 5) nutrition; 6) technology transfer, including communication strategies, and adoption; and 7) impact assessment. 49 3.1 Closing the yield gap for smallholder production While there have been successes in improving farmers’ grain legume yields based on research from the Legume Innovation Lab and its predecessors, the EET rightly identifies the need for a more focused effort to reduce the large yield gap that persists in grain legume production systems in developing countries. A comprehensive assessment of viable opportunities for yield gains, and of technologies accessible and demanded by farmers, should be part of the role of a future program, complementing and contributing to the efforts of the CGIAR and national agriculture research systems. 3.2 Plant breeding and genomics The value of integrating the use of modern genomics tools to improve the effectiveness and efficiency of crop breeding programs is now established, as indicated by the EET. The research investment in the recent past fostered the first steps toward the integration of breeding and genomics. Major advances will now be realized by significantly increasing the phenotype and genomics data necessary for fine-mapping and the development of broadly effective markers for important traits. These will be the drivers of a modern trait-based breeding strategy that includes yield under diverse agro-ecological conditions as a major target, while recognizing the need to develop varieties with broad production value and consumer/processor interest. As the EET notes, the Legume Innovation Lab has looked to innovative approaches for phenotyping, with PhotoSynQ and the MultiSpeQ instrument. It represents a “blue sky” project, where the potential yield gains through increased efficiency of carbon fixation and translocation by grain legumes under abiotic stress conditions are envisioned. With sophisticated phenotyping tools and analytics available to researchers, extension agents, farmers, and entrepreneurs in Africa and Latin America, this empowers host country scientists and programs to perform complex phenotyping experiments and provides the tools to analyze and make sense of the data for use in real world applications of improving yield levels of grain legume crops. 3.3 Integrated Pest Management (IPM) with emphasis on biological controls In agreement with the EET, the use of IPM-based pest control strategies in grain legume cropping systems represents a long-term strategy to achieve significant and sustainable yield gains. Building on the outcomes of Legume Innovation Lab, IPM strategies that use various options (biocontrols, biopesticides, and scalable communication strategies) separately and in various combinations have the potential to effectively realize yield gains in cowpeas and other grain legume cropping systems than the continued use of second generation pesticides, without the health and environment concerns of traditional insecticide sprays. 3.4 Sustainable Seed Systems With the investments in genomics and breeding, researchers have developed new varieties of grain legumes that can tolerate drought, be productive in degraded soils and possess disease and insect resistance traits. However, yields of grain legumes by smallholder farmers are not 50 improving at an acceptable rates due in large part to the lack of farmer use of quality seed of improved varieties. We agree with the EET recommendation that a multidisciplinary approach to seed systems should be a focus of both greater training and value chain investments in a future Innovation Lab. 3.5 Nutrition Research There is currently one project that focuses on nutrition, and as the EET report indicates, it is well-designed and implemented. The research project, based in Malawi, focuses on infant feeding and the role of grain legumes in the gut function and health. Looking to the future, FTF programs will continue to be expected to address nutritional insecurity and malnutrition of young children and women. Grain legumes are nutrient dense staple foods in diets of the poor around the world, thus being the logical choice for enhancing the nutritional quality of diets and improving nutrition and health. As recommended by the EET, future research priorities for grain legumes should include an emphasis on increasing knowledge of the nutritional and health value of grain legumes in diets, as well as on interdisciplinary research that seeks to enhance consumer access, preferences, and consumption of grain legumes for improved nutritional outcomes. 3.6 Technology transfer and decision tools Transfer of technologies broadly fall into two major categories, the embodied and disembodied technologies described by the EET. Embodied technologies are physical materials, including seeds and biopesticides that move through a value chain to reach users. Disembodied technologies are knowledge-based technologies, such as soil and pest management. In reality, a combination of both embodied and disembodied technologies is often the right mix for addressing problems. As the EET notes, engagement with both the formal and non-formal private sectors will be critical for the embodied technologies to become available through markets and exchanges. The Legume Innovation Lab engages economists and rural sociologists among others to help breeders and others understand the demand for inputs and outputs, as well as institutional linkages that may ensure the technologies are well-designed and placed for scaling up. Such strategies can certainly be evaluated and built upon in a next phase of this program. For disembodied technologies, including soil and pest management, the MO has encouraged cross-collaboration between the SAWBO team (originally based at University of Illinois and now at MSU), the impact assessment team (based at MSU) and the decision support tools team (based at Iowa State), in order to investigate scaling strategies for management tools. We find that the EET recommendation to use the RFP process and MO role to ensure the multidisciplinary approach is especially important with disembodied technologies for smallholder farmers. 4. Communications for coordination and outreach The EET raised several important points regarding communications, both in terms of challenges and emerging opportunities. Innovations in communication are needed to inform stakeholders 51 of technologies and knowledge generated by research on grain legumes, of lessons learned from research and development (R&D), and of leveraging opportunities. Unfortunately, many stakeholders (e.g., USAID Missions, other scientists, industrial partners, etc.) are not well informed of outputs and their potential to be scaled up for wide developmental outcomes, of possible opportunities for collaboration, and of new funding opportunities. A new Innovation Lab will need to have an overarching communications strategy to address these weaknesses. Finally, in keeping with the recommendations of the EET, a next phase of this program should continue to investigate cost-effective outreach models. 5. Role of Impact and Adoption Research Impact assessment research (IAR) is an essential part of publicly funded research programs, both for accountability and learning. In multi-disciplinary R&D initiatives, IAR needs to be well integrated into project design so that data are appropriately collected and evaluated to document outputs, outcomes (adoption), and pilot-scaling initiatives. The EET notes the challenges of collecting the necessary baseline data and information essential to understand constraints to adoption. With existing linkages with the CGIAR Standing Panel on Impact Assessment (SPIA), we believe that progress can be made in IAR. In addition to conducting ex post impact assessment of past successes that document the adoption of new technologies, we believe that a future Legume Innovation Lab program should invest adequate resources to assess constraints to adoption and to evaluating what works and what does not. This would also links to the EET recommendation to diagnose the gaps to identify priority areas needed to reduce the yield gaps. The goal of IAR should be to increase the overall impact of publicly funded programs by inculcating an impact culture within each of the multi-disciplinary teams and by helping projects to prioritize and to scale up the dissemination of outputs in ways that are effective and based on an understanding of socio-economic factors. 7. Capacity Strengthening and Training The Legume Innovation Lab has a fundamental commitment to work with our host countries to support and strengthen human capacity and the research environment to achieve their objectives. As the EET noted, there is a balance between long-term training and short-term training, for both men and women, as well as for researchers and technicians. The capacity building funds contribute to the local institutions filling in gaps. These efforts may be complemented by special or leveraged funding for regional activities or cross-project initiatives. 8. Summary The Management Office of the Legume Innovation Lab believes that considerable opportunities exist to build on the technical progress and outcomes of the current program and meet the challenges identified in the EET report. We endorse the EET’s recommendations for a future program and concur that collaborations between the international multi-disciplinary community of scientists and institutions engaged in grain legume research should continue to be strengthened. New iterations of the program could certainly benefit from 1) greater linkages between African and Central American projects (to exchange both biological materials and lessons learned), 2) clearer communications regarding achieving impact, 3) priority-setting exercises that may allow for identification of research gaps, obstacles, and opportunities for 52 increased impact, and 4) tighter linkages between groups that are at different “ends of the R4D continuum”. Based on resources, additional research themes may include climate smart agriculture exploitation of grain legume diversity for more productive, resilient and sustainable farming systems; and systems to better integrate agricultural outcomes with direct improvements in human health through both nutrition and consumer demand. Grain legumes are multifunctional, offering options for human and soil nutrition, income, food security, and risk reduction, making them a logical investment under Feed the Future. 53 APPENDICES Appendix A: Scope of Work 54 Appendix A: Scope of Work EXTERNAL EVALUATION OF THE FEED THE FUTURE INNOVATION LAB FOR COLLABORATIVE RESEARCH ON GRAIN LEGUMES: SCOPE OF WORK I. BACKGROUND INFORMATION A) Identifying Information 1. Project/Activity Title: Feed the Future Innovation Lab for Collaborative Research on Grain Legumes (Legume Innovation Lab) 2. Award Number: AID-EDH-A--00-07-00005 3. Award Dates: 4/1/2013 to 9/29/2017 (based on original award of 9/30/2007 as modified on 3/28/2013 4. Project/Activity Funding: $25,500,000.00 (Total $39,514,000.00) 5. Implementing Organization: Michigan State University 6. Project/Activity COR/AOR: Jennifer “Vern” Long B) Development Context 1. Opportunity Addressed by the Project/Activity being Evaluated General Background The Feed the Future (FTF) Food Security Research Strategy goals are to advance the productivity frontier, to transform key production systems and to enhance nutrition and food safety through agriculture to advance FTF’s overarching goal of sustainably reducing global poverty and hunger. The Feed the Future Innovation Labs with U.S. universities (the “Innovation Labs,” formerly called CRSPs) were created under Title XII of the International Development and Food Assistance Act of 1975, which authorized USAID to engage U.S. land grant and other eligible universities to address the needs of developing nations while also contributing to U.S. food security and agricultural development. In 2000, Title XII was reauthorized, enabling these U.S. university research efforts to continue “to achieve the mutual goals among nations of ensuring food security, human health, agricultural growth, trade expansion, and the wise and sustainable use of natural resources”. The Innovation Labs are an integral part of the new Feed the Future Food Security Innovation Center, established to implement the Feed the Future Global Hunger and Food Security Research Strategy and to respond to two key recommendations from a Board for International Food and Agricultural Development (BIFAD) commissioned CRSP review18: 18 http://transition.usaid.gov/our_work/agriculture/bifad/BIFADREVIEW_CRSP_August2012.pdf Appendix A: Scope of Work 55 • To develop an overarching and coordinated strategy for engaging U.S. universities in agriculture and food security research and human and institutional capacity development that includes the CRSPs as a central component; and • To leverage the impact of CRSP investments by strengthening links across universities, U.S. government, global programs, foundations, and other donors. The launch of the Food Security Innovation Center in 2012 enables USAID to manage its research, policy and capacity-strengthening portfolio through the following seven thematic areas rather than by institutional home: • Program for Research on Climate Resilient Cereals • Program for Research on Legume Productivity • Program for Advanced Approaches to Combat Pests and Diseases • Program for Research on Nutritious and Safe Foods • Program for Markets and Policy Research and Support • Program for Sustainable Intensification • Program for Human and Institutional Capacity Development The Innovation Labs funding of these themes is a central component to develop an overarching and coordinated strategy for engaging U.S. universities in agriculture and food security research and human and institutional capacity development and to leverage the impact of those investments by strengthening links across universities, U.S. government, global programs, foundations, and other donors. The Program for Research on Legume Productivity, which includes the Legume and other Innovation Labs , aims to increase the production and consumption of critical, nutrient-rich legumes, by developing disease and stress tolerant, high-yielding varieties, improve market linkages and post-harvest processing, and integrate legumes into major farming systems to improve household nutrition and incomes, especially for women. Description of the Legume Innovation Lab The Feed the Future Innovation Lab for Collaborative Research on Grain Legumes (the “Legume IL”), began as the Bean/Cowpea CRSP (1980–2007) and then became the Dry Grain Pulses CRSP (2007- 2012) with funding from USAID. With extension of the latter award in 2013 and rebranding under FTF, these former CRSPs became known as the Legume IL. Purpose The global program goal of the Legume Innovation Lab is to substantively increase the potential for grain legume productivity and sustainable intensification of smallholder farm systems as a way to increase the availability of affordable grain in domestic markets, increase consumption of legumes by the poor, and improve nutrition and nutritional security in developing countries. Research & Administration The Legume IL meets its research goals with projects geared toward four strategic objectives: • S01: “Advancing the Productivity Frontier” to substantively and sustainably increase grain legume productivity by improving adaptation to diverse agroecologies and reducing smallholder farmer vulnerability to climate change, with special consideration for the livelihoods of women, including o S01.A: To substantively enhance the genetic yield potential of common bean and cowpea by exploiting new research tools afforded by genomics and molecular breeding approaches, with a focus on improving resistances to economically important abiotic and Appendix A: Scope of Work 56 biotic constraints that limit yield in strategic cropping systems in Africa and Latin America where grain legumes are extensively grown (5 Projects), and o S01.B To sustainably reduce the yield gap for selected grain legume crops produced by smallholder, resource-poor farmers in strategic cropping systems (1 Project); • S02: “Transforming Grain Legume Systems and Value Chains” to transform grain legume-based systems through improved smallholder production management decision making and more effectual governance management of legume value chains by stakeholders, including smallholder farmers and consumers (2 Projects); • S03: “Enhancing Nutrition” to improve the nutritional quality of diets and to enhance the nutritional and health status of the poor through the consumption of edible grain legume-based foods (1 Project); and • S04: “Improving Outcomes of Research and Capacity Building” to improve outcomes of Legume Innovation Lab research and capacity building projects and to assess impacts to improve decision making regarding future investments (1 Project). The Legume IL is administered through its Management Office (MO) at Michigan State University under the direction of its director, deputy director and other administrative staff in conjunction with its Technical Management Advisory Committee (TMAC) that monitors the program’s strategic planning functions. An External Advisory Panel is convened for the review of proposals and the selection of projects for funding. 2. Target Areas and Groups Key U.S. Partners The Legume IL projects are led by University of California, Riverside; University of Illinois at Urbana– Champaign; Iowa State University; Kansas State University; Michigan State University; North Dakota State University; University of Puerto Rico; and Washington University, St. Louis. Geographic Reach The Legume IL project countries are Benin, Burkina Faso, Ghana, Guatemala, Haiti, Honduras, Malawi, Mozambique, Niger, Senegal, Tanzania, Uganda and Zambia. C) Documents For this evaluation, the evaluation team will review a wide variety of documents provided by BFS, the MO and by the sub-award partners. Other documents may be requested during the fieldwork preparation phase, or while the evaluation team is in the field. The primary documents and types of documents that will be provided are listed below. • Technical Application proposal, • Legume IL annual reports, • Sub-Awards project briefs, reports and annual work plans • Program publications including success stories, research publications, HICD outputs, and impact briefs (with links to data), • Other Legume Lab IL website materials and information (legumelab.msu.edu), and • External review report for first five years of the program (Dry Grain Pulses CRSP, 2007-2012) • Feed the Future Research Strategy Appendix A: Scope of Work 57 II. EVALUATION RATIONALE A) Evaluation Purpose The purpose of this external performance evaluation of the Legume IL is to provide empirical evidence to respond to evaluation questions designed to support learning and continuous improvement for BFS’ work. The evaluation will assess progress toward outcomes, the quality of the research program and its outputs, what is and is not working well in implementation, including the effectiveness and contributions of the program management entity, and will provide information and recommendations that BFS can use to inform the development of new investments in legumes research for development (from research directions to management styles), inform the development of new investments in legumes research for development, improve activity effectiveness and better achieve intended outcomes. B) Audience and Intended Uses These results are to be used by USAID/BFS/ARP-R to establish a future RFA (including program design) to address outstanding research questions connected to grain legume productivity for smallholders in developing countries and associated value chains. The evaluation results may also be applicable to others who are involved in designing research for development programs. C) Evaluation Questions Research Program 1. What contributions have the Legume IL research activities made to developing research outputs that could contribute to higher level impacts on the program goals of a. substantively increasing grain legume productivity through sustainable intensification of smallholder farm systems, b. increasing the availability of affordable grain in domestic markets, c. increasing consumption of legumes by the poor, and d. improving nutrition and nutritional security in developing countries? 2. To what extent did the project generate robust and quality research outputs using disciplinary￾appropriate metrics? Are the outputs relevant for a research-for-development project (e.g., did they generate new breeding lines of relevance to FTF countries)? Were the projects undertaken using the right tools/technologies and were they well executed? 3. What research-related challenges has the Legume IL faced during research design and implementation? In what ways have those challenges been addressed? What opportunities are there to improve research outputs moving forward? 4. Given the dual benefit mandate of Title XII authorized programs, such as the Legume IL, is there a sufficient balance between research efforts directed towards priorities of the domestic bean/cowpea stakeholder community and the bean/cowpea research priorities required to advance global food security goals, particularly in FTF countries? 5. Has the Legume IL met its academic training and technical capacity strengthening targets? Are the appropriate type and number of people being targeted for the right kind of training? In what ways has the program identified and addressed academic and technical capacity needs? In what ways could this be improved? Appendix A: Scope of Work 58 Program Structure & Management 1. How effectively has the Legume IL MO facilitated communication, coordination and implementation amongst sub-awardees and stakeholders to carry out the research and training activities in the U.S., Latin America and Africa? In what ways could communications and coordination be improved as a way to advance project goals and objectives, create a shared research agenda, meet reporting requirements and encourage additional relevant partners? 2. In what ways did the model of having one lead university serve as the MO to manage a global program of multiple interdisciplinary projects contribute to effective implementation and program objectives? In what ways could this model be improved to better meet the Legume IL’s objectives? What strengths or synergies did this model encourage, e.g., among the individual projects to ensure complementarity of effort? For example, did the MO use research outputs to shape its outreach or did the MO serve in its role of bringing information to people's attention as appropriate across the projects? Institutional Capacity Collaboration 1. What types of partnerships were established to address the different aspects of the activity, including any partnerships with USAID Missions, CGIAR centers and NGOs? What types of productive collaborations were established with other stakeholders, such as host country government and academic institutions, local NGOs, other FTF Innovation Labs and other relevant USAID activities in target countries? Other relevant international research programs working in the target countries (e.g., PABRA (CIAT) and IITA)? What type of opportunities are there for other partnerships and collaborations to further the Legume IL’s objectives? Program Future 1. What types of changes in implementation would more effectively achieve outcomes in the following components: research (i.e., design, implementation, communications, stakeholder involvement), capacity development (i.e., student recruitment and selection, content, location) and/or institutional collaboration? 2. Based on this program and the general knowledge generated across the legumes research community, what specific technical areas within grain legumes merit new or continuing research investment (e.g., breeding investments vs economic analysis vs nutrition research) or geographic considerations (e.g., what is value of having Central American and African researchers in one program vs two separate investments?) Appendix A: Scope of Work 59 III. EVALUATION DESIGN AND METHODOLOGY A) Evaluation Design The evaluation team will finalize the evaluation approach in the evaluation plan. However, a mixed methods or process evaluation approach, collecting and analyzing qualitative data is recommended. Qualitative data will be synthesized, analyzed, and triangulated to provide robust, objective evidence to answer the evaluation questions. B) Data Collection Methods This evaluation is expected to collect data through both qualitative and quantitative methods from a variety of stakeholders. At a minimum, qualitative data will be collected through key informant interviews and observation. Data collection methods will be finalized in the Evaluation Plan developed by the evaluation team. Data collection protocols will address the experiences, perceptions, opinions, motivations, and knowledge of the stakeholders interviewed, the context within which the activity is operating; and the factors (causal mechanisms) leading to observed results (or non-results). In-depth conversations guided by structured and semi-structured key informant interview instruments provide the opportunity for a flow of information that is not constrained by pre-determined response categories. This allows for exploration of subjects that surface during the interview and elicits information that provides nuance and insight for addressing the evaluation questions. C) Methodological strengths and limitations Limitations will be addressed in the finalized evaluation plan. IV. EVALUATION PRODUCTS A) Deliverables The specific deliverables, the timetable for delivery and dissemination plans are described in detail in Section VI and include: • Knowledge Gap Table (optional) February 10, 2016 • Draft Evaluation Plan February 19, 2016 • Final Evaluation Plan March 15, 2016 • Travel Completion Date May 15, 2016 • Preliminary findings June 1, 2016 • Draft Evaluation Report June 15, 2016 • Final Evaluation Report July 1, 2016 B) Reporting Guidelines Reporting should be done in accordance with USAID’s Criteria to Ensure the Quality of the Evaluation Report (copied below) and found in Appendix I of the USAID Evaluation Policy (attached). More specifically, • the Knowledge Gap Table, if provided, should be in the form provided in Appendix A; • the Evaluation Plan should be in the form provided in Appendix B; • Travel Debriefs should be by phone call or email prior to country departure; Appendix A: Scope of Work 60 • the Draft and Final Evaluation Reports should follow the format and length described in Appendix C. V. TEAM COMPOSITION The technical qualifications of EET members must be matched with the technical areas of focus of the Legume IL projects. Team members must have the expertise necessary to evaluate the many sub-award projects and to address the Scope of Work topics. Each member is requested to submit a CV that demonstrates relevant experience in technical, evaluation and management skills. USAID will designate one team member as the Team Leader. Team Evaluation Planning Lead (1): a senior-level evaluator experienced in international agricultural research for development and technology dissemination with technical expertise in monitoring and evaluation methodology and a minimum of 15 years of experience. The preferred candidate will be familiar with USAID (or other donor) funded programs. The candidate will also have: a) the capacity to conduct independent program evaluation; b) a thorough understanding of research methodology; c) the ability to analyze issues and formulate concrete recommendations orally and in writing; d) be available to travel and meet the timelines for completion of the evaluation; and e) not have any conflicts of interest. Administrative/management member (1): A senior-level evaluator with a minimum of ten years of experience managing and/or evaluating multifaceted international development research and/or university-based programs. The preferred candidate will be familiar with USAID (or other donor) funded programs. A background in agricultural development, with technical expertise in a field relevant to grain legume agronomic systems in either Latin America or Sub-Saharan Africa is recommended. The candidate will also have: a) a demonstrated capacity to conduct independent research program Appendix A: Scope of Work 61 evaluation; b) an understanding of USAID’s foreign assistance goals, and its particular objectives related to collaborative research, agricultural development and food security; and c) the ability to analyze issues and formulate concrete recommendations orally and in writing. Technical team members (3): Must be experienced experts in international agricultural research for development and technology dissemination. One must have a strong background in plant breeding, preferably in legume breeding, with at least 10 years experience. Technical team members will also have demonstrated the following: a) the capacity to conduct independent program evaluation; b) a thorough understanding of research methodology; and c) the ability to analyze issues and formulate concrete recommendations orally and in writing. Disciplines of all members (5): The team members need familiarity with Latin America, Caribbean and/or Sub-Saharan Africa’s agricultural systems with the following required composition of skill sets among them: organizational development, quantitative and qualitative evaluation, agronomist/agricultural systems, social/economics background, nutrition, integrated pest management and legume breeding (or other plant breeding skills). VI. EVALUATION MANAGEMENT A) Logistics & Scheduling 1) Conference call with USAID - between February 1-5, 2016 A conference call will be scheduled between the EET and the USAID Evaluation Manager, the Legume IL’s Agreement Officer’s Representative (AOR), and other officials in the Research and Monitoring & Evaluation Divisions of the Bureau for Food Security to review the scope of work and answer questions concerning the implementation and delivery of the evaluation. 2) Desk review - between February 5-February 10, 2016 The EET will conduct a desk review of the Legume IL’s publications and materials. The purpose of the desk review is to obtain needed background and context about the Legume IL and USAID in order to complete the Knowledge Gap Table and the Evaluation Plan (see below). Documents to be reviewed will include, but are not limited to, the RFPs (request for proposals), approved program proposals, the Leader with Associates Cooperative Agreement, annual reports, work plans, program operation documentation, and funded research proposals. Team members will also familiarize themselves with the Feed the Future Global Food Security Research Strategy19 and the USAID Evaluation Policy20. 3) Knowledge Gap Table – due February 10, 2016 Based on the desk review and draft evaluation plan, if needed, the EET will provide the USAID Evaluation Management a completed Knowledge Gap Table (in the form provided in Appendix A or as otherwise agreed). 4) Evaluation Plan – Draft due February 19, 2016, Final due March 15; The EET will submit to the USAID Evaluation Manager the Evaluation Plan (see Appendix B or in such form as agreed). The purpose of the Evaluation Plan is, in part, for the EET to present their evaluation design which includes, in part, research questions, methodology for quantitative and qualitative data 19 http://pdf.usaid.gov/pdf_docs/PDACR702.pdf 20 http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf Appendix A: Scope of Work 62 collection and data analysis, work plan, timeline and proposed domestic and international travel. The Evaluation Plan should be approved by the USAID before the EET can travel and begin their field work. USAID will provide approval or request changes. 5) Domestic and international travel – to be completed by May 15, 2016 The EET will need to travel domestically and internationally to gather the needed information to implement the evaluation plan and complete this scope of work. Domestic travel is limited to one trip, up to two days excluding transit, to visit the Legume IL MO at MSU. This visit should precede all international travel other than the Pan-African Legume Meeting from February 28-March 4, 2016. International travel is limited to two separate trips per evaluator to visit international collaborators and stakeholders with the Leguem IL. The USAID Evaluation Manager must pre-approve all travel. All travel will be arranged for the EET by the USDA Invitational Travel mechanism and must be in accordance with U.S. Government travel regulations. 6) International travel debriefs – prior to country departure A short summary of data collected and preliminary findings will be sent to the USAID Evaluation Manager for each country visited before departure from that country. This is not to be a trip report, nor should time be billed to write a trip report. Instead, it is meant to provide the USAID Evaluation Manager with progress made against the Evaluation Plan. Mission outbriefs shoud be included as applicable. 7) Preliminary findings – due May 31, 2016 The EET will provide to the USAID Evaluation Manager a written summary of the preliminary findings that will be used to develop the draft evaluation report. The summary can be presented as a powerpoint presentation or other suitable written form. 8) Draft evaluation report – due June 15, 2016 A draft of the evaluation report will be submitted electronically in MS Word format to the USAID Evaluation Manager in the form provided in Appendix C (or as otherwise agreed). USAID will review the draft for content. The MO will review the draft for accuracy. All comments, corrections and suggestions for consideration should be sent to the EET by June 22, 2016. 9) Final evaluation report – due July 1, 2016 The final evaluation report, subject to approval by USAID, should sufficiently address all comments and corrections provided to the draft report and be USAID-508 compliant. An optional presentation and/or discussion of the findings may be scheduled. Appendix A: Scope of Work 63 B) Level of Effort The period of performance is February 1, 2016 through July 31, 2016. Tasks Eval. Planning Lead (1) Administrative Team Member (1) Technical Team Member (3) LOE Days LOE Days LOE Days Conference Call/Desk Review 3 4 4 Knowledge Gap Table 1 1 1 Evaluation Plan 4 3 3 Travel & Travel Debriefs 9 21 21 Preliminary Findings 0 4 4 Draft Report 2 8 5 Final Report 1 4 2 TOTAL LOE 20 45 40 Appendix A: Scope of Work 64 SOW Appendix A: Knowledge Gap Table Key Knowledge Knowledge Gaps Program Management Technical leadership Administration Financial management Research Program Depth, breath, rigor Collaboration, outreach, technology dissemination HICD Future of Program Appendix A: Scope of Work 65 SOW Appendix B: Evaluation Plan FTF Activity/Mechanism Name: FTF Activity Country/Countries: Evaluation Lead Investigator: USAID Evaluation Manager: Approximate start date: Preface This document describes the components needed to complete an Evaluation Plan for Feed the Future (FTF) Activities. A. FTF Project Evaluation Design a. FTF Activity/Mechanism Description Describe the FTF activity/mechanism being evaluated. Provide enough detail to make clear the justification for the proposed methodology. Include the following items: activity/mechanism goals and objectives, main program components/interventions and delivery mechanisms, key activity/mechanism outcomes and indicators, target areas and target population groups, criteria for selecting target areas, criteria for selecting program participants, program implementation plan (start date, duration, deployment plan and timeline).(Note: much of this material can come from project documents.) b. Program Logic Please include either a diagram and/or a narrative that describes the program logic and articulates the causal pathways from activity implementation to the desired impacts. The description should include intermediate outcomes that would change along the way to final impacts or objectives of the project. (Note: this should also be available in project documents.) c. Evaluation Research Questions Succinctly state the primary questions that the evaluation will seek to answer.(Note: this should be available in the evaluation SOW.) d. Methodology for Quantitative and Qualitative Data Collection Please indicate briefly the methods and plans for data collection. This section should include all methods for primary collection (interviews, surveys, direct observation, etc.) and secondary data collection (project documents, performance reports, etc.). Provide the timing of any qualitative and quantitative data collection and explain how the two will be integrated. Include the number of planned survey rounds as well as the expected local data collection partner if applicable. Appendix A: Scope of Work 66 e. Methodology for Quantitative and Qualitative Data Analysis Describe the methods you will use to analyze the quantitative and qualitative information collected. Analysis methods should be described in detail for both quantitative (descriptive statistics, regression analyses, etc.) and qualitative (domain analysis, network analysis, etc.). Also, specific software that will be used should be mentioned (SPSS, STATA, ATLAS, etc.). f. Outcome Measures Briefly discuss the outcome measures that will be used for this study (quantitative and qualitative) and relate them to the evaluation research questions. Explain which evaluation questions the quantitative and qualitative data will help address and how. Define the variables or indicators that will be used to measure these outcomes. (A quantitative example would be an outcome measure of “Greater access to new technologies among partner developing countries” and corresponding indicator “Number of new technologies under research, field testing or made available for transfer”. A qualitative example would be an outcome measure of “Effective management” and corresponding indicator of “Communication processes are well-established”.) g. Additional Pertinent Information Use this section to describe any further information that is pertinent to this particular evaluation and should be considered as part of the evaluation design. For example, this section could be used to discuss collaboration agreements for analysis with other institutions or overlaps with other evaluations and coordination with those evaluations. Appendix A: Scope of Work 67 Evaluation Work Plan (adapt timeline as required) Activities Dates of Activity 1st Month 2nd Month 1 2 3 4 5 6 7 8 TASK 1. Develop evaluation design and implementation plan Activity 1: Activity 2: etc. TASK 2: Data Collection Activity 1: Activity 2: etc. TASK 3: Data Analysis Activity 1: Activity 2: etc. TASK 4: Report Writing Activity 1: Activity 2: etc. Appendix A: Scope of Work 68 B. Evaluation Budget (if applicable) Submit a detailed budget with the evaluation design covering all costs related to conducting the evaluation, including data collection, labor, travel, and communications. 1. Budget Summary (adapt timeline as required) Category Month 1 Month 2 Total Labor Travel and subsistence 21 Data collection Equipment Other costs Sub-total Indirect costs Total C. Data Collection and Management Plan 1. Interviewer/Enumerator Training (if any) Describe the plans for training for all data collection (if any), including length of training, location, expected number of participants, topics covered, and the approach to piloting or field testing during training. 2. Data Management and Security Describe how all data collected will be gathered, entered, managed, and stored. Please specify how data will be kept secure. 3. Data Collection Approvals Describe the process and results of all data collection approvals. D. Data Collection Instruments Submit a draft of any data collection instruments that will be used for the evaluation. 21 USAID will provide the airfare costs. All per diem and M&IE are to be based on U.S. Government rates. Appendix A: Scope of Work 69 SOW Appendix C: Report Format Title Page Table of Contents List of Acronyms List of Tables List of Figures Executive Summary (3 pages) Program Management (15 pages) - Findings - Conclusions - Lessons Learned - Recommendations Research Program (15 pages) - Findings - Conclusions - Lessons Learned - Recommendations Program Future (5 pages) - Program Management - Research Program Appendices A. Scope of work B. Evaluation Plan C. Travel itinerary, locations and dates of field visits D. List of persons contacted [EDITOR’S NOTE: Redacted for confidentiality] E. List of materials reviewed F. Photographs: high resolution with caption and photo credit (5 photographs) Appendix B: Evaluation Plan 70 Appendix B: Evaluation Plan FTF Activity/Mechanism Name: Legume Innovation Lab FTF Activity Country/Countries: Benin, Zambia, Malawi, Honduras, Tanzania, Uganda, Haiti, Mozambique, Niger, Senegal, Burkina Faso, Ghana, Guatemala Evaluation Lead Institution: Virginia Evaluation Team: Jeffrey Alwang, Steven Temple, James Myers, Jessica Fanzo, Eric Welch USAID Counterparts: Jennifer Long; Lisa Wilson Approximate start date: Feb. 9, 2016 Date originally submitted: Feb 19, 2016 Date of revision (version): Sep 2016 Appendix B: Evaluation Plan 71 A. FTF Project Evaluation Design 1. FTF Activity/Mechanism Description a. Introduction In compliance with the USAID Evaluation Policy (2011) and the Scope of Work for the Evaluation of the Feed the Future Innovation Labs for Collaborative Research on Grain Legumes (2013-2017) (LIL) this document: • presents an overview of the LIL including: current organization of the projects and their leadership, project goals and objectives, and activities to date; • proposes a set of key indicators and data sources necessary to complete the evaluation; • identifies information already available through project documents and interviews with project leadership; • identifies gaps in information necessary to evaluate the programs including questions to be used in evaluation and additional sources of data necessary to address the indicators. • outlines a design for the collection of additional data and a plan for analysis. The Scope of Work (SOW) for this evaluation instructed the review team to evaluate: (a) the research program, (b) LIL program structure and management, (c) the institutional capacity collaboration, and (d) future directions for the LIL. The SOW has identified evaluation questions for each of the four evaluation topics (a-d) which inform this evaluation plan and provide guidance to the External Evaluation Team (EET). The evaluation team will produce an evaluation report that will be used by USAID/BFS/ARP-R as a mechanism to demonstrate accountability to stakeholders and to inform a future RFA to address outstanding research questions connected to grain legume productivity for smallholders in developing countries and associated value chains (SOW 2016). b. The program and projects The Legume IL, began as the Bean/Cowpea CRSP (1980–2007) and then became the Dry Grain Pulses CRSP (2007-2012) with funding from USAID. The Legume IL was awarded $14,014,000 in 2013 as a Leader with Associates Cooperative Agreement for a period of 4.5 years. The Legume IL research is both similar and different from its predecessor. The Legume IL proposed to refocus and redirect research on priority areas with high promise for significant technological gains and the potential for widespread impacts on legume value chains in developing countries. Important structural considerations for the evaluation of the Legume Innovation Lab program for FY 2013-2017 include: a. Partial change in research project structure in which seven of the then projects are continuations of prior projects with significant revision and redirection as recommended Appendix B: Evaluation Plan 72 by the TMAC and requested by USAID. Three new projects were started as part of the 2013 LIL funding. For the seven continuing projects, the PIs remained in place. b. A focus on priority legume-based cropping systems and countries/regions that are aligned with FTF priorities, and c. An effort to broaden the expertise to include a new generation of US and partner country scientists; greater embedded multidisciplinarity in some projects including the addition of social scientists; and improved integration with CGIAR partners as a means of achieving broader global impacts. d. Specific innovations for the new LIL program: § Phenometrics for identifying new physiological and biochemical traits to enhance photosynthesis and assimilate transport to reproductive structures, and to improve the effectiveness of the plant to acquire water and nutrients from the soil under agroecological conditions typical of smallholder, resource-poor farms § Identification of Single Nucleotide Polymorphism (SNP) markers for QTLs of important physiological traits utilizing association mapping § Introgression of genes for drought tolerance and disease resistances from other legume species (e.g., Phaseolus accutifolius, P. coccineus) § Development of dynamic, virtual, interactive internet-based maps of insect pest population changes and crop damage in West Africa § Development of decision tools for low-literacy farmers to improve IPM and sustainable integrated soil fertility management of legume-based cropping systems § Identification of appropriate governance structures to improve legume value chain functions and to increase market access for smallholder grain legume farmers § Increased understanding of the role of grain legumes in diets for gut microbiome health and function (nutrient absorption) to enable the development of more effective education and food interventions to improve nutrition for infants and children, and women of childbearing age. The overarching goal of the Legume IL – to substantively increase the potential for grain legume productivity and sustainable intensification of smallholder farm systems as a way to increase the availability of affordable grain in domestic markets, increase consumption of legumes by the poor, and improve nutrition and nutritional security in developing countries – is operationalized through four strategic objectives. These include: a. Increasing grain legume productivity by improving adaptation to diverse agroecologies and reducing smallholder farmer vulnerability to climate change, with special consideration for the livelihoods of women (SO1); Appendix B: Evaluation Plan 73 b. Transforming grain legume-based systems through improved smallholder production management decision making and more effectual governance management of legume value chains (SO2); c. Enhancing nutrition to improve the nutritional quality of diets and to enhance the nutritional and health status of the poor through the consumption of edible grain legume-based foods (SO3); d. Improving outcomes of Legume Innovation Lab research and capacity building projects and to assess impacts to improve decision making regarding future investments (SO4). These four objectives are addressed through research and other activities undertaken in one or more of the ten funded LIL projects. An organization chart presenting the structure of the project is presented in Figure 1. The evaluation will focus primarily at the project level to assess how research activities under the various SOs are progressing and to examine outputs and outcomes of the research. Additionally, the EET will assess program level outcomes and cross-project interaction and activities, outputs and outcomes. Other cross-cutting components and aims are embedded within the LIL program description including: (a) commitment to training and capacity development (b) establishment of associate awards, (c) focus on gender, (d) strengthening capacity of agriculture research institutions in USAID FTF priority countries, and (e) increased coordination with the CGIAR through the CRP3.5 on Grain Legumes and Africa RISING. The evaluation will also consider these cross￾cutting program elements. Despite the history of USAID funded research, the scope of this evaluation comprises only the activities, outputs and outcomes of the LIL from 2013 when the LIL was funded by USAID until the present. Appendix B: Evaluation Plan 74 Figure 1. Organization Chart of the Legume Innovation Lab Managing Office Michigan State University Director and Program PI: Dr. Irvin Widders Co-Director: Dr. Cynthia Donovan Technical Management Advisory Committee External Evaluation Committee Project team leads US Institutions Project team international partner institutions SO1: Advancing the Productivity Frontier SO2: Transforming Grain Legume Systems & Value Chains SO3: Enhancing Nutrition SO4: Improving Outcomes of Research & Capacity Building Strategic Objective Other Integrated Project Dimensions Associate Award: MASFRIJOL Legume Scholars Program Leveraged Grant: Precision Integrated Pest Management for Smallholder Cowpea Farmers in West Africa (BMGF) SO1.A1 NDSU ICTA Guatemala Other US Team Partner Institutions SO1.A2 MSU UNZA Zambia ZARI Zambia UC Riverside SO1.A3 MSU ZARI Zambia NaCCRI Uganda Univ. of Neb. USDA-ARS SO1.A4 UPR EAP Zamoro Honduras ICTA Guatemala NSS Haiti USDA – ARS NDSU SO1.A5 UCR INERA Burkina Faso SARI Ghana ISRA CERRAS Senegal USDA - ARS SO1.B1 UIUC IITA Benin INERA B. Faso INRAN Niger CSIR-SARI Ghana INRAB Benin SO2.1 ISU Makerere U. Uganda NARL Uganda IIAM Mozambique U. Hawaii; UIUC SO2.2 KSU UNZA Zambia LUANAR Malawi SUA Tanzania SO3.1 WUSL U. Malawi Malawi SO4.1 MSU Other Legume Labs Appendix B: Evaluation Plan 75 2. Theory of Change a. Adapting the Agriculture for Development Model of Change for the Legume Innovation Lab The research program operating through the Legume Innovation Lab (LIL) comprises multiple projects organized under strategic objectives and complemented with Associate Awards and other activities. Each of the different activities is built on a logic of its own. In particular, each of the ten projects has developed an individualized impact pathway that links inputs, outputs, and outcomes. The LIL projects and associated activities can be integrated within a broader framework that maps the linkages that occur over time among research activity with key outcomes including the uptake and diffusion of research outputs to address broader aims as increasing agricultural productivity, improving nutrition, developing new crop varieties to address the effects of climate change and reducing poverty. The Agriculture Innovation System Model of Change (AISMC) (Arnold, E., and M. Bell. 2001; World Bank 2007) provides one way to depict the relationships among the demand environment, research, intermediaries and support structures and target groups found in the enterprise domain. Adapting this generic framework, which is taken from the A4D literature, for the Legume Innovation Lab provides a necessary transition between theoretical and operational for the evaluation. Figure 2 presents an adapted version of the AISMC specifically tailored for the LIL. The model recognizes the important partnership basis of the project, but also links specific research activities outlined in the impact models. US and partner country collaborators are involved in the design and execution of research, research is multidisciplinary including social science and gender specific inquiry is often part of the explicit research process. Partnership based research is guided by broad needs identified by the USAID Feed the Future program that include increased agricultural output, gender specific strategies, effective responses to climate change, improved nutrition and reduced poverty, among others. Recognizing both the USAID programmatic needs and the contexts within which partner organizations are based, LIL teams undertake translation efforts to bridge research outputs for improved uptake and diffusion. Such activities are project specific, location specific, gender specific and often aimed, through IARCs and others, for diffusion outside of the LIL countries. The model also recognizes the importance of education and training activities that are important both as enablers and for attaining LIL’s capacity development aims. Appendix B: Evaluation Plan 76 Figure 2. Legume Innovation Lab Partnership Model of Change b. Linking the Theory of Change to the LIL Project Logic Models Impact pathways (or logic models) represent broadly accepted approach to linking inputs and activities to outputs to outcomes to impacts. Inputs are the financial, scientific and human resources, financial, human necessary to implement the proposed research. Activities are broadly conceived as mechanisms that link objectives to actions. Outputs are immediate products of project activities while outcomes represent the effect that outputs have on the broader context or environment at which the activities are aimed or embedded. Impacts are the project results that, once diffused, affect broader and longer term goals such as poverty reduction, in the case of US AID Innovation Labs. Note: it is unlikely that the evaluation will discern specific impacts as these are long term and the project was established approximately three years ago. A simple impact pathway model, developed by Dr. Mywish Maredia, PI of the Assessment of Dry Grain Pulses CRSP Investments in Research, Institutional Capacity Building Diffusion, uptake and eventual impact Uptake of explicit knowledge, products and processes; gender relevant 1. Diffusion of new varieties 2. Availability to farmers in partner countries and elsewhere 3. Farmers purchase of new varieties 4. Productivity increases Demand / Need 1. Feed the future priority countries 2. Need for greater agricultural output 3. Need for nutritional value 4. Climate related challenges LIL Translation –partnerships with research organizations, NGOs, farmers, and others including Associate Award institutions. Partner based integration of knowledge outputs to design viable intermediary activities, processes and products 1. Demonstration, participatory appraisals 2. Diagnostic aids 3. Gender specific approaches 4. Use of research markers, varieties, etc. by breeders 5. Variety release LIL Research – project based research partnerships Partnership based collaboration, US and international universities, national research institutes and organizations, associate programs, other partners including IARCs. Projects produce research outputs that contain new knowledge and are motivated by broader demand a. Collection of materials b. Experiments c. Selection d. disease screening e. field testing f. Identification of best lines g. Marker identification Support structures A. Professional and collaboration networks including trade and farmer associations B. Education system – training and capacity development C. Diffusion networks Appendix B: Evaluation Plan 77 and Technology Dissemination in Africa, Latin America and the U.S project under SO4 is presented in Figure 3. Based on this model each of the 10 different projects developed their own impact pathways. Figure 3. Impact Pathway Generic Model (Mywish Maredia and Byron Reyes, June 2013) c. Specific Impact Models as Guides The Legume Innovation Lab’s approach to a research process that links activities to outcomes and to the strategic objectives of the project are articulated in the program-level impact models. An example of one project impact model is presented in Figure 4. These impact models will provide guidance to the EET team as it collects data, identifies knowledge gaps and assesses project-level as well as program level progress. Inputs/Activities Investment in collaborative research projects Outputs Technologies Products Services Practices Intellectual property Outcomes •Increase production, consumption, dissemination of outputs, products and services •Decrease per unit costs of production, Impacts •Poverty reduction •Hunger reduction •Health / Nutrition improvement Adoption / Uptake / Influence Scaling up and scaling out Appendix B: Evaluation Plan 78 Figure 4. Example Impact Pathway Appendix B: Evaluation Plan 79 3. Focus of the Evaluation Four main areas of interest have been identified for this evaluation (from the Scope of Work): a. Research Program 1. What contributions have the Legume IL research activities made to developing research outputs that could contribute to higher level impacts on the program goals of: a. substantively increasing grain legume productivity through sustainable intensification of smallholder farm systems, b. increasing the availability of affordable grain in domestic markets, c. increasing consumption of legumes by the poor, and d. improving nutrition and nutritional security in developing countries? Note: Given the early stage of this research we will focus on outputs and outcomes while also assessing the potential for the LIL projects to accomplish anticipated longer term impacts. 2. To what extent did the project generate robust and quality research outputs using disciplinary-appropriate metrics? Are the outputs relevant for a research-for￾development project (e.g., did they generate new breeding lines of relevance to FTF countries)? Were the projects undertaken using the right tools/technologies and were they well executed? 3. What research-related challenges has the Legume IL faced during research design and implementation? In what ways have those challenges been addressed? What opportunities are there to improve research outputs moving forward? 4. Given the dual benefit mandate of Title XII authorized programs, such as the Legume IL, is there a sufficient balance between research efforts directed towards priorities of the domestic bean/cowpea stakeholder community and the bean/cowpea research priorities required to advance global food security goals, particularly in FTF countries? 5. Has the Legume IL met its academic training and technical capacity strengthening targets? Are the appropriate type and number of people being targeted for the right kind of training? In what ways has the program identified and addressed academic and technical capacity needs? In what ways could this be improved? b. Program Structure and Management 1. How effectively has the Legume IL MO facilitated communication, coordination and implementation amongst sub-awardees and stakeholders to carry out the research and training activities in the U.S., Latin America and Africa? In what ways could communications and coordination be improved as a way to advance project goals and objectives, create a shared research agenda, meet reporting requirements and encourage additional relevant partners? Appendix B: Evaluation Plan 80 2. In what ways did the model of having one lead university serve as the MO to manage a global program of multiple interdisciplinary projects contribute to effective implementation and program objectives? In what ways could this model be improved to better meet the Legume IL’s objectives? What strengths or synergies did this model encourage, e.g., among the individual projects to ensure complementarity of effort? For example, did the MO use research outputs to shape its outreach or did the MO serve in its role of bringing information to people's attention as appropriate across the projects? c. Institutional Capacity Collaboration 1. What types of partnerships were established to address the different aspects of the activity, including any partnerships with USAID Missions, CGIAR centers and NGOs? What types of productive collaborations were established with other stakeholders, such as host country government and academic institutions, local NGOs, other FTF Innovation Labs and other relevant USAID activities in target countries? Other relevant international research programs working in the target countries (e.g., PABRA (CIAT) and IITA)? What type of opportunities are there for other partnerships and collaborations to further the Legume IL’s objectives? d. Program Future 1. What types of changes in implementation would more effectively achieve outcomes in the following components: research (i.e., design, implementation, communications, stakeholder involvement), capacity development (i.e., student recruitment and selection, content, location) and/or institutional collaboration? 2. Based on this program and the general knowledge generated across the legumes research community, what specific technical areas within grain legumes merit new or continuing research investment (e.g., breeding investments vs economic analysis vs nutrition research) or geographic considerations (e.g., what is value of having Central American and African researchers in one program vs two separate investments?) e. Cross-cutting themes 1. To what extent are other cross-cutting components and aims effectively embedded within the LIL program. These include: (a) integration and effectiveness of Associate Awards and (b) distinctive consideration of gender Appendix B: Evaluation Plan 81 4. Methodology for Qualitative and Quantitative Data Collection a. Details of the data collection plans The scope of this review covers four main areas outlined in the Scope of Work (SOW) for the ten projects of the LIL. Each project review will be conducted using a similar methodology, guided by the same set of SOW questions. As each project is designed to address different strategic objectives and because the collaborating partner countries and institutions vary across projects, it is necessary to consider the unique trajectory and contribution of the projects. For all projects, the EET will conduct interviews of the PI, CoPI, collaborating scientists and stakeholders. Cross cutting dimensions of the program will also be examined as part of the evaluation. Each of the areas contains cross-cutting themes that will need to be evaluated at the project level. Approximately 50 interviews will also be conducted of students, postdocs as possible. Focus groups may be possible during some site visits, particularly if there are a large number of collaborators present. All interviews will be conducted using formally constructed interview protocols with main questions and probes. All interviewees will be assured of confidentiality. Interviews will be conducted in pairs; each EET member will write notes, either handwritten or entered in a computer, which will then be transformed into a formal record of the interview. More on the process of data collection is presented below. At the program level, the EET will also conduct a review of the management office (MO) and undertake interviews of MO personnel. Additionally, all project interviewees and stakeholders will be asked about program level accomplishments, outcomes and future expectations. EET team members will travel to MSU for the review. Project and program review data will also be collected at the Pan-African Grain Legume and World Cowpea Conference to be held in Livingstone, Zambia from February 28 to March 4, 2016. Given the scope of the conference, four of the EET members will travel to conduct interviews at the conference. The LIL has also developed two associate awards, one on integrated pest management (IPM) for smallholder cowpea farmers in West Africa and another called MASFRIJOL which designed to mobilize the expertise and experience of the Legume Innovation Lab and its network of partners to help the USAID Mission to Guatemala achieve its Feed the Future strategic development goals. Given that a separate evaluation review of the IPM project in Benin has already occurred, this evaluation will consider only the MASFRIJOL associate award. At least two members of the evaluation team will travel to each of two key countries included in this review: Uganda and Guatemala. Uganda is of importance as it is the location of two large LIL projects: S01.A3 and S02.1. Team members will spend 3 days each in country conducting interviews addressing questions outlined in the Scope of Work. Table 1 presents the overall evaluation questions linked to indicators and proposed sources of data. The evaluation will draw from multiple data sources - program documents, interviews with key informants, focus groups, and other outputs. Appendix B: Evaluation Plan 82 Table 1. Linking Evaluation Questions, Knowledge, Indicator and Source of Data Knowledge/Outcome Illustrative Indicators Source of Data Research Program RP 1. Contributions of the Legume IL research activities to developing research outputs that could contribute to outcomes related to the program goals. (note that long term impacts are likely not visible given the 2-3 year duration of the LIL.) - Increased grain productivity through sustainable intensification Production quantities; project and partner activity Varieties released, production differential Annual reports, interviews with PIs and Management Office - Increased availability of affordable grain in domestic markets Existence of and access to affordable grain in project areas; availability of new varieties; arrangements to increase availability; willingness of farmers to take up new varieties; dissemination of new varieties to other non-project localities Production quantity, distribution patterns, programs for access, prices, partner identification of initiatives and efforts Review reports; interviews with PIs, CoPIs, Partners - Increased consumption of legumes by the poor Consumption patterns of the poor; program design and implementation Evidence of changes in consumption in project areas; whether alignment between factor known to affect consumption and project aims Interviews and possibly focus groups of local stakeholders, international partners; reports and other documentation - Improving nutrition and nutritional security in developing countries? To what extent have project outputs addressed nutrition; how are programs progressing toward accomplishing this Research outputs related to nutrition; program components aimed at diffusion and uptake Interviews of PIs, CoPIs, partners; bibliometric studies RP 2. To what extent did the project generate robust and quality research outputs using disciplinary-appropriate metrics? - Are the outputs relevant for a research-for-development project (e.g., did they generate new breeding lines of relevance to FTF countries)? Outputs by category, benchmarks from other projects Varieties released by relevance and appropriateness, etc. Review of reports, interviews with PIs, CoPIs, Partners, TMAC. Site visits. - What knowledge outputs have been produced? Knowledge outputs, dissemination and uptake. publications, especially peer reviewed, reports, Bibliometric analysis Appendix B: Evaluation Plan 83 - Were the projects undertaken using the right tools/technologies and were they well executed? Information about tools and techniques available; selection decisions; information on results Effectiveness of project activity; problems encountered; options not taken Interviews of PIs, CoPIs, partners, TMAC; project documents. Site visits. RP 3. What research-related challenges has the Legume IL faced during research design and implementation? In what ways have those challenges been addressed? What opportunities are there to improve research outputs moving forward? Obstacles or challenges faced Evidence of challenges from interviews, reports. Success of work-arounds. New directions, opportunities lost. Interviews of PIs, CoPIs, partners, TMAC; project documents. Site visits. RP 4. Given the dual benefit mandate of Title XII authorized programs, such as the Legume IL, is there a sufficient balance between research efforts directed towards priorities of the domestic bean/cowpea stakeholder community and the bean/cowpea research priorities required to advance global food security goals, particularly in FTF countries? Range of outputs and applicability to local as well as broader contexts Evidence of locally relevant outputs, uptake. Evidence of IARC collaboration, use of outputs. Evidence of value of outputs to domestic US research Interviews of PIs, CoPIs, in country partners and other stakeholders, TMAC; progress reports, highlights. Site visits. RP 5 Has the Legume IL met its academic training and technical capacity strengthening targets? Are the appropriate type and number of people being targeted for the right kind of training? In what ways has the program identified and addressed academic and technical capacity needs? How could this be improved? Training types and numbers relative to targets. Legume Scholars Program. Degrees received, current training, non-degree training. Scholarships awarded. Students enrolled. Review reports; interviews with PIs. Student interviews. Site visits. Targeted training How potential degree recipients are targeted, how degree training is prioritized. Legume Scholars Program. Program of study for LSP. Evidence of training activity. Effectiveness/outcomes of training. Review reports; interviews with PIs (US and HC PIs) Partner interviews. Site visits. Program Structure and Management PS 1. How effectively has the Legume IL MO facilitated communication, coordination and implementation amongst sub￾awardees and stakeholders to carry out the research and training activities in the U.S., Latin America and Africa? In what ways could communications and coordination be improved as a way to: Advance project goals and objectives Project and program communication processes; Are PIs aware of what is being done at other sites Evidence of communication and coordination effectiveness; problems; team perception Documents; review reports; interviews with PIs (US and HC PIs) Partner interviews. Site visits. Appendix B: Evaluation Plan 84 Shared research agenda Decision making processes Perceptions about inclusion; perceptions about team approach, sharing of priorities, results, findings? Documents; review reports; interviews with PIs (US and HC PIs) Partner interviews. Site visits. Meet reporting requirements Reporting requirements & associated communication Timeliness, completeness and distribution of reporting Documents; review reports; interviews with PIs (US and HC PIs) Partner interviews. Site visits. USAID interviews. Encourage additional relevant partners Outreach and engagement activity Evidence of potential partners not included; problems with external perception of program Review reports; interviews with PIs (US and HC PIs) Partner/stakeholder interviews. Site visits. PS 2. In what ways did the model of having one lead university serve as the MO to manage a global program of multiple interdisciplinary projects contribute to effective implementation and program objectives? In what ways could this model be improved to better meet the Legume IL’s objectives? What strengths or synergies did this model encourage, e.g., among the individual projects to ensure complementarity of effort? For example, did the MO use research outputs to shape its outreach or did the MO serve in its role of bringing information to people's attention as appropriate across the projects? Institutional Capacity Collaboration IC 1. What types of partnerships were established to address the different aspects of the activity, including any partnerships with USAID Missions, CGIAR centers and NGOs? What types of productive collaborations were established with other stakeholders, such as host country government and academic institutions, local NGOs, other FTF Innovation Labs and other relevant USAID activities in target countries? Other relevant international research programs working in the target countries (e.g., PABRA (CIAT) and IITA)? What type of opportunities are there for other partnerships and collaborations to further the Legume IL’s objectives? Existence and quality of partnerships; partnership activities and outcomes Quantity of partnerships by type and function; length of partnership; partnerships gained and lost; perceptions about partnership quality, roles, and effectiveness; future potential for growth and new directions; needs and missed opportunities; evidence of scale up and dissemination of project outputs by partners Interviews with PIs, CoPIs, Collaborators, Partners, TMAC, USAID. Site visits. Other organizations as identified during the interviews or document review. Appendix B: Evaluation Plan 85 Program future PF 1. What types of changes in implementation would more effectively achieve outcomes in the following components: research (i.e., design, implementation, communications, stakeholder involvement), capacity development (i.e., student recruitment and selection, content, location) and/or institutional collaboration? Knowledge about program directions relative to needs; understanding about scientific directions; assessment of training needs; new food security threats or concerns; recognition of lack of attention paid by other research efforts; partner capacity; program level outcomes Gaps, barriers, opportunities, challenges, strengths and weaknesses of projects; opportunity to leverage capacity of partners toward new directions; evidence of cross￾project innovations; Interviews with PIs, CoPIs, Collaborators, Partners, TMAC, USAID PF 2. Based on this program and the general knowledge generated across the legumes research community, what specific technical areas within grain legumes merit new or continuing research investment (e.g., breeding investments vs economic analysis vs nutrition research) or Geographic considerations: e.g., what is value of having Central American and African researchers in one program vs two separate investments? Cross-cutting themes To what extent are other cross-cutting components and aims effectively embedded within the LIL program. These include: (a) integration and effectiveness of Associate Awards and (b) distinctive consideration of gender Knowledge about integration of cross￾cutting themes Evidence of Associate Award activity and outputs; evidence of gender integration Associate Award interviews; site visits; PI CoPI interviews Appendix B: Evaluation Plan 86 b. Document review and secondary sources The evaluation team is in the process of conducting a full review of program and project documents, including: • program-level technical application • project-level technical applications • annual reports • technical progress report (program and projects) • annual work plans • program highlights and project briefs • other documents summarizing project level work collected from the website and the MO • DGP CRSP archives, CRSP evaluation, response to evaluation The team has also reviewed the LIL website, its organization and the materials provided on the website. Some members of the research team will review a selection of the journal articles published by LIL collaborators. Depending on the findings from the full review, the EET will request additional materials from the MO and project PIs as needed. The EET will also request any additional contact information not contained in the directory posted on the LIL website. For example, the EET may request contact information for graduate students or postdocs employed on the project. Additionally, the EET may request additional information on stakeholder names and contact information. The EET will conduct a desk review of all collected documents to identify key knowledge gaps. The desk review of each project will be conducted according to the project review template (Table 2). Summary knowledge gaps will be entered into the table which will help guide the remainder of the project. Appendix B: Evaluation Plan 87 Table 2. Project Review Template Outline To facilitate broad and efficient review of the IL projects, EET members will take on specific tasks as primary or secondary reviewer of each of the 10 research projects (RP) (see table). The primary and secondary reviewers will be responsible for the in-depth analysis of the RP in question. The review itself will be comprised of the desk review and follow-up interviews. The purpose of the desk review is to help identify knowledge gaps, answer evaluation questions, and to inform the subsequent interviews. The following outline will help make comparisons and ensure that the review is consistent across projects. The two reviews – primary and secondary - will be compared for consistency. When there is disagreement or gaps, the two reviewers will confer to discuss rationales and support for findings. The two reviewers will make final decisions based on consensus. 1) Development impact potential. Contributions or potential contributions of the Legume IL research activities to developing research outputs that could contribute to higher level impacts on the program goals (refer to four higher-level goals). Please make an attempt to demonstrate how progress against specific objectives will contribute to change as per theory of change (which we should have soon). 2) Quality of science. To what extent did the project generate robust and quality research outputs using disciplinary-appropriate metrics? What evidence is there of this? Are the outputs relevant for a research-for-development project (e.g., did they generate new breeding lines of relevance to FTF countries)? Please highlight and describe most important outputs. Were the projects undertaken using the right tools/technologies and were they well executed? 3) Challenges faced. Please document these, where possible, during desk review. Interviews should build on these findings. 4) Impacts in the US. Is there any evidence of IL-related outputs benefiting US producers or consumers? 5) Training accomplishments. Information needed on degrees awarded and current positions of graduates (host-country, other developing country, US). Non-degree training (type and number) will also be evaluated. Please comment on the balance and appropriateness of this training. • Appropriate mix of training topics and appropriate training method • Gender balance 6) Partnerships. Please describe existing partnerships and their quality, identify potential partnerships. Identify any sharing of information and spillovers from IL partners. 7) Project-specific questions for interviews. Appendix B: Evaluation Plan 88 c. Key informant interviews and focus groups The Evaluation team will conduct a series of semi-structured interviews with key informants stakeholders in Zambia, Uganda, Guatemala, and the USA (Table 4). Interviewees will include: • Project PIs and CoPIs • US collaborators on research projects • International collaborators on research projects • Partner organizations including IARCs, local government and NGO partners, and other stakeholders • TMAC members • MO personnel including the LIL director • USAID Mission and other in-country staff The evaluation team already has conducted an informational unstructured interview with the LIL program director. Members of the team, including the EET lead have been able to discuss the evaluation with the USAID representatives. The EET has met by conference call and will meet in Zambia to review all plans before beginning the interviews on February 29, 2016. When, during site visits, there are a large number of key informants present it may be appropriate to focus groups with small sets of similar stakeholders – students, international collaborators, local stakeholders. Focus groups would not substitute for key informant interviews. Table 3 presents the outline of data collection activities, particularly those related to interviews, focus groups and site visits. Table 4 lists individuals representing stakeholder groups from which the EET will select a sample of key informants for interviews. Section C contains an initial set of preliminary interview protocols to be used in semi-structured interviews with key informants. d. Bibliometric Analysis The LIL program, including all projects has produced a number of journal articles. As part of the desk review and follow-up inquiry, the EET will collect the full set of journal article references attributed to the LIL by its collaborators. The EET will then collect bibliometric data publication and citation data related to the journal articles. Bibliometric data are drawn from a standard source: Thomson Reuters ISI Web of Science (WoS). The team will also search Google Scholar to assess the extent to which LIL academic and gray literature (reports, white papers, etc…) are cited. Appendix B: Evaluation Plan 89 Analysis will be of three types: quantity of outputs, quality of the outlets of LIL publications as measured by journal impact factors and citation patterns of LIL outputs. Journal impact factors are one measure representing quality. For this analysis, we will only examine the number (total citations) and quality of citing journals. Because LIL has only be operational for around three years, and given the time required for citations, the analysis will be able to estimate near term impact. Table 3. Project specific data collection plan Project Name Interviews of PI and Co￾PIs during Pan-African Legume Conference (PALC) Anticipated site visits to review achievements Possible additional interviews of collaborating scientists in the US after PALC Interviews of collaborating international scientists Possible focus group of international scientists at PALC if attendees numerous Genetic Improvement of Guatemalan Climbing Beans for Efficient Production in the Highland S01.A1–NDSU ü Travel to Guatemala Phone/skype ü ü Improving Photosynthesis in Grain Legumes with New Plant Phenotyping Technologies S01.A2-MSU ü Possible during MSU visit ü ü MSU Improving Genetic Yield Potential of Andean Beans with Increased Resistances to Drought and Major Foliar Diseases and Enhanced Biological Nitrogen Fixation (BNF) S01.A3-MSU ü Travel to Uganda Possible during MSU visit ü ü Development and implementation of robust molecular markers and genetic improvement of common and tepary beans to increase grain legume production in Central America and Haiti S01.A4-UPR ü Travel to Guatemala Phone/skype ü ü Genetic improvement of cowpea to overcome drought and biotic constraints to grain productivity S01.A5-UCR ü Phone/skype ü ü IPM-omics: Scalable and sustainable solutions for pest management of insect pests of cowpea in Africa S01.B1-UIUC ü Phone/skype ü ü Farmer Decision Making Strategies for Improved Soil Fertility Management in Maize-Bean Production Systems S02-1- ISU ü Travel to Uganda Phone/skype; TMAC member, Johannes Lehman, soil scientist at Cornell ü ü Enhancing Value-Chain Performance through Improved Understanding of Consumer Behavior and Decision￾Making S02-2-KSU ü Phone/skype ü ü Legumes, Environmental Enteropathy, the Microbiome and Child Growth in Malawi S03-1-WUSL ü Phone/skype ü Impact Assessment of Dry Grain Pulses CRSP Investments in Research, Institutional Capacity Building and Technology Dissemination for Improved Program Efforts S04-1-MSU ü Possible during MSU visit Phone/skype Appendix B: Evaluation Plan 90 5. Methodology for Quantitative and Qualitative Data Analysis Semi-structured interviews will be conducted following traditional interview methods in which interviewers aim to engage the interviewee in a substantive conversation about their research. In-depth conversations guided by interview instruments provide the opportunity for a flow of information that is not constrained by pre-determined response categories. Interviewees are both able to follow lines of inquiry unique to an individual, but also collect equivalent data across interviews. This approach allows exploration of subjects that surface during the interview and elicits information that provides nuance and insight for addressing the evaluation questions. A written interview protocol including primary questions and follow-up probes will be used as a structured guide for the interviewer to follow and to ensure all questions are asked of all respondents. Example interview protocols are presented in Section D below. Interviewers are experienced and will allow respondents to respond freely to questions. The interview protocol will be further revised based on the ongoing review of documents described above. Ideally interviews will be conducted in teams of two. The use of teams enhances creativity, enables for the convergence of observations and increases confidence in the findings, and allows the interview material to be viewed from different perspectives. In some cases, team-based interviews may not be possible. When conducted, focus groups will also follow the outlines of the interview protocol and be conducted with two members of the EET present. Interviews and focus groups with key informants will be captured in notes, which will be expanded into field notes at the end of each day of interviewing. Notes will be saved as text files and summarized at the end of the interview. Summaries should include specific insights or key findings from the interviews and focus groups. Cross-interview findings can also be noted for later analysis. On site visits, EET members will meet each evening to review notes, summarize key findings for the day and identify key questions for further exploration. During other phases of data collection, such as skype interviews of US collaborators, EET members will schedule meetings to review findings. Each of the ten projects will be assigned to one of the members of the EET as lead to synthesize notes and summarize findings. The primary EET member will collect all notes from interviews as well as all background materials for the particular project as input. Project write￾ups will address the evaluation questions established for the evaluation, integrating key indicators from Table 1 with evidence from interviews. All individuals who attended the interviews will be asked to review the individual project write-ups, include additional comments and identify inconsistencies. When all project write-ups are complete, the team lead will develop an overall program assessment that takes into account the individual projects and the overall program-level and cross-project outputs and outcomes. Bibliometric analysis will also be integrated. Conclusions will include accomplishments and shortcomings. Based on the evidence and considering project goals, the EET will develop a set of recommendations to assist USAID and LIL in the future. Appendix B: Evaluation Plan 91 6. Methodological Limitations a. Sampling and participant selection bias In order to develop a list of potential interviewees from among the various stakeholders in these programs, we have solicited a list of active and past staff, investigators, partners, host country officials and academics. The LIL website and documents provide lists of stakeholders. The list can be found in Table 4. Additional names and contact information will be collected throughout the evaluation for follow-up. We have identified a number of critical informants with whom we will be speaking. We will interview all US PIs and CoPIs, International CoPIs and other important US and international collaborators. We will also interview TMAC members, IARC partner representatives, USAID in-country staff, and key representatives of participating government ministries. For other stakeholder groups identified as critical to the evaluation, we will select individuals on a purposive basis considering 1) level of involvement with the project, 2) breadth of knowledge of program activities, 3) representativeness of the range of diversity of stakeholders, 4) availability during the time available to the evaluation team in each host country or by Skype, 5) referrals. Given the complex nature of this program and the wide range of stakeholders, current and former, development of an exhaustive population of actors involved with LIL is not possible. Moreover, given substantial differences in involvement of stakeholders, any other selection process, such as random sampling, is likely to miss important sources of information. Hence, purposive sampling is preferred both for feasibility and data quality reasons. Given the large number of planned interviews, concern about sample bias is negligible. b. Interviewee recall bias Key informant interviewing will focus primarily on the interviewees’ opinions, actions and assessments of program activities as participants and collaborators since project inception in 2013. We do not anticipate problems with recall bias given that the project has only recently begun and is still in operation. The EET can identify inconsistencies through comparison across interviews. Questions of fact can be cross-checked against documentation of events and activities available in other project materials. c. Interviewee response bias Interview based data is subject to self-report bias. We have structured this evaluation in ways to reduce the likelihood of three types of systematic response bias: acquiescence bias (i.e., the tendency to agree), demand characteristics (i.e., modifications to responses because of being “studied”), and social desirability bias (i.e., ascribing favorable traits, even if this is untrue). Interview protocols ask open-ended questions and probes are constructed to inquire about multiple perspectives on an issue and similar questions are asked of multiple project members such that it will be possible to recognize issues of response bias. In analysis we will review Appendix B: Evaluation Plan 92 responses to interview questions such that the responses of different stakeholder’s and different individuals from within stakeholder groups can be cross-checked for convergence or disagreement. Through triangulation of assessments and understandings of different individuals we can come to a set of conclusions that identify areas of consensus and areas of disagreement. While eliminating response bias is difficult, triangulation greatly reduces the concern. d. Interviewer bias Several sources of interviewer bias are particularly important to be aware of during interviews. Facial expression or inflection of the interviewer can bias responses, encouraging or discouraging particular lines of conversation. In international contexts, race and ethnicity of the interviewer can bias responses of interviewees such that interviewees may be more open with individuals who are more like them. Interviewees can also make the mistake of presenting leading questions or ‘putting words into people’s mouths’. Interpretation of data can become skewed toward one perspective or another although evidence does not merit it. Finally, prior involvement with a particular group or scientist can lead to biased interpretation of data. Although these forms of interviewer bias are difficult to manage, most of them can be addressed through training, interview experience and interview structure. For this evaluation, the EET has substantial interview experience which minimizes these concerns. Additionally, interviews will be team-based when possible, written materials will be shared among EET members and analysis will be vetted as a group. Additionally, to avoid any concerns about bias, the team is careful not to assign individuals to site visits when those individuals have had a high level of prior collaboration. Appendix B: Evaluation Plan 93 Table 4: List of Interviews Project Lab Position Location Administrative Advisory Committee AAC Administrative Advisory Committee Iowa AAC Administrative Advisory Committee Pennsylvania AAC Administrative Advisory Committee California AAC Administrative Advisory Committee Michigan AAC Administrative Advisory Committee Puerto Rico AAC Administrative Advisory Committee Illinois Partner Institutions CIAT Partner IARC Malawi NGO Michigan Bean Commission Michigan TMAC / IITA IARC Partner Nigeria TMAC / PABRA Partner IARC Uganda CIAT Partner IARC Colombia TMAC / IITA IARC Partner Nigeria TMAC Other member New York TMAC Other member Virginia TMAC Other member North Carolina TMAC / PABRA IARC Partner Uganda SO1.A1 / SO1.A4 / TMAC Collaborating Scientist North Dakota SO3 / TMAC Other member Missouri SO1.B1 / TMAC Collaborating Scientist Burkina Faso LIL Management Office MO Manager, Communications and Strategic Directions Michigan MO Assistant Professor Michigan MO Deputy Director Michigan MO Director, Center for Global Connections Michigan MO Assistant Professor Michigan MO Communications Manager Michigan MO Feed the Future Legume Innovation Lab Michigan MO Administrative Assistant Michigan MO Accountant Michigan MO Director Michigan LIL Collaborators and PIs S01.A1 / SO1.A4 PI / Collaborating Scientist North Dakota SO1.A5 / S01.A2 PI / Collaborating Scientist California SO1.A1 Collaborating Scientist Guatemala SO1.A1 Collaborating Scientist Guatemala SO1.A1 Collaborating Scientist Guatemala SO1.A1 / SO1.A4 Collaborating Scientist Guatemala SO1.A1 / SO1.A4 / TMAC Collaborating Scientist North Dakota SO1.A2 PI Michigan SO1.A2 Collaborating Scientist Michigan SO1.A2 Collaborating Scientist Zambia SO1.A3 / SO1.A2 PI / Collaborating Scientist Michigan SO1.A2 / SO1.A3 Collaborating Scientist Zambia SO1.A2 / SO1.A3 Collaborating Scientist Michigan SO1.A2 / SO1.A5 Collaborating Scientist California Appendix B: Evaluation Plan 94 SO1.A3 Collaborating Scientist Nebraska SO1.A3 Collaborating Scientis t Nebraska SO1.A3 Collaborating Scientist Michigan SO1.A3 Collaborating Scientist Uganda SO1.A4 Collaborating Scientist Puerto Rico SO1.A4 Collaborating Scientist Haiti SO1.A4 PI Puerto Rico SO1.A4 Collaborating Scientist Honduras SO1.A4 Collaborating Scientist Puerto Rico SO1.A5 Collaborating Scientist Ghana SO1.A5 Collaborating Scientist Ghana SO1.A5 Collaborating Scientist Burkina Faso SO1.A5 Collaborating Scientist Burkina Faso SO1.A5 Collaborating Scientist Senegal SO1.A5 Collaborating Scientist Burkina Faso SO1.B1 Collaborating Scientist Niger SO1.B1 Collaborating Scientist Benin SO1.B1 Collaborating Scientist Burkina Faso SO1.B1 Collaborating Scientist Benin SO1.B1 Collaborating Scientist Ghana SO1.B1 Collaborating Scientist Benin SO1.B1 Collaborating Scientist Niger SO1.B1 Collaborating Scientist Ghana SO1.B1 / SO2.1 PI / Collaborating Scientist Illinois SO1.B1 / SO2.1 Collaborating Scientist Illinois SO1.B1 / TMAC Collaborating Scientist Burkina Faso SO2.1 Collaborating Scientist Mozambique SO2.1 Collaborating Scientist Iowa SO2.1 Collaborating Scientist Iowa SO2.1 Collaborating Scientist Iowa SO2.1 Collaborating Scientist Uganda SO2.1 Collaborating Scientist Uganda SO2.1 Collaborating Scientist Mozambique SO2.1 Collaborating Scientist Uganda SO2.1 PI Iowa SO2.1 Collaborating Scientist Hawaii SO2.2 Collaborating Scientist Kansas SO2.2 Collaborating Scientist Tanzania SO2.2 Collaborating Scientist Kansas SO2.2 PI Kansas SO2.2 Collaborating Scientist Zambia SO2.2 Collaborating Scientist Malawi SO3 Collaborating Scientist Malawi SO3 Collaborating Scientist Missouri SO3 Collaborating Scientist Malawi SO3 / TMAC Missouri SO4 Collaborating Scientist Michigan SO4 PI Michigan Appendix B: Evaluation Plan 95 7. Evaluation Work Plan Activities Dates of Activity 1st Month 2nd Month 3rd Month 4th Month 5th Month 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 TASK 1. Develop evaluation design and implementation plan Feb 9-Feb 27 Activity 1: Desk review & initial interview with MSU management entity Feb 9 – Feb 27 X X X Activity 2: Development of evaluation design Feb 9 – Feb 27 X X X Activity 3: Select stakeholders to interview; plan travel to PAN Conf. Feb 9 – Feb 27 X X X TASK 2: Data Collection Feb 28 – Apr 15 Activity 1: Travel to Zambia; interview PIs, Co-PIs, stakeholders Feb 28 – Mar 5 X X X Activity 2: Site visit to MO at MSU Mar 21-Mar 30 X X Activity 3: Site visit to Uganda Apr 1–Apr 15 X X Activity 4: Additional Skype interviews as needed Apr 1 – Apr 30 X X X X Activity 5: Site visit to Guatemala for interviews Apr 1 – Apr 15 X X Activity 6: Collection of Mar 15 – Apr 15 X X X X Appendix B: Evaluation Plan 96 publication data TASK 3: Data Analysis Apr 16 –May 31 Activity 1: Review data collection for gaps Apr 16 –May 31 X X Activity 2: Organize findings, discuss and develop evaluation themes May 1 – May 10 X X Activity 3: Compile preliminary findings for ppt presentation May 31 May 1-May 31 X X X TASK 4: Report Writing June 1 – July 1 Activity 1: Draft report prepared and submitted to USAID by June 15 June 1-June 15 X X X Activity 2: Final report submitted by July 1 June 15 - July 1 X Appendix B: Evaluation Plan 97 B. Data Collection and Management Plan 1. Survey Training Evaluators are experienced in interview techniques. No special training is necessary to use the interview protocols. Interviews will be conducted in English. If a translator is necessary, the EET will arrange for one, verify the qualifications needed and provide any needed training to the translator. 2. Data Management and Security Data collection and management have been described in Section 4a. Evaluators are experienced with the confidentiality requirements needed for work with human subjects. All responses are considered confidential. If there is a need to reference a particular PI or co-PI, the EET will request permission from the individual. All notes and findings will be collated anonymously and no direct reference will be made to any individual respondent. Evaluators will not identify individual respondents either in reports or to USAID during debriefings. 3. Data Collection Approvals Data collection approvals will not be needed as this data collection is not for research, and the only information collected is that given by the subjects themselves with their agreement. Confidentiality of responses will be maintained, and no names or identification will be attached to responses. Interview notes will be coded with the general category of stakeholder and individual names kept separately to avoid potential for disclosure. 4. Data Collection Instruments The following section contains two interview protocols as examples of the type of the format and types of questions that will be asked during interviews. Interview protocols will be developed for each type of group interviewed: Stakeholders, TMAC members, PIs and CoPIs, other collaborators, students and postdocs. Responses to these questions will be collected and analyzed as presented in Section A. a. Protocol for Interviews of Project PIs and CoPIs Introduction The purpose of this interview is to assess the perspectives of member of the research teams funding under the Legume Innovation Lab in general and the XX project specifically. We are conducting confidential interviews of PIs, Co-PIs, Collaborating researchers in the US and in partner countries, as well as other relevant stakeholders. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. Our aim is to better understand the how research activities are proceeding, how the various LIL projects interact with each other and external groups and what factors might improve LIL’s Appendix B: Evaluation Plan 98 ability to attain its objectives. I have several questions that will take about 45 minutes of your time. Do you have any questions before we begin? Questions 1. Could you first describe the research that you are doing with the LIL? Prompt: Can you help me understand how your project addresses one or more of the strategic objectives of the legume innovation lab? 2. Could you please describe your one or two most important research findings? How has this research contributed to change or better conditions in your country? Prompt: Do you make adjustments to your activities based on good performance/important research of research teams? Prompt: How do you judge the quality of the various research activities under your project? Prompt: Do you provide incentives for publications? Please describe. Prompt: Could you describe the relevance of the research findings to the particular countries you are working in? What about for other countries or populations? 3. Thinking back to the beginning of the project in 2013, is the research progressing as you expected? Prompt: Can you point to some particular activities that highlight your progress? (e.g. journal articles, new breeding lines, etc.). Prompt: Are there activities that are ahead of schedule? Is anything falling behind? Prompt: Can you give me some examples? What do you think are the main reasons why these LIL activities are not progressing as planned? Or ahead of schedule? Prompt: Have you had to develop workarounds to address the challenges? Is there anything you can think of that would help you to better accomplish your LIL research? Prompt: Looking ahead, are there research activities that were proposed that you may not be able to accomplish? 4. Are there any significant new directions on the project due either to significant challenges you have encountered or due to important findings or discoveries from the research? Prompt: Can you give me some examples? Prompt: Do you see possibilities for new directions in a new research program? Explain? Prompt: Have there been any missed opportunities? Can you give me some examples? 5. Does program structure allow short-term changes to facilitate investigation of newly emerging research themes? Prompt: Can you provide examples of emerging issues that might be investigated and how you would adjust your research program to address them? Prompt: Would the project (MO) be able to provide additional resources, would you Appendix B: Evaluation Plan 99 have to cut back on other activities? How would your current project be adjusted to meet these new themes? 6. How has your project included consideration of gender in its design and implementation? Can you give us some detail on any gender specific findings, outputs or outcomes of your LIL work? Prompt: How many women work on the.project? As scientists, laboratory assistants, etc. Do female farmers attend project workshops? 7. This project includes people from multiple countries and institutions. Is there sufficient communication and coordination among LIL collaborators? Prompt: To what extent is decision making include perspectives from multiple partners or collaborators? Prompt: How well does the international research collaboration work? Are there ways you think that collaborative interaction could be improved? Prompt: To what extent to you feel like there is a team-based approach to the research? Prompt: Who on the project is your principal contact? Prompt: On average, how often do you receive communications from the MO/PI? What proportion of these communications is a request for information? What proportion is giving you information? Prompt: Please describe the process by which your project assembles its annual workplan and annual report. Is there a meeting of all scientists? 8. This project is one of many under the LIL program. How well does the management office at MSU manage the LIL program? Prompt: What is your experience with the management of the LIL? a. Professional/collegial interactions b. Fiscal interactions (promptness, clarity) c. Promptness and effectiveness of managing problems d. Quality of the solutions to problems e. Effectiveness and efficiency of the sub-award process f. Clarity of goals and objectives g. Degree to which planning is collaborative h. Degree to which data collection and analysis are supported i. Degree to which findings are appropriately disseminated Prompt: What could the MO do to more effectively meet the Legume IL’s objectives? Prompt: What strengths or synergies did this model encourage, e.g., among the individual projects to ensure complementarity of effort? 9. Is it important for your project to establish partnerships with local organizations or agencies, universities, USAID Missions, CGIAR centers and NGOs? Why or why not? Appendix B: Evaluation Plan 100 Prompt: How often do you meet with USAID mission representatives? How do these meetings transpire? Prompt: Can you provide an example of when your project responded to a USAID mission need? Prompt: If so, what types of connections have you established? What have you done with partner organizations that advance the strategic objectives of your project? Prompt: Do you think the partnerships work well? Are there partnerships that seem to work better than others? Why? Prompt: Have there been important contributions from these stakeholders or partners to your research? Could you give us some examples? 10. From your perspective, has there been much cross-project interaction and learning? Can you point to cross-project or program-level results or outcomes that have occurred? Prompt: How do you interact with scientist from other Legume IL research projects? Do meetings regularly occur, and how do they come about? 11. One of the objectives of the IL is to build local capacity for research. From your perspective is the program accomplishing this? Prompt: Have you received additional training as a result of the project? If so, please describe. If not, have you asked for such training? What was the resolution of your request? Prompt: How does the project identify perspective students for advanced study (e.g at a national or international university)? Is gender a consideration? How has the gender balance been? 12. What is your assessment of the strengths and weaknesses of the LIL in general, and your specific project (s) in particular? a. Protocol for Interviews of Non-PI/CoPI Collaborators (US and International) Introduction The purpose of this interview is to assess the perspectives of member of the research teams funding under the Legume Innovation Lab in general and the XX project specifically. We are conducting confidential interviews of research PIs, Co-PIs, Collaborating researchers in the US and in partner countries, as well as other relevant stakeholders. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. Our aim is to better understand the how research activities are proceeding, how the various LIL projects interact with each other and external groups and what factors might improve LIL’s ability to attain its objectives. Appendix B: Evaluation Plan 101 I have several questions that will take about 45 minutes of your time. Do you have any questions before we begin? Questions 1. Could you first describe the research that you are doing with the LIL? Prompt: Can you help me understand how your project addresses one or more of the strategic objectives of the legume innovation lab? 2. Thinking back to the beginning of the project in 2013, is the research progressing as you expected? Prompt: Can you point to some particular activities that highlight your progress? (eg journal articles, new breeding lines, etc.). Prompt: Are there activities that are ahead of schedule? Is anything falling behind? Prompt: Can you give me some examples? What do you think are the main reasons why these LIL activities are not progressing as planned? Or ahead of schedule? Prompt: Have you had to develop workarounds to address the challenges? Is there anything you can think of that would help you to better accomplish your LIL research? Prompt: Looking ahead, are there research activities that were proposed that you may not be able to accomplish? 3. Are there any significant new directions on the project due either to significant challenges you have encountered or due to important findings or discoveries from the research? Prompt: Can you give me some examples? Prompt: Do you see possibilities for new directions in a new research program? Explain? Prompt: Have there been any missed opportunities? Can you give me some examples? 4. Does program structure allow short-term changes to facilitate investigation of newly emerging research themes? Prompt: Can you provide examples of emerging issues that might be investigated and how you would adjust your research program to address them? Prompt: Would the project be able to provide additional resources, would you have to cut back on other activities? How would your current project be adjusted to meet these new themes? 5. How has your project included consideration of gender in its design and implementation? Can you give us some detail on any gender specific findings, outputs or outcomes of your LIL work? 6. This project includes people from multiple countries and institutions. Is there sufficient communication and coordination among LIL collaborators? Appendix B: Evaluation Plan 102 Prompt: To what extent is decision making include perspectives from multiple partners or collaborators? Prompt: How well does the international research collaboration work? Are there ways you think that collaborative interaction could be improved? Prompt: Would you say there is a decision making structure? Is there a need to engage more researchers project decisions? Do you feel sufficiently included in decisions? Prompt: How well does international collaboration work on this project? Do you feel like there is a team-based approach to the research? Why or why not? Prompt: Do project leaders ask you for input on project direction? Do you think your ideas are considered important? Do you feel like a member of the LIL project team? This is a big project, but do you get the intellectual credit you deserve? Prompt: Who on the project is your principal contact? Do you ever communicate directly with the MO? Prompt: On average, how often do you receive communications from the PI/co-PI? What proportion of these communications is a request for information? What proportion is giving you information? Prompt: Please describe the process by which your project assembles its annual workplan and annual report. Is there a meeting of all scientists? Prompt: Are you provided incentives for producing high-quality research? Specific examples of quality might be international journal articles, etc. 7. Is it important for your project to establish partnerships with local organizations or agencies, universities, USAID Missions, CGIAR centers and NGOs? Why or why not? Prompt: If so, what types of connections have you established? What have you done with partner organizations that advances the strategic objectives of your project? Prompt: Do you think the partnerships work well? Are there partnerships that seem to work better than others? Why? Prompt: Have there been important contributions from these stakeholders or partners to your research? Could you give us some examples? 8. From your perspective, has there been much cross-project interaction and learning? Can you point to cross-project or program-level results or outcomes that have occurred? Prompt: Have you attended international meetings of the project where scientists from other projects/countries have been present? When/where? Did you find that these interactions helped you in your research? 9. One of the objectives of the IL is to build local capacity for research. From your perspective is the program accomplishing this? Prompt: Have you received additional training as a result of the project? If so, please describe. If not, have you asked for such training? What was the resolution of your request? Appendix B: Evaluation Plan 103 Prompt: How does the project identify perspective students for advanced study (e.g at a national or international university)? 10. What is your assessment of the strengths and weaknesses of the LIL in general, and your specific project (s) in particular? b. Protocol for Interviews of Project Stakeholders and Partners Introduction The purpose of this interview is to assess the perspectives of stakeholders and partners involved with or knowledgeable about the Legume Innovation Lab. We are also conducting interviews of other researchers involved in the innovation lab. Our aim is to better understand the how research activities are proceeding, how the various LIL projects interact with each other and external groups and what factors might improve LIL’s ability to attain its objectives. I have several questions that will take about 45 minutes of your time. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. We expect that you are here on a voluntary basis. Please let me know if you prefer not to conduct this interview. Do you have any questions before we begin? Questions for non-LIL key informants (partner or potential partner) 1. Could you first discuss how you have been involved with (or know about) the Legume Innovation Lab. Prompt: Do you work with a particular project or projects? Prompt: Who do you work with? Prompt: What activities have you been involved in? 2. From your perspective, how is the project going? Is it proceeding as it was planned? Prompt: Do you know about any key research findings from the project? Could you provide specific examples? Prompt: Can you provide specific examples of LIL technologies or research findings that were taken up or used by other organizations? Prompt: What about other efforts, such as the dissemination of information, knowledge or other outputs? Prompt: Is the size and capacity of the project in line with its aims? 3. From your perspective, how is the project meeting its potential? What could it be doing differently that would improve its eventual impact? Prompt: Are the research themes addressed by LIL scientists appropriate from your perspective? Could you provide some detail? Prompt: Are there topics or directions that are not being address that should be? Prompt: Are there stakeholders that are not involved that should be? Are there some partnerships that are not productive or should be either reframed or dropped? Prompt: What do you think could be improved to boost the quality of the research? Appendix B: Evaluation Plan 104 4. How does the LIL research project complement and or advance actions/research programs of other organizations whether national or international? Prompt: Please provide an example or two of that complementarity. Prompt: Can you provide specific examples of LIL technologies or research findings that were taken up or used by other organizations? Prompt: Does the work contribute to the field in a significant way? If so, how? What niche is it filling that other projects are not? 5. From your knowledge, in what ways does the LIL enable and/or provide training opportunities for researchers and others? Prompt: Does your organization participate in LIL research or training? Please describe the modalities of this participation? Prompt: Are the proper people being trained in the proper fields and at the proper level? Prompt: Are women adequately represented in the training program? Prompt: Are researchers working in the developing world well represented in the training program? Prompt: Is the LIL properly prioritizing its training (undergraduate/graduate)? Are certain disciplines under-represented? Prompt: Have you seen any outcomes of the training – such as people from your organization who have been able to contribute to new research or activity? 6. How well does LIL manage coordination and communication among LIL partners or stakeholders? Prompt: How often do you communicate with LIL? Is that communication mostly in the form of dissemination of information or are you included in dialogue? Prompt: To what extent do you feel like your perspectives and knowledge is considered valuable by LIL? Could the communication be better? How? Prompt: To what extent to you feel like there is a team-based approach to the research? Do you feel like it is a true collaboration? 7. Has the LIL and your partnership run into any obstacles, constraints or challenges? Prompt: Could you provide an example? Prompt: What did LIL do to address it? Do you think LIL was responsive? Prompt: Were the issues well resolved? Are there issues still to resolve? We have talked about many different things. Could you help us understand more about any important outcomes, advances or issues that we have not discussed? Appendix B: Evaluation Plan 105 C. Descriptions of Research Projects SO1. Advancing the Productivity Frontier for Grain Legumes 1. Genetic Improvement of Middle-American Climbing Beans for Guatemala (SO1.A1) Lead PI: Juan M. Osorno, North Dakota State University Co-PIs: Phil McClean, North Dakota State University; Julio C. Villatoro, Fernando Aldana, Karla Ponciano, Julio Martinez, and Edgardo Carrillo, Instituto de Ciencia y Tecnología Agrícola (ICTA), Guatemala The highlands of Guatemala are a unique bean producing region where intercropping (milpa) is still the main production system, mostly with maize–bean association. The system involves planting climbing beans that grow and develop concurrently with maize or planting in relay. On-farm productivity of climbing bean landraces is approximately one-third of their genetic yield potential, mostly due to the lack of improved cultivars that are able to withstand biotic and abiotic stresses. This low productivity adversely affects food and nutritional security of households of largely indigenous Mayan families in the region, especially women and children. Historically, climbing beans have received less attention and breeding efforts in comparison with the bush-type beans commonly grown in the lowlands. In addition, the Guatemalan Middle American climbing beans are a unique group of germplasm that has not been studied extensively and could offer new genetic variation for traits of economic importance. This new Legume Innovation Lab project will focus on developing genetically improved varieties of climbing beans that can enhance the productivity of the highland milpa system. The main objectives are: 1. to breed varieties with improved disease resistance and agronomic performance, 2. to characterize the genetic diversity of this unique set of germplasm, 3. to improve understanding of constraints to increasing climbing bean productivity in an intercropped system, and 4. to train the next generation of plant breeders for Guatemala. 2. Improving Photosynthesis in Grain Legumes with New Plant Phenotyping Technologies (S01.A2) Lead PI: David M. Kramer, Michigan State University, Collaborating Scientists: Timothy J. Close, University of California, Riverside; Maren Friesen, Michigan State University; James Kelly, Michigan State University; Wayne Loescher, Michigan State University; Phil Roberts, University of California, Riverside; Kennedy Muimui, ZARI; Zambia Kelvin Kamfwa, University of Zambia; Kennedy Muimui, ZARI, Zambia Photosynthesis provides the energy to generate all food on the planet. However, the realized efficiency of photosynthesis is far below its theoretical limit partly because it is inherently potentially dangerous to plants, particularly under fluctuating (rapidly changing) environmental conditions. Improving photosynthesis requires selection for multiple traits simultaneously that both increase yield and resilience to natural, fluctuating environmental conditions. Importantly, there are large natural genetic and breeding-induced variations in the responses of photosynthesis across species and germplasm, leading to large differences in productivity in specific environments. These variations may be exploited for plant Appendix B: Evaluation Plan 106 improvement, provided that we can identify the genetic loci and processes controlling these traits. The objectives of this project are to: 1) develop and evaluate innovative new technologies (DEPI and PhotosynQ) for improvement of grain legumes both under controlled yet dynamic and field conditions. 2) employ these technologies in proof-of￾concept projects to identify QTLs in cowpea and common beans that modulate the efficiency of photosynthesis and its responses to changing environmental conditions; 3) establish and enable an African-USA community of networked scientists, extension agents, students and growers to address field-level research and production questions; and 4) establish and enable an Africa–USA community of networked scientists. 3. Improving Genetic Yield Potential of Andean Beans with Increased Resistances to Drought and Major Foliar Diseases and Enhanced Biological Nitrogen Fixation (BNF) (S01.A3 Lead PI: James Kelly, Michigan State University Co-PIs: Karen Cichy, USDA-ARS, Wayne Loescher, Michigan State University, James Steadman and Carlos Urrea, University of Nebraska; Stanley Nkalubo, National Crop Resources Research Institute (NaCCRI), Uganda; Kennedy Muimui, Zambia Agriculture Research Institute (ZARI), Zambia. Common bean (Phaseolus vulgaris L.) is the most important grain legume consumed in Ecuador, Uganda, and Zambia. Building on international bean germplasm, an opportunity exists to develop and to deploy improved bean varieties using a combination of traditional and the latest molecular plant improvement techniques. An improved understanding of plant traits and genotypes with resistance to multiple stresses from abiotic (drought) and biotic (root rot and foliar pathogens) sources will provide unique genetic materials for enhanced plant breeding methods and sources to study plant tolerance mechanisms. Improvements in current understanding of the physiology of drought and evapotranspiration as well as the genetics of drought tolerance in common bean and the development of effective molecular and quantitative methods for the selection of drought tolerance are needed. The development of improved bean varieties and germplasm with high yield potential, healthy root systems, improved biological nitrogen fixation (BNF) with resistance to multiple diseases, and sustained or improved water use efficiency under limited soil water conditions are needed to increase profit margins and lower production costs. The project will use QTL (Quantitative Trait Loci) analysis and SNP (Single Nucleotide Polymorphism)-based genomewide association mapping to uncover regions associated with drought tolerance, disease resistance, enhanced BNF, and shorter cooking time. 4. Development and Implementation of Robust Molecular Markers and Genetic Improvement of Common and Tepary Beans to Increase Grain Legume Production in Central America and Haiti (S01.A4) Lead PI: James Beaver, University of Puerto Rico Co-PIs: Consuelo Estevez de Jensen, University of Puerto Rico; Timothy Porch, USDA/ARS/TRAS; Juan Osorno and Phil McClean, North Dakota State University (NDSU); Juan Carlos Rosas, Escuela Agrícola Panamericana (Zamorano), Honduras; Julio Cesar Villatoro, Instituto de Ciencia y Appendix B: Evaluation Plan 107 Tecnología Agrícola (ICTA), Guatemala; Emmanuel Prophete, National Seed Service, Ministry of Agriculture, Haiti This project will develop, release, and disseminate improved common bean cultivars with enhanced disease and/or pest resistance and greater tolerance to abiotic stresses for the lowland tropics in Central America and Haiti. Drought and heat tolerant tepary bean lines (Phaseolus acutifolius) will be bred for improved seed and agronomic traits and greater disease resistance. Bruchid resistance genes (arcelin 2 and null phaseolin) resulting from an interspecific cross will be introgressed into black, small red and white beans with resistances to BCMV, BCMNV, and BGYM for Central America and the Caribbean regions. In Guatemala, breeding efforts will focus on increasing resistance to common bacterial blight and web blight in humid tropical lowland regions, such as the Petén. The next generation of breeder-friendly molecular markers linked with pathogen-resistant genes will be developed. These markers will improve the efficiency and effectiveness of selection for traits of economic importance. Graduate and undergraduate training will be supported by the project to address the need to strengthen bean research programs in the target countries. In-service training and workshops will be sponsored to improve the quality and reliability of field research and to permit target country breeding programs to take advantage of molecular tools to increase the efficiency of the selection of economically important traits. 5. Genetic Improvement of Cowpea to Overcome Drought and Biotic Constraints to Grain Productivity (S01.A5) Lead PI: Philip Roberts, University of California, Riverside Co-PIs: Timothy Close, University of California, Riverside; Issa Drabo and Jean-Baptiste Tignegre, Institut de l’Environnement et de Recherches Agricoles (INERA), Burkina Faso; Ibrahim Atokple and Francis Kusi, Savanna Agricultural Research Institute (SARI), Ghana; Ndiaga Cisse, Centre National Recherches Agronomie, Bambey, Institut Senegalais de Recherches Agricole (ISRA), Senegal Cowpea is a highly nutritious grain legume crop vitally important to food security in sub￾Saharan Africa, especially for women and children, where it complements cereals in the diet. However, in the Sudano–Sahel region of West Africa, typical smallholder farmer cowpea yields are only 10–20 percent of known yield potential. Biotic stresses caused by insect pests, and diseases caused by pathogens, parasitic plants, and nematodes, and abiotic stresses from drought and low-fertility soils are primary constraints to cowpea grain production. This project will focus on cowpea breeding with an emphasis on insect tolerance and resistance traits, combined where feasible with drought tolerance and disease resistance traits. More specifically, tolerance or resistance to aphids, flower thrips, and pod￾sucking bugs will be pursued through trait discovery and molecular-driven breeding selection to generate improved cowpea varieties. Field and lab-based phenotyping in Burkina Faso, Ghana, and Senegal will be matched with SNP marker high-throughput genotyping to identify and select for target QTL. Advanced breeding lines will be tested regionally across the host countries to broaden their release potential. In addition, several near-release advanced lines will be performance tested for full release decisions in Burkina Faso and Senegal, capitalizing on previous USAID CRSP investments. In California, cowpea dry grain Appendix B: Evaluation Plan 108 novel market classes of breeding lines will be advanced together with leveraged funding in support of the U.S. dry bean industry. 6. IPM-omics: Scalable and Sustainable Biological Solutions for Pest Management of Insect Pests of Cowpea in Africa (SO1.B1) Lead PI: Barry Pittendrigh, University of Illinois at Urbana–Champaign Co-PIs: Manuele Tamò, International Institute for Tropical Agriculture (IITA), Benin; Clémentine Dabiré￾Binso, l’Institut de l’Environnement et de Recherches Agricoles (INERA), Burkina Faso; Ibrahim Baoua, L’Institut National de la Recherche Agronomique du Niger (INRAN), Niger; Stephen Asante, Savannah Agriculture Research Institute (SARI), Ghana; Haruna Braimah, Crops Research Institute, Ghana; Leonard Hinnou, Institut National des Recherches Agricoles du Benin (INRAB), Benin; Julia Bello-Bravo, University of Illinois at Urbana– Champaign The major biotic constraint on cowpea crops in West Africa is an insect pest complex. Pesticides and/or transgenics will not provide the long-term solutions needed to manage these pest populations. The logical strategy is Integrated Pest Management (IPM), involving a pipeline of diverse biological control situations. This project focuses on the development and deployment of scalable pest control solutions involving a combination of traditional pest control and deployment strategies and cutting-edge technologies, including genomics and GIS (Geographic Information Systems) to help direct the most effective deployment of these approaches, as well as testing and deploying cutting-edge ICT (Information and Communications Technology) tools as part of the scaling of these solutions. IPM-omics involves defining the pest problems, bringing forward appropriate solutions through a biocontrol/biopesticide pipeline, and scaling of solutions through multipronged strategies that will include farmer field flora, ICT approaches, women’s cooperatives, and partnerships with small-scale industries. The project will continue to develop online interfaces that make solution options easily available to other groups that can benefit from the materials. The project will also develop, deploy, and test training/technology packages/programs that will be passed o¤ to groups (e.g., NGOs and national/international agencies). Strategic Objective 2 (S02): Transforming Grain Legume Systems and Value Chains 7. Farmer Decision-Making Strategies for Improved Soil Fertility Management in Maize–Bean Production Systems (S02.1) Lead PI: Robert E. Mazur, Iowa State University Co-PIs: Eric Abbott, Andrew Lenssen, and Ebby Luvaga, Iowa State University; Russell Yost, University of Hawaii at Manoa; Julia Bello￾Bravo and Barry R. Pittendrigh, University of Illinois at Urbana–Champaign; Moses Tenywa, Makerere University, Uganda; Onesimus Semalulu, National Agricultural Research Laboratories (NARL), Uganda; Ricardo Maria, Instituto de Investigação Agrária de Moçambique (IIAM) Appendix B: Evaluation Plan 109 Poor and declining soil fertility is the primary constraint to common bean productivity among smallholder farmers in Africa, affecting cropping systems, food security, nutrition, incomes, and livelihoods. Adoption of improved crop management practices, particularly regarding soil fertility, has been modest. The central premise of this project is that addressing soil-related constraints requires understanding farmers’ current practices and enhancing their capabilities in diagnosing and finding solutions to yield constraints. To contribute to widespread and sustainable improvements in bean productivity and soil fertility, the research objectives are to: 1. characterize smallholder farmers’ agricultural motivations, current knowledge, and practices; problem diagnoses; and livelihood and risk management strategies; 2. develop and refine models about their decision making; 3. develop and validate appropriate diagnostic and decision support aids; and 4. develop and assess the effectiveness of innovative approaches for disseminating information and decision support aids, training, and follow-up technical support. Working with smallholder farmers in rainfed maize– bean cropping systems in the Masaka district in Uganda, and the Gurué district in Mozambique, at 1000–1200m altitude with annual rainfall 1000–1500mm, the approach and methods involve: 1. participatory rural appraisal and baseline surveys for activity planning; taking into account critical social, economic, and cultural factors that impact decision making and adoption of new strategies and technologies; and monitoring changes over time; 2. farmer innovator and scientific analyses of soil-related constraints; 3. participatory on-farm studies using identified possible solutions; 4. participatory, gender equitable development and validation of diagnostic and decision support aids; and 5. development and pilot-testing of innovative sociotechnical approaches for communication, dissemination, and scaling up. 8. Enhancing Pulse Value-Chain Performance through Improved Understanding of Consumer Behavior and Decision Making (S02.2) Lead PI: Vincent Amanor-Boadu, Kansas State University Co-PIs: Gelson Tembo, University of Zambia; Lawrence Mapemba, Lilongwe University of Agriculture and Natural Resources; Fredy Kilima, Sokoine University of Agriculture; Allen Featherstone and Kara Ross, Kansas State University Despite knowledge about their high nutritional benefits and profile, grain legumes are not prominent on the food hierarchy in Malawi, Tanzania, and Zambia. The challenge confronting grain legume producers and their supply chains is how to enhance their competitiveness in their local markets and gain the necessary policy support from their government to sustain it. This project seeks to make two critical contributions to this challenge. First, it will seek to develop a clearer appreciation of the factors influencing grain legume consumption in these three countries to provide empirical direction for market and policy development. Second, it will also seek to provide training and capacity building support for the industry’s stakeholders to seize identified opportunities and address existing and emerging challenges. When this project succeeds, it will contribute to creating value in the grain legume value chain. The project has three integrated dimensions: an Appendix B: Evaluation Plan 110 empirical foundation for understanding factors and their influence on food choices; application of the empirical results in crafting policies and facilitating knowledge and skill development in managing value chains; and developing and delivering outreach programs to help both private and public stakeholders improve the performance of grain legume value chains. These activities contribute to supporting strategies and initiatives that enhance the well-being of smallholder producers. This project, therefore, provides innovative and unique pathways that bring smallholder producers and public and private stakeholders together to help achieve the underlying objectives of the Feed the Future initiative. Strategic Objective 3 (S03): Enhancing Nutrition 9. Legumes, Environmental Enteropathy, the Microbiome and Child Growth in Malawi (SO3.1) Lead PI: Mark Manary, Washington University School of Medicine in St. Louis Co-PIs: Ken Maleta and Chrissie Thakwalakwa, University of Malawi College of Medicine; Indi Trehan, Washington University School of Medicine in St. Louis Because millions of children die annually due to undernutrition and hundreds of millions more are stunted, interventions that decrease the burden of childhood malnutrition are urgently needed. Environmental enteropathy (EE), a pervasive, chronic, subclinical, inflammatory condition among children when complementary foods are introduced, places them at high risk for stunting, malabsorption, and poor oral vaccine efficacy. Here we propose two randomized, controlled clinical trials to determine if common beans or cowpeas improve growth, ameliorate EE, and alter the intestinal microbiome during this high-risk period. The first study involves 6–11-month-old children who will receive common beans, cowpeas, or standard local complementary foods for six months. Anthropometry will be compared among the three groups. EE will be assessed using a urine dual-sugar absorption test and by quantifying human intestinal mRNA for inflammatory messages, and the intestinal microbiota characterized by deep sequencing of fecal DNA to enumerate the host microbial populations and their metabolic capacity. The second randomized, controlled trial will enroll 12–35-month-old children and follow them for 12 months; each subject will receive dietary interventions, either legume-based or control. Anthropometric, host inflammatory, and gut microbiota analyses will be conducted similar to the first study. The studies will also facilitate the training of two doctoral-candidate nutrition students from the University of Malawi and two food science master’s level students from the national agricultural university. By amalgamating the power of the clinical trial and advanced biological analyses, we will elucidate the potential of legumes to significantly impact child health in sub-Saharan Africa. Strategic Objective 4 (S04): Improving Outcomes of Research and Capacity Building Impact 10. Assessment of Dry Grain Pulses CRSP Investments in Research, Institutional Capacity Building and Technology Dissemination in Africa, Latin America and the U.S. (S04.1) Appendix B: Evaluation Plan 111 Lead PI: Mywish Maredia, Michigan State University Co-PIs: Eric Crawford and Byron Reyes, Michigan State University Building on the momentum and experience gained over the last three years, this project proposes to contribute to evidence-based, rigorous ex ante (i.e., potential) and ex post (i.e., realized) assessments of outputs, outcomes, and impacts of research with the goal of assisting the Legume Innovation Lab program and its Management Office (MO) to achieve two important goals—accountability and learning. Greater accountability (and strategic validation) is a prerequisite for continued financial support from USAID and better learning is crucial for improving the effectiveness of development projects and ensuring that the lessons from experience—both positive and negative—are heeded. Integrating this culture of impact assessment into publicly funded programs such as the Legume Innovation Lab will ultimately help increase the overall impact of such investments. The project team proposes to provide technical leadership in the design, analysis, and collection of baseline and endline data to conduct ex ante and ex post impact assessment of the Legume Innovation Lab’s investments in research, institutional capacity building, and technology dissemination in Africa, Latin America, and the U.S. It also proposes to conduct systematic analysis of existing data or to conduct field studies to address strategic research questions on the role of grain legumes in household food security, nutrition, and income. Appendix C: Travel Itinerary 112 Appendix C: Travel Itinerary Zambia: Alwang, Myers, Temple and Welch travel to Pan African Grain Legume and World Cowpea Conference. Date Activity 27-February Alwang, Myers and Temple travel to South Africa; team meeting at hotel 28- February Arrive in Zambia; entire team meets with USAID 29-February-4 March Interviews with project PIs, LIL-supported students and external stakeholders 5-March Return to US Michigan State University: Alwang, Myers and Temple Date Activity 20-March Travel to East Lansing 21-March Alwang, Myers and Temple meet with MO team, with representatives of the Michigan Dry Bean Commission, and with project PIs and graduate students 22-March Alwang, Myers and Temple meet with College and University Administration, grants and contracts officials, and project PIs and graduate students 23-March Return to home Guatemala: Alwang and Temple Date Activity 24-April Travel to Guatemala City 25-April Alwang and Temple, together with Luis Flores (Director MASFRIJOL), Juan Osorio (PI, NDSU) and Juan Carlos Rosas (PI, Zamorano) to USAID Mission. Met with three USAID representatives. Traveled to ICTA and met with Julio Cesar Villatoro (PI, Guatemala) and Elias Raymundo (DG of ICTA) 26-April Team (Alwang, Temple, Flores, Osorio, Rosas and Villatoro) traveled to Chimaltenango and visited LIL experiments at ICTA experiment stations. Traveled to Quetzaltenango and had long meeting with MASFRIJOL partners. 27-April Visited ICTA experiment station to review field trials for Osorno LIL project in Quetzaltenango. Experiment station work appears to be going well. Visited 2 farmer fields in outskirts of Quetzaltenango where trials are being conducted on farmer fields. Traveled to San Miguel. Series of long interviews with MASFRIJOL collaborators on functioning of program. Separate interviews were conducted by Temple and Alwang. 28-April Visited 4 community seedbanks in areas around San Miguel. Interviewed participating farmers about their experience with improved seeds and MASFRIJOL in general. Team returned to Guatemala City. 29-April Alwang and Osorio returned to US; Rosas returned to Honduras 30-April Temple returned to US Appendix C: Travel Itinerary 113 Uganda: Myers and Temple Date Activity 6-May Departed U.S. 7-May Arrived in Uganda, travel to Kampala. 8-May Rest and preparation for the visit. 9-May Traveled with Stanley Nkalubo, plant breeder of the MSU SO1.3 project, to the USAID Mission in Kampala. Met for 1 hour with 5 officials of the USAID office in Kampala. Then traveled to the NaCCRI station at Kawanda outside Kampala, and toured the facilities, met some NaCCRI colleagues, and spent several hours discussing progress and plans with Stanley. 10-May Transported to the NARO Kawanda research station, and spent several hours talking with 3 members of the CIAT East African team. Then met Onesimus Semalulu, NARO soil scientist of the ISU project SO2.1, and discussed project activities and plans. 11-May Met by Richard Miiro of ISU project SO 2.1, and traveled to Makere University, where we spent the day meeting/discussions with most of the Makerere team and visiting the facilities and meeting with station director. 12-May Met by Onesimus Semalulu soil scientist of NARO and member of the Ugandan SO2.1 team, and student participants. Traveled to the Masaka site where SO2.1 is conducting the IP activities, and spent a full day visiting numerous field sites in two of the three “sub-counties” in the Masaka district where IP is under way. Overnight in Masaka. 13-May Attended meeting of some 40 farmers and 20 local “input participants”, who traveled to the Masaka district office for the activity. Returned to Kampala in the evening. 14-May Consolidated notes during the day and traveled to Entebbe in the evening to begin the journey back to the U.S. 15-May Arrived in U.S. Appendix D: List of Materials Reviewed 114 Appendix D: List of Materials Reviewed Technical Application proposal Legume IL annual reports Sub-Awards project briefs, reports and annual work plans Program publications including success stories, research publications, HICD outputs, and impact briefs (with links to data) Other Legume Lab IL website materials and information (legumelab.msu.edu) External review report for first five years of the program (Dry Grain Pulses CRSP, 2007-2012) Feed the Future Research Strategy Appendix E: Photos 115 Appendix E: Photos Figure 2. Climbing bean greenhouse trials. Bolonillo bean trials in greenhouse, Chimaltenango, Guatemala. The SO1.A1 project, recently introduced into the LIL portfolio is engaged in producing new germplasm that should greatly improve bean yields and is particularly (because it is a climbing bean variety) suited for livelihood systems in Guatemala’s extremely poor Western Highlands. Photo courtesy of Luis Flores. Appendix E: Photos 116 Figure 3. Soil fertility research on beans in a farmer's field. Dry beans in a farmer's field near Masaka, Uganda. Those on the right have a chicken manure + fertilizer treatment and those on the left are untreated and typical of most farmers’ fields in the region. Research on soil fertility is being conducted by SO2.1. Photo courtesy of James Myers. Appendix E: Photos 117 Figure 4. Sampling of bean diseases. A sampling of diseases observed in SO2.1 Innovation Platform dry bean fields growing near Maskaka, Uganda in 2016. A. Anthracnose; B. Floury Leafspot; and C. Halo blight. Anthracnose and Halo blight are seed borne diseases and all can reduce yields. Photo courtesy of James Myers.