AN EXTERNAL PERFORMANCE EVALUATION OF THE FEED THE FUTURE INNOVATION LABS FOR APPLIED WHEAT GENOMICS AND CLIMATE RESILIENT CHICKPEA 21 March 2018 Award Numbers: AID-OAA-A-13-00051 (Applied Wheat Genomics) and AID-OAA-A-14-00008 (Climate Resilient Chickpea). This document was produced at the request of the United States Agency for International Development. It was prepared by Jim Register, Jackie Rudd, and Alex Winter-Nelson. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Acknowledgements This evaluation relied on the cooperation, knowledge and expertise of many people. The External Evaluation Team is grateful to all the individuals who participated in interviews and meetings or responded to questions over e-mail. We are also grateful to USAID staff who provided guidance and support. The content of this report does not reflect positions of USAID or its employees. This report provides the authors’ evaluation based on interpretation of the data we collected in the evaluation process. The authors are grateful to all individuals who provided information, and the authors are responsible for the interpretation of the information as presented in this report. Table of Contents List of Acronyms ............................................................................................................................................. i List of Tables ................................................................................................................................................ iii Executive Summary ...................................................................................................................................... iv Introduction .................................................................................................................................................. 1 Evaluation ..................................................................................................................................................... 3 Research Program: Applied Wheat Genomics IL ...................................................................................... 3 Findings ................................................................................................................................................. 3 Conclusions .........................................................................................................................................11 Lessons Learned ..................................................................................................................................11 Recommendations ..............................................................................................................................12 Research Program: Climate Resilient Chickpea IL ...................................................................................12 Findings ...............................................................................................................................................12 Conclusions .........................................................................................................................................20 Lessons Learned ..................................................................................................................................20 Recommendations ..............................................................................................................................20 Program Structure & Management ........................................................................................................20 Findings ...............................................................................................................................................20 Conclusions .........................................................................................................................................23 Lessons Learned ..................................................................................................................................23 Recommendations ..............................................................................................................................23 Partnership & Leveraging Opportunities ................................................................................................23 Findings ...............................................................................................................................................23 Conclusions .........................................................................................................................................26 Lessons Learned ..................................................................................................................................27 Recommendations ..............................................................................................................................27 Conclusions .................................................................................................................................................28 Appendices ..................................................................................................................................................31 Scope of work .........................................................................................................................................31 Evaluation Plan........................................................................................................................................32 List of persons contacted ........................................................................................................................70 List of materials reviewed .......................................................................................................................70 Responses from Project Directors ……………………………………………………………………………………………………….71 i List of Acronyms ABI Advanced Backcross Introgression AOR Agreement Officer’s Representative AWG Applied Wheat Genomics AWGIL Applied Wheat Genomics Innovation Lab AARI Ayub Agricultural Research Institute BFS Bureau for Food Security BISA Borlaug Institute for South Asia CGIAR Consultative Group for International Agricultural Research CIMMYT International Center for Maize and Wheat Improvement CRC Climate Resilient Chickpea CRCIL Climate Resilient Chickpea Innovation Lab EAP External Advisory Panel EET External Evaluation Team FAO Food and Agriculture Organization GEBV Genomic-Estimated Breeding Value GRDC Grains Research and Development Corporation GS Genomic Selection HTP High Throughput Phenotyping ICAR Indian Council for Agricultural Research ICARDA International Center for Agricultural Research in the Dry Areas ICRISAT International Crops Research Institute for Semi-Arid Tropics KSU Kansas State University NARS National Agricultural Research Service PI Principal Investigator revIL Reverse Introgressed lines RP Research Program SOW Statement of Work ii UCD University of California at Davis USAID United States Agency for International Development iii List of Tables Table 1: Long-term Trainees, AWGIL…………………………………………………………………………. 7 Table 2: Publications, AWGIL…………………………………………………………………………………….. 9 Table 3: Long-term Trainees, CRCIL………………………………………………………………………….. 17 Table 4: Publications, CRCIL……………………………………………………………………………………… 19 Table 5: Extent of network and funds leveraged for Climate Resilient Chickpea IL.….. 26 iv Executive Summary The Feed the Future Innovation Lab for Applied Wheat Genomics (AWGIL) and the Feed the Future Innovation Lab for Climate Resilient Chickpea (CRCIL) were established as five-year awards ending in August and December 2018, respectively. The AWGIL was launched with an initial grant of $ 5,000,000 to Kansas State University (KSU) while the CRCIL received a $ 6,000,000 award through the University of California at Davis (UCD). These labs are similar in structure in that they both represent consortium Innovation Labs (IL) rather than Management Entity approaches. The two ILs have some thematic similarity in that both intend to use advanced methods to create resources for improved crop breeding and ultimately reduce food insecurity. The AWGIL is doing this through creation and dissemination of wheat varieties that can potentially be advanced for grower use or, at a minimum, enhance breeding programs. The CRCIL works farther upstream and most outputs from the CRCIL will require additional research and development before they can be commercialized or contribute directly to chickpea breeding programs. This evaluation, conducted November 2017 through March 2018, follows an evaluation plan approved by the United States Agency for International Development (USAID). It involved a review of background literature and project documents and a series of interviews with more than 20 project participants including program directors, collaborating project scientists, graduate students, other scientists and USAID staff. The main findings of the External Evaluation Team (EET) relate to performance of the ILs in execution of proposed research programs, in management of activities, and in leveraging of other opportunities and cooperation with other entities. Findings in each of these areas are summarized below. Findings Concerning research outcomes, both ILs have met or exceeded many of their initial goals. In some cases, the programs have had to adjust in the face of unforeseeable difficulties or changing circumstances and opportunities. The adjustments have been adopted after proper consultation and likely enhance the long-term impact of the labs. Both lab directors have been reasonably nimble in adapting to changes and challenges. The AWGIL and the CRCIL have faced at least two common challenges in their research programs. In both cases the capacity to create data seems to exceed the capacity to manage those data and optimize their value to the program and to other scientists. This situation reflects a broad issue in the scientific community today, and it is a significant challenge to the work conducted in these ILs. v Consistent with the priorities of USAID, the AWGIL and the CRCIL have attempted to be gender￾sensitive in their research activities. The nature of the research leaves little scope for engagement with women as users of the final technologies or in systems that might reach the farm level. Therefore, the primary mechanism whereby these labs can engage women is through capacity building of female scientists. The underrepresentation of women in science, particularly in agricultural sciences, in the target countries has made it difficult for both ILs to identify and recruit highly qualified women into their degree training programs. The AWGIL and CRCIL made thoughtful efforts to recruit women. While both ILs had and met overall quantitative targets for trainees, establishment and ongoing management of gender targets between USAID and IL leadership was inconsistent. Both the AWGIL and the CRCIL have conducted numerous short duration trainings. Systematic pre- or post-testing assessments or other mechanisms that would allow evaluation of learning outcomes were not used in either case. As a result, the impact of the trainings is unobservable for the EET. While participants may express satisfaction with the trainings, we cannot discern whether or how their capabilities were improved through them. Neither the AWGIL nor the CRCIL were expected to produce research outputs that would be applied on-farm within five years. The practical impact of the work in these labs is therefore difficult to assess at the present time, however this evaluation suggests that the work in both ILs is on track to have impact, but that has not yet been proven. Concerning management, these ILs present somewhat different models. The AWGIL employs a full-time Project Coordinator in addition to the Project Director, while in the CRCIL, the director absorbs all administrative duties in addition to taking on intellectual leadership. The CRCIL employs an External Advisory Panel (EAP) to give high-level strategic and scientific input to the Project Director, while the AWGIL operates with no formal advisory body. The EET is sympathetic to the CRCIL Director’s decision to allocate funds to scientists rather than administrative support. That said, the absence of a Project Coordinator may result in dissipation of a Director’s effort and some delays in communication among program partners. The CRCIL has made good use of its EAP to provide guidance in some decisions. The relatively high degree of focus in many consortium-model ILs as opposed to Management Entity ILs appears to reduce the need for an EAP in some cases. The CRCIL and AWGIL approached leveraging and partnerships in different ways. Work done within the AWGIL stayed very close to the original plan, with researchers from KSU and the International Center for Maize and Wheat Improvement (CIMMYT) developing an apparently deep multifaceted scientific relationship that extended to include researchers in India, Pakistan, and ultimately Bangladesh. Expansion into Bangladesh was facilitated by the presence of a Bangladeshi graduate student at KSU, who was able to serve as the field lead for this part of the IL activity. This helped the IL overcome the lack of expertise and capacity for conducting large￾scale field trials and High Throughput Phenotyping (HTP) that existed before the IL began vi working in Bangladesh. The AWGIL has not made a priority of leveraging or expanding beyond the original plan, although there are some preliminary plans to do so before the current funding period ends. In contrast, the CRCIL has actively sought additional resources (funds, people, and germplasm) from a range of sources to extend its impact. The CRCIL more than tripled its financial resources through additional grants as well as by developing partnerships with other groups with independent sources of funding. These resources were used to extend efforts into additional countries and areas of research. Given the focus of the program on creation of germplasm collections, this geographic expansion enhanced the value of work for all countries involved, reinforcing the activities of the original USAID award. Recommendations The two labs reviewed here have made significant advances in accordance with the initial proposals. Achieving the development impacts on smallholder farmers that USAID seeks will require considerably more time and continued efforts along the lines that each IL has been pursuing. The EET recommends that future activities emphasize: 1. Data management and software systems to ensure optimal use of the resources developed under the ILs. 2. Increased efforts to engage partners from South Asia in degree training and in the use of Genomic-Estimated Breeding Value (GEBV) modeling for Genomic Selection (GS) in order to achieve lasting impact in wheat. 3. Efforts to leverage new chickpea resources developed over the last five years to move from pre-breeding to breeding activities. 4. Establishment of mechanisms for systematic assessment of learning outcomes from training efforts, especially short-term training activities. 5. Careful consideration of the optimal distribution of funding for administrative personnel and scientific personnel if a program attempts to move chickpea activities from pre￾breeding to breeding. In general, it would appear that interruption in the support to these ILs might undermine the effectiveness and impact of the investments that have already been made by diminishing the capacity to consolidate knowledge and breaking momentum in moving research findings to application. 1 Introduction Program Background The Applied Wheat Genomics Innovation Lab (AWGIL) was established in October 2013 with a cooperative agreement (AID-OAA-A-13-00051) of $4,990,218 for a period of five years. The AWGIL aims to develop heat-tolerant, high-yielding, and farmer-accepted varieties for South Asia, while simultaneously increasing the research for development capacity of the global wheat improvement system through application of cutting-edge genomics in applied wheat improvement. Specifically, the AWGIL focuses on the challenges of wheat production and climate change in South Asia, by applying genomics, physiology, and international phenotyping networks to maximize the value of wheat varieties in the region. The AWGIL is organized as a consortium led by Prof. Jesse Poland at Kansas State University (KSU) and includes the International Maize and Wheat Improvement Center (CIMMYT) in Mexico and Pakistan, the Borlaug Institute for South Asia (BISA) in India, the Indian Council for Agricultural Research (ICAR), the Wheat Research Center of the Ayub Agricultural Research Institute (AARI) in Pakistan, and Cornell University. The AWGIL director is supported by a Project Coordinator and group of “Lab Members” from Cornell University, CIMMYT and BISA. The AWGIL does not have an external advisory panel. Specific objectives of the AWGIL against which outputs and outcomes may be compared include: a) Early and expanded evaluation in India and Pakistan of four successive annual cohorts of CIMMYT candidate wheat varieties. b) Expanded field phenotyping of candidate wheat varieties for heat tolerance in South Asia. c) Genotyping: CIMMYT and National Agricultural Research Service (NARS) wheat breeders have access to detailed whole‐genome characterization of candidate wheat varieties. d) Genomic selection of improved candidate wheat varieties. e) Yield and heat tolerance are improved by application of Genomic Estimated Breeding Values (GEBVs) to selection of candidate wheat varieties in NARS testing and emerging from CIMMYT’s main breeding stream. f) Capacity development for heat stress breeding and application of genomic selection methodology. These specific objectives contribute to the overall development goal of delivering improved heat tolerant varieties with superior yield potential for improved food security and increased farmer income. 2 The Climate Resilient Chickpea Innovation Lab (CRCIL) was established in January 2014 with a cooperative agreement (AID-OAA-A-14-00008) of $5,999,952 for a program running through December 31, 2018. Chickpea (Cicer arietinum) is the world’s second most important pulse legume and, like most cultivated legumes, has narrow genetic and phenotypic diversity, making it challenging to breed for climate resilience or other high-value traits. The CRCIL aims to: characterize wild Cicer species from a representative range of environments to introduce wild diversity into phenology-normalized backgrounds for trait assessment and breeding; characterize the material by systematic phenotyping; develop a digital information network that explicitly identifies and quantifies the contributions of agronomically useful alleles; and develop improved chickpea varieties using an international consortium of chickpea breeders. This program also has a goal of collecting and characterizing a geographically diverse set of Mesorhizobium strains and making them available for further research or commercialization. The CRCIL is organized as a consortium led by Prof. Douglas Cook at the University of California, Davis. The CRCIL has several core partners, including the University of Vermont (where Dr. Bishop von Wettburg moved to upon leaving Florida International University; USA), University of Southern California (USA), Ethiopian Institute of Agricultural Research (Ethiopia), International Crops Research Institute for the Semi-Arid Tropics (ICRISAT), and Harran University (Turkey). The program has also established a wide network of leveraged partners, including Addis Ababa University (Ethiopia), Aegean Agricultural Research Institute (Turkey), Assam Agricultural University (India), Banaras Hindu University (India), Dicle University (Turkey), EGE University (Turkey), Ethiopian Institute of Agricultural Research (Ethiopia), Grains Research and Development Corporation (Australia), International Center for Agricultural Research in the Dry Areas (ICARDA), Legume Federation, University of Saskatchewan (Canada), and University of Edinburgh (UK). The CRCIL has a small External Advisory Panel (EAP). The Director absorbs the project management duties with no specific staff support beyond a webmaster. The original objectives of the project were to: a) Characterize a comprehensive collection of wild species focused on C. reticulatum, the wild progenitor of cultivated chickpea. b) Create reverse-introgression and advanced backcross introgression lines to (a) remove phenological barriers that otherwise impede the use of wild germplasm in breeding, (b) establish a resource for association mapping of climate-resilience traits, and (c) initiate breeding with superior wild alleles. c) Phenotype reverse-introgressed and advanced backcross introgression lines for a range of high-priority traits related to developing high-yielding, climate-resilient chickpea. 3 d) Develop a predictive network of genotype-phenotype associations that identifies genes and genome regions from wild species that improve chickpea’s yield resilience to climatic extremes. e) Provide data management and public-facing bioinformatics. f) Train and educate a gender-diverse group of young scientists from target countries. Evaluation Overview/Methodology For each of these two Feed the Future Innovation Labs, the EET was charged with assessing:  progress toward outputs and outcomes,  quality of the research program and its outputs,  what is and is not working well in implementation, including the effectiveness and contributions of the consortium management model,  information and recommendations that the Bureau for Food Security (BFS) can use to inform the development of current programming and the design of future programs to support productive and efficient research-for-development partnerships with developing-country scientists, including an understanding of how consortium-model ILs formed and managed their partnerships in pursuit of program goals. The evaluation questions for the EET relate to (1) research program, (2) program structure and management, (3) partnership and leveraging opportunities. A detailed evaluation plan, submitted on December 15, 2017, is attached in an appendix. The evaluation is based on desk review of annual reports and other documents provided by the AWGIL and CRCIL, a review of publications resulting from the activities of these labs, and interviews with individuals associated with the labs, including investigators, directors, staff, students and USAID contacts. Evaluation Research Program: Applied Wheat Genomics IL Findings The approved evaluation plan specified questions related to the AWGIL Research Program. These questions are addressed below:  AWG-RP 1. How effective has the annual transfer of lines from CIMMYT to India/Pakistan (1,000) and from NARS to CIMMYT (100-200) been? Have numerical goals been met? Are materials appropriately annotated? Transfers from Mexico to India and Pakistan are according to or exceeding the plan. In particular, an additional shipment was made in Year 1 that accelerated the program. 4 Numerical goals have been met or exceeded and all material has been annotated appropriately for subsequent use. Activity 1.2 in the original proposal stated that 100-200 lines annually would be transferred from India and Pakistan NARS to CIMMYT. From discussions with the Program Director and primary principal investigator (PI) in Pakistan it appears that this was a misstatement of intent. Instead, the plan was to transfer DNA from this target number of lines annually to CIMMYT for genotypic characterization. This would enable inclusion of these lines in modeling, as desired. Some DNA transfers have been made from Pakistan. Although they have not been as regular as proposed, the transfers have been sufficient to meet overall IL objectives. Transfers from India have not been possible due to Indian government restrictions.  AWG-RP 2. Has annual seed increase, initial phenotyping, and subsequent phenotyping under heat stress at the four locations across India and Pakistan taken place as planned and have data been put into a database? Numerical goals have been met or exceeded. Interviews consistently support the claim that data quality has been sufficient for subsequent application. Database creation, management, and curation has been done, but is suboptimal for achieving long-term outcomes; tools for subsequent data analysis are a challenge (see RP 6, RP 8).  AWG-RP 3. Has genotyping at KSU taken place as planned and have the resulting data been archived appropriately for use? Genotyping has been conducted as planned with data archived as planned. Anchoring markers to a physical map to indicate particular chromosome locations is a positive step.  AWG-RP 4. What is the status of determining genomic-estimated breeding values (GEBVs) for yield under stress and development and use of prediction models? What is the demonstrated utility of the exercise? Determining GEBVs remains a work in progress. It is significant that, despite data management challenges, prediction models have been run every cycle. One lesson from early work is that reliable prediction models require very large data sets. The multi￾location, multi-trait data sets that can enable high quality modeling have only recently become available. While some use of GEBV’s can now be made, the exact applications and the impact of using GEBVs on breeding program results is yet to be determined. Currently, GEBV predictions are effective within related germplasm pools and for particular locations, but less useful for more diverse germplasm pools or outside of environments used to create the models. The GEBV predictions can be expected to become more robust and broadly applicable as more germplasm and more field data are included, but predictions across diverse germplasm and fluctuating environments 5 will require continuous model updating. For the time being, modeling is being done by expert modelers in the US and at CIMMYT. The impact of the tools used in the AWGIL will be far greater if, in the future, the approach can be applied by researchers trained to use the models, rather than remaining only in the hands of researchers with expertise in building the models. The current approach seems appropriate for now, given that this type of modeling is not yet routine. Some types of data, including climate and some environmental data from Asia, have yet to be fully incorporated into models, but this work is to be completed in 2018. It is unclear how germplasm selections / advancements made using GS compare with those made using traditional phenotypic characterization. Indirect comparisons based on progress over time are ongoing. Although it is too early to draw conclusions, the EET anticipates that GS will increase the efficient utilization of the increased diversity created through this project.  AWG-RP 5. Have best varieties identified as outcomes of Objectives 1-4 been transferred to multiple partners for further testing as planned and what are the results of this testing? Are the models generated from Objective 4 being used more broadly in the CIMMYT breeding program? Has any impact on cycle time been seen from use of genomic selection? Concerning the transfer of varieties, interviews and information gleaned from annual reports indicate that expansion to the Bangladesh NARS has been largely successful. Field trials at a vastly larger scale than previously conducted in the country have been carried out at multiple locations over multiple years and desired phenotypic data have been obtained. Dissemination of selected germplasm to universities, research institutes, and industry in India and Pakistan is taking place. Some lines (selected before GS was being used) have entered and advanced within the national trial systems in India and Pakistan, supporting claims that germplasm from the AWGIL will be of value in Asia. Interviewees noted that the status and progress of varieties in these national trial systems can be difficult to observe from outside of the systems. Consequently, the exact status of lines from the AWGIL in these trials is presently unclear, but it is believed that the first approval for commercial release may take place within the next year. Providing target countries access to a wider range of germplasm has been a success of the AWGIL. Overall deployment of HTP tools has gone well in India, Pakistan and, more recently, Bangladesh. Perhaps the most significant accomplishment is the increased capacity to conduct large-scale field trials. The context and challenges differ across these target countries. In Pakistan there existed a good foundation in research capacity on which the AWGIL could build, resulting in an 6 increase in the scale of activity compared to historical norms. In India, the AWGIL met initial conditions and pre-existing capacity that varied across state and national programs, but again the project resulted in a significant overall expansion of activity. The AWGIL started from a lower baseline in Bangladesh in terms of scale, field equipment, computer hardware and software, infrastructure, and expertise, but multiple interviewees noted that due to the commitment of key researchers, major strides have been made and all field trial and data collection objectives to date have been achieved. Annual reports and interviews noted challenges using drones in Pakistan and Bangladesh, but these constraints were not a significant limitation to the AWGIL as researchers found workarounds to obtain the desired data. It is clear that the CIMMYT wheat breeding program is making use of GS as models are developed. Reliance on the new methods is being tempered appropriately until models are more fully validated. Given that GS models are still being developed and validated, it is too early to assess whether use of GS will have an impact on breeding cycle time.  AWG-RP 6. What is the status of deploying electronic data capture tools (hardware and software)? What is the status of implementing bar coding for seed inventory and management? Data capture tools (eg. Android OS based digital handheld tools) are being effectively deployed in all countries. Seed inventory and management tools are being deployed successfully, with caveats noted in AWG-RP 8. The AWGIL has contributed to significantly increased breeding capacity in all target countries. This success represents a major accomplishment for the AWGIL. Sustainability of this increased capacity is in question in the absence of improved infrastructure in Bangladesh and the lack of effective data management tools in all target countries. Despite not being involved to-date in making predictions from models, all partners are using these predictions to make their own germplasm selections for advancement. In addition, each annual cohort of germplasm sent out from CIMMYT has increased value to the target countries since target country data are used to create the GS models used to select the cohorts. This will result in a higher frequency of superior lines that can be selected by the breeders. The AWGIL has also succeeded in developing hands-on skills and capacity for HTP in all target countries. Based on interview feedback it appears that general familiarity with GS has increased across all partners but hands-on expertise outside of CIMMYT and US scientists remains a future goal. Infrastructure needs in Bangladesh have been noted. These needs exist in the field (equipment) and lab (computers, internet connections). Since much of the field work is 7 accomplished through manual labor, the introduction of small plot research equipment could greatly increase capacity and reduce costs. Field plot equipment has been built in India but attempts to move the equipment into Bangladesh have not been successful. There are plans to find machinists in Bangladesh to fabricate similar equipment. Addressing information handling gaps is either beyond the scope of this program (internet) or will require additional future funding (computers).  AWG-RP 7. Has the AWGIL met its academic training and technical capacity strengthening targets? Are the appropriate type and number of people being targeted for the right kind of training? In what ways has the program identified and addressed academic and technical capacity needs? How could this be improved? Long-term trainee statistics (numbers, demographics, gender) are shown in Table 1. Of the six PhD students listed in the 2016 Annual Report of the AWGIL, two were female and three were from outside the USA. No women from outside of the USA received degree-training through this IL. A very limited pool of qualified female candidates exists in target countries, essentially making it impossible to achieve gender balance in a single step. Moreover, no specific targets for gender inclusion were set for the AWGIL. The EET found that the AWGIL made significant efforts and achieved meaningful progress in the area of gender inclusion and an imbalance should not be viewed as resulting from neglect of the issue. Across interviews, assessments of the technical quality of the training were almost uniformly positive. Going forward, the impact of the training will depend on whether local conditions in the target countries (cultural, competing priorities, infrastructure) will allow trainees and local researchers to fully utilize what they learned while in the AWGIL. Table 1. Long-term Trainees, AWGIL Gender University Degree Major Program End Date Home Country Male KSU Ph.D. Genetics 2018 India Male KSU Ph.D. Genetics 2019 Bangladesh Female KSU Ph.D. Genetics 2019 USA Male KSU Ph.D. Genetics 2016 USA Male TERI University Ph.D. Plant Biotechnology 2015 India Female Cornell University Ph.D. Plant Breeding & Genetics 2019 USA Source: Applied Wheat Genomics Innovation Lab. (2016) Poland, J. (ed). 8 Short-term training sessions were held in target countries for people outside the AWGIL. The primary purpose of these trainings was to increase awareness of the germplasm being made available. The FTFMS reporting indicates that 211 individuals participated in short-term training between 2014 and 2016, compared to a target of 220. Unfortunately, records of who attended the sessions are incomplete and no formal feedback was obtained after the sessions. Consequently, there is no evidence beyond the generally positive impressions of trainers that can indicate the success of these trainings. Other sessions for researchers from national programs and universities did target attendees and covered topics including electronic data capture (and tools) and phenotyping. No record of attendees were kept nor was formal feedback obtained, but adoption of tools like HTP methods and electronic field books were cited as measures of some success. National programs expressed desire for more training, which was cited as evidence that such trainings should be emphasized in the future.  AWG-RP 8. What research-related challenges has the AWG Project faced during research design and implementation? In what ways have those challenges been addressed? What opportunities are there to improve research outputs moving forward? Data management presents the most significant challenge not covered adequately elsewhere in this review. The scale of data being captured and then utilized by the AWGIL (terabytes of new data annually) has outpaced existing management software for all partners. The resulting gaps have been addressed within the AWGIL using short￾term workarounds, but these gaps will increasingly be impediments for AWGIL researchers as the total amount of data continues to increase. Data management constraints will be even more problematic for researchers outside the IL who want to work with the materials and potentially apply GS. Solutions that had been planned (software developed at CIMMYT) did not materialize. A clear resolution does not exist for this issue and addressing it is beyond the scope or funding of the AWGIL. As noted in RP7, infrastructure needs in Bangladesh represent another challenge to the AWGIL. To some degree, the AWGIL developed workarounds to pursue its immediate goals in the face of these challenges, but unless addressed, continued infrastructure deficiencies will likely undermine the long-term impact of the AWGIL’s work in Bangladesh. Uncertainty about ongoing funding for work in Pakistan presents a challenge. With Pakistan’s wheat productivity program ending, the ability to leverage those funds for this program will come to a close.  AWG-RP 9. Given the dual benefit mandate of Title XII authorized programs, such as the AWGIL, is there a sufficient balance between research efforts directed towards priorities of the domestic wheat stakeholder community and the wheat research priorities required to advance global food security goals, particularly in FTF countries? 9 Publications are available globally and indicate the broad reach of the AWGIL. To date, the AWGIL has generated more than 25 publications (Table 2). The mean Impact Factor of journals for the highest ranked 10 publications is 8.39. Even though these publications are recent, many of them are already widely cited. However, we do not present citation counts for specific articles at this time, as such measures are not reliable markers of quality for work as recent as the publications of the AWGIL and CRCIL. Table 2. Publications, AWGIL Author (year) Title Journal Journal Impact Factor Crossa, J. et al. (2017) Genomic Selection in Plant Breeding: Methods, Models, and Perspectives. Trends in Plant Science 13.44 Lado, B., et al. (2017) Strategies for selecting crosses using genomic prediction in two wheat breeding programs. The Plant Genome 10(2) 2.74 Montesinos López, O.A, et al. (2017) Prediction of multiple-trait and multiple-environment genomic data using recommender systems. G3 :Genes | Genomes | Genetics 2.91 Dunckel, S. et al. (2017) Genomic selection for increased yield in synthetic-derived wheat. Crop Science, 57(2) 1.62 Juliana, P. et al. (2017) Genomic and pedigree-based prediction for leaf, stem, and stripe rust resistance in wheat. Theoretical and Applied Genetics, 1–16 4.13 Juliana, P. et al. (2017) Comparison of models and whole￾genome profiling approaches for genomic-enabled prediction of Septoria tritici blotch, Stagonospora nodorum blotch, and tan spot resistance in wheat. The Plant Genome, 10(2) 2.74 Pérez￾Rodríguez, P. et al. (2017) Single-step genomic and pedigree genotype × environment interaction models for predicting wheat lines in international environments. The Plant Genome, 10(2) 2.74 Sehgal, D. et al. (2017) Identification of genomic regions for grain yield and yield stability and their epistatic interactions. Scientific Reports, 7, 41578 4.26 Tanger, P. et al. (2017) Field-based high throughput phenotyping rapidly identifies genomic regions controlling yield components in rice. Scientific Reports, 7, 42839 4.26 10 Guzman, C. et al. (2016) Wheat quality improvement at CIMMYT and the use of genomic selection on it. Applied and translational Genomics Tack, J. et al. (2016) Quantifying variety-specific heat resistance and the potential for adaptation to climate change. Global Change Biology, 22(8), 2904–2912 8.44 Rutkoski, J.et al. (2016) Canopy temperature and vegetation indices from high-throughput phenotyping improve accuracy of pedigree and genomic selection for grain yield in wheat. G3: Genes| Genomes| Genetics 2.91 Poland, J. and J. Rutkoski (2016) Advances and challenges in genomic selection for disease resistance. Annual Review of Phytopathology 54(1) 12.35 Battenfield, S. et al (2016) Genomic selection for processing and end-use quality traits in the CIMMYT spring bread wheat breeding program The Plant Genome 2.74 Crain, J. et al. (2016) Development and deployment of a portable field phenotyping platform. Crop Science 56(3): 965- 975 1.62 Montesinos￾López, O. A., et al. (2016) A genomic Bayesian multi-trait and multi-environment model G3:Genes | Genomes | Genetics. 6:2725-2744 2.91 Haghighattalab, A., L.et al.(2016) Application of unmanned aerial systems for high throughput phenotyping of large wheat breeding nurseries. Plant Methods 12(1): 1- 15 4.14 Edae, E., R. Bowden, and J. Poland (2015) Application of population sequencing (PopSeq) for ordering and imputing genotyping-by￾sequencing markers in hexaploid wheat. G3: Genes|Genomics|Genet ics 5(12): 2547-2553 2.91 Montesinos￾López, O. A., et al. (2016) Predicting grain yield using canopy hyperspectral reflectance in wheat breeding data. Plant Methods (accepted) 4.14 Juliana P. et al. (2015) Genome-wide association mapping for leaf tip necrosis and pseudo￾black chaff in relation to durable rust resistance in wheat. The Plant Genome. Vol. 8, No. 2 2.736 Dunckel, S., et al. (2015) Genetic mapping of race-specific stem rust resistance in the synthetic hexaploid W7984 x Opata M85 mapping population. Crop Science. 55:1–9 1.62 Poland, J. (2015) Breeding assisted genomics. Current Opinion in Plant Biology: 24,119–124 7.357 11 Rife, T. et al. (2015) Spiked GBS: a unified, open platform for single marker genotyping and whole-genome profiling. BMC Genomics 16(1): 248 3.729 Rutkoski, J., et.al. (2015) Genetic gain from phenotypic and genomic selection for quantitative resistance to stem rust of wheat. The Plant Genome 8(2) 2.736 Rutkoski, J.,et al. (2015) Efficient use of historical data for genomic selection: A case study of stem rust resistance in wheat. The Plant Genome 8(1) 2.736 M. Lopez‐Cruz, J. et al. (2015) Increased prediction accuracy in wheat breeding trials using a marker × environment interaction genomic selection model. G3: Genes|Genomes|Geneti cs 2.91 Genomic selection has not been adopted broadly for wheat breeding anywhere. If use of GS in the AWGIL is successful, and it is still too early to make such a broad unqualified claim, the precedents being set will demonstrate through publications and the personal involvement of researchers how it can be utilized. In countries with sufficient infrastructure (mainly developed), this will enable use by interested researchers in the public and private sectors. Infrastructure issues along with competing priorities will make use in developing countries a decision for case-by-case evaluation. Conclusions  The AWGIL has succeeded in meeting initial research objectives: increasing access in India, Pakistan, and Bangladesh to CIMMYT wheat germplasm; increasing field trial capacity and associated use of HTP tools in these countries; and deployment of electronic data capture systems. It has made good progress towards more forward￾looking objectives like development and use of GEBV models for GS.  While progress has been made, demonstration of the overall value of GS to wheat breeding is a work in progress.  The capacity of the CIMMYT wheat breeding program has increased through its implementation of HTP and GS. Although not yet fully demonstrated, the EET does agree with the AWGIL that this should increase genetic gain over time, thus benefiting not only South Asia, but wheat improvement globally. Lessons Learned  Achieving gender balance is a long-term challenge that is inhibited by contextual issues beyond the control of any Innovation Lab. Human capacity building is essential, but stronger institutions and infrastructure may be needed to capitalize on human capacity building in target countries. Creating environments and institutions that support inclusion of women in advanced training and allow application of increased human 12 capacity upon completion of training both require timeframe and funding levels that exceed those associated with ILs.  Pitfalls seemingly beyond the scope of the AWGIL should be clearly shared with all concerned partners and sponsors as soon as they are identified and understood. Examples include shipment of materials from India, data management system and infrastructure (field equipment, computer-related) challenges. Based on the materials provided to the EET and from interviews it appears this may have taken place inconsistently (although it may also be true that in some cases all relevant parties were made aware of issues but were unable to address them within the timeframe of the IL). Recommendations The EET recommends that the following areas receive increased emphasis in the future:  Limitations to data storage, management, and accessibility need to be considered and addressed.  Now that large-scale field trials have been established in the target countries: o the number of target countries can be expanded; o HTP platforms can be expanded past hand-held; and, o routine use of GEBV modeling for GS can be expanded beyond the experts in the US and at CIMMYT.  Improving infrastructure in Bangladesh and new target countries.  Including funding for work in Pakistan.  Improving and formalizing records for and feedback from short-term training. Research Program: Climate Resilient Chickpea IL Findings The approved evaluation plan specified questions related to the CRCIL research program. These questions are addressed below:  CRC-RP 1. Establishing collections: Has the intended sequencing of C.reticulatum accessions been done? Has collected germplasm been increased / preserved as planned? Have the desired Rhizobium strains been collected? The quantitative goals for germplasm collection (plant and microbe) and sequencing have been exceeded. This was accomplished both by making additional collections and by leveraging materials previously collected and archived by others, but not generally available (e.g. Vavilov chickpea relative collection in Russia). New collections have been annotated and deposited as planned. 13  CRC-RP 2. Environmental characterization: For all accessions, were climatic, soil and other environmental data obtained and archived as planned? For all accessions, all essential environmental data were obtained and archived largely as planned. Some interviewees noted that obtaining additional data from Turkey was planned and would have been valuable; however, the civil war in nearby Syria prevented the necessary researchers from traveling to Turkey at appropriate times.  CRC-RP 3. Genotyping: Were all plant genotypes obtained as planned? (Phase I – deep sequencing of 250 individuals from 20 wild populations to guide parent selection; phase II - <1X coverage of <1000 individuals from populations then created to look at allele diversity.) Was planned subset of Rhizobium isolates (480) genotyped / sequenced? What was learned about G x E associations for C. reticulatum? The CRCIL met or exceeded all plant and microbe genotyping and sequencing goals. Significant progress understanding GxE associations for drought and initial progress for heat tolerance, yield and tolerance to some diseases has been made. Drought associations are being used for identification and selection of specific segregating populations for further study and development.  CRC-RP 4. Population creation: Were reverse introgressed lines (revILs) for flowering time and apical dominance traits generated as planned (5 cultivated x 20 wild accessions, w/ 50 indiv per pop)? Did this produce materials that breeders could use for subsequent pop development? Were advanced backcross introgression (ABI) lines generated in both Indian and Ethiopian cultivars as planned? Were F2s genotyped as planned? Was desired representation of chromosomal segments achieved? Progress with use of original revIL crosses for ABI development? Although the details of revIL population creation have changed since the project plan was developed, what has been done or is underway remains consistent with the overall strategy and intent for development of these populations. Fewer cultivated genotypes have been used for the most advanced materials than was initially planned, but the size of the populations created is larger than initially proposed. This has some negative downstream implications for direct breeding application of the most advanced materials in some locations and environments but significantly increases the genetic resolution of the populations. Populations created using additional cultivated genotypes are being developed but are generations behind the most advanced. The overall number of populations created has exceeded the plan, which is a mark of success but creates downstream logistic and management challenges. These challenges are being met by leveraging additional support (funding, personnel, etc.) in some cases (e.g. from Mars 14 and Grains Research and Development Corporation (GRDC)) and prioritizing advancement of lines in others. Based on interview feedback and considering the upstream nature of the CRCIL’s primary goals, creating materials of direct interest to breeders was always a stretch goal for this IL. Nonetheless, some of the populations created have already been accessed. ICRISAT breeders in India have obtained one population for evaluation and Ethiopian researchers have shown interest in some F5 materials. Multiple interviewees noted that populations need to be advanced several more generations before they can be more broadly considered for cultivation. Based on resource constraints and perceived overall significance of ABI materials to CRCIL objectives (relative to the revIL populations), a decision was made during the second year of the project not to create the ABI materials. It is clear from interviews that significant discussion went into this decision and that all key parties were aligned around the decision. At around the same time a decision was also made not to fix populations for flowering time and architecture as originally planned. The rationale was that the advantages of minimizing associated recombination drag outweighed the disadvantage of the increased ongoing need for phenotyping, particularly phenotyping of flowering time. While there may have been differences of opinion during these discussions, all key parties appear to have ultimately supported this decision. The quality of the chickpea physical map has limited the extent to which representation of chromosomal segments can be determined to date, with about 50 percent of the populations anchored, and the rest remaining works-in-progress.  CRC-RP 5. Phenotyping: Status of revILs phenotyping. Were desired number of lines evaluated for desired traits and across desired locations (California, India, Ethiopia)? Is the status of Rhizobium N fixation efficiency evaluations (multiple revIL’s x multiple Rhiz isolates) appropriate? Is the status of seed nutrient characterizations of 300 revILs from Ethiopian parent consistent with plans? Has work facilitated selection of lines for breeding pops? What is the status of ABI line phenotyping? Was desired characterization of agronomic and yield traits done? Phenotyping of revIL population has been consistent with or has exceeded planned targets in all respects. Evaluation of N fixation efficiency of collected symbiotes remains in progress. Strains have not yet been identified for product development but experiments with commercial partners are planned for 2018. This aspect of the CRCIL is an extension from what was originally planned and has the potential to expand the commercial inoculum industry in India and help develop such an industry in Ethiopia. 15 Some seed nutrient characterization has been done in collaboration with partner researchers at the Danforth Foundation and in Canada. The results of such studies are of potential relevance to partners like Nestlé, who is evaluating use of chickpea as a processed protein source. The studies do not provide insight into nutrients in chickpea as used directly for food in countries like Ethiopia and India. Future studies would be needed to assess this. The completed studies were suboptimal due to use of different seed sources and analytical methods by each partner and the results obtained have not been of value for any population selections. Given the decision not to pursue creation of ABI populations, there is nothing to report concerning ABI line phenotyping.  CRC-RP 6. Associations: What is the status of using revILs to make genotype-phenotype associations for climate resilience traits? Have revILs with desirable (wild) traits (fixed or not) been identified? Have potential undesirable cultivated x wild allele interactions been identified? Association of drought tolerance from some wild accessions with specific revILs was noted in the 2017 report. Further work with select lines has been initiated but nothing more can be said at this time. Initial work with multiple biotic and other abiotic stresses has been done but, consistent with development of populations segregating for these traits, is running behind the drought work. Some genomic regions fixed or enriched for during domestication have been identified.  CRC-RP 7. Electronic data management: Are tools being used for data repository and management working as desired? Are data being deposited in timely manner? Utility / accessibility of data? Have desired improvements emerged? What is the status of creating public-facing system(s)? Tools appear for the most part to be available and are being used. Interviewees indicated that the amount of phenotype data now being collected has strained systems in at least one target country (India), temporarily impacting use of the data. Work to address this is reported to be ongoing. Interviewees also noted that some of the systems and tools working acceptably today for researchers in the CRCIL will not be as useful in the future if they must support researchers outside the IL who may want to work with the materials and associated data. The IL is attempting to address this during 2018. Better software systems from the Consultative Group for International Agricultural Research (CGIAR) system would help address this gap. While all data have not yet been made public, this objective was repeatedly referred to as a priority during interviews and plans are to have a public-facing system running before the end of the current phase of funding. 16  CRC-RP 8. Has the CRCIL met its academic training and technical capacity strengthening targets? Has the desired diversity amongst project participants (including gender) been achieved? Is training targeting the right people? How has the program identified and addressed academic and technical capacity needs? How could this be improved? What is the status of the planned short-term fellowship program? An interesting strategy for long-term trainee recruitment was noted in multiple interviews. CRCIL leadership recognized early on that a training challenge inherent in programs like this is post-training “reintegration” of trainees from target countries into roles in those countries where they can continue to make use of their training. Given this, a decision was made to focus on recruiting trainees with existing positions at established target country research institutions and have their primary affiliation during training remain within the target country (typically at the same institute). While recognizing that this could impact the quality of training received (although there has been an attempt to overcome this challenge through liberal use of short-term training stints with experts at all IL locations) and limit the diversity of the pool of candidates (particularly in areas like gender), the potential “re-entry” upside and knowledge the trainees would have of how to operate most effectively within their home institutions made this an experiment worth running. As shown in Table 3, the CRCIL was able to recruit multiple female trainees, consistent with established targets for gender inclusion. Although not shown in Table 3, a total of more than 40 long-term trainees (with a nearly equal male-female distribution) have been associated with the CRCIL when those supported using leveraged funds are included. It is too early to determine if the hoped-for upside of the CRCIL’s recruitment strategy will be realized, but the EET commends the IL for trying this approach and encourages USAID to monitor outcomes over time as trainees leave the IL and go back to their full-time roles. As noted, short-term training sessions on novel techniques and tools were conducted for long-term trainees outside their home labs. Because only informal oral feedback was obtained, nothing beyond the generally positive impressions of trainers and positive impressions obtained by the EET during an interview with the trainees can be used to measure success. Given the “pre-breeding” focus of the IL, training beyond program participants and partners has not been pursued and this seems appropriate. Expanding the scope of training would be a reasonable objective if the program transitions in more applied directions in the future. 17 Table 3. Long-term Trainees, CRCIL Gender University Level Major / Area of Focus Program End Date Home Country Male* Addis Ababa University PhD Plant Biotech TBD Ethiopia Female* Addis Ababa University PhD Microbiology 2018 Ethiopia Male* Addis Ababa University PhD Plant Genetics 2018 Ethiopia Male* Addis Ababa University PhD Plant Genetics 2018 Ethiopia Male* Addis Ababa University PhD Plant Genetics TBD Ethiopia Male* Hawassa University MS Plant Genetics 2017 Ethiopia Female* Addis Ababa University PhD Microbiology 2020 Ethiopia Female* Addis Ababa University PhD Plant Genetics 2020 Ethiopia Male* Haromaya University PhD Plant Pathology TBD Ethiopia Female** University Southern California Post-Doc Computational Biology Not applicable (N/A) India Female** University Southern California Post-Doc Genomics N/A Portugal Male** UC-Davis Post-Doc Population Development N/A Ethiopia Male** UC-Davis PhD Plant Pathology 2018 USA * Included on 2017 FTFMS list ** Not included on 2017 FTFMS list 18  CRC-RP 9. What research-related challenges has the CRCIL project faced during research design and implementation? In what ways have those challenges been addressed? What opportunities are there to improve research outputs moving forward? Several challenges are noted elsewhere in this section. Delays initiating work in the first program year (timing of fund availability; visa challenges in Turkey and Ethiopia) had a variety of impacts. Many of these adverse effects have been addressed but some have had ongoing implications. For example, population development has not fully recovered from time lost in the first program year. The CRCIL encountered multiple institutional and structural barriers to conducting trials of the desired scope and scale in Africa and Asia. It appears from interviews that while, in retrospect, some of these barriers could possibly have been better anticipated, the scale and scope of the majority of work was somewhat novel to all participants, who repeatedly modified plans and/or found workarounds that enabled major objectives to be accomplished. Significant differences in flowering time between cultivated chickpea and wild relatives also created challenges for, in particular, making wild relative x cultivated chickpea crosses in the field in Ethiopia. The CRCIL responded to this challenge by moving more work to the University of California at Davis and having researchers from multiple participating labs do short stints in Davis to help with this work.  CRC-RP 10. Given the dual benefit mandate of Title XII authorized programs, such as the CRC IL, is there a sufficient balance between research efforts directed towards priorities of the domestic chickpea stakeholder community and the chickpea research priorities required to advance global food security goals, particularly in Feed the Future countries? The IL has taken multiple steps to balance efforts targeted at Feed the Future countries, the USA, and other developed countries. The CRCIL has generated research publications in widely available outlets. The mean Impact Factor of journals in which the CRCIL’s six journal publications appear is 5.167 (Table 4). Additional publications are expected before the end of the current funding cycle. 19 Table 4. Publications, CRCIL Climate Resilient Chickpea Lab: Publications and Impact Journal Journal Impact Factor Wettberg et al. (2017) Ecology and genomics of an important crop wild relative as a prelude to agricultural innovation Nature Communications 12.12 Syed Gul Abbas Shah Sani et al. (2017) Exploring the genetic diversity and population structure in Chickpea (Cicer arietinum) cultivated germplasm from Pakistan. Plant Genome 2.74 Plekhanova, E. et al. (2017) Genomics and phenotypic analysis of Vavilov’s historic landraces reveals the impact of environment and genomic islands of agronomic traits Scientific Reports 7: 4816 4.26 Kahraman, A. et al. (2017) Distinct subgroups of Cicer echiospermum are associated with hybrid sterility and hybrid breakdown in interspecific crosses with cultivated chickpea. Crop Science 1.63 Bisseling T. and Cook D. (2017) Live on Air: Rhizobium and legumes, the nitrogen fixing nodule symbiosis. Microanaon Foundation, The Netherlands none found Mirlan Tilap et al. (2017) Agro-morphological traits of Cicer reticulatum Ladizinsky in comparison to C. echinospermum P.H. Davis in terms of potential to improve cultivated chickpea (Cicer arietinum L.) Genetic Resources and Crop Evolution. In press. none found R. Varma Penmetsa et al. (2016) Multiple post-domestication origins of kabuli chickpea through allelic variation in a diversification￾associated transcription factor. New Phytologist 211:1440- 1451 7.67 Douglas R. Cook (2016) Molecular breeding in legumes for resource-poor farmers: Chickpea for Ethiopia and India. Proceedings of the FAO International Symposium on the Role of Agricultural Biotechnologies in Sustainable Food Systems and Nutrition. FAO, Rome, Italy none found Toker C. et al. (2014) Cicer reticulatum Ladizinsky, progenitor of the cultivated chickpea (C. arietinum L.) Legume Perspectives 5: 26- 27 none found Warschefsky E. et al. (2014) Back to the wilds: tapping evolutionary adaptations for resilient crops through systematic hybridization with crop wild relatives. American Journal of Botany 101: 1791-1800 2.59 20 Although it is too early for comprehensive assessment, the partnerships established with Canadian and Australian researchers will very likely result in outputs from the IL impacting chickpea research and development broadly in developed countries. Downstream, interest from commercial partners like Nestle could also evolve into IL impact on global processed protein use. Conclusions  The CRCIL succeeded in meeting its initial objectives – increasing the genetic diversity of broadly available chickpea (including wild relatives) and associated microbial symbiotes.  Efforts to meet more forward-looking objectives, such as development of segregating populations for direct use in breeding, development of GxE associations, and advancing symbiotes towards commercialization, remain underway.  The CRCIL took an innovative approach to recruiting long-term trainees. It will be important to follow the long-term outcomes of this approach. Lessons Learned  If expectations between researchers and USAID are aligned and remain aligned through execution of a program, USAID can effectively sponsor and support upstream / “pre￾breeding” research like the CRCIL.  Programs with a pre-breeding focus, like the CRCIL, can reasonably be expected to have more shifts in objectives than is typical in more applied programs. Recommendations The EET recommends that the following areas receive increased emphasis in the future:  Increase its emphasis on chickpea breeding applications as well as on chickpea consumption by people in target Feed the Future countries.  Develop more formal feedback systems from short-term training sessions.  Consider use of a “Program Coordinator” who could assist the Program Director ensuring timely follow-up on action items from meetings and potentially addressing administrative, logistical/tactical issues as they arise. Specific responsibilities should be based on objectives and needs identified by the Program Director. Program Structure & Management Findings Specific questions regarding program structure are specified in the evaluation plan. These questions are reproduced and addressed in this section. 21  AWG-PS 1. How effectively has the AWGIL Program Director managed research and training activities across partners and stakeholders? In what ways has the PI supported coordination and communication across partners to ensure complementarity of effort and contribution to AWG’s objectives? In what ways could communications and coordination be improved as a way to better achieve outcomes? All parties interviewed cited the partnerships between KSU, CIMMYT, and key organizations in India, Pakistan and Bangladesh as being strong and effective. The EET found no evidence to suggest otherwise. All partners reported that the quality and quantity of internal communication has been good. Several interviews noted that the quality and consistency of communication improved when the Project Coordinator was hired. Several interviewees felt external communications other than academic publications, including press releases, information on the program web site, and some communication with funding agencies, is an area of weakness. The Project Coordinator intends to make such external communication a focus of effort during the last year of the current funding cycle. All partners appear satisfied with decision-making in the program, feeling that decisions are made at appropriate levels, with the Program Director effectively making the decisions most appropriately made at that level.  CRC-PS 1. How effectively has the CRC IL Program Director managed research and training activities across partners and stakeholders? In what ways has the PI supported coordination and communication across partners to ensure complementarity of effort and contribution to CRC’s objectives? In what ways could communications and coordination be improved as a way to better achieve outcomes? Overall, the Program Director received high marks in multiple interviews regarding his leadership of the CRCIL in all areas (communication, coordination and prioritization of activities, decision-making, budgets, commitment to dissemination of information, etc.). As noted, communication across CRCIL partners and with USAID staff appears to be good. One comment that did arise from multiple interviews was that the Program Director is spread so thinly that communication is sometimes less prompt or effective across partners than might be desired. That said, the EET found no evidence that occasional lapses in communications were due to any lack of commitment on the Director’s part or that they resulted in failure to advance or achieve key CRCIL objectives. Regarding decision-making, multiple interviewees noted the Program Director’s commitment to ensuring that decisions are made at appropriate levels. One challenge 22 inherent with “upstream” research is that there is limited evidence for prioritizing one strategic decision over another. In a completely consensus-driven organization, this can lead to gridlock. The Program Director has avoided this by taking a strong leadership role over such decisions. This centralization seems appropriate for strategic decisions and has probably contributed to the CRCIL’s progress to date. Multiple interviewees indicated that relieving the Director of some very tactical or logistical responsibilities (or as some termed them, “administrative” responsibilities) might open up some time for other activities. The CRCIL does not utilize anyone outside the Program Director with specific project “management” or “coordination” responsibilities. As noted above, centralizing authority with the Director has advantages in a program like this, but the EET believes there are a number of more tactical or administrative tasks that could be delegated to someone else, under the direction of the Director. Delegating specific administrative duties would help ensure that the Director has more time for higher level contributions needed to run the program effectively. Implications of this go beyond the CRCIL and details of how this could work are discussed further in the Evaluation Conclusions section.  CRC-PS 2. What role has or could the External Advisory Panel play in contributing to program effectiveness and advice relating to broad research priorities in the wider breeding field? The CRCIL does make use of a two-member EAP. Their charge is informal and engagements are relatively infrequent. They attend annual meetings and informally provide verbal advice to IL leadership during those meetings. Some infrequent informal conversations have also taken place outside of those sessions. While it is clear that the EAP has provided useful input to the CRCIL, it is not clear that all smaller, consortium-model, ILs need an EAP. An EAP can potentially provide valuable input to a program like this in three areas: (i) technical, (ii) target country understanding, and (iii) mentorship for leading an IL. Key technical as well as target country needs should be met through IL participants, not advisors. The EET understands that a committee of all IL Directors exists that could serve the mentorship function. Thus, while an EAP can provide value, it is one more program component the Director has to manage. The EET feels that for consortium-model ILs, decisions about whether to utilize an EAP and what areas of expertise to have on an EAP should be made on a case￾by-case basis. Where used, a fairly informal, advisory role does seem appropriate but as noted by some interviewees, some written input should be obtained from the EAP after engagements of any significance. This would encourage the EAP members to formulate their thoughts and would provide a better resource for the Program Director and other team members. 23 Conclusions Concerning program structure and management, the EET evaluation observed the following:  Good communication and coordination was the norm across both ILs.  Both ILs have effective Program Directors.  The EAP utilized by the CRCIL clearly added value to that program. It is not clear that all consortium-model ILs need an EAP.  The AWGIL made good use of the “Project Coordinator” role. Lessons Learned  If an EAP is to be used, firm commitments from members and clear expectations for the EAP prior to engagement will increase the value of such a group. Recommendations  The CRCIL should consider employing a non-researcher as a program coordinator to handle a number of administrative and organizational tasks.  The AWGIL should evaluate how the program coordinator role can evolve so it best meets the ongoing needs of the IL and the Program Director.  The AWGIL should increase use and funding of short-term training sessions to bring knowledge of the research tools and germplasm to broader and more applied audiences. Better record keeping for participation and use of a formal feedback mechanism will aid in evaluating the value of such sessions. Partnership & Leveraging Opportunities Findings Specific questions regarding partnership and leveraging are specified in the approved Evaluation Plan and addressed here.  AWG-IC 1. In what ways, if any, did the location-based efforts (Mexico, US, India, Bangladesh and Pakistan) complement other wheat research activities in these regions? What are the lessons learned that can improve the complementarity of future USAID research and development investments in wheat breeding to other ongoing programs in the regions? In what ways can USAID programs in wheat breeding more effectively engage developing country partners in research? 24 Integration of new tools and technologies (including electronic data capture and GS) needed to be achieved first at CIMMYT, which could then “model” for the Asian researchers. Close partnership with CIMMYT has allowed this integration to take place. Overall deployment of HTP tools and related techniques has gone well in India, Pakistan and, recently Bangladesh (see details under Research Program). The increased capacity to conduct and phenotype large-scale field trials will have a long-term impact not only in wheat variety development, but also in other crop and management research.  AWG-IC 2. What types of changes in implementation would more effectively achieve outcomes in the following components: research (design, implementation, communications, stakeholder involvement), capacity development (student recruitment and selection, content, location) and/or institutional collaboration? Research gaps are discussed elsewhere in this evaluation as are communication and stakeholder involvement. Overall design and implementation are viewed as having been very successful. Most planned objectives were met and those that have not yet been fully accomplished were reasonably considered to require a timeframe of longer than five years. As noted elsewhere, student training was viewed by interviewees as successful. The EET largely agrees with this assessment. The IL recruited a good set of students but the absence of female long-term trainees from outside the US is a gap. It is worth noting that a Bangladeshi student at KSU is leading the Bangladesh field trial component of the IL. The ability to bring expertise not already present in that target country was cited as one reason expansion of the AWGIL to Bangladesh has gone as well as it has. While having a Bangladeshi student at KSU was fortuitous in this case and may be hard to achieve across programs and time, it is a potential ingredient for success that may benefit other programs. Increased involvement of all national programs would facilitate adoption of tools and approaches, and this was noted specifically for Pakistan in interviews  AWG-IC 3. What leveraging opportunities have developed from the AWGIL and what advantage has been taken of these opportunities? Complementary funding opportunities or grant applications? Engagement with private sector partners? Use of research outputs or program capital investments beyond the IL? Interviewees noted some examples of leveraging opportunities, but this has not been a priority of the AWGIL to date. One example of leveraging with a British International Aid Agency-Bill and Melinda Gates Foundation project was cited and a few less well-defined or emergent efforts and future plans were noted. An uncertain US funding environment 25 was cited by some interviewees as an issue of concern that has impacted how much time they have put into leveraging.  CRC-IC 1. In what ways, if any, did the location-based efforts (US, India, Ethiopia) complement other chickpea research activities in these regions? What are the lessons learned that can improve the complementarity of future USAID research and development investments in chickpea breeding to other ongoing programs in the regions? In what ways can USAID programs in chickpea breeding more effectively engage developing country partners in research? Feedback from interviews, including individuals from target countries, supports that the CRCIL is delivering on the objective of providing significant new diversity to the chickpea research community (wild chickpea relatives, introgression populations and symbiotes) and that the plant materials generated will be useful to breeding programs, particularly in India and Ethiopia, moving forward. Given the upstream focus of the CRCIL, it is difficult to see how, at this time, broader engagement of the chickpea research community in developing countries would be of value. Broader downstream engagement should become a greater focus of more applied efforts moving forward.  CRC-IC 2. What types of changes in implementation would more effectively achieve outcomes in the following components: research (i.e., design, implementation, communications, stakeholder involvement), capacity development (i.e., student recruitment and selection, content, location) and/or institutional collaboration? This subject is addressed elsewhere in the evaluation within responses to other questions.  CRC-IC 3. What leveraging opportunities have developed from the CRC IL and what advantage has been taken of these opportunities? Complementary funding opportunities or grant applications? Engagement with private sector partners? Use of research outputs or program capital investments beyond the IL? The network of researchers and institutions developed as part of the CRCIL is impressive (Table 5). The additional resources including people, funding, and accessions, which the Program Director has brought to the IL are significant and have increased the critical mass of researchers and associated partners working together in the area. The development of this extensive network has resulted in a broadening of the original objectives, with some of the apparent broadening being associated with work taking place outside the USAID-funded project. Interviews show that shifts in objectives and scope were discussed openly and were acceptable to all parties involved, including 26 USAID points of contact. It will be important for any future efforts to consider how to maintain this community. Further leveraging opportunities are under discussion, including support for continued population development and management as well as acquisition of additional plant genetic diversity. Table 5. Extent of network and funds leveraged for Climate Resilient Chickpea IL (from CRCIL 2017 annual report). Sponsors Funding 1USAID (Washington, India, Ethiopia) $5,999,999 2US National Science Foundation $3,347,609 2USAID-sponsored Partnership for Enhanced Engagement in Research $239,000 2USAID-sponsored Legume Innovation Lab $235,000 2USAID-sponsored US Borlaug Fellows in Global Food Security program $22,000 2USAID-sponsored US-Pakistan Center for Advanced Studies $66,000 2 Pakistan Higher Education Commission $36,000 2Global Crop Development Trust $414,000 2 TATA Trusts $150,000 (est) 2 2Blades Foundation $75,000 2Mars Incorporated $15,000 2 CGIAR Consortium Research Program 3.5 on Grain Legumes $29,000 2 Indo-US Science and Technology Forum $32,000 1,3, 4Australian Grains Research and Development Corporation $5,175,000 3 Saskatchewan (Ministry of Agriculture and Pulse Growers Ass’n) $500,000 (est) 3Nestle SA $75,000 (est) 3UK Biotechnology and Biology Research Council $970,000 (est) 3 Chinese Sch Council $48,000 TOTAL >$17,000,000 1 Core Innovation Lab funding. 2 Full or partial funding and/or management responsibilities at UC Davis. 3 Parallel funding, not involving UC Davis. 4 $175,000 core at UC-D; $5,000,000 (est) parallel funding, within Australia Conclusions  Capacity building for large field trials and HTP in the AWGIL has been a success. Capacity building for routine use of GS is a work in progress.  The CRCIL Program Director has identified and leveraged a tremendous range of opportunities.  Having someone in the AWGIL from Bangladesh but working in the Project Director’s lab helped enable expansion of that program into Bangladesh. 27 Lessons Learned  Individuals from target countries seeking advanced degree training through ILs may enable expansion of IL activities.  Leveraging through addition of a multiplicity of funders and partners can expand the impact of an IL, but also raises risks of diluting efforts and straining capacity. Successful leveraging requires thoughtful consideration of opportunities. Recommendations  As more understanding is gained about how best to use GS for wheat breeding, a strategy for educating the global wheat breeding community would be beneficial. Although this emerging technology is far from routine, the lessons learned by the AWGIL can provide important guidance as other plant improvement programs begin to implement GS  In the future, a clear strategy for maintaining the network of chickpea partners will be needed to address new objectives. Maintaining such relationships is often harder than forming them. 28 Conclusions In addition to observations and conclusions made for each IL noted above, the EET made several observations across ILs that merit highlighting. Both ILs have foundational, as well as more forward-looking, downstream objectives. For the Applied Wheat Genomics Innovation Lab, the first category includes (i) making available to Asian researchers a broader array of CIMMYT germplasm (ii) more uniform broad use of advanced phenotyping methods and (iii) increasing use of electronic data collection systems while the second category includes development and routine application of genomic selection and prediction models. For the Climate Resilient Chickpea Innovation Lab, the first category includes increasing the genetic diversity of characterized wild chickpea relatives and related microbial symbiotes in publicly accessible germplasm banks and the second includes creation of well characterized segregating wild x cultivated chickpea populations and advancing selected symbiotes for commercial use. In both cases, success in the foundational areas was required before progress could be made against the downstream objectives and in both cases outstanding progress against the foundational objectives was made, but less consistent progress has been made against the more forward-looking goals. Achieving the foundational goals alone were major successes for each IL and should be recognized as such. Consistent with USAID expectations, both ILs attempted to address gender issues by recruiting women among their long-term trainees. While this is important, it is a small component of addressing gender roles and opportunities overall, particularly in agriculture and in Asian and African settings. The ILs had mixed effectiveness in providing degree training for women from target countries. Success in this very broad area will require significant time and investment and likely would require changes in attitudes and practices on a scale that extend beyond the manageable interest of any Innovation Lab. It would be advantageous for programs like these if USAID (perhaps with other major funders) established clearer guidelines and procedures under which the gender-related efforts within research and development programs could operate. Such guidelines could provide a framework to allow progress and challenges to be evaluated consistently and in a broader context. ILs will likely be more able to support USAID’s goals for female inclusion if they are given well-specified and consistent objectives around which they can make management decisions. The USAID crosscutting theme of nutrition is addressed in both of these ILs as they both seek to contribute to increased availability of important food crops in regions with high rates of nutritional insecurity. As described in CRC-RP5, the CRCIL made one attempt to address directly chickpea nutritional composition but then moved in other directions. 29 One of the ILs evaluated had an individual in a project management role (AWGIL) while the other did not. Based on what the EET learned during this evaluation as well as our own experiences, we believe that having an individual formally assigned such a role in consortium￾model programs (as contrasted with management entity programs) is a positive, but that the details of what this role should entail: (i) will likely differ from those of a “project management professional” (PMP) that would be familiar to industry and perhaps warranted for a management entity program, and (ii) are best established on a case-by-case basis. Responsibilities that should be considered for such a role and are viewed by the EET as well as multiple interviewees as being most helpful to the Director and PIs include: (i) setting up regular meetings and developing meeting agendas, (ii) following up meetings by capturing and tracking action items, (iii) managing logistics like travel and budgets, (iv) assisting the Director with report writing and communication. Based on perspectives shared during interviews regarding the value that a Project Coordinator brought to the AWGIL and regarding the administrative management of the CRCIL, as well as the experience of some members of the EET, the EET recommends that this role be filled full￾time by an individual with appropriate administrative, communications and project management experience, as opposed to being a part-time assignment for a researcher. Challenges in the area of data management and software systems go beyond the scope of any single program like either of these two ILs. Based on this evaluation the EET believes that, for many reasons, it would be optimal wherever possible to utilize systems in use by CGIAR member organizations. The major downside of this approach as seen for the AWGIL is that it leaves management of this issue, which is key to program success, outside the control of the IL. Success here will likely require some coordinated effort by multiple funders perhaps above the level of individual CGIAR organizations like CIMMYT or ICRISAT. Another alternative might be for USAID to support establishment of a university-based data sciences resource center to be shared by ILs. Expectations for record keeping and for getting feedback on short-term training do not appear to be well established and this was associated with a lack of data or meaningful feedback for any short-term training that was not a part of the longer-term student efforts. USAID would benefit from establishing expectations for outcomes assessment in short-term trainings. The two labs reviewed here have made significant advances in accordance with the initial proposals. Achieving the development impacts on smallholder farmers that USAID seeks will require considerably more time and continued efforts along the lines that each IL has been pursuing. The EET recommends that future activities emphasize: 30 1. Data management and software systems to ensure optimal use of the resources developed under the ILs. 2. Increased efforts to engage partners from South Asia in degree training and in the use of GEBV modeling for GS in order to achieve lasting impact in wheat. 3. Efforts to leverage new chickpea resources developed over the last five years to move from pre-breeding to breeding activities. 4. Establishment of mechanisms for systematic assessment of learning outcomes from training efforts, especially short-term training activities. 5. Careful consideration of the optimal distribution of funding for administrative personnel and scientific personnel if a program attempts to move chickpea activities from pre￾breeding to breeding. In general, it would appear that interruption in the support to these ILs might undermine the effectiveness and impact of the investments that have already been made by diminishing the capacity to consolidate knowledge and breaking momentum in moving research findings to application. 31 Appendices Scope of work Evaluation Purpose The purpose of this external evaluation is to provide empirical evidence that supports learning and continuous improvement of BFS’s work. The evaluation will assess progress toward outputs and outcomes, the quality of the research program and its outputs, what is and is not working well in implementation, including the effectiveness and contributions of the consortium management model, and will provide information and recommendations that BFS can use to inform the development of current programming and the design of future programs to support productive and efficient research-for-development partnerships with developing￾country scientists, including an understanding of how consortium model ILs formed and managed their partnerships in pursuit of their program goals. Audience and Intended Uses These results are to be used by USAID’s Research Division, Office of Agricultural Research & Policy in the Bureau for Food Security (USAID/BFS/ARP-R) to establish whether continued funding of the current awards are warranted and to inform design, management and implementation of current and future research programs. Evaluation Questions Research Program 1. To what extent did each IL generate robust and quality research outputs using disciplinary-appropriate metrics? Are the outputs relevant for a research-for￾development project (e.g., did they generate new breeding lines of relevance for the relevant countries)? Were the projects undertaken using the right tools/technologies and were they well executed? 2. What research-related challenges did each IL face during research implementation? In what ways were these challenges addressed? 3. Given the dual benefit mandate of Title XII authorized programs, such as these ILs, is there a sufficient balance between research efforts directed towards priorities of the domestic stakeholder community and the research priorities required to advance global food security goals, particularly in FTF countries? How has the project contributed to domestic US stakeholder priorities? 4. To what extent has each IL met its academic training and technical capacity strengthening targets? What improvements, if any, are needed in the ways: a) The program identifies and addresses academic and technical capacity needs? 32 b) Academic and technical capacity strengthening activities are implemented? c) People are targeted and selected for training? Program Structure & Management 5. How effectively has the Program Director managed the research and training activities among the awardees and stakeholders in the U.S. and partner countries? In what ways did the Program Director support coordination among the individual projects to ensure complementarity of effort and contribute to the IL’s objectives? How was communication and engagement handled? What were the pitfalls and challenges in coordinating among the different partners and among the different projects to better achieve outputs and outcomes? 6. If these IL’s have external advisors or an external panel, who served as advisors or on the panel? Were any of these persons from a country where the project worked? What role did the advisors or group play in advising or otherwise supporting the Innovation Lab? How did this contribute to program effectiveness and project implementation? 7. How did these consortium-style programs address, as relevant, cross-cutting issues of gender and nutrition? Did the programs have sufficient expertise, staff and resources to adequately address the relevant cross-cutting issues in project implementation? Partnership & Leveraging Opportunities 8. Did any of the partners from these project teams undertake or pursue other complementary funding opportunities or grant applications together (e.g., NSF BREAD? Others?) If so, were any successfully funded and leveraged existing funding? 9. What engagement, if any, did each IL have with private-sector partners? How effectively, and in what ways, did this involvement advance the project’s research and/or development goals? 10. As applicable, are there plans by the developing country partners to leverage capital investments by the Innovation Lab (equipment, facilities, etc.) for existing and future research activities? 11. What are the developing-country partners’ plans to further enhance, develop or disseminate research outputs produced by the Innovation Lab? Evaluation Plan A. Evaluation Design 1. Ativity/Mechanism Description a. Introduction In compliance with the USAID Evaluation Policy (2011) and the Scope of Work for the Combined External Evaluation of Feed the Future Innovation Lab for Applied Wheat Genomics (2013-2018) (AWG IL) and the Feed the Future Innovation Lab for Climate Resilient Chickpea (2014-2018) (CRC IL) this document: 33 1. Presents an overview of the AWG IL and CRC IL including: current organization of each IL and their leadership, goals, objectives, and activities to date; 2. Proposes a set of key indicators and data sources necessary to complete the evaluation; 3. Identifies information already available through project documents ; 4. Identifies gaps in information necessary to evaluate the programs including questions to be used in evaluation and additional sources of data necessary to address the indicators. 5. Outlines a design for the collection of additional data and a plan for analysis. The Scope of Work (SOW) for this evaluation instructed the review team to evaluate: (a) the research program of both ILs, (b) program structure and management, and (c) opportunities for partnership and leveraging. The SOW has identified evaluation questions for each of these three evaluation topics (a-c) which inform this evaluation plan and provide guidance to the External Evaluation Team (EET). The evaluation team will produce an evaluation report that will be used by USAID/BFS/ARP-R as a mechanism to provide empirical evidence that supports learning and continuous improvement of BFS’s work. The evaluation will assess progress toward outputs and outcomes, the quality of the research program and its outputs, what is and is not working well in implementation, and will provide information and recommendations that BFS can use to inform the development of current programming and the design of future programs to support productive and efficient research-for-development partnerships with developing-country scientists (SOW 2017). b. The Programs i. AWG-IL The Applied Wheat Genomics Innovation Lab was established in October 2013 with a cooperative agreement (AID-OAA-A-13-00051) of $4,990,218 for a period of 5 years. AWG-IL aims to develop heat tolerant, high yielding, and farmer-accepted varieties for South Asia, while simultaneously increasing the research for development capacity of the global wheat improvement system through application of cutting-edge genomics in applied wheat improvement. Specifically, the AWG-IL focuses on the challenges of wheat production and climate change in South Asia, taking a logical but innovative approach in applying genomics, physiology, and international phenotyping networks to maximize the value of wheat varieties in South Asia. (Applied Wheat Genomics Innovation Lab. September 2014. Poland, J. (ed). First Annual Report. Applied Wheat Genomics Innovation Lab, Kansas State University. Manhattan, KS. 30 pp). The AWG-IL is organized as a consortium led by Prof. Jesse Poland at Kansas State University and including the International Maize and Wheat Improvement Center (CIMMYT) in Mexico, The Borlaug Institute for South Asia (BISA) in India, The Indian Council for Agricultural Research (ICAR), The Wheat Research Center of the Ayub Agricultural Research Institute (AARI) in Pakistan, and Cornell University. The AWG-IL director is supported by a Project Manager, Haley Ahlers, and group of “Lab Members” from Cornell University, CIMMYT and BISA. The AWG-IL does not have an external advisory board. 34 Important considerations for the evaluation of the AWG-IL program include: a) Intention to enhance the value of the CIMMYT breeding pipeline via use of genomic selection (GS) with the specific target of increasing yield potential under extreme heat b) Intention to achieve earlier and larger distributions of candidate wheat varieties from CIMMYT-Mexico to India and Pakistan, where phenotyping and local breeding expertise can benefit from Genomic Estimated Breeding Values (GEBVs) to improve the performance of wheat in national trials, seed systems, and farmers’ fields. c) Intention to generate the largest public resource of elite candidate wheat varieties with well‐characterized phenotype and genotype information, along with seed and DNA in wheat history. d) Objectives of raising capacity for using cutting edge methods to accelerate breeding programs in South Asia e) Intention to work with the national agricultural research programs and private seed companies to promote early injection of larger numbers of candidate lines. Specific objectives of the AWG-IL against which outcomes may be compared include: g) Early and expanded evaluation in India and Pakistan of four successive annual cohorts of CIMMYT candidate wheat varieties. h) Expanded field phenotyping of candidate wheat varieties for heat tolerance in South Asia. i) Genotyping to ensure that CIMMYT and NARS wheat breeders have access to detailed whole‐genome characterization of candidate wheat varieties. j) Genomic selection of improved candidate wheat varieties. k) Yield and heat tolerance are improved by application of Genomic Estimated Breeding Values to selection of candidate wheat varieties in NARS testing and emerging from CIMMYT’s main breeding stream. l) Capacity development for heat stress breeding and application of genomic selection methodology. These specific objectives contribute to the overall development objective of delivering improved heat tolerant varieties with superior yield potential for improved food security and increased farmer income. ii. CRC-IL The Climate Resilient Chickpea Innovation Lab was established in January 2014 with a cooperative agreement (AID-OAA-A-14-00008) of $5,999,952 for a program running through December 31, 2018. Chickpea (Cicer arietinum) is the world’s second most important pulse legume and, like most cultivated legumes has narrow genetic and phenotypic diversity, making it challenging to breed for climate resilience or other high value traits. The CRC-IL aims at: characterizing wild Cicer species from a representative range of environments to introduce wild diversity into phenology-normalized backgrounds for trait assessment and breeding; characterizing the material by systematic phenotyping; developing a digital information network that explicitly identifies and quantifies the contributions of agronomically useful alleles; and developing improved chickpea varieties using an international consortium of 35 chickpea breeders. This program also has a goal of collecting and characterizing a geographically diverse set of Mesorhizobium strains and making them available for further research or commercialization. The CRC-IL is organized as a consortium led by Prof. Douglas Cook at the University of California, Davis. CRC-IL partners with many institutions including: Agginnovations (USA), Florida International University (USA), University of Southern California (USA), Addis Ababa University (Ethiopia), Aegean Agricultural Research Institute (Turkey), Assam Agricultural University (India), Banaras Hindu University (India), Dicle University (Turkey), EGE University (Turkey), Ethiopian Institute of Agricultural Research (Ethiopia), Grains Research Development Corporation (Australia), Harran University (Turkey), International Center for Agricultural Research in the Dry Areas (ICARDA), International Crops Research Institute for the Semi-Arid Tropics (ICRISAT), Legume Federation, and University of Saskatchewan (Canada). The CRC-IL has a small external advisory board. The Director absorbs the project management duties with no specific staff support beyond a webmaster. Specific objectives of the project are to: g) Characterize a comprehensive collection of wild species focused on C. reticulatum, the wild progenitor of cultivated chickpea. h) Create reverse-introgression and advanced backcross introgression lines to (a) remove phenological barriers that otherwise impede the use of wild germplasm in breeding, (b) establish a resource for association mapping of climate-resilience traits, and (c) initiate breeding with superior wild alleles. i) Phenotype reverse-introgressed and advanced backcross introgression lines for a range of high-priority traits related to developing high-yielding, climate-resilient chickpea. j) Develop a predictive network of genotype-phenotype associations that identifies genes and genome regions from wild species that improve chickpea’s yield resilience to climatic extremes. k) Provide data management and public-facing bioinformatics. l) Train and educate a gender-diverse group of young scientists from target countries. 2. Theory of Change and Program Logic a. AWG-IL Theory of Change and Program Logic The specific objectives of the AWG-IL reflect a logic model that links to program activities to the intended outcome of delivering improved heat tolerant varieties with superior yield potential for improved food security and increased farmer income. The logic model emphasizes the development of a system for tapping the genetic resources of the CIMMYT wheat program and up-to-date genomic analysis to create capacities and systems for more rapid development of wheat varieties that are high yielding and heat tolerant in South Asian settings. The model emphasizes that such systems will provide smallholder farmers, and especially women, with seed varieties that can improve their livelihoods. Gender is therefore a cross cutting issue that needs to be addressed at all stages of the process in order to assure impact from the intervention. The approach and methods used in the AWG-IL program are in use in other 36 organizations and with other crops and have been demonstrated to work. The scale-up at CIMMYT and connections with CIMMYT's downstream partners in Asia are aspects unique to this program. The Theory of Change, described graphically in Figure 1, is motivated by the observations that growth in wheat yields in South Asia has been below the pace needed to meet growing demand and that climate change models suggest increasing heat stress that threatens growth in yields. The challenge of developing heat tolerant wheat varieties that can promote renewed growth in smallholder yields can be addressed by mobilizing the combined genetic resources found in CIMMYT and National Research Institutes in South Asia with advanced genomic selection methods. This approach will accelerate the release of candidate varieties into national systems for phenotyping, leading to faster emergence of improved candidate wheat varieties in NARS testing systems. By applying advanced genomic selection and GEBV with partners in South Asia, the program will simultaneously accelerate the release of improved varieties and create capacity in NARS for effectively using genomic selection for continued development of new varieties. Once varieties are developed existing systems of multiplication and distribution are to be used to provide access to smallholder farmers. As varieties that are high yielding under heat stress are identified, participatory breeding, including women, will increase the likelihood that smallholders adopt the varieties. (Immediate Output) (Immediate Outcome) (Intermediate Outcome) (Ultimate Outcome) Figure 1: AWG-IL Theory of Change 37 A logic model is a broadly accepted approach to linking inputs and activities to outputs to outcomes to impacts. Inputs are the financial, scientific and human resources necessary to implement the proposed research. Activities are broadly conceived as mechanisms that link objectives to actions. Outputs are immediate products of project activities while outcomes represent the effect that outputs have on the broader context or environment at which the activities are aimed or embedded. Impacts are the project results that, once diffused, affect broad development goals, such as poverty reduction. The logic model for the AWG-IL presented in Figure 2 starts with financial, intellectual, and genetic inputs that are used to undertake research and capacity building activities to produce a flow of outputs in the form of candidate varieties and more effective systems for selection in the NARS. These outputs are expected to yield initial outcomes in greater availability of improved varieties that are high yielding under heat stress and acceptable to smallholder farmers in South Asia. Subsequently, the project outcomes are to include improved farmer livelihoods and regional food security. This ultimate outcome depends on activities in seed multiplication and distribution as well as extension that are beyond the scope of the project. It is unlikely that the AWG-IL evaluation will discern ultimate or even intermediate outcomes as AWG-IL was established too recently for these outcomes to emerge. Nevertheless, the evaluation will make every effort to understand whether and to what extent AWG-IL is on track to accomplish these longer-term outcomes. Figure 2. AWG-IL Logic Model 38 b. CRC-IL Theory of Change and Program Logic Chickpea (cider arietinum) is of critical importance to food security in many low-income countries in Africa and South Asia. Narrow genetic and phenotypic diversity in cultivated chickpea significantly limits the ability of scientists to adapt the crop to diverse and changing environmental settings. Effective use of wild germplasm in chickpea improvement will require new and systematic surveys of genotypes from natural environments, identification of adaptive alleles to environmental extremes, and incorporation of the diversity of wild alleles into phenology-normalized backgrounds for trait analysis and breeding. The CRC-IL aims to apply modern trait selection methods with a focus on generating breeding stocks with superior abiotic stress tolerance and nutrient density to ultimately benefit smallholder farmers and food consumers in Africa, other low-income regions of the world, and globally. The process by which outcomes are to be obtained is represented in the theory of change in Figure 3. The initial activity is to develop and characterize a comprehensive collection of wild species from a range of environments. From that sample, two purpose-driven populations that are synergistic for trait discovery and breeding are to be created: One population is developed through introgression of selected alleles for flowering time and plant architecture from cultivated genotypes into a range of wild species genomic diversity (these are called “reverse introgression” lines; revILs) and the other through introgression of genomic segments from wild species into cultivated genotypes (these are called advanced backcross introgression lines; ABLs). The revILs can then be phenotyped by an international consortium of breeders for a range of high priority traits, while scientists at UC Davis phenotype the ABI lines. Phenotyping will then allow development of a predictive network of genotype-phenotype associations that is the basis for the output of this effort: a resource that can provide genotype-phenotype (Output) (Immediate Outcome) (Intermediate Outcome) (Ultimate Outcome) Figure 3. CRC-IL Theory of Change 39 associations relevant to breeders globally. The primary outcome of this IL is the establishment of a properly implemented, well-utilized collection of wild germplasm that could provide crop phenotypes of importance in the developing world (e.g. heat tolerance, increased nutrient density) and add value to production from developed countries including the United States. That bioinformatic resource (rather than a physical seed bank) plus trained scientists constitute the basis for expanded chickpea breeding around the world and ultimately supports desired outcomes of a greater range of chickpea varieties available and enhanced productivity and food security among chickpea growers and consumers. The logic model for the CRC-IL presented in Figure 4 starts with financial, intellectual, and genetic inputs that are used to undertake research and capacity building activities. The initial activities involve creation of lines, genotyping and phenotyping to achieve the output of a predictive network of genotype-phenotype associations as a resource for breeders and others around the world. This output is expected to yield the initial outcome of greater capacity for chickpea breeding globally due to availability of this bioinformatics resource and the training of scientists. This ultimate outcome of increased productivity and food security depends on activities beyond the scope of the project. It is unlikely that the CRC-IL evaluation will discern ultimate or even intermediate outcomes as CRC-IL was established too recently for these outcomes to emerge. Nevertheless, the evaluation will make every effort to understand whether the CRC-IL is on track to accomplish these longer-term outcomes. Figure 4. CRC-IL Logic Model 3. Evaluation Research Questions The same set of questions applies to the evaluations of both the AWG-IL and the CRC-IL. The SOW for this evaluation identifies three areas for emphasis: Research Program; Program Structure and Management; and Partnership and Leveraging Opportunities. 1) Research Program 40 For each IL, the evaluation will pose the following questions concerning research activities: 1. To what extent did each IL generate robust and quality research outputs using disciplinary-appropriate metrics? Are the outputs relevant for a research-for￾development project (e.g., did they generate new breeding lines of relevance for the relevant countries)? Were the projects undertaken using the right tools/technologies and were they well executed? 2. What research-related challenges did each IL face during research implementation? In what ways were these challenges addressed? 3. Given the dual benefit mandate of Title XII authorized programs, such as these ILs, is there a sufficient balance between research efforts directed towards priorities of the domestic stakeholder community and the research priorities required to advance global food security goals, particularly in FTF countries? How has the project contributed to domestic US stakeholder priorities? 4. To what extent has each IL met its academic training and technical capacity strengthening targets? What improvements, if any, are needed in the ways: b. the program identifies and addresses academic and technical capacity needs? c. academic and technical capacity strengthening activities are implemented? d. people are targeted and selected for training? 2) Program Structure & Management Concerning program structure and management the evaluation will ask: 5. How effectively has the Program Director managed the research and training activities among the awardees and stakeholders in the U.S. and partner countries? In what ways did the Program Director support coordination among the individual projects to ensure complementarity of effort and contribute to the IL’s objectives? How was communication and engagement handled? What were the pitfalls and challenges in coordinating among the different partners and among the different projects to better achieve outputs and outcomes? 6. If these IL’s have external advisors or an external panel, who served as advisors or on the panel? Were any of these persons from a country where the project worked? What role did the advisors or group play in advising or otherwise supporting the Innovation Lab? what evidence is there as to whether the presence or absence of an external advisory panel or project manager affected the program implementation or effectiveness? 7. How did these consortium-style programs address, as relevant, cross-cutting issues of gender and nutrition? Did the programs have sufficient expertise, staff and resources to adequately address the relevant cross-cutting issues in project implementation? 3) Partnership & Leveraging Opportunities Concerning partnerships and leveraging the EET will ask: 8. Did any of the partners from these project teams undertake or pursue other complementary funding opportunities or grant applications together (e.g., NSF BREAD? 41 Others?) If so, were any successfully funded and leveraged existing funding? 9. What engagement, if any, did each IL have with private-sector partners? How effectively, and in what ways, did this involvement advance the project’s research and/or development goals? 10.As applicable, are there plans by the developing country partners to leverage capital investments by the Innovation Lab (equipment, facilities, etc.) for existing and future research activities? 11.What are the developing-country partners’ plans to further enhance, develop or disseminate research outputs produced by the Innovation Lab? 4. Methodology for Qualitative and Quantitative Data Collection The scope of this review covers the three main areas of research, management, and leveraging for the AWG-IL and the CRC-IL. Each project review will use a similar methodology, guided by the same set of questions, indicated above. As each IL addresses distinct research questions and objectives and because the collaborating partner countries and institutions vary across the ILs, the evaluation will consider the unique activities, outputs and near-term outcomes of each IL. All interviewees and stakeholders will be asked to identify program level accomplishments, outcomes and future expectations. Tables 1 and 2 present the evaluation questions linked to indicators and proposed sources of data for the AWG-IL and the CRC-IL respectively. The evaluation will draw from program documents, interviews with key informants, web-based surveys, and other outputs. 42 Table 1. Linking Evaluation Questions, Knowledge, Indicator and Source of Data: AWG-IL Knowledge/Outcome Illustrative Indicators Source of Data Research Program RP 1. How effective has the annual transfer of lines from CIMMYT to India/Pakistan (1000) and from NARS to CIMMYT (100-200) been? Have numerical goals been met? Are materials appropriately annotated? Whether materials and information required for initial project execution available. Differences by location. Transfer records, utility and completeness of annotations. Review of reports, online data available as described in management plan, interviews with PIs, CoPIs, Project Manager, collaborators. RP 2. Has annual seed increase, initial phenotyping and subsequent phenotyping under heat stress at the 4 India / Pakistan locations taken place as planned and have data been put into a database? Quantitative assessment of annual research progress. Multiple perspectives on data management. Quantitative records, evidence of database utility. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 3. Has genotyping at KSU taken place as planned and have the resulting data been archived appropriately for use? Quantitative assessment of annual research progress. Multiple perspectives on data management. Quantitative records, evidence of database utility. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 4. Status of determining GEBVs for yield under stress and development and use of prediction models. Utility? Whether quality of genotype and phenotype data is enabling desired modeling. Evidence of development and use of predictive models within program. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 5. Have best varieties identified as outcomes of Objectives 1-4 been transferred to multiple partners for further testing as planned and what are the results of this testing? Are the models generated from Obj 4 being used more broadly in the CIMMYT breeding program? Has any impact on cycle time yet seen from use of genomic selection? Initial impact of project beyond primary collaborators and on overall CIMMYT wheat breeding program. Specific evidence that lines selected using data generated from this project are impacting programs outside of India and Pakistan and that tools developed are being used broadly within CIMMYT. It is probably too early to see an impact on cycle time but early subjective signs may be available. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 6. Status of deploying electronic data capture tools (hardware and software)? Status of implementing bar coding for seed inventory and management? Tool availability. Existence and use of tools at one or more project locations. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 7. Has the AWG IL met its academic training and technical capacity strengthening targets? Are the Training of CIMMYT and Asian partners. Post doc status and Training types and numbers relative to targets. Students or post docs, Review reports; interviews with PIs. Student or post 43 appropriate type and number of people being targeted for the right kind of training? In what ways has the program identified and addressed academic and technical capacity needs? How could this be improved? progress. How potential degree recipients are targeted, how degree training is prioritized. Training statistics. PMIL/LIL Scholars Program. degrees received, current training, non-degree training, scholarships awarded. Evidence of training activity. Job placement. doc interviews. RP 8. What research-related challenges has the AWG Project faced during research design and implementation? In what ways have those challenges been addressed? What opportunities are there to improve research outputs moving forward? Obstacles or challenges faced Evidence of challenges from interviews, reports. Success of work-arounds. New directions, opportunities lost. Interviews of PIs, CoPIs, partner collaborators; project documents. RP 9. Given the dual benefit mandate of Title XII authorized programs, such as the AWG IL, is there a sufficient balance between research efforts directed towards priorities of the domestic wheat stakeholder community and the wheat research priorities required to advance global food security goals, particularly in FTF countries? Range of outputs and applicability to local as well as broader contexts Evidence of locally relevant outputs, uptake, awareness. Evidence of collaboration with international agriculture research centers, use of outputs. Evidence of value of outputs to domestic US research Interviews of PIs, CoPIs, partner collaborators and other stakeholders; annual and progress reports. Program Structure and Management PS 1. How effectively has the AWG IL Program Director managed research and training activities across partners and stakeholders? In what ways has the PI supported coordination and communication across partners to ensure complementarity of effort and contribution to AWG’s objectives? In what ways could communications and coordination be improved as a way to better achieve outcomes? Project and program communication processes. Are all key partners in Mexico, India, Bangladesh, and Pakistan aware of what is being done at other sites; do they understand how their research trajectories are integrated; do they direct efforts to communicate about and undertake integrative research activity (across the value chain and thematically)? Evidence of communication and coordination effectiveness; problems; team perception Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. Decision-making processes Perceptions about inclusion; perceptions about team approach, sharing of priorities, results, findings. Perceptions about whether research priorities and funding align with integration across themes Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. Reporting requirements & Timeliness, completeness and Documents; review reports; 44 associated communication distribution of reporting interviews with PIs, CoPIs, collaborating researchers and stakeholders. USAID interviews. Outreach and engagement activity Evidence of potential partners not included; problems with external perception of program Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. Leveraging and Partnership IC 1. In what ways, if any, did the location-based efforts (Mexico, US, India, Bangladesh and Pakistan) complement other wheat research activities in these regions? What are the lessons learned that can improve the complementarity of future USAID research and development investments in wheat breeding to other ongoing programs in the regions? In what ways can USAID programs in wheat breeding more effectively engage developing country partners in research? Local bridging and interaction between location-based projects and other regional efforts; local partnership existence, activities and outcomes; evidence of cross￾theme awareness and integration in local areas; opportunities for bridging and furthering complementarity locally. Quantity of partnerships by type and function; length of partnership; partnerships gained and lost; perceptions about partnership quality, roles, and effectiveness; future potential for growth and new directions; needs and missed opportunities; evidence of integrating project outputs by partners in local areas. Potential partners not linked. Interviews with PIs, CoPIs, Collaborators, Partners, USAID. Other organizations as identified during the interviews or document review. IC 2. What types of changes in implementation would more effectively achieve outcomes in the following components: research (i.e., design, implementation, communications, stakeholder involvement), capacity development (i.e., student recruitment and selection, content, location) and/or institutional collaboration? Knowledge about program directions relative to needs; understanding about complementary scientific directions; new food security threats or concerns; recognition of lack of attention paid by other research efforts; gaps to fill Gaps, barriers, opportunities, challenges, strengths and weaknesses of projects; opportunity to leverage capacity of partners toward new directions; evidence of cross-theme complementarities addressed and not addressed. Interviews with PIs, CoPIs, Collaborators, Partners, USAID. Other organizations as identified during the interviews or document review. 45 Table 2. Linking Evaluation Questions, Knowledge, Indicator and Source of Data: CRC-IL Knowledge/Outcome Illustrative Indicators Source of Data Research Program RP 1. Establishing collections. a. Has the intended sequencing of C.reticulatum accessions been done? b. Has collected germplasm been increased / preserved as planned? c. Have the desired Rhizobium strains been collected? Whether materials and information required for project execution are available. Transfer records, utility and completeness of annotations. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 2. Environmental characterization. For all accessions, were climatic, soil and other environmental data obtained and archived as planned? Data availability. Quantitative records, evidence of database utility. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 3. Genotyping. a. Were all plant genotypes obtained as planned? (Phase I – deep sequencing of 250 indiv from 20 wild pops to guide parent selection; phase II - <1X coverage of <1000 individuals from pops then created to look at allele diversity.) b. Was planned subset of Rhizobium isolates (480) genotyped / sequenced? c. What was learned about GxE associations for C.reticulatum from this work? Sequencing –data availability and accessability. GxE associations – Whether associations found; qualitative assessment of how next phases of program enabled. Sequencing – Records and evidence of database utility. GxE associations - Evidence of development and use within program. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 4. Population creation. a. Were reVIL’s (reverse introgressed lines) for flowering time and apical dominance traits generated as planned (5 cultivated x 20 wild accessions, w/ 50 indiv per pop)? Did this produce materials that breeders could use for subsequent pop development? b. Were ABI (advanced backcross introgression) lines generated in both Indian and Ethiopian cultivars as planned? Were F2’s genotyped as planned? Was desired representation of chromosomal segments achieved? c. Progress with use of original reVIL crosses for ABI devt? Availability of lines (reVIL and ABI) for subsequent use. Utility of lines for desired applications. Records and seed availability; evidence of subsequent use of lines as planned. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. 46 RP 5. Phenotyping. a. Status of reVIL’s phenotyping. Were desired number of lines evaluated? For desired traits? Across desired locations (California, India, Ethiopia)? b. Status of Rhizobium N fixation efficiency evaluations (multiple reVIL’s x multiple Rhiz isolates). c. Status of seed nutrient characterizations of 300 reVILs from Ethiopian parent? Has this facilitated selection of lines for breeding pops? d. Status of ABI line phentyping. Was desired characterization of agronomic and yield traits done? Data availability, accessibility and utility. Overall – availability and use of data and materials. Specific evidence that nutrient characterization data were used for breeding population development. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 6. Associations. a. Status of using reVIL’s to make genotype-phenotype associations for climate resilience traits. b. Have reVIL’s with desirable (wild) traits (fixed or not) been identified? c. Have potential undesirable cultivated x wild allele interactions been identified? Associations established and being used to guide breeding population development. Evidence from design of populations. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 7. Electronic data management. a. Are tools being used for data repository and management working as desired? Are data being deposited in timely manner? Utility / accessibility of data? Desired improvements? b. Status of creating public-facing system(s)? Data and tool availability and utility. Existence of tools and use across project locations. Review of reports, interviews with PIs, CoPIs, Project Manager, collaborators. RP 8. Has the CRC IL met its academic training and technical capacity strengthening targets? a. Has the desired diversity amongst project participants (including gender) been achieved? b. Is training targeting the right people? c. How has the program identified and addressed academic and technical capacity needs? How could this be improved? d. Status of planned short-term fellowship program. Training of partners. Status of any students or post doc’s involved with the program. Training statistics. Training types and numbers relative to targets. Students or post docs, degrees received, current training, non-degree training, scholarships awarded. Evidence of training activity. Job placement. Review reports; interviews with PIs. Student or post doc interviews. RP 9. What research-related challenges has the CRCIL Project faced during research design and implementation? In what ways have those challenges been addressed? What opportunities are there to improve research outputs moving forward? Obstacles or challenges faced Evidence of challenges from interviews, reports. Success of work-arounds. New directions, opportunities lost. Interviews of PIs, CoPIs, partner collaborators; project documents. RP 10. Given the dual benefit mandate of Title XII authorized Range of outputs and Evidence of locally relevant Interviews of PIs, CoPIs, 47 programs, such as the CRC IL, is there a sufficient balance between research efforts directed towards priorities of the domestic wheat stakeholder community and the wheat research priorities required to advance global food security goals, particularly in FTF countries? applicability to local as well as broader contexts outputs, uptake, awareness. Evidence of collaboration with international agriculture research centers, use of outputs. Evidence of value of outputs to domestic US research partner collaborators and other stakeholders; annual and progress reports. Program Structure and Management PS 1. How effectively has the CRC IL Program Director managed research and training activities across partners and stakeholders? In what ways has the PI supported coordination and communication across partners to ensure complementarity of effort and contribution to CRC’s objectives? In what ways could communications and coordination be improved as a way to better achieve outcomes? Project and program communication processes. Are all key partners in the US, India and Ethiopia aware of what is being done at other sites; do they understand how their research trajectories are integrated; do they direct efforts to communicate about and undertake integrative research activity (across the value chain and thematically)? Evidence of communication and coordination effectiveness; problems; team perception Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. Decision-making processes Perceptions about inclusion; perceptions about team approach, sharing of priorities, results, findings. Perceptions about whether research priorities and funding align with integration across themes Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. Reporting requirements & associated communication Timeliness, completeness and distribution of reporting Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. USAID interviews. Outreach and engagement activity Evidence of potential partners not included; problems with external perception of program Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. PS 2. What role has or could the External Advisory Panel play in contributing to program effectiveness and advice relating to broad research priorities in the Role of EAP; assistance with program review and direction; impact of advice on direction Evidence that meetings and other communications have taken place; broad program-wide efforts; Interviews with PIs, EAP, USAID. 48 wider wheat breeding field? awareness and understanding of program goals and project integration efforts/potential Partnership and Leveraging IC 1. In what ways, if any, did the location-based efforts (US, India, Ethiopia) complement other chickpea research activities in these regions? What are the lessons learned that can improve the complementarity of future USAID research and development investments in chickpea breeding to other ongoing programs in the regions? In what ways can USAID programs in chickpea breeding more effectively engage developing country partners in research? Local bridging and interaction between location-based projects and other regional efforts; local partnership existence, activities and outcomes; evidence of cross￾theme awareness and integration in local areas; opportunities for bridging and furthering complementarity locally. Quantity of partnerships by type and function; length of partnership; partnerships gained and lost; perceptions about partnership quality, roles, and effectiveness; future potential for growth and new directions; needs and missed opportunities; evidence of integrating project outputs by partners in local areas. Potential partners not linked. Interviews with PIs, CoPIs, Collaborators, Partners, EAP, USAID. Other organizations as identified during the interviews or document review. IC 2. What types of changes in implementation would more effectively achieve outcomes in the following components: research (i.e., design, implementation, communications, stakeholder involvement), capacity development (i.e., student recruitment and selection, content, location) and/or institutional collaboration? Knowledge about program directions relative to needs; understanding about complementary scientific directions; new food security threats or concerns; recognition of lack of attention paid by other research efforts; gaps to fill Gaps, barriers, opportunities, challenges, strengths and weaknesses of projects; opportunity to leverage capacity of partners toward new directions; evidence of cross-theme complementarities addressed and not addressed. Interviews with PIs, CoPIs, Collaborators, Partners, EAP, USAID. Other organizations as identified during the interviews or document review. IC 3. What leveraging opportunities have developed from the CRC IL and what advantage has been taken of these opportunities? Complementary funding opportunities or grant applications? Engagement with private sector partners? Use of research outputs or program capital investments beyond the IL? Knowledge about program directions relative to needs; understanding about complementary scientific directions; new food security threats or concerns; recognition of lack of attention paid by other research efforts; gaps to fill Gaps, barriers, opportunities, challenges, strengths and weaknesses of projects; opportunity to leverage capacity of partners toward new directions; evidence of cross-theme complementarities addressed and not addressed. Interviews with PIs, CoPIs, Collaborators, Partners, EAP, USAID. Other organizations as identified during the interviews or document review. 49 | P a g e a. Key Informant Interviews The EET will conduct interviews of the Directors, program manager or other staff, collaborating scientists and stakeholders for each innovation lab. The EET estimates that it will conduct ten to fifteen interviews for each innovation lab. Additionally, the EET will conduct interviews of students, trainees, and collaborators as appropriate and members of the External Advisory Board if one exists. The EET will also contact the field USAID missions in the primary target countries. Interviews will be conducted by telephone and will typically include only one respondent at a time, an exception being a focus group interview with students and post docs in the CRC-IL. If an interview cannot be arranged in a timely manner or if supplemental information is needed before or after an interview, information will be collected by e-mail. The EET intends to send interviewees the questions to be asked in the interview prior to the each interview. These questions will cover the primary areas of query as well as specific details to be covered during the interview. The purpose of sending the questions in advance is to facilitate preparation for the interview to maximize efficiency of the interviews. In general, the EET will not expect a written response to these questions. The questions are identical to those used to structure the interviews and examples of questions are provided in the Section B4. All interviews will be conducted using formal interview protocols with main questions and probes. All interviewees will be assured of confidentiality. To the extent possible, interviews will be conducted by two EET members; one EET member will be designated as the primary note taker. That person will write either handwritten notes or enter the notes in a laptop computer. If the notes are handwritten, the EET member responsible will enter them into a laptop at a later date such that all notes will be preserved electronically. If there are subjects that cannot be interviewed due to logistical constraints, the email response to the list of questions will be used as a substitute for the interview. A written interview protocol including primary questions and follow-up probes will be used as a structured guide for the interviewer to follow and to ensure all questions are asked of all respondents. Example interview protocols are presented in Section B4. Interviewers will allow respondents to respond freely to questions. The interview protocol will be further revised based on the ongoing review of documents described above. Section E lists individuals the EET will select for interviews. b. Document Review and Secondary Sources The evaluation team is in the process of conducting an initial review of program documents and data from AWG-IL and CRC-IL, including:  program-level technical application  annual and semi-annual reports  annual work plans  program highlights, success stories, and project briefs  websites 50 | P a g e  posted genotypic and phenotypic data emerging from the activities, including: AWG-IL related data at: https://data.cimmyt.org/dataset.xhtml?persistentId=hdl:11529/10696 and at: https://data.cimmyt.org/dataset.xhtml?persistentId=hdl:11529/10695  other published materials These documents have been made available by the evaluation manager or are on either the IL websites or their partners’ websites. Based on this initial review the EET may request additional documentation or records referred to or identified in available records. The EET may also request additional contact information not contained in the IL directories. For example, the EET may request contact information for graduate students, trainees, or postdocs involved with the labs or for stakeholder or collaborator names and contact information. The aim of this desk review is to identify key knowledge gaps. The desk review of each lab will be conducted according to the project review template in Table 3. Summary knowledge gaps will be entered into the table in Section F which will help guide the remainder of the project. c. Bibliometric Analysis The AWG-IL and CRC-IL have produced a number of journal articles. As part of its assessment of the quality of science, the EET will collect data suggesting the quality and impact of these publications. Publications from each IL are documented in the annual reports provided to the EET. To assess quality and impact of these publications the EET will identify whether the publication is in an outlet that has an official impact factor, how that impact factor compares to the leading journals in the field, and evidence of downloads or citations (in google scholar) for that article. Because the labs have only been operational for three to four years, and given the time required for citations, we do not expect to find high citation rates, but we would expect evidence of downloads and some publication in outlets with impact factors. 51 | P a g e Table 3. Project Review Template Outline The purpose of the desk review is to help identify knowledge gaps, answer evaluation questions, and to inform the subsequent interviews. The following outline will help make comparisons and ensure that the review is consistent. The team members will complete this template for each lab. A secondary consultation among EET members will confirm and revise the template to ensure accuracy. When there is disagreement or gaps, the EET members will confer to discuss evidence and support before finalizing the template findings. The EET will make final decisions based on consensus.  Development impact potential. Contributions or potential contributions of the IL research activities to developing research outputs that could contribute immediate and intermediate outcomes. Reviewers should link findings to theory of change and logic model.  Quality of science. To what extent did the IL generate robust and quality research outputs using disciplinary￾appropriate metrics? What evidence is there of this? Are the outputs relevant for a research-for-development project (e.g., did they generate new breeding lines of relevance to FTF countries)? Please highlight and describe most important outputs. Were the projects undertaken using the right tools/technologies and were they well executed?  Challenges faced. Evaluators document these, where possible, during desk review. Interviews should build on these findings.  Impacts in the US. Is there any evidence of IL-related outputs benefiting US producers, consumers or research community?  Training accomplishments. Information needed on degrees awarded and current positions of graduates (partner country, other developing country, US). Non-degree training (type and number) will also be evaluated. Please comment on the balance and appropriateness of this training. 1. Appropriate mix of training topics and appropriate training method 2. Considerations of gender balance  Partnerships. Evaluators to describe existing partnerships and their quality, identify potential partnerships. Identify any sharing of information and spillovers from IL partners.  Project-specific questions for interviews. 5. Methodology for Quantitative and Qualitative Data Analysis Semi-structured interviews will be conducted following traditional interview methods in which interviewers aim to engage the interviewee in a substantive conversation about their research. In-depth conversations guided by interview instruments provide the opportunity for a flow of information that is not constrained by pre-determined response categories. Interviewees are both able to follow lines of inquiry unique to an individual, but also collect equivalent data across interviews. This approach allows exploration of subjects that surface during the interview and elicits information that provides nuance and insight for addressing the evaluation questions. 52 | P a g e Interviews with key informants will be captured in notes, which will be expanded at the end of each day of interviewing. Notes will be saved as text files and summarized at the end of the interview. Summaries should include specific insights or key findings from the interviews. Cross-interview findings can also be noted for later analysis. EET members will schedule meetings to review findings from each interview. Project write-ups will address the evaluation questions established for the evaluation, integrating key indicators from the desk review listed in Tables 1 and 2 with evidence from interviews. All individuals who attended the interviews will be asked to review the individual project write-ups, include additional comments and identify inconsistencies. When all project write-ups are complete, the team will develop an overall program assessment concerning the delivery of outputs and outcomes. Bibliometric analysis and materials from desk review will be integrated into the assessment. Conclusions will include accomplishments and shortcomings. Based on the evidence and considering project goals, the EET will develop a set of recommendations to assist USAID and the two innovation labs in the future. After considering the evidence of effectiveness of each of the Innovation Labs under review, the EET will attempt to synthesize and present any learning that may have applicability across the programs. 6. Methodological Limitations a. Publication Bias Evidence of scientific quality may not be well represented from the publication trail. An absence of any published output would be a sign of low scientific merit, but it is possible for the publication count to be inflated by multiple publications with little differentiation or impact. By the same token, output of great scientific merit may published only with considerable delay or may remain in the gray literature. Output from the bibliometric analysis will be treated with care and in the context of other evidence of quality. b. Sampling and Participant Selection Bias Given substantial differences in involvement of stakeholders and persons interacting with these labs, it is likely that we will miss important sources of information. Moreover, as we must rely on the labs themselves to provide lists of people to interview, other than those named in reports from the labs, there is a potential for bias. c. Interviewee Recall Bias Key informant interviewing will focus primarily on the interviewees’ opinions, actions and assessments of program activities as participants and collaborators since project inception in 2013. We do not anticipate problems with recall bias given that the project has only recently begun and is still in operation. The EET can identify inconsistencies through comparison across interviews. Questions of fact can be cross-checked against documentation of events and activities available in other project materials. 53 | P a g e d. Interviewee Response Bias Interview based data is subject to self-report bias. We have structured this evaluation in ways to reduce the likelihood of three types of systematic response bias: acquiescence bias (i.e., the tendency to agree), demand characteristics (i.e., modifications to responses because of being “studied”), and social desirability bias (i.e., ascribing favorable traits, even if this is untrue). Interview protocols ask open-ended questions and probes are constructed to inquire about multiple perspectives on an issue and similar questions are asked of multiple project members such that it will be possible to recognize issues of response bias. In analysis we will review responses to interview questions such that the responses of different stakeholder’s and different individuals from within stakeholder groups can be cross-checked for convergence or disagreement. Through triangulation of assessments and understandings of different individuals we can come to a set of conclusions that identify areas of consensus and areas of disagreement. While eliminating response bias is difficult, triangulation greatly reduces the concern. e. Interviewer Bias Several sources of interviewer bias are particularly important to be aware of during interviews. Inflection of the interviewer can bias responses, encouraging or discouraging particular lines of conversation. In international contexts, race and ethnicity of the interviewer can bias responses of interviewees such that interviewees may be more open with individuals who are more like them. Interviewees can also make the mistake of presenting leading questions or ‘putting words into people’s mouths’. Interpretation of data can become skewed toward one perspective or another although evidence does not merit it. Finally, prior involvement with a particular group or scientist can lead to biased interpretation of data. Although these forms of interviewer bias are difficult to manage, most of them can be addressed through training, interview experience and interview structure. For this evaluation, the EET has substantial interview experience which minimizes these concerns. Additionally, interviews will be team-based when possible, written materials will be shared among EET members and analysis will be vetted as a group. Additionally, to avoid any concerns about bias, the team is careful not to assign individuals to individuals have had a high level of prior collaboration. 7. Outcome Measures Outcomes measures related to research, excluding training and bibliographic data, are largely specific to each IL while those related to management and partnership are more uniform. Selected measures include:  Numbers of lines transferred, compared to target (AWG-IL) indicated in annual reports, will suggest the effectiveness of the program in reaching research targets.  Success generating phenotypic data and the quality of those data across locations in South Asia (AWG-IL and CRC-IL), indicated in annual reports, will progress in meeting research goals. 54 | P a g e  Evidence of performance of selected lines (AWG-IL) will imply success of IL in meeting research goals.  Evidence of effective use of tools at partner institutions (AWG-IL) may come in quantitative or qualitative forms and will suggest effectiveness of research program.  Growth in germplasm collections (CRC-IL) will indicate effectiveness of research program.  Quality of environmental characterization in accessions (CRC-IL) will indicate usefulness of research for domestic and international researchers.  Quantitative evidence of availability of reVIL and ABL populations for use within and outside of the program will indicate impact of research (CRC-IL).  Successful generation of quality genotype data will suggest both management effectiveness and research success.  Evidence of value of outputs to domestic (US) research will be suggested by qualitative impressions of stakeholders and through analysis of citations and presentations data.  Number of students in graduate degree training by gender, degree program and institution. This measure will indicate progress towards academic technical capacity building targets, gender sensitivity, and depth of institutional partnerships  Number of research publications in peer reviewed journals, impact factors, and citation counts. This will indicate the volume and quality of research output  Number and value of additional research grants. This will indicate quality and social value of research programs and success in leveraging.  Perceptions of effective communication and inclusion. This qualitative data will indicate quality of management.  Timeliness and completeness of reporting. This will indicate quality of management.  Qualitative assessment of ease of use of webpage will indicate effectiveness of communication.  Observed inclusion of diverse partners including universities, government agencies, private sector and others. This will indicate adequacy of management and program structure. 55 | P a g e 8. Evaluation Work Plan Activities Dates of Activity 1 st Month 2 nd Month 3 rd Month 1 2 3 4 1 2 3 4 1 2 3 4 TASK 1. Develop evaluation design and implementation plan Nov 15-Dec 15 Activity 1: Desk Review and initial interviews with lab directors and USAID managers Nov 15-Dec 15 X X X X Activity 2: Development of Evaluation Design Nov 15-Dec 15 X X X X Activity 3: Develop Schedule of Interviews Dec 1-Dec 15 X X TASK 2: Data Collection Activity 1: Send Surveys Dec 15-Dec 21 X X Activity 2: Conduct interviews Dec 15-Jan 7 X X X Activity 3: Collect publication data, bibliometrics Dec 15- Dec 31 X X TASK 3: Data Analysis Activity 1: Review Data Gaps Dec 15-Jan15 X X X X Activity 2:Organize Findings and Discuss to Develop Evaluation Output Jan 1-Jan 15 X X TASK 4: Report Writing Activity 1: Draft Initial Findings Jan 7-15 X Activity 2: Draft Report Jan 15-Feb 15 X X X X Activity 3: Submit Final Report March 15 B. Data Collection and Management Plan 1. Survey Training Evaluators are experienced in interview techniques. No special training is necessary to use the interview protocols. Interviews will be conducted in English. If a translator is necessary, the EET will arrange for one, verify the qualifications needed and provide any needed training to the translator. 2. Data Management and Security Evaluators are experienced with the confidentiality requirements needed for work with human subjects. All responses are considered confidential. If there is a need to reference a particular PI or co-PI, the EET will request permission from the individual. All notes and findings will be collated anonymously and no direct reference will be made to any individual respondent. Evaluators will not identify individual respondents either in reports or to USAID during debriefings. 3. Data Collection Approvals Data collection approvals will not be needed as this data collection is not for research, and the 56 | P a g e only information collected is that given by the subjects themselves with their agreement. Confidentiality of responses will be maintained, and no names or identification will be attached to responses. Interview notes will be coded with the general category of stakeholder and individual names kept separately to avoid potential for disclosure. 4. Data Collection Instruments The following section contains interview protocols as examples of the format and types of questions that will be asked during interviews. Interview protocols will be tailored to each type of group interviewed: (i) Program Director, (ii) Project Manager, (iii) PIs and Co-PIs and (iv) USAID Mission Staff, and (v) others. The need to interview other collaborators and stakeholders, including students and postdocs will be determined based on learnings from report reviews and initial interviews. This protocol will be revised based on information from the desk review or other sources. Responses to these questions will be collected and analyzed as presented in Section A. a. AWG-IL Protocol for Initial Interview of Program Director (Multiple interviews with the Program Director may be required, with the protocol(s) for subsequent interviews based on specific questions that arise during the evaluation) Introduction The purpose of this interview is to assess the perspectives of the Program Director for the Applied Wheat Genomics Innovation Lab. We are also conducting confidential interviews of PIs, Co-PIs, Collaborating researchers in the US and in partner countries, as well as other relevant stakeholders. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. Our aim is to better understand how research activities are proceeding, how the multiple AWG partners interact with each other and external groups and what factors might improve AWG’s ability to attain its objectives. I have several questions that will take about 60 minutes of your time. Do you have any questions before we begin? Questions 1. Looking at the program overall, since its beginning in 2013, how would you characterize research progress compared to initial expectations? Prompt: Can you point to some particular accomplishments that highlight this progress? (e.g. journal articles, deployment of new tools, advancement of new breeding lines, etc.). Prompt: Are there activities that are ahead of schedule? Is anything falling behind? Prompt: Can you give me some examples? What do you think are the main reasons why these AWG-IL activities are not progressing as planned? Or are ahead of schedule? 57 | P a g e Prompt: Have you had to develop workarounds to address the challenges? Is there anything you can think of that would help you to better accomplish the AWG project goals? Prompt: Looking ahead, are there research activities that were proposed that you may not be able to accomplish? 2. What would you highlight as the 1-2 most significant accomplishments to date? (eg. in terms of future applications for research in the US or partner research programs?) Prompt: How do you judge the quality of the various research activities within the program? Prompt: In what ways are the research findings being integrated/adopted by local partners? Prompt: What are the developing country partners’ plans for further enhancement, development or dissemination of the outputs produced by the Innovation Lab? 3. Are there or do you foresee any significant changes to the program due either to significant challenges you have encountered or due to important findings or discoveries from the research? Prompt: Can you give me some examples? Prompt: Do you see possibilities for new directions in a new research program? Explain? Prompt: Have there been any missed opportunities? Can you give me some examples? Prompt: For any identified changes, how could the current project be adjusted to meet any new objectives? Would you have to cut back on other activities? 4. In what ways does the AWG-IL program consider gender? Prompt: Can you give us some detail on any gender specific findings, outputs or outcomes of from the program? Prompt: What about participation of women in the project - how are women integrated in the various parts of project? 5. This program includes people from multiple countries and institutions. Describe the level of communication and coordination among AWG-IL collaborators? Prompt: To what extent does decision-making include perspectives from multiple partners or collaborators? Prompt: How well does the international research collaboration work? Are there ways you think that collaborative interaction could be improved? 58 | P a g e Prompt: Are there avenues of communication that seem to work better than others? Why? Prompt: To what extent to you feel like there is a team-based approach to the research? Prompt: On average, how often do you receive communications from the PI’s? What proportion of these communications is a request for information? What proportion is giving you information? Prompt: Please describe the process by which your project assembles its annual workplan and annual report. Is there a meeting of all scientists? Prompt: What has been the Project Manager’s role in facilitating cooperation. 6. Is it important for the AWG-IL program to establish partnerships with local organizations or agencies, universities, and/or NGOs? Why or why not? Prompt: If so, what types of connections have you established? What have you done with partner organizations that advance the strategic objectives of he program? Prompt: Do you think the partnerships work well? Are there partnerships that seem to work better than others? Why? Prompt: Have there been important contributions from these stakeholders or partners to your research? Could you give us some examples? 7. From your perspective, has there been sufficient interaction and learning across program partners and institutions? Can you point to any results or outcomes that have occurred due to such interactions? Prompt: How do you interact with others working on the AWG IL research program? Do meetings regularly occur, and how do they come about? 8. One of the objectives of the IL is to build local capacity for research. From your perspective is the program accomplishing this? Prompt: What training has been provided as part of the program? What training is planned but not yet completed? Is this training meeting objectives (yours and the trainees)? Are there unmet needs? Prompt: One program objective was to have 4-6 post docs. How has that gone? Are these researchers gaining experience that will allow them to function independently following completion of the program? What is the gender make-up of this group? Is this appropriate from your perspective? 9. Have you or any of the partners in this program undertaken or pursued together other complementary funding opportunities, grant applications or collaboration with other parties? If so, were any successfully funded and leveraged existing funding? 59 | P a g e Prompt: Please provide examples and/or, as appropriate, other places we can get this information. Prompt: What about private sector partners? Prompt: Are there plans by the developing country partners to leverage capital investments by the Innovation Lab (equipment, facilities, etc.) for existing and future research activities? Explain. 10. Thinking about the program as a whole, what is your assessment of its strengths and weaknesses? b. AWG-IL Protocol for Interview of a PI or Key Collaborator Introduction The purpose of this interview is to assess the perspectives of the Key Collaborators as part of the Applied Wheat Genomics Innovation Lab. We are also conducting confidential interviews of the Program Director, PIs, Co-PIs, Collaborating researchers in the US and in partner countries, as well as other relevant stakeholders. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. Our aim is to better understand the how research activities are proceeding, how the multiple AWG-IL partners interact with each other and external groups and what factors might improve AWG-IL’s ability to attain its objectives. I have several questions that will take about 60 minutes of your time. Do you have any questions before we begin? Questions 1. Could you first describe the research that you are doing within the AWG IL program? Prompt: Can you help me understand how your project addresses one or more of the strategic objectives of the AWGIL? 2. Since the beginning of the program in 2013, has the research in your area progressed as you expected? Prompt: Can you point to some particular accomplishments that highlight this progress? (e.g. journal articles, deployment of new tools, advancement of new breeding lines, etc.). Prompt: Are there activities that are ahead of schedule? Is anything falling behind? Prompt: Can you give me some examples? What do you think are the main reasons why these AWG activities are not progressing as planned? Or ahead of schedule? 60 | P a g e Prompt: Have you had to develop workarounds to address the challenges? Is there anything you can think of that would help you to better accomplish the AWG project goals? Prompt: Looking ahead, are there research activities that were proposed that you may not be able to accomplish? 3. What would you highlight as the 1-2 most significant accomplishments to date within the program overall? Within your part of the program? What is your outlook for future application / deployment within your organization of the materials and tools developed? Prompt: What do you feel are the main issues preventing (or inhibiting) even greater application / deployment? Prompt: In what ways are developments from the program being integrated / uptaken by your organization and by local partners? Prompt (FOR DEVELOPING COUNTRY PARTNERS): What are your plans for further enhancement, development or dissemination of the outputs produced by the Innovation Lab? 4. This program includes people from multiple countries and institutions. Is there sufficient communication and coordination among AWG collaborators? Prompt: To what extent does decision-making include perspectives from multiple partners or collaborators? Prompt: Do you have a principal contact? Who else to you communicate / coordinate with? Prompt: How well does the international research collaboration work? Are there ways you think that collaborative interaction could be improved? Prompt: To what extent to you feel like there is a team-based approach to the research? Prompt: On average, how often do you receive communications from the Program Director? What proportion of these communications is a request for information? What proportion is giving you information? Prompt: Please describe the process by which your project assembles its annual workplan and annual report. Is there a meeting of all scientists? Prompt: How would you describe the Project Manager’s role in facilitating cooperation. 5. The AWG-IL program operates across multiple countries and organizations. How well does the Program Director manage the program? Prompt: What is your experience with the management of the program? 61 | P a g e a. Professional/collegial interactions b. Fiscal interactions (promptness, clarity) c. Promptness and effectiveness of managing problems d. Quality of the solutions to problems e. Clarity of goals and objectives f. Degree to which planning is collaborative g. Degree to which data collection and analysis are supported h. Degree to which findings are appropriately disseminated Prompt: What could the Program Director do to more effectively meet the AWG IL’s objectives? Prompt: What strengths or synergies has the program encouraged, e.g., across locations and/or organizations to ensure complementarity of effort? 6. From your perspective, has there been sufficient interaction and learning across program partners and institutions? Can you point to any results or outcomes that have occurred due to such interactions? Prompt: How do you interact with other researchers who are part of the program? Do meetings regularly occur, and how do they come about? 7. One of the objectives of the IL is to build local capacity for research. From your perspective is the program accomplishing this? Prompt: Have you (or others at your location) received additional training as a result of the program? If so, please describe. If not, have you asked for such training? What was the resolution of your request? Prompt: One program objective was to have 4-6 post docs. How has that gone? Are these researchers gaining experience that will allow them to function independently following completion of the program? What is the gender make-up of this group? Is this appropriate from your perspective? Is gender a consideration? How has the gender balance been? 8. Where appropriate, ask about pursuit of any other complementary funding opportunities, grant applications or collaboration with other parties. 9. Thinking about the program as a whole, what is your assessment of the strengths and weaknesses of the AWG-IL program, as well as your specific part of the program? c. CRC-IL Protocol for Initial Interview of Program Director (Multiple interviews with the PD may be required, with the protocol(s) for subsequent interviews based on specific questions that arise during the evaluation.) Introduction 62 | P a g e The purpose of this interview is to assess the perspectives of the Program Director for the Climate Resilient Chickpea Innovation Lab. We are also conducting confidential interviews of PIs, Co-PIs, Collaborating researchers in the US and in partner countries, as well as other relevant stakeholders, including EAP members. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. Our aim is to better understand the how research activities are proceeding, how the multiple CRC partners interact with each other and external groups and what factors might improve CRC’s ability to attain its objectives. I have several questions that will take about 60 minutes of your time. Do you have any questions before we begin? Questions 1. Let’s start by talking about progress against the research program objectives and how things are progressing relative to plans and expectations. In all cases where appropriate, please comment on activities that are ahead or behind schedule, any workarounds you’ve had to employ and any activities that now may not be accomplished. Prompt: Regarding establishment of collections, have the intended chickpea and Rhizobium materials been collected? Has the intended sequencing of C.reticulatum accessions been done? Prompt: Regarding environmental characterizations, were climatic, soil and other data obtained and archived as planned? Prompt: Regarding genotyping, have all plant genotypes (Phases I and II) been obtained as planned? What about Rhizobium isolates? How would you summarize what has been learned to-date about genotype by environment associations from this work? Prompt: Were the reVIL’s generated as planned? Have breeders used these lines for subsequent population development? Were the ABI lines generated as planned? Were F2’s genotyped as planned? What representation of chromosomal segments was achieved? What progress has been made using the original reVIL crosses for ABI devt? Prompt: For phenotyping, what is the status of reVIL’s phenotyping? Were desired number of lines evaluated, for desired traits, across desired locations? What is the status of Rhizobium N fixation efficiency evaluations? What is the status of reVIL seed nutrient characterization? Has this facilitated selection of lines for breeding pops? What is the status of ABI line phentyping? Prompt: What is the status of using reVIL’s to make genotype-phenotype associations for climate resilience traits? Please comment on any identified desirable or undesirable traits. 63 | P a g e Prompt: Are data management tools working as desired? Are data being deposited in timely manner? Are the data meeting accessibility and utility needs of users? What improvements do you still plan on making? Have the intended public-facing system(s) been created? 2. What would you highlight as the 1-2 most significant accomplishments to date? Future applications for research in the US? Indian and/or Ethiopian partner research programs? Prompt: How do you judge the quality of the various research activities within the program? Prompt: In what ways are the research findings being integrated/uptaken by local partners? Prompt: What are the developing country partners’ plans for further enhancement, development or dissemination of the outputs produced by the Innovation Lab? 3. Are there or do you foresee any significant changes to the program due either to significant challenges you have encountered or due to important findings or discoveries from the research? Prompt: Can you give me some examples? Prompt: Do you see possibilities for new directions in a new research program? Explain? Prompt: Have there been any missed opportunities? Can you give me some examples? Prompt: For any identified changes, how could the current project be adjusted to meet any new objectives? Would you have to cut back on other activities? 4. In what ways does the CRC program consider gender? Prompt: Can you give us some detail on any gender specific findings, outputs or outcomes of from the program? Prompt: What about participation of women in the project - how are women integrated in the various parts of project? 5. This program includes people from multiple countries and institutions. Is there sufficient communication and coordination among CRC collaborators? Prompt: To what extent does decision-making include perspectives from multiple partners or collaborators? Prompt: How well does the international research collaboration work? Are there ways you think that collaborative interaction could be improved? 64 | P a g e Prompt: Are there avenues of communication that seem to work better than others? Why? Prompt: To what extent to you feel like there is a team-based approach to the research? Prompt: On average, how often do you receive communications from the PI’s? What proportion of these communications is a request for information? What proportion is giving you information? Prompt: Please describe the process by which your project assembles its annual workplan and annual report. Is there a meeting of all scientists? 6. The CRC IL program has established partnerships with a range of local organizations or agencies, universities, and NGOs. Please describe their roles and importance to the program. Prompt: If so, what types of connections have you established? What have you done with partner organizations that advance the strategic objectives of he program? Prompt: Do you think the partnerships work well? Are there partnerships that seem to work better than others? Why? Prompt: Have there been important contributions from these stakeholders or partners to your research? Could you give us some examples? 7. From your perspective, has there been sufficient interaction and learning across program partners and institutions? Can you point to any results or outcomes that have occurred due to such interactions? Prompt: How do you interact with others working on the CRC IL research program? Do meetings regularly occur, and how do they come about? 8. One of the objectives of the IL is to build local capacity for research. From your perspective is the program accomplishing this? Prompt: What training has been provided as part of the program? What training is planned but not yet completed? Is this training meeting objectives (yours and the trainees)? Are there unmet needs? Prompt: One program objective was a short-term fellowship program. How has that gone? What level of independence will researchers who have been part of this program be prepared for following completion of the CRC IL program? What is the gender make-up of this trainee group? Is this appropriate from your perspective? 9. How does the EAP function? Is it providing you and the team with valuable input? Prompt: Please provide examples of useful input received and how this input improved the program. 65 | P a g e Prompt: Is there anything you would change about the make-up or operation of the EAP? 10. Have you or any of the partners in this program undertaken or pursued together other complementary funding opportunities, grant applications or collaboration with other parties? If so, were any successfully funded and leveraged existing funding? Prompt: Please provide examples and/or, as appropriate, other places we can get this information. Prompt: What about private sector partners? Prompt: Are there plans by the developing country partners to leverage capital investments by the Innovation Lab (equipment, facilities, etc.) for existing and future research activities? Explain. 11. Thinking about the program as a whole, what is your assessment of its strengths and weaknesses? d. CRC-IL Protocol for Interview of a PI or Key Collaborator Introduction The purpose of this interview is to assess the perspectives of key collaborators as part of the Climate Resilient Chickpea Innovation Lab. We are also conducting confidential interviews of the Program Director, PIs, Co-PIs, Collaborating researchers in the US and in partner countries, as well as other relevant stakeholders, including EAP members. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. Our aim is to better understand the how research activities are proceeding, how the multiple CRC partners interact with each other and external groups and what factors might improve CRC’s ability to attain its objectives. I have several questions that will take about 60 minutes of your time. Do you have any questions before we begin? Questions 1. Could you first describe the research that you are doing within the CRC IL program? Prompt: Can you help me understand how your project addresses one or more of the strategic objectives of the CRC IL? 2. Since the beginning of the program in 2013, has the research in your area progressed as you expected? Prompt: Can you point to some particular accomplishments that highlight this progress? (e.g. journal articles, deployment of new tools, advancement of new breeding lines, etc.). 66 | P a g e Prompt: Are there activities that are ahead of schedule? Is anything falling behind? Prompt: Can you give me some examples? What do you think are the main reasons why these AWG activities are not progressing as planned? Or ahead of schedule? Prompt: Have you had to develop workarounds to address the challenges? Is there anything you can think of that would help you to better accomplish the AWG project goals? Prompt: Looking ahead, are there research activities that were proposed that you may not be able to accomplish? 3. What would you highlight as the 1-2 most significant accomplishments to date within the program overall? Within your part of the program? What is your outlook for future application / deployment within your organization of the materials and tools developed? Prompt: What do you feel are the main issues preventing (or inhibiting) even greater application / deployment? Prompt: In what ways are developments from the program being integrated / uptaken by your organization and by local partners? Prompt (FOR DEVELOPING COUNTRY PARTNERS): What are your plans for further enhancement, development or dissemination of the outputs produced by the Innovation Lab? 4. This program includes people from multiple countries and institutions. Is there sufficient communication and coordination among CRC collaborators? Prompt: To what extent does decision-making include perspectives from multiple partners or collaborators? Prompt: Do you have a principal contact? Who else to you communicate / coordinate with? Prompt: How well does the international research collaboration work? Are there ways you think that collaborative interaction could be improved? Prompt: To what extent to you feel like there is a team-based approach to the research? Prompt: On average, how often do you receive communications from the Program Director? What proportion of these communications is a request for information? What proportion is giving you information? 67 | P a g e Prompt: Please describe the process by which your project assembles its annual workplan and annual report. Is there a meeting of all scientists? 5. The CRC IL program operates across multiple countries and organizations. How well does the Program Director manage the program? Prompt: What is your experience with the management of the program? a. Professional/collegial interactions b. Fiscal interactions (promptness, clarity) c. Promptness and effectiveness of managing problems d. Quality of the solutions to problems e. Clarity of goals and objectives f. Degree to which planning is collaborative g. Degree to which data collection and analysis are supported h. Degree to which findings are appropriately disseminated Prompt: What could the Program Director do to more effectively meet the CRC IL’s objectives? Prompt: What strengths or synergies has the program encouraged, e.g., across locations and/or organizations to ensure complementarity of effort? 6. From your perspective, has there been sufficient interaction and learning across program partners and institutions? Can you point to any results or outcomes that have occurred due to such interactions? Prompt: How do you interact with other researchers who are part of the program? Do meetings regularly occur, and how do they come about? 7. One of the objectives of the IL is to build local capacity for research. From your perspective is the program accomplishing this? Prompt: Have you (or others at your location) received additional training as a result of the program? If so, please describe. If not, have you asked for such training? What was the resolution of your request? Prompt: One program objective establishment of a short-term fellowship program. Have researchers from your location participated? How has that gone? Are these researchers gaining experience that will allow them to function independently following completion of the program? What is the gender make-up of this group? Is this appropriate from your perspective? Is gender a consideration? How has the gender balance been? 8. Where appropriate, ask about pursuit of any other complementary funding opportunities, grant applications or collaboration with other parties. 9. Thinking about the program as a whole, what is your assessment of the strengths and weaknesses of the CRC IL program, as well as your specific part of the program? 68 | P a g e e. CRC-IL Protocol for Focus Group Discussion with Students and Postdocs Introduction You are entering the last year of the USAID-funded Climate Resilient Chickpea Innovation Lab program. It is standard USAID practice to conduct a review of all funded programs at this stage and this interview is one part of that. The purpose of this interview is to obtain the perspectives of students and postdoctoral trainees regarding a number of aspects of the program. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. We are also conducting confidential interviews of the Program Director, PIs, key collaborating researchers in the US and in partner countries, and other relevant stakeholders. USAID has recruited three of us to conduct this program review: Introduce Jim, Jackie, Alex with couple-sentence bio’s. This review is being conducted through desk reviews of available written (or audio or video) materials as well as through interviews. Thus, the interviews are central to our review. Our objective today is to get your perspectives regarding a number of aspects of the program that are particularly relevant and important to you as trainees. Because we want to allow time for broad participation, we have limited the number of questions we will ask and have reserved 60 minutes of your time. Do you have any questions before we begin? Questions 1. Building research expertise in target countries through academic training, with a particular emphasis on training women, is a key objective of this program. Students and postdocs are at the center of this. Do you believe that objective is being met? Why (or why not)? Prompt: Give a brief background of yourself and describe the role(s) you have played in the program (get several inputs)? Prompt: Will the training you are receiving allow you to continue working in this area of research independently after the training or program ends? If not, what are the gaps and how could they be addressed? Prompt: From your perspectives, has communication across the program been managed effectively? Do you feel sufficiently tied in to the program overall? Prompt: If you had to identify one thing that the program could do better regarding training, what would it be and why? 2. A related objective of the IL is building local capacity for research. From your perspective is the program accomplishing this? Prompt: From your perspectives, will the research being done as part of the program be able to continue beyond whenever the program ends? Why (or why not)? 69 | P a g e Prompt: For those of you from Ethiopia, India, Pakistan or Turkey, are you planning to return to your home country following training? Do you plan to continue working in this research area? Why or why not? 3. As you think about the people who are receiving or who have received training as part of this program, do you feel it has targeted the right people? Why or why not? Prompt: Has the desired diversity amongst project participants (including gender) been achieved? Prompt: (If any gaps are identified) What is the biggest obstacle to achieving this objective? 4. Before we end this interview, is there anything else that any of you would like to share with us regarding learnings, challenges or anything else associated with your experience as part of the Climate Resilient Chickpea Innovation Lab? f. CRC-IL and AWG-IL Protocol for Interviews with USAID Mission Staff Introduction The purpose of this interview is to assess the perspectives of key partners as part of the Climate Resilient Chickpea (Applied Wheat Genomics) Innovation Lab. We are also conducting confidential interviews of the Program Director, PIs, Co-PIs, Collaborating researchers in the US and in partner countries, as well as other relevant stakeholders. Interviews are confidential. Our aim is to better understand the how research activities are proceeding and what factors might improve Innovation Lab’s ability to attain its objectives. I have several questions that will take about 60 minutes of your time. Do you have any questions before we begin? 11. What is your role as USAID Mission staff officer responsible for this Innovation Lab? 12. For each of the following areas, please give me at least one achievement/success and one suggestion for future improvement for this Innovation Lab: a. Management and implementation b. Communication and coordination for the achievement of project goals and objectives c. Partnerships with other sectors d. Research and training e. Dissemination of research results/products f. Institutional collaboration 13. What have been the biggest challenges within/facing the program and have they been overcome dealt with? 14. What are the outputs of this Innovation Lab that you consider to have had greatest impact (or potential to achieve impact in future) and how are they relevant to demand-driven R4D? 15. In general, how can the outputs/products and the process of generating those outputs be improved in the future, i.e. in terms of both increasing quality and generating appropriate outputs for Feed the Future countries? 70 | P a g e List of persons contacted AWGIL CRCIL Interviewed (as of Feb 22, 2018) Interviewed (as of Feb 22, 2018) IL Director IL Director IL Project Coordinator 1 EAP member 4 Collaborating Scientists 2 Collaborating scientists 1 USAID AOR 1 USAID AOR 1 Long-term trainee 15 long-term trainees* *Some trainees were associated with, but not funded by, the USAID IL List of materials reviewed Applied Wheat Genomics Innovation Lab. (2016) Poland, J. (ed). Third Annual Report. Applied Wheat Genomics Innovation Lab, Kansas State University. Manhattan, KS. 30 pp. Applied Wheat Genomics Innovation Lab. (2015) Poland, J. (ed). Second Annual Report. Applied Wheat Genomics Innovation Lab, Kansas State University. Manhattan, KS. 30 pp. Applied Wheat Genomics Innovation Lab. (2014) Poland, J. (ed). First Annual Report. Applied Wheat Genomics Innovation Lab, Kansas State University. Manhattan, KS. 30 pp. Climate Resilient Chickpea Innovation Lab. (2014) Doug Cook (ed.), Annual Report: January 1, 2014 – September 30, 2014. University of California at Davis, Davis, CA. 12 pp. Climate Resilient Chickpea Innovation Lab. (2014) Doug Cook (ed.), Annual Report: October 1, 2014 – September 30, 2015. University of California at Davis, Davis, CA. 11 pp. Climate Resilient Chickpea Innovation Lab. (2014) Doug Cook (ed.), Annual Report: October 1, 2015 – September 30, 2016. University of California at Davis, Davis, CA. 13 pp. Climate Resilient Chickpea Innovation Lab. (2014) Doug Cook (ed.), Annual Report: October 1, 2016 – September 30, 2017. University of California at Davis, Davis, CA. 12 pp. https://www.k-state.edu/wheat-innovation-lab/, Feed the Future Innovation Lab for Applied Wheat Genomics website. Accessed repeatedly between Nov 15, 2017 and February 15, 2018. http://chickpealab.ucdavis.edu/, Feed the Future Lab for Climate Resilient Chickpea website. Accessed repeatedly between November 15, 2017 and February 15, 2018. USAID Cooperative Agreement No AID-OAA-A-13-00051 (Applied Wheat Genomics), August 2013. USAID Cooperative Agreement No AID-OAA-A-14-00008.mod 3(Applied Wheat Genomics), January 2014. 71 | P a g e Responses from Project Directors A. Applied Wheat Genomics Innovation Lab (AWGIL): Jesse Poland, Project Director, Kansas State University Responses are listed according to the table of contents in the AWGIL Report. 1. Executive Summary, Findings, Page iv - v Engaging Women Through Capacity Building This is mentioned multiple times throughout the document as a shortcoming of our project and it was realized that this was due to a reporting misunderstanding on our part more than lack of training women scientists. Due to staff and AOR shifts during the project and TraiNet visa issues, reporting of long-term training was inconsistent and inaccurate. We have trained multiple female scientists and continue to train and engage women scientists in our project since the start of the Applied Wheat Genomics Innovation Lab (AWGIL) in 2013. Long-term Trainees, AWGIL (REVISED) Gender University Degree Major Program End Date Home Country Male KSU Ph.D. Genetics 2018 India Male KSU Ph.D. Genetics 2019 Bangladesh Female KSU Ph.D. Genetics 2019 USA Male KSU Ph.D. Genetics 2016 USA Female Cornell University Ph.D. Plant Breeding & Genetics 2017 India Female CIMMYT Post Doc Wheat Breeding NA India Male TERI University Ph.D. Plant Biotechnology 2015 India Female Cornell University Ph.D. Plant Breeding & Genetics 2019 USA Female Cornell University Post Doc Plant Breeding & Genetics NA China Male Punjab Agricultural University Ph.D. Biotechnology 2021 India Male Punjab Agricultural University Ph.D. Breeding 2021 India Male KSU Post Doc Genetics NA Nepal Leveraging Funding The AWGIL has in fact put significant effort into, and has seen significant returns from leveraging funding from other resources. We were never asked to include a list of complementary and leveraged projects in our reporting, which is why that information is missing. Due to the structure of the AWG IL we are not authorized to access USAID Mission funding. While we have engaged with the USAID missions in-country, we have not engaged with the missions for funding. This is a potential untapped opportunity if authorized that we 72 | P a g e plan to leverage in phase two, but have been able to leverage many other funding opportunities. Our progress over the past four years has been a collective effort and investment by many federal, state, and international agencies. Support from USDA’s Wheat Production Enhancement Program (WPEP) has provided foundational infrastructure and funded the field work in Pakistan. WPEP created infrastructure development for plot experiments and has provided equipment including plot planters and combines that enable large scale field testing and experiments. Parallel support and funding from other federal and state agencies and grower groups such as US National Science Foundation, NIFA IWYP and the Kansas Wheat Commission has allowed us to run parallel trials and research in the U.S. implementing the same technologies and approaches on US winter wheat—leading to more rapid overall progress toward developing improved breeding technologies that are applicable to breeding programs anywhere in the world. An example of this interagency compliment is that we are now using an automated unmanned aerial vehicles (UAV) analysis pipeline to process data from trials in South Asia that was developed using data from Kansas Wheat Commission funded trials last summer in Kansas. Support from the Gates Foundation, the McKnight Foundation, and NSF has allowed us to develop and deploy phenoApps around the globe to improve data collection and integrity, enabling data collection on much larger experiments, and providing more rapid turn￾around on phenotypic data. 2. Introduction, Program Background, Page 1 Total AWGIL Funding The document notes that we received a total of $4,990,218 for a period of five years. The actual funding level received was $4,548,544. 3. Evaluation, Pages 5, 7, and 9-10 Bangladesh Infrastructure It is noted on page 5 for AWG-RP 5 that our trials were carried out in multiple locations in Bangladesh, which is incorrect. We have a single trial location site at Jamalpur, Bangladesh. On page 7, under AWG-RP 6—we have completed the fabrication of a thresher in-country in Bangladesh. A photo of the equipment in progress is shown below. 73 | P a g e 4. Publications Below are additional publications from the AWGIL team not mentioned in the EET report. Sun, J., J.E. Rutkoski, J.A. Poland, J. Crossa, J-L Jannink, and M.E. Sorrells. 2017. Multitrait, Random Regression, or Simple Repeatability Model in High-Throughput Phenotyping Data Improve Genomic Prediction for Wheat Grain Yield. The Plant Genome. doi:10.3835/plantgenome2016.11.0111. Suneel Kumar, Marion S. Röder, Ravi P. Singh, Arun K. Joshi, Uttam Kumar (2016) Mapping of spot blotch disease resistance using NDVI as substitute to visual observation in wheat (Triticum aestivum, L.). Mol Breeding 36:95 Crain, J., Mondal, S., Rutkoski, J., Singh, R. P., & Poland, J. (2018). Combining High-Throughput Phenotyping and Genomic Information to Increase Prediction and Selection Accuracy in Wheat Breeding. The Plant Genome, 11(1), 0. https://doi.org/10.3835/plantgenome2017.05.0043 Montesinos-López, O.A., Montesinos-López, A., Crossa, J., Montesinos-López, J.C., Mota￾Sanchez, D., Estrada-Gonzalez, F., Gilberg, J., Singh, R., Mondal, S., Juliana, P. 2018. Prediction of multiple-trait and multiple-environment genomic data using recommender systems. G3: Genes, Genomes, Genetics, 8(1), 131-147. 5. Recommendations, Page 12 High-Throughput Platforms We are already actively expanding high-throughput phenotyping platforms past hand-held sensors with unmanned aerial vehicles (UAVs) in countries where allowed. The paperwork and approvals are in progress for Bangladesh and Pakistan and UAV operations have been actively conducted for several years in India and Mexico. Geographic Reach We are not currently working in Nepal. 74 | P a g e B. Climate Resilient Chickpea Innovation Lab (CRCIL): Douglas R Cook, Project Director, University of California Davis To help orient the responses, the relevant text and location within the review document are listed first to which each of the responses relates. 1. Bottom, Page iv Report text: “The AWGIL and the CRCIL have faced at least two common challenges in their research programs. In both cases the capacity to create data seems to exceed the capacity to manage those data and optimize their value to the program and to other scientists. This situation reflects a broad issue in the scientific community today, and it is a significant challenge to the work conducted in these ILs.” Response: Here I respectfully disagree. We have been very effective in generating data, managing that data, and turning the data into knowledge. But this is an ongoing task, and one that will continue through the full duration of the project. We have systems for archiving and access to large genomic data sets now active at NCBI, and numerous instances in which we have shared data sets with external groups interested in using the information. Our computational biology team has sophisticated activities underway to understand the genetic architecture of yield-related traits, and to deliver Genomic Prediction models to breeders by the end of the project. Our “public facing” informatics activity now includes collaborators at the James Hutton Institute, with funding from the Crop Trust, with which we have developed a relational database to handle seed information including passport data, genotypes, phenotypes and a range of collection site metadata. We are actively working to curate the data and place the data into the proper format for upload into the database. Perhaps the sheer volume of data, and the fact that much data collection are still being collected and curated, may have given the review team the notion that we were struggling in this regard, which is certainly not the case. 2. First Paragraph, Page v Report text: “The underrepresentation of women in science, particularly in agricultural sciences, in the target countries has made it difficult for both ILs to identify and recruit highly qualified women into their degree training programs.” Response: It is true that the underrepresentation of women in our recruitment pools was initially a challenge. But with continued effort throughout the first year of the project, we have achieved ~50% representation of women. Our initial set of “TraiNet-recognized” students included only one woman and five men. But with the advent of additional resources (a Legume Innovation Lab scholarship, funding from the PEER funds, and support from the Crop Trust, for examples), we have involved a larger set of women trainees – for example, two additional female PhD students from Ethiopia were recruited under PEER, one female student from Kenya 75 | P a g e was recruited under the Legume Innovation Lab fellowship, and one female Turkish trainee was recruited under the Crop Trust award. The results of these and other recruitments are that we have balanced gender outcomes. 3. First Paragraph, Page v Report text: “While both ILs had and met overall quantitative targets for trainees, establishment and ongoing management of gender targets between USAID and IL leadership was inconsistent.” Response: Certainly we need to provide verifiable, quantitative data about training, which we have done in the form of tables submitted with the 2017 year progress report. That table documents that we have significantly exceeded our training goals, and achieved gender equity in doing so. In earlier years, our reporting may have been less clear, but at least from this Director’s perspective, reporting formats were difficult to understand and did not appear provide latitude to fully report and explain our training activities. Such confusion on my part perhaps led to a lack of clarity in reporting of outcomes, and if so then I take responsibility. Nevertheless, I do not believe that solution is to create more onerous reporting schema – rather, I would like to work with USAID to determine how best to provide our current data in a format that can meet their reporting needs. 4. Second Paragraph, Page v Report text: “Both the AWGIL and the CRCIL have conducted numerous short duration trainings. Systematic pre- or post-testing assessments or other mechanisms that would allow evaluation of learning outcomes were not used in either case.” Response: True. But this does not mean that we have not been effective. I believe that we have been unusually successful in creating both disciplinary and critical thinking skills among our trainees, and we have given them world-class learning environments. While I support the need to have assessments, I simultaneously worry about creating a situation where such reporting requirements become so substantive that they impede training itself, by distracting us from our training activities. Again, this may be an area where dialogue between the Innovation Labs and USAID could result in better mechanism. 5. Third Paragraph, Page 14. Report text: “Fewer cultivated genotypes have been used for the most advanced materials than was initially planned, but the size of the populations created is larger than initially proposed.” Response: This is incorrect. We proposed 5 cultivated elite parents, and we have exceeded that number: Ethiopia (2), Turkey (1), India (1), Canada (1), and Australia (1). The “crossability” of 76 | P a g e accessions differed, so the balance among elite genotypes was uneven in two of the six cases. As the reviewers point out, the total number of crosses also exceeds the initial objective. 6. Second Paragraph, Page 15 Report text: “The quality of the chickpea physical map has limited the extent to which representation of chromosomal segments can be determined to date, with about 50 percent of the populations anchored, and the rest remaining works-in-progress.” Response: There is confusion about technical details here, leading to a misinterpretation. The initial (2013) chickpea genome was a draft genome sequence (no physical map), containing only ~50% of the genome anchored and ~60% present. The Chickpea Innovation Lab started in 2014, and one objective was to improve the reference genome. To date, the project has generated three new reference genomes (two wild, and one cultivated). These genome references capture 40-50% more of the genome than the 2013 draft, essentially complete genomes, with assembly statistics improved by ~10-fold. All data have been deposited to NCBI. Currently we are using Hi-C technology to finalize chromosomal level assemblies that we anticipate releasing in the summer of 2018. We expect these revised assemblies to have a huge impact on our ability to understand and interpret data, and to provide a superior baseline from which to guide breeding efforts. 7. Second Paragraph from Bottom, Page 15 Report text: “Given the decision not to pursue creation of ABI populations, there is nothing to report concerning ABI line phenotyping.” Response: The ABI populations were replaced by an increased effort on recombinant inbred (RIL) populations. We have made great progress phenotyping for a range of traits, both in the wild parental material and in the segregating pre-breeding materials. Field phenotyping and controlled-environment assessment of traits continues, and breeding is underway for some traits. Importantly, we are beginning the process of building quantitative models for breeding values for yield-related traits. 8. Fifth Paragraph, Page 16 Report text: “Better software systems from the Consultative Group for International Agricultural Research (CGIAR) system would help address this gap.” Response: We are not limited by software systems, and in any case it is debatable whether CGIAR systems are the ones to use. The fact is that data generation is ongoing and as new data types are generated we appropriately revise data structures and analyses, which is to be expected. We spent an entire session at our annual meeting discussing this topic. Our current trajectory is to move much of our data into the Germinate database in Scotland, from which users can download information about germplasm, genoptypes and phenotypes. We are also 77 | P a g e implementing a field phenotyping system and database, using the CSIRO FieldPrime data collection application and database. The advantage of FieldPrime is its simplicity and the fact that it is in use by project collaborator in Australia, providing a simple means to integrate data among partners. If the project were to be funded beyond its current end date, then we will implement breeding-related data tools, for which the BMGF-funded (CGIAR associated) Breeding Management System is one of several options to consider. There are numerous solutions available, each with advantages and disadvantages. 9. Sixth Paragraph, Page 16 Report text: “While all data have not yet been made public, this objective was repeatedly referred to as a priority during interviews and plans are to have a public-facing system running before the end of the current phase of funding.” Response: The vast majority of the genomic and genetic data are already available at NCBI, and we are actively curating germplasm and phenotype data to enter the Germinate database. “Public-facing” data, with mature data sets (e.g., SNP tables rather than raw reads), will be a key deliverable by the end of the project. 10. Second Paragraph, Page 27. Report text: “Given the upstream focus of the CRCIL, it is difficult to see how, at this time, broader engagement of the chickpea research community in developing countries would be of value. Broader downstream engagement should become a greater focus of more applied efforts moving forward.” Response: We have been very effective at engaging the broader chickpea community. Our annual meetings have been at least doubled in size based on the attendance of local chickpea researchers (first in Ethiopia, then Turkey, then India). Nearly every major chickpea research activity in the world has been engaged with the Chickpea Innovation Lab at one point or another, and numerous of these groups are now active users of our data and germplasm resources. 11. Last Paragraph, Page 30. Report text: “Based on this evaluation the EET believes that, for many reasons, it would be optimal wherever possible to utilize systems in use by CGIAR member organizations.” Response: The CGIAR data systems are one of several systems to manage genetic and breeding data, and the CGIAR systems have limitations the same as others. Chickpea Innovation Lab staff have invested considerable time and energy into exploring these issues, and have collected input from numerous stakeholders. For purposes of the first phase of the project, our data organization resources are principally NCBI (genomic data) and Germinate (germplasm and phenotypes, with some genetic/genomic content), both of which are fully publicly accessible. 78 | P a g e Our data are also being organized and delivered to user communities through the NSF-funded Legume Federation at Iowa State and the National Center for Genome Resources. If there is a second phase of the project, then we will evaluate potential solutions and make a rationale, need based decision about best practices, which may or may not involve CGIAR platforms.