External Evaluation of Feed the Future Innovation Lab for Collaborative Research on Peanut Productivity and Mycotoxin Control August 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Joan Fulton (Chair), Farid Waliyar, Medson Chisi, and Eric Welch. ii External Evaluation of Feed the Future Innovation Lab for Collaborative Research on Peanut Productivity and Mycotoxin Control August 2016 Cover photo: Village groundnut farmers, Ghana, June 2016. Photo courtesy of Medson Chisi. The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii Contents ACRONYMS.............................................................................................................................................. v EXECUTIVE SUMMARY..................................................................................................................... viii INTRODUCTION .................................................................................................................................... 1 RESEARCH PROGRAM.......................................................................................................................... 1 Introduction .............................................................................................................................................. 1 A. Peanut Germplasm Development........................................................................................................ 2 A1. Translational genomics to reduce pre-harvest Aflatoxin contamination of peanut ...................... 2 A2. Silencing of Aflatoxin Synthesis through RNA Interference (RNAi) in Peanut Plants................... 3 A3. An integrated Global Breeding and Genomics Approach to Intensifying Peanut Production and Quality................................................................................................................................................... 4 B. Mycotoxin Detection and Peanut Nutritional Studies......................................................................... 6 B1. AflaGoggles for Screening Aflatoxin Contamination in Maize ....................................................... 6 B2. Development and Validation of Methods for Detection of Mycotoxins Exposure in Dried Blood Spot Samples......................................................................................................................................... 7 B3. Aflatoxin in Peanut and Peanut Products: Comparative Study on Analytical Methods for Detection of Aflatoxin .......................................................................................................................... 8 B4. Randomized Controlled Trial of the Impact of Treating Moderately Malnourished Women in Pregnancy .............................................................................................................................................. 9 C. Peanut Value Chain Interventions......................................................................................................10 C1. Production to Consumption – Technologies to Improve Peanut Production, Processing and Utilization in Haiti ...............................................................................................................................10 C2. Using Applied Research and Technology Transfer to Minimize Aflatoxin Contamination and Increase Production, Quality and Marketing of Peanut in Ghana ......................................................13 C3. Producer and Consumer Interventions to Decrease Peanut Mycotoxin Risk in Ghana ............16 C4. Aflatoxin Management Interventions, Education and Analysis at Various Steps along the Peanut Value Chain in Malawi, Mozambique and Zambia...............................................................................18 C5. Productivity and Profitability Growth in Peanut Production: A Farm Level Analysis in Malawi, Mozambique and Zambia ....................................................................................................................20 PMIL Publication Outputs.......................................................................................................................21 PROGRAM MANAGEMENT...............................................................................................................22 Findings....................................................................................................................................................22 Conclusions.............................................................................................................................................26 iv Response to SOW Questions................................................................................................................27 Lessons Learned......................................................................................................................................27 Recommendations ..................................................................................................................................28 PROGRAM FUTURE.............................................................................................................................28 Program Management Recommendations..............................................................................................28 Core Mission.......................................................................................................................................29 Project Design Structure ....................................................................................................................29 Sub-contract Management ..................................................................................................................30 Host Country Scientists......................................................................................................................30 Research Program Recommendations....................................................................................................30 PMIL Project Organization..................................................................................................................30 Seed Availability ..................................................................................................................................31 Gender Integration .............................................................................................................................31 Training focus......................................................................................................................................31 Technology Transfer...........................................................................................................................32 Priority and Focus...............................................................................................................................32 Small Equipment..................................................................................................................................32 Value Chain .........................................................................................................................................32 Peanut Markets ...................................................................................................................................33 Peanut Products..................................................................................................................................33 APPENDICES..........................................................................................................................................34 APPENDIX A. SCOPE OF WORK ........................................................................................................35 APPENDIX B. EVALUATION PLAN .....................................................................................................49 APPENDIX C. TRAVEL ITINERARY, LOCATIONS AND DATES ......................................................95 APPENDIX D. LIST OF PERSONS CONTACTED...............................................................................96 APPENDIX E. LIST OF MATERIALS REVIEWED..................................................................................97 MANAGEMENT ENTITY RESPONSE TO THE EXTERNAL EVALUATION REPORT ....98 v ACRONYMS CGIAR Consultative Group on International Agricultural Research co-PI co-Principal Investigator CRI Crops Research Institute CRSP Collaborative Research Support Project CSIR Council for Scientific and Industrial Research DBS Dried Blood Spot EAP External Advisory Panel EET External Evaluation Team HPLC High-Performance Liquid Chromatography ICRISAT International Crops Research Institute for the Semi-Arid Tropics IITA International Institute of Tropical Agriculture IPM Integrated Pest Management JCR Journal Citation Report KNUST Kwame Nkrumah University of Science and Technology LUANAR Lilongwe University of Agriculture and Natural Resources ME Management Entity MFK Meds and Food for Kids MOFA Ministry of Food and Agriculture MSc Master of Science MWK Malawian Kwacha NARO National Agricultural Research Organization NARS National Agricultural Research Systems NCSU North Carolina State University NGO Non-Governmental Organization PhD Doctor of Philosophy PI Principal Investigator PICS Purdue Improved Crops Storage PMIL Peanut and Mycotoxin Innovation Lab RCT Randomized Control Trial RUTF Ready-to-Use Therapeutic Food SARI Savanna Agricultural Research Institute SNP Single Nucleotide Polymorphism SOW Scope of Work UDS University for Development Studies UF University of Florida UGA University of Georgia UNZA University of Zambia US United States USAID Unites States Agency for International Development VC Value Chain ZARI Zambia Agricultural Research Institute viii EXECUTIVE SUMMARY The overarching mission of the Feed the Future Innovation Lab for Collaborative Research on Peanut Productivity & Mycotoxin Control (PMIL) is to apply leading innovative US science to improve peanut production and use, raise nutrition awareness and increase food safety in developing countries. PMIL aims to integrate two major themes – peanut production and mycotoxin research – under one roof as part of a value chain approach (http://pmil.caes.uga.edu/about/index.html). This report is the result of a review of PMIL by an External Evaluation Team (EET). This report is based on desk review, presentations made by scientists, site visits, and Skype and phone calls. During site visits EET members met with faculty at several universities, scientists in National Agricultural Research Systems (NARS), farmers, field visits, discussion with representatives from non-governmental organizations (NGOs), other USAID supported programs such as peanut scale-up and all USAID offices in the visited countries. The report focuses on three main areas: program management; research program; and program future. PMIL is a multidisciplinary and multi-organizational project that is building on many aspects of the previous Peanut Collaborative Research Support Project (CRSP). PMIL is focusing on five Feed the Future countries – Haiti, Ghana, Malawi, Mozambique and Zambia. PMIL is structured around twelve research projects, which were determined through a combination of commissioned and competitive proposals and evaluation. The twelve projects fall into three areas, grouped as A, B and C projects. The A projects are under the heading of Peanut Germplasm Development; the B projects under the heading of Mycotoxin Detection and Peanut Nutritional Studies; and the C projects under the heading of Peanut Value Chain Interventions. Crop production research projects are incorporated into the C projects. Each of the twelve projects has a Principal Investigator (PI) from a US university, Co-Principal Investigators (co-PIs) and Partners from other US universities, CGIAR centers, universities and research services in partner countries as well as the private sector. The A projects utilize genotyping and associating peanut molecular variation with resistance to pre-harvest aflatoxin contamination. This information is used in breeding programs to develop new peanut varieties with improved yields, drought tolerance, pest and disease resistance and value-added traits. The EET found good research from these projects. PMIL should encourage PIs and co-PIs to exploit all possible linkages with other organizations, and especially the CGIAR centers. The EET encourages additional capacity building, both with respect to more training and improving equipment and facilities in host country institutions. ix The B projects cover the two broad areas of mycotoxin detection and nutrition. The EET concludes that aflatoxin testing and detection is important and support efforts to develop a detection technique that is easy to use in host countries. The EET notes that the full cost of strip testing may be high and recommends this be examined. The EET expressed concern that project B3 was undertaken without an adequate literature review with the final results not being as relevant as they could be. The projects B2 and B4, dealing with dried blood spot samples and nutrition in pregnant women respectively, involve interesting research with good quality science. The EET notes that these two projects are distinct from the other PMIL projects such that it is difficult to see the synergies. The EET encourages PMIL to evaluate projects from the perspective of how they fit with PMIL’s core mission. The C projects have a value chain approach with C1 focusing on Haiti, C2 and C3 on Ghana and C4 and C5 on Malawi, Zambia and Mozambique. There is interesting quality research in each of the projects. It is interesting to note that in Ghana and the southern and eastern African countries the economics project is separate from the value chain project, a sign that there is not a good understanding of a value chain approach by the scientists. The work in Haiti has some important value chain aspects, but there is still room for greater communication and integration with the work of the economists. The EET examined the effectiveness and efficiency associated with sub-contracts and funding flow to the projects. Specifically two important factors were timely flow of funds and ability to use the funds for project expenses. Since both of these issues were a constraint at the start of PMIL the ME worked with the UGA sponsored programs/contracts office to develop a much more effective model. This new model resulted in a fixed price contract between UGA and the partner country organizations. The ME of PMIL received consistently positive comments from the people the EET interviewed during the review. The EET found that the value chain projects have value chain as part of their title but often not part of the work of the project. Gender is supposed to be a cross-cutting theme, but the EET found that this was often a topic that was glossed over. The EET did find significant emphasis on capacity building through the training of students, often in host country universities. The EET is impressed with the successes of PMIL to date, but also notes that there are areas for improvement. In looking to the future, the EET recommends that PMIL revisit and redefine their core mission; develop and implement a fully transparent process for project selection; further improve the efficiency of management of sub-contracts; increase the influence and participation of host country scientists in PMIL; structure the organization of their projects to promote strong NARS in the HCs; address the problem of a lack of breeder and foundation seed; incorporate gender directly into the program; continue its focus on training while x evaluating the advantages and disadvantages of training in host country institutions; include technology transfer as part of PMIL; engage in research on small equipment and machinery that promotes small businesses for women; embrace a value chain approach, enhance the understanding of peanut markets; and develop new peanut products to grow the demand for peanuts. Page | 1 INTRODUCTION This external evaluation of the Feed the Future Innovation Lab for Collaborative Research on Peanut Productivity & Mycotoxin Control (PMIL) was conducted by a four member External Evaluation Team (EET). The team had several meetings before initiating the process of the review. They laid out a plan and proceeded to review the research activities of PMIL in phases. This started with team members reading the major documents provided by USAID (EET 2007 and 2012, full proposal, annual reports, etc.). The team members also visited the public web site of PMIL as well as a secured drive where team members were given access to different project documents. Two of the EET members visited the Management Entity (ME) at the University of Georgia (UGA), meeting with the management team, UGA officials, and a number of PIs and co-PIs. The EET heard presentations from several project scientists on almost all of the PMIL projects. Some scientists were present at UGA while others made their presentations using video-conferencing. Following the visit to the ME the EET made site visits to four of the host countries, Ghana, Malawi, Zambia and Haiti. Each visit involved two team members, a format that brought different disciplinary perspectives to the review and evaluation and a method of verifying observations and conclusions. This report is based on desk review, presentations made by project scientists, and site visits. During site visits EET members met with faculty at several universities, scientists in National Agricultural Research Systems (NARS), farmers’ field visits, discussions with representatives from NGOs, other USAID supported projects such as groundnut scale-up and all USAID offices. In the following section the research program is reviewed. This section includes a sub-section for each of the twelve research projects. At the end of each sub-section the Scope of Work (SOW) questions relating to productivity and mycotoxin control, quality of research, research challenges, the dual benefit mandate of Title XII authorized programs, and training for technical capacity and academic improvement are addressed. The EET also completed a review of published materials from PMIL that covered all publications, regardless of specific project. Thus, a section discussing research publications is presented separately following the twelve research sections. Next the program structure and management is reviewed, with the SOW questions relating to program management specifically addressed at the end of that section. The vision the EET identified for the future of the PMIL program comprises the final section. RESEARCH PROGRAM Introduction This section of the report includes the EET’s evaluation of the research progress and technology development as well as some transfer activities implemented from 2014 to date. The team conducted desk review, site visits of the ME at UGA, site visits to host countries, and conducted telephone/Skype interviews with the PIs, co-PIs, and some members of the External Advisory Panel (EAP). Team members analyzed all data and information gained, held numerous telephone calls to generate the observations and conclusions associated with the research and presented below. Page | 2 The EET undertook the review from the perspective that PMIL’s overriding objective is to generate research outputs that when applied, increases the productivity and profitability of peanut production for smallholder farmer and to reduce the negative impacts of mycotoxin contamination along the value chain of peanut and other crops in five Feed the Future countries – Haiti, Ghana, Malawi, Mozambique and Zambia. PMIL focuses on research related to peanut production, mycotoxin control, seed production, post-harvest handling and processing, market opportunities, gender, as well as some instructional workshops and training. The PMIL projects are divided into three areas and labeled as ‘A’ projects (Peanut Germplasm Development), ‘B’ projects (Mycotoxin Detection and Peanut Nutritional Studies) and ‘C’ projects (Peanut Value Chain Interventions). The projects are evaluated below. A. Peanut Germplasm Development The A projects utilize genotyping and associating peanut molecular variation with resistance to pre-harvest aflatoxin contamination. This information is intended to be used in breeding programs to develop new peanut varieties with improved yields, drought tolerance, pest resistance and value-added traits. In addition the information will be used in genetic transformation of peanut plants using RNAi technology to reduce aflatoxin in peanuts. A1. Translational genomics to reduce pre-harvest Aflatoxin contamination of peanut The discovery and use of high throughput DNA sequencing, improved bioinformatics and statistical analyses has brought about significant advances in the field of molecular genetics. Researchers in breeding programs working on various crops are now able to investigate genome-wide variations in DNA sequences and link them to the inheritance of complex traits controlled by many genes such as yield. Consistent with this, PMIL’s mission is to apply innovative science to improve peanut production and use, raise nutrition awareness, and increase food safety in developing countries. The aim of the project (A1) is to associate molecular variation with resistance to pre￾harvest aflatoxin contamination on a genome wide scale and link this with classical breeding methods. The process of genotyping is being enabled by sequencing information from both tetraploid (cultivated) as well as diploid (wild) progenitors of the tetraploids. This work is being carried out in the US and the host countries India, Senegal and Niger. A number of promising lines are being evaluated in these and other countries. A multi-disciplinary approach is used in this project as the PI is a molecular geneticist and the co-PI is a classical plant breeder. A lot of work has been accomplished within a short period. Sequencing of additional genotypes is on-going to expand the pool of Single Nucleotide Polymorphisms (SNPs) that can be used for genotyping. The work on the SNP Chip measuring polymorphism and variability is a plus for PMIL and will assist in the identification of rosette virus resistant varieties in the project. The PI and co-PIs have established excellent collaborations with many NARS and international centers. The project achievements are Page | 3 positive and noteworthy as the new breeding lines will help farmers as well as other breeding programs. The EET learned that there are currently two graduate students, from Ghana and Kenya, supported by this project and studying in the US. A Visiting Scientist from Senegal has spent several months in the PI’s laboratory. The exposure to molecular techniques in addition to classical methods should assist host country researchers in developing high quality varieties that are associated with disease and pest resistance and reduced aflatoxin levels. Response to SOW Questions There is an excellent link between peanut productivity and mycotoxin control in this project as the core objective is to develop new varieties with resistance to aflatoxin. The use, in Project A1, of applied genomics in breeding represents good and appropriate research yielding positive results. Specifically, this project has made progress in molecular plant breeding with the following effect:  Shortening the time it takes to domesticate new crops from wild relatives.  Tailoring existing crops to meet new requirements, such as nutritional enhancement.  Incorporating valuable traits from wild relatives into established crops. The major challenge for the research concerns facilities in host country institutions that will enable them in the long term to carry out biotechnological research locally. Additionally, the delayed disbursement of funds to host country collaborators was a challenge. The output of this research will benefit the peanut industry in the US as well as in the host countries. Building local capacity is one of the main accomplishments of the project and there are already two graduate students in the USA receiving training and some are on the way. Recommendations i. Continue work using applied genomics in breeding. ii. Continue training for capacity building. iii. Set up and improve basic equipment for the host country scientists to carry out their research in the host country institutions. iv. Continue this work that has already produced exciting results from the work done so far and is critical to finding molecular variation in peanut. v. Encourage PIs and co-PIs to exploit all possible linkages with other organizations, and especially the CGIAR centers. A2. Silencing of Aflatoxin Synthesis through RNA Interference (RNAi) in Peanut Plants The main emphasis of this project is to use RNA interference (RNAi) to reduce aflatoxin in peanut seeds. The focus of this project is to look at the genetic diversity of Aspergillus species and the transformation of the plants using RNAi. It is interesting to note that partners in this work include ICRISAT in Malawi (from the CGIAR), Kenyatta University in Kenya, and the National Agricultural Research Organization (NARO) in Uganda. Another Page | 4 important finding is the quantification of aflatoxins in peanut seeds using ultra high performance liquid chromatography. The EET identifies some fundamental questions associated with the potential for success for this project that utilized genetic modification. EET members, starting from the knowledge that there are some countries that do not want genetically modified products, asked the PI how she would arrange for these genetically modified products to be tested in Africa. The EET was concerned when her answer was that “they would go to the countries that would agree.” In particular they question the robustness of the research design. While this particular research involves work with Kenyatta University to produce RNAi transformants for testing in Kenya the research could have limited impact. If no other countries allow the product to be tested the question of research relevance will be paramount. Response to SOW Questions This project has a core objective to link peanut productivity and mycotoxin control by breeding new peanut seeds that have reduced aflatoxin. The EET found that good quality research was being undertaken in this project, but noted concerns about whether the products could be tested in the appropriate countries due to differing policies across countries concerning genetically modified products. The output of this research has potential to benefit both US peanut production and peanut productivity in host countries. Training at graduate and technical levels is an important component of this project. For genetic diversity of aflatoxigenic Aspergillus species funded by PMIL, training is offered in DNA extraction, bioinformatics and isolating and identifying different strains. Peanut transformation is funded by Norman Borlaug Commemorative Research Initiative (NBCRI) and students are trained in molecular techniques for screening transgenic plants. A graduate student from Kenya is receiving training in peanut genetic transformation and molecular tools and an additional PhD student from Haramaya University in Ethiopia has also been trained at the lab in Dawson, GA. Recommendations i. Promote collaboration, expanding to other institutions when possible to ensure the effectiveness of the project. ii. Engage in capacity building in the host countries to improve equipment and facilities that are necessary for quality research. A3. An integrated Global Breeding and Genomics Approach to Intensifying Peanut Production and Quality The project is developing new improved varieties and transferring these improved varieties and management practices to end-users. Emphasis is placed on developing high yielding peanut varieties with high oleic and oil content. Biotic and abiotic stresses and other factors that mitigate aflatoxin contamination are the focus. Uganda, Zambia and to some extent Ghana were countries visited and talked to by the EET and they have active breeding programs. These programs have successfully developed and Page | 5 released new and improved varieties that are high yielding, drought tolerant, resistant to rosette virus, leaf spot, with increased micronutrient (zinc and iron), high oleic and oil content. Breeding projects and activities take time, are a continuing process and are slow to show results in terms of new varieties. The five-year cycle for PMIL is very short in terms of a breeding timeline. All of the crosses being made now that are incorporating wild genes for disease resistance and drought traits will be released long after the end of the project. The EET members were also surprised that farmers were requesting seed of released varieties from CGIAR centers when the country has a national peanut breeding program, and typically new seed varieties are released by a national program. The EET observed that the breeding programs are on track with new varieties in the pipeline. The EET learned that the Uganda breeding program has continued with the breeding activities from the previous Peanut CRSP to PMIL. This program works closely with the PI at the UGA on developing and improving high yielding and culturally preferred varieties. Biotic stresses are addressed by focusing on developing varieties with resistance to rosette virus. The problem of ‘Leaf miner’ on peanuts in East and Central Africa was noted. EET members also found it positive that breeding activities have also included the quality traits of high oleic content and high oil. The EET was pleased to identify that the level of interactions among the host country researchers has gone beyond annual planning meetings and workshops. The host country researchers are now exchanging germplasm and visits. Technology transfer activities are few in number, in large part due to the fact that linkages with agricultural extension are weak. In cases where the various NGOs were working to reach farmers, there was an observed shortage of seed. This observation suggests that farmers are willing to pay for new and improved seed and a need exists to expand the seed duplication and distribution system in the host countries. While the focus of PMIL is research, the EET identifies that without activities associated with moving from new seed varieties through registration and approval and then seed multiplication the impact of the research cannot be achieved. Similarly without extension activities farmers will not know about the benefits from the new varieties, another factor negatively affecting the adoption of improved technology. Response to SOW Questions This project links peanut productivity with mycotoxin control as part of its core objective. This project has a lot of potential to generate outputs and impacts. A number of varieties have been developed with drought tolerance, high oleic and oil content, resistance to late leaf spot, groundnut rosette disease, large seed and quality traits such as zinc and iron. The project has made several presentations and has released a number of papers in journals. In many cases, the issue of contracts affected the implementation of the project where there was no funding upfront. In addition, lack of infrastructure such as greenhouses and poorly equipped laboratories were some of the challenges faced. It will be important that partners in this project get the necessary support to be able work on genomics so that varietal selection can take place in-country. The output of this research will benefit the peanut industry in the US as well as in the host countries as some of the required traits may be the same for the US and the partner Page | 6 countries. In addition, improved sources for those traits can be used by US scientists as well as partner country scientists. Capacity training under this project is being offered by the co-PIs at undergraduate level and the PIs at the graduate level. Recommendations i. Encourage PMIL scientists to utilize the gains with respect to new varieties from the Peanut CRSP breeding program to take advantage of the carryover and more quickly achieve the benefits from new varieties within PMIL. ii. In order to achieve the full potential from this project there needs to be greater availability of improved seed for the farmers to plant. The EET recommends that the NARS could provide breeder seed. PMIL needs to recognize that some aspects of seed production and distribution need to be embraced by the project to show success. iii. Training in molecular breeding needs to be accompanied with facilitation of rudimentary equipment to practice this. iv. PMIL is utilizing the state of the art advances in plant genomics. While attempts are being made to build human capacity in the host countries, the facilities should adapt to accompany the same since without adequate facilities the project cannot reach full potential. v. PMIL PIs and co-PIs should seek out partners and either through joint effort, or in partnership ensure that the necessary extension programs are in place so farmers learn about the new technology and adopt it. vi. Continue to build human capacity building as there are inadequate numbers in the NARS. B. Mycotoxin Detection and Peanut Nutritional Studies B1. AflaGoggles for Screening Aflatoxin Contamination in Maize The main objective of this project is to develop a rapid and portable technology for Aflatoxin detection. The research team hopes to have a product where you can run a sorting process to sort out the contaminated product and leave the good product for sale and commercial use. They began by developing a goggle device, but recently they have been exploring a Box Detector. They have also explored using the strip tests with a tablet scanner, which was demonstrated to the EET during the site visit to UGA. They started with basic research and are now seeing applications, specifically for PMIL uses. The PI noted that the budget that is program receives from PMIL is providing only about 10-15% of his overall research program budget. The EET observed that the research team is significantly contributing to the PMIL objectives. However, there is a need to focus on developing a technology that detects aflatoxin in the field. The PI reported that aflatoxin detection technique using strips is receiving serious consideration across all components of PMIL. The EET notes that although the extraction was simplified, the strip test remains costly. A central problem of the strip testing method is when the levels of aflatoxin are unknown, it may be necessary to use several strips to identify the aflatoxin level. This increases the cost of testing and the cost increases with each strip. The EET Page | 7 concluded that the strip test detection technique has potential for field use but methods to reduce costs need to be figured out. During site visits the EET met several partners who are using the strip test technique with tablet readers, and these partners are very satisfied. There is extensive interest in getting more units so that they can run several tests at the same time. There is also a need for high￾level training for the technicians and anyone running the tests. The training should not be limited to detection technology but should also cover sampling methods and very importantly safety measures for detection of aflatoxin. During one site visit the EET noticed that some students were using the toxin extract without taking safety precautions. The EET understands that some of the activities associated with the strip tests were initiated and led by the ME and not directly associated with any one project. The comments about the strip tests are included in this section because, following the site visits and phone interviews, the EET was left with the impression that this project was involved with the strip tests. In addition, the EET feels the comments and recommendations about the strip tests are important from the overall research component of PMIL. Response to SOW Questions The development of new detection technologies is contributing to the overall goal of the PMIL as it will help PIs, co-PIs and partners under the other PMIL projects (A, B and C) to carry out aflatoxin testing. The quality of research is excellent and the outcomes benefit scientists and other stakeholders with quality aflatoxin testing technology. One student was part of short-term training at Mississippi State University. Recommendations i. Evaluate the full cost of the strip test method of aflatoxin level determination. ii. Provide high-level training in the host countries for technicians and anyone running the tests. These training sessions need to be led by scientists with extensive knowledge in aflatoxins, testing, sampling methods and safety measures. iii. Ensure project titles appropriately reflect the work that is being done (e.g., in Project B1 change from “aflagoggles” to “new tools for aflatoxin testing”). iv. The EET recommends that funds should be used in developing a detection technique that is easy to use in host countries. This is particularly important at the locations where traders in the marketplace interact (e.g. weekly village markets). B2. Development and Validation of Methods for Detection of Mycotoxins Exposure in Dried Blood Spot Samples The goal of this project is to establish and validate methods for measuring major mycotoxin biomarkers, especially for aflatoxin-lysine adduct, in human Dried Blood Spot (DBS) samples for supporting urgent needs of nutrition impact and intervention studies conducted in Asia and Africa countries by PMIL, as well as the Nutrition Innovation Laboratory at Tufts University. The methods will be validated and applied to assess susceptibility factors in the determination of human aflatoxicosis, to evaluate the linkage between aflatoxin exposure and human nutrition deficiency and growth retardation and developmental inhibition in children. Page | 8 The project has made significant progress as the DBS protocol has been validated and the technology can be used for field studies to assess aflatoxin B1 exposure in different populations. The EET noted that the results are very good and the technology can be used by several organizations involved in nutrition/health studies. The EET feels that this technique is more related to the medical side than the agricultural side of aflatoxin contamination. The decision of whether to fund this project through PMIL should be made as part of the decisions regarding the strategic direction of PMIL. Response to SOW Questions The achievements under this project are very important for future health research as the test can be used to study the importance of mycotoxin in humans and animals as well as the interaction of mycotoxins with micro-nutrients. The test can be used both in the US and also in all of the host countries. The major challenge is how to transfer this technology so that diverse partners can use it. Some students have done their degrees in this topic at UGA. Recommendation i. If this work is consistent with the identified strategic direction, such that health issues are an integral part of the strategy, then this project should continue to be funded. B3. Aflatoxin in Peanut and Peanut Products: Comparative Study on Analytical Methods for Detection of Aflatoxin The EET was limited in its ability to review this project as the only information they had was from the PMIL annual report. This project is justified in the PMIL documents as follows: There are numerous methods to measure the toxicity of fungal infection in various crops. A primary limitation for aflatoxin determination in peanuts is the lack of generally accepted and standardized methods for farmers to screen or for testing laboratories to quantify the level of contamination. Even among PMIL collaborators, different evaluation methods have been reported in individual studies, making the comparison of results difficult. This project conducted a systematic comparative study to evaluate and report existing/emerging analytical methods for aflatoxin determination in peanuts and peanut products. A blind test, in which a variety of peanut products was naturally and artificially contaminated with aflatoxin, was prepared to test the current available analytical methods within the collaborating institutions/analysis laboratories. Results from the project were helpful to document the existing methods, the advantages/disadvantages of each method, and which method is best for each objective. During the EET meetings and visits there was no mention about this work except the information we got from the 2015 annual report. The comparison between several detection techniques is reported in the annual report. The understanding of the EET is that all of these methods have been previously developed by reputed scientists around the world and have been published. The cost involved with these technologies is well known as there are several papers comparing these technologies. These techniques are currently used by many organizations and most of them are Page | 9 commercially available. The EET is concerned that this research was undertaken without a proper review of the literature to ensure that the research filled a gap in understanding or provided a technological advancement. It is possible that the EET did not obtain all of the information about this project during the site visits and review of documents. However, the EET still questions the value that a project like this can add to the program, given the published literature that already exists. Response to SOW Questions This is routine research that tried to compare the cost of different technologies to detect aflatoxin in peanuts. The use of any of these technologies are helping scientist to accurately detect aflatoxin levels. Recommendation i. This project is finished and as the EET does not see how this research is not bringing any new information no further funds should be directed to other activities. B4. Randomized Controlled Trial of the Impact of Treating Moderately Malnourished Women in Pregnancy The objective of this project is to determine the benefits of treating moderately malnourished pregnant women with a peanut butter-based nutritional supplement. The trial is a randomized, investigator-blinded, controlled clinical effectiveness trial on pregnant women with moderate malnutrition, with and without HIV-infection, in southern Malawi. The trial used three different nutritional supplements for comparison with one being a peanut based Ready￾to-Use Supplementary Food. The aim of the study is to provide significant evidence that using a peanut-based supplementary food will reduce maternal mortality and improve infant growth and development. Results will provide national and international agencies with evidence to recommend and promote the use of peanut-based products for maternal health, as well as purchase some for use in their nutrition programs. This study was undertaken to understand the effect of under-nutrition during pregnancy. The results of this investigation did not show any affect in response to micronutrients. The PI indicated that the results suggest a need for combined intervention with a food product and treatment to decrease infections and poor health. The EET observed that the PI was doing a good job of leveraging funds, noting that this is one way for PMIL to be involved in the nutrition part of value chain at a relatively low cost. However, the EET also observed that the project is independent from all of the other projects so there are no synergies being experienced. The PI reported to the EET that he was asked to develop a proposal with a very short turnaround time. Thus, he had identified a project that would fit with his existing work. He was pleased with the research, but also noted that if he had had more time he could have designed a better project. Page | 10 Response to SOW Questions The focus of this project was on the use of peanut based products in nutrition of pregnant women and thus did not specifically link to mycotoxins. The EET observed good quality research from this project and no research challenges were identified by the PI or co￾PIs. The target audience for this research was pregnant women in Malawi so the benefits are greatest to the host country. There is opportunity for the US peanut industry as the benefits of peanut based ready to eat foods for development programs will create greater world-wide demand for peanuts, which will benefit US producers. This is a small part of the PI’s total research program. It has provided training to students from Cal Poly, trained local staff in the clinics and supported the PhD of one person at the University of Malawi. Recommendations i. Revisit the nature of this study to see how it fits with the core mission of PMIL. ii. Decide whether there is a need for continuing this work or redirect the funds towards other health studies. C. Peanut Value Chain Interventions C1. Production to Consumption – Technologies to Improve Peanut Production, Processing and Utilization in Haiti The project is developing a comprehensive production, processing and utilization strategy for peanuts in Haiti. All phases of peanut production are being evaluated, including varieties specific to the region and market influences. A seed-increase program and facilities to maintain genetic resources through curation of important peanut germplasm has been established. Capacity building is being promoted through the introduction of labor saving devices and harvesting equipment and procedures. In addition, activities are in place to evaluate the infrastructure to improve peanut handling, drying and long-term storage. Once these improvements have been evaluated, the project team plans to take the best management practices and strategies to the grower level in several villages and communities in the region, particularly through the depot network partnership with the Acceso Peanut Enterprise Corporation. The project team is providing training and infrastructure support to realize these improvements and ensure long-term capacity building. Aflatoxin and the role of women in the peanut value chain is being measured/surveyed throughout the duration and in all phases of the project. The project team is also establishing aflatoxin-testing facilities and re-training Haitians to measure aflatoxin and recognize the importance of avoiding aflatoxin in their diet. Another important capacity-building measure is the creation of alternative products/markets for high aflatoxin contaminated peanuts. An important component of this project is led by economists. Their goals are to evaluate the impact of the project across different segments of the peanut value chain, including: conducting a baseline survey to analyze current conditions, determine the impacts from improved varieties and other agronomic practices (row planting, herbicide use, fertilizer use, harvest maturity), determine the impacts from post-harvest practices (storage facilities, use of bags, tarps), economic feasibly of using bad peanuts to make safe animal feed and/or Page | 11 manufacturing fuel patties, establishing an insurance product using land quality and rainfall data, and determining the impact of gender. Findings The project PI from the University of Florida (UF) and co-PIs from the University of Georgia have established an important network of partners to implement the project in Haiti. This project is using a value chain approach and bringing together the 2 major components – crop production and aflatoxin management. There are several components of the project in the country: crop improvement, agronomy (fertilizer and weed control), crop protection (pest and diseases), pre and post￾harvest management of aflatoxin, processing and utilization. In terms of crop improvement, the project is introducing lines from the US, India and Africa, including released lines from Uganda (A3), and testing them under the different agro-ecological zones in Haiti. The EET learned during its visit to Haiti that rust and leafspots are severe in Haiti and that rust causes most of the damage and reductions to yield. The breeder from UF (who is a co-PI on the breeding project A3) mentioned that rust is not very severe in the US. They are taking the opportunity to screen lines in Haiti that can be used in peanut improvement programs in the US in case the disease become serious. In order to control the disease, several trials are been conducted on the efficiency of fungicide application. The breeder from UF is keen about training the newly hired legume breeder at CHIBAS and Quisqueya University. CHIBAS is an institute/research center on bio-energy and sustainability and a not-for-profit organization based in Haiti. The new legume breeder will initiate an appropriate breeding program in Haiti, which is important given that currently there is no program or organization in charge of peanut breeding in Haiti. There is no organization and no supporting structure for varietal release and registration of seeds in Haiti. The only organization in place is the National Seed Services. A Director from CHIBAS noted that CHIBAS is trying to play a key role by working to produce a national catalog of peanut varieties and establish a common record as a first step toward identifying specific varieties as Haitian varieties. Furthermore, there are no standards in place in Haiti for allowable aflatoxin levels. The research program on peanuts in Haiti is new and therefore has limited experience. The Director of CHIBAS explained to the EET that they are developing the scientific base for carrying out the required research. They have actually seven students among them three MSc students who are going to continue their PhD research later. One of the students will start soon at the UF in the US. CHIBAS has good collaboration with several intuitions in the US As well as in Europe. They are sending students outside the country but making sure that their field work is carried out in Haiti. The Director of CHIBAS noted that Haiti has large variation in agro-ecological zones (different soil types; low land to highlands; very dry to wet environments). There are important opportunities for research that could be carried out in Haiti and applied to other regions of the world, particularly the US. While the EET was positively impressed with these opportunities they noted that the facilities are not adequate for aflatoxin testing. The EET observed aflatoxin Page | 12 testing being carried out without appropriate safety precautions, most likely due to insufficient equipment and a lack of comprehensive training on aflatoxin testing. The economics component is making good progress. The baseline survey is completed. The data is collected for the randomized control trials to assess the effects of introducing microcredit along with new technology management practices and pre- and post-harvest handling procedures. This analysis was done in conjunction with Acceso. The EET observed that the economics component is using a current and appropriate methodology for analysis. The EET did observe that much of the economics work is done “on its own” and that increased collaboration among the economists and the rest of the team would enable them to incorporate the results of the economics research more quickly. This team has effectively connected with Meds and Food for Kids (MFK) and Acceso to incorporate the market aspects and enable a value chain approach. Acceso is providing farmers with input packages (e.g. seed, other inputs and credit) and buying peanuts from the farmers at harvest. Acceso partners with MFK in Haiti to supply them with quality (low aflatoxin) peanuts that MFK uses in their production of RUTF products. There are other aspects of peanut marketing that are not being explored by this project. In particular, further evaluation of how the typical open markets for peanuts operate would be beneficial, as well as a better understanding of how aflatoxin is managed and controlled in the marketplace. The EET was positively impressed with how PMIL has brought together key individuals and organizations associated with peanut research in Haiti, which is important given that the National Agriculture Research system is not involved in peanut research and seed registration/dissemination in Haiti. The EET observed that the PMIL team is fairly well connected with each other. They use phone and Skype conversations to develop their work plan. They are in regular communication with each other if there is any problem to be solved. The research activities are progressing well. PMIL (both in Haiti and the US) has a core group of good scientists and this is very important strength. The interactions with the economists that are working on the Haiti project are not as frequent as among the other members of the team, resulting in missed opportunities for a complete value chain project. Response to SOW Questions The EET did observe many positive aspects of the Haiti project. The project has embraced many of the aspects of a value chain approach. As stated in various sections excellent research is ongoing in Haiti. Both crop production and aflatoxin management practices have been successfully implemented. Most of the project activities are well planned, on time and the objectives are achievable. Many farmers are participating in the Acceso initiative. There are reports of farmers who tripled their yield of peanut in Haiti. Acceso has created new markets for peanuts. The project has been evaluating varieties and other pest and diseases management options. Several students have been trained. The EET notes that there is good potential for large impact from the project. The research will help both Haiti and the US. One important example is how the researchers documented the first examples of Tomato Cholorotic Virus in peanut. They can identify and test different Page | 13 management strategies for this problem in Haiti. This is a great opportunity to observe new pests on peanuts and get data on relative susceptibility of cultivars before they reach US. An important challenge the project should explore relates to the market issues, including a more comprehensive understanding of the demand for peanuts, price incentives for aflatoxin free peanuts and new peanut products. Recommendations i. Assist Haiti in establishing a breeding program in the country and develop capacity particularly in breeding and mycotoxin detection. ii. Assist CHIBAS in build the required infrastructure and facilities for aflatoxin testing & and breeding and pathology. iii. Increase the communication and collaboration between the economists and the other members of the PMIL team in Haiti. iv. Carry out an impact assessment study to document the impact of research carried out in Haiti. C2. Using Applied Research and Technology Transfer to Minimize Aflatoxin Contamination and Increase Production, Quality and Marketing of Peanut in Ghana A wide range of abiotic and biotic stresses negatively impact peanut production in the field and generally contributes to the reduced quality of marketed peanut in Ghana and West Africa. Aflatoxin contamination can occur and increase at all steps of the peanut supply chain including production in the field, storage in fields and villages, and in processed products. Interventions at each step of the supply chain can minimize aflatoxin contamination. Improved production in the field, including pest resistant cultivars, adequate soil fertility and plant nutrition, and synchronization of peanut pod growth phase with adequate soil moisture, can increase peanut yield and quality and minimize aflatoxin contamination. Adequate and timely drying of farmer stock peanut minimizes additional production of aflatoxin during storage in villages prior to marketing. Effective processing of farmer stock and shelled stock peanut can also reduce aflatoxin prior to purchase and consumption. Determining current practices by farmers, conducting research to mitigate aflatoxin and improve peanut quality, and transferring appropriate technology to farmers are needed to improve productivity, profits, and quality of peanut and to increase safety of peanut products consumed by humans and livestock. The primary platform being used to research aflatoxin contamination of peanut in the supply chain in Ghana is taking place in nine villages in northern and central Ghana. Interventions at each step of the supply chain are being implemented and aflatoxin contamination determined. Research is conducted at two institutions associated with the Council for Scientific and Industrial Research (CSIR), the Savanna Agricultural Research Institute (CSIR-SARI) and at the Crops Research Institute (CSIR-CRI) to develop appropriate production and pest management strategies and to evaluate new germplasm suitable for the region. Results from efforts at villages and research stations are presented to farmers using the Farmer Field School approach and appropriate posters, bulletins and manuals. Graduate student training is closely linked to activities in villages and research stations. Results from the project are providing farmers in Ghana with information on documented interventions that reduce aflatoxin contamination of peanuts throughout the Page | 14 supply chain. Improved productivity and quality of peanut coupled with acceptable levels of aflatoxin in peanut products improve access to local, regional, national and international markets leading to enhanced economic viability of farmers and their communities. Findings: Peanut research, through the Peanut CRSP and then PMIL has been active in Ghana for a long time. Many institutions from the US and Ghana are involved in the project which has six major objectives. The following objectives are implemented in Ghana: Evaluation of on-farm interventions during crop production, drying, storage and processing; pre and post-harvest technologies to reduce aflatoxin; evaluation of new germplasm from different institutions; technology dissemination; economic analyses of aflatoxin reduction; and survey of aflatoxin contamination. Several institutions including CSIR-SARI, CSIR-CRI, Kwame Nkrumah University of Science and Technology (KNUST), University of Ghana, Ministry of Food & Agriculture (MOFA), North Carolina State University (NCSU), Virginia Tech and UGA are involved in the implementation of the project. IPM strategies are developed by CSIR-SARI and CSIR-CRI in collaboration with NCSU. The scientists reported that the project includes socio-economic studies and data was collected in 2015, but not yet analyzed. In reality the baseline surveys should have been carried out even before the start of the project so that the results could be incorporated into the other projects. The EET noted that the project title includes “marketing of peanut” in the title, but did not observe any project work related to marketing. When the EET asked the farmers about marketing options the response was that nothing had been proposed. The EET visited a research location in Ejura-Nkwanta village where the research was focusing on good agricultural practices (Alata soap treatment, oyster powder applications and two weedings) and aflatoxin management. The results were compared to traditional famer practices. In term of yield performance, the improved technology produced more than farmers’ practice. At the time of the EET visit they were still waiting for the results of aflatoxin tests from the 2015 harvest (seven months after submission). The inability to get timely test results is a major constraint for maintaining timely and relevant research progress. The results showed that the practice of drying peanuts on tarps reduced the incidence of aflatoxin by 70-100%. There was approximately 0.61-24.96 ppb when dried on the ground. While this reduction in aflatoxin levels is encouraging there is still little to no understanding of why the farmers are not using the tarps, especially when the knowledge has been around for over eight years. The breeding program is using participatory varietal selection and breeder seed production. There is a need to look at the entire process of production and utilization. New breeding lines are being tested and a number of lines from the past breeding efforts under the Peanut CRSP have been released. However, there is no data available on the uptake of these varieties in Ghana. One of the stakeholders of PMIL told us that this is a major issue in Ghana as the seed (these varieties as well as other improved varieties) are not available for farmers so the impact of these varieties cannot be assessed. Page | 15 The EET visited PMIL activities in villages throughout the country of Ghana. They found: farmers were aware of PMIL; farmers had learned about aflatoxin control from PMIL; and farmers’ yields had increased by 70-100% and families were better off with the resulting increase in food for family consumption and income from selling peanuts. A next step is better storage so farmers can maintain high quality peanuts and sell later in the year when market prices are higher. It would be important to coordinate this work with economists to ensure appropriate marketing strategies are part of this. The farmers expressed interest in testing new technologies, particularly new varieties. Farmers were also interested in using triple bag technology but they have difficulties getting PICS bags for storage as the bags are not always available in the market. While the EET supports the use of on-farm demonstrations there are a couple of points that concerned the EET. The EET was disappointed with the quality of the field demonstrations. The fields were not well prepared and there was a lot of irregularity in the fields. A demonstration plot for farmers should be of high quality and appealing. The project has also developed a solar dryer that can contribute to maintaining high quality peanuts with low levels of aflatoxin with current dryers having a capacity of 125-200 kg (using 4 or 5 racks). Since it takes about four days to reduce moisture to below 10%, for a farmer with one solar dryer, drying one ton of harvested peanuts will take 20 days, a time frame that is too long to ensure quality peanuts. While the EET observed that the PMIL team is still evaluating the size and capacity of the solar dryer, the true cost associated with using the solar dryer needs to be thoroughly evaluated in order to appropriately determine its potential. A survey of post-harvest aflatoxin contamination in peanut and peanut products is being carried out from the University of Ghana in Accra. This is the only activity based in Accra. The main focus of the program is oil processing to identify levels of aflatoxin. The project is supported by the activities of two students. The EET noted with interest that there is a bottleneck with respect to aflatoxin testing in Ghana. The scientist in charge of aflatoxin analyses explained that he is getting samples from all the project partners in Ghana and the lab is overwhelmed. He was using high performance liquid chromatography (HPLC) and also the mobile assay but still does not have sufficient capacity to process all of the samples for the program in Ghana. The EET noted that progress towards the research objectives for PMIL is not being met because of this hold up with aflatoxin testing. The EET observed a lack of understanding of how a value chain approach would be implemented and the benefits that would result from that. A number of scientists did not know what this approach is about. The only time during the EET visit to different research organizations in Ghana that the EET found a good understanding of the value chain approach was the Director of CRI. She has excellent knowledge and would be very useful for PMIL training in Ghana such as a day seminar to the scientists from different organizations involved in PMIL. She also has as very good understanding of innovation platforms, something that PMIL could benefit from participation in. The EET received copies of three articles published in peer review journals of good quality. The scientists mentioned that they get good support from the PI for publications. Page | 16 Response to SOW Questions The research in Ghana incorporates both components of peanut production and aflatoxin managements. The scientists did not understand how a value chain approach would be implemented and the benefits that would result from that. In some areas there is good quality research. Three journal articles have been published in good quality peer-review journals. A major challenge in the context of Ghana is to bring everyone on the same level of understanding of the value chain approach. A number of technologies such as disease resistant lines are being developed that are going to be of use by both US and Ghana farmers. A number of students both from the US and Ghana have been trained and are obtaining their degrees from US universities. Recommendations i. Implement a decentralized aflatoxin testing system for Ghana. Specific actions that will aid this include: a) In addition to mobile assay and HPLC, provide the ELISA method to assist in getting a larger number of samples tested in a short period of time. KNUST would benefit from collaboration with ICRISAT in having access to ELISA developed by ICRISAT at a lower price than commercially available ELISA Kits. b) Train more staff on detection of aflatoxin. Training on aflatoxin sampling and testing should be provided to staff in CSIR-SARI and CSIR-CRI. c) Decentralize the detection work. Detection facilities could also be housed at CSIR-SARI in Tamale. d) Better equip the University for Development Studies (UDS) with detection facilities and more involved in some of research activities e) Distribute the workload associated with aflatoxin testing by conducting the grinding and sample preparation locally in each location before sending to the central location for aflatoxin extraction and testing. f) Add a testing lab in Tamale using ELISA and mobile assay so there is a testing location in the northern area of the country. ii. Implement a program to train and build awareness about the value chain approach to research for PMIL researchers and partners. iii. Design and implement, in close collaboration with the economists, appropriate marketing strategies for peanuts grown by small holder farmers in Ghana either through collaboration with partners (if available) or using PMIL funds. iv. Seek out and secure an engagement for PMIL scientists with innovation platforms in each region. v. Carry out an impact assessment in Ghana to look at the achievements and impact of many years of investment in this country. C3. Producer and Consumer Interventions to Decrease Peanut Mycotoxin Risk in Ghana This project focuses on technological and market interventions that can mitigate aflatoxin in peanuts in northern Ghana and investigates the relative and combined impacts of the interventions. The researchers worked with local experts to identify simple low-cost technological preventative measures that had the best potential for long-term and affordable Page | 17 solutions to aflatoxin reduction. They also examined market interventions that ensure a premium for low aflatoxin levels. They worked with local peanut buyers to offer a premium for low-level aflatoxin peanuts. A randomized control trial (RCT) methodology was used where farmers were randomly selected and assigned to different interventions. Producers who were selected to receive the market intervention were made aware of potential customers for low￾level aflatoxin peanuts and the required standards to receive the premium. Producers selected for the drying on tarps intervention were provided with training on how to effectively dry. Drying on racks was also incorporated into the first year of the project, but that method was found to not be effective as farmers had multiple layers of peanuts on a given rack and when the rains came the resulting mass of wet peanuts was not good. Gender differences were specifically incorporated into the research design. Gender, individual assets and joint asset ownership were built into the baseline study. This enabled the researchers to capture gender dynamics related to adoption of interventions to reduce aflatoxins in peanuts. Their methodology was structured to include an initial baseline survey, a mid-project survey and an endline survey. Over 1000 farmers were sampled in the baseline survey in late 2014 and early 2015. In 2015 tarps were procured (for drying on tarps as the technology intervention). Peanut samples were tested for aflatoxin at each stage, as part of the research protocol. The environmental conditions were such that low aflatoxin levels existed the year of the endline survey. Thus the researchers did not have all of the data they were expecting, but given they had conducted the mid project survey incorporating the gender relevant variables they were able to conduct important analysis and obtain relevant results. The researchers found that farmers are selling higher quality (lower aflatoxin level) peanuts and keeping the poorer quality peanuts (those with higher aflatoxin levels) for family consumption. Women have less access to agricultural inputs compared to men, due to financial and cash flow constraints. Peanuts have relatively lower input costs so work well for women. In spite of these lower input costs, women still have a lack of access to inputs. They observed that women were more likely to purchase the tarps, which were sold at a subsidized price, than men. Response to SOW Questions This research deliberately and effectively incorporates aflatoxin and peanuts examining how technological and market interventions can influence farmer behavior in the peanut value chain. The use of RCT is the appropriate current methodology for this sort of research in applied economics. The researchers have been able to present the research at conferences and are preparing papers for journal articles. Given the timing of the receipt of the funds, surveys and data collection this research is on track. These are all indicators of excellent quality research. The main challenge faced was the low aflatoxin levels, resulting from the unusual good conditions, with the endline survey. The researchers did get other excellent information that they have used to complete appropriate analysis. This research has focused on Ghana, with no Page | 18 obvious direct connections to the US peanut industry. This project involved training of two MPhil students at UDS Recommendations i. The projects in Ghana need to be coordinated. Currently this economics component is a separate project from an administrative perspective. It is important for the project. scientists to better coordinate with each other, share results (preliminary and final) and adapt on-going research in light of the results obtained. ii. Develop a true value chain project in Ghana. As noted in the section above, this is lacking. Better connections among the economists and other scientists working in Ghana is an important first step towards achieving a true value chain project in Ghana. This would then be followed with better integration of the results from the economics research. C4. Aflatoxin Management Interventions, Education and Analysis at Various Steps along the Peanut Value Chain in Malawi, Mozambique and Zambia This project is designed to address a number of issues of peanut production, post￾harvest handling, and processing issues in peanuts in Malawi, Mozambique and Zambia. The objective is to alleviate the constraints associated with these issues, which in turn will decrease aflatoxin contamination levels and also result in higher yields and increased profits for farmers. In Malawi, Zambia and particularly in Mozambique the implementation of research activities was delayed by a year mainly because of contracting issues, and lack of PI nomination. Therefore funds arrived late and in some locations funds did not arrive until late 2015. This project has too many components, which has created coordination challenges. The EET visited the University of Zambia and met the co-PIs and several of the students. The EET was shown the research work on various aspects of aflatoxin contamination management. The EET was encouraged to observe that there is a lot of cooperation between the University of Zambia (UNZA), ZARI at Msekera Research Station in Chipata and ICRISAT Malawi. UNZA is focused on pre- and post-harvest research activities and examining soil amendments, processing technologies and drying procedures to reduce aflatoxin contamination. The research is conducted by students as part of their training at the undergraduate and graduate levels. The ZARI Msekera Research Station is engaged in activities related to the development of peanut varieties with improved yields, drought tolerance, pest resistance and value-added traits. Research and demonstrations on agronomic practices is conducted in collaboration with the extension and NGOs. It was important to note that ZARI has established a laboratory for detecting and testing aflatoxin presence, which is accredited with the Zambia Bureau of Standards and the Botswana Bureau of Standards as collaborators. HPLC, ELISA and the Rapid Strip Tests with the tablet are used to detect aflatoxin levels. ZARI has close and positive collaborations with Comaco and Eastern Province Farmers Cooperatives. These two organizations process peanuts and have been progressive in producing products with low Page | 19 aflatoxin levels. The linkages between ZARI-Msekera Research Station and the peanut processing companies who are working to lower aflatoxin levels in food products represents an important move toward a value chain focused research program. The visit to Lilongwe University of Agriculture and Natural Resources (LUANAR), department of Food Science, Bunda Campus by the EET revealed further activities on both pre￾harvest and post-harvest projects some in conjunction with ICRISAT. LUANAR is evaluating peanut butter, peanut flour standards, shelf life and storage. There are currently no standards for peanut flour – thus they have made contact with the Malawi Standards Office and hope the research results can aid as future standards are formed. The other activities are focusing on the effect of drought, pests and diseases on aflatoxin levels. Preliminary results indicate that when plants undergo more stress there are increased levels of aflatoxin. Similar work on crop rotation found that when peanuts follow sorghum there are lower levels of aflatoxin. One of the objectives of this research component is to find alternative uses for the “grade-out” peanuts so that farmers will be willing to sort out the “bad” peanuts. The research is examining different conditions for processing. The hope is that they can produce oil that will be pure enough to have no aflatoxin. Then a high value consumer product can be achieved from the “grade-out” product. The EET inquired about the current price of “grade-out” and “regular” peanuts in the market. There is reported a relatively small difference in market prices for peanuts that are “grade-out” (a price of 250-300MWK) and the regular peanuts (a price of 400MWK). In other words, the market is not providing incentives for business decision makers along the value chain to take actions to reduce aflatoxin levels. The EET visited ICRISAT and discussed the collaborative work with PMIL. It was noted that PMIL (and Peanut CRSP) has a big emphasis on training students so the partnerships with LUANAR and UNZA are a good fit. During the visit to Malawi, the EET was able to link to the private sector partner, Exagris. Exagris is a multifaceted organization that operates a commercial farm. They produce seed that is sold to small-holder farmers. They purchase peanuts from small-holder farmers and make processed products (peanut butter, oil and high nutritious bars for UNICEF (RUTF)). The organization is closely working with LUANAR (Bunda) in conducting research trials as part of PMIL. It is a good example of a value chain perspective for PMIL in Malawi. Response to SOW Questions This project is an example of multi-stakeholder involvement and involves ICRISAT Malawi, the University of Zambia, Lilongwe University of Agriculture and Natural Resources (LUANAR) in Malawi, and Eduardo Mondlane University and Instituto de Investigacao Agraria de Mozambique (IIAM) in Mozambique. It also involves a number of partners such as Exagris and NASFAM in Malawi. The emphasis of the project is production, post-harvest handling and processing as part of the value chain steps. The sub-contracts were not awarded in time and therefore delayed the commencement of the project. The output of this research will benefit the peanut industry in the US as well as in the host countries. Capacity building is an important aspect of the project. The co-PIs have engaged students in the project working on various aspects of pre and post-harvest awareness campaigns. A PhD student at the University of Zambia is developing risk indices following the Auburn University model for peanuts and other pre- and post-harvest aflatoxin mitigation measures. Page | 20 Recommendations i. Support the researchers to continue this important research and encourage the collaborations to continue and expand whenever possible. ii. Assist the researchers from the project to communicate and exchange research results with the economists from C5. C5. Productivity and Profitability Growth in Peanut Production: A Farm Level Analysis in Malawi, Mozambique and Zambia This project provides important and essential cost of production and profitability analysis. This project utilizes well-proven production economics methodology to provide essential information on productivity and profitability, which is essential for the researchers at other stages of the value chain. The objectives for this project are: examine the costs and benefits of interventions designed to decrease aflatoxin and improve productivity and profits; analyze improved peanut varieties and farm productivity in Malawi, Mozambique and Zambia; and conduct training activities in the three HCs focusing on the economics of peanut production. The researchers worked with researchers from other components to get plot-based observations from which they could conduct cost of production and profitability analysis. They obtained 96 plot-based observations from two locations in Malawi. In Mozambique, they obtained plot-based observations for two research stations and five farm plot locations. They have not received data from Zambia. Their research analysis has enabled them to show a technological gap with significantly improved productivity from using improved varieties of peanut seed. They also show that the technology gap is actually relatively low and a significant increase in productivity could occur by lessening the management gap. To achieve the third objective the researchers shared with the EET they had a half-day training session planned for June 2016. They also had several additional surveys planned for the remainder of this year and next year. These will all contribute to achieving the project objectives. Response to SOW Questions The research links aflatoxin and peanuts as noted in the first objective. The research here is very appropriate for the needs of the overall project in the countries involved. The researchers seem to be working quite well with researchers from the other projects to obtain necessary data. Their analysis is appropriate economic analysis. They have presented results at professional conferences and are preparing manuscripts for journal submission, which are expected to result in publication. Work to through June 2016 related to this project, and two other projects within PMIL, has resulted in four presentations at professional meetings, three journal articles and one working paper. Based on EET visits, it was not evident that the results of the economics research are getting back to project in ways that provide insight to the research. The noted lack of data from Zambia is one challenge this project faced. The EET did not hear how this was addressed. It is not clear that there is any link of this research for US peanut Page | 21 production. The researchers reported that a PhD candidate in agricultural and resource economics at the University of Connecticut was being trained as part of PMIL. Additional capacity building also resulted from the training in the HCs. Recommendations i. Maintain this important research that is on track, conducting appropriate research that is important research to complement the breeding and agronomic research. ii. Incorporate the training of a host country agricultural economist who will add to the scientific capacity in the region after graduation. PMIL Publication Outputs The EET evaluated research using a variety of approaches including visiting labs and field plots, discussing research objectives and ongoing results with PMIL scientists, review of reports and one-on-one discussions with scientists directly involved in PMIL and others associated with PMIL. Given that refereed journal articles are an important measure of research quality the EET collected and analyzed journal article publication data. The EET acknowledges that there are a number of challenges associated with using journal article publications with the most significant being that there is a time lag between the completion of quality research and journal article publications. Data were provided from three main sources, the PMIL Management Entity, PMIL Annual Reports (2014 and 2015), and individual researchers during or after interviews. Basic analysis was undertaken to count the number of publications and assess the quality and relevance of journal outlets in which PMIL publications have occurred. Given that data collection stopped in July 2016, publications reported in 2016 are incomplete. Additionally, it is important to note that research is a continuous process and that many of the research trajectories existed prior to the start of PMIL. As a result, it is difficult to attribute specific publications to PMIL, especially those published at the early stages of the project. Nevertheless, the publications presented in this analysis were confirmed to be produced under and attributable to PMIL. PMIL researchers have produced a total of 41 publications across the four years of the project: five in 2013, sixteen in 2014, ten in 2015 and ten in 2016. More publication were produced in 2014 than in other years, although it appears that publication will likely increase in 2016 given that ten have been reported in the first half of the year. Overall, publication outputs are moderate given the project’s size and that the data reflect only the midpoint of the research. Interviewees indicated that data are still being collected and that publications are planned for the future. In addition to the quantity of articles, it is also relevant to examine the quality and relevance of the publication outlets. To assess journal quality, the EET examined the Journal Impact Factor for each of the journals in which PMIL researchers have published so far.1 Although many of the articles produced by PMIL are not included in the Web of Science (WoS), 1 Definitions of the three indicators (http://admin-apps.webofknowledge.com/JCR/help/h_index.htm): Journal Impact Factor is the average number of times articles from the journal published in the past two years have been cited in the Journal Citation Report (JCR) year. The Impact Factor is calculated by dividing the number of citations in the JCR year by the total number of articles published in the two previous years. Page | 22 available statistics show that many of the journals have an impact factor of near or greater than two.2 This indicates that PMIL researchers are publishing in reasonably good quality journals as outlets. In addition to assessing quality, it is also important to assess relevance of journal outlets within the research for development framework. From this perspective, we see that many of the journals selected by PMIL researchers are African regional journals. This indicates that PMIL researchers are doing a good job considering a variety of outlets and recognizing that regional outlets are as or perhaps more important for some types of publication outputs. It is also noteworthy that PMIL is publishing journal articles in many different subject categories, including those that are considered ‘multidisciplinary’. The breadth of the research undertaken by PMIL is producing published output that contributes knowledge to numerous disciplines and in some cases is relevant across disciplines. Finally, based on the list of non-journal article publications, abstracts, media reports and other outputs, PMIL is very actively producing products and outputs important for communicating the research efforts. In sum, although publication quantities are only moderate for this stage of the project, PMIL research is being recognized in the literature and PMIL researchers are publishing in good journals and in journals relevant for research for development. PROGRAM MANAGEMENT Findings PMIL began in 2013, building on many aspects of the previous Peanut Collaborative Research Support Project (CRSP). It is a multidisciplinary and multi-organizational project with a goal to “increase the productivity and profitability of peanut production for smallholder farmers and to reduce the negative impacts of mycotoxin contamination along the value chain of peanut and other crops in five Feed the Future countries – Haiti, Ghana, Malawi, Mozambique and Zambia.” (2015 PMIL Annual Report) After finalizing the PMIL at the University of Georgia (UGA), a search was conducted for the Project Director and David Hoisington was hired for that position. Then a process of soliciting, reviewing, selecting and awarding the projects that make up PMIL was carried out. Most of the activities of PMIL started in 2014. The Management Entity (ME) of PMIL is located at the UGA on the main campus in Athens, Georgia. David Hoisington serves as the Project Director and James Rhoads serves as the Assistant Director. The ME office is also comprised of a Business Manager, a Communications Coordinator, a Web Developer/Computer Support and an Administrative Specialist. As evidence of the integration of the PMIL project in the UGA, the Director and Assistant Director have research faculty positions in the Crop and Soil Sciences Department in UGA’s College of Agriculture and Environmental Sciences. The Web Developer position is part of the 2 According to one report, approximately 34% of all journals in Web of Science (WoS) have an impact factor greater than one, while about 18% of all journals in WoS have an impact factor greater than two (http://mdanderson.libanswers.com/faq/26159). Page | 23 Office of Information Technology group, but the individual’s salary is paid by PMIL and office space is located with the PMIL offices. The Administrative Specialist is shared with the College of Agriculture and Environmental Sciences’ Office of Global Programs so is thus a part-time position in PMIL. Hoisington and Rhoads have different disciplinary backgrounds (Hoisington – biotechnology and Rhoads – anthropology). These disciplinary differences are helpful enabling the management team to understand the projects from different perspectives. The PMIL office is set up as part of the College of Agriculture and Environmental Sciences and housed in the same area as the College’s Office of Global Programs. The PMIL Director reports directly to the Dean of the College. The other members of the PMIL ME report to the PMIL Director. The External Advisory Panel (EAP) is an important part of the PMIL organization. This group of nine individuals comes from the U.S. and host countries, representing industry, universities in the United States, the Bill and Melinda Gates Foundation and other research organizations. The EAP meets annually, in conjunction with the PMIL annual meeting. They reviewed proposals at the start and during the buildup of PMIL. They also review annual and progress reports. The External Evaluation Team (EET) interviewed several members of the EAP by phone or Skype. The EAP members were very positive about PMIL and the ME. The EAP members did have some suggestions that would improve PMIL. They recommended that there be increased transparency in all aspects of the proposal solicitation, review and award. They also suggested that that PMIL would benefit from increased integration of the projects across disciplines and more of a value chain perspective for PMIL. PMIL is structured around 12 research projects as shown in the Evaluation Plan (Table 1 lists the projects and Figure 1 shows them in organizational chart format). These projects, along with the scientists and associated institutions, were determined through a combination of commissioned and competitive proposals and evaluation. The 12 projects fall into three areas, grouped as A, B and C projects. The A projects are under the heading of Peanut Germplasm Development; the B projects under the heading of Mycotoxin Detection and Peanut Nutritional Studies; and the C projects under the heading of Peanut Value Chain Interventions. Crop production research projects are incorporated into the third grouping. Each of the 12 projects has a Principal Investigator (PI) from a US university. Co-Principal Investigators (co-PIs) and Partners from other US Universities, CGIAR centers, Universities and Research Services in Partner Countries, as well as private sector. In general, funding flows from USAID to UGA who then establish sub-contracts for each of the 12 projects. The PI and sub-contract is with another U.S. University. Each of the 12 projects has co-PIs and often partners. Funding for co-PIs and partners generally flows through to their respective organizations. In some cases there were several levels of sub-contracts (e.g. UGA to NCSU; NCSU to Virginia Tech; Virginia Tech to ICRISAT; ICRISAT to University of Zambia). The PMIL Director and Associate Director, as well as the co-PIs in the partner countries realized that there were several problems associated with this contract/sub-contract model. Two important factors Page | 24 related to sub-contracts are timely flow of funds and ability to use the funds for project expenses. Timely flow of funding is essential for any project, but especially the case for those that involve planting and growing peanut crops. With one crop per year, if funding is not available to purchase supplies and pay staff for planting, the research is delayed by a whole year. This issue proved to be important for PMIL, as in some cases the time it took for each sub-contract to get established resulted in a situation where sub-contracts were not yet established in time for critical research activities. Having several layers of sub-contracts can result in these inefficiencies. The observations of a co-PI in Ghana illustrate this point. The researcher noted that at the early stage of the project she supported the activity of the 1st student through funds from another project but the second student’s research has been postponed. The PI was informed but the solution was still not found. However, the program has finally received the funds. The PI of Project B1 described the challenges he was encountering as he tried to engage in collaboration with his colleague from the International Institute of Tropical Agriculture (IITA) in Nigeria due to delays in setting up sub-contracts and then having systems in place so the money could be spent on research. The second challenge, related to the ability to use the funds, can be linked to the type of sub-contract that is in place. The typical way that the sub-contracts were set up was “cost reimbursable.” Cost reimbursable contracts require the receiving institution to pay for the research expenses (to purchase inputs, hire labor, etc.) and then submit receipts for reimbursement. This often does not work well for host country universities and research institutions. These organizations are small and do not have the resources to fund expenditures and wait for reimbursement. Furthermore, when exchange rates are volatile (and if the move in the exchange rate is against the host country currency) the university or research organization can incur a “loss” associated with the exchange rate change. The ME, led by the Director and Assistant Director, sought a solution for the problems of getting the financial resources to the partner country researchers in a timely manner. They worked with the sponsored programs/contracts office at the UGA and determined that they could use a fixed price contract between the UGA and the partner country organizations. Thus, the review team observed that there are, in essence, two models for PMIL contract with host country institutions. With US universities all of the sub-contracts are cost reimbursable contracts. Some of the sub contracts (and the ones in Ghana fall into this category) are cost reimbursable contracts. Those are from the universities other than the UGA. These are structured so that the scientists purchase the supplies (or hire the people) and then submit the receipts for reimbursement. As we have heard many times (both at UGA and host country site visits) cost reimbursable contracts often do not work well for host country institutions because they do not have funds to float the expenditures and then get reimbursed. In some cases an advance is provided to the host country institution – which often helps somewhat. These contracts are annual contracts – so from the perspective of the scientist it functions as though they spend money, submit receipts and receive additional money to spend. Reporting is annual Page | 25 and it would be expected that if the scientist in the host country was not performing then there would be changes (or no new contract) the subsequent year. The other type of contract is the fixed price contract. These were set up by the ME at UGA directly with host country institutions and are found in Malawi, Mozambique and Zambia. The ME set these up directly when they learned that the cost reimbursable contracts were not working and they needed to do something to enable the projects to proceed. The PMIL director indicated that they are now visiting with the US universities to expand the use of fixed price contacts if there is a next round of PMIL. The fixed price contracts are annual contracts. At the start, one-quarter of the funds are transferred to the host country institution. At the end of the quarter, the scientist prepares a progress report that is sent to the ME (where the PMIL Business Manager reviews it and forwards it to the associated PI). After the report is approved by the PI, the PMIL Business Manager forwards the next quarter of funds to the host country partner. These quarterly reports are usually in bullet point format and focus on whether the activities that were in the SOW have been completed. The fixed price contracts are working well and can be the preferred way to structure contracts with host country institutions. The PIs in the US can focus on research activities and not have to worry about setting up sub-contracts, since the ME takes care of those details. Many of the delays in getting contracts set up are avoided, and the host country researchers are able to spend the money and conduct the research. To further evaluate the question of whether PMIL is efficiently and effectively using its funding to meet its research objectives, current spending was compared against the budgeted amounts for the first three years on a project by project basis. With the exception of one closed out project that spent virtually all of the budgeted funds, all of the other projects are significantly behind on expenditures for this period. Thus, to date, one-quarter of the projects have spent 70 percent or more of the available funds, whereas the remaining projects have only spent between 55 and 70 percent of available funds to yield an average overall underspending of 35 percent during the first three years of PMIL. This finding is consistent with messages received from PMIL researchers. The EET heard comments of frustration from researchers concerning their inability to conduct the research due to problems with getting access to the funds. Similar messages of concern are noted in PMIL annual reports. The Director and Associate Director of PMIL have fostered synergies within the UGA as well as outside of the university. Representatives from the UGA’s College of Agricultural and Environmental Sciences specifically noted the positive synergies that the college was experiencing as a result of having PMIL as an integral part of the college and physically located adjacent to the Office of Global Programs. This integration comes in the form of joint interactions when hosting foreign visitors as well as intellectual collaboration on projects. The PMIL research projects at UGA are an important part of the college’s research portfolio. It is important to note that the UGA research projects are independent from the PMIL ME. The external evaluation team interviewed several administrators from UGA College of Agricultural and Environmental Sciences as well as staff from the university sponsored programs. All individuals noted the positive interactions with PMIL and how PMIL brought value to the university. Page | 26 The Executive Director of the Peanut Foundation (a U.S. peanut industry organization) highlighted the important synergies between PMIL and the Peanut Foundation. He noted that he has regular communication with the PMIL Director enabling them to identify where the two organizations have common goals and that they were exploring ways to fund projects jointly, in areas where the goals are common. Effective communication among PMIL researchers and across the PMIL projects is in part a responsibility of the ME. The ME utilizes a regular newsletter as well as the annual meetings to achieve the communication goals. The PI’s, co-PI’s and partners consistently reported that the annual meetings were serving as a good venue to both tell the other researchers about their projects and also learn about the other research projects. Members of the EAP also attend the annual meetings and reported that the annual meetings were a useful venue for good communication across PMIL. The EET did hear one suggestion to further enhance communication across PMIL. The PI noted that he had previously been involved in an end-of￾project review, which was organized by the ME, and involved a small team of scientists. This was found to be very helpful and he suggested that the ME could organize more of these reviews to facilitate greater interaction amongst the researchers. The EET did hear a request from researchers in some of the host countries, when they noted that if they could get access to some on-line journals it would greatly facilitate their research. The cross-cutting issues of gender, climate change, capacity building and nutrition and health are part of the set of objectives for PMIL. Given that these are cross-cutting issues it is important for the ME to be proactive in working toward these objectives, as without the central agency these will fall “between the cracks.” With the exception of the research undertaken by economists in Ghana and Haiti, gender is not directly integrated into PMIL. Similarly the EET did not find climate change as a priority that PMIL researchers were paying significant attention. Capacity building, with an emphasis on human capacity and degree programming, was an important part of many of the PMIL projects. Nutrition and Health is receiving some attention in PMIL, albeit with small projects where there could be a more direct linkage with the rest of the PMIL projects. Conclusions The move from the Peanut CRSP to PMIL has been a big change and with it an associated big transition. The individual serving as interim director during the transition is now Executive Director of the Peanut Foundation. The EET heard from several people that the changes have been very positive, the Interim Director did a good job in the transition and the current ME of PMIL received nothing but positive comments from the people the EET interviewed in the course of this review. The stated focus of PMIL is a value chain approach with programmatic themes of: peanut production, aflatoxin management, seed production, post-harvest handling and processing and production development and nutritional benefits. Title XII of the International Development and Food Assistance Act of 1975, which authorized USAID to engage US land grant and other eligible universities to address the needs of developing nations while also contributing to US food security and agricultural development, is also relevant here. As noted at the beginning of this section of the report, the current goal statement for PMIL focuses on host countries, with no mention of benefit to the U.S. peanut industry. Page | 27 Response to SOW Questions The first question in the SOW relates to how effectively the ME manages the research and training activities of PMIL along with what opportunities there are for improvement. The EET found that the ME is doing a good job of managing the research and training activities. PIs and co-PIs regularly provided praise for the work of the ME and were pleased with the support that the ME provided for them. The main area for improvement has been noted above and relates to the timely implementation of sub-contracts with host country partners that facilitate spending the money and conducting good research. While the ME was proactive in seeking out the fixed price contracts for some of the host country partners, this represents and area for additional change to benefit all host country partners. The second question in the SOW relates to the role of the EAP. The EET found that the roster of the EAP is made up of individuals with great expertise, experience and insight for PMIL. The main venue where input from the EAP is received is the annual meeting. The EET found that the members of the EAP are attending the annual meetings and playing important and positive roles. That said, the EET found that further insights could be gained from the EAP, for example, the EAP could take a proactive role in all aspects of the project proposals (design of call for proposals, review of proposals, review of progress). We heard directly from the EAP members that they felt that greater transparency in all of these aspects was important. Lessons Learned A number of “lessons learned” from this review of PMIL are noted:  Multiple contracts and sub-contracts to several institutions is very inefficient and often results in a delay in the research or an inability to undertake the research.  Cost reimbursable contracts are generally not effective for host country partner organizations. Fixed price contracts have been much more efficient from the perspective of the scientists and the ME.  Value chain projects in the third (C Projects) component of PMIL have Value Chain as part of their titles. The EET found, however, that value chain was part of the title but often not part of the project. These projects were really a set of projects addressing different nodes along the peanut value chain with little to no evidence of interaction amongst the scientists from the different projects. It seemed evident to the EET that the scientists do not understand what a value chain approach should and can be.  Gender: The evaluators asked researchers how they were incorporating gender into the projects. While the scientists (especially in the partner countries) often reported that peanuts are a women’s crop their follow-on responses noted the number of female students and co-PIs they had on the project. They were not designing and developing the research to consider the role of women and men in each of the aspects of peanut production, marketing and processing. Many of the scientists do not seem to understand how gender can be integrated into a multidisciplinary research project such as this. Some of the economics projects were effectively including gender in the data collection and analysis.  Climate Change: The EET did not identify direct consideration of climate change in the design and development of any of the projects. Page | 28  Capacity Building: There was significant emphasis on training of students in PMIL. Many of the students are being trained at partner country universities, which is much less costly compared to training students at US universities. In several cases, scientists in partner countries reported that they needed to build their physical infrastructure in order to effectively carry out research (e.g. fix and expand greenhouse facilities; upgrade lab facilities – especially lab facilities dedicated for aflatoxin analysis and testing.)  Nutrition and Human Health: There is one project (B4) dealing with nutrition and health with a randomized controlled trial of the impact of treating pregnant women with a peanut-butter based product. Currently there is no link between this project and the other PMIL projects. Recommendations The EET has the following recommendations with respect to program management: - Formulate and adhere to the core focus. With limited resources available, greater outputs and impact will be achieved when the scarce resources are focused rather than spread out too thin to be useful. It is recommended that this be a “value chain approach for enhancing peanut production and aflatoxin management to increase incomes and human health” - Engage in training on value chain analysis for all involved, for example, consider evaluating other development projects that have successfully incorporated a value chain analysis. The PIs and co-PIs could benefit from training on how value chain analysis is implemented into a project and the outcomes that result for the benefit of the project and long term impact. - Incorporate training on how to effectively incorporate gender into all aspects of research. This could be incorporated into the annual meeting, or training opportunities between annual meetings. - Ensure there is no interruption in the flow of funding for research so that research can continue uninterrupted. The EET recommends that unused funds could be beneficially used in the following: - Develop and provide training, as noted above, in the areas of value chain and gender. - Enhance the physical facilities necessary for the completion of quality and timely research (e.g. fixing up and expanding greenhouse facilities; upgrading dedicated lab facilities, etc.). This is especially the case for partners in host countries (NARS and universities) PROGRAM FUTURE Program Management Recommendations The EET is impressed with the successes of PMIL to date, but also notes that with any program there are areas for improvement. Suggestions for specific changes are noted in the recommendations sections for each of the components of PMIL, noted above in the report. In looking to the future the EET makes the following recommendations: Page | 29 Core Mission Revisit the Core Mission of PMIL and Identify Implications for PMIL The EET recommends that the ME revisit, using a typical strategic planning process, the mission of PMIL. The current goal of PMIL is stated as “increase the productivity and profitability of peanut production for smallholder farmers and to reduce the negative impacts of mycotoxin contamination along the value chain of peanut and other crops in five Feed-the￾Future countries – Haiti, Ghana, Malawi, Mozambique and Zambia.” The PMIL ME should evaluate the extent to which they are addressing Title XII of the International Development and Food Assistance Act of 1975, which authorized USAID to engage U.S. land grant and other eligible universities to address the needs of developing nations while also contributing to U.S. food security and agricultural development.” Questions they should consider are: Do they want to change the goal/mission statement for PMIL? How broadly or narrowly do they want to interpret the goal/mission of PMIL that will serve as guidance for selecting projects for a future phase of PMIL. Project Design Structure Develop and Implement a Transparent Process for Project Design, Development and Delivery that leads to Quality and Impactful Programs that contribute to PMILs Core Mission The EET concludes that the PMIL project has reached a level of maturity such that it can now embrace a completely transparent process for project design, development and delivery. The result will be a set of projects and overall program that are of highest quality, achieve PMILs core mission and positively impact peanut farmers and others along the peanut value chain worldwide. Specifically: - there is greater involvement by the EAP in the design and establishment of the call for proposals, the review of the proposals and selection of project teams, in the annual review of the projects and in suggesting adaptations and changes during the life of the projects. - there is full transparency in the call for proposals. The call is widespread to attract the greatest number of submissions. As part of the call for proposals researchers are informed that proposals will be reviewed based on the criteria of: importance of the topic; review of the literature; the degree to which the work of the project fills an important gap in the literature; whether the proposed plan of work is achievable given time and budget; the quality of researchers; and the likelihood of achieving objectives. - a thorough review of the proposals, to be carried out by external individuals, using the above criteria, is conducted prior to final selection of the projects for funding. - all projects are continuously monitored for satisfactory progress annually and achievement of objectives by the end of each project. Page | 30 Sub-contract Management Further Improve on the Efficiency of Management of the Sub-contracts for PMIL projects The EET recommends that the ME take a greater role in the administration of the sub￾contracts that make up the PMIL projects. The resulting structure should involve: - projects with other US universities will be a sub contract between UGA and the other university. These contracts will be cost reimbursable contracts. - projects with HC universities and NARS will be fixed price contracts between UGA and the HC institution. This will: o ensure consistency and efficiency with sub-contracts with HC institutions. o enable the sub-contracts with HC institutions to be in place in a more timely manner. UGA has a template already developed for this. There will be no need to wait for the sub contract between UGA and the US university to be finalized before the sub contract with the HC institution to be initiated. o PIs at US universities are scientists and do not have a comparative advantage in administration of sub-contracts. Contract administration activities are better left to the ME. The project PIs can still have control over progress to research goals and quality of work through the process of approving quarterly reports. Host Country Scientists Increase the Influence and Participation of Host Country Scientists in PMIL The EET encourages projects to have HC scientists as PIs. There are many scientists in HC institutions that have the expertise and experience to serve this role. The EET recommends greater involvement of HC scientists on the EAP. These EAP members need to be in a position to represent the perspective of the NARS. The EET recommends there be one for Western Africa and one for Eastern and Southern Africa. Individuals from the regional organizations (e.g. CORAF in West Africa) would have the appropriate expertise and perspective for this role. Research Program Recommendations The EET recommends the following for the research program of PMIL: PMIL Project Organization Structure the Organization of the PMIL Projects to Promote Strong NARS in the HCs The NARS are the organizations in the HCs that provide the continuity over time and link research and technology transfer. When participating NARS are stronger, they are better able to fulfill their missions which include facilitating dissemination of seed, education for technology transfer and responding to emergency needs. This also benefits PMIL. Some specific actions that will serve to promote strong interaction between the NARS and CGIAR and HC universities and CGIAR include: - Paying a portion of the salary of co-PIs at NARS and HC universities, just as they sometimes pay salaries from co-PIs from some US scientists. These resources will Page | 31 enable scientists at HC NARS and universities to give resources to their institutions and “buy out” their time to devote more complete attention to the PMIL research. - Providing support to develop and improve the infrastructure at the NARS and HC universities (e.g. labs, greenhouses, vehicles, computing needs, access to on-line journals and other academic references). There is inherently a resource difference between the HC institutions and the CGIAR centers. The suggestions here would serve to alleviate some of the most pressing constraints for the scientists in the HC institutions to carry out good science. Seed Availability The EET identified a problem with a lack of availability of breeder seed and foundation seed. It is recommended that to increase activities in these areas as otherwise the results of the breeding work will not be able to be realized by the farmers and the investment in the breeding activity will have been for not. Gender Integration The EET recommends that the issue of gender be moved from that of a topic that is “glossed over” to one where gender is an integral part of the projects. It is not enough to simply say that peanuts are a “women’s crop” or to count the number of female PIs, co-PIs and students in PMIL. Gender needs to be integrated into PMIL projects. An integration of gender into the PMIL program will enable a full understanding of how women and men work with peanuts at all stages of the value chain. Gaining this understanding is the first step for research on gender and the output can then be incorporated into the research on breeding, agronomy, and marketing etc. Knowing how the preferences of women and men differ when it comes to farming practices can influence the traits that breeders select for and the best agronomic practices. This could be achieved by having a specific Women in Development (WID) project. The PI would be a gender specialist, e.g., someone from a Woman’s Study program with experience in development. It would be critical that one of the goals for that project be “integration and interaction” with the other PMIL projects. The EET recommends this over having gender integrated into each of the projects since the scientists in the other PMIL projects do not have expertise in gender and would not have the skills to carry it out. Training focus The EET recommends the continued focus on training. The EET does suggest that PMIL examine some trade-offs associated with training. In particular, the PMIL ME should evaluate the advantages and disadvantages of having degree training take place in host countries versus the US. Factors to consider include: - cost per student can be much lower in host country universities - there may be a higher rate of maintaining talent in the HC if students study locally, rather than when they study at US universities they are recruited by other US universities or international agencies - students that study at host country universities do not get the benefit of taking courses and interacting with some of the top ranked scientists at the US universities Page | 32 - explore the potential of students spending some time (perhaps a semester) at one of the US universities that is a partner in the associated project - explore the potential to have US PIs (and co-PIs) as members of the advisory and examining committees for the students studying in HC universities Technology Transfer Including Technology Transfer (Extension) as part of PMIL The EET recommends devotion of more resources to technology transfer. The PIs and the ME could deliberately seek out and link with other projects that are upscaling technologies so that PMIL technologies are adopted and the full impact of the research investment is experienced. Priority and Focus Examine Priorities and Focus on the Projects that are Core to PMIL Mission The EET recommends eliminating projects that are not part of the core set of host Countries (e.g. Burkina Faso, Senegal, Uganda, etc.). In these “non-core” countries PMIL provides very small amounts of money to a country or institute. For these, the investigators have to incur the costs of administering the sub contract, preparing reports with very little funds to carry out any science. The payoff is simply not there. Small Equipment Research on Small Equipment and Machinery for Peanut Production and Processing The EET recommends taking advantage of numerous opportunities that exist in projects that will combine women and small business development with new (or adapted) small equipment and machinery. Evidence of the fact that there is demand for these programs is illustrated by comments to the EET from farmers in Ghana who requested this. The outcome here could be successful business operations (some operating as individual proprietors and some from collective or cooperative organization) that are providing quality peanuts and peanut-based products to consumers in the market and also income to these business owners. Value Chain The need to have PMIL projects embrace a value chain approach has been delineated in several places in this report. Some of the PMIL researchers understand what is required for a VC approach, but most do not. The project in Haiti has achieved many of the benefits of a VC approach, but they could also benefit from greater interaction with the economists, sharing and incorporating ideas into the projects. The other projects could learn from this. PMIL should initiate training for PIs on VC approaches for projects. This training could have three components. The first component would involve bringing in outside experts on VC work to lead a workshop. The topics covered would include defining what a VC approach to research is; demonstrating why a VC approach to research is beneficial, showing how to implement a VC approach. The second component would involve hearing the stories from PMIL where parts of a VC approach are being implemented, such as in Haiti. The third component would involve participants working in their project groups to adapt their programs Page | 33 to be VC projects. Then they need to insist that new projects be true VC projects such that evaluation of projects reflects how well the VC approach is being implemented. Peanut Markets Enhance the Understanding the Peanut Markets The EET recommends that expanding the analysis from economists to include projects that evaluate the nature of the peanut markets to determine:  Annual market price patterns and the potential for gain if farmers store peanuts without loss of quality and sell later in the market season  Elasticities of demand to gain an understanding of how much peanut production can be expanded before there is an excess of peanuts resulting in lower prices. Examine the points in the markets where there exist incentives (higher prices) for decision makers (farmers, processors) to produce and trade low aflatoxin peanuts. There are currently very few places in the peanut markets where farmers have incentive to sell low aflatoxin peanuts in the market. Project C3 in Ghana is starting to collect appropriate data and conduct good analysis. It is important to build on this and ensure integration with the rest of PMIL Peanut Products Developing New Peanut Products to Grow Demand for Peanuts Develop new peanut products for consumer use in Host Countries and beyond that will expand the demand for peanuts and create the market for the additional volume of peanuts that will be produced as new varieties, technologies and production practices are implemented. Following the VC approach this component would involve considerable interaction with the breeders and agronomists with respect to traits and characteristics of peanuts as well as the economists with respect to consumer preferences. These interactions and this work would be iterative in nature. Initially the researchers would learn about the current peanut characteristics and consumer preferences and design value added products for consumers. The lessons learned would be shared with researchers all along the VC to influence on-going research. Thus, the breeders would adapt the set of traits they are breeding for as they learn about the requirements for processing peanuts. The economists would evaluate consumer preferences, and thus demand, for new value added peanut products and the information gained will influence ongoing research on new product development. The result of this iterative work is products that consumers demand and the peanut industry can expand. Page | 34 APPENDICES Page | 35 APPENDIX A. SCOPE OF WORK EXTERNAL EVALUATION OF THE FEED THE FUTURE INNOVATION LAB FOR COLLABORATIVE RESEARCH ON PEANUT PRODUCTIVITY AND MYCOTOXIN CONTROL: SCOPE OF WORK I. BACKGROUND INFORMATION A) Identifying Information 1. Project/Activity Title: Feed the Future Innovation Lab for Collaborative Research on Peanut Productivity & Mycotoxin Control (Peanut & Mycotoxin Innovation Lab) 2. Award Number: AID-ECG-A-00-07-00001 3. Award Dates: 3/31/2013 to 7/30/2017 (based on original award of 7/31/2007 as modified on 3/28/2013) 4. Project/Activity Funding: $26,000,000.00 (Total $39,865,000.00) 5. Implementing Organization: University of Georgia Research Foundation, Inc. 6. Project/Activity COR/AOR: Jennifer “Vern” Long B) Development Context 1. Opportunity Addressed by the Project/Activity being Evaluated General Background The Feed the Future (FTF) Food Security Research Strategy goals are to advance the productivity frontier, to transform key production systems and to enhance nutrition and food safety through agriculture to advance FTF’s overarching goal of sustainably reducing global poverty and hunger. The Feed the Future Innovation Labs with U.S. universities (the “Innovation Labs,” formerly called CRSPs) were created under Title XII of the International Development and Food Assistance Act of 1975, which authorized USAID to engage U.S. land grant and other eligible universities to address the needs of developing nations while also contributing to U.S. food security and agricultural development. In 2000, Title XII was reauthorized, enabling these U.S. university research efforts to continue “to achieve the mutual goals among nations of ensuring food security, human health, agricultural growth, trade expansion, and the wise and sustainable use of natural resources”. The launch of the Food Security Innovation Center in 2012 enables USAID to manage its research, policy and capacity-strengthening portfolio through the following seven thematic areas rather than by institutional home:  Program for Research on Climate Resilient Cereals  Program for Research on Legume Productivity  Program for Advanced Approaches to Combat Pests and Diseases Page | 36  Program for Research on Nutritious and Safe Foods  Program for Markets and Policy Research and Support  Program for Sustainable Intensification  Program for Human and Institutional Capacity Development The Innovation Labs funding of these themes is a central component to develop an overarching and coordinated strategy for engaging U.S. universities in agriculture and food security research and human and institutional capacity development and to leverage the impact of those investments by strengthening links across universities, U.S. government, global programs, foundations, and other donors. The Program for Research on Legume Productivity aims to increase the production and consumption of critical, protein-rich legumes, by developing disease and stress tolerant, high-yielding varieties, improve market linkages and post-harvest processing, and integrate legumes into major farming systems to improve household nutrition and incomes, especially for women. Description of the Peanut & Mycotoxin Innovation Lab The Feed the Future Innovation Lab for Collaborative Research on Peanut Productivity & Mycotoxin Control (Peanut & Mycotoxin Innovation Lab or “PMIL”) falls under the Program for Research on Legume Productivity and is the successor to the Peanut Collaborative Research Support Program (1982-2012). The present program was established in 2007 for five years and for an additional program period to end in 2017. The latter program period is the subject of this evaluation. Purpose The current program mission of PMIL is to apply innovative U.S. science to improve peanut production and use, raise nutrition awareness, and increase food safety in developing countries. Research & Administration The PMIL management entity (ME) is the University of Georgia Research Foundation and is administered by a program director, an assistant director and five administrative staff members. A 10-member External Advisory Panel composed of industry, government and academic experts aims to provide unbiased and independent advice on technical matters within the research portfolio and to identify further research directions and opportunities aligned with PMIL’s goals. The creation of PMIL coincided with significant administrative changes from the Peanut CRSP to implement and coordinate research programs which align with the FTF Food Security Research Strategy of 2012. These changes delayed many subaward launches. The Peanut & Mycotoxin Innovation Lab research focuses on peanut production, mycotoxin control, seed production, post-harvest handling and processing, market opportunities, gender, and instructional workshops and training. The twelve projects managed by PMIL can be broken down into four broad research areas that include aspects covering multiple focus areas.  Peanut Breeding and Production o These projects relate to genotyping and associating peanut molecular variation with resistance to pre-harvest aflatoxin contamination and using this information in breeding programs; developing new peanut varieties with improved yields, drought tolerance, pest resistance and value-added traits; and transforming peanut plants with RNAi to reduce aflatoxin in peanuts..  Peanut Agronomy, Production and Value Chain Interventions Page | 37 o These projects relate to reducing aflatoxin contamination of peanut in the supply chain by determining current pre- and post-harvest practices by peanut farmers, conducting research to mitigate aflatoxin and improve peanut quality, and transferring appropriate technology to farmers in Ghana; addressing and mitigating key constraints to peanut production and use in Haiti; and addressing production, post-harvest handling, and processing of peanuts in Malawi, Zambia and Mozambique with respect to aflatoxin contamination levels, yield and profitability.  Mycotoxin Detection Projects o These projects include development of rapid, low cost fluorescence-based detection methods for aflatoxin in maize; establishment and validation of methods to detect mycotoxin biomarkers in human dried blood and evaluation of the linkages between aflatoxin exposure and nutritional deficiencies; and a comparative study of analytical methods for detecting aflatoxin in peanuts and peanut products.  Economic and Nutrition Studies o These projects include investigating the relative and combined impact of technological and market aflatoxin mitigation interventions for groundnuts; determining the benefits of treating moderately malnourished pregnant women with a peanut butter-based nutritional supplement; and generating and transferring the economic knowledge needed to intensify groundnut production, and its subsequent use, to significantly increase productivity and farm profits, while reducing the risk of aflatoxin contamination 2. Target Areas and Groups Key U.S. Partners The PMIL projects are led by University of Connecticut; University of Georgia; University of Florida; Mississippi State University; North Carolina State University; USDA-ARS National Peanut Research Laboratory; Virginia Polytechnic Institute & State University; and Washington University, St. Louis. Geographic Reach The PMIL project countries are Ghana, Haiti, India, Kenya, Malawi, Mali, Mozambique, Senegal, Uganda and Zambia. C) Documents For this evaluation, the team will review a wide variety of documents provided by BFS, the ME and by the sub-award partners. Other documents may be requested during the fieldwork preparation phase, or while the evaluation team is in the field. The primary documents and types of documents that will be provided are listed below.  Technical Application proposal,  PMIL annual reports,  Subaward project briefs, reports and annual work plans  Program publications including success stories, research publications, HICD outputs, and impact briefs (with links to data),  Other PMIL website materials and information (http://pmil.caes.uga.edu/), and  External review report for first five years of the program (Peanut CRSP, 2007-2012).  Feed the Future Research Strategy Page | 38 II. EVALUATION RATIONALE A) Evaluation Purpose The purpose of this external performance evaluation of PMIL is to provide empirical evidence to respond to evaluation questions designed to support learning and continuous improvement for BFS’ work. The evaluation will assess progress toward outcomes, the quality of the research program and its outputs, what is and is not working well in implementation, including the effectiveness and contributions of the program management entity, and will provide information and recommendations that BFS can use to inform the development of new investments in peanut production and mycotoxin control research for development (from research directions to management styles), inform the development of new investments in such research for development, improve activity effectiveness and better achieve intended outcomes. B) Audience and Intended Uses These results are to be used by USAID/BFS/ARP-R to establish a future RFA (including program design) to address outstanding research and development questions connected to (i) peanut productivity for smallholders in developing countries and associated value chains and (ii) broad-based mycotoxin detection, prevention and control in foodstuffs and animal feed relevant to small holder farmers in developing countries (and not limited to aflatoxin in peanuts and maize). The evaluation results may also be applicable to others who are involved in designing research for development programs. Evaluation Questions Research Program 1. The PMIL research combines two thematic areas, peanut productivity and mycotoxin control. Program activities are organized around a value chain approach. How effective has this approach, combining productivity and mycotoxin control throughout the program, been in achieving research outcomes in both areas? In what ways did combining these two research themes in one program strengthen or weaken the focus on the most important questions in both of these research areas? Were these areas adequately addressed to justify this combination of research areas? Why or why not, and is it equally the case for both thematic areas? 2. To what extent did each project generate robust and quality research outputs using disciplinary￾appropriate metrics? Are the outputs relevant for a research-for-development project (e.g., did they generate new breeding lines of relevance to FTF countries or aflatoxin detection methods that respond to local context in FTF countries)? Were the projects undertaken using the right tools/technologies and were they well executed? 3. What research-related challenges has PMIL faced during research design and implementation? In what ways have these challenges been addressed? 4. Given the dual benefit mandate of Title XII authorized programs, such as PMIL, is there a sufficient balance between research efforts directed towards priorities of the domestic peanut and mycotoxin stakeholder community and the peanut and mycotoxin research priorities required to advance global food security goals, particularly in FTF countries? 5. To what extent has PMIL met its academic training and technical capacity strengthening targets? What improvements, if any, are needed in the ways: Page | 39 a. the program identifies and addresses academic and technical capacity needs? b. academic and technical capacity strengthening activities are implemented? c. people are targeted and selected for training? Program Structure & Management 1. How effectively has the PMIL ME managed research and training activities amongst sub￾awardees and stakeholders in the U.S., Latin America, Africa and Asia? And across the different thematic areas? In what ways has the ME supported coordination among the individual projects to ensure complementarity of effort and contribution to the PMIL’s objectives? For example, did the ME use the research outputs to shape the kind of outreach it undertook (e.g., was the ME the interlocutor of information generated among various projects and did the ME serve in its role of bringing information to people's attention as appropriate across the projects)? What opportunities are there to improve the coordination amongst the different partners and among the different thematic areas to better achieve outcomes? 2. What role has or could the External Advisory Panel play in contributing to program effectiveness and advice relating to broad research priorities in the wider peanut production and mycotoxin control fields? Program Future 1. In what ways, if any, did the location-based projects (Ghana, Haiti, and Southern Africa projects) complement other groundnut and/or mycotoxin research activities in these regions? What are the lessons learned that can improve the complementarity of future USAID research and development investments in groundnuts and mycotoxins to other ongoing programs in the regions? In what ways can USAID programs in groundnut and mycotoxins more effectively engage developing country partners in research? 2. What types of changes in implementation would more effectively achieve outcomes in the following components: research (i.e., design, implementation, communications, stakeholder involvement), capacity development (i.e., student recruitment and selection, content, location) and/or institutional collaboration? III. EVALUATION DESIGN AND METHODOLOGY A) Evaluation Design The evaluation team will develop and elaborate the evaluation approach in the evaluation plan. However, a mixed methods or process evaluation approach to collecting and analyzing qualitative data is recommended. Qualitative data will be synthesized, analyzed and triangulated to provide robust, objective evidence to answer the evaluation questions. Quantitative data, including secondary sources, program performance monitoring data, among others will also be used to support the findings and conclusions. B) Data Collection Methods Data collection protocols will address the experiences, perceptions, opinions, motivations, and knowledge of the stakeholders interviewed, the context within which the activity is operating; and the factors (causal mechanisms) leading to observed results (or non-results). In-depth conversations guided by structured Page | 40 and semi-structured key informant interview instruments provide the opportunity for a flow of information that is not constrained by pre-determined response categories. This allows for exploration of subjects that surface during the interview and elicits information that provides nuance and insight for addressing the evaluation questions. C) Methodological strengths and limitations Limitations will be addressed in the finalized evaluation plan. IV. EVALUATION PRODUCTS A) Deliverables The specific deliverables and timetable for delivery are described in detail in Section VI and include:  Knowledge Gap Table (optional) April 5, 2016  Evaluation Plan (at least 2 revisions) April 10, 2016  Travel Completion Date June 15, 2016  Preliminary Findings June 30, 2015  Draft Evaluation Report July 15, 2016  Final Evaluation Report August 1, 2016 B) Reporting Guidelines Reporting should be done in accordance with USAID’s Criteria to Ensure the Quality of the Evaluation Report (copied below) and found in Appendix I of the USAID Evaluation Policy (attached). More specifically,  Knowledge Gap Table should be in the form provided in Appendix A or as otherwise agreed;  Evaluation Plan should be in the form provided in Appendix B or as otherwise agreed;  Travel Debriefs should be by phone call or email prior to country departure;  Draft and Final Evaluation Reports should follow the format and length described in Appendix C or as otherwise agreed. V. TEAM COMPOSITION The technical qualifications of EET members must be matched with the technical areas of focus of the PMIL projects. Team members must have the expertise necessary to evaluate the many sub-award projects and to address the Scope of Work topics. Each member is requested to submit a CV that demonstrates relevant experience in technical, evaluation and management skills. USAID will designate one team member as the Team Leader. Evaluation Planning Lead (1): a senior-level evaluator experienced in international agricultural research for development and technology dissemination with technical expertise in monitoring and evaluation methodology and a minimum of 15 years of experience. The preferred candidate will be familiar with USAID (or other donor) funded programs. The candidate will also have: a) the capacity to conduct independent program evaluation; b) a thorough understanding of research methodology; c) the ability to Page | 41 analyze issues and formulate concrete recommendations orally and in writing; d) be available to travel and meet the timelines for completion of the evaluation; and e) not have any conflicts of interest. Team Administrative/management Lead (1): A senior-level evaluator with a minimum of ten years of experience managing and/or evaluating multifaceted international development research and/or university-based programs. The preferred candidate will be familiar with USAID (or other donor) funded programs. A background in agricultural development, with technical expertise in a field relevant to peanut agronomic systems in either Latin America, Sub-Saharan Africa or Asia is recommended. The candidate will also have: a) a demonstrated capacity to conduct independent research program evaluation; b) an understanding of USAID’s foreign assistance goals, and its particular objectives related to collaborative research, agricultural development and food security; and c) the ability to analyze issues and formulate concrete recommendations orally and in writing. Technical team members (2): Must be experienced experts in international agricultural research for development and technology dissemination. Technical team members will also have demonstrated the following: a) the capacity to conduct independent program evaluation; b) a thorough understanding of research methodology; and c) the ability to analyze issues and formulate concrete recommendations orally and in writing. Disciplines of all members (4): The team members need familiarity with Caribbean, Sub-Saharan Africa or Asia’s agricultural systems with the following required composition of skill sets among them: organizational development, quantitative and qualitative evaluation, agronomist/agricultural systems, social/economics background, aflatoxin detection and management, and peanut breeding or genomics (or other plant breeding skills). VI. EVALUATION MANAGEMENT A) Logistics & Scheduling 1) Conference call with USAID - between March 5-March 15, 2016 A conference call will be scheduled between the EET and the USAID Evaluation Manager, the PMIL’s Agreement Officer’s Representative (AOR), and other officials in the Research and Monitoring & Evaluation Divisions of the Bureau for Food Security to review the scope of work and answer questions concerning the implementation and delivery of the evaluation. 2) Desk review - between March 5-April 1, 2016 The EET will conduct a desk review of the PMIL’s publications and materials. The purpose of the desk review is to obtain needed background and context about the PMIL and USAID in order to complete the Knowledge Gap Table and the Evaluation Plan (see below). Documents to be reviewed will include, but are not limited to, the RFPs (request for proposals), approved program proposals, the Leader Cooperative Agreements, annual reports, work plans, program operation documentation, and funded research proposals. Team members will also familiarize themselves with the Feed the Future Global Food Security Research Strategy3 and the USAID Evaluation Policy4 . 3) Knowledge Gap Table – due April 5, 2016 3 http://pdf.usaid.gov/pdf_docs/PDACR702.pdf 4 http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf Page | 42 Based on the desk review, the EET will provide the USAID Evaluation Management the completed Knowledge Gap Table (see Appendix A). 4) Evaluation Plan - due April 10-12, 2016 The EET will submit to the USAID Evaluation Manager the Evaluation Plan (see Appendix B). The purpose of the Evaluation Plan is, in part, for the EET to present their evaluation design which includes, in part, research questions, methodology for quantitative and qualitative data collection and data analysis, work plan, timeline and proposed domestic and international travel. The Evaluation Plan must be approved by the USAID before the EET can travel and begin their field work. USAID will provide approval or request changes. 5) Domestic and international travel – to be completed by June 15, 2016 The EET will need to travel domestically and internationally to gather the needed information to implement the evaluation plan and complete this scope of work. Domestic travel is limited to one trip, up to two days excluding transit, to visit the PMIL ME at University of Georgia. This visit should precede all international travel. International travel is limited to two separate trips per evaluator to visit international collaborators and stakeholders with the PMIL. The USAID Evaluation Manager must pre-approve all travel. All travel will be arranged for the EET by the USDA Invitational Travel mechanism and must be in accordance with U.S. Government travel regulations. 6) International travel debriefs – prior to country departure A short summary of data collected and preliminary findings will be sent to the USAID Evaluation Manager for each country visited before departure from that country. This is not to be a trip report, nor should time be billed to write a trip report. Instead, it is meant to provide the USAID Evaluation Manager with progress made against the Evaluation Plan. Mission outbriefs should be included as applicable. 7) Preliminary findings – due June 30, 2016 The EET will provide to the USAID Evaluation Manager a written summary of the preliminary findings that will be used to develop the draft evaluation report. The summary can be presented as a PowerPoint presentation or other suitable written form. 8) Draft evaluation report – due July 15, 2016 A draft of the evaluation report will be submitted electronically in MS Word format to the USAID Evaluation Manager. USAID will review the draft for content. The ME will review the draft for accuracy. All comments, corrections and suggestions for consideration will be sent to the EET by June 22, 2016. 9) Final evaluation report – due August 1, 2016 The final evaluation report, subject to approval by USAID, should sufficiently address all comments and corrections provided to the draft report and by USAID-508 compliant. An optional presentation and/or discussion of the finding may be scheduled. B) Level of Effort The period of performance is March 1, 2016 through August 31, 2016. Page | 43 Tasks Eval. Planning Lead (1) Administrative Team Member (1) Technical Team Member (2) LOE Days LOE Days LOE Days Conference Call/Desk Review 3 4 4 Knowledge Gap Table 1 1 1 Evaluation Plan 4 3 3 Travel & Travel Debriefs 5 21 21 Preliminary Findings 0 4 4 Draft Report 2 8 5 Final Report 1 4 2 TOTAL LOE 16 45 40 Page | 44 Appendix A: Knowledge Gap Table Key Knowledge Knowledge Gaps Program Management Technical leadership Administration Financial management Research Program Depth, breath, rigor Collaboration, outreach, technology dissemination HICD Future of Program Appendix B: Evaluation Plan FTF Activity/Mechanism Name: FTF Activity Country/Countries: Evaluation Lead Investigator: USAID Evaluation Manager: Approximate start date: Page | 45 Preface This document describes the components needed to complete an Evaluation Plan for Feed the Future (FTF) Activities. A. FTF Project Evaluation Design 1. FTF Activity/Mechanism Description Describe the FTF activity/mechanism being evaluated. Provide enough detail to make clear the justification for the proposed methodology. Include the following items: activity/mechanism goals and objectives, main program components/interventions and delivery mechanisms, key activity/mechanism outcomes and indicators, target areas and target population groups, criteria for selecting target areas, criteria for selecting program participants, program implementation plan (start date, duration, deployment plan and timeline).(Note: much of this material can come from project documents.) 2. Program Logic Please include either a diagram and/or a narrative that describes the program logic and articulates the causal pathways from activity implementation to the desired impacts. The description should include intermediate outcomes that would change along the way to final impacts or objectives of the project. (Note: this should also be available in project documents.) 3. Evaluation Research Questions Succinctly state the primary questions that the evaluation will seek to answer.(Note: this should be available in the evaluation SOW.) 4. Methodology for Quantitative and Qualitative Data Collection Please indicate briefly the methods and plans for data collection. This section should include all methods for primary collection (interviews, surveys, direct observation, etc.) and secondary data collection (project documents, performance reports, etc.). Provide the timing of any qualitative and quantitative data collection and explain how the two will be integrated. Include the number of planned survey rounds as well as the expected local data collection partner if applicable. 5. Methodology for Quantitative and Qualitative Data Analysis Analysis methods should be described in detail for both quantitative (descriptive statistics, regression analyses, etc.) and qualitative (domain analysis, network analysis, etc.). Also, specific software that will be used should be mentioned (SPSS, STATA, ATLAS, etc.). 6. Outcome Measures Briefly discuss the outcome measures that will be used for this study (quantitative and qualitative) and relate them to the evaluation research questions. Explain which evaluation questions the quantitative and qualitative data will help address and how. Define the variables or indicators that will be used to measure these outcomes. (A quantitative example would be an outcome measure of “Greater access to new technologies among partner developing countries” and corresponding indicator “Number of new technologies under research, field testing or made available for transfer”. A qualitative example would be an outcome measure of “Effective management” and corresponding indicator of “Communication processes are well-established”.) Page | 46 7. Additional Pertinent Information Use this section to describe any further information that is pertinent to this particular evaluation and should be considered as part of the evaluation design. For example, this section could be used to discuss collaboration agreements for analysis with other institutions or overlaps with other evaluations and coordination with those evaluations. Evaluation Work Plan (adapt timeline as required) Activities Dates of Activity 1 st Month 2 nd Month 1 2 3 4 5 6 7 8 TASK 1. Develop evaluation design and implementation plan Activity 1: Activity 2: etc. TASK 2: Data Collection Activity 1: Activity 2: etc. TASK 3: Data Analysis Activity 1: Activity 2: etc. TASK 4: Report Writing Activity 1: Activity 2: etc. Page | 47 B. Evaluation Budget (if applicable) Submit a detailed budget with the evaluation design covering all costs related to conducting the evaluation, including data collection, labor, travel, and communications. 1. Budget Summary (adapt timeline as required) Category Month 1 Month 2 Total Labor Travel and subsistence 5 Data collection Equipment Other costs Sub-total Indirect costs Total C. Data Collection and Management Plan 2. Interviewer/Enumerator Training (if any) Describe the plans for training for all data collection (if any), including length of training, location, expected number of participants, topics covered, and the approach to piloting or field testing during training. 3. Data Management and Security Describe how all data collected will be gathered, entered, managed, and stored. Please specify how data will be kept secure. 4. Data Collection Approvals Describe the process and results of all data collection approvals. D. Data Collection Instruments Submit a draft of any data collection instruments that will be used for the evaluation. 5 USAID will provide the airfare costs. All per diem and M&IE are to be based on U.S. Government rates. Page | 48 Appendix C: Report Format Title Page Table of Contents List of Acronyms List of Tables List of Figures Executive Summary (3 pages) Program Management (15 pages) - Findings - Conclusions - Lessons Learned - Recommendations Research Program (15 pages) - Findings - Conclusions - Lessons Learned - Recommendations Program Future (5 pages) - Program Management - Research Program Appendices A. Scope of work B. Evaluation Plan C. Travel itinerary, locations and dates of field visits D. List of persons contacted E. List of materials reviewed F. Photographs: high resolution with caption and photo credit (5 photographs) Page | 49 APPENDIX B. EVALUATION PLAN FEED THE FUTURE EVALUATION PLAN FTF Activity/Mechanism Name: Peanut and Mycotoxin Innovation Lab (PMIL) FTF Activity Country/Countries: Target Countries: Ghana, Malawi, Haiti, Mozambique, Zambia; Partnership Countries: Burkina Faso, India, Kenya, Mali, Niger, Nigeria, Senegal and Uganda Evaluation Lead Institution: Purdue Evaluation Team: Joan Fulton, Farid Waliyar, Medson Chisis, Eric Welch USAID Counterparts: Jennifer Long; Lisa Wilson Approximate start date: March 15, 2016 Date submitted: March 31, 2016 2 Table of Contents A. FTF Project Evaluation Design ............................................................................................ 3 1. FTF Activity/Mechanism Description ...........................................................................................3 a. Introduction ...................................................................................................................................... 3 b. The program and projects ................................................................................................................ 3 2. Theory of Change and Program Logic ..........................................................................................8 a. Peanut and Mycotoxin Innovation Lab Integrated Theory of Change ............................................. 8 b. Activating the Theory of Change: Linking Program Components to PMIL Projects ......................... 9 c. Traditional Logic Model Guidance for PMIL Evaluation ................................................................... 9 3. Focus of the Evaluation ............................................................................................................11 a. Research Program .....................................................................................................................11 b. Program Structure and Management .......................................................................................12 c. Institutional Capacity Collaboration ..........................................................................................12 4. Methodology for Qualitative and Quantitative Data Collection ..................................................13 a. Details of the data collection plans ................................................................................................13 b. Document review and secondary sources .....................................................................................18 c. Key informant interviews ...............................................................................................................20 d. Bibliometric Analysis ......................................................................................................................20 5. Methodology for Quantitative and Qualitative Data Analysis ....................................................22 6. Methodological Limitations ......................................................................................................23 a. Sampling and participant selection bias ....................................................................................23 b. Interviewee recall bias ...............................................................................................................24 c. Interviewee response bias .........................................................................................................24 d. Interviewer bias .........................................................................................................................25 6. Evaluation Work Plan ...........................................................................................................26 B. Data Collection and Management Plan ............................................................................ 27 1. Survey Training ....................................................................................................................27 2. Data Management and Security ...........................................................................................27 3. Data Collection Approvals ....................................................................................................27 4. Data Collection Instruments ................................................................................................. 27 a. Protocol for Interviews of Project PIs, CoPIs and Collaborating Researchers ...........................28 b. Protocol for Interviews of ME personnel ..................................................................................31 c. Protocol for Interviews of Project Stakeholders and Partners ..................................................34 C. Curriculum Vitae of Researchers ................................................................................... 37 D. Evaluation Scope of Work ............................................................................................ 37 E. Knowledge Gaps Table ................................................................................................. 37 F. Abstract of the Twelve PMIL Projects ........................................................................... 37 PAGE | 50 3 A. FTF Project Evaluation Design 1. FTF Activity/Mechanism Description a. Introduction In compliance with the USAID Evaluation Policy (2011) and the Scope of Work for the Evaluation of the Feed the Future Innovation Labs for Collaborative Research on Peanut and Mycotoxin (2013-2017) (PMIL) this document:  presents an overview of the PMIL including: current organization of the projects and their leadership, project goals and objectives, and activities to date;  proposes a set of key indicators and data sources necessary to complete the evaluation;  identifies information already available through project documents and interviews with project leadership;  identifies gaps in information necessary to evaluate the programs including questions to be used in evaluation and additional sources of data necessary to address the indicators.  outlines a design for the collection of additional data and a plan for analysis. The Scope of Work (SOW) for this evaluation instructed the review team to evaluate: (a) the research program, (b) PMIL program structure and management, and (c) future directions for the PMIL. The SOW has identified evaluation questions for each of the four evaluation topics (a￾c) which inform this evaluation plan and provide guidance to the External Evaluation Team (EET). The evaluation team will produce an evaluation report that will be used by USAID/BFS/ARP-R as a mechanism to demonstrate accountability to stakeholders and to inform a future RFA to address outstanding research questions connected to peanut productivity and mycotoxin control (SOW 2016). b. The program and projects The Peanut and Mycotoxin IL comprises the most recent US AID funded program in peanut research. Begun in 1982, US AID programmatic funding for peanut research and has continued uninterrupted to the present, including the Peanut Collaborative Research Support Program (Peanut CRSP; 2007-2012) which preceded PMIL. In 2013, the Peanut and Mycotoxin IL was awarded a Leader with Associates Cooperative Agreement of $26,000,000 for a period of 4.5 years. PAGE | 51 4 The overarching mission of the Peanut and Mycotoxin IL is to apply leading innovative US science to improve peanut production and use, raise nutrition awareness and increase food safety in developing countries (http://pmil.caes.uga.edu/about/index.html). The PMIL program includes research topics similar to its predecessor – the Peanut CRSP. But the Peanut and Mycotoxin IL offers a significant modification by aiming to integrate two major themes – peanut production and mycotoxin research – under one roof as part of a value chain approach. The revised approach, funded over five years (FY 2013-2017) hopes to produce contributions and outcomes in both thematic areas as well as leverage the cross-theme design to produce innovative new insights and directions. Key dimension of PMIL are found in the Evaluation Scope of Work in Appendix D of this document. Important structural considerations for the evaluation of the Peanut and Mycotoxin Innovation Lab program include: a. The project portfolio is organized around a “value chain” approach in order to develop a programmatic coherence from project activities. b. Because effective functioning of the peanut value chain depends on management of aflatoxin contamination and mitigation, aflatoxin research is integrated across the value chain through various research and intervention activities in ways that can contribute to a reduction of the aflatoxin burden. c. Primary target countries include: Ghana, Haiti, Malawi, Mozambique and Zambia. In addition to the primary countries, PMIL has research activities in Burkina Faso, India, Kenya, Mali, Nigeria, Senegal and Uganda. Other components of the PMIL program include: (a) commitment to training and capacity development (b) integration of gender and nutrition, (c) strengthening capacity of agriculture research institutions in USAID FTF priority countries. The evaluation will also consider these program components. Despite the history of USAID funded research, the scope of this evaluation comprises only the activities, outputs and outcomes of the PMIL from 2013 when it was funded by USAID until the present. The PMIL program includes twelve projects organized into three groups: peanut production, mycotoxin detection and value chain intervention. Table 1 presents the list of projects, responsible primary investigators (PIs) and lead institutions. Figures 1 and 2 present graphically the organization of the PMIL program. PAGE | 52 5 Table 1. PMIL Projects, PIs and Lead Institutions Research Project Title Project Investigator Lead Institution A. Peanut Germplasm Development A1. Translational Genomics to Reduce Pre- harvest Aflatoxin Contamination of Peanut Peggy Ozias-Akins University of Georgia A2. Silencing of Aflatoxin Synthesis through RNA Interference (RNAi) in Peanut Plants Renee Arias USDA-ARS National Peanut Research Lab A3. An Integrated Global Breeding and Genomics Approach to Intensifying Peanut Production and Quality Mike Deom University of Georgia B. Mycotoxin Detection and Peanut Nutritional Studies B1. AflaGoggles for Screening Aflatoxin Contamination in Maize Haibo Yao Mississippi State University B2. Development and Validation of Methods for Detection of Mycotoxins Exposure in Dried Spotted Blood Samples Jia-Sheng Wang University of Georgia B3. Aflatoxin in Peanut and Peanut Products: Comparative Study on Analytical Methods for Detection of Aflatoxin Kumar Mallikarjunan Virginia Polytechnic Institute and State U. B4. Randomized Controlled Trial of the Impact of Treating Moderately Malnourished Women in Pregnancy Mark Manari Washington University, St. Louis C. Peanut Value Chain Interventions C1. Production to Consumption – Technologies to Improve Peanut Production, Processing and Utilization in Haiti Greg MacDonald University of Florida C2. Using Applied Research and Technology Transfer to Minimize Aflatoxin Contamination and Increase Production, Quality and Marketing of Peanut in Ghana David Jordan North Carolina State University C3. Producer and Consumer Interventions to Decrease Peanut Mycotoxin Risk in Ghana Nicholas Magnan University of Georgia C4. Aflatoxin Management Interventions, Education and Analysis at Various Steps Along the Peanut Value Chain in Malawi, Mozambique and Zambia Rick Brandenburg North Carolina State University C5. Productivity and Profitability Growth in Peanut Production: A Farm Level Analysis in Malawi, Mozambique and Zambia Boris Bravo-Ureta University of Connecticut PAGE | 53 6 Figure 1. Organization Chart of the Peanut and Mycotoxin Innovation Lab Managing Entity (ME) University of Georgia Director and Program PI: Dr. David Hoisington Co-Director: Dr. Jaime Rhoads External Advisory Panel (EAP): Byanyima, Cardwell, Ehlers, Emmott, Johnson, McDermott, Minde, Nigam, Wright, (Long, USAID, ex officio) External Evaluation Committee (EEC) Project Sub￾categories and Numbers Project team partner institutions (US, I=International L=Lead) A: Peanut and Germplasm Development B. Mycotoxin Detection & Peanut Nutritional Studies Project Category PMIL/LIL Scholar Program; Other local Other Integrated Project Dimensions partner and stakeholder relationships Research, intervention and dissemination activities in other non-target countries. A1 UGA-207 US: UGA (L), USDA-ARS; I: ICRISAT, U. Brasilia, ISRA A2 USDA-201 US: USDA￾ARS (L); Auburn University I: ICRISAT, Kenyatta U., NaCRRI (Uganda) A3 UGA-203 US: UGA (L), USDA-ARS, UConn, A&M, VTech, UFL, NMSU; I: NaSARRI (Uganda), IIAM (Mozambique), TNAU (India), SARI (Ghana) B1 MSU-209 US: Miss Sate Univ (L), USDA-ARS; I: IITA Nigeria B2 UGA-210 US: UGA (L), Tufts; I: University of Ghana B3 VT-211 US: Virginia Polytechnic Institute & St. University (L) B4 WU-206 US: Washington University (L), California Polytechnic State University; I: University of Malawi C1 UF-204 US: UFL (L), UGA, Cornell, NCSU, M & F for Kids, Frank’s Design for Peanuts; I: Kreyol Inc., Acesso Corp., M&F for Kids (Haiti) C2 NCSU-208 US: NCSU (L), VPI, UConn, UFL, UGA; I: KNUST (Ghana); CSIR (Ghana), U. Ghana C5 UTC-212 US: UConn (L), NCSU; I: IIAM (Mozambique), DARS (Malawi), NARO (Uganda) C3 UGA-205 US: UGA (L), IFPRI; I: University for Development Studies (Ghana) C4 NCSU-202 US: NCSU (L), UGA, VPI; I: ICRISAT, Lilongwe U., CARS & IIAM, (Mozambique), U. Zambia, Exagris Ltd. (Malawi) C. Peanut Value Chain Interventions PAGE | 54 7 Figure 2. Project-Region-Researcher Institution Two-Mode Network Depictions of PMIL Structure Color Key: Dark Blue = PMIL Projects; Bright Green = US; Bright Pink = South Africa; Red = West Africa; Purple = East Africa; Bright Blue = Caribbean; Yellow = Europe; Salmon = Multiple Region; Medium Blue = South Asia; Medium Green = Latin America Size Key: Larger notes have more connections PAGE | 55 8 2. Theory of Change and Program Logic a. Peanut and Mycotoxin Innovation Lab Integrated Theory of Change The research program operating through the Peanut and Mycotoxin Innovation Lab (PMIL) comprises multiple projects that are integrated from two perspectives: value chain for the peanut and mycotoxin program that extends from upstream R&D through the end consumer. Figure 3 presents one model of change provided by PMIL to integrate the research, intervention and impact areas of the program. These include research for development to identify/assess/integrate new genetic diversity, markers and genes which, though breeding produces improved varieties. Pre-harvest options are identified through development and evaluation of improved cultivars across regions, transfer of technology and knowledge across regions, evaluation of inputs and biotic/abiotic stress on aflatoxin control, among others. Figure 3. Mycotoxin PMIL Model of Change Post-harvest options are identified through development and evaluation of drying, sorting and storage practices/technologies that are effective for aflatoxin management, identification of best practices appropriate for SMEs and assessment of aflatoxin waste stream management efforts, among others. In addition, the program aims to develop standardized methods and PAGE | 56 9 sampling protocols for aflatoxin detection that are simple and cost effective. Training for mycotoxin detection is also important. Finally, though not presented in this diagram, PMIL incorporates activities that assess the impact of different peanut nutrients on treating moderately malnourished women in pregnancy. In each of these areas of intervention and research, PMIL aims to produce optional technologies for transfer to target countries. b. Activating the Theory of Change: Linking Program Components to PMIL Projects In addition to the connection between program components and the integrated value chain presented in the previous section, PMIL is designed such that each of its projects contribute differently to the various program components. Shown in Table 2 all projects in the research portfolio are relevant to two or more program components. Given that these linkages are conceptualized and designed by PMIL, the EET will examine not only whether each individual project is accomplishing it objectives, but also the extent to which there is vertical learning and integration among projects (A1 to C5) and horizontal integration across program components (e.g. improved varieties – mycotoxin management – seed production). Table 2. Linking Program Components in Theory of Change with Projects PMIL Program Component / Projects Improved peanut varieties Mycotoxin management Seed production Post-harvest handling & processing Market opportunities  Genomics (A1)  RNAi (A2)  Breeding (A3)  Aflagoggles (B1)  Blood samples (B2)  Detection options, Sampling strategies (B3)  Prenatal nutrition study (B4)  Haiti value chain (C1)  Ghana value chain (C2, C3)  Malawi/Mozambique/ Zambia value chain (C4, C5)  Pre-harvest economics (C1-C5) c. Traditional Logic Model Guidance for PMIL Evaluation A logic model is a broadly accepted approach to linking inputs and activities to outputs to outcomes to impacts. Inputs are the financial, scientific and human resources, financial, human necessary to implement the proposed research. Activities are broadly conceived as PAGE | 57 10 mechanisms that link objectives to actions. Outputs are immediate products of project activities while outcomes represent the effect that outputs have on the broader context or environment at which the activities are aimed or embedded. Impacts are the project results that, once diffused, affect broader and longer term goals such as poverty reduction, in the case of US AID Innovation Labs. The Peanut and Mycotoxin Innovation Lab has also developed a more traditional logic model that characterizes the connection between inputs, activities, outputs and outcomes. Presented in Figure 4, financial and intellectual inputs are used to undertake research and other activities to produce a variety of outputs including varieties, post-harvest options, aflatoxin detection and control technologies and students. These outputs are expected to have initial effects, particularly in the target countries, on the availability of improved varieties, higher yields, and improved detection of aflatoxin. Subsequently, the project outputs, if taken up as expected will improve farmer livelihoods, reduce aflatoxin consumption and, ultimately, reduce poverty and stunting. It is unlikely that the PMIL evaluation will discern ultimate or even intermediate outcomes as PMIL was established approximately three years ago and many of projects were not approved until 2014. Nevertheless, the evaluation will make every effort to understand whether and to what extent PMIL is on track to accomplish these longer term outcomes. PAGE | 58 11 Figure 4. PMIL Logic Model (PMIL 2015 Annual Report, January 2016) 3. Focus of the Evaluation Three main areas of interest have been identified for this evaluation (from the Scope of Work): a. Research Program 1. The PMIL research combines two thematic areas, peanut productivity and mycotoxin control. Program activities are organized around a value chain approach. How effective has this approach, combining productivity and mycotoxin control throughout the program, been in achieving research outcomes in both areas? In what ways did combining these two research themes in one program strengthen or weaken the focus on the most important questions in both of these research areas? Were these areas adequately addressed to justify this combination of research areas? Why or why not, and is it equally the case for both thematic areas? 2. To what extent did each project generate robust and quality research outputs using disciplinary-appropriate metrics? Are the outputs relevant for a research-for￾development project (e.g., did they generate new breeding lines of relevance to FTF countries or aflatoxin detection methods that respond to local context in FTF PAGE | 59 12 countries)? Were the projects undertaken using the right tools/technologies and were they well executed? 3. What research-related challenges has PMIL faced during research design and implementation? In what ways have these challenges been addressed? 4. Given the dual benefit mandate of Title XII authorized programs, such as PMIL, is there a sufficient balance between research efforts directed towards priorities of the domestic peanut and mycotoxin stakeholder community and the peanut and mycotoxin research priorities required to advance global food security goals, particularly in FTF countries? 5. To what extent has PMIL met its academic training and technical capacity strengthening targets? What improvements, if any, are needed in the ways: a. the program identifies and addresses academic and technical capacity needs? b. academic and technical capacity strengthening activities are implemented? c. people are targeted and selected for training? b. Program Structure and Management 1. How effectively has the PMIL ME managed research and training activities amongst sub￾awardees and stakeholders in the U.S., Latin America, Africa and Asia? And across the different thematic areas? In what ways has the ME supported coordination among the individual projects to ensure complementarity of effort and contribution to the PMIL’s objectives? For example, did the ME use the research outputs to shape the kind of outreach it undertook (e.g., was the ME the interlocutor of information generated among various projects and did the ME serve in its role of bringing information to people's attention as appropriate across the projects)? What opportunities are there to improve the coordination amongst the different partners and among the different thematic areas to better achieve outcomes? 2. What role has or could the External Advisory Panel play in contributing to program effectiveness and advice relating to broad research priorities in the wider peanut production and mycotoxin control fields? c. Institutional Capacity Collaboration 1. In what ways, if any, did the location-based projects (Ghana, Haiti, and Southern Africa projects) complement other groundnut and/or mycotoxin research activities in these regions? What are the lessons learned that can improve the complementarity of future USAID research and development investments in groundnuts and mycotoxins to other PAGE | 60 13 ongoing programs in the regions? In what ways can USAID programs in groundnut and mycotoxins more effectively engage developing country partners in research? 2. What types of changes in implementation would more effectively achieve outcomes in the following components: research (i.e., design, implementation, communications, stakeholder involvement), capacity development (i.e., student recruitment and selection, content, location) and/or institutional collaboration? 4. Methodology for Qualitative and Quantitative Data Collection a. Details of the data collection plans The scope of this review covers four main areas outlined in the Scope of Work (SOW) for the twelve projects of the PMIL. Each project review will be conducted using a similar methodology, guided by the same set of SOW questions. As each project is designed to address different research questions and objectives and because the collaborating partner countries and institutions vary across projects, the evaluation will consider the unique activities, outputs and and near-term outcomes of each project. In addition to project-level analysis, the evaluation will examine program-level interconnections and resulting program-level outputs and outcomes, particularly those where peanut and mycotoxin research efforts interact or are supposed to interact. All project interviewees and stakeholders will be asked to identify program level accomplishments, outcomes and future expectations. The EET will conduct interviews of the PI, CoPI, collaborating scientists and stakeholders for each project. The EET estimates that it will conduct approximately four or five interviews for each project as well as four to five interviews of the ME (Management Entity) team (overall PMIL Director and Co-director) and staff. Additionally, the EET will conduct interviews of students and stakeholders as appropriate and relevant for each project. In total, the EET will interview over 60 individuals associated with PMIL. While it may be necessary for logistical or convenience reasons to interview two or three individuals in one session, the EET will aim to maximize one-on-one interviews, particularly given confidentiality issues discussed below. All interviews will be conducted using formally constructed interview protocols with main questions and probes. All interviewees will be assured of confidentiality. To the extent possible, interviews will be conducted by two EET members; one EET member will be designated as the primary note taker. That person will write either handwritten notes or enter the notes in a laptop computer. If the notes are handwritten, the EET member responsible will enter them into a laptop at a later date such that all notes will be preserved electronically. PAGE | 61 14 More detail about the data collection process is presented below. At the program level, the EET will also conduct a review of the Management Entity (ME) and undertake interviews of ME personnel. This will include some level of budgetary review as well as assessment of proposal review processes, communication, external advisory team management, reporting and other key ME functions. EET team members will travel to University of Georgia for the first round of interviews, currently scheduled for April 27 and 28, 2016. Project and program review data will also be collected on site visits to different projects sites in Haiti, Ghana, Zambia and Malawi. These trips will be scheduled in consultation with US AID and the Management Entity. Table 3 presents the overall evaluation questions linked to indicators and proposed sources of data. The evaluation will draw from multiple data sources - program documents, interviews with key informants, focus groups, and other outputs. PAGE | 62 15 Table 3. Linking Evaluation Questions, Knowledge, Indicator and Source of Data Knowledge/Outcome Illustrative Indicators Source of Data Research Program RP 1. How effective is the combined peanut productivity and mycotoxin control value chain approach? In what ways did combining the two research themes in one program strengthen or weaken the focus on the most important questions in both of these research areas? Topics addressed for the first time; knowledge absorption and integration; new insights; etc. Specific evidence that the integrated design has leveraged knowledge otherwise not available; new collaborations; new projects and research directions; changes in approaches in the field and research. Interviews of PIs, CoPIs, partner collaborators, EAP; project documents. Site visits. RP 2. To what extent did each project generate robust and quality research outputs using disciplinary appropriate metrics? Are the outputs relevant for a research-for-development project (e.g. did they generate new breeding lines of relevance to FTF countries or aflatoxin detection methods that respond to local contexts in FTF countries)? Outputs by category, benchmarks from other projects New breeding lines produced, methods of aflatoxin detection produced, etc. Review of reports, interviews with PIs, CoPIs, partner collaborators, EAP. Site visits. Knowledge outputs, dissemination and uptake. publications, especially peer reviewed, reports, Bibliometric analysis Information about tools and techniques available; selection decisions; information on results Evidence of effective project efforts in FTF countries; problems encountered; etc. Interviews of PIs, CoPIs, partner collaborators, EAP; project documents. Site visits. Options available to target countries – technologies, pre-harvest and postharvest techniques, etc. Evidence of availability; local awareness of availability or near future availability; local preparation use of new technologies Interviews and possibly focus groups of local stakeholders, international partners; reports and other documentation To what extent have project outputs addressed nutrition; how are projects progressing Research outputs related to nutrition; program components aimed at improving nutrition Interviews of PIs, CoPIs, partners RP 3. What research-related challenges has the PMIL faced during research design and implementation? In what ways have those challenges been addressed? What opportunities are there to improve research outputs moving forward? Obstacles or challenges faced Evidence of challenges from interviews, reports. Success of work-arounds. New directions, opportunities lost. Interviews of PIs, CoPIs, partner collaborators, EAP; project documents. Site visits. RP 4. Given the dual benefit mandate of Title XII authorized programs, such as the Peanut and Mycotoxin IL, is there a sufficient balance between research efforts directed towards priorities of the domestic peanut and mycotoxin stakeholder community and the peanut and mycotoxin research priorities required to advance global food security goals, particularly in FTF countries? Range of outputs and applicability to local as well as broader contexts Evidence of locally relevant outputs, uptake, awareness. Evidence of collaboration with international agriculture research centers (IARCS), use of outputs. Evidence of value of outputs to domestic US research Interviews of PIs, CoPIs, partner collaborators and other stakeholders, EAP; annual and progress reports. Site visits. RP 5 Has the Peanut and Mycotoxin IL met its academic training and technical capacity strengthening targets? Are the appropriate type and number of people being targeted for the right kind of training? In what ways has the program identified and addressed academic and technical capacity needs? How could this be improved? How potential degree recipients are targeted, how degree training is prioritized. Training statistics. PMIL/LIL Scholars Program. Training types and numbers relative to targets. Students enrolled, degrees received, current training, non-degree training, scholarships awarded. Program of study for PMIL/LIL SP. Evidence of training activity. Job placement. Review reports; interviews with PIs. Student interviews. Site visits. PAGE | 63 16 Program Structure and Management PS 1. How effectively has the Peanut and Mycotoxin IL ME managed research and training activities amongst sub￾awardees and stakeholders in the U.S., Latin America, Africa and Asia? How effectively has the ME managed across thematic areas? In what ways has the ME supported coordination and communication among individual projects to ensure complementarity of effort and contribution to PMIL’s objectives? In what ways could communications and coordination be improved as a way to better achieve outcomes? Project and program communication processes; Are PIs aware of what is being done at other sites; do they understand how their research trajectories are integrated; do they direct efforts to communicate about and undertake integrative research activity (across the value chain and thematically)? Evidence of communication and coordination effectiveness; problems; team perception Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. Site visits. Decision making processes Perceptions about inclusion; perceptions about team approach, sharing of priorities, results, findings. Perceptions about whether research priorities and funding align with integration across themes Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. Site visits. Reporting requirements & associated communication Timeliness, completeness and distribution of reporting Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. Site visits. USAID interviews. Outreach and engagement activity Evidence of potential partners not included; problems with external perception of program Documents; review reports; interviews with PIs, CoPIs, collaborating researchers and stakeholders. Site visits. PS 2. What role has or could the External Advisory Panel play in contributing to program effectiveness and advice relating to broad research priorities in the wider peanut production and mycotoxin control fields? Role of EAP; assistance with program review and direction; impact of advice on direction Evidence that meetings and other communications have taken place; broad program-wide efforts; awareness and understanding of program goals and project integration efforts/potential Interviews with ME, PIs, EAP, USAID. Institutional Capacity Collaboration IC 1. In what ways, if any, did the location-based projects (Ghana, Haiti, and Southern Africa projects) complement other groundnut and/or mycotoxin research activities in these regions? What are the lessons learned that can improve the complementarity of future USAID research and development investments in groundnuts and mycotoxins to other ongoing programs in the regions? In what ways can USAID programs in groundnut and mycotoxins more effectively engage developing country partners in research? Local bridging and interaction between location-based projects and other regional efforts; local partnership existence, activities and outcomes; evidence of cross-theme awareness and integration in local areas; opportunities for bridging and furthering complementarity locally. Quantity of partnerships by type and function; length of partnership; partnerships gained and lost; perceptions about partnership quality, roles, and effectiveness; future potential for growth and new directions; needs and missed opportunities; evidence of integrating project outputs by partners in local areas. Potential partners not linked. Interviews with PIs, CoPIs, Collaborators, Partners, EAP, USAID. Site visits. Other organizations as identified during the interviews or document review. IC 2. What types of changes in implementation would more Knowledge about program directions Gaps, barriers, opportunities, challenges, Interviews with PIs, CoPIs, PAGE | 64 17 effectively achieve outcomes in the following components: research (i.e., design, implementation, communications, stakeholder involvement), capacity development (i.e., student recruitment and selection, content, location) and/or institutional collaboration? relative to needs; understanding about complementary scientific directions; new food security threats or concerns; recognition of lack of attention paid by other research efforts; gaps to fill strengths and weaknesses of projects; opportunity to leverage capacity of partners toward new directions; evidence of cross￾theme complementarities addressed and not addressed. Collaborators, Partners, EAP, USAID. Site visits. Other organizations as identified during the interviews or document review. PAGE | 65 18 b. Document review and secondary sources The evaluation team is in the process of conducting an initial review of PMIL program and project documents, including:  program-level technical application  project-level technical applications / proposals  annual reports (2014 and 2015)  technical progress report (program and projects)  annual work plans  program highlights and project briefs  other documents summarizing project level work collected from the website and the Management Entity  Prior CRSP evaluation, response to evaluation Most of these documents are either available on the PMIL secure website or the PMIL website. It is based on these documents that the EET gleaned its understanding of the PMIL organization presented in Figures 1 and 2. Based on this initial review the EET may request additional documentation or records referred to or identified in available records. The EET may also request additional contact information not contained in the PMIL directory. For example, the EET may request contact information for graduate students or postdocs employed on the project or for stakeholder names and contact information. The aim of this desk review is identify key knowledge gaps. The desk review of each project will be conducted according to the project review template (Table 4). Summary knowledge gaps will be entered into the table in Appendix F which will help guide the remainder of the project. PAGE | 66 19 Table 4. Project Review Template Outline To facilitate broad and efficient review of the PMIL projects, EET members will take on specific tasks as primary or secondary reviewer of each of the 12 research projects (see Table 1). The primary and secondary reviewers will be responsible for the in-depth analysis of their assigned projects. The review itself will be comprised of the desk review and follow-up interviews. The purpose of the desk review is to help identify knowledge gaps, answer evaluation questions, and to inform the subsequent interviews. The following outline will help make comparisons and ensure that the review is consistent across projects. The primary reviewer will be responsible for completing the template for the assigned project. The secondary review, confirm and revise the template to ensure accuracy. When there is disagreement or gaps, the two reviewers will confer to discuss evidence and support before finalizing the template findings. The two reviewers will make final decisions based on consensus. 1) Development impact potential. Contributions or potential contributions of the Peanut and Mycotoxin IL research activities to developing research outputs that could contribute immediate and intermediate outcomes. Reviewers should link findings to theory of change and logic model. 2) Quality of science. To what extent did the project generate robust and quality research outputs using disciplinary￾appropriate metrics? What evidence is there of this? Are the outputs relevant for a research-for-development project (e.g., did they generate new breeding lines of relevance to FTF countries)? Please highlight and describe most important outputs. Were the projects undertaken using the right tools/technologies and were they well executed? 3) Challenges faced. Evaluators document these, where possible, during desk review. Interviews should build on these findings. 4) Impacts in the US. Is there any evidence of IL-related outputs benefiting US producers, consumers or research community? 5) Training accomplishments. Information needed on degrees awarded and current positions of graduates (partner country, other developing country, US). Non-degree training (type and number) will also be evaluated. Please comment on the balance and appropriateness of this training.  Appropriate mix of training topics and appropriate training method  Considerations of gender balance 6) Partnerships. Evaluators to describe existing partnerships and their quality, identify potential partnerships. Identify any sharing of information and spillovers from IL partners. 7) Project-specific questions for interviews. PAGE | 67 20 c. Key informant interviews The EET will conduct a series of semi-structured interviews with key project researchers, students, advisory panel members, and other relevant stakeholders. Interviewees will include individuals from the following categories:  Project PIs and CoPIs  US collaborators on research projects  International collaborators on research projects  External advisory board members  Partner organizations local government and NGO partners, and other stakeholders  EAP members  ME personnel including the PMIL director and Co-director  USAID Mission and other in-country staff The evaluation team lead has already conducted an informational unstructured interview with the PMIL program director. Also, members of the team, have been able to discuss the evaluation with the USAID representatives. The EET has met once on a conference call and will meet at PMIL headquarters at the University of Georgia (UGA) to undertake a first wave of interviews on April 27-28, 2016. Additional interviews of US faculty and others will either be conducted during site visits to partner US institutions or by phone/skype. During the UGA visit, it may be possible for some team members to travel to Athens for interviews. Although, as noted above, given scheduling and other factors, it may be necessary to interview a few PMIL researchers in groups of two or three, the EET will try to avoid these. Table 5 presents the outline of data collection activities, particularly those related to interviews and site visits. Table 6 lists individuals representing stakeholder groups from which the EET will select a sample of key informants for interviews. Section C contains an initial set of preliminary interview protocols to be used in semi-structured interviews with key informants. d. Bibliometric Analysis The PMIL program, including all projects has produced a number of journal articles. As part of the desk review and follow-up inquiry, the EET will collect the full set of journal article references attributed to the PMIL by its collaborators. The EET will then collect bibliometric data publication and citation data related to the journal articles. Bibliometric data are drawn from a standard source: Thomson Reuters ISI Web of Science (WoS). The team will also search Google Scholar to assess the extent to which PMIL academic and gray literature (reports, white papers, etc…) are cited. PAGE | 68 21 The EET will conduct three general types of bibliometric analysis: quantity of outputs, quality of the publication and citation patterns of PMIL outputs. To measure quantity of outputs, we will document the total number of peer reviewed publications, reports and other grey literature, and other written products. We will ask the PMIL ME to provide a complete list of publications since 2013. During interviews with faculty or in follow up emails, the EET will ask PMIL authors to indicate whether a particular journal article produced since 2013 included research findings or insights gained from the current PMIL project. This step is essential for attribution purposes. Journal impact factors, which are publicly available, represent one well recognized way to measure the scientific quality of research outputs. To measure publication quality, we will also collect journal article citation data for all PMIL publications (Google Scholar and Web of Science). Because PMIL has only be operational for around three years, and given the time required for citations, the EET is not expecting to find high citation rates and will be careful to place too much emphasis on this last metric. Table 5. Project specific data collection plan Project Name Desk review of project documents Interviews of PI and Co￾PIs, collaborators and staff during UGA site visit Anticipated international site visits to review achievements Interviews of international and US collaborating scientists Bibliometric data collection A1. Translational Genomics to Reduce Pre- harvest Aflatoxin Contamination of Peanut   tbd By skype, phone or visit, tbd  A2. Silencing of Aflatoxin Synthesis through RNA Interference (RNAi) in Peanut Plants   tbd By skype, phone or visit, tbd  A3. An Integrated Global Breeding and Genomics Approach to Intensifying Peanut Production and Quality   tbd By skype, phone or visit, tbd  B1. AflaGoggles for Screening Aflatoxin Contamination in Maize   tbd By skype, phone or visit, tbd  B2. Development and Validation of Methods for Detection of Mycotoxins Exposure in Dried Spotted Blood Samples   tbd By skype, phone or visit, tbd  B3. Aflatoxin in Peanut and Peanut Products: Comparative Study on Analytical Methods for Detection of Aflatoxin   tbd By skype, phone or visit, tbd  B4. Randomized Controlled Trial of the Impact of Treating Moderately Malnourished   tbd By skype, phone or visit, tbd  PAGE | 69 22 Women in Pregnancy C1. Production to Consumption – Technologies to Improve Peanut Production, Processing and Utilization in Haiti   tbd By skype, phone or visit, tbd  C2. Using Applied Research and Technology Transfer to Minimize Aflatoxin Contamination and Increase Production, Quality and Marketing of Peanut in Ghana   tbd By skype, phone or visit, tbd  C3. Producer and Consumer Interventions to Decrease Peanut Mycotoxin Risk in Ghana   tbd By skype, phone or visit, tbd  C4. Aflatoxin Management Interventions, Education and Analysis at Various Steps Along the Peanut Value Chain in Malawi, Mozambique and Zambia   tbd By skype, phone or visit, tbd  C5. Productivity and Profitability Growth in Peanut Production: A Farm Level Analysis in Malawi, Mozambique and Zambia   tbd By skype, phone or visit, tbd  5. Methodology for Quantitative and Qualitative Data Analysis Semi-structured interviews will be conducted following traditional interview methods in which interviewers aim to engage the interviewee in a substantive conversation about their research. In-depth conversations guided by interview instruments provide the opportunity for a flow of information that is not constrained by pre-determined response categories. Interviewees are both able to follow lines of inquiry unique to an individual, but also collect equivalent data across interviews. This approach allows exploration of subjects that surface during the interview and elicits information that provides nuance and insight for addressing the evaluation questions. A written interview protocol including primary questions and follow-up probes will be used as a structured guide for the interviewer to follow and to ensure all questions are asked of all respondents. Example interview protocols are presented in Section D below. Interviewers are experienced and will allow respondents to respond freely to questions. The interview protocol will be further revised based on the ongoing review of documents described above. Ideally interviews will be conducted in teams of two. The use of teams enhances creativity, enables for the convergence of observations and increases confidence in the findings, and allows the interview material to be viewed from different perspectives. In some cases, team-based interviews may not be possible. PAGE | 70 23 Interviews with key informants will be captured in notes, which will be expanded into field notes at the end of each day of interviewing. Notes will be saved as text files and summarized at the end of the interview. Summaries should include specific insights or key findings from the interviews and focus groups. Cross-interview findings can also be noted for later analysis. On site visits, EET members will meet each evening to review notes, summarize key findings for the day and identify key questions for further exploration. During other phases of data collection, such as skype interviews of US collaborators, EET members will schedule meetings to review findings. Each of the twelve PMIL projects will be assigned to one of the three main members of the EET as lead to synthesize notes and summarize findings. The primary EET member will collect all notes from interviews as well as all background materials for the particular project as input. Project write-ups will address the evaluation questions established for the evaluation, integrating key indicators from Table 3 with evidence from interviews. All individuals who attended the interviews will be asked to review the individual project write-ups, include additional comments and identify inconsistencies. When all project write-ups are complete, the team lead will develop an overall program assessment that takes into account the individual projects and the overall program-level and cross-project outputs and outcomes. Bibliometric analysis will also be integrated. Conclusions will include accomplishments and shortcomings. Based on the evidence and considering project goals, the EET will develop a set of recommendations to assist USAID and PMIL in the future. 6. Methodological Limitations a. Sampling and participant selection bias In order to develop a list of potential interviewees from among the various stakeholders in these programs, we have solicited a list of active and past staff, investigators, partners, host country officials and academics. The PMIL website and documents provide lists of stakeholders. The list can be found in Table 6. Additional names and contact information will be collected throughout the evaluation for follow-up. We have identified a number of critical informants with whom we will be speaking. We will interview all US PIs and CoPIs, International CoPIs and other important US and international collaborators. We will also interview EAC members, USAID in-country staff, and key representatives of partners stakeholders. PAGE | 71 24 For other stakeholder groups identified as critical to the evaluation, we will select individuals on a purposive basis considering 1) level of involvement with the project, 2) breadth of knowledge of program activities, 3) representativeness of the range of diversity of stakeholders, 4) availability during the time available to the evaluation team in each host country or by Skype, 5) referrals. Given the complex nature of this program and the wide range of stakeholders, current and former, development of an exhaustive population of actors involved with PMIL is not possible. Moreover, given substantial differences in involvement of stakeholders, any other selection process, such as random sampling, is likely to miss important sources of information. Hence, purposive sampling is preferred both for feasibility and data quality reasons. Given the large number of planned interviews, concern about sample bias is negligible. b. Interviewee recall bias Key informant interviewing will focus primarily on the interviewees’ opinions, actions and assessments of program activities as participants and collaborators since project inception in 2013. We do not anticipate problems with recall bias given that the project has only recently begun and is still in operation. The EET can identify inconsistencies through comparison across interviews. Questions of fact can be cross-checked against documentation of events and activities available in other project materials. c. Interviewee response bias Interview based data is subject to self-report bias. We have structured this evaluation in ways to reduce the likelihood of three types of systematic response bias: acquiescence bias (i.e., the tendency to agree), demand characteristics (i.e., modifications to responses because of being “studied”), and social desirability bias (i.e., ascribing favorable traits, even if this is untrue). Interview protocols ask open-ended questions and probes are constructed to inquire about multiple perspectives on an issue and similar questions are asked of multiple project members such that it will be possible to recognize issues of response bias. In analysis we will review responses to interview questions such that the responses of different stakeholder’s and different individuals from within stakeholder groups can be cross-checked for convergence or disagreement. Through triangulation of assessments and understandings of different individuals we can come to a set of conclusions that identify areas of consensus and areas of disagreement. While eliminating response bias is difficult, triangulation greatly reduces the PAGE | 72 25 concern. d. Interviewer bias Several sources of interviewer bias are particularly important to be aware of during interviews. Facial expression or inflection of the interviewer can bias responses, encouraging or discouraging particular lines of conversation. In international contexts, race and ethnicity of the interviewer can bias responses of interviewees such that interviewees may be more open with individuals who are more like them. Interviewees can also make the mistake of presenting leading questions or ‘putting words into people’s mouths’. Interpretation of data can become skewed toward one perspective or another although evidence does not merit it. Finally, prior involvement with a particular group or scientist can lead to biased interpretation of data. Although these forms of interviewer bias are difficult to manage, most of them can be addressed through training, interview experience and interview structure. For this evaluation, the EET has substantial interview experience which minimizes these concerns. Additionally, interviews will be team-based when possible, written materials will be shared among EET members and analysis will be vetted as a group. Additionally, to avoid any concerns about bias, the team is careful not to assign individuals to site visits when those individuals have had a high level of prior collaboration. Table 5: List of Interviews Note -- The evaluation team interviewed a wide range of individuals associated with the project. These include members of the Management Entity, PIs and CoPIs, researchers in the US and in host countries, stakeholder organizations and others. The full list of names and contact information has been redacted to preserve confidentiality. PAGE | 73 26 6. Evaluation Work Plan Activities Dates of Activity 1 st Month 2nd Month 3rd Month 4th Month 5th Month 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 TASK 1. Develop evaluation design and implementation plan April 1 -April 25 Activity 1: Desk review & initial interview with PMIL ME April 1 -April 25 X X X Activity 2: Development of evaluation design April 1 -April 25 X X X Activity 3: Develop agenda for PMIL visit and interviews April 1 -April 25 X X X TASK 2: Data Collection April 27 – June 30 Activity 1: Site visit to PMIL ME April 27 – April 28 X Activity 2: Site visit 1, Haiti May 20 – May 30 X X Activity 3: Site visit 2, Ghana May 25 – June 5 X X Activity 4: Site visit 3, S. Africa June 1 – June 10 X X Activity 4: Skype interviews of US and international researchers May 1 – May 30 X X X X X X X X Activity 6: Collection of publication data, bibliometrics May 15 – June 15 X X X X X X TASK 3: Data Analysis May 25 – July 30 Activity 1: Review data collection for gaps (ongoing) May 25 – July 15 X X X X X X X Activity 2: Organize findings, discuss and develop evaluation themes (ongoing) May 25 – July 15 X X X X X X X Activity 3: Compile preliminary findings; ppt presentation July 31 July 7 – July 31 X X X TASK 4: Report Writing August 1 – August 31 Activity 1: Draft report prepared & submitted to USAID by August 21 August 1 – August 21 X X X Activity 2: Final report submitted by August 31 August 15 - August 31 X X PAGE | 74 27 B. Data Collection and Management Plan 1. Survey Training Evaluators are experienced in interview techniques. No special training is necessary to use the interview protocols. Interviews will be conducted in English. If a translator is necessary, the EET will arrange for one, verify the qualifications needed and provide any needed training to the translator. 2. Data Management and Security Data collection and management have been described in Section 4a. Evaluators are experienced with the confidentiality requirements needed for work with human subjects. All responses are considered confidential. If there is a need to reference a particular PI or co-PI, the EET will request permission from the individual. All notes and findings will be collated anonymously and no direct reference will be made to any individual respondent. Evaluators will not identify individual respondents either in reports or to USAID during debriefings. 3. Data Collection Approvals Data collection approvals will not be needed as this data collection is not for research, and the only information collected is that given by the subjects themselves with their agreement. Confidentiality of responses will be maintained, and no names or identification will be attached to responses. Interview notes will be coded with the general category of stakeholder and individual names kept separately to avoid potential for disclosure. 4. Data Collection Instruments The following section contains two interview protocols as examples of the type of the format and types of questions that will be asked during interviews. Interview protocols will be developed for each type of group interviewed: Management Entity, Stakeholders, EAP members, PIs and CoPIs, other collaborators, students and postdocs. Responses to these questions will be collected and analyzed as presented in Section A. PAGE | 75 28 a. Protocol for Interviews of Project PIs, CoPIs and Collaborating Researchers Introduction The purpose of this interview is to assess the perspectives of member of the research teams funding under the Peanut and Mycotoxin Innovation Lab in general and the XX project specifically. We are conducting confidential interviews of PIs, Co-PIs, Collaborating researchers in the US and in partner countries, as well as other relevant stakeholders. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. Our aim is to better understand the how research activities are proceeding, how the various PMIL projects interact with each other and external groups and what factors might improve PMIL’s ability to attain its objectives. I have several questions that will take about 45 minutes of your time. Do you have any questions before we begin? Questions 1. Could you first describe the research that you are doing with the PMIL? Prompt: Can you help me understand how your project addresses one or more of the strategic objectives of the Peanut and Mycotoxin Innovation Lab? 2. Thinking back to the beginning of the project in 2013, is the research progressing as you expected? Prompt: Can you point to some particular activities that highlight your progress? (e.g. journal articles, new breeding lines, etc.). Prompt: Are there activities that are ahead of schedule? Is anything falling behind? Prompt: Can you give me some examples? What do you think are the main reasons why these PMIL activities are not progressing as planned? Or ahead of schedule? Prompt: Have you had to develop workarounds to address the challenges? Is there anything you can think of that would help you to better accomplish your PMIL research? Prompt: Looking ahead, are there research activities that were proposed that you may not be able to accomplish? 3. Could you please describe your one or two most important research findings for US research in Peanuts or Mycotoxin? PAGE | 76 29 Prompt: How do you judge the quality of the various research activities under your project? 4. The project is built on a value chain approach. In what ways does your research feed into or affect the value chain in target countries (or the target country you are most involved with)? Prompt: How has your research contributed to change or better conditions in the PMIL target country? Prompt: In what ways are the research findings being integrated/uptaken by local actors? Prompt: Could you describe the relevance of the research findings to the particular countries you are working in? What about for other countries or populations? 5. Are there any significant changes to the project due either to significant challenges you have encountered or due to important findings or discoveries from the research? Prompt: Can you give me some examples? Prompt: Do you see possibilities for new directions in a new research program? Explain? Prompt: Have there been any missed opportunities? Can you give me some examples? Prompt: How would your current project be adjusted to meet these new themes? Would the project (ME) be able to provide additional resources, would you have to cut back on other activities? 6. In what ways does your research project or the PMIL program in general consider gender? Prompt: Can you give us some detail on any gender specific findings, outputs or outcomes of your PMIL work? Prompt: What about participation of women in the project either as researchers or as local partners within the value chain. How are women integrated in the various parts of project? 7. This project includes people from multiple countries and institutions. Is there sufficient communication and coordination among PMIL collaborators? Prompt: To what extent is decision making include perspectives from multiple partners or collaborators? PAGE | 77 30 Prompt: How well does the international research collaboration work? Are there ways you think that collaborative interaction could be improved? Prompt: To what extent to you feel like there is a team-based approach to the research? Prompt: Who on the project is your principal contact? Who else to you communicate/coordinate with in other projects? Prompt: On average, how often do you receive communications from the ME/PI? What proportion of these communications is a request for information? What proportion is giving you information? Prompt: Please describe the process by which your project assembles its annual workplan and annual report. Is there a meeting of all scientists? 8. This project is one of many under the PMIL program. How well does the management office at UGA manage the PMIL program? Prompt: What is your experience with the management of the PMIL? a. Professional/collegial interactions b. Fiscal interactions (promptness, clarity) c. Promptness and effectiveness of managing problems d. Quality of the solutions to problems e. Effectiveness and efficiency of the sub-award process f. Clarity of goals and objectives g. Degree to which planning is collaborative h. Degree to which data collection and analysis are supported i. Degree to which findings are appropriately disseminated Prompt: What could the ME do to more effectively meet the Peanut and Mycotoxin IL’s objectives? Prompt: What strengths or synergies did this model encourage, e.g., among the individual projects to ensure complementarity of effort? 9. Is it important for your project to establish partnerships with local organizations or agencies, universities, partners and other and NGOs? Why or why not? Prompt: If so, what types of connections have you established? What have you done with partner organizations that advance the strategic objectives of your project? Prompt: Do you think the partnerships work well? Are there partnerships that seem to work better than others? Why? Prompt: Have there been important contributions from these stakeholders or partners to your research? Could you give us some examples? PAGE | 78 31 10. From your perspective, has there been much cross-project interaction and learning? Can you point to cross-project or program-level results or outcomes that have occurred? Prompt: How do you interact with scientist from other Peanut and Mycotoxin IL research projects? Do meetings regularly occur, and how do they come about? 11. One of the objectives of the IL is to build local capacity for research. From your perspective is the program accomplishing this? Prompt: Have you received additional training as a result of the project? If so, please describe. If not, have you asked for such training? What was the resolution of your request? Prompt: How does the project identify perspective students for advanced study (e.g at a national or international university)? Is gender a consideration? How has the gender balance been? 12. What is your assessment of the strengths and weaknesses of the PMIL in general, and your specific project (s) in particular? b. Protocol for Interviews of ME personnel Introduction The purpose of this interview is to assess the perspective of the management entity and the operations of the PMIL team in general, and the progress of the project. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. Our aim is to better understand the how research activities are proceeding, how the various PMIL projects interact with each other and external groups and what factors might improve PMIL’s ability to attain its objectives. I have several questions that will take about 45 minutes of your time. Do you have any questions before we begin? Questions 1. Could you first describe the overall program objectives? PAGE | 79 32 2. Thinking back to the beginning of the project in 2013, is the PMIL research progressing as you expected? Why or why not? Prompt: Can you point to some particular activities that highlight your progress? (eg journal articles, new breeding lines, etc.). Prompt: Are there activities that are ahead of schedule? Is anything falling behind? Prompt: Can you give me some examples? What do you think are the main reasons why these PMIL activities are not progressing as planned? Or ahead of schedule? Prompt: Have you had to develop workarounds to address the challenges? Is there anything you can think of that would help you to better accomplish your PMIL research? Prompt: Looking ahead, are there research activities that were proposed that you may not be able to accomplish? 3. In what ways is research in the two themes of peanut production and mycotoxin detection being integrated? Prompt: What are the key areas that you see leading to advances or contributions for US science and agriculture? Prompt: To what extent is it important to have linkages between the peanut and mycotoxin themes in order to produce advances? Can you help us understand how the two themes leverage each other on the research side? Prompt: In what ways has PMIL been an effective catalyst for this integration? Prompt: What could be done differently to improve the integration of the two for research purposes? 4. The project is built on a value chain approach. How has the research made a difference in the peanut value chain in target countries? Prompt: How has the research contributed to change or better conditions in the PMIL target country? Prompt: In what ways are the research findings being integrated/uptaken by local actors? Prompt: Could you describe the relevance of the research findings to the particular countries PMIL has targeted? What about for other countries or populations? 5. Are there any significant new directions on PMIL due either to significant challenges encountered or to important findings or discoveries from the research? Prompt: How has PMIL been able to provide additional resources toward new opportunities or promising directions? PAGE | 80 33 Prompt: Does the PMIL program structure or budget allow short-term changes to facilitate investigation of newly emerging research themes? Prompt: Have there been any missed opportunities? Can you give me some examples? 6. How has PMIL included consideration of gender in its design and implementation? Prompt: Can you give us some detail on any gender specific findings, outputs or outcomes of PMIL’s work? 7. This project includes people from multiple countries and institutions. Is there sufficient communication and coordination among PMIL collaborators? Prompt: To what extent is decision making include perspectives from multiple partners or collaborators? Prompt: How well does international collaboration work on this project? Do you feel like there is a team-based approach to the research? Why or why not? Are there ways you think that collaborative interaction could be improved? Prompt: Please describe the process by which PMIL assembles its annual workplan and annual report. Is there a meeting of all scientists? 8. How successful has PMIL been in fostering partnerships with local organizations or agencies, universities, other stakeholders and NGOs? Prompt: What types of connections has PMIL established? What has PMIL done with partner organizations to advance the aims of the project? Prompt: Do you think the partnerships work well? Are there partnerships that seem to work better than others? Why? Prompt: Have there been important contributions from these stakeholders or partners to PMIL research? Could you give us some examples? Prompt: Are there important countries or partners that are being left out of the PMIL program? How does that affect the project? 9. From your perspective, has there been much cross-project interaction and learning? Can you point to cross-project or program-level results or outcomes that have occurred? Prompt: How much do scientists from the different Peanut and Mycotoxin IL research projects interact? Do meetings regularly occur, and how do they come about? What do they produce? PAGE | 81 34 10. One of the objectives of the IL is to enable training and build local capacity. From your perspective is the program accomplishing this? Prompt: How does the project identify perspective students for advanced study (e.g at a national or international university)? Prompt: How does the PMIL/LIL Scholar Program work? What are the accomplishments? What are the challenges? 11. What is your assessment of the strengths and weaknesses of the PMIL both from a research perspective and from a value chain intervention perspective? c. Protocol for Interviews of Project Stakeholders and Partners Introduction The purpose of this interview is to assess the perspectives of stakeholders and partners involved with or knowledgeable about the Peanut and Mycotoxin Innovation Lab. We are also conducting interviews of other researchers involved in the innovation lab. Our aim is to better understand the how research activities are proceeding, how the various PMIL projects interact with each other and external groups and what factors might improve PMIL’s ability to attain its objectives. I have several questions that will take about 45 minutes of your time. Interviews are confidential; results will be generalized such that it will not be possible to attribute comments to individuals. We expect that you are here on a voluntary basis. Please let me know if you prefer not to conduct this interview. Do you have any questions before we begin? Questions for non-PMIL key informants (partner or potential partner) 1. Could you first discuss how you have been involved with (or know about) the Peanut and Mycotoxin Innovation Lab. Prompt: Do you work with a particular project or projects? Prompt: Who do you work with? What activities have you been involved in? Prompt: Do you consider your work to be linked to a particular part of the peanut value chain? Is your work with PMIL related to that part of the value chain? 2. From your perspective, how is the project going? Is it proceeding as it was planned? Prompt: How do the two themes – peanut production and mycotoxin detection – PAGE | 82 35 interact from your perspective? In what ways has this had a positive impact in your country? Are there any particular challenges or gaps? Please explain. Prompt: Do you know about any key research findings from the project? Could you provide specific examples? Prompt: Can you provide specific examples of PMIL technologies or research findings that were taken up or used by other organizations? Prompt: What about other efforts, such as the dissemination of information, knowledge or other outputs? Prompt: Is the size and capacity of the project in line with its aims? 3. From your perspective, how is the project meeting its potential? What could it be doing differently that would improve its eventual impact? Prompt: Are the research themes addressed by PMIL scientists appropriate from your perspective? Could you provide some detail? Prompt: Are there topics or directions that are not being address that should be? Prompt: Are there stakeholders that are not involved that should be? Are there some partnerships that are not productive or should be either reframed or dropped? Prompt: What do you think could be improved to boost the quality of the research? 4. How does the PMIL research project complement and or advance actions/research programs of other organizations whether national or international? Prompt: Please provide an example or two of that complementarity. Prompt: Can you provide specific examples of PMIL technologies or research findings that were taken up or used by other organizations? Prompt: Does the work contribute to the field in a significant way? If so, how? What niche is it filling that other projects are not? 5. From your knowledge, in what ways does the PMIL enable and/or provide training opportunities for researchers and others? Prompt: Does your organization participate in PMIL research or training? Please describe the modalities of this participation? Prompt: Are the proper people being trained in the proper fields and at the proper level? Prompt: Are women adequately represented in the training program? PAGE | 83 36 Prompt: Are researchers working in the developing world well represented in the training program? Prompt: Is the PMIL properly prioritizing its training (undergraduate/graduate)? Are certain disciplines under-represented? Prompt: Have you seen any outcomes of the training – such as people from your organization who have been able to contribute to new research or activity? 6. How well does PMIL manage coordination and communication among PMIL partners or stakeholders? Prompt: How often do you communicate with PMIL? Do you feel like you are included in substantive dialogue about the program, findings, interventions and ideas? Prompt: To what extent do you feel like your perspectives and knowledge is considered valuable by PMIL? Could the communication be better? How? Prompt: To what extent to you feel like there is a team-based approach to the research? Do you feel like it is a true collaboration? 7. Has the PMIL and your partnership run into any obstacles, constraints or challenges? Prompt: Could you provide an example? Prompt: What did PMIL do to address it? Do you think PMIL was responsive? Prompt: Were the issues well resolved? Are there issues still to resolve? 8. We have talked about many different things. Could you help us understand more about any important outcomes, advances or issues that we have not discussed? PAGE | 84 37 C.Curriculum Vitae of Researchers [Ed. Note: REDACTED] D.Evaluation Scope of Work [Ed. Note: See Appendix A] E. Knowledge Gaps Table [Ed. Note: See Appendix A] F. Abstract of the Twelve PMIL Projects A. Peanut Germplasm Development 1. A1. Translational Genomics to Reduce Pre- harvest Aflatoxin Contamination of Peanut Lead PI: Peggy Ozias-Akins, University of Georgia The goal of the project is to associate molecular variation with resistance to pre￾harvest aflatoxin contamination on a genome-wide scale and to begin to utilize this information in breeding programs. To achieve the goal, both genotyping and highly replicated phenotyping of genetic resources and populations for aflatoxin contamination are being pursued. Genotyping with genome-wide SNP (single￾nucleotide polymorphism) markers is being enabled by peanut genome sequence information, both from cultivated tetraploid genotypes as well as diploid progenitors PAGE | 85 38 of the tetraploid. Genetic populations are being developed in India, Senegal and the USA. Phenotyping is being done under controlled field conditions in Niger, Senegal and the USA. 2. A2. Silencing of Aflatoxin Synthesis through RNA Interference (RNAi) in Peanut Plants Lead PI: Renee Arias, USDA-ARS National Peanut Research Laboratory The overall goal of this project is to use RNA interference (RNAi) to reduce aflatoxin contamination of peanut seeds. The research has two main objectives funded by complementary sources: 1) the study of genetic diversity of aflatoxigenic Aspergillus species funded by PMIL, and 2) the genetic transformation of peanut plants using RNA interference that is funded by NBCRI. For the genetic diversity studies, samples are being analyzed from Ethiopia, Kenya, Malawi, Uganda, Zambia and the USA and fingerprinted using sequences within the aflatoxin synthesis gene cluster at the National Peanut Research Laboratory (NPRL) in Dawson, Georgia. Three African peanut varieties (CG 7, JL 24 and ICGV 90704) are being transformed at Kenyatta University in Nairobi, Kenya, using RNAi molecular constructs provided by NPRL. Scientists at the NPRL are providing training and backstopping to the African scientists in the project, many of whom have visited the NPRL for hands-on training. 3. A3. An Integrated Global Breeding and Genomics Approach to Intensifying Peanut Production and Quality Lead PI: Mike Deom, University of Georgia The overall goal of this project is to use conventional and molecular breeding to enhance the productivity, quality and marketability of peanut in PMIL target countries. The proposed research focuses on intensifying the biotic resistance, abiotic tolerance and quality aspects of peanut varieties through partnerships with USA and developing country breeding programs. Biotic stresses include resistance to economically important pathogens and pests, while the primary abiotic stress addressed is drought tolerance and avoidance, a trait that factors into mitigating aflatoxin contamination. The breeding programs also focus on value added traits, including high oleic content (nutrition and shelf-life), increased micronutrient density (iron and zinc), high oil content (cooking oil and butter) and large seeds (edible market). Outreach programs are being used to stress technology transfer and the PAGE | 86 39 value of new cultivars and system considerations for utilizing appropriate crop￾management strategies. Considerable resources are directed to host countries for capacity building, including student training, scientist training and infrastructure improvements. As advanced varieties become available, they are distributed to PMIL target country collaborators and PMIL value chain projects for evaluation as well as other developing countries that request the material. The outcomes of the research include increased yields and increased quality. Subsequent benefits result in improved peanut value chains, increased food security, better nutritional and dietary traits and increased income throughout PMIL target countries as well as other developing countries. Capacity building results in in-country knowledge, expertise and improved infrastructure, which build a foundation to continue improving peanut yields and quality. B: Mycotoxin Detection and Peanut Nutritional Studies 4. B1. AflaGoggles for Screening Aflatoxin Contamination in Maize Lead PI: Haibo Yao, Mississippi State University Aflatoxin contamination in maize and peanut is a major food safety issue worldwide. The problem is of special importance in African countries because these crops, among others, are staple foods. A primary limitation to controlling ingestion of contaminated food in these countries is the lack of affordable and feasible methods for farmers on small village farms to screen for aflatoxin contamination. Due to the high cost associated with any existing aflatoxin detection methods and the need for sample processing and detection, there is an urgent need to develop portable, rapid, and non￾invasive technology for aflatoxin detection in maize and peanut for these farmers. Therefore, the goal of the project is to develop portable, fluorescence spectral-based technology for rapid and non-invasive aflatoxin detection in maize (and peanut). A detection device for this purpose will be developed in the project. 5. B2. Development and Validation of Methods for Detection of Mycotoxins Exposure in Dried Spotted Blood Samples Lead PI: Jia-Sheng Wang, University of Georgia PAGE | 87 40 The goal of this project is to establish and validate methods for measuring major mycotoxin biomarkers, especially for aflatoxin-lysine adduct, in human dried blood spot (DBS) samples for supporting urgent needs of nutrition impact and intervention studies conducted in Asia and Africa countries by PMIL, as well as the Nutrition Innovation Laboratory at Tufts University. The methods will be validated and applied to assess susceptibility factors in determination of human aflatoxicosis, to evaluate the linkage between aflatoxin exposure and human nutrition deficiency and growth retardation and developmental inhibition in children. 6. B3. Aflatoxin in Peanut and Peanut Products: Comparative Study on Analytical Methods for Detection of Aflatoxin Lead PI: Kumar Mallikarjunan, Virginia Polytechnic Institute and State University There are numerous methods to measure the toxicity of fungal infection in various crops. A primary limitation for aflatoxin determination in peanuts is the lack of generally accepted and standardized methods for farmers to screen or for testing laboratories to quantify the level of contamination. Even among PMIL collaborators, different evaluation methods have been reported in individual studies, making the comparison of results difficult. This project conducted a systematic comparative study to evaluate and report existing/emerging analytical methods for aflatoxin determination in peanuts and peanut products. A blind test, in which the variety of peanut products was naturally and artificially contaminated with aflatoxin, was prepared to test the current available analytical methods within the collaborating institutions/analysis laboratories. Results from the project were helpful to document the existing methods, the advantages/disadvantages of each method, and which method is best for each objective. 7. B4. Randomized Controlled Trial of the Impact of Treating Moderately Malnourished Women in Pregnancy Lead PI: Mark Manari, Washington University, Saint Louis The objective of this project is to determine the benefits of treating moderately malnourished pregnant women with a peanut butter-based nutritional supplement. The trial is a randomized, investigator- blinded controlled clinical effectiveness trial in pregnant women with moderate malnutrition, with and without HIV-infection, in PAGE | 88 41 southern Malawi. The trial is using three different nutritional supplements for comparison: (1) a Ready-to-Use Supplementary Food (RUSF) formulated to deliver about 200% of the RDA of most micronutrients in pregnancy (RUSF-P); (2) fortified corn soy blend (also known as CSB+ or super-cereal) with a multiple micronutrient tablet chosen to deliver about 200% of the RDA of most micronutrients (CSB-P); or (3) the standard of care which is a fortified corn soy blend, vegetable oil and sugar with supplementary iron and folic acid tablets (CSB), delivering between 0-350% of the RDA. The primary outcomes for this study are both maternal; recovery and Mid￾Upper Arm Circumference (MUAC) change, as well as infant outcomes in mean birth weight, mean birth length, and percentage of premature delivery. The aim of the study is to provide significant evidence that using a peanut-based supplementary food will reduce maternal mortality and improve infant growth and development. This will provide national and international agencies with evidence to recommend and promote the use of peanut-based products for maternal health, as well as purchase some for use in their nutrition programs. C. Peanut Value Chain Interventions 8. C1. Production to Consumption – Technologies to Improve Peanut Production, Processing and Utilization in Haiti Lead PI: Greg MacDonald, University of Florida The overall goal of this project is to address and mitigate key constraints to peanut production and utilization in Haiti. Peanuts have been and continue to be an important part of Haitian diet and culture. In addition, peanuts provide an important source of cash income. To combat malnutrition in the country, certain NGOs have developed facilities to produce peanut-based Ready-to-Use Therapeutic Food or RUTF. To date, however, there has been limited utilization of locally grown peanut due to issues with productivity, quality and aflatoxin contamination. In this project, we are developing a comprehensive production, processing and utilization strategy for peanuts in Haiti. All phases of peanut production are being evaluated, including varieties specific to the region and market influences. We are instituting a seed-increase program and developing facilities to maintain genetic resources through curation of important peanut germplasm. Capacity building through the introduction of labor saving devices and harvesting equipment and PAGE | 89 42 procedures is underway, along with evaluating the infrastructure to improve peanut handling, drying and long-term storage. Once these improvements have been evaluated, we take the best management practices and strategies to the grower level at several villages and communities in the region, particularly through the depot network partnership with the Acceso Peanut Enterprise Corporation. We are providing training and infrastructure support to realize these improvements and ensure long-term capacity building. Aflatoxin and the role of women in the peanut value chain is being measured/surveyed throughout the duration and in all phases of the project. We are also establishing aflatoxin-testing facilities and re-training Haitians in how to measure and the importance of avoiding aflatoxin in their diet. Another important capacity-building measure is the creation of alternative products/markets for high aflatoxin contaminated peanuts. 9. C2. Using Applied Research and Technology Transfer to Minimize Aflatoxin Contamination and Increase Production, Quality and Marketing of Peanut in Ghana Lead PI: David Jordan, North Carolina State University A wide range of abiotic and biotic stresses negatively impact peanut production in the field and generally contributes to the reduced quality of marketed peanut in Ghana and West Africa. Aflatoxin contamination can occur and increase at all steps of the peanut supply chain including production in the field, storage in fields and villages, and in processed products. Interventions at each step of the supply chain can minimize aflatoxin contamination. Improved production in the field including pest resistant cultivars, adequate soil fertility and plant nutrition, and synchronization of peanut pod growth phase with adequate soil moisture can increase peanut yield and quality and minimize aflatoxin contamination. Adequate and timely drying of farmer stock peanut minimizes additional production of aflatoxin during storage in villages prior to marketing. Effective processing of farmer stock and shelled stock peanut can also reduce aflatoxin prior to purchase and consumption. Determining current practices by farmers, conducting research to mitigate aflatoxin and improve peanut quality, and transferring appropriate technology to farmers are needed to improve productivity, profits, and quality of peanut and to increase safety of peanut products consumed by humans and livestock. PAGE | 90 43 The primary platform being used to research aflatoxin contamination of peanut in the supply chain in Ghana is taking place in nine villages in northern and central Ghana. Interventions at each step of the supply chain are being implemented and aflatoxin contamination determined. Research is conducted at two institutions associated with the Savanna Agricultural Research Institute (SARI) and at the Crops Research Institute (CRI) to develop appropriate production and pest management strategies and to evaluate new germplasm suitable for the region. Results from efforts at villages and research stations are presented to farmers using the Farmer Field School approach and appropriate posters, bulletins and manuals. Graduate student training is closely linked to activities in villages and research stations. Results from the project are providing farmers in Ghana with information on documented interventions that reduce aflatoxin contamination of peanut throughout the supply chain. Improved productivity and quality of peanut coupled with acceptable levels of aflatoxin in peanut products improve access to local, regional, national and international markets leading to enhanced economic viability of farmers and their communities. 10. C3. Producer and Consumer Interventions to Decrease Peanut Mycotoxin Risk in Ghana Lead PI: Nicholas Magnan, University of Georgia The goal of this project is to investigate the relative and combined impact of technological and market aflatoxin mitigation interventions for groundnuts in northern Ghana. The technological intervention will facilitate the adoption of simple and low-cost aflatoxin prevention technologies. Essentially, we are giving a randomly selected subset of study farmers the materials and information necessary to adopt. We worked with local experts to identify the preventative measures with the best potential to provide long-term and affordable solutions. The market intervention ensures a premium for a different and partially overlapping randomly selected subset of study farmers. To do this, we work with local groundnut buyers to offer a premium for groundnuts tested by the project that pass a safety criterion. Producers selected to receive the market intervention are made aware of the potential customers for safe groundnuts, and what the standards are to qualify for the price premium. PAGE | 91 44 In Ghana, women constitute over 48% of the agricultural labor force. Furthermore, women are the main purchasers of groundnuts, and then use them to make paste and extract oil. Hence, when designing the questionnaires and intervention we considered gender differences. We built in modules on gender, individual assets and joint asset ownership at baseline. In this way, we have attempted to capture the gender dynamic around reasons why/why not individuals or households adopt control measures. 11. C4. Aflatoxin Management Interventions, Education and Analysis at Various Steps Along the Peanut Value Chain in Malawi, Mozambique and Zambia Lead PI: Rick Brandenburg, North Carolina State University This project addresses a wide range of production, post-harvest handling, and processing issues relative to peanuts in Malawi, Zambia, and Mozambique that can impact aflatoxin contamination levels, yield, and profitability. The strength of this project is that interventions are being evaluated throughout the value chain and the cumulative effect of these efforts is being measured against traditional production and marketing practices. Through linkages with various partners, farmer education will be emphasized and extended linkages with various industries and marketing groups will help accelerate aflatoxin mitigation and market development. Malawi has a strong history of research on peanut through ICRISAT, the Department of Agriculture Research at Chitedze Research Station, and Lilongwe University of Agriculture and Natural Resources (LUANAR), but the ability of farmers to produce high yielding, high quality peanuts with consistently low aflatoxin levels is still quite limited. Additional agencies such as NASFAM (National Small Farmer Association of Malawi), the Ministry of Agriculture, Exagris, Afri-Nut, TWIN of the U.K., the Clinton Development Initiative and others are all engaged in further evaluation of production, processing, and marketing strategies as well as farmer education. Improved cultivars are available, but the lack of an effective seed program limits availability. Limited marketing due to high aflatoxin contamination levels exacerbates the problem by reducing farmer incentive to implement current production recommendations and limits commercial processing and marketing. Our project, with its multidisciplinary team, takes a comprehensive approach to problem-solving research and effective technology transfer through key partnerships PAGE | 92 45 with in-country research counterparts and NGOs. The higher level of peanut research in Malawi will be expanded and emphasis placed on implementation and additional research efforts will be rapidly phased in to Zambia and Mozambique creating a regional project providing research data with even wider scale application. Key components include taking advantage of improved germplasm already available, in￾country aflatoxin testing equipment and technicians already in place, key production, processing, marketing and technology transfer partners. Our project is addressing the challenges from production to processing including information transfer and creating aflatoxin awareness along the whole value chain. 12. C5. Productivity and Profitability Growth in Peanut Production: A Farm Level Analysis in Malawi, Mozambique and Zambia Lead PI: Boris Bravo-Ureta, University of Connecticut The overarching objective of this project is to generate and transfer economic knowledge needed to intensify groundnut production, and its subsequent use, so as to significantly increase productivity and farm profits, while reducing the risk of aflatoxin contamination in the harvested crop. The end goal is to boost productivity growth in groundnut farming systems as a way to increase food safety, food security, and farm income in Malawi, Mozambique, and Zambia. This work will be done in close collaboration with the Southern Africa Value Chain and Integrated Breeding Projects. A fundamental underpinning of the project proposed here is that a major constraint to a healthy groundnut value chain in much of Africa is low levels of farm productivity and profits. Productivity and profits can be improved in various ways, including gains in marketable yields. Thus, the primary focus of this project is to analyze the farm level costs and benefits of alternative treatments designed to reduce the aflatoxin levels with the goal of increasing peanut quality and prices received by farmers. A second area of work is to utilize available data from the World Bank Living Standard Measurement Studies-Integrated Surveys on Agriculture (LSMS-ISA) and variety data generated by the Integrated Breeding Project to evaluate the farm benefits of improved seed varieties, particularly in Uganda and Malawi. PAGE | 93 46 A third area of work will be to undertake human capacity building through workshops in various topics including production economics, farm management principles and/or impact evaluation techniques. PAGE | 94 Page | 95 APPENDIX C. TRAVEL ITINERARY, LOCATIONS AND DATES Country Team Members Dates University of Georgia Joan Fulton and Farid Waliyar April 27, 2016 – April 28, 2016 Ghana Medson Chisi and Farid Waliyar May 26, 2016 – June 2, 2016 Zambia/ Malawi Joan Fulton and Medson Chisi June 1, 2016 – June 9, 2016 Haiti Joan Fulton and Farid Waliyar June 15, 2016 – June 22, 2016 Page | 96 APPENDIX D. LIST OF PERSONS CONTACTED The evaluation team interviewed over 50 individuals having a wide range responsibilities and roles on the PMIL project. As anticipated in the evaluation plan, interviews were conducted in person on site visits to the ME at UGA and in host countries, by telephone and on skype. Interviews included individuals from the following groups: Project PIs and CoPIs  US collaborators on research projects  International collaborators on research projects  External advisory panel members  Partner organizations local government and NGO partners, and other stakeholders  EAP members  ME personnel including the PMIL director and Co-director  USAID Mission and other in-country staff The full list of names and contact information has been redacted to preserve confidentiality. Page | 97 APPENDIX E. LIST OF MATERIALS REVIEWED Annual Reports Annual Report, Feed the Future Innovation Lab for Collaborative Research on Peanut Productivity and Mycotoxin Control, 2014 Annual Report, Feed the Future Innovation Lab for Collaborative Research on Peanut Productivity and Mycotoxin Control, 2015 Other Documents Groundnut Seed Production Guidelines (English) or Ndondomeko Zakalimidwe Ka Nshaba (Chewa), 2016 Controlling Aflatoxins (English) 2016 The Feed the Future Peanut & Mycotoxin Innovation Lab - Facilitating U.S. Scientists to Solve Global Problems, 2016 Feed the Future Aflatoxin Interventions (English), 2015 Peanut & Mycotoxin Innovation Lab Program Brochure (English), Published: 2016 Validation and adoption of a novel method of aflatoxin detection in peanut butter using a tablet, Published: 2016 Why control aflatoxin? (English), Meds & Foods for Kids, 2015 A Guide to Peanut Production on the Rupununi Savannas (English) or Gid pou Pwodiksyon Pistach nan Savann Rupununi (Creole), The Beacon Foundation and Peanut Growers of the Rupununi Savannas Guyana, South America, 2007, 2014 PMIL Mycotoxin Webinar: Part 1 Designing Mycotoxin Sampling Plans, Aflatoxin Sampling Webinar Raleigh, NC USA December 2014 PMIL Mycotoxin Webinar: Part 2 Representative Sampling for Mycotoxins, Aflatoxin Sampling Webinar, Raleigh, NC USA December 2014 PMIL Mycotoxin Webinar: Part 3 FAO Mycotoxin Sampling Tool, Aflatoxin Sampling Webinar, Raleigh, NC USA December 2014 Page | 98 MANAGEMENT ENTITY RESPONSE TO THE EXTERNAL EVALUATION REPORT The Feed the Future Innovation for Collaborative Research on Peanut Productivity and Mycotoxin Control (Peanut & Mycotoxin Innovation Lab, PMIL) Management Entity (ME) is grateful for the thorough and thoughtful report prepared by the External Evaluation Team (EET) and recognizes the significant effort made by the team to provide suggestions on ways to improve the quality of research being conducted within the Innovation Lab. In general, the ME is in agreement with most points and recommendations made in the report, both complimentary and critical. The following comments follow the report organization by project or thematic area and are followed by responses to some of the general recommendations. Genetic Resources and Breeding Projects The ME is fully supportive of the comments in the areas of genetic resources and breeding, especially in regard to improvement of infrastructure for collaborating international scientists in areas of genomic selection and for improvement of breeding programs. The recent availability of new tools for genotyping and phenotyping offer exciting opportunities to enhance the efficiencies of all breeding programs. The ME would like to emphasize the emerging success of the regional breeding programs being led by Host Country (HC) scientists in recognition of the suggestion to further include HC scientists in project design and implementation. We agree that this should be a focus in all aspects of the project, not just breeding. Regarding the efforts to address aflatoxin contamination via an RNAi strategy, the ME notes that the objective of this project was to develop a proof-of-concept for a potential “game changing” technology. A thorough review of the future potential of this technology is important and would need to integrate suggestions regarding local regulatory acceptance and capacity to develop/evaluate/release the technology in target countries. The comments regarding integration of research on seed systems and scaling of adoption are valid. PMIL will continue to reach out to existing initiatives currently underway in target countries aimed at improving seed systems and scaling of improved varieties, especially those funded by USAID missions. PMIL scientists are involved with several of these initiatives and have worked to include varieties developed and evaluated by PMIL to leverage greater impact. However, PMIL must maintain a balance between research objectives and scaled development objectives given limited resources. National program collaborators also must work within the structure of their national initiatives and regulations on variety release and seed multiplication. Where possible, linkages with the private sector are critical for long-term sustainability of seed systems. Page | 99 Mycotoxin Detection Technologies During the initial project meetings of PMIL, concerns were raised over differing methods of sampling and detection for aflatoxin across projects and the potential for variability that may limit comparison of results. The high cost of analysis and difficulties associated with shipping of samples and bottlenecks in analysis were also raised as problem areas. As a result, the ME began two initiatives: 1) instructional materials on sampling and sample preparation based on the work of the USDA experts, now publically available on the PMIL website; and 2) the validation of the Mobile Assay mReader and Neogen lateral flow strips as an “off the shelf” technology that met many of the specified criteria of low cost, safety, ease of use, potential for field level analysis, and accuracy of detection. This validation was done as an addition within two projects with assistance from the ME. Once validation for peanut products was completed, training was conducted in several locations to facilitate the appropriate use of this technology where desired. In several cases, it has reduced budget amounts committed to analysis and opened the potential for analysis in diverse locations, without the need for extensive laboratory investments or shipping to a separate lab. The ME agrees with the suggestion for further research in low/no cost detection methods, such as the “Aflagoggles” technology, but also in novel ways of tackling the issue of extreme variability in sampling and sample preparation as the primary sources of error, as noted by the EET. Concerning the relative outlier status of the Dried Blood Spot (UGA-210) and Prenatal Nutrition in Malawi (WU-206) projects, the ME recognizes the value of intellectual diversity across the overall PMIL. Inclusion of these collaborators has helped maintain PMIL’s alignment with the Feed the Future focus on nutrition and health impacts, provided a diversity of viewpoints during project evaluation at annual meetings and offered opportunities for collaboration. PMIL feels that inclusion of nutrition projects may also emphasize the health benefits of peanut consumption to counterbalance the negative perception of peanuts due to risks of potential aflatoxin contamination. Value Chain Projects Haiti The ME agrees with the comments on the Haiti project. The limited focus and number of actors on the team has improved the integration of the value chain aspects of the project. Improvements could be made with further development of local capacity and infrastructure in breeding, mycotoxin detection, and technology evaluation, and should be pursued in the future to enhance current collaboration. The ME is planning to conduct an impact assessment of this project before the end of the current phase. Ghana The ME recognizes the limitation of a more integrated value chain approach in the Ghana project and welcomes the ideas of conducting a value chain workshop for collaborators and investing in the use of innovation hubs. Further private sector integration could assist with a more direct “pull” factor that may help align the disparate projects along the value chain. Page | 100 Though PMIL has assisted with development or improvement of labs with detection capacity at both Kwame Nkrumah University for Science and Technology (KNUST) and University for Development Studies (UDS), the bottleneck in the analytical step may be resolvable using other technology or adding further capacity and should be evaluated, as suggested by the EET. Though the two projects (NCSU-208 and UGA-205) are geographically co-located and scientists have collaborated as necessary, they also have maintained a balance of focus on their specific research objectives. The ME intends to conduct an impact assessment of this project before the end of current phase. Southern Africa The ME agrees that support is needed as the project continues to mature. Overall The EET noted the limited integration of Value Chain projects in general. Learning from this current phase, future programs should consider various approaches to value chain research to better develop the proposals. The ME notes that current projects range from a relatively small and targeted program in Haiti, to a larger and more complex, but somewhat focused geography in Ghana, to extremely complex, multiple geographies in Southern Africa. Our observation is the relative scale and complexities correspond to the level of integration and collaboration within the value chain as required by the number of actors involved and difficulties and expense of managing large, diverse groups. Future projects may enhance integration by narrowing the geography and product targeting. Further ME engagement in direct management of value chain issues, as suggested by the EET, is also a reasonable option given the limitations of scientists to their specific fields, desire to focus on research and training, and availability. Clarity around these roles and responsibilities has positively evolved as the current projects have matured. Administrative and managerial roles of Principal Investigators should be reduced in favor of simplifying contracts and greater ME integration when possible. Comments on Recommendations The ME will continue its efforts to improve upon the Title XII dual benefits to host countries and the US, both in communicating benefits to the US under the current phase, and also to seek input for areas of improvement in the future. Much of the ME communications focus has been in this area and will continue to improve integration with US industry events to share the global perspective and align research interests. The ME supports the comments on use of fixed term contracts for overseas partners and will continue to work on simplifying these processes with collaborator administrators to assure timely flow of funds. The ME acknowledges the comments concerning the limited integration of gender issues. While disaggregating data based on gender and training a high proportion of female students is important, attention should be placed on better integrating gender analysis in future work. The recommendations of including gender specialists as part of the EAP and/or within focused projects would greatly improve integration of this component. Page | 101 The ME acknowledges the comments concerning the trade-offs of training options. In the future, students being trained in the US universities should maintain relevance to the HC research requirements and conduct research in that context. In addition, HC-university trainees should be given greater US/International exposure, such as attending international meetings, having the opportunity to spend time at an US/international institute, and requiring US researchers on thesis committees, to deepen their experience and improve their professional development. The inclusion of students in all PMIL meetings and allowing them to make presentations and present posters has allowed for more interactions with all PMIL scientists and provide the students with excellent opportunities to make formal presentations. We believe this should be continued and strengthened where possible. Regarding comments on maintaining the core focus of PMIL, the ME welcomes the potential of removing mycotoxins as a focus. This will help fully address the range of mycotoxin related issues elsewhere by USAID and to better address research needs to improve peanut productivity and recognize/improve the nutritional and health benefits of peanuts, rather than focus on potential risks. Clearly, any program working on peanuts would need to address the risks of aflatoxin, but could do so in a balanced way. As noted in the breeding section, the ME fully agrees and is working to enhance the role of HC scientists in designing and implementing the future research projects. In regard to increasing transfer of technologies, PMIL continues to work with development partners with more resources for scaling technologies and will continue to work with Missions to make them aware of technologies developed by PMIL to assure integration in future projects. The ME does not agree entirely with the EET comments concerning limiting research only to targeted countries. PMIL has engaged with scientists with specific capacities or advantages outside of the target countries for the benefit of the regions that they serve. The limited capacity and resources available to the peanut research community require regional integration and collaboration to reduce unnecessary duplication. In some cases, PMIL has built on long￾term relations that pay dividends towards research goals over time, such as working with former students or successful collaborators from previous projects. To abandon these relationships in favor of seeking new collaborators may prove more wasteful of resources than functioning with limited budgets in non-target countries. We do agree though that funding should be in proportion to the research agenda. Regarding the recommendation to invest in developing new peanut-based products, the ME seeks to balance the need to improve quality and expand capacity for demand of existing products with development of new products. New product development is a relatively high-risk investment to assure impact; whereas tweaks to existing products, such as improved packaging or better value addition to reduce home preparation, are more likely to succeed. However, integration of private sector actors in either process is required to assure better targeting and future commercialization and this is an area where active collaboration between the PMIL and the private sector is worth exploring.