iThe U.S. Government's Glohal Hunger & Food Securicx Initiative USAID FROM THE AMERICAN PEOPLE MID-AWARD EXTERNAL PERFORMANCE EVALUATION FEED THE FUTURE INNOVATION LAB FOR FOOD SAFETY 2019-2022 Mid-Award External Performance Evaluation of the Feed the Future Innovation Lab for Food Safety 2019-2022 This report was prepared independently by: Dena Bunnel March 20, 2023 Table of Contents Acronyms ........................................................................................................................................................................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... ............................... iii Executive Summary........................................................................................................... Introduction and Scope.................................................................................................... Evaluation Methodology................................................................................................... Technical Program Performance.................................................................................... Research Progress and Performance ....................................................................... Capacity Building Performance.................................................................................. Cross-cutting themes................................................................................................... Program Management Performance.............................................................................. Program Administration.............................................................................................. RFA process................................................................................................................... Partnerships and Collaboration................................................................................. Compliance and USAID relationship........................................................................ Findings and Recommendations..................................................................................... Findings............................................................................................................................ Recommendations ........................................................................................................ Conclusion .......................................................................................................................... Appendix A: List of Interviewees................................................................................... Appendix B: PI/Co-PI Interview Questions................................................................. ................ 1 ................ 3 ................ 3 ................ 5 ................ 5 .............. 10 .............. 11 .............. 11 .............. 11 .............. 13 .............. 14 .............. 14 .............. 15 .............. 15 .............. 16 .............. 17 .............. 18 .............. 21 ii Acronyms AoI Area(s) of Inquiry AOR Agreement Officer’s Representative EMMP Environmental Mitigation and Monitoring Plan FS Food Safety FSIL Innovation Lab for Food Safety FY Fiscal Year GFSS Global Food Security Strategy IBC Institutional Biosafety Committee IL Innovation Lab IRB Institutional Review Board ME Management Entity MEL Monitoring, Evaluation, and Learning MSI Minority Serving Institution PI Principal Investigator RFA Request for Applications USG United States Government iii Executive Summary The Feed the Future Innovation Lab for Food Safety (FSIL) is a cooperative agreement funded by the United States Agency for International Development and led jointly by Purdue University and Cornell University. The FSIL program launched June 25, 2019, and the current award will run until June 24, 2024. The program’s core mandate is “to create enhanced agriculture sustainability and resilience and global food security through research and engagement that increases production of and access to safe and nutritious foods, leading to well-nourished communities, especially among women, youth, and vulnerable populations.” In order to assess program performance to date, the FSIL Management Entity commissioned an external performance evaluation to assess three primary factors of program performance: (1) the research program performance, (2) the capacity building efforts, and (3) overall management. The evaluation was conducted from January 1, 2023, to March 31, 2023, with an assessment window of June 24, 2019, to December 31, 2022, representing three and half years, or 70 percent of the life of the cooperative agreement. The evaluation reviewed key program documents and conducted virtual interviews with FSIL ME, subawardees, USAID personnel and other stakeholders. The primary findings of the evaluation were: 1) The Food Safety Innovation Lab has a high functioning management entity that successfully meets the needs of the FSIL subawardees and USAID. 2) The ME remains open to and understanding of unforeseen changes and facilitates a culture of openness that results in effective adaptive management. 3) The Food Safety Innovation Lab is making good progress towards its objectives, with highly active research projects, and is largely on track to meet its objectives by the end of the current award period (June 24, 2024). a. This is despite early setbacks caused by the COVID-19 pandemic that prohibited international travel and in some cases in-country travel for project teams. b. One project has faced serious challenges in receiving Institutional Review Board approval for its research to the point of potentially compromising the ability for the project to meet its research objectives within the life of the award. However, the project team and FSIL ME are actively working toward rectification. 4) FSIL has established a unified research strategy to address challenges in global food safety, with a particular focus on awareness of microbial food safety threats. 5) FSIL has set a precedent for other Innovation Labs and similar programs to effectively engage Minority Serving Institutions (MSIs) in international research, though barriers still remain. 6) FSIL has committed significant resources toward the inclusion of gender-sensitive programming in its project portfolio and exhibited a strong understanding of the important role of women in the food system and in food safety interventions. 7) University systems remain sluggish in their administrative systems which results in delays in contracting, funding issuance, research approvals, etc. The evaluation led to key recommendations for FSIL and USAID for the remainder of the award period and for future research initiatives in food safety. These include: 1 1) The overarching recommendation is that food safety is a critical component in improving food security and nutrition in low-income, vulnerable communities. USAID should increase or at least maintain its investments in food safety. With a well-established, highly functioning management entity and well-defined research portfolio, continuing future investments in the Food Safety Innovation Lab will represent a best use of U.S. Government funds to achieve the Feed the Future and USAID objectives. 2) Though the food safety landscape is large, funding is constrained. Therefore, FSIL should prioritize and condense its portfolio in order to achieve greater impact in targeted areas. This could include a smaller geographic reach and/or a smaller value chain focus, while maintaining a focus on food safety awareness and in identifying interventions that can be feasibly implemented. 3) USAID should continue to invest in making international research more accessible to MSIs. This includes additional funding to support the administrative support mechanisms that universities require to implement international research projects. 4) Constraints on graduate student funding present a challenge for effective research implementation at U.S. universities. This is presumed to be an even greater challenge for MSIs than for larger institutions. USAID should continue to work toward more open graduate student funding guidelines. 5) Under USAID’s localization strategy, there is risk and opportunity for Innovation Labs. USAID should engage in dialogue with university institutions on how to manage these risks so as to continue to leverage the expertise and commitment of U.S. universities while still supporting greater leadership in project countries. 6) For the remainder of the award period, FSIL is moving full steam ahead to make up for lost time during the COVID-19 pandemic and achieve its stated objectives by June 2024. Some projects are still in danger of not meeting their objectives within the project period. Because publication of results in particular is likely to lag behind, the ME should make plans for ensuring researchers have publishing needs met outside of the project period. 2 Introduction and Scope The Feed the Future Innovation Lab for Food Safety (FSIL) was awarded to Purdue University and Cornell University by the United States Agency for International Development (USAID) under the United States Government’s (USG) Feed the Future Initiative on June 25, 2019, for a period of five years, until June 24, 2024. Program activities started in August 2019. The primary outcome objectives of FSIL are “to create enhanced agriculture sustainability and resilience and global food security through research and engagement that increases production of and access to safe and nutritious foods, leading to well-nourished communities, especially among women, youth, and vulnerable populations.” The Activity has funded research projects in six focus countries through competitive selection processes. In order to assess the performance to date of FSIL against its key objectives, this performance evaluation was commissioned by the FSIL Management Entity (ME). The period of the review covers from the start of the cooperative agreement to December 31, 2022, representing three and half years, or 70 percent, of the five-year award. During the period of review, two requests for application (RFA) periods were opened and six projects awarded funding as subawards. This is in addition to four “Quick Start” foundational research grants, which were implemented in the first year of the program, and one Buy-In award focused on COVID-19 response. The evaluation assessed the following program components: (1) the research program performance, (2) the capacity building efforts, and (3) overall management. The evaluation did not assess financial performance of the Innovation Lab. The performance evaluation considered the implementation of the global research program, including: incorporation of the core program components and cross-cutting themes; progress of the research towards objectives; the degree to which the research activities achieve integration and are relevant to development in the host countries and more broadly; and human and institutional strengthening results of the program. It also evaluated the administrative and management effectiveness of the Management Entity (ME) including the relationship between the ME and sub￾recipients/partners; the relationship and communication with USAID Washington and missions; and the outreach and intellectual leadership activities undertaken by the ME. Evaluation Methodology To conduct this performance evaluation, a series of virtual interviews were conducted, alongside a desk review of relevant program documents. The evaluation sought to assess performance to date of FSIL and its research projects. The evaluation did not seek to make judgements on the quality or importance of research projects. Rather it sought to assess progress against stated project objectives and relevance to FSIL’s broader objectives and cross-cutting themes. Interviews were conducted via video Zoom in 30 minute to one-hour periods during which the evaluator asked a series of questions relating to the administration of FSIL and/or specific country projects and progress against stated objectives, including research, capacity building and policy outcomes and cross-cutting themes. Interviews were conducted with the FSIL Management Entity, Principal Investigators (PI), in-country leaders, co-PIs, technical experts, external advisory board members and USAID AOR and Activity Manager. A full list of those interviewed can be found in Appendix A. The documents reviewed included the following: • FSIL prime agreement • FSIL technical application (i.e., original proposal) 3 • FSIL subaward technical narratives • FSIL MEL plan • FSIL EMMP • FSIL EMMP compliance documentation • FSIL work plans • FSIL RFAs • FSIL annual reports – FY2020, FY2021, FY2022 • FSIL semi-annual reports – FY2020, FY2021, FY2022 • FSIL news stories After review of documents and conducting interviews, progress and performance were mapped against the evaluation objectives of 1) the research program performance, 2) the capacity building efforts, and 3) overall management. Following the FSIL MEL Plan, the evaluation considered: MANAGEMENT APPROACH TECHNICAL APPROACH Management Practices: How effectively has Alignment with Theory of Change: How the Management Entity implemented and closely do research and engagement activities managed the FSIL’s respective research and and results align with the FSIL theory of change capacity development activities? and objectives? Collaboration with other Development Adaptive Research Portfolio: How proactive Partners: How effectively has the management has the management entity, in partnership with entity established partnerships with USG-funded its subawardees, been in responding to research projects/programs, Feed the Future Innovation challenges and opportunities and adjusting Labs, and other donors (e.g., DFID/BMGF) to activities accordingly? identify synergistic collaborations, reduce duplication, and reinforce outcomes by coordinating efforts and sharing knowledge? Private Sector Engagement: To what extent Technical Capacity Development: How has the management entity established well has the management entity identified and meaningful engagement with the private sector addressed academic and technical capacity needs to help achieve its objectives and support of host country stakeholders as it relates to sustainability, scalability, and empowerment of food safety? people and partners on their journey to self￾reliance? The resulting evaluation report is a narrative report that provides an overview of FSIL’s key activities and priorities, assesses progress to date, and highlights particular achievements and challenges. It concludes with some focused recommendations for future work. The evaluation was conducted by Dena Bunnel, an independent consultant with thirteen years of experience in international agricultural development and research for development, including previously serving as the Assistant Director of the Feed the Future Innovation Lab for the Reduction of Post￾Harvest Loss. 4 Technical Program Performance The primary outcome objective of FSIL is to create enhanced agriculture sustainability and resilience and global food security through research and engagement that increases production of and access to safe and nutritious foods, leading to well-nourished communities, especially among women, youth, and vulnerable populations. FSIL is guided by five primary objectives and four cross-cutting themes. Objective 1: Increase stakeholder awareness of food safety issues, impacts, and measures to reduce food safety risks in households, communities, government agencies, and food production firms Objective 2: Build local research capacity and conduct research on regional food safety challenges Objective 3: Support translation and dissemination networks among researchers, government, private sector, NGOS, and the public to develop policies and engagement structures Objective 4: Enhance local capacity to translate food safety research into training, guidelines, and commercialized products to deliver technologies and best practices at scale Objective 5: Achieve and maintain high standards in management performance through effective structures, a dynamic and adaptive personnel team, and a culture of open communication within the ME, FSIL and with internal and external stakeholders The cross-cutting themes of FSIL include gender equity, youth engagement, human and institutional capacity development and food safety enabling environments. A major subtheme of the human and institutional capacity development theme is investment in Minority Serving Institutions (MSI) for research and development. The research portfolio further falls under three areas of inquiry: 1) improved nutrition and human outcomes, 2) reduce and mitigate risk for enhanced resilience, and 3) advancing the productivity frontier through economic development, which align with the U.S. Government’s Global Food Security Strategy objectives. Research Progress and Performance Over the course of the award to date, FSIL has funded 11 research projects: four foundational research grants (Quick Starts), four long-term subawards, two short-term subawards, and one Buy-In award. These projects have a wide geographic scope covering West and East Africa and South and Southeast Asia. They also cover a diversity of value chains, including fresh vegetables, poultry, fish and dairy. At program design, an intentional decision was made by the FSIL ME to not invest in specific technologies or equipment. Rather, priority would be placed on awareness of food safety issues, social and economic factors in food safety practices, and life science research that is sustainable and replicable within the project countries’ systems. This is reflected across the subaward project objectives. At the proposal stage, FSIL made clear that they intended to focus primarily on microbiological food safety research, with less emphasis on chemical hazards. This was due to an identified gap in food safety research where microbial food safety research had been underinvested in, despite foodborne illness being a serious, if poorly characterized, issue in developing country systems. Chemical hazards are not excluded from the research portfolio, with the Senegal QuickStart assessing mycotoxin contamination in 5 " E 0 !l :, 0 t :, fr :, 0 .,, :! ·;; .: u <( Enhanced agriculture sustainability and resilience and global food security through research and engagement that increases production of and access to safe arnd nutritious foods, leading to well-nourished communities, especially among women, youth, and vulnerable populations. The burden of food borne disease from biological & chemical hazards is significantly reduced through high-impact academic, private sector, government, non-government and public collaboration that produces transformational food safety research and the policy and engagement practices that ensure translation of FSIL research and dissemination to households, communities, and throughout the production chain. ♦ Increased stakeholder awareness of food safety issues, impacts, and measures to reduce food safety risks FSIL research • and outreach activities focus on awareness raising and problem identification to reduce barriers to FS policy & practice ♦ Local researchers conduct priority FS research in collaboration with stakeholders at all levels ♦ FSIL builds local research capacity and facilitates researcher/stakeholder engagement for research priority setting ♦ Policies and engagement structures support translation, dissemination, and implementation of FS research FSI L supports • translation and dissemination networks between researchers, government, private sector, NGOs, and the public ♦ Technologies and best practices are available/used by ho1Useholds, communities, and commercial stakeholders FSI L enhances • local capacity to translate FS research into trainings, guidelines, and commercialized products Foundational Activities: Conduct site-specific assessment and research of FS problems, opportunities, and challenges: establish researcher engagement with government, private sector, NGOs, and the public; quantify and map barriers to improved FS; collaboratively establish FS research priorities and fund research, translation, and dissemination activities. groundnuts and the Bangladesh long-term subaward assessing chemical contaminants in fish, for example, but microbial contaminants form a larger share of the portfolio. Figure 1 - Results Framework for the Feed the Future Innovation Lab for Food Safety Quick Starts: At the start of FSIL activities, four Quick Start projects were awarded as pre-selected in the proposal process. These Quick Starts were in the four focus countries of Bangladesh, Cambodia, Kenya, and Senegal. Interviews with FSIL ME and USAID staff indicated that the Quick Starts were effective in gaining an understanding of the scope of food safety research and challenges in the focus countries. In some cases, namely Cambodia and Bangladesh, project PIs and co-PIs stated the Quick Start projects were helpful in the development of long-term research project proposals in the first round of Request for Applications (RFAs). While Quick Start projects were designed to launch quickly and allow for in-country project activities to start soon after FSIL’s project start, the tradeoff of such projects is that they still take a significant amount of time for contracting and administration, they are fairly costly given their scope and compared to longer-term projects, and their short duration limits their outcomes. One Quick Start award planned for Ethiopia was cancelled when the proposed project team was unable to meet work plan development and sub-contracting requirements to deliver a timely, effective program. The cancellation was ultimately mutually agreed upon by the FSIL ME and the subawardee institution, as well in consultation with the USAID AOR, and does not seem to have negatively affected program performance. Competitively awarded subaward projects: Two RFAs were issued under FSIL. The first was issued in April 2020 and resulted in four research projects in the same focus countries as the Quick Starts. These projects run for 3.5 years, from October 1, 2020, to March 31, 2024. The second round RFA was issued 6 in April 2021 to fund two-year research projects led by Minority Serving Institutions (MSIs). While not part of the original workplan, this RFA was added to increase engagement of MSIs in international food systems research, a priority of USAID and FSIL, and was made possible due to available funds from the cancellation of the Ethiopia Quick Start and cost-saving from the lack of travel caused by pandemic restrictions. This RFA resulted in funding of two projects, one in Nepal and one in Nigeria, running from March 1, 2022, to February 28, 2024, and May 1, 2022, to April 30, 2024, respectively. Subaward projects should align with the objectives set out in FSIL’s broader proposal yet provide specific objectives to address food safety challenges in the focus country. A review of each subaward’s goals and objectives reveals this continuity across all the projects. The result is a cohesive and substantive research portfolio that covers critical needs in food safety in Feed the Future target countries. Goals and Objectives of each FSIL subaward project: Bangladesh Overarching goal: to promote informed decisions and actions that enhance food safety quality in farmed fish (tilapia, pangasius, and rohu) and a chicken product (frozen uncooked drumsticks) value chains in Bangladesh by integrating multidisciplinary research and training on food safety. Objective 1: Analysis of the Knowledge, Attitude and Practices (KAPs) regarding food safety and risk issues related to fish and selected frozen chicken products in a gender and age segregated representative sample of Bangladeshi consumers and major value chain actors. Objective 2: Estimation of Bangladeshi consumers’ willingness to pay (WTP) for a general reduction in exposure to potentially harmful microorganisms and chemicals, and for safety certification in fish and chicken. Objective 3: Evaluation of the impact of information related to food safety on consumers’ WTP for a general reduction in exposure to potentially harmful microorganisms and chemicals in fish and chicken. Objective 4: Evaluation of the welfare impacts of a general reduction in exposure to harmful microorganisms and chemicals in fish and chicken. Cambodia Overarching goal: to measurably reduce the prevalence and incidence of foodborne pathogen contamination of vegetables produced and sold in Cambodia. Objective 1: Identification of Critical Control Points (1) Identify two prominent bacterial pathogens associated with vegetable￾borne disease(s). (2) Conduct a longitudinal study to map and characterize bacterial pathogen contamination points, persistence, and transmission in vegetable￾chains. (3) Utilize previously collected data to create a shared research agenda among all Objective 2: Creation of Targeted Interventions (1) Identify and/or design interventions to reduce microbial contamination. (2) Assess food safety awareness and willingness-to-adopt potential interventions for specific critical control points. (3) Establish and strengthen food safety networks and public-private partnerships to promote adoptable interventions, Objective 3: Delivery of Data￾driven Engagement. (1) Create and deliver engagement programs to foster greater adoption of food safety interventions by farmers, collectors, vendors, and market management groups. (2) Deliver engagement programs that improve food safety awareness among consumers. (3) Measure impacts and efficacy of all engagement programs and refine outreach to 7 partners in terms of critical control points to be targeted for high-impact interventions. identify early adopters, and help in positioning interventions. produce greater reductions in foodborne disease risks. Kenya Overarching goal: to improve food security and nutrition in Kenya by developing capacity for systems-based, risk-informed approaches to food safety that reduce risk of foodborne disease, increase production of safe food, and improve economic outcomes. Objective 1: Identify, Objective 2: Objective 3: Develop Objective 4: Estimate in collaboration with Characterize and evaluate the efficacy the public health stakeholders, food Salmonella enterica of culturally and gender impact and evaluate the safety priorities for (SALM) and appropriate, practical, benefits and costs from poultry value chains in Campylobacter spp. and scalable intervention selected Kenya using a risk- (CAMPY) in poultry strategies for mitigating intervention strategies informed approach. value chains managed by women and youth farmers in peri-urban areas of Kenya. risk of SALM and CAMPY in poultry that effectively account for gendered roles in poultry production. to inform public and private decision￾making. Senegal Overarching goal: to improve the microbiological safety and quality of the dairy value chain via research and human and institutional capacity development. Objective 1: Raise Objective 2: Objective 3: Provide Objective 4: awareness of Develop and conduct the food industry with Coordinate stakeholders on food research-based food knowledge on cost- development and safety issues and their safety training benefit propositions for implementation of public health impact. programs for capacity building. implementing food safety interventions. comprehensive food safety regulations. Nepal Overarching goal: to enhance the food safety economic research and outreach in Nepal in the fresh produce sector to advance knowledge and enable stakeholders to make informed and prioritized decisions on food safety investments. Objective 1: Objective 2: Objective 3: Objective Objective 5: Objective 6: Assess the Understand the Investigate the 4: Analyze Outreach and Food-safety indicators of demand for incentives for the awareness research contamination safer quality safer quality production training on findings risks and fresh produce production by and food safety and dissemination foodborne through examining food consumption health hazard and outreach illness assessment of safety practices differences reduction to involving incidences consumer among fresh by gender small and government among consciousness, produce and locations. medium-sized stakeholders consumers consumer growers and producers and private and fresh willingness to impacts on emphasizing agribusinesses produce pay, and diet costs, revenues, young and to advocate growers. diversity of households. and well-being. female entrepreneurs. prioritized food-safety strategies and 8 informed investment decisions. Nigeria Overarching goal: to understand facilitators and barriers to reducing the prevalence of foodborne illness and its associated nutritional outcomes (i.e., stunting) among households with young children from the perspectives of children, mothers, primary health care providers, community development personnel in Local Government Areas, representatives from the State Ministry and civil societies, and community-based organizations. Objective 1: Evaluate household vulnerability to foodborne illnesses by monitoring food safety practices and assessing environmental sanitation using a mixed methods approach. Objective 2: Understand the extent to which strategies described in The Nigerian National Policy on Food Safety, published in 2014, are recognized and implemented by key stakeholders. Objective 3: Identify priority areas for food safety intervention programing, using a novel data-driven approach based on GIS mapping. Objective 4: Using a Nominal Group Technique, bring together key stakeholders to identify and prioritize programs and policy actions to improve household food safety through cross-sector collaboration. Beyond physical science research, which is robust, FSIL research projects also seek to assess societal and behavioral aspects of food safety, focusing on consumer and value chain actor behavior. This is an important component in considering feasibility to bring real change to the food safety landscape in the project countries. This included a willingness to pay economic survey in Bangladesh to assess if consumers would pay a premium price for fish of higher quality, a survey of consumer and vendor perceptions in vegetables markets in Cambodia, and a risk prioritization workshop that included stakeholders from farmers to government officials in Kenya. As co-PI for the Cambodia project Dr. Paul Ebner stated regarding identifying and measuring effective food safety interventions, you need to answer the question, “Will they start doing it, and will they do it over time?” Most projects expressed confidence, if cautiously, about being on track to complete their objectives by the end of the award period, despite delays caused by COVID-19 restrictions. Data collection is ongoing or complete in most projects, and U.S. based researchers have been or will soon travel to their project countries. Virtual trainings and workshops allowed capacity building activities to move forward during the restrictions, and in-person trainings, workshops, and stakeholder events are actively taking place. The project of most concern is the subaward project in Kenya, which due to a number of challenges at the institutional level has yet to received Institutional Review Board (IRB) approval for data collection on the project. With two sets of data collection still to take place under the objectives of this project, there is considerable concern for this project to be able to complete its objectives by the end of the subaward. Nevertheless, all project partners and the FSIL ME are assertively and collaboratively working to try to resolve the institutional barriers preventing IRB approval. Senegal, too, is collecting data in FY2023, which creates a narrow timeline for data collection, analysis, and results within the current subaward period, set to end March 31, 2024. The Senegal PI in fact stated that the project was “on track to catch up,” and expressed cautious optimism about being able to meet project objectives within the award period. The short-term Nigeria project seems to be making good 9 progress, after some delays due to SAM registration of the in-country university partner (Bowen University). Data collection has begun but is scheduled to continue in spring 2023. With Nigerian elections taking place Feb 24-Mar 18, 2023, the risk of further delay is high, which could negatively impact the ability to complete the project objectives. However, the Nigeria project team identified on￾the-ground enumerators required and has obtained IRB approval for data collection. IBC acknowledgement and lab training is still outstanding (per the most recent EMMP tracking report). Buy-In project: FSIL additionally was awarded a Buy-In project from USAID that sought to respond to the COVID-19 pandemic’s impacts on the food system. The project worked across FSIL focus countries of Bangladesh, Cambodia, Kenya, and Senegal, as well as Nepal, through the creation of an “international task force” that sought to provide COVID-19 resources and science-based messaging for the food industry through Frequently Asked Questions videos, translated resources, and live virtual office hours. This project engaged more than 1,000 stakeholders across the food system, and, in several instances, local partners have sought additional funding to continue the activities after the Buy-In project ended. Capacity Building Performance Capacity building is a critical component of USAID programming and FSIL’s program objectives. Human and institutional capacity building for FSIL can largely be broken into three categories: 1) capacity building of institutions in project countries, including government institutions and policymakers, 2) long￾term training in the form of master’s and PhD students at subawardee institutions, both U.S. and international, and 3) capacity building of Minority Serving Institutions in the U.S. To date, 27 graduate students are being supported through FSIL. Of these students, 22 are graduate student researchers at universities in their home country where the projects are being implemented. Five are American students at U.S. universities. Capacity building of graduate students at U.S. research institutions is limited due to an inability to fund American students in most cases for these awards and cumbersome and restrictive requirements of the USAID-sponsored J1 visa required for international students to undertake USAID-funded degree programs at U.S. universities. This was expressed as a concern by several interviewees, stating the restrictions on funding graduate students makes it much more difficult to implement USAID-funded research. Some PIs have resorted to funding graduate students on other, non-USAID funds but still dedicate some of that student’s time to FSIL research activities. However, this is not always feasible, and particularly is unlikely to be an accessible option for MSI-based researchers with more limited funding. Multiple researchers referenced their role in capacity building as not only of building skills and providing resources in their technical area of food safety research, but also in developing strong research methods among project country partners, including quantitative and qualitative research methods, good survey design, and adherence to research ethics. In the Cambodia project, this was taken one step further with the development of virtual courses on quantitative and qualitative research methods which were delivered to partners within the project and to other external stakeholders. A major capacity building effort of FSIL is the focus on encouraging research and leadership at Minority Serving Institutions. FSIL engaged the Associate Dean for Diversity, Equity, Inclusion and Faculty Affairs at Purdue University as a technical expert to help FSIL understand the barriers for MSIs, design an effective RFA, and facilitate outreach for the RFA, as well as serving on the review panel for proposal selection. Many interviewees spoke about the dedicated investment in MSIs through the second RFA. It was expressed that this was a heavy lift but a worthwhile one, and this sentiment was shared amongst the ME, MSI subaward recipients, and USAID. For the Nepal and Nigeria projects funded under this 10 second RFA, all subaward agreements are held by Purdue University, rather than having secondary subawards held by the PI’s institution. This was done to reduce the burden on MSIs in the administration of these research projects. FSIL researchers expressed that the investment in MSIs has already had positive contributions outside of FSIL as well. The Agriculture and Forestry University in Nepal is a research partner on the Tennessee State University-led subaward in Nepal. Their participation in the MSI-led RFA process has led additional researchers there to apply for other USAID-funded projects, such as the competitive RFA from the Horticulture Innovation Lab. Cross-cutting themes Gender: A strong component of FSIL gender programming was the completion of the LASER PULSE Gender Analysis in Research and Application course as a requirement prior to submitting research proposals. FSIL further required that all long-term subawards include a thorough gender strategy and at least one gender specialist for each project. FSIL has established a Gender Working Group to facilitate collaboration and thinking on gender aspects across its projects. In FY2022, this group developed a logic model to guide its efforts, which may help guide greater impact. There seems to be a strong understanding across the research portfolio of the important role of women in creating change in food safety practices. This includes women as the purchasers and preparers of food for the home. The Nigeria project highlights this particularly well with survey data focusing on mothers. Women also have key roles in the value chains of FSIL research projects, such as women as the primary vendors at vegetable markets in Cambodia and in the dairy sector in Senegal. Youth engagement is also a cross-cutting theme; however, it has less focus across the research projects than gender. Yet, with the role of young people as enumerators and graduate students, and the role of young people in some value chain functions included in FSIL projects, youth is still a consideration and will likely have a positive impact from the project. Food safety enabling environment: The potential for policy-level impact varies across the projects; however, in every project there exists some form of policy and enabling environment-focused activity. Stakeholders across the value chain are included in project activities, including private-sector stakeholders and government officials, both key parties in policy change. There is a recognition of the need for a multi-part intervention strategy that targets consumer behavior, government policy, regulation and enforcement, and private sector engagement was found across the FSIL research projects. Program Management Performance Program Administration Management entity function: Every subawardee partner interviewed expressed high confidence in the Management Entity and spoke to their responsiveness to project needs. A theme across the interviews was the realistic approach the ME takes to the management of international research projects. The ME remains open to and understanding of unforeseen changes and facilitates a culture of openness that results in effective adaptive management. The FSIL ME is unique in that it is led by co-Directors at different universities, with Dr. Oliver at Purdue University listed as the Director and Dr. Worobo at Cornell University as the Associate Director. The 11 co-Directors work smoothly together by all accounts, facilitated by a mutual willingness for shared decision-making and flexibility. Rather than separating into distinct roles, the co-Directors trade off arising needs of FSIL based on their availability, which serves them well as both also maintain active research portfolios at their universities outside of FSIL. Dr. Oliver takes the lead on most activities with USAID and other Innovation Lab partners. Neither co-Director has 100 percent of their time dedicated to the lab. An advantage of this approach is that they are able to remain engaged in their universities and with additional research within the land grant mission; however, it comes with the risk of being overburdened by the combined Innovation Lab, departmental and research responsibilities. Drs. Oliver and Worobo are able to balance their FSIL and other university requirements, without sacrificing program oversight and effectiveness, by having a well-staffed ME of highly capable professionals. The ME is characterized by a hierarchically flat and highly collaborative team structure. The Managing Director facilitates the day-to-day operations of the lab, such as USAID reporting, meeting organization, subaward management, handling ad hoc USAID requests, etc. The Managing Director is supported by the Operations Specialist who handles the many logistical and administrative components of FSIL management. The ME also consists of a business manager, with a 50 percent time allocation, and a communications manager with an 80 percent time allocation. The ME will be adding an additional 75%-time program manager position in spring 2023. The ME is well-staffed and intentionally done so. The FSIL Director shared that an active goal is to have the ME operating at 80 percent total capacity for their primary FSIL administration needs. This strategy creates room for the ME to respond meaningfully to ad hoc requests by USAID or program partners and to have staffing available to administer potential future buy-in or associate award projects. The ME further stated that sharing the leadership of FSIL across two institutions allows them to double their pool of resources. They can draw from two talent pools of university faculty and staff and utilize the university system that works most effectively for a given purpose. For example, when FSIL hosted a series of webinars, Cornell University was better equipped with the support system to run a webinar, and so the series was hosted out of Cornell University, rather than Purdue. Subawardee engagement: The shared mantra of the Management Entity is that they are not afraid to make significant changes if something is not working well. This was exemplified by the cancellation of a Quick Start project in Ethiopia following prolonged challenges with project start up. At the same time, they respond quickly and realistically to unforeseen challenges, working with research partners to resolve issues or adjust course. Monthly meetings are held with the ME and each project team, lasting approximately 30 minutes each meeting. The ME and research partners expressed the usefulness of these calls in keeping projects on track, being able to deal with emerging issues and reducing the need for extensive email correspondence. The ME utilizes the PieStar DPx program management software to facilitate reporting and travel requests by subawardees. Piestar RFx was also used in the RFA process for proposal submission. The ME values the system to streamline the reporting process, store historical program information and increase accountability of the subawardees. Subawardees largely found the PieStar system to be easy to use and helpful. There have been no stated challenges or delay in the receipt of USAID funds to FSIL. The biggest challenge in administration of subawards was the time for subcontracting from Purdue University to the subawardees or from a subawardee to a secondary subawardee, especially international subawards. 12 Working through burdensome university systems for other research components such as IRB approval for field research has also been a challenge, as previously highlighted. The revamping of the USG SAM system was a serious hurdle faced by multiple subawardees which led to a delay in contracting and provision of funds. Any institution receiving USG funds must be registered in the SAM system; however, many international institutions may not have a central office to handle such administrative requirements. In the face of this challenge, all parties reported dedicated efforts by the ME to help subawardees overcome the SAM registration challenges. To date, all subawardees have up￾to-date SAM registrations and are able to receive USG funds. Communications: The FSIL also places a priority on communicating clearly to highlight key messages. They maintain active Twitter and LinkedIn accounts. Project briefs exist for each subaward project, as do news and success stories about the projects. The FSIL ME has hosted multiple webinar series. A challenge of research for development communication is teasing out compelling stories, which can be difficult for ongoing research projects, particularly with an emphasis from USAID on showcasing the impact of programming. Research projects often do not show impact until they reach their conclusion, and researchers will be resistant to drawing conclusions until they can validate their findings. Despite this challenge, FSIL has done an excellent job of communicating their work and successes. RFA process FSIL has issued two rounds of requests for applications to fund food safety research projects. The first RFA established long-term subawards, each lasting three and one-half years. In interviews with the successful proposal PIs, it was expressed that the process was transparent and clear. One PI expressed that it was one of the smoothest RFA processes he had experienced. The first round RFA was issued as the COVID-19 pandemic was first unfolding. While the movement restrictions of COVID-19 affected the Quick Start projects, and later some of the implementation of the long-term research subaward projects, the FSIL Director noted that for the proposal process, FSIL actually benefited. Because researchers were working from home and unable to travel or even conduct laboratory research in some cases, they were able to spend more time developing very strong research proposals. Presciently and cleverly, for the long-term subawards, the FSIL ME prohibited international travel in the first year of the project. This helped to reduce the disruptions caused by pandemic restrictions. A second round RFA was issued in April 2021 and limited exclusively to projects led by researchers at Minority Serving Institutions. The FSIL ME took an intentional approach to provide additional resources and feedback through the three-step process, which first required a letter of interest and mandatory attendance in a “pre-competitive virtual ideation session” before concept notes were submitted. In order to encourage young researchers but ensure university-level support, FSIL also required support letters from the PI’s research leadership. The ideation session was followed by in-depth feedback to concept notes and the opportunity to engage with FSIL technical experts in the development of their full proposal. Drawing lessons from the previously cancelled Quick Start, which was meant to be housed at an MSI institution, the ME began working with the PI institution’s sponsored programs offices at the full proposal stage, prior to final award announcements. As previously mentioned, both primary and secondary subawards for these projects are held by Purdue University in order to reduce the administrative burden on the awarded MSI’s. 13 Both RFA review panelists, which include FSIL external advisory board members and technical experts, and the research teams who applied expressed that the process was clear and transparent. Particularly in the 2nd round RFA, researchers communicated that they appreciated the detailed feedback and collaboration sessions that took place during the process and praised the opening of greater opportunities for MSIs. Partnerships and Collaboration FSIL maintains an external advisory council that boasts leading experts across the food safety and research spectrum. This group has been engaged in RFA proposal review and other ad hoc requests to engage with FSIL research teams. FSIL also draws from a multi-disciplinary group of researchers at Purdue and Cornell universities to serve as technical experts in the development of RFAs, review of proposals, research design, publication review, and other technical needs of the program. FSIL is also an active member of the Innovation Lab community, attending events and engaging in relevant communities of practice. They have incorporated researchers from other Innovation Labs in their external advisory board, technical expert pool, and in the research projects themselves. They are also cognizant of other development partners within the research projects. They intentionally collaborate with other stakeholders doing work in the respective project countries, such as the Cambodia project’s partnership with the Institut Pasteur du Cambodge, to reduce duplication and expand impact. Alternately, they look to avoid areas where duplication seems likely, which was listed as an additional reason for cancellation of the Ethiopia QuickStart. Compliance and USAID relationship The FSIL ME appears to have a strong collaborative relationship with USAID through the AOR and Activity Manager. Major decisions or changes to program components are discussed and agreed with USAID. The FSIL ME is responsive to USAID requests and proactive in supporting USAID priorities. FSIL has proven itself to be a leader in the Innovation Lab community for its support and amplification of USAID priorities. The greatest example of this is the second RFA call that was targeted exclusively to MSI-led projects. Multiple interviewees expressed how FSIL took this priority on and “showed how this could be done effectively.” FSIL also collaborates with USAID on communications strategic outputs, such as the development and delivery of a series of webinars at USAID’s request, and responds promptly to requests for information from USAID. Engagement with the USAID Missions varies across countries; however, this seems to be largely determined by the Mission staff rather than FSIL researchers. Travel limitations and remote work as a result of the COVID-19 pandemic reduced communication with the USAID Missions in the early stages of the project. With travel and activities now resumed, all project PIs reported communicating with Mission staff prior to international travel and any major in-country events. If unable to meet in person during an in-country visit, PIs reported offering virtual update meetings to Mission staff. There were no concerns found regarding compliance with USAID. FSIL maintains an updated MEL Plan and Environment Monitoring and Mitigation Plan and Reports. Semi-annual and annual reports are complete for all reporting periods. This evaluation did not assess the public availability of research data, namely because research projects are ongoing, and thus finalized, cleaned data sets are not to be expected at this stage. The evaluation also did not directly review annual reporting of standard Feed the Future indicators, but no concerns were expressed by USAID on their progress against these indicators. 14 The only compliance issue found related to field data collection prior to receipt of IRB approval for one project. This issue was reported in the 2022 Annual Report and elevated to university administration of the PI’s institution, which ultimately determined that it was not a major event. The team reached agreement that the data cannot be used, and communication appears to be open and productive in working to receive full IRB approval. Findings and Recommendations Findings Overall, the Food Safety Innovation Lab proved to be a well-functioning program. Key findings of the evaluation were: 1) The Food Safety Innovation Lab has a high functioning management entity that successfully meets the needs of the FSIL subawardees and USAID. a. The ME takes an intentional approach to ME staffing that enables them to respond effectively as needs arise and to avoid burnout of staff. This approach should be encouraged and replicated by similar programs. Effective management of complex research programs requires a staff that is prepared and nimble. This approach facilitates that. b. The dual university and directorship of FSIL works extremely well as a result of a joint commitment to shared governance and a trusting working relationship between the co￾Directors and across the ME. Other programs considering this model would need to be similarly committed to this partnership style. c. Having the co-Directors not 100% dedicated to the lab makes leadership of such a program more open to researchers with dynamic portfolios, particularly younger researchers moving into leadership roles. However, it should also be accompanied by a well-staffed management entity, and time-commitments associated with Directors’ roles should be negotiated with the university to ensure shared expectations and feasible workloads. 2) The ME remains open to and understanding of unforeseen changes and facilitates a culture of openness that results in effective adaptive management. 3) The Food Safety Innovation Lab is making good progress towards its objectives, with highly active research projects, and is largely on track to meet its objectives by the end of the current award period (June 24, 2024). a. This is despite early setbacks caused by the COVID-19 pandemic that prohibited international travel and in some cases in-country travel for project teams. Most interviewees estimated that the COVID-19 pandemic set back project activities by one to one-and-a-half years. Nevertheless, most projects have been able to make progress on their objectives and are accelerating activities with the return to more or less normal travel and work operations. b. One project has faced serious challenges in receiving IRB approval for its research to the point of potentially compromising the ability for the project to meet its research objectives within the life of the award. However, the project team and FSIL ME are actively working toward rectification. 4) FSIL has established a unified research strategy to address challenges in global food safety, with a particular focus on awareness of microbial food safety threats. 15 5) FSIL has set a precedent for other Innovation Labs and similar programs to effectively engage Minority Serving Institutions in international research, though barriers still remain for MSIs seeking to manage international research projects. 6) FSIL has committed significant resources toward the inclusion of gender-sensitive programming in its project portfolio and exhibited a strong understanding of the important role of women in the food system and in food safety interventions. a. This commitment has included required training, planning and staffing of their research projects, dedicated and intensive gender analyses, and a Gender Working Group. b. However, there is still work to be done on establishing shared and robust standards for gender mainstreaming, and the effectiveness of the Gender Working Group remains to be seen. 7) University systems remain sluggish in their administrative systems which results in delays in contracting, funding issuance, research approvals, etc. Recommendations The overarching recommendation of this report is that food safety is a critical component in improving food security and nutrition in low-income, vulnerable communities. USAID should increase or at least maintain its investments in food safety. With a well-established, highly functioning management entity and well-defined research portfolio, continuing future investments in the Food Safety Innovation Lab will represent a best use of U.S. Government funds to achieve the Feed the Future and USAID objectives. FSIL has succeeded in establishing a cohesive research portfolio despite working across multiple geographies and value chains. Nevertheless, the resource pool, at $10MM in the current agreement, is quite limited. Any future research activities after the current award period should consider reducing scope in either geography, value chain, or other area of focus, if available funding remains at a similar level. While this would reduce the area of coverage, it would allow for more in-depth research and intervention development at targeted regions or challenges in food safety. For example, the Kenya project is currently being implemented in only one county. This limits the ability to have a full understanding of the landscape across Kenya, which may limit the interest of policy makers and limit access to some necessary stakeholders in the poultry value chain in Kenya. A longer and more in-depth study would strengthen the relationships with local stakeholders needed for lasting change. Food safety reaches across many development sectors, including food security and nutrition. The development community should continue to increase its understanding of how food safety is inextricably linked to food and health outcomes and prioritize research and interventions in food safety that can increase awareness and systems change in the policy and the private sector. The FSIL ME and research partners have a large role to play in continuing to communicate this linkage and advocate for investment. To that end, investment in food safety awareness and in identifying interventions that can be feasibly implemented – and measuring their effectiveness – should be a continued focus of future programming, as has been identified by the FSIL ME and several research partners. For the remaining project period: Most projects have done an admirable job of catching up following the delays in project implementation caused by COVID-19. However, projects are still in danger of not completing their objectives by the end of their subaward periods. This will likely be particularly true for the publication of results. The FSIL ME and USAID may need to consider some targeted no-cost extensions to ensure that all projects can successfully meet their objectives. Contingencies should be established for costs related to publishing of research outside of the award period. 16 Localization strategy: Recommendations for USAID: As a cooperative partner, USAID has ample opportunity to increase the effectiveness of the Innovation Labs, including FSIL. In addition to supporting the research agenda described above, USAID can also support effective research for development programming in the following additional ways: MSI support: USAID should continue to invest in making international research more accessible to MSIs. This may include providing additional funding to support the administrative support mechanisms that universities require to implement USG-funded international research projects. Graduate Student support: Good research is critical to the mission of the Innovation Labs, and good research requires the mentoring of graduate student researchers by established principal investigators. Ideally, these graduate students would come from a broad talent pool, which includes U.S. students interested in international food systems, international graduate students currently enrolled in U.S. universities, international students seeking graduate research opportunities in the U.S., and domestic students in the countries of Innovation Lab projects. All of these students should have access to the capacity building opportunities that Innovation Lab research provides. Constraints on graduate student funding present a challenge for effective research implementation at U.S. universities. This is presumed to be an even greater challenge for MSIs than for larger institutions, where researchers are less likely to be able to fund graduate students out of other research funds. USAID should continue to work toward more open graduate student funding guidelines, with recognition of the limitation that some requirements are congressionally bound. Under USAID’s localization strategy, there is risk and opportunity for Innovation Labs. USAID should be aware of potential limitations with U.S. universities that may be hesitant to serve as subawardees or place major financial burdens and responsibilities on local institutions. USAID should engage in dialogue with university institutions on how to manage these risks so as to continue to leverage the expertise and commitment of U.S. universities while still supporting greater leadership in project countries. Conclusion In summary, the Food Safety Innovation has had an impressive initial three and half years of programming, in spite of the unforeseen challenges of the COVID-19 pandemic. The FSIL Management Entity has shown resolve and support for its research teams. Across the board, FSIL research partners spoke positively about the FSIL ME. Across the projects, research objectives were clear and included the engagement of policymakers and stakeholders across the value chain. Capacity building has spanned beyond the traditional role of training graduate students and providing workshop-style trainings (though it does this too) to efforts that support the long-term engagement of the stakeholders in the project countries. By prioritizing access to research opportunities and supporting the process by carrying more of the administrative burden, FSIL has been a leader in advancing MSIs in the global research space. There is a strong consensus amongst the researchers associated with FSIL that awareness is a primary barrier in food safety research and interventions. It therefore seems intuitive that future programming on food safety should focus its limited resources here and continue to make ground in the policy space, both in the U.S. and in the project countries, to advance the understanding that food safety is food security. Drs. Oliver and Worobo are extremely well-positioned to continue this investment through future funding of the Food Safety Innovation Lab. 17 FSIL Management Entity FSIL U.S.-based Principal Investigators Appendix A: List of Interviewees Dr. Haley Oliver Director Professor of Food Science Purdue University Dr. Randy Worobo Associate Director Professor of Food Microbiology Cornell University Julie Hancock Operations Specialist Purdue University Dr. Amanda Garris Communications Specialist Cornell University Allison Staley International Post Award Business Manager Purdue University Dr. Andrea Bersamin (Nigeria) Associate Professor of Nutrition University of Alaska, Fairbanks Dr. Aditya Khanal (Nepal) Associate Professor, Agribusiness, Agricultural Economics Tennessee State University Dr. Barbara Kowalcyk (Kenya) Assistant Professor of Food Safety and Public Health Director of the Center for Foodborne Illness Research and Prevention The Ohio State University Dr. Madan Dey (Bangladesh) Professor and Chair Department of Agriculture Texas State University Dr. Manpreet Singh (Senegal) Professor, Poultry Science 18 FSIL in-country Co-PIs FSIL U.S.-based Co-PIs USAID University of Georgia Dr. Martin Wiedmann (Global, COVID-19) Gellert Family Professor in Food Safety Cornell University Dr. Lyda Hok (Cambodia) Director, Soil Science Center of Excellence on Sustainable Agricultural Intensification and Nutrition Royal University of Agriculture, Cambodia Dr. Robert Onsare (Kenya) Head of Unit, Salmonella/Antimicrobial Resistance Surveillance Centre for Microbiology Research Kenya Medical Research Institute Dr. Ram Timilsina (Nepal) Assistant Professor, Agricultural Extension and Rural Sociology Agriculture and Forestry University, Nepal Dr. Md. Saidur Rahman (Bangladesh) Director, Institute of Agribusiness and Development Studies Bangladesh Agricultural University Dr. Kathy Colverson (Kenya, Gender) Associate Research Scientist, Animal Sciences University of Florida Dr. Claudia Gasner (Kenya) Research Assistant Scientist, Animal Sciences University of Florida Dr. Arie Havelaar (Kenya) Preeminent Professor of Global Food Safety and Zoonoses, Animal Sciences Institute for Sustainable Food Systems, Emerging Pathogens Institute University of Florida Dr. Ahmed Kablan Agreement Officer Representative Dr. Meera Chandra Activity Manager 19 Purdue University Purdue Sponsored Programs FSIL Technical Experts FSIL Advisory Committee Beth Siple Assistant Director of Financial Affairs Sponsored Program Services Purdue University Dr. Jacob Ricker-Gilbert Professor of Agricultural Economics Purdue University Dr. Paul Ebner Professor of Animal Sciences Purdue University Dr. Levon Esters Associate Dean of Diversity, Equity, Inclusion and Faculty Affairs, Polytechnic Institute Purdue University Dr. Kathryn Boor Dean of the Graduate School and Vice Provost for Graduate Education Cornell University Dr. Shibani Ghosh Associate Director, Feed the Future Innovation Lab for Food Systems for Nutrition Research Associate Professor, Friedman School of Nutrition Science and Policy Tufts University Howard Popoola Vice President, Corporate Food Technology and Regulatory Compliance The Kroger Co., GFSI Board Member *Select project interviews were recorded for accuracy. All recorded interviews were done with the verbal consent of the interviewees. 20 Appendix B: PI/Co-PI Interview Questions Interview questions were tailored the individual interviewee/project, but a standard set of questions was included for all PIs/Co-PIs. These questions are listed below: 1. Does the FSIL ME provide the level of support you need to successfully conduct your research? What do they do well and what could they improve? 2. Do your home/subawardee institution(s) provide the level of support you need to successfully conduct your research? What do they do well and what could they improve? 3. How was the RFA process? Was it transparent; were the research priorities clear? Were there areas for improvement? 4. What has been the biggest success and the biggest challenge in your FSIL project(s) to date? a. How did COVID-19 impact your project activities, and how did you adapt? 5. Are you on track to achieve your project objectives by the end of the award period? 6. What do you think will be most important research outcome from your project? 7. What do you think will be most important capacity building outcome? 8. Have you been able to impact policy? 9. How do you incorporate gender into your work? 10. What level of engagement do you have with the USAID Mission? 11. How would you structure/prioritize future work? 21